fftools/ffmpeg_filter: Fix autorotation

In case of an orthogonal transformation av_display_rotation_get()
returns the (anticlockwise) degree that the unit vector in x-direction
gets rotated by; get_rotation in cmdutils.c makes a clockwise degree
out of this. So if one inserts a transpose filter corresponding to
this degree, then the x-vector gets mapped correctly and there are
two possibilities for image of the y-vector, namely the two unit
vectors orthogonal to the image of the x-vector.

E.g. if the x-vector gets rotated by 90° clockwise, then the two
possibilities for the y-vector are the unit vector in x direction
or its opposite. The latter case is a simple 90° rotation for both
vectors* whereas the former is a simple 90° clockwise rotation followed
by a horizontal flip. These two cases can be distinguished by looking
at the x-coordinate of the image of the y-vector, i.e. by looking
at displaymatrix[3]. Similarly for the case of a 270° clockwise
rotation.

These two cases were previously wrong (they were made to match
wrongly parsed exif rotation tag values).

*: For display matrices, the y-axis points downward.

Signed-off-by: Andreas Rheinhardt <[email protected]>

fftools/ffmpeg_filter.c

@@ -761,12 +761,12 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter,
         theta = get_rotation(displaymatrix);
 
         if (fabs(theta - 90) < 1.0) {
+            ret = insert_filter(&last_filter, &pad_idx, "transpose", "clock");
             if (displaymatrix[3] > 0) {
                 ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL);
                 if (ret < 0)
                     return ret;
             }
-            ret = insert_filter(&last_filter, &pad_idx, "transpose", "clock");
         } else if (fabs(theta - 180) < 1.0) {
             if (displaymatrix[0] < 0) {
                 ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL);
@@ -777,12 +777,12 @@ static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter,
                 ret = insert_filter(&last_filter, &pad_idx, "vflip", NULL);
             }
         } else if (fabs(theta - 270) < 1.0) {
+            ret = insert_filter(&last_filter, &pad_idx, "transpose", "cclock");
             if (displaymatrix[3] < 0) {
                 ret = insert_filter(&last_filter, &pad_idx, "hflip", NULL);
                 if (ret < 0)
                     return ret;
             }
-            ret = insert_filter(&last_filter, &pad_idx, "transpose", "cclock");
         } else if (fabs(theta) > 1.0) {
             char rotate_buf[64];
             snprintf(rotate_buf, sizeof(rotate_buf), "%f*PI/180", theta);