Re: Android camera rotation

[Сергей Митрофанов <goretz.m@xxxxxxxxx>]

Hi Ming

I know that old ticket, and I have some info from my experience for you here.

I'm developing a free video chat using your superb library. Really, it is awesome in comparison to undocumented linphone or non-sip solutions (laggy red5, proprietary adobe air or deprecated vitamio).

But pjsip for sure has an issue with the camera orientation at least in android platforms.

First of all we have a lot of devices to test on, so I'm talking with the real world results in my hands now.

For example if I use your sample app as is with the most recent svn version of pjsip on major variety of our devices the front camera preview and stream appears upside down in any orientation, while the back camera shows normal.

This is because for the most devices initial rotation value from android camera API is 90 for both front and back cameras. While other devices (tablets, TV - from 20% to 30%) can have other values: 0 or 90 or 270 for any of cameras.

As it is obvious from pjsip code, you don't use at all the actual camera rotation from android hardware.camera API.
Looks like you suggested "all back cameras are always rotated for 90 and all front cameras are always rotated for 270". And that why you map android device orientation with 90 offset to the library orientation in the test app.

But in fact this simple mapping is not always true and device can also have external plugged in cameras (in case of TVs).

So to make it work correctly I have to make strange crutches (And I done it before) to figure out the real initial rotation of device and fix the orientation before passing it to the library. And since there is no straight mapping between android cameras IDs and pjsip capture devices IDs it becomes very ugly and unreliable.

Speaking specifically that how I had solve this issue before to make it work as intended on all devices:

1. Get the list of android cameras from hardware.camera API

2. Get rotation for front and back cameras (only 2 values, just for front and back, since we have no straight mapping to pjsip devices here)

3. Tweak rotation values: rotationBack = 90 - cameraInfo.orientation and rotationFront = 270 - cameraInfo.orientation 

4. Get list of the pjsip devices

5. Create an Array of POJOs to keep the rotation values for pjsip devices

6. Assign the rotationBack to the pjsip devices that contain in their name the word "back"
7. Assign the rotationFront to the pjsip devices that contain in their name the word "front"

8. Remove non-camera devices (driver is not "Android" or direction is not "PJMEDIA_DIR_ENCODING")

And we have a list of pjsip camera IDs with their rotations values packed in POJOs

Then at the time of rotation those one used like this:

1. figure out the actualRotation = screenOrientation - cameraRotation (from camera POJO - either rotationBack or rotationFront)

2. Normalize the result value, so it will be between 0 and 360 (+ or - 360 to the value)

3. Make the same mapping as in your sample application (0 = 270DEG, 90 = NATUAL etc.)

As you see it is a long and unstable way to achieve this.

After tweaking the android_dev.c I now can use the camera rotation functionality without any extra crutches and wheels. And it works on any of android devices that is out there now.

The main issue in the current pjsip work with cameras - it does not respect the real initial rotation of camera devices.

If this is not a problem for other platforms it doesn't mean that there is no problem at all.

And if you want to keep behavior the same with other platforms to the prejudice of library usability on other platforms, then it appears sad for me...

Anyway I just pointed out the big annoying issue with current camera rotation calculations scheme on Android devices and I don't force you to use this solution for the main branch.

Maybe later you will find a nice solution that will work as simple as with my patch (without any excessive code-magic) across all platforms supported by your team. And I will be very glad when you will make it and will make a standing ovation for you guys!

But until this happen I will use this patch.

Maybe it will be useful for someone else, that why I publish it here...

Best Regards,

Sergey

вт, 5 апр. 2016 г. в 13:06, Ming <ming@xxxxxxxxx>:

Hi Сергей,

Unfortunately this will make the behavior different with other
platforms. And FYI, we just changed the spec of pjmedia_orient in
ticket #1880 (https://trac.pjsip.org/repos/ticket/1880) -->
https://trac.pjsip.org/repos/browser/pjproject/trunk/pjmedia/include/pjmedia/types.h#L197

Regards,
Ming

On Mon, Apr 4, 2016 at 7:42 PM, Сергей Митрофанов <goretz.m@xxxxxxxxx> wrote:
> Yesterday I have published a patch, but today rechecked and it produces
> incorrect rotations for back camera.
> Here is the correct one patch.
> It works without any additional rotation offset calculations in app and with
> it pjsip rotations are mapped to android screen orientations directly.
>
> Was in test app:
>
> wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> display = wm.getDefaultDisplay();
> rotation = display.getRotation();
> Log.d("CallActivity", "Device orientation changed: " + rotation);
> switch (rotation) {
>     case Surface.ROTATION_0:   // Portrait
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
>         break;
>     case Surface.ROTATION_90:  // Landscape, home button on the right
>         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
>         break;
>     case Surface.ROTATION_180:
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
>         break;
>     case Surface.ROTATION_270: // Landscape, home button on the left
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
>         break;
>     default:
>         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> }
>
> With this patch:
>
> wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> display = wm.getDefaultDisplay();
> rotation = display.getRotation();
> Log.d("CallActivity", "Device orientation changed: " + rotation);
> switch (rotation) {
>     case Surface.ROTATION_0:   // Portrait
>         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
>         break;
>     case Surface.ROTATION_90:  // Landscape, home button on the right
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
>         break;
>     case Surface.ROTATION_180:
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
>         break;
>     case Surface.ROTATION_270: // Landscape, home button on the left
>         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
>         break;
>     default:
>         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> }
>
> And this works correct for both facing and back cameras without any
> additional work in application.
>
>
> вс, 3 апр. 2016 г. в 17:25, Сергей Митрофанов <goretz.m@xxxxxxxxx>:
>>
>> Hi!
>>
>> I have found that pjsip does not respect initial camera rotation in
>> android devices.
>> Sure the android_dev.c itself extracts the rotation value from android
>> camera devices,
>> but for now this value is actually not used, resulting wrong capture video
>> orientation.
>> I made a simple patch (a little bit raw code style) that resolves this
>> bug.
>> It takes the camera rotation value in the game and make PJSUA2 easier to
>> use - developer need no more think about camera initial rotation and tweak
>> the rotation value before giving it to PJSUA2 itself.
>>
>> Here is the diff patch for the changes:
>>
>> From 1ea38ce9103f6fadd9812f635c83442994fd37ea Mon Sep 17 00:00:00 2001
>>
>> From: Sergey Mitrofanov <GOretZ.M@xxxxxxxxx>
>>
>> Date: Sun, 3 Apr 2016 17:21:48 +0300
>>
>> Subject: [PATCH] Fixing to respect initial camera rotation in android
>> devices.
>>
>>
>> ---
>>
>>  pjmedia/src/pjmedia-videodev/android_dev.c | 36
>> ++++++++++++++++++++++++++++++
>>
>>  1 file changed, 36 insertions(+)
>>
>>
>> diff --git a/pjmedia/src/pjmedia-videodev/android_dev.c
>> b/pjmedia/src/pjmedia-videodev/android_dev.c
>>
>> index 1da7261..fd6e61d 100644
>>
>> --- a/pjmedia/src/pjmedia-videodev/android_dev.c
>>
>> +++ b/pjmedia/src/pjmedia-videodev/android_dev.c
>>
>> @@ -72,6 +72,7 @@ typedef struct and_dev_info
>>
>>      pjmedia_vid_dev_info info; /**< Base info         */
>>
>>      unsigned dev_idx; /**< Original dev ID   */
>>
>>      pj_bool_t facing; /**< Front/back camera?*/
>>
>> +    unsigned rotation; /**< initial camera rotation*/
>>
>>      unsigned sup_size_cnt; /**< # of supp'd size  */
>>
>>      pjmedia_rect_size *sup_size; /**< Supported size    */
>>
>>      unsigned sup_fps_cnt; /**< # of supp'd FPS   */
>>
>> @@ -532,6 +533,8 @@ static pj_status_t
>> and_factory_refresh(pjmedia_vid_dev_factory *ff)
>>
>>   } else {
>>
>>       pj_ansi_strncpy(vdi->name, "Front camera", sizeof(vdi->name));
>>
>>   }
>>
>> + adi->rotation = (*jni_env)->GetIntField(jni_env, jdev_info,
>>
>> + jobjs.cam_info.f_orient);
>>
>>
>>
>>   /* Get supported sizes */
>>
>>   jtmp = (*jni_env)->GetObjectField(jni_env, jdev_info,
>>
>> @@ -1001,6 +1004,39 @@ static pj_status_t
>> and_stream_set_cap(pjmedia_vid_dev_stream *s,
>>
>>   else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>>
>>       eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>>
>>       }
>>
>> +     /* Normalize the orientation for rotated camera */
>>
>> +     switch(adi->rotation){
>>
>> + case 90:
>>
>> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>>
>> + break;
>>
>> + case 180:
>>
>> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>>
>> + break;
>>
>> + case 270:
>>
>> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>>
>> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>>
>> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>>
>> + break;
>>
>> +     }
>>
>>       pjmedia_vid_dev_conv_set_rotation(&strm->conv, eff_ori);
>>
>>
>>
>>       PJ_LOG(4, (THIS_FILE, "Video capture orientation set to %d",
>>
>> --
>>
>> 2.6.4 (Apple Git-63)
>>
>>
>
> _______________________________________________
> Visit our blog: http://blog.pjsip.org
>
> pjsip mailing list
> pjsip@xxxxxxxxxxxxxxx
> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>

_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org