Re: Android camera rotation

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Ming,

You are welcome.
I agree, you have to do more testings by yourself to see what is going on.
Btw, I suppose I should point out my building system details.
So I build the pjsip in MacBook Pro (Retina 13", early 2015)
With OS X El Capitan 10.11.4 (15E65) on board.
I also use libyuv-android and openh264-1.0.0 according to the video guide tickets
Also I use the most recent opus snapshot and stripped version of webrtc EC (from ticket about EC).
I'm building for armeabi-v7a only for now
And this is my configure options

TARGET_ABI=armeabi-v7a APP_PLATFORM=android-16 ./configure-android --use-ndk-cflags --disable-speex-aec --disable-g711-codec --disable-l16-codec --disable-gsm-codec --disable-g722-codec --disable-g7221-codec --disable-speex-codec --disable-ilbc-codec --disable-ffmpeg --disable-opencore-amr --disable-silk --with-openh264=/Users/sergej/Projects/openh264-1.0.0/android --with-libyuv=/Users/sergej/Projects/libyuv-android/jni --with-opus=/Users/sergej/Projects/opus/build --disable-webrtc

Ask freely if you need some other details about my building system.

Regards,
Sergey

ср, 6 апр. 2016 г. в 6:04, Ming <ming@xxxxxxxxx>:
Hi Sergey,

Thanks for your valuable insight and feedback.
Let us do more testing first with this.

Regards,
Ming

On Tue, Apr 5, 2016 at 8:14 PM, Сергей Митрофанов <goretz.m@xxxxxxxxx> wrote:
> Hi Ming
>
> I know that old ticket, and I have some info from my experience for you
> here.
> I'm developing a free video chat using your superb library. Really, it is
> awesome in comparison to undocumented linphone or non-sip solutions (laggy
> red5, proprietary adobe air or deprecated vitamio).
> But pjsip for sure has an issue with the camera orientation at least in
> android platforms.
> First of all we have a lot of devices to test on, so I'm talking with the
> real world results in my hands now.
>
> For example if I use your sample app as is with the most recent svn version
> of pjsip on major variety of our devices the front camera preview and stream
> appears upside down in any orientation, while the back camera shows normal.
>
> This is because for the most devices initial rotation value from android
> camera API is 90 for both front and back cameras. While other devices
> (tablets, TV - from 20% to 30%) can have other values: 0 or 90 or 270 for
> any of cameras.
>
> As it is obvious from pjsip code, you don't use at all the actual camera
> rotation from android hardware.camera API.
> Looks like you suggested "all back cameras are always rotated for 90 and all
> front cameras are always rotated for 270". And that why you map android
> device orientation with 90 offset to the library orientation in the test
> app.
> But in fact this simple mapping is not always true and device can also have
> external plugged in cameras (in case of TVs).
>
> So to make it work correctly I have to make strange crutches (And I done it
> before) to figure out the real initial rotation of device and fix the
> orientation before passing it to the library. And since there is no straight
> mapping between android cameras IDs and pjsip capture devices IDs it becomes
> very ugly and unreliable.
> Speaking specifically that how I had solve this issue before to make it work
> as intended on all devices:
> 1. Get the list of android cameras from hardware.camera API
> 2. Get rotation for front and back cameras (only 2 values, just for front
> and back, since we have no straight mapping to pjsip devices here)
> 3. Tweak rotation values: rotationBack = 90 - cameraInfo.orientation and
> rotationFront = 270 - cameraInfo.orientation
> 4. Get list of the pjsip devices
> 5. Create an Array of POJOs to keep the rotation values for pjsip devices
> 6. Assign the rotationBack to the pjsip devices that contain in their name
> the word "back"
> 7. Assign the rotationFront to the pjsip devices that contain in their name
> the word "front"
> 8. Remove non-camera devices (driver is not "Android" or direction is not
> "PJMEDIA_DIR_ENCODING")
> And we have a list of pjsip camera IDs with their rotations values packed in
> POJOs
> Then at the time of rotation those one used like this:
> 1. figure out the actualRotation = screenOrientation - cameraRotation (from
> camera POJO - either rotationBack or rotationFront)
> 2. Normalize the result value, so it will be between 0 and 360 (+ or - 360
> to the value)
> 3. Make the same mapping as in your sample application (0 = 270DEG, 90 =
> NATUAL etc.)
> As you see it is a long and unstable way to achieve this.
> After tweaking the android_dev.c I now can use the camera rotation
> functionality without any extra crutches and wheels. And it works on any of
> android devices that is out there now.
>
> The main issue in the current pjsip work with cameras - it does not respect
> the real initial rotation of camera devices.
> If this is not a problem for other platforms it doesn't mean that there is
> no problem at all.
> And if you want to keep behavior the same with other platforms to the
> prejudice of library usability on other platforms, then it appears sad for
> me...
>
> Anyway I just pointed out the big annoying issue with current camera
> rotation calculations scheme on Android devices and I don't force you to use
> this solution for the main branch.
> Maybe later you will find a nice solution that will work as simple as with
> my patch (without any excessive code-magic) across all platforms supported
> by your team. And I will be very glad when you will make it and will make a
> standing ovation for you guys!
> But until this happen I will use this patch.
> Maybe it will be useful for someone else, that why I publish it here...
>
> Best Regards,
> Sergey
>
> вт, 5 апр. 2016 г. в 13:06, Ming <ming@xxxxxxxxx>:
>>
>> Hi Сергей,
>>
>> Unfortunately this will make the behavior different with other
>> platforms. And FYI, we just changed the spec of pjmedia_orient in
>> ticket #1880 (https://trac.pjsip.org/repos/ticket/1880) -->
>>
>> https://trac.pjsip.org/repos/browser/pjproject/trunk/pjmedia/include/pjmedia/types.h#L197
>>
>> Regards,
>> Ming
>>
>> On Mon, Apr 4, 2016 at 7:42 PM, Сергей Митрофанов <goretz.m@xxxxxxxxx>
>> wrote:
>> > Yesterday I have published a patch, but today rechecked and it produces
>> > incorrect rotations for back camera.
>> > Here is the correct one patch.
>> > It works without any additional rotation offset calculations in app and
>> > with
>> > it pjsip rotations are mapped to android screen orientations directly.
>> >
>> > Was in test app:
>> >
>> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
>> > display = wm.getDefaultDisplay();
>> > rotation = display.getRotation();
>> > Log.d("CallActivity", "Device orientation changed: " + rotation);
>> > switch (rotation) {
>> >     case Surface.ROTATION_0:   // Portrait
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
>> >         break;
>> >     case Surface.ROTATION_90:  // Landscape, home button on the right
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
>> >         break;
>> >     case Surface.ROTATION_180:
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
>> >         break;
>> >     case Surface.ROTATION_270: // Landscape, home button on the left
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
>> >         break;
>> >     default:
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
>> > }
>> >
>> > With this patch:
>> >
>> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
>> > display = wm.getDefaultDisplay();
>> > rotation = display.getRotation();
>> > Log.d("CallActivity", "Device orientation changed: " + rotation);
>> > switch (rotation) {
>> >     case Surface.ROTATION_0:   // Portrait
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
>> >         break;
>> >     case Surface.ROTATION_90:  // Landscape, home button on the right
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
>> >         break;
>> >     case Surface.ROTATION_180:
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
>> >         break;
>> >     case Surface.ROTATION_270: // Landscape, home button on the left
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
>> >         break;
>> >     default:
>> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
>> > }
>> >
>> > And this works correct for both facing and back cameras without any
>> > additional work in application.
>> >
>> >
>> > вс, 3 апр. 2016 г. в 17:25, Сергей Митрофанов <goretz.m@xxxxxxxxx>:
>> >>
>> >> Hi!
>> >>
>> >> I have found that pjsip does not respect initial camera rotation in
>> >> android devices.
>> >> Sure the android_dev.c itself extracts the rotation value from android
>> >> camera devices,
>> >> but for now this value is actually not used, resulting wrong capture
>> >> video
>> >> orientation.
>> >> I made a simple patch (a little bit raw code style) that resolves this
>> >> bug.
>> >> It takes the camera rotation value in the game and make PJSUA2 easier
>> >> to
>> >> use - developer need no more think about camera initial rotation and
>> >> tweak
>> >> the rotation value before giving it to PJSUA2 itself.
>> >>
>> >> Here is the diff patch for the changes:
>> >>
>> >> From 1ea38ce9103f6fadd9812f635c83442994fd37ea Mon Sep 17 00:00:00 2001
>> >>
>> >> From: Sergey Mitrofanov <GOretZ.M@xxxxxxxxx>
>> >>
>> >> Date: Sun, 3 Apr 2016 17:21:48 +0300
>> >>
>> >> Subject: [PATCH] Fixing to respect initial camera rotation in android
>> >> devices.
>> >>
>> >>
>> >> ---
>> >>
>> >>  pjmedia/src/pjmedia-videodev/android_dev.c | 36
>> >> ++++++++++++++++++++++++++++++
>> >>
>> >>  1 file changed, 36 insertions(+)
>> >>
>> >>
>> >> diff --git a/pjmedia/src/pjmedia-videodev/android_dev.c
>> >> b/pjmedia/src/pjmedia-videodev/android_dev.c
>> >>
>> >> index 1da7261..fd6e61d 100644
>> >>
>> >> --- a/pjmedia/src/pjmedia-videodev/android_dev.c
>> >>
>> >> +++ b/pjmedia/src/pjmedia-videodev/android_dev.c
>> >>
>> >> @@ -72,6 +72,7 @@ typedef struct and_dev_info
>> >>
>> >>      pjmedia_vid_dev_info info; /**< Base info         */
>> >>
>> >>      unsigned dev_idx; /**< Original dev ID   */
>> >>
>> >>      pj_bool_t facing; /**< Front/back camera?*/
>> >>
>> >> +    unsigned rotation; /**< initial camera rotation*/
>> >>
>> >>      unsigned sup_size_cnt; /**< # of supp'd size  */
>> >>
>> >>      pjmedia_rect_size *sup_size; /**< Supported size    */
>> >>
>> >>      unsigned sup_fps_cnt; /**< # of supp'd FPS   */
>> >>
>> >> @@ -532,6 +533,8 @@ static pj_status_t
>> >> and_factory_refresh(pjmedia_vid_dev_factory *ff)
>> >>
>> >>   } else {
>> >>
>> >>       pj_ansi_strncpy(vdi->name, "Front camera", sizeof(vdi->name));
>> >>
>> >>   }
>> >>
>> >> + adi->rotation = (*jni_env)->GetIntField(jni_env, jdev_info,
>> >>
>> >> + jobjs.cam_info.f_orient);
>> >>
>> >>
>> >>
>> >>   /* Get supported sizes */
>> >>
>> >>   jtmp = (*jni_env)->GetObjectField(jni_env, jdev_info,
>> >>
>> >> @@ -1001,6 +1004,39 @@ static pj_status_t
>> >> and_stream_set_cap(pjmedia_vid_dev_stream *s,
>> >>
>> >>   else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>> >>
>> >>       eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>> >>
>> >>       }
>> >>
>> >> +     /* Normalize the orientation for rotated camera */
>> >>
>> >> +     switch(adi->rotation){
>> >>
>> >> + case 90:
>> >>
>> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>> >>
>> >> + break;
>> >>
>> >> + case 180:
>> >>
>> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>> >>
>> >> + break;
>> >>
>> >> + case 270:
>> >>
>> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
>> >>
>> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
>> >>
>> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
>> >>
>> >> + break;
>> >>
>> >> +     }
>> >>
>> >>       pjmedia_vid_dev_conv_set_rotation(&strm->conv, eff_ori);
>> >>
>> >>
>> >>
>> >>       PJ_LOG(4, (THIS_FILE, "Video capture orientation set to %d",
>> >>
>> >> --
>> >>
>> >> 2.6.4 (Apple Git-63)
>> >>
>> >>
>> >
>> > _______________________________________________
>> > Visit our blog: http://blog.pjsip.org
>> >
>> > pjsip mailing list
>> > pjsip@xxxxxxxxxxxxxxx
>> > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>> >
>>
>> _______________________________________________
>> Visit our blog: http://blog.pjsip.org
>>
>> pjsip mailing list
>> pjsip@xxxxxxxxxxxxxxx
>> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>
>
> _______________________________________________
> Visit our blog: http://blog.pjsip.org
>
> pjsip mailing list
> pjsip@xxxxxxxxxxxxxxx
> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>

_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
[Index of Archives]     [Asterisk Users]     [Asterisk App Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [Linux API]
  Powered by Linux