Hi,
I can do videocall with pjsua, but video latency is unacceptable and
frame rate is really too slow.
Since gstreamer supports video acceleration while ffmpeg does not but
uses libx264 that causes an intolerable cpu load, I've made a simple
attempt to substitute pjsip video streamer with gstreamer pipeline.
I first launced:
$ raspivid -t 999999 -w 384 -h 256 -fps 15 -hf -b 1000000 -o - |
gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96
! gdppay ! udpsink host=192.168.1.99 port=9078
it shows:
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
[....]
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0: timestamp = 3672856811
/GstPipeline:pipeline0/GstRtpH264Pay:rtph264pay0: seqnum = 15147
New clock: GstSystemClock
then, in the same ARM platform, with another ssh terminal, I launched:
$ ./pjsua --video
Since camera driver is already in use by gstreamer pipeline, pjsua is
forced to avoid using it:
pjsua_vid.c ..Initializing video subsystem..
v4l2_dev.c ...Video4Linux2 has 1 devices
colorbar_dev.c ...Colorbar video src initialized with 1 device(s):
colorbar_dev.c ... 0: Colorbar generator
this is the log while videocall is starting: note that now pjsua detect
nmal video camera driver and tries to start v4l2 video stream from it,
but it clearly fails, because the driver is already in use by raspivid.
m=video 9078 RTP/AVP 97
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=428014
pjsua_app.c ....Call 0 state changed to CONNECTING
pjsua_media.c ....Call 0: updating media..
pjsua_aud.c .....Audio channel update..
strm0x63ed7c ......VAD temporarily disabled
strm0x63ed7c ......Encoder stream started
strm0x63ed7c ......Decoder stream started
pjsua_media.c .....Audio updated, stream #0: speex (sendrecv)
pjsua_vid.c .....Video channel update..
vstenc0x65112c ......Encoder stream started
vstdec0x65112c ......Decoder stream started
pjsua_vid.c ......Setting up RX..
pjsua_vid.c .......Creating video window: type=stream, cap_id=-1, rend_id=2
vid_port.c ........Opening device SDL renderer [SDL] for render:
format=I420, size=768x512 @22:1 fps
vid_port.c ........Device SDL renderer [SDL] opened: format=I420,
size=768x512 @22:1 fps
pjsua_vid.c ........stream window id 0 created for cap_dev=-1 rend_dev=2
pjsua_vid.c ........Window 0 created
sdl_dev.c .......Starting sdl video stream
pjsua_vid.c ......Setting up TX..
pjsua_vid.c .......Creating video window: type=preview, cap_id=0, rend_id=2
vid_port.c ........Opening device mmal service 15.1 [v4l2] for capture:
format=YUY2, size=384x256 @15:1 fps
vid_port.c ........Device mmal service 15.1 [v4l2] opened: format=YUY2,
size=384x256 @15:1 fps
vid_port.c ........Opening device SDL renderer [SDL] for render:
format=I420, size=384x256 @15:1 fps
vid_port.c ........Device SDL renderer [SDL] opened: format=I420,
size=384x256 @15:1 fps
pjsua_vid.c ........preview window id 1 created for cap_dev=0 rend_dev=2
pjsua_vid.c ........Window 1 created
v4l2_dev.c .......Starting v4l2 video stream mmal service 15.1
libv4l2: error turning on stream: Invalid argument
libv4l2: error dequeuing buf: Invalid argument
libv4l2: error dequeuing buf: Invalid argument
libv4l2: error dequeuing buf: Invalid argument
v4l2_dev.c .......Stopping v4l2 video stream mmal service 15.1
pjsua_media.c .....pjsua_vid_channel_update() failed for call_id 0
media 1: Invalid argument
pjsua_app.c ....Call 0 media 0 [type=audio], status is Active
The situation at this point is:
- a frozen window on remote user agent (Linphone) that means that it is
not able to catch the H264 videostream from gstreamer pipeline, although
video streaming matches negotiated datas (H264, 384x256, video RTP
port=9078, )
- sdl vindow continues to be rendered to full video, instead to be
384x256 in size as it should be: this could mean that
on_call_media_event() is not called (when connection closes it writes:
"TX last update:never")
When I close communication:
[DISCONNCTD] To: sip:toto at 192.168.1.107;tag=29298
Call time: 00h:01m:00s, 1st res in 65 ms, conn in 4868ms
#0 audio speex @16kHz, sendrecv, peer=192.168.1.107:7078
SRTP status: Not active Crypto-suite:
RX pt=98, last update:00h:00m:01.212s ago
total 3.0Kpkt 211.1KB (331.8KB +IP hdr) @avg=27.7Kbps/43.6Kbps
pkt loss=1 (0.0%), discrd=1 (0.0%), dup=0 (0.0%), reord=0 (0.0%)
(msec) min avg max last dev
loss period: 20.000 20.000 20.000 20.000 0.000
jitter : 1.187 21.201 33.437 25.125 1.862
TX pt=98, ptime=20, last update:00h:00m:00.300s ago
total 2.1Kpkt 51.6KB (136.1KB +IP hdr) @avg=6.7Kbps/17.9Kbps
pkt loss=0 (0.0%), dup=0 (0.0%), reorder=0 (0.0%)
(msec) min avg max last dev
loss period: 0.000 0.000 0.000 0.000 0.000
jitter : 11.125 14.729 20.187 13.562 2.299
RTT msec : 9.033 32.027 66.000 37.704 18.107
#1 video H264, inactive, peer=192.168.1.107:9078
SRTP status: Not active Crypto-suite:
RX last update:00h:00m:59.270s ago
total 549pkt 500.5KB (522.5KB +IP hdr) @avg=66.1Kbps/69.0Kbps
pkt loss=0 (0.0%), discrd=1 (0.2%), dup=0 (0.0%), reord=0 (0.0%)
(msec) min avg max last dev
loss period: 0.000 0.000 0.000 0.000 0.000
jitter : 11.766 19.658 38.666 15.688 5.035
TX last update:never
total 0pkt 0B (0B +IP hdr) @avg=0bps/0bps
pkt loss=0 (0.0%), dup=0 (0.0%), reorder=0 (0.0%)
(msec) min avg max last dev
loss period: 0.000 0.000 0.000 0.000 0.000
jitter : 0.000 0.000 0.000 0.000 0.000
RTT msec : 0.000 0.000 0.000 0.000 0.000
Is there anyone that could tell me why remote user agent is not able to
show gstreamer H264 video stream?
Thanks in advance
Best regards
Silvio
