everything i learned about streaming video in linux

-nemo

demo

let's talk about X11

+-------------------------+
|    input devices        |
|  keyboard | mouse       |
+-------^---+-^-----------+
        |     |
        |     |                 +------------------------+
      +-v-----v-------+-------->+                        |
      |               |         |                        |
      |   X Server    +-------->+                        |
      |               |         |                        |
      +--+---------+--+-------->+      Screen            |
         ^         ^            |                        |
         |         |            |                        |
         |         |            +------------------------+
         v         v
 +-------+---+  +--+-------+
 | browser   |  |          |
 | X client  |  | xterm    |
 |           |  |          |
 +-----------+  +----------+

x11 - displays and screens

  • You "select" a display with the :$DISPLAY:SCREEN syntax.

  • So the first screen on your main display server would be :0.0

  • You can set the display an application launches with DISPLAY environment variable

export DISPLAY=:0.1 leafpad

Xvfb

Xvfb or X virtual framebuffer is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in virtual memory without showing any screen output.

However, no output is shown. This virtual server does not require the computer it is running on to have any kind of graphics adapter, a screen or any input device.

xwd

xwd - dump an image of an X window

xwd -root image.xwd
convert image.xwd image.png

ffmpeg

can "grab" X11 windows

ffmpeg -f x11grab -framerate 10 -i :1.0 \
  -vaapi_device /dev/dri/renderD128 \
  -vf 'format=nv12,hwupload,scale_vaapi=w=854:h=480' \
  -c:v h264_vaapi -qp:v 26 -bf 0 -tune zerolatency -f mpegts \
  udp://localhost:12345

video devices

  • /dev/video*
v4l2-ctl  --device /dev/video2  --all
Driver Info:
  Driver name      : uvcvideo
  Card type        : HD Pro Webcam C920
  Bus info         : usb-0000:00:14.0-4
  Driver version   : 5.7.9
  Capabilities     : 0x84a00001
    Video Capture
    Metadata Capture
    Streaming
    Extended Pix Format
    Device Capabilities
  Device Caps      : 0x04a00000
    Metadata Capture
    Streaming
    Extended Pix Format
Priority: 2
Format Metadata Capture:
  Sample Format   : 'UVCH' (UVC Payload Header Metadata)
  Buffer Size     : 1024

v4l2loopback

v4l2loopback - a kernel module to create V4L2 loopback devices

this module allows you to create "virtual video devices" normal (v4l2) applications will read these devices as if they were ordinary video devices, but the video will not be read from e.g. a capture card but instead it is generated by another application. this allows you for instance to apply some nifty video effects on your Skype video... it also allows some more serious things (e.g. I've been using it to add streaming capabilities to an application by the means of hooking GStreamer into the loopback devices).

create new webcams!

modprobe v4l2loopback devices=4

/dev/video3,4,5,6,7

ffmpeg can "write" to v4l

ffmpeg -f x11grab \
       -framerate 4 \
       -s 854x480 \
       -i :1.0 \
       -pix_fmt yuv420p \
       -f v4l2 /dev/video3

ffmpeg -pix_fmts

aside: MJPEG over HTTP

ffmpeg -i ... -f mjpg -o file.jpg

Content-Type: multipart/x-mixed-replace; boundary="JPGBoundary"

--JPGBoundary
Content-Type: image/jpeg

<jpeg bytes>
--JPGBoundary
Content-Type: image/jpeg

<jpeg bytes>
--JPGBoundary
... and so on ...
--JPGBoundary--

MJPEG streaming is simple

  • mjpeg-server (uses ffmpeg -f mjpg)
  • ustreamer (only supports /dev/video*)
+---------------------+        +---------------------------+
|                     |        |                           |
|     Xvfb Virtual    +------> |      ffmpeg -f x11grab    |
+---------+-----------+        |     -o /de^/video3        |
          ^                    +---------------------------+
   +------+----------+                     |
   |  X Server       +<---+      +---------+-----------+
   +---+-------------+    |      |  /dev/video3        |
       ^                  |      |                     |-???->USTREAMER->Viewers ???
   +---+---+              |      +-------+-------------+
   | baba  |              |              |
   | is    |        +-----+---------+    |   +-----------+
   | you   |        |               |    +-->+  Zoom     |
   |       |        |  xdotool      |        |           |
   +-------+        +---------------+        +-----------+

security

YOLO!

Pitch: RFCs We Love!