Audio/Video mixers

Índex

General

Brave (BBC)

  • Instal·lació / Installation
    • Dependències / Dependencies
      • Mageia 7
        • urpmi git python3-virtualenv
        • urpmi libgirepository-devel gobject-introspection python3-cairo-devel
        • urpmi gstreamer1.0-nice gstreamer1.0-x264 gstreamer1.0-rtmp gstreamer1.0-srtp gstreamer1.0-vp8 lib64gstwebrtc-gir1.0
      • CentOS 7
        • yum install git ...
        • ...
    • cd ~/src
    • git clone https://github.com/bbc/brave.git
    • cd brave
    • install and activate python 3 virtualenv
    • source env/bin/activate
    • pip install --upgrade pip
    • pip install pyyml pygobject uvloop sanic psutil
  • Ús / Usage
  • REST API

    • types list create update delete
      general
      • # get all elements
        curl -X GET http://127.0.0.1:5000/api/all
      • # restart
        curl -X POST -d '{"config":"original"}'  http://localhost:5000/api/restart



      inputs
      • uri
      • image
      • TCP Client
      • test_video
      • test_audio
      • # get all inputs curl -X GET http://localhost:5000/api/inputs
      • # create an input with a test video
        curl -X PUT -d '{"type": "test_video"}' http://localhost:5000/api/inputs
      • # pause input1
        curl -X POST -d '{"state": "PAUSED"}' http://localhost:5000/api/inputs/1
      • # modify pattern in input1
        curl -X POST -d '{"pattern": 18}' http://localhost:5000/api/inputs/1
      • # play input1
        curl -X POST -d '{"state": "PLAYING"}' http://localhost:5000/api/inputs/1
      • # resize input3 curl -X POST -d '{"width": 100, "height": 100}' http://localhost:5000/api/inputs/3

      overlays
      • # get all overlays curl -X GET http://localhost:5000/api/overlays
      • # create an overlay (text) over mixer1
        curl -X PUT -d '{"type": "text", "source": "mixer1"}' http://localhost:5000/api/overlays
      • # create an overlay (clock) over input1
        curl -X PUT -d '{"type": "text", "source": "input1", "text": "camera 1"}' http://localhost:5000/api/overlays
      • # create an overlay (effect) over input2 and make it visible curl -X PUT -d '{"type": "effect", "source": "input2", "effect_name": "agingtv", "visible": true}' http://localhost:5000/api/overlays
      • # make overlay visible
        curl -X POST -d '{"visible": true}' http://localhost:5000/api/overlays/1
      • # make overlay invisible
        curl -X POST -d '{"visible": false}' http://localhost:5000/api/overlays/1

      mixers
      • # get all mixers
        curl -X GET
        http://localhost:5000/api/mixers

      • # modify pattern in mixer1
        curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1
      • # cut (switch) to input1
        curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source
      • # overlay to input2 (input1 is not removed from mixer)
        curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source

      outputs
      • rtmp
      • tcp
      • file
      • image
      • webrtc
      • kvs
      • local
      • # get all outputs
        curl -X GET
        http://localhost:5000/api/outputs
      • # create a webrtc output curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs



    • Example
      • # create a webrtc output
        curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs
        # modify pattern (snow) in mixer1
        curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1
        # create an input with a test video (colour bars)
        curl -X PUT -d '{"type": "test_video", "pattern": 0}' http://localhost:5000/api/inputs
        # create an input with a test video (balls)
        curl -X PUT -d '{"type": "test_video", "pattern": 18}' http://localhost:5000/api/inputs

        # cut (switch) to input1
        curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source
        # cut (switch) to input2
        curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/cut_to_source

        # create an image input curl -X PUT -d '{"type": "image", "uri": "file:///tmp/logo.png"}' http://localhost:5000/api/inputs # put image as a DOG
        curl -X POST -d '{"uid": "input3", "zorder": 100}' http://localhost:5000/api/mixers/1/overlay_source

        # select (overlay) input1
        curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/overlay_source
        # select (overlay) input2
        curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source

        # restart
        curl -X POST -d '{"config":"original"}'  http://localhost:5000/api/restart
        ...
    • ...

Snowmix

  • Snowmix - Live Video Mixer
  • Instal·lació / Installation
    • Dependencies
      • CentOS
        • yum install nmap-ncat bc dejavu-lgc-sans-fonts
        • all other dependencies are automatically managed by bootstrap
      • Mageia
        • urpmi lib64SDL-devel lib64png-devel lib64cairo-devel lib64pango1.0-devel lib64tcl-devel bc
        • urpmi xterm bwidget
        • urpmi netcat-traditional
    • Get source
      • curl -o Snowmix-0.5.1.tar.gz -L "http://downloads.sourceforge.net/project/snowmix/Snowmix-0.5.1.tar.gz?r=&use_mirror=netix"
      • tar xzf Snowmix-0.5.1.tar.gz
      • Mageia (to avoid non-closing connection for nc)
        • sed -i 's/debian )/debian | mageia )/g' scripts/snowmix-settings
      • CentOS (not critical)
        • sed -i 's/fedora |/fedora | centos |/g' scripts/snowmix-settings
    • Install
      • cd Snowmix-0.5.1
      • ./bootstrap
      • or use non interactive script bootstrap_ni.sh
        • #!/usr/bin/expect

          spawn ./bootstrap
          expect "your password to sudo (Y/N)?"
          sleep 2
          send "y\r"

          expect "Press return to execute bootstrapd/bootstrap-centos"
          sleep 2
          send "\r"

          expect "update your Fedora installation (Y/N) ?"
          sleep 2
          send "n\r"

          expect "Update and install (y=yes, n=no, q=quit) ?"
          sleep 2
          send "y\r"

          expect "Press return to continue"
          sleep 2
          send "\r"
      • make
      • sudo make install
  • Additional tools: not dependencies, but used by demo script
  • Usage
    • echo "export SNOWMIX=/usr/local/lib/Snowmix-0.5.1" >>~/.bashrc
    • demo
      • snowmix_demo
    • complete minimal examples
      • using snowmix/ini/minimal-1feed.ini
        • start snowmix:
          • snowmix ini/minimal-1feed.ini
        • check status
          • echo 'system info' | nc 127.0.0.1 9999
        • show output
          • scripts/av_output2screen
        • send file to feed 1
          • ...
        • show inputs
        • ...
      • snowmix/minimal.ini
        • # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved
          # verbose
          require version 0.5.0
          system control port 9999
          system geometry 640 360 BGRA
          system frame rate 25
          system socket /tmp/mixer1
      • snowmix ...
    • regular usage:
      • Snowmix Libraries
      • start snowmix (example ini files at ini dir)
        • snowmix ini/minimal-1feed.ini
        • snowmix ini/demo-scenes
        • snowmix ini/my.ini
          • my.ini
            • # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved
              # verbose
              require version 0.5.0

              # include basis setup
              include ini/basis.ini

              DigitalClockInit
              ClockInit

              # include audio setup
              include ini/audio
        • minimal_scenes.ini
          • ...
      • output script (using gstreamer)
        • scripts/av_output2screen
        • Problemes / Problems
          • no audio
            • Solució / Solution
              • av_output2screen
                •           autovideosink \
                            fdsrc fd=0 do-timestamp=true ! \
                            $AUDIOFORMAT ! \
      • input script (using gstreamer)
        • scripts/input2feed <video_feed_id> [<audio_feed_id>] ...
        • get gstreamer-related variables (called from input2feed)
          • scripts/gstreamer-settings

          • output variables
            snowmix
            • SNOWMIXVIDEOSRC=shmsrc
            • SNOWMIXVIDEOSINK=shmsink
            filters
            • VIDEOCONVERT=videoconvert
            caps
            • VIDEO=video/x-raw
            • VIDEOBGRA=video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive
            • VIDEOI420=video/x-raw,format=I420,pixel-aspect-ratio=1/1,interlaced-mode=progressive
            • AUDIO=audio/x-raw
            • AUDIOS16LE=audio/x-raw,format=S16LE,layout=interleaved
        • get snowmix-related variables (called from input2feed)
          • scripts/snowmix-settings


          • input variables (optional)
            output variables
            example
            system

            • SNOWMIX_IP (default: 127.0.0.1)
            • SNOWMIX_PORT (default: 9999)
            • snowmix

            • feed_id=1 source ./snowmix-settings;
              echo $geometry
              ...
            • get_snowmix-settings.sh
              •  #!/bin/bash

                # usage: get_snowmix-settings <feed_id> [<audio_feed_id>]

                feed_id=$1
                audio_feed_id=$2

                source ./snowmix-settings

                echo
                echo "SNOWMIX_IP: $SNOWMIX_IP"
                echo "SNOWMIX_PORT: $SNOWMIX_PORT"
                echo "  snowmix:           $snowmix"
                echo
                echo "video input (feed_id: $feed_id):"
                echo "  feed_control_pipe: $feed_control_pipe"
                echo "  feed_width:        $feed_width"
                echo "  feed_height:       $feed_height"
                echo
                echo "video output:"
                echo "  geometry:          $geometry"
                echo "  framerate:         $framerate"
                echo "  ctrsocket:         $ctrsocket"
                echo "  system_width:      $system_width"
                echo "  system_height:     $system_height"
                echo "  ratefraction:      $ratefraction"
                echo
                echo "audio input (audio_feed_id: $audio_feed_id):"
                echo "  feed_rate:         $feed_rate"
                echo "  feed_channels:     $feed_channels"
                echo
                echo "audio output (audio_sink_id: $audio_sink_id)"
                echo "  rate:              $rate"
                echo "  channels:          $channels"
                echo "  AUDIOFORMAT:       $AUDIOFORMAT"

                exit 0
            video
            input
            • feed_id
            • feed_control_pipe
            • feed_width
            • feed_height
            output
            • geometry
            • framerate
            • ctrsocket
            • system_width
            • system_height
            • ratefraction
            audio
            input
            • audio_feed_id
            • feed_rate
            • feed_channels
            output
            • audio_sink_id (default: 1)
            • channels
            • rate
            • AUDIOFORMAT
      • ls -l /dev/shm/shmpipe*
      • output
        • scene #1
          • frame #1.1
          • frame #1.2
          • ...
        • scene #2
          • frame #2.1
          • ...
        • ...
    • control using GUI (0.5.1)
      • tcl/
        included files
        (when modified, you must run: sudo make install)
        uses: echo <command> | nc <server> <port>, where command is:
        snowscene.tcl control and mix video scenes
        • cs_feed_commands.inc
        • scenes.inc

        snowaudio.tcl control and mix audio
        • graph.inc
        • audio.inc
        snowoutput.tcl monitor the output statistics:
        • Top graphic:
        • Bottom graphic:
        • Right graphic:
        • Legend:
          • "In Use" white
          • "Min" red
          • "Avg" green
          • "Max" yellow
          • "Last" blue
          • "Duration" cyan
        • graph.inc
        • output.inc
        • tcl eval snowmix message [snowmix info system status format]
        • tcl eval snowmix message [snowmix info system status]
        • system output status
        snowrelay.tcl
        • graph.inc
        • cs_feed_commands.inc
        • relay.inc
        • relay_analyzer.inc

        snowcommentator.tcl
        • commentator.inc

        snowcub.tcl


      • to control/monitor a remote snowmix server:
        • on remote server, allow your local address (/32) or subnet:
          • snowmix.ini
            • system host allow 127.0.0.1 192.168.1.0/24
        • when calling tcl locally
          • add server:
            • <ip_address_of_instance_running_snowmix>
    • control using netcat / nc
      • group command


        response
        system system ...


        STATS: ...
        video feed ...



        audio audio
        feed ...



        mixer ...



        sink ...

        TCL tcl reset




        help

        MSG: ...


        exec




        eval snowmix message <text>




        snowmix parse <snowmix command>




        snowmix parse silent <snowmix command>




        snowmix parses <snowmix command>




        snowmix info audio (feed | sink) (info | status | extended | syntax) (format | ids | maxid | nextavail | <id_list>)




        audio mixer (info | status | extended | source info | source status | source extended | syntax) (format | ids | maxid | nextavail | <id_list>)




        command ( names | list | at | syntax ) [ format | <name> ]




        feed (geometry | status | extended | state | syntax) (format | ids | maxid | nextavail | <id_list>)




        image (load | place | move | extended | syntax) (format | ids | maxid | nextavail | <id_list>)




        text (string | font | place | move | backgr | linpat | syntax) (format | ids | maxid | nextavail | <id_list>)




        (vfeed | virtual feed) (place | move | extended | syntax) (format | ids | maxid | nextavail | <id_list>)




        shape (info | list | place | move) ( format | ids | maxid | nextavail | <id_list> )




        system (info | status | maxplaces | overlay | syntax) [ format ]




        shape ( syntax | ( ( info | list | place | move ) ( format | ids | maxid | nextavail | <id_list>) )
      • bash
        • echo '<command>' | nc 127.0.0.1 9999
      • Python
        • dependencies
        • snowmix_info.py
          • import nclib

            # python 3

            nc = nclib.Netcat(('127.0.0.1', 2000))
            nc.recv_until(b'Snowmix version 0.5.1.\n')

            command = 'system info'
            nc.send_line(command.encode())
            response = nc.recv_until(b'STAT: \n')
            print(response.decode())

            command = 'system output info'
            nc.send_line(command.encode())
            response = nc.recv_until(b'STAT: \n')
            print(response.decode())

            command = 'audio mixer source'
            nc.send_line(command.encode())
            response = nc.recv_until(b'STAT: \n')
            print(response.decode())

            command = 'audio feed status'
            nc.send_line(command.encode())
            response = nc.recv_until(b'STAT: \n')
            print(response.decode())

            nc.close()
    • system commands (ini file)

      • command / line in ini file
        echo 'system info' | nc [-q 1] 127.0.0.1 9999
        echo 'system output info' | nc 127.0.0.1 9999 variable returned by
        scripts/snowmix-settings


        Snowmix version 0.5.1.
        STAT:  System info
        STAT:  Snowmix version      : 0.5.1
        STAT:  System name          : localhost.localdomain
        STAT:  Ini file             : .../minimal.ini
        STAT:  Block size mmap      : 2359296 (=1024*576*4)
        STAT:  Frame sequence no.   : 0
        STAT:  Lagged frames        : 0
        STAT:  Open Control conns.  : 2
        STAT:  Ctr verbose          : no
        STAT:  Ctr broadcast        : no
        STAT:  Output ctr fd        : master 6 client -1
        STAT:  Output settings      : mode 0 delay 1
        STAT:  Output frames inuse  : 0 of 20
        STAT:  Output shm name      : /shmpipe.25528.    0
        STAT:  Video feeds          : loaded
        STAT:  Virtual video feeds  : no
        STAT:  Video text           : no
        STAT:  Video image          : no
        STAT:  Video shapes         : no
        STAT:  Command interface    : no
        STAT:  Audio feeds          : no
        STAT:  Audio mixers         : no
        STAT:  Audio sinks          : no
        STAT:  TCL interpreter      : no
        STAT:  Stack                : 0
        STAT:
        Snowmix version 0.5.1.
        STAT: system output info
        STAT:   Verbose           = 1
        STAT:   Socket master     = 6
        STAT:   Socket client     = -1
        STAT:   Memory handle     = /shmpipe.25528.    0
        STAT:   Memory size       = 47185920 (20 blocks, block size 2359296)
        STAT:   Output mode       = 0 (standard, no thread)
        STAT:   Freeze            = no (0 frames, 0.00 secs)
        STAT:   Delay             = 1 frames (40.0 ms)
        STAT:


        minimal
        system control port 9999
        STAT:  Control port number  : 9999


        system host allow 127.0.0.1 192.168.1.0/24
        STAT:  Host allow           : 127.0.0.1/32 192.168.1.0/24

        system geometry 1024 576 BGRA
        STAT:  System geometry      : 1024x576 pixels
        STAT:  Pixel format         : BGRA
        STAT:  Bytes per pixel      : 4

        • $system_width
        • $system_height
        system frame rate 25
        STAT:  Frame rate           : 25.000

        • $ratefraction
        system socket /tmp/mixer1
        STAT:  Output ctr sock name : /tmp/mixer1
        STAT:   Socket name       = /tmp/mixer1
        • $ctrsocket

        stack ...



        NOTE: deprecated; only for quick setups.
    • video commands

      • command (echo '<command>' | nc 127.0.0.1 9999) / line in ini file
        echo 'system info'
        | nc 127.0.0.1 9999
        scripts/snowmix-settings
        creation
        • feed add [<feed no> <feed name>]

        input:
        • $feed_id
        setup
        • feed socket <feed no> <file name>
        • feed geometry <feed no> <width> <height>
        • feed fast overlay <feed id> <col> <row> <feed col> <feed row> <cut cols> <cut rows> [ <scale 1> <scale 2> [ <par_w> <par_h> [ center ]]]
        • feed filter [ <feed no> (fast | good | best | nearest | bilinear | gaussian)]
        • feed idle <feed no> <timeout in frames> <idle image file>
        • feed name <feed no> <feed name>
        • feed overlay (<id> | <id>..<id> | all | end | <id>..end) [ (<id> | <id>..<id> | all | end | <id>..end) ] ...
        • feed par <feed no> <scale_1> <scale_2>
        • feed (live | recorded) <feed no>
        • feed cutout <feed no> <start col> <start row> <columns> <rows>
        • feed shift <feed no> <column> <row>
        • feed scale <feed no> <scale_1> <scale_2>

        output:
        • $feed_control_pipe
        • $feed_width
        • $feed_height
        info
        • feed help
        • feed buffers
          • DBG: Feed 0 Internal
            DBG: Feed 1 Feed #1
            ...
        • feed info
          • STAT: feed id : state islive oneshot geometry cutstart cutsize offset fifo good missed dropped <name>
            STAT: feed 0 : STALLED recorded continuously 320x180 0,0 320x180 0,0 0:0 0 266393 0 <Internal>
            STAT: feed 1 : PENDING live continuously 320x180 0,0 320x180 0,0 0:10 0 266604 0 <Feed #1>
            ...
        • feed list [verbose]
          • STAT: Feed ID 0  Name: Internal
            STAT: Feed ID 1  Name: Feed #1
            ...


        NOTE: deprecated; only for quick setups.
    • audio commands

      • Audio Feed Audio Mixer Audio Sink

        command
        (echo '<command>' | nc 127.0.0.1 9999) /
        line in ini file
        echo 'system info'
        | nc 127.0.0.1 9999
        scripts/snowmix-settings command / line in ini file
        command / line in ini file
        echo 'system info'
        | nc 127.0.0.1 9999

        scripts/snowmix-settings
        creation
        STAT: audio feed ... input:
        • $audio_feed_id
        STAT: audio sink ... input:
        • $audio_sink_id (default: 1)
        Formats
        Sample Rate
        Channels
        • audio feed format [<feed id> (8 | 16 | 24 | 32 | 64) (signed | unsigned | float)]
        • audio feed rate [<feed id> <rate>]
        • audio feed channels [<feed id> <channels>]

        output:
        • $feed_rate
        • $feed_channels
        Info:
        • audio mixer rate
          • STAT: audio mixer 1 rate 48000 Hz
        • audio mixer channels
          • STAT: audio mixer 1 channels 2
        • audio mixer source
          • STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal
            ...
        Set:
        • audio mixer rate [<mixer id> <rate>]
        • audio mixer channels [<mixer id> <channels>]
        • audio mixer source [map <mixer id> <source id> <map 1> <map 2> ... <map n>
        • audio sink format [<sink id> (8 | 16 | 24 | 32 | 64) (signed | unsigned | float)]
        • audio sink rate [<sink id> <rate>]
        • audio sink channels [<sink id> <channels>]
        • audio sink source mixer <...> <...>
        • audio sink file <...> <...>

        output:
        • $rate
        • $channels
        Starting
        • audio feed ctr isaudio <feed id>


        • audio mixer start [[soft ]<mixer id>]
        • audio sink start [<sink id>]
        • audio sink ctr isaudio <sink id>


        Muting
        Volume
        • audio feed mute [(on | off) <feed id>]
        • audio feed volume [<feed id> <volume 0> ... <volume n>]


        Info:
        • audio mixer mute
          • STAT: audio mixer  1 volume 1.000,1.000
            STAT: - source audio feed   1 volume 1.000,1.000 muted
            ...
        • audio mixer volume (same as mute)
          • STAT: audio mixer  1 volume 1.000,1.000
            STAT: - source audio feed   1 volume 1.000,1.000 muted
            ...
        • audio mixer source
          • STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal
            ...
        Set:
        • audio mixer mute [(on | off) <feed id>]
        • audio mixer source [mute (on|off) <mixer id> <source no>]
        • audio mixer volume [<mixer id> <volume 0> ... <volume n>]
        • audio mixer source [volume <mixer id> <source no> <volume 0>... <volume n>]
        Info:
        • audio sink mute
          • STAT: audio sink  0 volume 1.000,1.000 muted
            STAT: audio sink  1 volume 1.000,1.000
        • audio sink volume
        Set:
        • audio sink mute [(on | off) <feed id>]
        • audio sink volume [<sink id> <volume 0> ... <volume n>]


        Delay
        • audio feed delay
        • audio feed delay <feed_id> <ms>


        • audio mixer source mindelay <mixer_id> <source_no> <ms>
        •  audio mixer source maxdelay <mixer_id> <source_no> <ms>



        Silence
        • audio feed add silence <feed id> <ms>


        • audio mixer add silence <mixer id> <ms>
        • audio mixer source add silence <mixer id> <source no> <ms>
        • audio sink add silence <sink id> <ms>


        Drop
        • audio feed drop <feed id> <ms>


        • audio mixer drop <mixer id> <ms>
        • audio mixer drop <mixer id> <source no> <ms>
        • audio sink drop <sink id> <ms>


        Info and status
        • audio feed info
          • STAT:  audio feed id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues
            STAT:  - audio feed 1 : READY, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 1
            ...
        • audio feed status
          • STAT: feed_id : state samples samplespersecond avg_samplespersecond silence dropped clipped delay rms
            STAT: audio feed 1 : READY 0 0 0 0 0 0 0 0.0,0.0
            ...
        • audio feed verbose [<level>]
        • audio feed help


        • audio mixer info
          • STAT:  audio mixer id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues
            STAT:  - audio mixer 1 : RUNNING, 48000, 2, 4, signed, 1.000,1.000, unmuted, 16384, 0, 2
        • audio mixer status
          • STAT: mixer_id : state samples sps avg_sps silence dropped clipped delay rms
            STAT: audio mixer 1 : RUNNING 1057478812 99972 96020 0 0 0 0,0 0.0,0.0
            STAT: - source audio feed   1 : READY 0 1 0 0 0 0 0 0.0,0.0
            ...
        • audio mixer verbose [<level>]
        • audio mixer help
        • audio sink info
          • STAT:  audio sink id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues
            STAT:  - audio sink 0 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, muted, 8192, 0, 0
            STAT:  - audio sink 1 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 0
        • audio sink status
          • STAT: sink_id : state samples samplespersecond avg_samplespersecond silence dropped clipped queue rms
            STAT: audio sink 0 : RUNNING 0 0 0 0 1068699328 0 0 0.0,0.0
            STAT: audio sink 1 : RUNNING 1067938584 95071 96005 0 0 0 0,0 0.0,0.0
        • audio sink verbose [<level>]
        • audio sink help


        Pausing


        • audio mixer source [pause <mixer id> <source no> <frames>]
        • audio mixer source [rmsthreshold <mixer id> <source no> <level>]



    • Sincronització àudio/vídeo / Audio/video synchronisation
      • Audio
      • Understanding Audio and Video Sync for Snowmix
        • Manual AV Sync
          • audio is ahead of video
            • determine optimal delay (temporarily)
              • add delay/silence several times (+10/20) until audio is in sync -> t1 (e.g. t1=60), t_audio_is_in_sync
              • add delay/silence several times (+10/20) until video is ahead of audio / audio is behind of video -> t2 (e.g. t2=170), audio_is_behind
              • optimal_delay = (2*t1 + t2) / 2 (e.g. optimal_delay=(2*60 + 170)/2=145)
                optimal_delay = (2*t_audio_is_in_sync + t_audio_is_behind) / 2
            • set optimal delay ()
              • audio feed delay 1 <optimal_delay>
              • audio feed delay 1 145
                audio mixer source mindelay 1 2 145
                audio mixer source maxdelay 1 2 325
      • Output buffer problem...
        • audio is ahead of video
          • audio feed delay 1 <t_mean>
          • determine optimal delay with Snowcub (temporarily):
            • add delay several times (+50, audio feed add silence 1 50) until audio is behind of video -> t_audio_is_behind
            • remove delay several times (-30, audio feed ...) until audio is ahead of video -> t_audio_is_ahead
            • optimal_delay = (t_audio_is_ahead + t_audio_is_behind) / 2
          • set optimal delay ()
            • audio feed delay 1 <optimal_delay>
              ...
        • audio is behind the video
          • system output delay ...
    • AV input/output with Gstreamer (shm for video, fd for audio)

      • input (to first feed)
        output
        audio
        • ( echo 'audio feed ctr isaudio 1';
          gst-launch-1.0 -v audiotestsrc is-live=true !\
          'audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2' !\
          fdsink fd=3 sync=true 3>&1 1>&2
          ) | nc 127.0.0.1 9999
        • audio_feed_id=1
          source ./snowmix-settings
          # audio
          ( echo "audio feed ctr isaudio $audio_feed_id";
            gst-launch-1.0 -v audiotestsrc is-live=true ! $AUDIOFORMAT ! fdsink fd=3 sync=true 3>&1 1>&2
          ) | nc $SNOWMIX_IP $SNOWMIX_PORT
        (
          echo 'audio sink ctr isaudio 1'; cat >/dev/null
        ) | nc 127.0.0.1 9999 \
        (
          head -1
          gst-launch-1.0 -v fdsrc fd=0 !\
            'audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2' !\
            audioconvert !\
            autoaudiosink
        )

        video
        • gst-launch-1.0 -q filesrc location=toto.mp4 ! decodebin name=decoder ! videoconvert ! videoscale ! videoconvert ! 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive, width=704, height=576' ! \
          shmsink
          socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc -l` wait-for-connection=1 sync=true
        • feed_id=1
          source ./snowmix-settings
          # video
          VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"
          gst-launch-1.0 -v videotestsrc pattern=ball ! $VIDEOCAPS ! videoconvert ! videoscale ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true
        • X11 output
          • gst-launch-1.0 -q shmsrcsocket-path=/tmp/mixer1 do-timestamp=true is-live=true !\ 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive,width=(int)1024,height=(int)576,framerate=(fraction)25/1' !\ videoconvert ! queue ! autovideosink
        • H.264 + RTP multicast
          • gst-launch-1.0 -v shmsrc socket-path=/tmp/mixer1 do-timestamp=true is-live=true ! video/x-raw,width=1280,height=720,framerate='25/1',format=BGRA ! videoconvert ! x264enc ! rtph264pay config-interval=10 pt=96 ! udpsink host=224.0.0.0 port=5008
        audio + video
        • test2feed.sh
        • ( echo 'audio feed ctr isaudio 1';
            gst-launch-1.0 -v \
                videotestsrc pattern=ball ! 'video/x-raw,framerate=25/1,width=704,height=576,format=BGRA' ! videoconvert ! videoscale ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc` wait-for-connection=1 sync=true \
                audiotestsrc wave=5 is-live=true ! 'audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2' ! fdsink fd=3 sync=true 3>&1 1>&2
          ) | nc  127.0.0.1 9999
        • feed_id=1
          audio_feed_id=1
          source $SNOWMIX/scripts/snowmix-settings
          VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"
          AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio from test
          ( echo "audio feed ctr isaudio $audio_feed_id";
            gst-launch-1.0 -q \
                videotestsrc pattern=ball ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \
                audiotestsrc is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2
          ) | nc $SNOWMIX_IP $SNOWMIX_PORT
        • feed_id=1
          audio_feed_id=1
          source $SNOWMIX/scripts/snowmix-settings
          VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"

          AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels"
          # video and audio from sdp
          ( echo "audio feed ctr isaudio $audio_feed_id";
            gst-launch-1.0 -q \
                filesrc location=/tmp/toto.sdp ! sdpdemux name=demux \
                demux. ! queue ! "application/x-rtp, media=(string)video" ! decodebin ! videoconvert ! videoscale ! videorate ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \
                demux. ! queue ! "application/x-rtp, media=(string)audio" ! decodebin ! audioconvert ! audioresample ! $AUDIOCAPS ! fdsink fd=3 sync=true \
                3>&1 1>&2
          ) | nc $SNOWMIX_IP $SNOWMIX_PORT
        • default video + audio output
          • source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1"
            AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels"
          • ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \
            ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \
            ( head -1
                $gstlaunch -q \
                shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! autovideosink \
                fdsrc fd=0 do-timestamp=true ! $AUDIOCAPS ! queue ! audiorate ! audioconvert ! audioresample ! queue ! autoaudiosink \
            )
        • H.264 + opus to RTP
          • source $SNOWMIX/scripts/snowmix-settings
            # video output
            VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1"
            # video coder
            gop_size=25
            video_profile=baseline
            video_bitrate=1024000
            # video rtp
            config_interval=-1
            rtp_video_payload_type=96
            # audio output
            AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels"
            echo "AUDIOCAPS: $AUDIOCAPS"
            # audio coder
            audio_bitrate=200000
            # audio rtp
            rtp_audio_payload_type=$(( rtp_video_payload_type + 1 ))
            #audio_media_subtype="mpeg4-generic"
            audio_media_subtype=opus
          • # sdp
            cat >$sdp_path <<EOF
            v=0
            c=IN IP4 $multicast_address
            m=video $video_rtp_port RTP/AVP $rtp_video_payload_type
            a=rtpmap:$rtp_video_payload_type H264/90000
            m=audio $audio_rtp_port RTP/AVP $rtp_audio_payload_type
            a=rtpmap:$rtp_audio_payload_type ${audio_media_subtype}/${rate}/${channels}
            EOF
          • ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \
            ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \
            ( head -1
                $gstlaunch -q \
                shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! x264enc bitrate=$video_bitrate key-int-max=$gop_size ! video/x-h264,profile=${video_profile} ! rtph264pay config-interval=$config_interval pt=$rtp_video_payload_type ! udpsink host=$multicast_address port=$video_rtp_port \
                fdsrc fd=0 do-timestamp=true ! $AUDIOCAPS ! queue ! audiorate ! audioconvert ! audioresample ! queue ! opusenc bitrate=$audio_bitrate ! rtpopuspay pt=$rtp_audio_payload_type ! udpsink host=$multicast_address port=$audio_rtp_port \
            )
      • test2feed.sh
        • #!/bin/bash

          MIN_ARGS=2
          MAX_ARGS=6
          if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
          then
              cat <<EOF
          Usage: `basename $0` [options] feed_id audio_feed_id

          Options:
          -p, --pattern video_pattern       video generated pattern (1) (default: snow)
          -w, --wave audio_wave             audio generated wave (2) (default: 6 (pink noise))

          (1) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videotestsrc.html#GstVideoTestSrcPattern
          (2) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html#GstAudioTestSrcWave

          EOF
              exit 1
          fi

          # defaults
          # video
          pattern=snow
          # audio
          wave=6

          # options
          eval set -- $(getopt -o p:w: --long pattern:,wave: -n $0 -- "$@")

          while true
          do
            case "$1" in
              -p | --pattern ) pattern="$2"; shift 2;;
              -w | --wave ) wave="$2"; shift 2;;
              -- ) shift; break ;;
              * ) break ;;
            esac
          done

          # parameters
          feed_id=$1
          audio_feed_id=$2


          if [ "X$SNOWMIX" = X ] ; then
            echo "You need to set the SNOWMIX environment variable"
            exit 1
          fi

          # get snowmix settings
          source $SNOWMIX/scripts/snowmix-settings

          echo
          echo "SNOWMIX_IP: $SNOWMIX_IP"
          echo "SNOWMIX_PORT: $SNOWMIX_PORT"
          echo "  snowmix:           $snowmix"
          echo
          echo "video input (feed_id: $feed_id):"
          echo "  feed_control_pipe: $feed_control_pipe"
          echo "  feed_width:        $feed_width"
          echo "  feed_height:       $feed_height"
          echo
          echo "video output:"
          echo "  geometry:          $geometry"
          echo "  framerate:         $framerate"
          echo "  ctrsocket:         $ctrsocket"
          echo "  system_width:      $system_width"
          echo "  system_height:     $system_height"
          echo "  ratefraction:      $ratefraction"
          echo
          echo "audio input (audio_feed_id: $audio_feed_id):"
          echo "  feed_rate:         $feed_rate"
          echo "  feed_channels:     $feed_channels"
          echo
          echo "audio output (audio_sink_id: $audio_sink_id)"
          echo "  rate:              $rate"
          echo "  channels:          $channels"
          echo "  AUDIOFORMAT:       $AUDIOFORMAT"


          # video
          VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"

          # audio
          AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels"

          # video and audio
          rm -f $feed_control_pipe
          (
            echo "audio feed ctr isaudio $audio_feed_id";
            gst-launch-1.0 -q \
                videotestsrc pattern=$pattern ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \
                audiotestsrc wave=$wave is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2
          ) | nc $SNOWMIX_IP $SNOWMIX_PORT


          exit 0
  • Problemes / Problems

Voctomix

  • Instal·lació / Installation
    • Dependencies
      • Mageia
        • urpmi...
    • cd ~/src
    • git clone https://github.com/voc/voctomix.git
    • cd voctomix
    • virtualenv env
    • source env/bin/activate
    • pip install pygobject
  • Ús / Usage
    • cd ~/src/voctomix/
    • ./voctocore/voctocore.py -vv
    • ./voctogui/voctogui.py -vv
  • API
    • ...

http://www.francescpinyol.cat/audio_video_mixers.html
Primera versió: / First version: 4.V.2020
Darrera modificació: 25 de maig de 2020 / Last update: 25th May 2020

Valid HTML 4.01!

Cap a casa / Back home.