This content originally appeared on DEV Community and was authored by Leo
Since the inception of video production, we have faced a significant challenge: processing vast amounts of data. Initially, computers were too slow, leading to the emergence of an industry focused on building specialized hardware boxes for various tasks such as encoding, decoding, transmitting, receiving, switching, and displaying video. These boxes became indispensable, making video production an expensive endeavor.
Today, however, we have powerful general-purpose hardware, such as GPUs and CPUs, capable of handling the video and audio data we need to process. But the industry is still behind on this. Why do video encoder boxes still exist when GPUs can perform these tasks?
So what is the problem?
Integration and Software. We have little bits and pieces of software that can do some things, but not others. For simple recording tasks, tools like OBS suffice. For more complex switching, tools like vMix are used. All those tools have one thing in common: They are not as flexible as plugging in some wires wherever you want. You are stuck with what is handed to you.
The Solution
We make boxes and wires in software. Instead of buying encoder boxes, you encode your video with little programs and connect these little programs with virtual wires. This it the idea of streamtasks. You can even change what codec you want, what resolution, what pixel formats, whatever this little piece of software („Task“) allows you to change. To get a better understanding of how the software works I recommend watching the overview video.
Streamtasks allows you to connect topics, which are displayed as colored circles, with eachother. During operation, the task will process the input data in realtime. For example, a video encoder will encode the raw video it receives and then output the encoded video on the output topic.
What you can do with streamtasks depends on two things:
- the tasks available
- your imagination
While I cannot improve your imagination, I can make tasks and simplify the process of creating new ones.
Right now, there are over 50 different tasks integrated into streamtasks. These range from very simple to more complex and powerful tasks, many of which are documented here.
The general idea is to give the user as much control as possible. If a task can be split into two simpler ones, it is usually a good idea.
If you can write software, you can write your own tasks. The simplest task can be written in 4 lines of code (even less, if you really try). If you are interested in making your own tasks, I refer you to the documentation: https://leopf.github.io/streamtasks/custom-task.html
AI
One of the greatest advancements of our time is AI, which has not yet been widely integrated into video production. Streamtasks provides a strong framework for integrating AI and other kinds of software into your production pipelines. You can implement AI into you workflows today and build things like:
- Live Audio Transcription
- Live Face Detection
- Face Tracking
- Text To Speech
You can watch this Demo of a chatbot with TTS to get an Idea of what is possible.
Distributing Data and Workloads
While our computers are really fast today, there is a limit to their performance. In order to build larger production pipelines that process more data, you will have to distribute data and workload accross multiple machines.
This is possible with streamtasks today.
You can have mutliple instances of streamtasks running on different machines and connect them by either using the connection manager UI, or specifying a connect/serve url on the command line.
# the main instance (started with -C) accepting tcp connections on 9002
streamtasks -C --serve tcp://0.0.0.0:9002
# a sub instance connecting to the main system (in this case running on the same machine)
streamtasks --connect tcp://127.0.0.1:9002
Thats it. Now you can create your pipelines with all tasks available on either machine. The only apparent difference between tasks running different machines is a dotted line connecting their topics.
Each task available on both machines will now be displayed twice in the task selection. Once for each node.
As a result you can now start tasks on different machines and connect them seemlessly.
In this example we have two different audio inputs (like microphones) one running on node/machine "demo1" and the other running on "demo2". This audio is mixed and then played on an audio output on "demo1". Be cautious! The audio data from the audio input on demo2 is being sent over TCP. If you send raw video over a TCP connection you will likely not get the result you wish for. 1080p 30fps RGB24 video is 1492Mb/s of data. If you do not have enough bandwidth, it won't work.
Distributing Workloads
By having tasks run on different machines, you can distribute heavy workloads, like encoding video. This can be done while still extracting useful information like the amount of activity in a video (how many and how much pixels have changed), before the video is encoded.
In the following example we have two video inputs (like cameras). One on each node. The video activity is measured, in order to switch between the video feeds, and the video is encoded on each node. The encoded video data from demo2 is then sent over to demo1, where we switch between the video feeds based on their activity. The most active video is then displayed in a video viewer.
This allows us to use one machine as an encoder for each video feed. When using GPU Hardware encoders, each machine can usually encode much more than just one video feed. As we scale up, we will want to distribute the load across machines.
Distributing Data
Local internet connections are often very limited. Even if you are lucky to get 1Gb/s upload speed, it still does not compare to the kind of speed you can archive from a server running in a datacenter.
When streaming to multiple streaming services at the same time, you might want to first send your video to a server in a datacenter and from there distribute it to multiple streaming services.
In the following examples we have a video input running on our computer. The video data is encoded and then sent to 3 different output containers. These output containers will then each publish the video feed to a streaming service as a rtmp stream (see the live streaming demo).
The streamtasks network is smart enough to only send the video feed once to our server running in the datacenter, despite there being three tasks receiving the data. As a result we are only using a third of the upload bandwidth on our local machine, while streaming to three different services.
Building Dashboards
The more complex your production pipelines are, the more oversight you need in order to monitor them. Streamtasks allows you to create dashboards that consist of windows, each window is the "display" of a task. This can be the video from a video viewer, the text of a text display, the switch of a switch UI.
The following example is a kill switch. The timestamp updater moves the time of the video feed 5s into the future. When synchronized with the other video feeds this causes the actual feed to be pushed back 5s in time. That way you can preview the video in the live view and then switch the output to a static image 5s before it is displayed.
The video viewers each display a video, which can be arranged in the dashboard. The switch UI, used to turn on/off the video, displays a control element.
More Examples
Since you are free to combine any tasks as you wish, you can built awesome things with it. Here are some examples:
- You can implement video switching by having multiple video streams and using a switch to switch between them. All you need to actually switch is a way of controlling the switch like radio buttons, a little UI that displays some options and lets you select one. The switch will always switches to the video feed that has the largest number as its control signal.
- Reducing the volume of ambient sound (like stadium microphones) when someone is speaking.
- Switching the video of podcasts by always switching to the loudest speaker
- Playing sounds
- Creating chatbots that speak
If you are interested in seeing more working examples, you can watch them on Youtube.
This content originally appeared on DEV Community and was authored by Leo
Leo | Sciencx (2024-08-07T01:35:14+00:00) Video Production with Streamtasks. Retrieved from https://www.scien.cx/2024/08/07/video-production-with-streamtasks/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.