In this blog post, I*m going to describe how a Raspberry can be used for video encoding. The idea is to have recorded TV from a receiver converted into MP4 videos to be watched on a tablet or smartphone. The encoded file is upload to Google drive to reduce download time for the mobile device.
NOTE: This is work in progress - there are still some things to be done. As I don't want to wait until the last line of code is written, I'm publishing this with the intention to provide updates soon. At least the early publishing should keep things fresh for writing them down. Please the comment function for feedback or questions.
NOTE: This is work in progress - there are still some things to be done. As I don't want to wait until the last line of code is written, I'm publishing this with the intention to provide updates soon. At least the early publishing should keep things fresh for writing them down. Please the comment function for feedback or questions.
This is quite an extensive blog as there are many ingridients needed.
What you need:
a) Raspberry Pi. You should have the latest Raspberry PI 2 model B
b) USB Harddisk attached to the PI. (Alternative is a large & fast SD card / USB stick)
c) A receiver configured to download recorded files via FTP. Assumption the files are ".ts" - transport streams
d) a mail account to send mails from (using SMTP)
e) a Google Drive account
Approach:
The biggest technical issue to solve is the video enconding:
When I started with the project I initially looked at FFMPeg. I managed to get encoding to work, however the performance is not convincing. Despite using 4 cores the pi does process between 1 and 2 frames a second. Reducing the framerate to 25 fps means that encoding still takes 25 times of the recorded duration. (I did not really investigate the influence of the input framerate (50) vs the output frame rate (25).
Understanding that the performance of ffmpeg is just too slow, I investigated the hardware supported encoding on the PI. In order to use this, it is required to use the gstreamer libs. This turned out to be somewhat complicated. I got the h264 encoding to work, but was lacking audio. Playing around with different modules can create pipelines where some data piles up and the pi needs endless memory. Finally, I learned that gstreamer encodes an MP4, but seems to miss some header information. Such a file cannot be played be a media player. Apparently, the streamer stuff is more intended to process a life signal for streaming to a device rather than processing files. The performance of the hardware streaming is way faster compared to ffmeg, but still not near real time. Also I did notice that the gstreamer implementation on the raspi is behind its possibilities as it seems to use only one thread (CPU always at 97%, only 1 core).
In addition, I learned that there is a substantial problem with Audio and Video synchronization. I'm not sure that the solution I found here is the best as it accepts that GStreamer "looses" a couple of seconds.
In addition, I learned that there is a substantial problem with Audio and Video synchronization. I'm not sure that the solution I found here is the best as it accepts that GStreamer "looses" a couple of seconds.
In order to overcome the sync & format problems, I turned back to ffmpeg. The approach described here is to first encode the movie with gstreamer into separate audio and video files. In a second step the files are fixed & merged into a syntactically correct MP4 file using ffmpeg.
1. ffmpeg
A couple of steps have to be takes to make ffmpeg work on the Raspberry for this project. As I startet with encoding using ffmpeg, I did not verify whether there are less steps for the ffmpeg installation required for muxing only. I expect that ffmpeg needs to "understand" h264 to do a proper job.
In order to get an ffmpeg with the proper encoders. It is necessary to install l-mash and x264 prior to building ffmpeg. All these packages have to be build:
Step 1.1: l-smash
Perform the following steps on the Raspberry (root account needed for install):
git clone https://code.google.com/p/l-smash/
configure
make
make install
The l-mash stuff is needed by the x264 library.
Step 1.2: x264
Perform the following steps on the Raspberry (root account needed for install):
./configure --extra-cflags="-march=armv6 -mfloat-abi=hard -mfpu=vfp" --enable-static --disable-opencl --disable-asm
make
make install
Step 1.3: ffmpeg
Perform the following steps on the Raspberry (root account needed for install):
git clone git://source.ffmpeg.org/ffmpeg.git
cd ffmpeg
sudo ./configure --arch=armel --target-os=linux --enable-gpl --enable-libx264 --enable-nonfree
make
make install
After these steps you should have working installation of ffmpeg on the Raspberry. You may verify the installation with the following command:
DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (encoders: libx264 libx264rgb )
2 Gstreamer
In order to install gstreamer you need to install quite a number of packages from apt:
apt-get update
apt-get install libgstreamer1.0-0 libgstreamer1.0-0-dbg libgstreamer1.0-dev liborc-0.4-0 liborc-0.4-0-dbg liborc-0.4-dev liborc-0.4-doc gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer1.0-alsa gstreamer1.0-doc gstreamer1.0-omx gstreamer1.0-plugins-bad gstreamer1.0-plugins-bad-dbg gstreamer1.0-plugins-bad-doc gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-base-dbg gstreamer1.0-plugins-base-doc gstreamer1.0-plugins-good gstreamer1.0-plugins-good-dbg gstreamer1.0-plugins-good-doc gstreamer1.0-plugins-ugly gstreamer1.0-plugins-ugly-dbg gstreamer1.0-plugins-ugly-doc gstreamer1.0-pulseaudio gstreamer1.0-tools gstreamer1.0-x libgstreamer-plugins-bad1.0-0 libgstreamer-plugins-bad1.0-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev
This takes a moment, but then you should have the gstreamer stuff ready to go.
3 Google Drive
Now it is time to install also python as we will need ti later on:apt-get python python-crypto python-pip
For the installation of the google drive API you may need root:
pip install --upgrade google-api-python-client
In order to be able to upload processed movie files you need to have the credentials for the oAuth stuff:
Go to the google developer console: https://console.developers.google.com/
- Create a new project
- Select APIs & Auth
- Select Add Credentials -> Oauth 2.0 client ID
- Application Type: Web
- Now you should see the new credentials under OAuth 2.0 client ID. Select the credential by clocking on its name "Webclient ..".
- Under authorized redirect URLs enter: http://localhost:8080/. Important: Google matches the URLs by character, thus you need to make sure that you enter the last "/". Press "Save"
- Finally download the json file (download symbol appears when mouse move to the end of the line in the list of credentials. The file is needed later with the software installation.
4 NAS Movie Processor
Download the source code from source forge: https://sourceforge.net/projects/nasmovieprocessor/Unpack the file under your home directory (in my case /root/src). Before you start a couple of steps needs to be taken:
Step 4.1 Credentials
Copy the previously downloaded json to the src directory of the unpacked the source code. (I.e. /root/src/NASMovieProcessor/src/)
Step 4.2 Links & Directories
Create the following links in this directory (/root/src/NASMovieProcessor/src/):
ln -s client_secret_*,json client_secrets.json
ln -s eu/liebrand/movie/movieprocessor.ini .
On the file space where the movies are stored locally on the Raspberry:
create a root directory: "mc"
create an incoming directory: "mc/incoming"
create a log directory: "mc/logs"
Step 4.3 Config File
You need to provide some data for the config file to get things working finally:
Section Receiver:
host - IP or hostname of your TV receiver (this is where our ftp client will connect)
user - user on the receiver to be used for downloading the ts files
password - ftp password
sourceDir - directory on the receiver to find the recorded media. If unsure what this is, use ftp manually and find out what cwd command it takes to switch to the recordings
incomingDir - full path of the above created incoming directory
movieDir - full path of the above created root directory
gstCmd - normally no need to adjust, use "which get-launch-1.0" to verify that gstreamer can be found
ffmpegCmd - same as gstCmd - verify location
notify - enter your email address, where you want to be notified when processing has completed.
encLogFIle - a path and filename to store the gstreamer output. This is especially useful when turing debug on and the amount of data increases.
inventoryFile - path & name of a file that will contain already processed files (to avoid reprocessing)
Section Mail:
host - SMTP host of your mail provider
port - SMTP port - consult you mail provider for setting
user - user to authenticate with for outbound mails.
password - password for the above mentioned user
Section Movie:
Here you can specify what movies to look for. The idea is that you receiver records series or recurring content like news. To be encoded is only the latest content of a series (e.g. the last news). For the list of files a matching towards the end of the file name is done. Assumption is that the front part of the name contains the date & time information for each recording. This is at least true for linux images used on Dreambox and vuSolo devices. So you enter something like this:
[Movie]
<pattern to compare>=<outputfile.mp4>
Example:
KSBY News - Morning Update=Morning_Update.mp4
Syncing Audio & Video
It
appears that the Gstreamer output "looses" some seconds of the video
during encoding. Per 15 minutes movie, the loss is about 9 seconds. This means
a normal movie with 90 minutes, you most probably will not be able to follow to
ist end, if the voice track has any meaning to understanding what is going on.
I have not found out how to tell Gstreamer x264 encoder to stick to the
original length. In order to fix this with a post processing there are in
general two options. One is to stretch the video to the original length or to
shrink the audio to the video size. Clearly, fixing the video is the option to
go after. However, the ffmpeg re-encoding of the video stream basically eats up any performance
gain of hardware processing. A framerate of 2.5 fps / second is achieved. 15 minutes
of video need 2+ hours to re-encode. So I went for fixing the audio stream.
Step 1: Preparing Video
Gstreamer produces a
stream w/o headers which makes it impossible to determine the video length. We
have to run it once through ffmpeg:
ffmpeg -v video.mp4
-vcodec copy fixedvideo.mp4
Step 2: Get the size of Audio and Video
ffprobe seems to
come with ffmpeg:
ffprobe -i audio.ac3
-show_entries format=duration -v quiet -print_format json
ffprobe -i
fixedvideo.mp4 -show_entries format=duration -v quiet -print_format json
Both provide an
output like this:
{
"format": {
"duration": "890.944000"
}
}
The json format makes it easy to parse and use the numbers in a python script.
Step 3: Shrink the Audio
You need to
calculate from both outputs the factor. As you would like to shrink, the number
needs to be >1. Divide the audio duration by video duration and use this
factor to process audio:
ffmpeg -i audio.ac3
-filter:a atempo="#FACTOR#" shrinkedaudio.ac3
Where #FACTOR#=lengthOf(Video)/lengthOf(Audio).
Step 4: Muxing into final File:
Do your muxing as
usual:
ffmpeg -i
fixedvideo.mp4 -i shrinkedaudio.ac3 -vcodec copy -acodec copy output.mp4
You may notice that
the muxed file is slightly shorter than the original recording.
Things To Do
As indicated in the beginning - there are still some things to be done to make it a user-friendly end to end solution. Here is my current list of things to do:- Proper Google Drive Authentication Code: I tweaked the high level API in order to make a resumable upload. The upload needs to be resumable, otherwise upload won't work (MemoryError in Python). Currently, the code will break when Google changes the API internals.
- DONE: Upload w/ Google drive: There seem to be some kind of race condition. Creating a file and immediately afterwards trying to upload *sometime* failes. Re-trying a moment later works fine.
- DONE: Fix GStreaner Log File location (currently hardcoded)
- DONE: Name of upload file appear in Google as "untitled"
- Split & Thread: I'd like to find out whether the Raspberry can be used to run the hardware based encoding on four cores in parallel. That would gibe quite a performance boost. Update: I think, that should not work as not the different cores are relevant, but the GPU.
- DONE: Audio and Video seem to run out of sync.
Finally, some notes on Gstreamer
I consulted many pages in the web trying to find out how to construct the proper pipelines for gstreamer. I'd like to share a couple of insights - maybe that is of help for others:
- The hardware based video encoding (the name actually says it) is only doing video. I.e. you get a file, but no audio. Thus you need to construct a pipeline setup to process both.
- The "decode bin" output produces separate pipes for audio and video. You should have sinks for both. If you only process the audio part, you run soon out of memory as the video stream is piling up in a pipe without sink.
- When experimenting with the GStreamer components it is always a good idea to have an eye on the memory consumption. A pipe setup may start to work, but somewhere data is pilling up and you run out of memory. (The Raspberry gets very slow and more and more irresponsive)
- When recording from TV it is common to add a couple of minutes before and/or after to avoid missing parts of the film. It may happen that during the recording the audio or video format of the stream changes. I hope, the current pipe setup will handle this properly. In the event the processing breaks at some point - maybe it is because of a starting or ending move in connection with a format change.
- A nasty experience with gstreamer: the encoder won't work (negotiation problem). A setup working before no longer worked. Setting GST_DEBUG="*:3" revealed "insufficient resources". A reboot solved the problem. .. Hm...