Build for Android with NDK r10c

[Grinchman]

Trying to build Bento4 library to use in Android project.

My platform is: Ubuntu 14.04 x86-64
Android NDK is: android-ndk-r10c
Bento4 source: last commit of master branch

According to build instruction I run build command from directory Build/Targets/arm-android-linux. Build command is: scons -u build_config=Release target=arm-android-linux
Script doesn't found ndk toolchain, because it tries to locate it in linux-x86 but I have linux-x86_64 platform. I fix scrip to find proper toolchain but build command failed with this build log:

********* Configuring Build Target = arm-android-linux / Release ********
Building for Android:
ANDROID_HOST_SYSTEM = linux-x86_64
ANDROID_TOOLCHAIN = arm-linux-androideabi-4.6
ANDROID_PLATFORM = android-4
ANDROID_ARCH = arm
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated VariantDir targets: Build/Targets/arm-android-linux
arm-linux-androideabi-g++ -o Build/Targets/arm-android-linux/Release/C++/Apps/Aac2Mp4/Aac2Mp4.o -c -fno-exceptions -fno-rtti -Wall -D_REENTRANT -O3 -I/home/grinch/Android/android-ndk-r10c/sources/cxx-stl/system/include --sysroot /home/grinch/Android/android-ndk-r10c/platforms/android-4/arch-arm -msoft-float -fpic -mthumb-interwork -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -D_REENTRANT -DANDROID -DNPT_CONFIG_HAVE_SYSTEM_LOG_CONFIG -ISource/Config -ISource/C++/Core -ISource/C++/Crypto -ISource/C++/MetaData -ISource/C++/System/StdC -ISource/C++/System/Posix -ISource/C++/Codecs Source/C++/Apps/Aac2Mp4/Aac2Mp4.cpp
arm-linux-androideabi-g++ -o Build/Targets/arm-android-linux/Release/C++/Apps/AvcInfo/AvcInfo.o -c -fno-exceptions -fno-rtti -Wall -D_REENTRANT -O3 -I/home/grinch/Android/android-ndk-r10c/sources/cxx-stl/system/include --sysroot /home/grinch/Android/android-ndk-r10c/platforms/android-4/arch-arm -msoft-float -fpic -mthumb-interwork -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -D_REENTRANT -DANDROID -DNPT_CONFIG_HAVE_SYSTEM_LOG_CONFIG -ISource/Config -ISource/C++/Core -ISource/C++/Crypto -ISource/C++/MetaData -ISource/C++/System/StdC -ISource/C++/System/Posix -ISource/C++/Codecs Source/C++/Apps/AvcInfo/AvcInfo.cpp
Source/C++/Apps/AvcInfo/AvcInfo.cpp: In function 'int main(int, char)':
Source/C++/Apps/AvcInfo/AvcInfo.cpp:125:76: error: no matching function for call to 'AP4_FileByteStream::AP4_FileByteStream(char const&, AP4_FileByteStream::Mode)'
Source/C++/Apps/AvcInfo/AvcInfo.cpp:125:76: note: candidates are:
Source/C++/Core/Ap4FileByteStream.h:65:5: note: AP4_FileByteStream::AP4_FileByteStream(AP4_ByteStream_)
Source/C++/Core/Ap4FileByteStream.h:65:5: note: candidate expects 1 argument, 2 provided
Source/C++/Core/Ap4FileByteStream.h:42:7: note: AP4_FileByteStream::AP4_FileByteStream(const AP4_FileByteStream&)
Source/C++/Core/Ap4FileByteStream.h:42:7: note: candidate expects 1 argument, 2 provided
Source/C++/Apps/AvcInfo/AvcInfo.cpp:126:14: error: expected type-specifier before 'AP4_Exception'
Source/C++/Apps/AvcInfo/AvcInfo.cpp:126:14: error: exception handling disabled, use -fexceptions to enable
scons: *_* [Build/Targets/arm-android-linux/Release/C++/Apps/AvcInfo/AvcInfo.o] Error 1
scons: building terminated because of errors.

Please help to build your library to use in Android. Thanks.

[barbibulle]

The scons config file for Android does need to be updated a bit, you're right. It seems to still be working fine with NDK 10, but it could use an update.
(you need to set the env variable ANDROID_NDK_ROOT to point to the location where you downloaded the NDK. once that's done, just using:
scons -u target=arm-android-linux
should work (works on my system, MacOSX with NDK r10c and r10d)

The error you're seeing here are only in two command line example apps, AvcInfo and HevcInfo, which you don't really need to build (command lines apps on Android are not often useful!). So you can either just remove them from the build by editing 'Build.scons', or you can pull the latest commit from the 'develop' branch here where I have committed a fix that allows this two apps to build on Android.

[Grinchman]

Thanks! develop branch successfully built!

Next question is about Mp4Edit app. Specifically about command --insert. What exact I have put to that command to append one mp4 file (for example file_to_append.mp4) to another file (original_file.mp4)?

[barbibulle]

mp4edit allows you to insert/change/remove atoms (boxes) from an MP4 file, but not concatenating two files. None of the current command line apps actually implement concatenation at this point. It would be fairly easy to write a concatenator based on the Bento4 class library API (just like all the other command line apps are built on top of the core library).

[Grinchman]

Oooh, I see...Sad to read this...ok, will look how to implement such functionality with Bento4.
As I understand, first I have to open one file, parse it to atoms and put that atoms to second file? Is that right?

[barbibulle]

It's simpler than this.
What you need to do is to create the output file, then open the first input file, read its samples one by one and add them to the output, then open the second file, read its samples one by one and add them to the output. You'll also need to copy the sample descriptions from the input(s) to the output. If you know for sue that both inputs have the same sample descriptions, you can simply copy the info from the first file. If they may be different, you would add to the output both input sample descriptions (for each track), and when you add the samples from the second input you specify that the sample description index is 1 (instead of 0).
You can take a look at the mp4mux app for an example of how to create an output file and adding samples (and sample descriptions to it).

[Grinchman]

Thanks for tips! They partially helps ) But anyway I still stuck in merging of two mp4 files. I've looked in code of Mp4Mux app into method AddMp4Tracks. It looks quite simple: get input Tracks and add them to output Movie and nothing more. I've done the same but result video has wrong orientation and it plays frames only from first file. Can you look at my code? What I'm doing wrong? What about SampleDescription? I didn't found in method AddMp4Track some usage of Samples.

// create the first input stream
AP4_Result result;
AP4_ByteStream* input = NULL;
result = AP4_FileByteStream::Create(input_filename,   AP4_FileByteStream::STREAM_MODE_READ, input);
if (AP4_FAILED(result))
{
    sprintf (status, "ERROR: cannot open input file (%s)\n", input_filename);
    return env->NewStringUTF(status);
}

//Get Movie from first file
AP4_File first_file(*input);
input->Release();
AP4_Movie* input_movie = first_file.GetMovie();
if (input_movie == NULL)
{
    sprintf (status, "ERROR: input file (%s) doesn't have Movie object\n", input_filename);
    return env->NewStringUTF(status);
}
__android_log_print(ANDROID_LOG_ERROR, "Mp4Editor", "input file created");



// create the second input stream
AP4_ByteStream* input2 = NULL;
result = AP4_FileByteStream::Create(appended_filename, AP4_FileByteStream::STREAM_MODE_READ, input2);
if (AP4_FAILED(result))
{
    input->Release();
    sprintf (status, "ERROR: cannot open output file (%s)\n", appended_filename);

    return env->NewStringUTF(status);
}

//Get Movie from second(appended) file
AP4_File second_file(*input2);
input2->Release();
AP4_Movie* input_movie2 = second_file.GetMovie();
if (input_movie2 == NULL)
{
    sprintf (status, "ERROR: input file (%s) doesn't have Movie object\n", appended_filename);
    return env->NewStringUTF(status);
}
__android_log_print(ANDROID_LOG_ERROR, "Mp4Editor", "input2 file created");



// create the output stream
AP4_ByteStream* output = NULL;
result = AP4_FileByteStream::Create(output_filename, AP4_FileByteStream::STREAM_MODE_WRITE, output);
if (AP4_FAILED(result))
{
    sprintf (status, "ERROR: cannot open output file (%s)\n", output_filename);

    return env->NewStringUTF(status);
}

//Create output Movie
AP4_Movie* output_movie = new AP4_Movie();
if (output_movie == NULL)
{
    sprintf (status, "ERROR: output file (%s) doesn't have Movie object\n", output_filename);
    return env->NewStringUTF(status);
}
__android_log_print(ANDROID_LOG_ERROR, "Mp4Editor", "output file created");



//ADD TRACKS FROM TWO INPUT FILES TO OUTPUT FILE

//Put tracks from first input file to output Movie
AP4_List<AP4_Track>::Item* track_item = input_movie->GetTracks().FirstItem();
AP4_List<AP4_Track>::Item* track_item2 = input_movie2->GetTracks().FirstItem();

while (track_item)
{
    AP4_Track* track = track_item->GetData();
    track = track->Clone();
    // reset the track ID so that it can be re-assigned
    track->SetId(0);
    output_movie->AddTrack(track);
    track_item = track_item->GetNext();
}

//Put tracks from second input file to output Movie
while (track_item2)
{
    AP4_Track* track = track_item2->GetData();
    track = track->Clone();
    // reset the track ID so that it can be re-assigned
    track->SetId(0);
    output_movie->AddTrack(track);
    track_item2 = track_item2->GetNext();
}


// create a multimedia file
AP4_File file(output_movie);

// setup the brands
AP4_Array<AP4_UI32> brands;
brands.Append(AP4_FILE_BRAND_ISOM);
brands.Append(AP4_FILE_BRAND_MP42);

// set the file type
file.SetFileType(AP4_FILE_BRAND_MP42, 1);

// write the file to the output
AP4_FileWriter::Write(file, *output);

// cleanup
output->Release();

[Grinchman]

Hello!
I used mp4info app and found that my input is coded in H.264:

File:
major brand: isom
minor version: 0
compatible brand: isom
compatible brand: iso2
compatible brand: avc1

Movie:
duration: 9152 ms
time scale: 6000
fragments: no

Found 2 Tracks
Track 1:
flags: 15 ENABLED IN-MOVIE IN-PREVIEW
id: 1
type: Video
duration: 8879 ms
language: ```
media:
sample count: 264
timescale: 90000
duration: 799151 (media timescale units)
duration: 8879 (ms)
bitrate (computed): 41888.176 Kbps
display width: 3840.000000
display height: 2160.000000
frame rate (computed): 29.732
Sample Description 0
Coding: avc1 (H.264)
Width: 3840
Height: 2160
Depth: 24
AVC Profile: 66 (Baseline)
AVC Profile Compat: 80
AVC Level: 51
AVC NALU Length Size: 4
AVC SPS: [67428033da00f0010fa01b4284d4]
AVC PPS: [68ce06e2]
Codecs String: avc1.428033

Track 2:
flags: 15 ENABLED IN-MOVIE IN-PREVIEW
id: 2
type: Audio
duration: 9152 ms
language: ```
media:
sample count: 429
timescale: 48000
duration: 439296 (media timescale units)
duration: 9152 (ms)
bitrate (computed): 96.247 Kbps
Sample Description 0
Coding: mp4a (MPEG-4 Audio)
Stream Type: Audio
Object Type: MPEG-4 Audio
Max Bitrate: 96000
Avg Bitrate: 96000
Buffer Size: 768
MPEG-4 Audio Object Type: 2 (AAC Low Complexity)
MPEG-4 Audio Decoder Config:
Sampling Frequency: 48000
Channels: 1
Sample Rate: 48000
Sample Size: 16
Channels: 1

Is that right? Then I tried to use code of method AddH264Track from Mp4Mux.cpp just to put one input file to new output file, but get failed again. Still don't see clear how to manage my input files ((((

[Grinchman]

I tried another aproach to get correct output file. I decide to construct new AP4_Track with AP4_SampleTable in which I added all AP4_Samples from 2 input files. On that stage I've ignored AUDIO tracks (is that ok to do this?). So, the code is looks like this:

AP4_Track* track_video = input_movie->GetTrack(AP4_Track::TYPE_VIDEO);
track_video = track_video->Clone();
AP4_Track* track_video2 = input_movie2->GetTrack(AP4_Track::TYPE_VIDEO);
track_video2 = track_video2->Clone();

AP4_Track* track_audio = input_movie->GetTrack(AP4_Track::TYPE_AUDIO);
AP4_Track* track_audio2 = input_movie2->GetTrack(AP4_Track::TYPE_AUDIO);


// create a sample table
AP4_SyntheticSampleTable* sample_table = new AP4_SyntheticSampleTable();

AP4_SyntheticSampleTable* sample_video = (AP4_SyntheticSampleTable*)track_video->GetSampleTable();
AP4_SyntheticSampleTable* sample_video2 = (AP4_SyntheticSampleTable*)track_video2->GetSampleTable();
AP4_SyntheticSampleTable* sample_audio = (AP4_SyntheticSampleTable*)track_audio->GetSampleTable();
AP4_SyntheticSampleTable* sample_audio2 = (AP4_SyntheticSampleTable*)track_audio2->GetSampleTable();

AP4_SampleDescription* description_video = track_video->GetSampleDescription(0);
AP4_SampleDescription* description_video2 = track_video2->GetSampleDescription(0);
AP4_SampleDescription* description_audio = track_audio->GetSampleDescription(0);

unsigned int video_width = track_video->GetWidth();
unsigned int video_height = track_video->GetHeight();


for(int i = 0; i < sample_video->GetSampleCount(); i++)
{
    AP4_Sample sample;
    sample_video->GetSample(i, sample);
    sample_table->AddSample(sample);
}

for(int i = 0; i < sample_video2->GetSampleCount(); i++)
{
    AP4_Sample sample;
    sample_video2->GetSample(i, sample);
    sample_table->AddSample(sample);
}

sample_table->AddSampleDescription(description_video);

AP4_UI32 movie_timescale      = track_video->GetMovieTimeScale();
AP4_UI32 media_timescale      = track_video->GetMediaTimeScale();

AP4_UI32 duration_movie1     = input_movie->GetDurationMs();
AP4_UI32 duration_movie2     = input_movie2->GetDurationMs();

AP4_UI32 duration_media1      = track_video->GetMediaDuration();
AP4_UI32 duration_media2      = track_video2->GetMediaDuration();

AP4_UI64 video_track_duration = duration_movie1 + duration_movie2;
AP4_UI64 video_media_duration = duration_media1 + duration_media2;

// create a video track
AP4_Track* track = new AP4_Track(AP4_Track::TYPE_VIDEO,
                                 sample_table,
                                 0,                    // auto-select track id
                                 movie_timescale,      // movie time scale
                                 video_track_duration, // track duration
                                 media_timescale,     // media time scale
                                 video_media_duration, // media duration
                                 "```",                // language
                                 video_width<<16,      // width
                                 video_height<<16      // height
                                 );
output_movie->AddTrack(track);
// create a multimedia file
AP4_File file(output_movie);

// setup the brands
AP4_Array<AP4_UI32> brands;
brands.Append(AP4_FILE_BRAND_ISOM);
brands.Append(AP4_FILE_BRAND_MP42);
brands.Append(AP4_FILE_BRAND_AVC1);

// set the file type
file.SetFileType(AP4_FILE_BRAND_MP42, 0);

// write the file to the output
AP4_FileWriter::Write(file, *output);

// cleanup
output->Release();

But still no luck! Result file isn't played by videoplayer.

mp4Info app gets this:

File:
major brand: mp42
minor version: 0

Movie:
duration: 6784 ms
time scale: 90000
fragments: no

Found 1 Tracks
Track 1:
flags: 7 ENABLED IN-MOVIE IN-PREVIEW
id: 1
type: Video
duration: 6784 ms
language: ```
media:
sample count: 197
timescale: 90000
duration: 593966 (media timescale units)
duration: 6600 (ms)
bitrate (computed): 11886.091 Kbps
frame rate (computed): 29.850
Sample Description 0
Coding: avc1 (H.264)
Width: 1280
Height: 720
Depth: 24
AVC Profile: 66 (Baseline)
AVC Profile Compat: 80
AVC Level: 31
AVC NALU Length Size: 4
AVC SPS: [6742801fda014016e806d0a135]
AVC PPS: [68ce06e2]
Codecs String: avc1.42801F

[Grinchman]

Ok then ) Now I got video which is partially played. Video player shows 6 sec. duration but plays only 2 sec. + orientation of video is wrong.
Code is (I looked to AP4_Track.Clone() code to create own new version):

// create a sample table
AP4_SyntheticSampleTable* sample_table = new AP4_SyntheticSampleTable();

AP4_SyntheticSampleTable* sample_video = (AP4_SyntheticSampleTable*)track_video->GetSampleTable();
AP4_SyntheticSampleTable* sample_video2 = (AP4_SyntheticSampleTable*)track_video2->GetSampleTable();

AP4_SampleDescription* description_video = track_video->GetSampleDescription(0);


// add clones of the sample descriptions to the new sample table
for (unsigned int i=0; ;i++) {
    AP4_SampleDescription* sample_description = sample_video->GetSampleDescription(i);
    if (sample_description == NULL) break;
    sample_table->AddSampleDescription(sample_description->Clone());
}

AP4_Sample  sample;
AP4_Ordinal index = 0;
while (AP4_SUCCEEDED(sample_video->GetSample(index, sample)))
{
    AP4_ByteStream* data_stream;
    data_stream = sample.GetDataStream();
    sample_table->AddSample(*data_stream,
                            sample.GetOffset(),
                            sample.GetSize(),
                            sample.GetDuration(),
                            sample.GetDescriptionIndex(),
                            sample.GetDts(),
                            sample.GetCtsDelta(),
                            sample.IsSync());
    AP4_RELEASE(data_stream); // release our ref, the table has kept its own ref.
    index++;
}

index = 0;
while (AP4_SUCCEEDED(sample_video2->GetSample(index, sample)))
{
    AP4_ByteStream* data_stream;
    data_stream = sample.GetDataStream();
    sample_table->AddSample(*data_stream,
                            sample.GetOffset(),
                            sample.GetSize(),
                            sample.GetDuration(),
                            sample.GetDescriptionIndex(),
                            sample.GetDts(),
                            sample.GetCtsDelta(),
                            sample.IsSync());
    AP4_RELEASE(data_stream); // release our ref, the table has kept its own ref.
    index++;
}


AP4_UI32 duration_movie1     = track_video->GetDuration();
AP4_UI32 duration_movie2     = track_video2->GetDuration();

AP4_UI32 duration_media1      = track_video->GetMediaDuration();
AP4_UI32 duration_media2      = track_video2->GetMediaDuration();

// create the cloned track
AP4_Track* clone = new AP4_Track(track_video->GetType(),
                                 sample_table,
                                 0,
                                 track_video->GetMovieTimeScale(),
                                 duration_movie1 + duration_movie2,
                                 track_video->GetMediaTimeScale(),
                                 duration_media1 + duration_media2,
                                 track_video->GetTrackLanguage(),
                                 track_video->GetWidth(),
                                 track_video->GetHeight());

output_movie->AddTrack(clone);

// create a multimedia file
AP4_File file(output_movie);

// setup the brands
AP4_Array<AP4_UI32> brands;
brands.Append(AP4_FILE_BRAND_ISOM);
brands.Append(AP4_FILE_BRAND_MP42);
brands.Append(AP4_FILE_BRAND_AVC1);

// set the file type
file.SetFileType(AP4_FILE_BRAND_MP42, 0);

// write the file to the output
AP4_FileWriter::Write(file, *output);

// cleanup
output->Release();

mp4info app result:

File:
major brand: mp42
minor version: 0

Movie:
duration: 6164 ms
time scale: 1000
fragments: no

Found 1 Tracks
Track 1:
flags: 7 ENABLED IN-MOVIE IN-PREVIEW
id: 1
type: Video
duration: 6164 ms
language: ```
media:
sample count: 71
timescale: 90000
duration: 554767 (media timescale units)
duration: 6164 (ms)
bitrate (computed): 4649.342 Kbps
display width: 1280.000000
display height: 720.000000
frame rate (computed): 11.518
Sample Description 0
Coding: avc1 (H.264)
Width: 1280
Height: 720
Depth: 24
AVC Profile: 66 (Baseline)
AVC Profile Compat: 80
AVC Level: 31
AVC NALU Length Size: 4
AVC SPS: [6742801fda014016e806d0a135]
AVC PPS: [68ce06e2]
Codecs String: avc1.42801F

I feel that the Truth is near ))) Please help to stop my pains! )

[barbibulle]

I think you're almost there. I fear that there may be a small issue with calling sample_table->AddSample directly, but since you're now calling the more detailed AddSample() method, that should avoid that issue. The last remaining problem is probably because when you add the samples of the second track, the sample timestamps won't work, as they need to be shifted by the amount of the first track. What you should do it call sample.SetDts(0) before adding the samples (don't change the CtsDelta or the Duration). This way the DTS will be recomputed for you automatically. I'll try to find some time today and double check that this is the only missing part here.

[Grinchman]

Great! That works! Also I successfully added audio samples and now merged video is played well! BUT! There is one more problem: video orientation. I shot videos in portrait orientation while device's sensor is in landscape. And result video is rotated by -90 degrees (

[barbibulle]

The 90 degree rotation is probably related to some missing signaling in the sample description.
The video itself doesn't change, but there is sometimes as parameter in the track headers that provide a hint to the player that the video should be rotated when displayed.
Can you share an example of a video that exhibits the problem? What player are you using to display the video (I assume the player correctly displays the original videos that you are concatenating)

[Grinchman]

I share 2 input videos and result video in dropbox:
https://www.dropbox.com/s/9o370kl3dc6oou4/20150521_result_video.mp4?dl=0 - RESULT
https://www.dropbox.com/s/45d8prrx9hsvofa/20150521_video_01.mp4?dl=0 - INPUT 01
https://www.dropbox.com/s/zbnlhtt1zbzs3k5/20150521_video_02.mp4?dl=0 - INPUT 02

[barbibulle]

The reason the output video doesn't have the proper orientation is because the AP4_Track object was created without the same transform matrix as the original (not your fault, the sample code does the same).
I've added a new AP4_Track constructor that allows you to pass an existing AP4_Track object as a prototype to "inherit" the settings from, including the transform matrix (note that the notion of a non-default transform matrix is something that's specific to Quicktime, not all MP4 players will support it).
So at the point where you call new AP4_Track(), call this version of the constructor:
AP4_Track(AP4_SampleTable* sample_table, // ownership is transfered to the AP4_Track object
AP4_UI32 track_id,
AP4_UI32 movie_time_scale, // 0 = use default
AP4_UI64 track_duration, // in the movie timescale
AP4_UI32 media_time_scale,
AP4_UI64 media_duration, // in the media timescale
const AP4_Track* track_prototype); // prototype for the type and other track fields

(pass the original track pointer instead of language, width and height).

This is part of the latest commit in the master branch (not yet in the pre-compiled binaries)

[Grinchman]

Awesome! Seems result video now is absolutely correct! Thank you so much for you time!