Seeking / Scrubbing

[addroid]

I'm attempting to add a seek function to this. I've been attempting to manipulate the buffer index to seek, but haven't had much luck. Not sure I'm fully wrapping my head around the frame decoding aspect of the BitArray. Any chance you'll be adding a seek function? Preferably to a given time in the video?

[phoboslab]

Seeking to intra frames (those that don't rely on the previous frames) shouldn't be all too difficult. What I would do is loop through the whole buffer once and collect all START_PICTURE codes that are followed by the PICTURE_TYPE_I bits. Have a look at receiveSocketMessage - it has an option to skip anything that's not an intra frame.

Something like this would probably do (untested):

var intraFrames = [];

while( true ) {
    var code = this.buffer.findNextMPEGStartCode();
    if( code == START_PICTURE ) {
        // We found a START_PICTURE code. Let's check if it's an
        // intra frame
        this.buffer.advance(10); // skip temporalReference
        if( this.buffer.getBits(3) == PICTURE_TYPE_I ) {
            // It's an intra frame! Remember the position 13 bits back
            // at the temporalReference start.
            intraFrames.push(this.buffer.index - 13);
        }
    }
    else if( code == BitReader.NOT_FOUND ) {
        break;
    }
}

edit: fixed the code as @crowjonah notes below

So now intraFrames contains all indices of intra frames within the buffer. All you have to do to jump to one of those is to set this.buffer.index = intraFrames[n]; and call this.decodePicture(). Of course you can do a lot more to improve it. For instance count all START_PICTURE codes and calculate the time for each of the intra frames.

If you need finer grained seeking to non-intra frames, you have to jump to the closest intra frame before your jump target and then decode all pictures starting frome there.

[crowjonah]

This worked perfectly for me! Since I'm using static files (unstreamed), I gathered my intraFrames in loadCallback, and made sure to reset this.buffer.index = 0; before autoplaying. Also, just a note for anyone else copy+pasting: intraFrames = this.buffer.index - 13; should be intraFrames.push(this.buffer.index - 13);.

[shanytc]

I wasn't able to see to a specific Frame, i've tried the code above, however the PICTURE_TYPE_I returns only intra ? key frames, not ALL frames... any luck on having a function which can see to a specific frame?

[andyearnshaw]

@shanytc all non-intra frames rely on the previous frame being decoded first, a key factor in compression. If you want to seek to a specific frame, you'd first have to decode the intra frame before it, then all the frames following, skipping drawing until you get to the desired frame.

I wouldn't recommend doing this, however, as it could potentially take a long time to decode all those frames.

[shanytc]

@andyearnshaw
Does it mean that if I want to go to frame 54 for example,
I have to scan all Intra keys to the one most closes to 54 (say instra is 39 since the next one is 58) and then loop 15 times (54-39) the function findNextMPEGStartCode() just to be on frame 54? and then update the canvas with decodePicture()? I guess this.buffer.index points to the current frame i need to scan from after i find an intra right?

maybe @phoboslab can shed some light?

I basically want to attach a seek bar to the jsmpeg decoded file and scrub through the video back and forward in real time.

Other option I've had in mind is to loop all frames with findNextMPEGStartCode() and decodePicture() (do i still need to check for intra frames here?) and then dump the image from the canvas and store that image into an array of images so i can scrub through all images instead of the mpeg itself...

[phoboslab]

Modern encoders usually include one intra frame about every 30 frames (or once per second). So if you control the encoding step, you can make sure your videos include intra frames often enough. This would probably be good enough to implement a seekbar, depending on your use case. Many video hosters and their flash players only allow you to jump to intra frames as well.

Otherwise, yes, you have to jump to the closest intra frame and decode(!) all frames in between this intra frame and your jump target. The decoding is necessary, because all frames following an intra frame only encode changes based on the previous frame and rely on the previously decoded YCrCb buffers.

jsmpeg includes an option to skip the final YCrCb to RGB conversion when decoding, which speeds things up a bit: this.decodePicture(DECODE_SKIP_OUTPUT);

Your other solution, storing decoded frames, would need a lot of time to initially decode all frames and a lot of RAM to keep those images in memory. That's probably not feasible for videos longer than a few seconds.

[shanytc]

@phoboslab I will try to work on it! But it would be nice if you can (because you coded this awesome library) also include an option to seek a frame?

[phoboslab]

Looking into right now!

[shanytc]

@phoboslab THANK YOU!!!!!

[phoboslab]

// If you pass 'seekable: true' in the options, you can seek to a specific frame
// or time in the video.

var player = new jsmpeg('file.mpeg', {canvas: canvas, seekable: true});

player.seekToFrame(1200); // Seek to intra frame before frame 1200
player.seekToTime(20); // Seek to intra frame before 20sec

// seekToFrame() and seekToTime() only seek to the closest, previous intra frame by
// default. If you want to seek to the exact frame or time, pass 'true' as second
// parameter.
// Depending on the input video, this can be potentially slow, as jsmpeg has
// to decode all frames between the previous intra frame and the seek target

player.seekToFrame(1200, true); // Seek to frame 1200 exactly

[shanytc]

@phoboslab
Awesome update! it works awesome!!
I noticed thought that, when seeking to some frame towards the END of a movie and then seek to the next frame (using the same function), it won't display the right image.but when i hit PLAY, it WILL play the right frames after.

example:
total frames: 560
seek to frame: 555 (show an image)
seek to frame: 556 (shows same image)
but when hit play from frame 555, it will show the images just fine.

Also I've notived that when seeking to a frame next to an intra frame, it will show the same frame twice when seeking to the next frame.

example: seek to frame 13 -> seek to frame 14 (shows frame 13 instead of 14)

[phoboslab]

You're right! The above commit should fix this. Thanks!

[shanytc]

@phoboslab yes, that -1 removal indeed fixes that (in the beginning of a video), though i am still getting duplicated frames when seeking in mid video.. wonder why..

As for the end of the movie frames bug, did you manage to check it out? the play() function indeed plays through all the frames, but seeking to the few frames towards the end and then seeking to the next frame after will not display the next frame.

[phoboslab]

You're passing true to the seek function, to indicate exact seeking, right? I.e.:

player.seekToFrame(1234, true);

Do you have a test case? I can't seem to reproduce this with the Big Buck Bunny video.

[shanytc]

@phoboslab i am using this file for testing

https://dl.dropboxusercontent.com/u/3038723/movie.mpg

could be the ffmpeg conversion is not accurate?

and yes, i am using TRUE in the seekToFrame() function

[phoboslab]

So, I noticed you can't seek to or past the last intra frame in the file. Thanks again :)

[shanytc]

@phoboslab Yeah this works awesome!!!! Just checked it out ;)

As for the duplicated frames (intra shows twice?) what do you think?

You can see the duplicated frames in this video here:
https://dl.dropboxusercontent.com/u/3038723/movie3.mpg

frame #12 is duplicated when seeking to it (12, 13)
so I wonder if it's still a bug or frame rate conversion issue?

[phoboslab]

Looks like a frame rate conversion issue. Frame 37 & 38 and 61 & 62 are the same as well. Thats one dup every 25 frames - the frame rate of the mpeg is 23.98 so I assume the source video was 25fps.

Just encode the mpeg with 25fps as well:

ffmpeg -i movie.mp4 -r 25 movie.mpg

[shanytc]

Ok great, I suspected it might be a frame rate issue, just wanted to make sure. Thanks @phoboslab you're awesome!