Frame order from TS demux - mpeg2video vs h264

[jettoblack]

Hello,
I have a question about determining the display order of picture frames when demuxing a MPEG-2 transport stream that contains either MPEG-2 video or H.264 video.

I'm using ffmpeg git head latest as of the past couple days.  My code uses libavformat to demux the stream, add some metadata which must be in picture display order, then remux the stream.  I have a loop to get packets from the source, modify them if it's a video packet, and write packets back out to the output muxer.  This is working fine for MPEG-2 video streams but I'm having some issues getting the proper frame order when demuxing TS with H.264 video.  I tried decoding the codec to a picture and reading picture.coded_picture_number but this doesn't always give the right display order either.  Is there a better way?

Both test files were created with ffmpeg latest and from the same source, only difference is one is MPEG-2 video/audio and the other is H.264/AAC.

Here is the basic loop, edited to be concise (full compilable source code included in link below):

    while (1) {
        av_init_packet(&pkt);
        r = av_read_frame(in, &pkt);

        if (pkt.stream_index == videoStream) {
            avcodec_decode_video2(in->streams[videoStream]->codec, &picture, &got_picture, &pkt);
            if (got_picture) {
                printf("picture_packet %d, coded_picture_number %d, pts %"PRId64"\n", picture_packets, picture.coded_picture_number, pkt.pts);
               
                // very simple test to insert a byte that should increase in displayed frame order
                // note: I know this is not a proper way to insert extra data, this is for simple testing only
                av_grow_packet(&pkt, 1);
                pkt.data[pkt.size - 1] = (uint8_t)(picture.coded_picture_number & 0xff);
               
                picture_packets++;
            }
        }
       
        // write frame
        r = av_interleaved_write_frame(out, &pkt);
        av_free_packet(&pkt);        
    }

When the source file has MPEG-2 video, the frames come out of the demuxer in frame display order:
picture_packet 0, coded_picture_number 0, pts 129003
picture_packet 1, coded_picture_number 1, pts 132006
picture_packet 2, coded_picture_number 2, pts 135009
picture_packet 3, coded_picture_number 3, pts 138012

So my code is working fine for MPEG-2 video streams.  I can read the output stream and when demuxed back into picture display order, all of my metadata is in the proper order.

However when the source file contains H.264 video, the packets are not demuxing in picture display order, nor is the pattern consistent:
picture_packet 0, coded_picture_number 0, pts 156000
picture_packet 1, coded_picture_number 2, pts 153150
picture_packet 2, coded_picture_number 1, pts 162150
picture_packet 3, coded_picture_number 4, pts 159150
...
picture_packet 146, coded_picture_number 145, pts 595200
picture_packet 147, coded_picture_number 147, pts 592350
picture_packet 148, coded_picture_number 148, pts 601350
picture_packet 149, coded_picture_number 150, pts 598350
picture_packet 150, coded_picture_number 149, pts 607350


If I make my algorithm depend on the coded_picture_number, the results are right most of the time, but every 150 frames or so, I end up getting 2 frames swapped in order, as if the coded_picture_number is getting swapped once in a while.  E.g. when decoding the resulting output stream, my metadata ends up with the order "144 145 146 148 147 149 150..." instead of being in order.  Notice above that the pts for coded_picture_number 150 is less than the pts for coded_picture_number 148 or 149.

Any ideas?  What is the best way to reliably tell get the frames' picture display order when demuxing a MPEG-2 TS w/ H.264 video?  Is there a way to convert PTS back into a display-order frame number?

Source code: http://dl.dropbox.com/u/7730988/main.c.zip
Two sample videos (3MB): http://dl.dropbox.com/u/7730988/sotu11.zip

Thanks!

[Kalileo]

The decoder gives you the (decoded) pictures in the correct order.

The PTS value _should_ reflect that order.

I would try to attach the metadata in the order as you get it from the demuxer with the pictures as you get them out of the decoder.

Basically, it is exactly the same syncing you have to do with audio and video. With h264 the order is in the pts value (or as the decoded frames come out of the decoder.)


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[jettoblack]

On Mon, Apr 30, 2012 at 2:51 AM, Kalileo [via libav-users] <[hidden email]> wrote:

The decoder gives you the (decoded) pictures in the correct order.

That's what I expected, and it does for MPEG-2 codec video, but not for H.264 codec within a MPEG-2 transport stream.

 

I would try to attach the metadata in the order as you get it from the demuxer with the pictures as you get them out of the decoder.

I tried that and for H.264 it's definitely wrong if I use the frames in-order from the demuxer.  It does work fine for MPEG-2 video.

Should I file a bug report?

Thanks,

Jason

[Carl Eugen Hoyos]

jettoblack <jettoblack@...> writes:

> I tried that and for H.264 it's definitely wrong if I use the frames
> in-order from the demuxer.  It does work fine for MPEG-2 video.
>
> Should I file a bug report?

Is the problem reproducible with ffmpeg -i input -qscale 2 out.avi
or ffplay file ?

Carl Eugen

_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[jettoblack]

On Tue, May 1, 2012 at 2:20 AM, Carl Eugen Hoyos [via libav-users] <[hidden email]> wrote:

jettoblack <jettoblack@...> writes:

> I tried that and for H.264 it's definitely wrong if I use the frames
> in-order from the demuxer.  It does work fine for MPEG-2 video.
>
> Should I file a bug report?

Is the problem reproducible with ffmpeg -i input -qscale 2 out.avi
or ffplay file ?

Hi Carl, I really appreciate the response.

The output file plays/trancodes without any problems using ffplay or ffmpeg.  I don't know much about the internals of ffmpeg, but is it possible that the muxing process is reversing the issue? 

In any case, can you advise on how to determine the frame display index of a AVPacket?  I've tried:

A) the incremental order from the demuxer

B) using the pts value: int frame_num = (pkt.pts * av_q2d(in->streams[videoStream]->time_base)) / (1 / frameRate);

C) coded_picture_number (after decoding to AVPicture)

When the transport stream contains MPEG-2 video codec, all of these methods work equally well.  When it contains H.264 video codec, the pts and incremental order from the demuxer are definitely not in picture display order.  coded_picture_number is giving the closest result but it is wrong on approx. every 150th frame (key frames?).

Thanks!

Jason

[Carl Eugen Hoyos]

jettoblack <jettoblack@...> writes:

> When it contains H.264 video codec, the pts and
> incremental order from the demuxer are definitely not in
> picture display order.  coded_picture_number is giving the
> closest result but it is wrong on approx. every 150th
> frame (key frames?).

Are you using the parser?
(Do you see the same results with ffmpeg if you configure
with --disable-parser=h264 ?)

Carl Eugen

_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[Andrey Utkin]

Dear topicstarter.
You print the pts timestamps of demuxed packets. It is correct that
they are not sorted by pts. They are ordered by dts, the Decoding
Timestamp. They are in the order for being decoded. You should look at
pts of resulting frames (actually i use AVFrame.pkt_pts, as i see in
my apps sources). Decoders return frames in _pts_ order, that is what
you need.

--
Andrey Utkin
_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[Kalileo]

On Apr 30, 2012, at 23:53 , jettoblack wrote:

> On Mon, Apr 30, 2012 at 2:51 AM, Kalileo [via libav-users] <
> [hidden email]> wrote:
>
>>
>> The decoder gives you the (decoded) pictures in the correct order.
>>
>>
> That's what I expected, and it does for MPEG-2 codec video, but not for
> H.264 codec within a MPEG-2 transport stream.

It does. Otherwise there would be no way to display h264 video correctly ;)

As i said, and as also Andrey Utkin pointed out, the correct order for display is in the pts value, and after decoding (not after demuxing) the pictures/frames should be in pts order, ready for display.

>
>> I would try to attach the metadata in the order as you get it from the
>> demuxer with the pictures as you get them out of the decoder.
>>
>
> I tried that

No you haven't.

> and for H.264 it's definitely wrong if I use the frames
> in-order from the demuxed.

as said, for h264 with b frames you need the order _not_ "as demuxed" but "as decoded", and I'm not talking about any 'coded_picture_number' but about the order of the h264 frames as the decoder returns them. That is the correct order, and the pts should reflect that order. If it doesn't, trust the order the decoder gives you over the pts.

>  It does work fine for MPEG-2 video.

but not in h264 because in h264 the coded frames are reordered depending on which frame is needed for decoding first, not which frame is to be displayed first.

>
> Should I file a bug report?

No. You expect info from the demuxer which is not there.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[jettoblack]

Kalileo wrote

On Apr 30, 2012, at 23:53 , jettoblack wrote:

> On Mon, Apr 30, 2012 at 2:51 AM, Kalileo [via libav-users] <
> [hidden email]> wrote:
>
>>
>> The decoder gives you the (decoded) pictures in the correct order.
>>
>>
> That's what I expected, and it does for MPEG-2 codec video, but not for
> H.264 codec within a MPEG-2 transport stream.

It does. Otherwise there would be no way to display h264 video correctly ;)

As i said, and as also Andrey Utkin pointed out, the correct order for display is in the pts value, and after decoding (not after demuxing) the pictures/frames should be in pts order, ready for display.

Thanks Kalileo, I think I'm getting it now. :)

So, related question, I want to figure out what is the starting pts of the stream (lowest pts).  Is this info available from the demuxer by the time of demuxing the first picture packet, or do I have to wait until I receive some number of packets and look for the one with the lowest pts?

I have seen some files where the first packet is an I frame, and the next is a B frame that precedes the I frame (BBIBBP...), e.g.:
pkt 0: I frame dts 126000 pts 135009
pkt 1: B frame dts 129003 pts 129003
pkt 2: B frame dts 132006 pts 132006
pkt 3: P frame dts 135009 pts 144018
...

I suppose at the time of demuxing packet 0, there is no way to know yet that the next packet will have a lower pts?  

Would a workaround be to discard any frames with a pts lower than the first I frame?

I really appreciate the help, thanks again!
Jason

[Kalileo]

On May 4, 2012, at 00:16 , jettoblack wrote:

>
> Kalileo wrote
>>
>> On Apr 30, 2012, at 23:53 , jettoblack wrote:
>>
>>> On Mon, Apr 30, 2012 at 2:51 AM, Kalileo [via libav-users] <
>>> ml-node+s943685n4597541h88@.nabble> wrote:
>>>
>>>>
>>>> The decoder gives you the (decoded) pictures in the correct order.
>>>>
>>>>
>>> That's what I expected, and it does for MPEG-2 codec video, but not for
>>> H.264 codec within a MPEG-2 transport stream.
>>
>> It does. Otherwise there would be no way to display h264 video correctly
>> ;)
>>
>> As i said, and as also Andrey Utkin pointed out, the correct order for
>> display is in the pts value, and after decoding (not after demuxing) the
>> pictures/frames should be in pts order, ready for display.
>>
>>
>
> Thanks Kalileo, I think I'm getting it now. :)
>
> So, related question, I want to figure out what is the starting pts of the
> stream (lowest pts).  Is this info available from the demuxer by the time of
> demuxing the first picture packet, or do I have to wait until I receive some
> number of packets and look for the one with the lowest pts?
>
> I have seen some files where the first packet is an I frame, and the next is
> a B frame that precedes the I frame (BBIBBP...), e.g.:
> pkt 0: I frame dts 126000 pts 135009
> pkt 1: B frame dts 129003 pts 129003
> pkt 2: B frame dts 132006 pts 132006
> pkt 3: P frame dts 135009 pts 144018
> ...
>
> I suppose at the time of demuxing packet 0, there is no way to know yet that
> the next packet will have a lower its?  

 in a stream with h264, yes.

>
> Would a workaround be to discard any frames with a pts lower than the first
> I frame?
>
> I really appreciate the help, thanks again!
> Jason
>

That's how I do it, and it works for me. Wait for the first I-frame and start feeding the video frames to the decoder with this first I-frame.

I remember that I've seen video frames with a lower pts even after such an I-frame, however I ignore that pts, but not such a frame. After the first I-Frame they all go to the decoder.

The pts / dts of that first I-frame is what you sync with the dts / pts of the first audio or whatever frame you take and sync from there.
_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user

[Carl Eugen Hoyos]

Kalileo <kalileo@...> writes:

> That's how I do it, and it works for me. Wait for the first I-frame

Don't forget that valid H264 streams do not necessarily contain I frames.
(I don't know of such a sample.)

Carl Eugen

_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user