[Libav-user] MPEG2 decoding pixelation problem (but not in ffplay)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Libav-user] MPEG2 decoding pixelation problem (but not in ffplay)

Michael Goffioul
Hi,

I'm trying to use ffmpeg to decode a MPEG2 stream and I'm running into pixelation problem with some streams (not all of them). The fact is that the problematic streams play fine in ffplay (or vlc), so I figured the problem must be in the way I setup the decoder or feed data to it.

I've written a very simple code to illustrate the problem I'm having, it's available here:

It doesn't do anything fancy: just initialize the mpeg2video codec, feed the data (segmented on the picture start code boundary), drain frames, convert to RGB and save as PNG. It accepts 2 arguments: the elementary MPEG2 stream filename and the number of frames to extract.

I'm using this sample clip as test case, it's the elementary video stream extracted from a MPEG/TS clip (next link):

The pixelation problem is illustrated in this extracted frame:

By comparison, playing the sample clip (elementary stream) with ffplay gives this:

Clearly I must be doing something wrong. Does anybody have a hint or suggestion?

Thanks,
Michael.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: MPEG2 decoding pixelation problem (but not in ffplay)

Jaka Bac
Hi Michael,

You are parsing the mpeg2 stream by yourself looking for the 0x00 0x00 0x01 0x00 startcode which means that you will only feed the mpeg2 encoded pictures to the decoder.
This way your program will skip over the sequence headers which may include a lot of important information for the mpeg2 decoder (for example custom intra and non intra quantiser matrices, picture dimensions, aspect ratio,...)

The ffmpeg mpeg2video codec will try its best to decode the pictures, but with the missing info it may not be fully successful.
Your PussInBoots file for example, includes custom quantiser matrices which are probably slightly different from the default ones (I did not look in detail).

My suggestion would be to use libavformat to read the source files. This way you would get correctly parsed AVPacket(s).

If you for some reason can't use libavformat, you could at least use av_parser_parse2 function to parse the stream for you. (See example here: https://www.ffmpeg.org/doxygen/trunk/decode_video_8c-example.html)

Hope that helps,
Jaka

On 4 June 2017 at 05:12, Michael Goffioul <[hidden email]> wrote:
Hi,

I'm trying to use ffmpeg to decode a MPEG2 stream and I'm running into pixelation problem with some streams (not all of them). The fact is that the problematic streams play fine in ffplay (or vlc), so I figured the problem must be in the way I setup the decoder or feed data to it.

I've written a very simple code to illustrate the problem I'm having, it's available here:

It doesn't do anything fancy: just initialize the mpeg2video codec, feed the data (segmented on the picture start code boundary), drain frames, convert to RGB and save as PNG. It accepts 2 arguments: the elementary MPEG2 stream filename and the number of frames to extract.

I'm using this sample clip as test case, it's the elementary video stream extracted from a MPEG/TS clip (next link):

The pixelation problem is illustrated in this extracted frame:

By comparison, playing the sample clip (elementary stream) with ffplay gives this:

Clearly I must be doing something wrong. Does anybody have a hint or suggestion?

Thanks,
Michael.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: MPEG2 decoding pixelation problem (but not in ffplay)

Michael Goffioul
Hi Jaka,

Thanks for your answer. Even though I'm segmenting the data on 0x00 0x00 0x01 0x00, I'm still feeding the entire data between those boundaries to the decoder. So I'm not skipping any data from the stream. Referencing my sample code: p1 points to the current picture start code, p2 points to the next picture start code, feed data between p1 and p2, p1 = p2, restart.

Is there a particular way to segment the data that is expected by the mpeg2video decoder?

Michael.



On Tue, Jun 6, 2017 at 4:42 PM, Jaka Bac <[hidden email]> wrote:
Hi Michael,

You are parsing the mpeg2 stream by yourself looking for the 0x00 0x00 0x01 0x00 startcode which means that you will only feed the mpeg2 encoded pictures to the decoder.
This way your program will skip over the sequence headers which may include a lot of important information for the mpeg2 decoder (for example custom intra and non intra quantiser matrices, picture dimensions, aspect ratio,...)

The ffmpeg mpeg2video codec will try its best to decode the pictures, but with the missing info it may not be fully successful.
Your PussInBoots file for example, includes custom quantiser matrices which are probably slightly different from the default ones (I did not look in detail).

My suggestion would be to use libavformat to read the source files. This way you would get correctly parsed AVPacket(s).

If you for some reason can't use libavformat, you could at least use av_parser_parse2 function to parse the stream for you. (See example here: https://www.ffmpeg.org/doxygen/trunk/decode_video_8c-example.html)

Hope that helps,
Jaka

On 4 June 2017 at 05:12, Michael Goffioul <[hidden email]> wrote:
Hi,

I'm trying to use ffmpeg to decode a MPEG2 stream and I'm running into pixelation problem with some streams (not all of them). The fact is that the problematic streams play fine in ffplay (or vlc), so I figured the problem must be in the way I setup the decoder or feed data to it.

I've written a very simple code to illustrate the problem I'm having, it's available here:

It doesn't do anything fancy: just initialize the mpeg2video codec, feed the data (segmented on the picture start code boundary), drain frames, convert to RGB and save as PNG. It accepts 2 arguments: the elementary MPEG2 stream filename and the number of frames to extract.

I'm using this sample clip as test case, it's the elementary video stream extracted from a MPEG/TS clip (next link):

The pixelation problem is illustrated in this extracted frame:

By comparison, playing the sample clip (elementary stream) with ffplay gives this:

Clearly I must be doing something wrong. Does anybody have a hint or suggestion?

Thanks,
Michael.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: MPEG2 decoding pixelation problem (but not in ffplay)

Jaka Bac
Hi Michael,

mpeg2video wants the packets to be segmented in a way that logical things go together and that there is no extra "garbage" at the end.
One problem that you have is that your code will feed mpeg2video with last frame in a GOP and the sequence header of the next gop after it. That is not desired by mpeg2video.

The other problem that you have is that you are triggering undefined behaviour with this line in your code:
p2 = memmem(p1 + 4, n, "\x00\x00\x01\x00", 4);

from address at p1 there is at least n bytes in the buffer if you are doing the calculations correctly (which I think that you do).
but you are searching from p1+4 up n bytes which goes past the end of your buffer. If there are 00 after the end of your buffer memmem might find a "false" start code.

I think that the line in question should be:
p2 = (char*)memmem(p1 + 4, n-4, "\x00\x00\x01\x00", 4);

Probably you can get away with the first problem (headers of the next gop after end of previous gop) since you feed the sequence headers at the start of the stream to the decoder (the other sequence headers are just repeated values)

All in all, you would be better off with already tested parsers and demuxers from libavformat

Jaka


On 6 June 2017 at 22:56, Michael Goffioul <[hidden email]> wrote:
Hi Jaka,

Thanks for your answer. Even though I'm segmenting the data on 0x00 0x00 0x01 0x00, I'm still feeding the entire data between those boundaries to the decoder. So I'm not skipping any data from the stream. Referencing my sample code: p1 points to the current picture start code, p2 points to the next picture start code, feed data between p1 and p2, p1 = p2, restart.

Is there a particular way to segment the data that is expected by the mpeg2video decoder?

Michael.



On Tue, Jun 6, 2017 at 4:42 PM, Jaka Bac <[hidden email]> wrote:
Hi Michael,

You are parsing the mpeg2 stream by yourself looking for the 0x00 0x00 0x01 0x00 startcode which means that you will only feed the mpeg2 encoded pictures to the decoder.
This way your program will skip over the sequence headers which may include a lot of important information for the mpeg2 decoder (for example custom intra and non intra quantiser matrices, picture dimensions, aspect ratio,...)

The ffmpeg mpeg2video codec will try its best to decode the pictures, but with the missing info it may not be fully successful.
Your PussInBoots file for example, includes custom quantiser matrices which are probably slightly different from the default ones (I did not look in detail).

My suggestion would be to use libavformat to read the source files. This way you would get correctly parsed AVPacket(s).

If you for some reason can't use libavformat, you could at least use av_parser_parse2 function to parse the stream for you. (See example here: https://www.ffmpeg.org/doxygen/trunk/decode_video_8c-example.html)

Hope that helps,
Jaka

On 4 June 2017 at 05:12, Michael Goffioul <[hidden email]> wrote:
Hi,

I'm trying to use ffmpeg to decode a MPEG2 stream and I'm running into pixelation problem with some streams (not all of them). The fact is that the problematic streams play fine in ffplay (or vlc), so I figured the problem must be in the way I setup the decoder or feed data to it.

I've written a very simple code to illustrate the problem I'm having, it's available here:

It doesn't do anything fancy: just initialize the mpeg2video codec, feed the data (segmented on the picture start code boundary), drain frames, convert to RGB and save as PNG. It accepts 2 arguments: the elementary MPEG2 stream filename and the number of frames to extract.

I'm using this sample clip as test case, it's the elementary video stream extracted from a MPEG/TS clip (next link):

The pixelation problem is illustrated in this extracted frame:

By comparison, playing the sample clip (elementary stream) with ffplay gives this:

Clearly I must be doing something wrong. Does anybody have a hint or suggestion?

Thanks,
Michael.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user



_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: MPEG2 decoding pixelation problem (but not in ffplay)

Michael Goffioul
On Tue, Jun 6, 2017 at 7:05 PM, Jaka Bac <[hidden email]> wrote:
Hi Michael,

mpeg2video wants the packets to be segmented in a way that logical things go together and that there is no extra "garbage" at the end.
One problem that you have is that your code will feed mpeg2video with last frame in a GOP and the sequence header of the next gop after it. That is not desired by mpeg2video.

You're right. I had done some experiments in the meantime and it indeed appears one cannot feed data to mpeg2video codec without segmenting into logical units.
 

The other problem that you have is that you are triggering undefined behaviour with this line in your code:
p2 = memmem(p1 + 4, n, "\x00\x00\x01\x00", 4);

from address at p1 there is at least n bytes in the buffer if you are doing the calculations correctly (which I think that you do).
but you are searching from p1+4 up n bytes which goes past the end of your buffer. If there are 00 after the end of your buffer memmem might find a "false" start code.

Yup, I also caught that problem and had already fixed it.
 

I think that the line in question should be:
p2 = (char*)memmem(p1 + 4, n-4, "\x00\x00\x01\x00", 4);

Probably you can get away with the first problem (headers of the next gop after end of previous gop) since you feed the sequence headers at the start of the stream to the decoder (the other sequence headers are just repeated values)

I changed the way I do segmentation, such that sequence and other non-picture-related headers are isolated and fed separately to the decoder. This fixed the pixellation problem. 
 

All in all, you would be better off with already tested parsers and demuxers from libavformat

The sample code was just intended to illustrate the problem, it's not production code. The full context is that I'm trying to bridge ffmpeg and ExoPlayer for video decoding. The simple (and wrong) segmentation used in my sample code is based on the way ExoPlayer is segmenting a MPEG2 video stream (H262Reader).

The starting point was when I tried to use ExoPlayer with the Google software MPEG2 decoder available in Android (since Marshmallow). I observed bad pixellation. ExoPlayer devs put the blame on the MPEG2 decoder from Google. I then integrated ffmpeg with ExoPlayer for video decoding, but I was suprized to observe the same bad pixellation. That's how I ended up writing my sample code, to emulate what ExoPlayer and my ffmpeg integration were doing. Given that both Google's and ffmpeg's decoder show the same pixellation, I bet now that the root cause is ExoPlayer's segmentation.

Thanks again for your support, that really helped.

Michael.


_______________________________________________
Libav-user mailing list
[hidden email]
http://ffmpeg.org/mailman/listinfo/libav-user