Need help with using libav* in multithreaded code

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help with using libav* in multithreaded code

Alexander Benz
Hi there,

I'm trying to setup a partly multi threaded video pipeline but my result
images come out distorted from time to time.
It seems like some lines of the video contain lines from the previous
frames ... but I can't find the exact cause. Maybe
anyone has experienced a similar effect or knows the answer to this
problem.

In general, what I do is the following :

Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
avcodec_open etc.) then

1. Read a frame from the stream (av_read_frame)
2. Decode frame (avcodec_decode_video)
3. Do color conversion (YUV --> RGB)
4. Display Image

Steps 1 and 2 are done by libav*, step 3 is done by some different
library. Each step is one part in my video
pipeline where (1) delivers an AVPacket to (2), (2) delivers an AVFrame
to (3) and (3) delivers a RGB buffer
to (4). All those steps should run in parallel, so as soon as one frame
is read and delivered, the next frame will be
read while the current one is being decoded and so on. Of course, each
packet and each frame get their own
AVPacket / AVFrame buffer via "new AVPacket()" and "avcodec_alloc_frame"
respectively.
Unfortunately this produces partly distorted frames as described above.
As long as I keep the libav* stuff
in one thread (e.g. Read, decode, copy AVFrame to different buffer,
deliver result to color conversion),
everything is ok.
Any ideas ? I also tried to do step 1 and 2 in one and deliver the "new
AVFrame()" to the color conversion
component but this still results in distorted image parts.

Any help is highly appreciated :-)

Thanks in advance,
  ALex
_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

MtTracer
Alexander Benz schrieb:

> Hi there,
>
> I'm trying to setup a partly multi threaded video pipeline but my result
> images come out distorted from time to time.
> It seems like some lines of the video contain lines from the previous
> frames ... but I can't find the exact cause. Maybe
> anyone has experienced a similar effect or knows the answer to this
> problem.
>
> In general, what I do is the following :
>
> Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
> avcodec_open etc.) then
>
> 1. Read a frame from the stream (av_read_frame)
>  
Do you call av_dup_packet for the read packets in this step? Packets may
become invalid
after the next call to av_read_frame if you don't duplicate their data.

> 2. Decode frame (avcodec_decode_video)
> 3. Do color conversion (YUV --> RGB)
> 4. Display Image
>
> [...]
>
> Any help is highly appreciated :-)
>
> Thanks in advance,
>   ALex
> _______________________________________________
> libav-user mailing list
> [hidden email]
> https://lists.mplayerhq.hu/mailman/listinfo/libav-user
>
>  

_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

Michael Conrad-6
On Mon, 28 Jul 2008 08:40:56 -0400, MtTracer <[hidden email]> wrote:

> Alexander Benz schrieb:
>> Hi there,
>>
>> I'm trying to setup a partly multi threaded video pipeline but my result
>> images come out distorted from time to time.
>> It seems like some lines of the video contain lines from the previous
>> frames ... but I can't find the exact cause. Maybe
>> anyone has experienced a similar effect or knows the answer to this
>> problem.
>>
>> In general, what I do is the following :
>>
>> Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
>> avcodec_open etc.) then
>>
>> 1. Read a frame from the stream (av_read_frame)
>>
> Do you call av_dup_packet for the read packets in this step? Packets may
> become invalid
> after the next call to av_read_frame if you don't duplicate their data.

Actually its worse than this.  The AVFrame must also be duplicated,  
because some codecs re-use the pixel buffers.  Basically you can only rely  
on having a single AVFrame decoded at a time.  This is only an issue for  
some codecs, though.

>> 2. Decode frame (avcodec_decode_video)
>> 3. Do color conversion (YUV --> RGB)
>> 4. Display Image
>>
>> [...]
>>

--
Michael Conrad
IntelliTree Solutions llc.
513-552-6362
[hidden email]
_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

Alexander Benz
Hi,

So basically, it's best to avoid doing frame fetching and decoding in
parallel.
But thanks for the insight ! :-)

- ALex

Michael Conrad wrote:

> On Mon, 28 Jul 2008 08:40:56 -0400, MtTracer <[hidden email]> wrote:
>  
>> Alexander Benz schrieb:
>>    
>>> Hi there,
>>>
>>> I'm trying to setup a partly multi threaded video pipeline but my result
>>> images come out distorted from time to time.
>>> It seems like some lines of the video contain lines from the previous
>>> frames ... but I can't find the exact cause. Maybe
>>> anyone has experienced a similar effect or knows the answer to this
>>> problem.
>>>
>>> In general, what I do is the following :
>>>
>>> Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
>>> avcodec_open etc.) then
>>>
>>> 1. Read a frame from the stream (av_read_frame)
>>>
>>>      
>> Do you call av_dup_packet for the read packets in this step? Packets may
>> become invalid
>> after the next call to av_read_frame if you don't duplicate their data.
>>    
>
> Actually its worse than this.  The AVFrame must also be duplicated,  
> because some codecs re-use the pixel buffers.  Basically you can only rely  
> on having a single AVFrame decoded at a time.  This is only an issue for  
> some codecs, though.
>
>  
>>> 2. Decode frame (avcodec_decode_video)
>>> 3. Do color conversion (YUV --> RGB)
>>> 4. Display Image
>>>
>>> [...]
>>>
>>>      
>
>  


--
-------------------------------------------------------
Alexander Benz
ask - Innovative Visualisierungslösungen GmbH
Postfach 100 210
D-64202 Darmstadt
tel: ++49 (0) 6159 71713 57   fax: +49 6159/7171390
web: www.askvisual.de    E-Mail: [hidden email]
Registergericht: Amtsgericht Darmstadt, HRB 6177
Sitz: Messel
Geschäftsführer: Hermin Aftahi und Dr. Florian Schröder
-------------------------------------------------------

_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

Ulrich von Zadow-3
Hi,

I think that's a misunderstanding. Frame fetching can be done in a
separate demuxer thread. The demuxer thread duplicates the packets using
av_dup_packet() and puts them in a queue for the decoder to use.

Additionally, you can have separate decode and display threads,
connected by a queue of frames. You just have to make sure that the
frames that you put into the queue are not the original AVFrames that
you get from the decoder, because the decoder overwrites them.

Cheers,

   Uli

Alexander Benz wrote:

> Hi,
>
> So basically, it's best to avoid doing frame fetching and decoding in
> parallel.
> But thanks for the insight ! :-)
>
> - ALex
>
> Michael Conrad wrote:
>> On Mon, 28 Jul 2008 08:40:56 -0400, MtTracer <[hidden email]> wrote:
>>  
>>> Alexander Benz schrieb:
>>>    
>>>> Hi there,
>>>>
>>>> I'm trying to setup a partly multi threaded video pipeline but my result
>>>> images come out distorted from time to time.
>>>> It seems like some lines of the video contain lines from the previous
>>>> frames ... but I can't find the exact cause. Maybe
>>>> anyone has experienced a similar effect or knows the answer to this
>>>> problem.
>>>>
>>>> In general, what I do is the following :
>>>>
>>>> Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
>>>> avcodec_open etc.) then
>>>>
>>>> 1. Read a frame from the stream (av_read_frame)
>>>>
>>>>      
>>> Do you call av_dup_packet for the read packets in this step? Packets may
>>> become invalid
>>> after the next call to av_read_frame if you don't duplicate their data.
>>>    
>> Actually its worse than this.  The AVFrame must also be duplicated,  
>> because some codecs re-use the pixel buffers.  Basically you can only rely  
>> on having a single AVFrame decoded at a time.  This is only an issue for  
>> some codecs, though.
>>
>>  
>>>> 2. Decode frame (avcodec_decode_video)
>>>> 3. Do color conversion (YUV --> RGB)
>>>> 4. Display Image
>>>>
>>>> [...]
>>>>
>>>>      
>>  
>
>


--
Any technology distinguishable from magic is insufficiently advanced.

Ulrich von Zadow | +49-172-7872715
Jabber: [hidden email]
Skype: uzadow
_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

Michael Conrad-6
In reply to this post by Alexander Benz
On Wed, 30 Jul 2008 03:50:57 -0400, Alexander Benz  
<[hidden email]> wrote:
[...]
>> Actually its worse than this.  The AVFrame must also be duplicated,
>> because some codecs re-use the pixel buffers.  Basically you can
>> only rely on having a single AVFrame decoded at a time.  This is
>> only an issue for some codecs, though.
[...]
>
> So basically, it's best to avoid doing frame fetching and decoding
> in parallel.
> But thanks for the insight ! :-)

Actually, the design I chose for my needs was to have one thread that both  
reads packets and decodes them and copies video frames, a second thread  
that runs the YUV->RGB conversion and uploads the frame to video memory to  
be displayed, and a third thread which feeds the audio device.  In the  
first thread, I also check to see if either (video or audio) queue is  
running dry, and conditionally copy packets into a temporary queue so that  
I can find more of the depleted stream.

This requires more ram (and video frame copying) (but less packet copying)  
than the design of ffplay (which copies video packets so they can be  
decoded and displayed by the video thread, one at a time) but provides  
better buffering.  This way I can survive a small processing shortage  
without running out of either video or audio.  In an embedded realtime  
environment, you would likely want to stick with the ffplay design because  
you can guarantee that decoding one video frame will happen on time.


--
Michael Conrad
IntelliTree Solutions llc.
513-552-6362
[hidden email]
_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: Need help with using libav* in multithreaded code

Alexander Benz
In reply to this post by Ulrich von Zadow-3
Hey,

Well I wasn't sure if I can use av_dup_packet() safely because in the
header it is said that
"@warning This is a hack - the packet memory allocation stuff is broken...".
I will give it a try later.

Cheers,
  ALex

Ulrich von Zadow wrote:

> Hi,
>
> I think that's a misunderstanding. Frame fetching can be done in a
> separate demuxer thread. The demuxer thread duplicates the packets using
> av_dup_packet() and puts them in a queue for the decoder to use.
>
> Additionally, you can have separate decode and display threads,
> connected by a queue of frames. You just have to make sure that the
> frames that you put into the queue are not the original AVFrames that
> you get from the decoder, because the decoder overwrites them.
>
> Cheers,
>
>    Uli
>
> Alexander Benz wrote:
>  
>> Hi,
>>
>> So basically, it's best to avoid doing frame fetching and decoding in
>> parallel.
>> But thanks for the insight ! :-)
>>
>> - ALex
>>
>> Michael Conrad wrote:
>>    
>>> On Mon, 28 Jul 2008 08:40:56 -0400, MtTracer <[hidden email]> wrote:
>>>  
>>>      
>>>> Alexander Benz schrieb:
>>>>    
>>>>        
>>>>> Hi there,
>>>>>
>>>>> I'm trying to setup a partly multi threaded video pipeline but my result
>>>>> images come out distorted from time to time.
>>>>> It seems like some lines of the video contain lines from the previous
>>>>> frames ... but I can't find the exact cause. Maybe
>>>>> anyone has experienced a similar effect or knows the answer to this
>>>>> problem.
>>>>>
>>>>> In general, what I do is the following :
>>>>>
>>>>> Open the Mpeg2 video stream (av_open_input_file, avcodec_find_decoder,
>>>>> avcodec_open etc.) then
>>>>>
>>>>> 1. Read a frame from the stream (av_read_frame)
>>>>>
>>>>>      
>>>>>          
>>>> Do you call av_dup_packet for the read packets in this step? Packets may
>>>> become invalid
>>>> after the next call to av_read_frame if you don't duplicate their data.
>>>>    
>>>>        
>>> Actually its worse than this.  The AVFrame must also be duplicated,  
>>> because some codecs re-use the pixel buffers.  Basically you can only rely  
>>> on having a single AVFrame decoded at a time.  This is only an issue for  
>>> some codecs, though.
>>>
>>>  
>>>      
>>>>> 2. Decode frame (avcodec_decode_video)
>>>>> 3. Do color conversion (YUV --> RGB)
>>>>> 4. Display Image
>>>>>
>>>>> [...]
>>>>>
>>>>>      
>>>>>          
>>>  
>>>      
>>    
>
>
>  


--
-------------------------------------------------------
Alexander Benz
ask - Innovative Visualisierungslösungen GmbH
Postfach 100 210
D-64202 Darmstadt
tel: ++49 (0) 6159 71713 57   fax: +49 6159/7171390
web: www.askvisual.de    E-Mail: [hidden email]
Registergericht: Amtsgericht Darmstadt, HRB 6177
Sitz: Messel
Geschäftsführer: Hermin Aftahi und Dr. Florian Schröder
-------------------------------------------------------

_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user