crash on h264

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

crash on h264

Bugzilla from iscy@invalidip.com
Hi,

I'm experiencing a crash while decoding h264 videos. It seems to be much
easier to reproduce on win32, however valgrind is able to also identify the
problem on unix. The root cause of the problem is an invalid read out of the
buffer in one of the dsp util mmx/sse3 function.

This is one of the movie able to reproduce the issue:
http://movies.apple.com/movies/sony_pictures/hancock/hancock-tlr2r_h480p.mov

I attached the stack generated by GDB as well as the assembly where the crash
occurs.

When calling avcodec_decode_video(), the "put_h264_chroma_mc4_xxx" operation
can do an invalid read on the buffer 'src'. In my example, 'src' was first
initialized at "0x1bdf2bc" at the beginning of the function, then the last
value that it tried to read was "0x1be007c" and the last valid memory
location is at "0x1bdfffc". The loop was also on the last iteration when the
crash occured. It looks like this buffer should have been about 128 bytes
bigger to satisfy this function. This buffer is the one stored as
the 'ref_list' inside the H264Context struct. At the moment, I'm tracking it
down to see why it is too small for this assembly function.



_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user

stack.txt (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: crash on h264

Michael Conrad-6
On Fri, 25 Jul 2008 16:13:25 -0400, Pascal Patry <[hidden email]>  
wrote:
> The root cause of the problem is an invalid read out of the buffer in  
> one of the dsp util mmx/sse3 function.
[...]
> When calling avcodec_decode_video(), the "put_h264_chroma_mc4_xxx"  
> operation can do an invalid read on the buffer 'src'. In my example,
> 'src' was first initialized at "0x1bdf2bc" at the beginning of the
> function, then the last value that it tried to read was "0x1be007c"
> and the last valid memory location is at "0x1bdfffc". The loop was
> also on the last iteration when the crash occured.

Not familiar with the code in question, but are you making sure to  
allocate all of your buffers on multiples of 16 bytes?  For example,  
0x1bdf2bc is not an address that could be operated on by SSE instructions,  
but 0x1bdf2c0 is.

I'm pretty sure that all the libav functions which allocate buffers will  
give the appropriate alignment, but if you were allocating some yourself  
with plain old "malloc" you will get crashes any time the buffer isn't on  
a 16-byte boundary and an SSE-optimized routine hits it.

see
  posix_memalign
for unix, and
  __mingw_aligned_malloc
for windows, if you're using mingw.

It also isn't hard to "roll your own", but then you have to keep track of  
the original pointer.

--
Michael Conrad
IntelliTree Solutions llc.
513-552-6362
[hidden email]
_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user
Reply | Threaded
Open this post in threaded view
|

Re: crash on h264

Bugzilla from iscy@invalidip.com
On Saturday 26 July 2008 06:37:50 pm Michael Conrad wrote:

> On Fri, 25 Jul 2008 16:13:25 -0400, Pascal Patry <[hidden email]>  
> wrote:
> > The root cause of the problem is an invalid read out of the buffer in  
> > one of the dsp util mmx/sse3 function.
> [...]
> > When calling avcodec_decode_video(), the "put_h264_chroma_mc4_xxx"  
> > operation can do an invalid read on the buffer 'src'. In my example,
> > 'src' was first initialized at "0x1bdf2bc" at the beginning of the
> > function, then the last value that it tried to read was "0x1be007c"
> > and the last valid memory location is at "0x1bdfffc". The loop was
> > also on the last iteration when the crash occured.
>
> Not familiar with the code in question, but are you making sure to  
> allocate all of your buffers on multiples of 16 bytes?  For example,  
> 0x1bdf2bc is not an address that could be operated on by SSE instructions,  
> but 0x1bdf2c0 is.
>
> I'm pretty sure that all the libav functions which allocate buffers will  
> give the appropriate alignment, but if you were allocating some yourself  
> with plain old "malloc" you will get crashes any time the buffer isn't on  
> a 16-byte boundary and an SSE-optimized routine hits it.

[...]

Yeah, I already checked this out. The crashing opcode is movd which doesn't
require any alignment. Also, trying to print the data contained in this
address from a debugger shows that it was not allocated.


_______________________________________________
libav-user mailing list
[hidden email]
https://lists.mplayerhq.hu/mailman/listinfo/libav-user