[Libav-user] Extract AVFrameSideData

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Libav-user] Extract AVFrameSideData

lyncher
Hi, I'm trying to extract a A53CC captions from AVFrameSideData.
There are some filter that are aware of this kind of side data
(sidedata) but I cannot find a way to have access to it (unless using
lavfi+movie).

I would like to be able to retrieve the CC bytes from a previously
decoded AVFrame and using ccaption_dec to format it in SRT (or any other
supported format).
I was thinking to create a Filter that receives an AVFrame and returns
an AVPacket.... but this combination seems no to be possible.
How can I extract AVSideData from a AVFrame and output it in a filter as
AV_CODEC_ID_EIA_608?

Thanks
lyncher



_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

Devin Heitmueller
On Thu, Jun 6, 2019 at 3:19 PM <[hidden email]> wrote:

> I would like to be able to retrieve the CC bytes from a previously
> decoded AVFrame and using ccaption_dec to format it in SRT (or any other
> supported format).
> I was thinking to create a Filter that receives an AVFrame and returns
> an AVPacket.... but this combination seems no to be possible.
> How can I extract AVSideData from a AVFrame and output it in a filter as
> AV_CODEC_ID_EIA_608?

So filters can only have AVFrames as inputs and outputs.  You cannot
have a filter which takes in AVFrames and outputs AVPackets.
Something that takes in frames and outputs packets is technically an
encoder.  So in principle you could split the video with a filter and
feed the real video to whatever encoder you are using (e.g. x264), and
the second video feed goes into your fake encoder which outputs the
608 AVPackets.

One of these days I'll kill an afternoon and hack together such an
encoder, but just haven't had the need yet.

Devin

--
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

lyncher
Thanks Devin for your feedback.

That fake encoder will have a definition like:
>     .type           = AVMEDIA_TYPE_VIDEO,
>     .id             = AV_CODEC_ID_EIA_608,

But if it's defined like this, it will output a video with eia_608
type.... which doesn't t allow ccaption_dec to be applied.

If it's defined like:
>     .type           = AVMEDIA_TYPE_SUBTITLE,
>     .id             = AV_CODEC_ID_EIA_608,

No video packets will be sent to it.


What should be the correct definition to have this fake encoder
configured like:
     Video > Fake Encoder > eia_608 caption stream

Thanks
lyncher


On 06-06-2019 21:51, Devin Heitmueller wrote:

> On Thu, Jun 6, 2019 at 3:19 PM <[hidden email]> wrote:
>
>> I would like to be able to retrieve the CC bytes from a previously
>> decoded AVFrame and using ccaption_dec to format it in SRT (or any other
>> supported format).
>> I was thinking to create a Filter that receives an AVFrame and returns
>> an AVPacket.... but this combination seems no to be possible.
>> How can I extract AVSideData from a AVFrame and output it in a filter as
>> AV_CODEC_ID_EIA_608?
> So filters can only have AVFrames as inputs and outputs.  You cannot
> have a filter which takes in AVFrames and outputs AVPackets.
> Something that takes in frames and outputs packets is technically an
> encoder.  So in principle you could split the video with a filter and
> feed the real video to whatever encoder you are using (e.g. x264), and
> the second video feed goes into your fake encoder which outputs the
> 608 AVPackets.
>
> One of these days I'll kill an afternoon and hack together such an
> encoder, but just haven't had the need yet.
>
> Devin
>

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

Devin Heitmueller
> That fake encoder will have a definition like:
> >     .type           = AVMEDIA_TYPE_VIDEO,
> >     .id             = AV_CODEC_ID_EIA_608,

This should be what you need.  The output AVStream doesn't have a
discrete type like raw content does (e.g. where you have
audio/video/data).  The content of the packets is driven entirely by
the codec ID.

> But if it's defined like this, it will output a video with eia_608
> type.... which doesn't t allow ccaption_dec to be applied.

The output is defined by the codec.  If your encoder takes in video
AVFrames, throws away the actual video content, and uses the side data
to output packets with the codec AV_CODEC_ID_EIA_608, that should be
perfectly fine.

> If it's defined like:
> >     .type           = AVMEDIA_TYPE_SUBTITLE,
> >     .id             = AV_CODEC_ID_EIA_608,
>
> No video packets will be sent to it.

Right, that definitely won't work, because the ffmpeg core won't
deliver video AVFrames to it.

Did you actually try what you specified in the first scheme?  That
should work fine, so if you tried it and it didn't I would be
interested in seeing the errors you got.

Devin

--
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

lyncher
I'm not getting how can I define the codec ID to an AVPacket.....
Should I set something at AVCodecContext?

lyncher


On 07-06-2019 16:38, Devin Heitmueller wrote:

>> That fake encoder will have a definition like:
>>>      .type           = AVMEDIA_TYPE_VIDEO,
>>>      .id             = AV_CODEC_ID_EIA_608,
> This should be what you need.  The output AVStream doesn't have a
> discrete type like raw content does (e.g. where you have
> audio/video/data).  The content of the packets is driven entirely by
> the codec ID.
>
>> But if it's defined like this, it will output a video with eia_608
>> type.... which doesn't t allow ccaption_dec to be applied.
> The output is defined by the codec.  If your encoder takes in video
> AVFrames, throws away the actual video content, and uses the side data
> to output packets with the codec AV_CODEC_ID_EIA_608, that should be
> perfectly fine.
>
>> If it's defined like:
>>>      .type           = AVMEDIA_TYPE_SUBTITLE,
>>>      .id             = AV_CODEC_ID_EIA_608,
>> No video packets will be sent to it.
> Right, that definitely won't work, because the ffmpeg core won't
> deliver video AVFrames to it.
>
> Did you actually try what you specified in the first scheme?  That
> should work fine, so if you tried it and it didn't I would be
> interested in seeing the errors you got.
>
> Devin
>

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

Devin Heitmueller
Hi Lyncher,

On Fri, Jun 7, 2019 at 11:48 AM <[hidden email]> wrote:
>
> I'm not getting how can I define the codec ID to an AVPacket.....
> Should I set something at AVCodecContext?

So those fields are not describing the AVPacket.  They're describing
the codec.  That said, the codec itself has three different references
to the AVMediaType (the codec description, codec context, and codec
parameters).  While I know it uses at least one of those to know what
type of AVFrames can be dispatched to it (e.g. to prevent you from
sending audio AVFrames to a video encoder), I'm not sure which the .id
field ties directly to, or what happens if all three of those values
aren't the same.  At least the codec parameters needs to reflect the
subtitle type in order for subtitle decoders such as ccaption_dec to
be able to accept packets from it.

Interesting.  I would have to do some experimentation to see what the
right path is here.

Devin

--
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: Extract AVFrameSideData

lyncher
The code that I have:

>
>
> #include "libavutil/opt.h"
> #include "avcodec.h"
> #include "internal.h"
>
> typedef struct A53EncodeContext {
>     const AVClass *class;
> } A53EncodeContext;
>
>
> static int a53cc_encode(AVCodecContext *avctx, AVPacket *avpkt, const
> AVFrame *frame, int *got_packet_ptr)
> {
>     A53EncodeContext *ctx = avctx->priv_data;
>
>     int ret;
>
>     for (int i = 0; i < frame->nb_side_data; i++) {
>         AVFrameSideData *sd = frame->side_data[i];
>
>         av_log(ctx, AV_LOG_INFO, "  side data - ");
>         if (sd->type == AV_FRAME_DATA_A53_CC) {
>
>             //Create AVPacket to hold extracted CC
>             if ((ret = ff_alloc_packet2(avctx, avpkt, sd->size, 0)) < 0) {
>                 *got_packet_ptr = 0;
>                 return ret;
>             }
>
>             //Copy CC packet
>             memcpy(avpkt->data, sd->data, sd->size);
>             avpkt->stream_index = 0;
>             avpkt->pts = frame->pts;
>             avpkt->pos = frame->pkt_pos;
>
>             av_log(ctx, AV_LOG_INFO, "A/53 closed captions (%d
> bytes)\n", sd->size);
>             for (int offset = 0; offset < sd->size; offset++) {
>                 av_log(ctx, AV_LOG_INFO, "%0x", sd->data[offset]);
>             }
>
>             *got_packet_ptr = 1;
>             return 0;
>         }
>
>         av_log(ctx, AV_LOG_INFO, "\n");
>     }
>
>
>     *got_packet_ptr = 0;
>     return 0;
> }
>
>
> #define OFFSET(x) offsetof(A53EncodeContext, x)
> #define SE AV_OPT_FLAG_SUBTITLE_PARAM | AV_OPT_FLAG_ENCODING_PARAM
> static const AVOption a53cc_options[] = {
>     { NULL },
> };
>
> static const AVClass a53ccenc_class = {
>     .class_name = "A53 caption encoder",
>     .item_name  = av_default_item_name,
>     .option     = a53cc_options,
>     .version    = LIBAVUTIL_VERSION_INT,
> };
>
> AVCodec ff_a53cc_encoder = {
>     .name           = "a53cc",
>     .long_name      = NULL_IF_CONFIG_SMALL("A/53 captions"),
>     .type           = AVMEDIA_TYPE_VIDEO,
>     .id             = AV_CODEC_ID_EIA_608,
>     //.init           = a53cc_init,
>     .encode2        = a53cc_encode,
>     .priv_class     = &a53ccenc_class,
>     .priv_data_size = sizeof(A53EncodeContext),
> };
>

lyncher

On 07-06-2019 17:16, Devin Heitmueller wrote:

> Hi Lyncher,
>
> On Fri, Jun 7, 2019 at 11:48 AM <[hidden email]> wrote:
>> I'm not getting how can I define the codec ID to an AVPacket.....
>> Should I set something at AVCodecContext?
> So those fields are not describing the AVPacket.  They're describing
> the codec.  That said, the codec itself has three different references
> to the AVMediaType (the codec description, codec context, and codec
> parameters).  While I know it uses at least one of those to know what
> type of AVFrames can be dispatched to it (e.g. to prevent you from
> sending audio AVFrames to a video encoder), I'm not sure which the .id
> field ties directly to, or what happens if all three of those values
> aren't the same.  At least the codec parameters needs to reflect the
> subtitle type in order for subtitle decoders such as ccaption_dec to
> be able to accept packets from it.
>
> Interesting.  I would have to do some experimentation to see what the
> right path is here.
>
> Devin
>

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".