[Libav-user] [dnxhd] Internal error, put_bits buffer too small

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Libav-user] [dnxhd] Internal error, put_bits buffer too small

Jonathan Noble
Hi,

I am trying to encode to dnxhd with libavcodec. The result is quite glitchy with stuttering and anomalies on most frames.

During encoding I see the following message almost continuously.

> Internal error, put_bits buffer too small

I have tried to understand what this buffer is, how it is set up and I am struggling. From what I can understand the puts_bit buffer is DNXHDEncContext->m->pb which is initialised via init_put_bits() in dnxhd_encode_thread().

The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs and DNXHDEncContext->slice_size. These 2 are configured in dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width and MpegEncContext->mb_bits.

If my understanding is correct the 2 aforementioned variables in MpegEncContext are the variables I need to set. However, I have no idea what they mean or how they are accessed outside of AVCodecContext->priv_data.

Is my understanding correct? How does one setup the puts_bit buffer correctly for a dnxhd encode?

Kind regards

Jon Noble



_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: [dnxhd] Internal error, put_bits buffer too small

Paul B Mahol
Please provide input samples that allow to reproduce this issue with ffmpeg.

On 10/2/19, Jonathan Noble <[hidden email]> wrote:

> Hi,
>
> I am trying to encode to dnxhd with libavcodec. The result is quite glitchy
> with stuttering and anomalies on most frames.
>
> During encoding I see the following message almost continuously.
>
>> Internal error, put_bits buffer too small
>
> I have tried to understand what this buffer is, how it is set up and I am
> struggling. From what I can understand the puts_bit buffer is
> DNXHDEncContext->m->pb which is initialised via init_put_bits() in
> dnxhd_encode_thread().
>
> The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs and
> DNXHDEncContext->slice_size. These 2 are configured in
> dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width and
> MpegEncContext->mb_bits.
>
> If my understanding is correct the 2 aforementioned variables in
> MpegEncContext are the variables I need to set. However, I have no idea
> what they mean or how they are accessed outside of
> AVCodecContext->priv_data.
>
> Is my understanding correct? How does one setup the puts_bit buffer
> correctly for a dnxhd encode?
>
> Kind regards
>
> Jon Noble
>
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: [dnxhd] Internal error, put_bits buffer too small

Jonathan Noble
It isn't reproduceable with ffmpeg. Am i in the wrong mailing-list?

On Thu, 3 Oct 2019 at 07:47, Paul B Mahol <[hidden email]> wrote:
Please provide input samples that allow to reproduce this issue with ffmpeg.

On 10/2/19, Jonathan Noble <[hidden email]> wrote:
> Hi,
>
> I am trying to encode to dnxhd with libavcodec. The result is quite glitchy
> with stuttering and anomalies on most frames.
>
> During encoding I see the following message almost continuously.
>
>> Internal error, put_bits buffer too small
>
> I have tried to understand what this buffer is, how it is set up and I am
> struggling. From what I can understand the puts_bit buffer is
> DNXHDEncContext->m->pb which is initialised via init_put_bits() in
> dnxhd_encode_thread().
>
> The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs and
> DNXHDEncContext->slice_size. These 2 are configured in
> dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width and
> MpegEncContext->mb_bits.
>
> If my understanding is correct the 2 aforementioned variables in
> MpegEncContext are the variables I need to set. However, I have no idea
> what they mean or how they are accessed outside of
> AVCodecContext->priv_data.
>
> Is my understanding correct? How does one setup the puts_bit buffer
> correctly for a dnxhd encode?
>
> Kind regards
>
> Jon Noble
>
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: [dnxhd] Internal error, put_bits buffer too small

Paul B Mahol
On 10/3/19, Jonathan Noble <[hidden email]> wrote:
> It isn't reproduceable with ffmpeg. Am i in the wrong mailing-list?

Perhaps you reset frame_size somehow?

>
> On Thu, 3 Oct 2019 at 07:47, Paul B Mahol <[hidden email]> wrote:
>
>> Please provide input samples that allow to reproduce this issue with
>> ffmpeg.
>>
>> On 10/2/19, Jonathan Noble <[hidden email]> wrote:
>> > Hi,
>> >
>> > I am trying to encode to dnxhd with libavcodec. The result is quite
>> glitchy
>> > with stuttering and anomalies on most frames.
>> >
>> > During encoding I see the following message almost continuously.
>> >
>> >> Internal error, put_bits buffer too small
>> >
>> > I have tried to understand what this buffer is, how it is set up and I
>> > am
>> > struggling. From what I can understand the puts_bit buffer is
>> > DNXHDEncContext->m->pb which is initialised via init_put_bits() in
>> > dnxhd_encode_thread().
>> >
>> > The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs
>> and
>> > DNXHDEncContext->slice_size. These 2 are configured in
>> > dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width
>> > and
>> > MpegEncContext->mb_bits.
>> >
>> > If my understanding is correct the 2 aforementioned variables in
>> > MpegEncContext are the variables I need to set. However, I have no idea
>> > what they mean or how they are accessed outside of
>> > AVCodecContext->priv_data.
>> >
>> > Is my understanding correct? How does one setup the puts_bit buffer
>> > correctly for a dnxhd encode?
>> >
>> > Kind regards
>> >
>> > Jon Noble
>> >
>> _______________________________________________
>> Libav-user mailing list
>> [hidden email]
>> https://ffmpeg.org/mailman/listinfo/libav-user
>>
>> To unsubscribe, visit link above, or email
>> [hidden email] with subject "unsubscribe".
>
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: [dnxhd] Internal error, put_bits buffer too small

Jonathan Noble
I've just searched for any instance of AVCodecContext->frame_size and it is only read.

Here is a sample of how the AVCodecContext is setup

  // setup context
  vcodec_ctx->codec_id = static_cast<AVCodecID>(video_params_.codec);
  vcodec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;
  vcodec_ctx->width = video_params_.width;
  vcodec_ctx->height = video_params_.height;
  vcodec_ctx->sample_aspect_ratio = {1, 1};
  vcodec_ctx->pix_fmt = vcodec->pix_fmts[0]; // maybe be breakable code
  setupFrameRate(*vcodec_ctx, video_params_.frame_rate);
  if (video_params_.compression_type == CompressionType::CBR) {
    const auto brate = static_cast<int64_t>((video_params_.bitrate * 1E6) + 0.5);
    vcodec_ctx->bit_rate = brate;
    vcodec_ctx->rc_min_rate = brate;
    vcodec_ctx->rc_max_rate = brate;
  }
  vcodec_ctx->time_base = av_inv_q(vcodec_ctx->framerate);
  video_stream->time_base = vcodec_ctx->time_base;
  vcodec_ctx->gop_size = video_params_.gop_length_;
  vcodec_ctx->thread_count = static_cast<int>(std::thread::hardware_concurrency());
  // Do bare minimum before avcodec_open2
  switch (vcodec_ctx->codec_id) {
    case AV_CODEC_ID_H264:
      setupH264Encoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_MPEG2VIDEO:
      setupMPEG2Encoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_DNXHD:
      setupDNXHDEncoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_MPEG4:
      setupMPEG4Encoder(*vcodec_ctx, video_params_);
    default:
      // Nothing defined for these codecs yet
      break;
  }

  auto ret = avcodec_open2(vcodec_ctx, vcodec, &opts);
  if (ret < 0) {
    av_strerror(ret, err.data(), ERR_LEN);
    qCritical() << "Could not open output video encoder." << err.data();
    ed->export_error = tr("could not open output video encoder (%1)").arg(err.data());
    return false;
  }

  if (fmt_ctx->oformat->flags & AVFMT_GLOBALHEADER) {
    vcodec_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
  }

  vcodec_ctx->sample_aspect_ratio = {1, 1};
  vcodec_ctx->max_b_frames = video_params_.b_frames_;

  // copy video encoder parameters to output stream
  ret = avcodec_parameters_from_context(video_stream->codecpar, vcodec_ctx);
  if (ret < 0) {
    av_strerror(ret, err.data(), ERR_LEN);
    qCritical() << "Could not copy video encoder parameters to output stream, code=" << err.data();
    ed->export_error = tr("could not copy video encoder parameters to output stream (%1)").arg(QString::number(ret));
    return false;
  }

 void ExportThread::setupDNXHDEncoder(AVCodecContext& ctx, const Params& video_params) const
{
  ctx.profile = FF_PROFILE_DNXHD;
  // FIXME: DNXHDEncContext.pb needs setting up somehow

  if (video_params.profile_.endsWith("x")) {
    // dnxhdenc will deduce this as 10bits
    if (!video_params.pix_fmts_.empty())
    {
      auto fmt = video_params.pix_fmts_.front();
      switch (fmt)
      {
        case PixelFormat::YUV444:
          ctx.pix_fmt = AV_PIX_FMT_YUV444P10;
          break;
        case PixelFormat::YUV422:
          ctx.pix_fmt = AV_PIX_FMT_YUV422P10;
          break;
        case PixelFormat::YUV420:
          // this may not be in any profile
          ctx.pix_fmt = AV_PIX_FMT_YUV420P10;
          break;
      }
    }
  }
}

On Thu, 3 Oct 2019 at 09:47, Paul B Mahol <[hidden email]> wrote:
On 10/3/19, Jonathan Noble <[hidden email]> wrote:
> It isn't reproduceable with ffmpeg. Am i in the wrong mailing-list?

Perhaps you reset frame_size somehow?

>
> On Thu, 3 Oct 2019 at 07:47, Paul B Mahol <[hidden email]> wrote:
>
>> Please provide input samples that allow to reproduce this issue with
>> ffmpeg.
>>
>> On 10/2/19, Jonathan Noble <[hidden email]> wrote:
>> > Hi,
>> >
>> > I am trying to encode to dnxhd with libavcodec. The result is quite
>> glitchy
>> > with stuttering and anomalies on most frames.
>> >
>> > During encoding I see the following message almost continuously.
>> >
>> >> Internal error, put_bits buffer too small
>> >
>> > I have tried to understand what this buffer is, how it is set up and I
>> > am
>> > struggling. From what I can understand the puts_bit buffer is
>> > DNXHDEncContext->m->pb which is initialised via init_put_bits() in
>> > dnxhd_encode_thread().
>> >
>> > The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs
>> and
>> > DNXHDEncContext->slice_size. These 2 are configured in
>> > dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width
>> > and
>> > MpegEncContext->mb_bits.
>> >
>> > If my understanding is correct the 2 aforementioned variables in
>> > MpegEncContext are the variables I need to set. However, I have no idea
>> > what they mean or how they are accessed outside of
>> > AVCodecContext->priv_data.
>> >
>> > Is my understanding correct? How does one setup the puts_bit buffer
>> > correctly for a dnxhd encode?
>> >
>> > Kind regards
>> >
>> > Jon Noble
>> >
>> _______________________________________________
>> Libav-user mailing list
>> [hidden email]
>> https://ffmpeg.org/mailman/listinfo/libav-user
>>
>> To unsubscribe, visit link above, or email
>> [hidden email] with subject "unsubscribe".
>
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".
Reply | Threaded
Open this post in threaded view
|

Re: [dnxhd] Internal error, put_bits buffer too small

Jonathan Noble
I discovered what the issue was after commenting out sections of code until behaviour change.

AVFormatContext->thread_count was set but AVFormatContext->thread_type wasn't. Either unsetting thread_count or setting thread_type fixed the issue for me.

On Thu, 3 Oct 2019 at 20:35, Jonathan Noble <[hidden email]> wrote:
I've just searched for any instance of AVCodecContext->frame_size and it is only read.

Here is a sample of how the AVCodecContext is setup

  // setup context
  vcodec_ctx->codec_id = static_cast<AVCodecID>(video_params_.codec);
  vcodec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;
  vcodec_ctx->width = video_params_.width;
  vcodec_ctx->height = video_params_.height;
  vcodec_ctx->sample_aspect_ratio = {1, 1};
  vcodec_ctx->pix_fmt = vcodec->pix_fmts[0]; // maybe be breakable code
  setupFrameRate(*vcodec_ctx, video_params_.frame_rate);
  if (video_params_.compression_type == CompressionType::CBR) {
    const auto brate = static_cast<int64_t>((video_params_.bitrate * 1E6) + 0.5);
    vcodec_ctx->bit_rate = brate;
    vcodec_ctx->rc_min_rate = brate;
    vcodec_ctx->rc_max_rate = brate;
  }
  vcodec_ctx->time_base = av_inv_q(vcodec_ctx->framerate);
  video_stream->time_base = vcodec_ctx->time_base;
  vcodec_ctx->gop_size = video_params_.gop_length_;
  vcodec_ctx->thread_count = static_cast<int>(std::thread::hardware_concurrency());
  // Do bare minimum before avcodec_open2
  switch (vcodec_ctx->codec_id) {
    case AV_CODEC_ID_H264:
      setupH264Encoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_MPEG2VIDEO:
      setupMPEG2Encoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_DNXHD:
      setupDNXHDEncoder(*vcodec_ctx, video_params_);
      break;
    case AV_CODEC_ID_MPEG4:
      setupMPEG4Encoder(*vcodec_ctx, video_params_);
    default:
      // Nothing defined for these codecs yet
      break;
  }

  auto ret = avcodec_open2(vcodec_ctx, vcodec, &opts);
  if (ret < 0) {
    av_strerror(ret, err.data(), ERR_LEN);
    qCritical() << "Could not open output video encoder." << err.data();
    ed->export_error = tr("could not open output video encoder (%1)").arg(err.data());
    return false;
  }

  if (fmt_ctx->oformat->flags & AVFMT_GLOBALHEADER) {
    vcodec_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
  }

  vcodec_ctx->sample_aspect_ratio = {1, 1};
  vcodec_ctx->max_b_frames = video_params_.b_frames_;

  // copy video encoder parameters to output stream
  ret = avcodec_parameters_from_context(video_stream->codecpar, vcodec_ctx);
  if (ret < 0) {
    av_strerror(ret, err.data(), ERR_LEN);
    qCritical() << "Could not copy video encoder parameters to output stream, code=" << err.data();
    ed->export_error = tr("could not copy video encoder parameters to output stream (%1)").arg(QString::number(ret));
    return false;
  }

 void ExportThread::setupDNXHDEncoder(AVCodecContext& ctx, const Params& video_params) const
{
  ctx.profile = FF_PROFILE_DNXHD;
  // FIXME: DNXHDEncContext.pb needs setting up somehow

  if (video_params.profile_.endsWith("x")) {
    // dnxhdenc will deduce this as 10bits
    if (!video_params.pix_fmts_.empty())
    {
      auto fmt = video_params.pix_fmts_.front();
      switch (fmt)
      {
        case PixelFormat::YUV444:
          ctx.pix_fmt = AV_PIX_FMT_YUV444P10;
          break;
        case PixelFormat::YUV422:
          ctx.pix_fmt = AV_PIX_FMT_YUV422P10;
          break;
        case PixelFormat::YUV420:
          // this may not be in any profile
          ctx.pix_fmt = AV_PIX_FMT_YUV420P10;
          break;
      }
    }
  }
}

On Thu, 3 Oct 2019 at 09:47, Paul B Mahol <[hidden email]> wrote:
On 10/3/19, Jonathan Noble <[hidden email]> wrote:
> It isn't reproduceable with ffmpeg. Am i in the wrong mailing-list?

Perhaps you reset frame_size somehow?

>
> On Thu, 3 Oct 2019 at 07:47, Paul B Mahol <[hidden email]> wrote:
>
>> Please provide input samples that allow to reproduce this issue with
>> ffmpeg.
>>
>> On 10/2/19, Jonathan Noble <[hidden email]> wrote:
>> > Hi,
>> >
>> > I am trying to encode to dnxhd with libavcodec. The result is quite
>> glitchy
>> > with stuttering and anomalies on most frames.
>> >
>> > During encoding I see the following message almost continuously.
>> >
>> >> Internal error, put_bits buffer too small
>> >
>> > I have tried to understand what this buffer is, how it is set up and I
>> > am
>> > struggling. From what I can understand the puts_bit buffer is
>> > DNXHDEncContext->m->pb which is initialised via init_put_bits() in
>> > dnxhd_encode_thread().
>> >
>> > The 2 arguments used in init_put_bits() are DNXHDEncContext->slice_offs
>> and
>> > DNXHDEncContext->slice_size. These 2 are configured in
>> > dnxhd_setup_thread_slices() with 2 variables MpegEncContext->mb_width
>> > and
>> > MpegEncContext->mb_bits.
>> >
>> > If my understanding is correct the 2 aforementioned variables in
>> > MpegEncContext are the variables I need to set. However, I have no idea
>> > what they mean or how they are accessed outside of
>> > AVCodecContext->priv_data.
>> >
>> > Is my understanding correct? How does one setup the puts_bit buffer
>> > correctly for a dnxhd encode?
>> >
>> > Kind regards
>> >
>> > Jon Noble
>> >
>> _______________________________________________
>> Libav-user mailing list
>> [hidden email]
>> https://ffmpeg.org/mailman/listinfo/libav-user
>>
>> To unsubscribe, visit link above, or email
>> [hidden email] with subject "unsubscribe".
>
_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".

_______________________________________________
Libav-user mailing list
[hidden email]
https://ffmpeg.org/mailman/listinfo/libav-user

To unsubscribe, visit link above, or email
[hidden email] with subject "unsubscribe".