decavcodec: fix error when ffmpeg parser changes the codec_id

The ffmpeg mpeg2 parser changes AVCodecContext.codec_id on the fly based
on what it parses. Normally this results in correct switching of decoders
internally in ffmpeg. Due to some unfortunate ordering of how we initialize
things when HandBrake is using our own demuxers, avcodec_open gets called
with the original AVCodec and the AVCodecContext.codec_id that was changed
by the parser resulting in an error.

Further explanation... When using our own demuxers, there are some codecs that
we have to parse out the extradata for _prior_ to calling avcodec_open.
avcodec_open fails if extradata isn't initialized for these codecs prior to
the call. To initialize the extradata, we use parser->parser->split. I.e. the
parser must be initialized and used prior to calling avcodec_open. When
avcodec_open is called, it is using the original AVCodec that was used with
avcodec_alloc_context3 (it will fail if AVCodec is not the same between
avcodec_alloc_context3 and avccodec_open). The call to avcodec_open fails with
"Codec type or id mismatches" since AVCodecContext.codec_id no longer matches
AVCodec.id due to the parser changing the codec_id.

The solution is to reset AVCodec and reallocate the context when we
detect that the parser has changed codec_id on us.

1 files changed, 17 insertions, 4 deletions

diff --git a/libhb/decavcodec.c b/libhb/decavcodec.c
index 9afe318ed..957e2ae75 100644
--- a/libhb/decavcodec.c
+++ b/libhb/decavcodec.c
@@ -1669,10 +1669,6 @@ static int decodePacket( hb_work_object_t * w )
     // first frame because of M$ VC1 braindamage).
     if ( !pv->video_codec_opened )
     {
-        pv->context->workaround_bugs = FF_BUG_AUTODETECT;
-        pv->context->err_recognition = AV_EF_CRCCHECK;
-        pv->context->error_concealment = FF_EC_GUESS_MVS|FF_EC_DEBLOCK;
-
         if (setup_extradata(pv))
         {
             // we didn't find the headers needed to set up extradata.
@@ -1681,6 +1677,14 @@ static int decodePacket( hb_work_object_t * w )
             return HB_WORK_OK;
         }
 
+        hb_avcodec_free_context(&pv->context);
+        pv->context = avcodec_alloc_context3(pv->codec);
+
+        pv->context->workaround_bugs   = FF_BUG_AUTODETECT;
+        pv->context->err_recognition   = AV_EF_CRCCHECK;
+        pv->context->error_concealment = FF_EC_GUESS_MVS|FF_EC_DEBLOCK;
+
+
 #ifdef USE_QSV
         if (pv->qsv.decode &&
             pv->qsv.config.io_pattern == MFX_IOPATTERN_OUT_OPAQUE_MEMORY)
@@ -1848,6 +1852,7 @@ static int decavcodecvWork( hb_work_object_t * w, hb_buffer_t ** buf_in,
 
         if (pv->parser)
         {
+            int codec_id = pv->context->codec_id;
             len = av_parser_parse2(pv->parser, pv->context, &pout, &pout_len,
                                    in->data + pos, in->size - pos,
                                    pts, dts, 0 );
@@ -1855,6 +1860,14 @@ static int decavcodecvWork( hb_work_object_t * w, hb_buffer_t ** buf_in,
             parser_dts = pv->parser->dts;
             pts = AV_NOPTS_VALUE;
             dts = AV_NOPTS_VALUE;
+
+            if (codec_id != pv->context->codec_id)
+            {
+                // The parser has decided to change the decoder underneath
+                // us.  Update our context to match.  This can happen
+                // for MPEG-1/2 video, perhaps others
+                pv->codec = avcodec_find_decoder(pv->context->codec_id);
+            }
         }
         else
         {