VAAPI: Initial Patch (WIP)vaapi-encoder

- Enabling VAAPI for GNU/Linux (SYS_LINUX) only, adding libraries 'X11 va-drm va-x11 va'
  and USE_VAAPI define.

- hb.c:
  - Adding global static 'vaapi_device_ctx0' default device
    initialized via hb_avcodec_init() and released via hb_avcodec_free()

  - hb_avcodec_vaapi_set_hwframe_ctx() attaches the new AVHWFramesContext
    to the AVCodecContext

  - hb_avcodec_test_encoder(..) adds VAAPI branch,
    needs refinement

- encavcodec.c
  - adding VAAPI branches, enabling available, needs refinement

Discussion:

First of all, only on Intel hardware I was able to produce a correct
encoded result. On AMD we seemingly have YUV420 / NV12 artifacts.
Therefor I need to learn more details on the pix_fmt handling
as described below.

While adding the VAAPI ffmpeg support the following might be an issue.

Usually HB seems to operate with AV_PIX_FMT_YUV420P in general
being set in the AVCodecContext's pix_fmt.

VAAPI performs as follows:
  - It uses hardware frames on the target device, which are being transported
    from the software device. Both format should be allowed to differ??

  - AVCodecContext's pix_fmt uses AV_PIX_FMT_VAAPI

  - AVHWFramesContext: format uses AV_PIX_FMT_VAAPI and sw_format AV_PIX_FMT_NV12,
    the latter hinting on the actual hw-frame's target format.

  - AV_PIX_FMT_NV12 uses interleaved UV data, where AV_PIX_FMT_YUV420P uses
    seperated planes. Both use a separated Y plane upfront.
    Therefor both formats are not picture compatible, memory requirements are same.

  - Encode(..) hardcodes the source frame's to AV_PIX_FMT_YUV420P in VAAPI case,
    usually the AVCodecContext's pix_fmt is being used.
    I need to identify how HB uses AVCodecContext's pix_fmt, i.e. respects
    the desired pix_fmt.

    Then a hw_frame gets allocated and the source frame is being transported
    to this target. Probably it happens here in VAAPI AMD case, that the YUV420
    data is not being transformed to NV12.

    Now the hw_frame is being sent 'avcodec_send_frame' and the code-path re-aligns
    with non VAAPI.

Further fixes to do:
    Naming of VAAPI's profile and level .. figuring out which settings are generally accepted.
    Now the bf (b-frames) is set to zero, since it produces a failure otherwise.
    The latter should only happen in certain profiles, but .. it does even in 100=high

3 files changed, 7 insertions, 3 deletions

diff --git a/gtk/configure.ac b/gtk/configure.ac
index dcaa34faf..a6ada84b8 100644
--- a/gtk/configure.ac
+++ b/gtk/configure.ac
@@ -170,7 +170,7 @@ PKG_CHECK_MODULES(GHB, [$GHB_PACKAGES])
 
 GHB_CFLAGS="$HBINC $GHB_CFLAGS"
 
-HB_LIBS="-lhandbrake -lavresample -lavformat -lavfilter -lavcodec -lavutil -lswresample -lpostproc -ldvdnav -ldvdread -lmp3lame -lvorbis -lvorbisenc -logg -lsamplerate -lx264 -lswscale -ltheoraenc -ltheoradec -lvpx -lz -lbz2 -lbluray -lass -lfontconfig -lfreetype -lxml2 -ljansson -lopus"
+HB_LIBS="-lhandbrake -lavresample -lavformat -lavfilter -lavcodec -lavutil -lX11 -lva-drm -lva-x11 -lva -lswresample -lpostproc -ldvdnav -ldvdread -lmp3lame -lvorbis -lvorbisenc -logg -lsamplerate -lx264 -lswscale -ltheoraenc -ltheoradec -lvpx -lz -lbz2 -lbluray -lass -lfontconfig -lfreetype -lxml2 -ljansson -lopus"
 
 case $host in
   *-*-mingw*)
diff --git a/gtk/src/callbacks.c b/gtk/src/callbacks.c
index 9c5b846fb..15a3c6653 100644
--- a/gtk/src/callbacks.c
+++ b/gtk/src/callbacks.c
@@ -5485,6 +5485,10 @@ format_vquality_cb(GtkScale *scale, gdouble val, signal_user_data_t *ud)
         case HB_VCODEC_FFMPEG_VP9:
         case HB_VCODEC_FFMPEG_H264_NVENC:
         case HB_VCODEC_FFMPEG_H265_NVENC:
+        case HB_VCODEC_FFMPEG_H264_VAAPI:
+        case HB_VCODEC_FFMPEG_H265_VAAPI:
+        case HB_VCODEC_FFMPEG_VP8_VAAPI:
+        case HB_VCODEC_FFMPEG_VP9_VAAPI:
         case HB_VCODEC_THEORA:
         {
             return g_strdup_printf("%s: %d", vqname, (int)val);
diff --git a/gtk/src/makedeps.py b/gtk/src/makedeps.py
index 4c5cc5188..bc4d09e86 100644
--- a/gtk/src/makedeps.py
+++ b/gtk/src/makedeps.py
@@ -59,8 +59,8 @@ dep_map = (
     DepEntry("VideoEncoder", "x264FastDecode", "x264|x264_10bit", False, True),
     DepEntry("VideoEncoder", "x264UseAdvancedOptions", "x264|x264_10bit", False, True),
     DepEntry("HideAdvancedVideoSettings", "x264UseAdvancedOptions", "1", True, True),
-    DepEntry("VideoEncoder", "VideoOptionExtraWindow", "x264|x264_10bit|x265|x265_10bit|x265_12bit|x265_16bit|mpeg4|mpeg2|VP8|VP9|h264_nvenc|hevc_nvenc", False, True),
-    DepEntry("VideoEncoder", "VideoOptionExtraLabel", "x264|x264_10bit|x265|x265_10bit|x265_12bit|x265_16bit|mpeg4|mpeg2|VP8|VP9|h264_nvenc|hevc_nvenc", False, True),
+    DepEntry("VideoEncoder", "VideoOptionExtraWindow", "x264|x264_10bit|x265|x265_10bit|x265_12bit|x265_16bit|mpeg4|mpeg2|VP8|VP9|h264_nvenc|hevc_nvenc|h264_vaapi|hevc_vaapi|vp8_vaapi|vp9_vaapi", False, True),
+    DepEntry("VideoEncoder", "VideoOptionExtraLabel", "x264|x264_10bit|x265|x265_10bit|x265_12bit|x265_16bit|mpeg4|mpeg2|VP8|VP9|h264_nvenc|hevc_nvenc|h264_vaapi|hevc_vaapi|vp8_vaapi|vp9_vaapi", False, True),
     DepEntry("x264UseAdvancedOptions", "VideoSettingsTable", "1", True, False),
     DepEntry("VideoEncoder", "x264_box", "x264|x264_10bit", False, True),
     DepEntry("x264UseAdvancedOptions", "x264_box", "0", True, False),