Make I-frame max QP and encode quality level configurable

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.

[skirsten]

EncodeConfig grows two knobs, both defaulting to the previous hardcoded behavior:

• max_qp_i caps the QP of I/IDR frames in rate-controlled modes (H.264/H.265; the codec defaults of 42/51 apply when unset, values below the codec's min bound are raised to it, AV1 and CQP/disabled are unaffected).
• encode_quality_level sets the Vulkan video encode quality level, previously hardcoded to 0. It is applied consistently to session parameters creation and the first-frame RESET/RATE_CONTROL/QUALITY_LEVEL control command in all three codecs, since the spec requires the bound session parameters to match the session's effective quality level.

Motivation: on a cold encoder the driver's rate control opens the stream with a conservative QP estimate. On static content the skip-coded P-frames then carry that first coarse keyframe until the next IDR, which shows up as gradient banding for the whole first GOP. Bounding qp_i forces the first keyframe to spend enough bits regardless of the initial estimate; verified on NVIDIA (RTX 3080 dev, L40S production) with max_qp_i: 28 under VBR.

🤖 Generated with Claude Code

[hgaiser]

Hmm I tried these settings but had poor results:

1. max_qp_i did not change the encoded stream, it did not seem to have any effect at all (on RTX 5080, nvidia-open 610.57.04).
2. encode_quality_level did change the encoded stream, but hardly. PSNR changed by ~0.2 and the number of bytes by ~0.4%.

What were your results like?

[skirsten]

encode_quality_level also did not really change anything for me. Also could not really find any docs with the numbers mean. But afaik its the quality preset equivalent.

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

[hgaiser]

But max_qp_i=28 helped a lot. The first I-frame of the first GOP always had very bad quality for me which resulted in horrible thumbnails and other issues that depend on the I-frame. This fixed it by allowing it to use more bitrate on the I-frame while the driver's rate control is still adapting.

Do you have an isolated way to verify this? In my tests the output was the same, regardless of that setting.

[skirsten]

Just encode a static image with a large gop (e.g. 120 for 60fps content) and extract the very first iframe and compare.

Regardless, I created the PR just so users can configure the settings. It might not work on all GPUs and depend on other settings. That's why they are optional.