config: write api_key['BearerToken'] so v36+ SDK auth works by jmacek · Pull Request #2585 · kubernetes-client/python

jmacek · 2026-05-21T17:10:11Z

What this PR does

Updates the in-cluster and kubeconfig loaders in kubernetes/base/config/, plus kubernetes_asyncio/config/incluster_config.py, to write the bearer token under api_key['BearerToken'] (the v36+ lookup key) instead of api_key['authorization'] (the v35 lookup key). Updates the corresponding test assertions to read the new key. Adds a regression test that exercises the end-to-end flow (Configuration → ApiClient → outgoing Authorization header) which the existing tests did not cover.

Why

v36 rewrote Configuration.auth_settings() to look up the bearer-token credential under api_key['BearerToken'], matching the OpenAPI security scheme name. This rename was declared a breaking change in the project CHANGELOG as part of the openapi-generator v6.6.0 upgrade. The hand-written loaders in kubernetes/base/config/ and kubernetes_asyncio/config/incluster_config.py were not updated to follow the rename - they still write api_key['authorization']. (Asyncio's kube_config.py already writes 'BearerToken', so it isn't affected.)

Net effect on v36: every call to load_incluster_config() (and load_kube_config() with a static token, including the async equivalents) produces a Configuration whose auth_settings() yields no bearer credential, so outgoing API requests are sent without an Authorization header and the apiserver treats them as system:anonymous. The failure mode is silent - no warning, no exception, just 401s from every API call.

See #2582 for the user-side report and a minimal repro.

Test plan

Added test_load_incluster_sets_request_authorization_header to kubernetes/base/config/incluster_config_test.py. It drives a real ApiClient.update_params_for_auth() against a freshly-loaded Configuration and asserts the resulting headers contain an Authorization entry - the end-to-end invariant that v36 broke.
Verified the new regression test fails on kubernetes==36.0.0 against the unfixed loader (AssertionError: 'authorization' not found in {}) and passes with the fix.
Verified all 103 sync tests in kubernetes/base/config/ and 72 async tests in kubernetes_asyncio/config/ pass on v36 with the change applied.

Fixes#2582

Fix `load_incluster_config()` and `load_kube_config()` (sync and async, with a static token) so requests carry an `Authorization` header on `kubernetes-client/python` v36+. Without this fix, in-cluster pods upgrading to v36 silently send unauthenticated requests and the apiserver rejects them as `system:anonymous`.

linux-foundation-easycla · 2026-05-21T17:10:19Z

The committers listed above are authorized under a signed CLA.

✅ login: Jmacek / name: Jmacek (322667b)

k8s-ci-robot · 2026-05-21T17:10:21Z

Welcome @jmacek!

It looks like this is your first PR to kubernetes-client/python 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-client/python has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

jmacek · 2026-05-21T17:27:35Z

/kind bug

v36 rewrote Configuration.auth_settings() to look up the bearer-token credential under api_key['BearerToken'], matching the OpenAPI security scheme name. The in-cluster and kubeconfig loaders were not updated - they still write api_key['authorization'], the v35 lookup key. The same gap exists in kubernetes_asyncio/config/incluster_config.py. As a result, on v36 every call to load_incluster_config() (and load_kube_config() with a static token, including the async equivalents) produces a Configuration whose auth_settings() yields no bearer credential, so outgoing API requests are sent without an Authorization header and the apiserver treats them as system:anonymous. Write the token under both 'authorization' (v35) and 'BearerToken' (v36+) in all three affected loaders so requests carry the expected header. The old key is preserved as a backward-compatibility hedge for any third-party code introspecting api_key['authorization'] directly. Add a regression test in incluster_config_test that drives a real ApiClient.update_params_for_auth() against a freshly-loaded Configuration and asserts the resulting headers contain an Authorization entry - the end-to-end invariant that v36 quietly broke.

faust64 · 2026-05-21T20:18:03Z

 client_configuration.ssl_ca_cert = self.ssl_ca_cert
 if self.token is not None:
 client_configuration.api_key['authorization'] = self.token
+ client_configuration.api_key['BearerToken'] = self.token


Does it make sense to have 2 configurations with that same token?
Either one or the other would be fine. It has to be consistent everywhere.
Do we know why that BearerToken was introduced? What was the intention?

I think this is related https://github.com/kubernetes-client/python/blob/master/CHANGELOG.md#breaking-change-from-upgrading-openapi-generator-to-v660. It was introduced in upstream client generator. Same question here. I think upstream did a replacement, rather than keeping both. cc @yliaog

Thanks both - the original intent of writing both keys was to hedge backward compatibility for any third-party code introspecting api_key['authorization'] directly. But since the rename was already documented as a breaking change in the v6.6.0 generator upgrade, aligning the loaders with that decision is the right call. Switching to a straight replacement.

Address review feedback: the openapi-generator v6.6.0 upgrade declared this rename as a breaking change in the project CHANGELOG, so the loaders should follow the rename rather than preserve the v35 key for backward compatibility. Removes the api_key['authorization'] write in all three affected loaders (sync incluster, sync kube_config, async incluster) and updates the corresponding test assertions to read 'BearerToken'.

yliaog · 2026-05-21T22:38:33Z

thanks for the fix

/lgtm
/approve

k8s-ci-robot · 2026-05-21T22:38:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jmacek, yliaog

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [yliaog]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aojea · 2026-05-22T08:32:29Z

I thought that this was covered by current e2e tests, can we add one test case?

yliaog · 2026-05-22T15:33:00Z

Yes, the current e2e tests are all using config outside cluster, it's better to add one using in cluster config.

…test v36.0.0 rewrote Configuration.auth_settings() to look up the bearer credential under api_key['BearerToken'], but left the hand-written loaders in kubernetes/base/config/ and kubernetes_asyncio/config/ writing api_key['authorization']. Net effect: every load_incluster_config() on v36 produces a Configuration with no bearer credential, requests go out unauthenticated, and the apiserver rejects them as system:anonymous. Cherry-pick the two upstream commits from PR #2585 to write 'BearerToken' in all three loaders, and bump epoch to 1. Drop the cherry-picks once a v36.0.1 upstream release ships with the fix applied. Also add a subpackage test that drives InClusterConfigLoader._set_config and asserts api_key['BearerToken'] is populated, so the fix can't be silently dropped on a future bump. Refs: - Upstream PR: kubernetes-client/python#2585 - Upstream issue: kubernetes-client/python#2591 Signed-off-by: Francesco Bartolini <francesco.bartolini@chainguard.dev> Export: 6f0eda5c49c32b1ea5cb945c2c36e44ec0c20d35

### Context kubernetes v36 introduced two regressions that break local and staging operation: 1. `ApiClient.__deserialize_model()` now accesses `self.configuration` which is unset when constructed via `ApiClient.__new__()` (bypassing `__init__`). 2. `_set_config()` stores exec credentials under `api_key['authorization']` but `auth_settings()` was changed to check `api_key['BearerToken']`, leaving all requests unauthenticated (`system:anonymous`). ### Solution Pin to `<v36` ### References - [kubernetes-client/python v36.0.0 release](https://github.com/kubernetes-client/python/releases/tag/v36.0.0) - [CHANGELOG](https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md) - ![image.png](https://app.graphite.com/user-attachments/assets/091de1b7-34d2-45a1-82c2-733e26ed144f.png) - [Issue #2584 — BearerToken auth regression](kubernetes-client/python#2584) - [Issue #2582 — ApiClient constructor regression](kubernetes-client/python#2582) - [PR #2585 — Upstream fix for BearerToken (merged 2026-05-21)](kubernetes-client/python#2585)

@jmacek

##### [\`36.0.2\`](https://github.com/kubernetes-client/python/releases/tag/v36.0.2) Getting started: ```bash pip install --pre --upgrade kubernetes ``` Or from source, download attached zip file, then ```bash unzip client-python-v36.0.2.zip cd client-python-v36.0.2 python setup-release.py install ``` Then follow examples in <https://github.com/kubernetes-client/python/tree/release-36.0/examples> Changelog: <https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md> --- ##### [\`36.0.1\`](https://github.com/kubernetes-client/python/blob/HEAD/CHANGELOG.md#v3601) Kubernetes API Version: v1.36.1 ##### Bug or Regression - Fix `load_incluster_config()` and `load_kube_config()` (sync and async, with a static token) so requests carry an `Authorization` header on `kubernetes-client/python` v36+. Without this fix, in-cluster pods upgrading to v36 silently send unauthenticated requests and the apiserver rejects them as `system:anonymous`. ([#2585](kubernetes-client/python#2585), [@jmacek](https://github.com/Jmacek)) ##### Deprecation - Support new exec v5 websocket subprotocol ([#2486](kubernetes-client/python#2486), [@aojea](https://github.com/aojea))

@jmacek

##### [\`36.0.2\`](https://github.com/kubernetes-client/python/releases/tag/v36.0.2) Getting started: ```bash pip install --pre --upgrade kubernetes ``` Or from source, download attached zip file, then ```bash unzip client-python-v36.0.2.zip cd client-python-v36.0.2 python setup-release.py install ``` Then follow examples in <https://github.com/kubernetes-client/python/tree/release-36.0/examples> Changelog: <https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md> --- ##### [\`36.0.1\`](https://github.com/kubernetes-client/python/blob/HEAD/CHANGELOG.md#v3601) Kubernetes API Version: v1.36.1 ##### Bug or Regression - Fix `load_incluster_config()` and `load_kube_config()` (sync and async, with a static token) so requests carry an `Authorization` header on `kubernetes-client/python` v36+. Without this fix, in-cluster pods upgrading to v36 silently send unauthenticated requests and the apiserver rejects them as `system:anonymous`. ([#2585](kubernetes-client/python#2585), [@jmacek](https://github.com/Jmacek)) ##### Deprecation - Support new exec v5 websocket subprotocol ([#2486](kubernetes-client/python#2486), [@aojea](https://github.com/aojea))

kubernetes-client/python 36.0.0 regenerated Configuration.auth_settings() to read the token from api_key['BearerToken'], while load_incluster_config() still writes api_key['authorization']. auth_settings() therefore returns {} and every API request is sent without an Authorization header, which the apiserver rejects as system:anonymous (401/403 depending on cluster config). Fixed upstream in 36.0.1 (kubernetes-client/python#2585). This broke every scheduled run launch in prod since 0.30.1 shipped the downgrade from 36.0.2 to 36.0.0 (re-lock side effect of 5c38108). 36.0.1+ is currently unreachable: it requires aiohttp>=3.13.5, which the google-adk 2.3.0 stack caps below. Excluding 36.0.0 resolves to 35.0.0 with no other lock changes; drop the exclusion note once google-adk allows newer aiohttp and 36.0.2+ can be picked up. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Summary Allow Iris controller and test environments to use kubernetes-client v36 while excluding broken v36.0.0. Lock v36.0.3, which contains kubernetes-client/python#2585's bearer-token fix. The v36 lock also adds aiohttp to the existing dependency closure. ## Testing - A temporary in-cluster config loaded v36.0.3 from the lock and produced its bearer authorization header without network access. - 50 focused Kubernetes service tests passed. - 1,502 affected safe tests passed. - The Dockerfile's exact frozen controller sync installed v36.0.3. - CoreWeave live smoke passed on this head: the controller image built, deployed, exercised Kubernetes API operations, and completed the integration pipeline. - Iris unit, lint, integration, CodeQL, docs, Rust, and all ordinary unit shards passed. Opening the draft triggered the first live CoreWeave smoke before that mutation was intended; the current-head rerun was deliberate. ## Non-green CI - Levanter TPU reproduced the same pre-existing FP8 gradient assertion as #8360 (1,369 passed, 1 failed); aggregate `unit-tests` reflects it. Tracked in #8403. - GCP brought up a healthy controller, but both TPU zones reported no capacity. No workers became healthy, so smoke tests did not run. Cleanup passed, deleted the controller VM, and issued deletion requests for the remaining labeled TPUs. - Iris E2E hit its 10-minute cap while apt fetched Playwright dependencies, before tests. Levanter Torch hit its 15-minute cap after FFmpeg setup consumed most of the budget; tests reached 33% with no reported failure before cancellation. Tracked in #8404. Fixes#5924 Co-authored-by: yoblin <268258002+yoblin@users.noreply.github.com>

k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 21, 2026

k8s-ci-robot added the needs-kind Indicates a PR lacks a `kind/foo` label and requires one. label May 21, 2026

k8s-ci-robot requested review from fabianvf and roycaihw May 21, 2026 17:10

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 21, 2026

jmacek force-pushed the fix-incluster-auth-v36-bearertoken-key branch from f651938 to 322667bCompare May 21, 2026 17:12

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels May 21, 2026

jmacek force-pushed the fix-incluster-auth-v36-bearertoken-key branch from 322667b to c3608a4Compare May 21, 2026 18:29

faust64 reviewed May 21, 2026
View reviewed changes

roycaihw mentioned this pull request May 21, 2026
load_incluster_config() sets api_key['authorization'] but auth_settings() expects api_key['BearerToken'] in v36 #2584
Closed

k8s-ci-robot assigned yliaog May 21, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2026

k8s-ci-robot merged commit 46989af into kubernetes-client:masterMay 21, 2026
10 checks passed

aojea mentioned this pull request May 22, 2026
Regression in 36.0.0: in-cluster config no longer attaches Authorization header (every request 401s) #2591
Closed

Tenzer mentioned this pull request May 22, 2026
Update dependency kubernetes to v36 Memrise/copypod#144
Closed
1 task

starboyhassan mentioned this pull request May 22, 2026
v36.0.0 regression: load_incluster_config() sends requests as system:anonymous on GKE #2592
Closed

This was referenced May 22, 2026
[codex] Support kubernetes v36 in Iris marin-community/marin#5917
Closed
Track Iris upgrade past kubernetes-client v35 marin-community/marin#5924
Closed

This was referenced May 23, 2026
[WAITING] chore: Upgrade to kubernetes v36 and fix ApiClient.__new__ incompatibility TangleML/tangle#260
Draft
chore: Pin kubernetes below v36 to avoid breaking API changes TangleML/tangle#259
Merged

jmacek deleted the fix-incluster-auth-v36-bearertoken-key branch May 26, 2026 15:56

pcnudde mentioned this pull request May 27, 2026
Document and scripts for OpenShift deployment NVIDIA/NVFlare#4679
Merged
6 tasks

GK-07 mentioned this pull request May 28, 2026
v36.0.0 regression: exec WebSocket returns 401 Unauthorized on AKS (works in v35.0.0) #2595
Closed

alixirs mentioned this pull request Jun 1, 2026
cncf.kubernetes: support kubernetes client 36.x (>=36.0.1) - 35.x no_proxy regression breaks KubernetesPodOperator/hooks behind a corporate proxy apache/airflow#67845
Closed
2 tasks

This was referenced Jun 26, 2026
Restore kubernetes-client 36.x support in cncf.kubernetes and google providers (401 auth regression) apache/airflow#69023
Open
Cap kubernetes client to <36 to fix cluster auth 401 regression apache/airflow#69025
Closed

aaaaahaaaaa mentioned this pull request Jul 7, 2026
fix(k8s): exclude kubernetes client 36.0.0 with broken in-cluster auth digitl-cloud/interloper#125
Merged

yonromai mentioned this pull request Aug 18, 2026
[iris] Upgrade Kubernetes client to v36.0.3 marin-community/marin#8381
Merged

coderabbitaiBot mentioned this pull request Aug 18, 2026
multi-cluster-fleet: add multi-cluster warm-pool planner, part 1/4 kubernetes-sigs/agent-sandbox#1394
Open

Navigation Menu

Uh oh!

config: write api_key['BearerToken'] so v36+ SDK auth works - #2585

config: write api_key['BearerToken'] so v36+ SDK auth works#2585
k8s-ci-robot merged 2 commits into
kubernetes-client:masterfrom
jmacek:fix-incluster-auth-v36-bearertoken-key

jmacek commented May 21, 2026•
edited
Loading

Uh oh!

linux-foundation-easyclaBot commented May 21, 2026•
edited
Loading

Uh oh!

k8s-ci-robot commented May 21, 2026

Uh oh!

jmacek commented May 21, 2026

Uh oh!

faust64May 21, 2026

Uh oh!

roycaihwMay 21, 2026

Uh oh!

jmacekMay 21, 2026

Uh oh!

yliaog commented May 21, 2026

Uh oh!

k8s-ci-robot commented May 21, 2026

Uh oh!

Uh oh!

aojea commented May 22, 2026

Uh oh!

yliaog commented May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

jmacek commented May 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Why

Test plan

Uh oh!

linux-foundation-easyclaBot commented May 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented May 21, 2026

Uh oh!

jmacek commented May 21, 2026

Uh oh!

faust64May 21, 2026

Choose a reason for hiding this comment

Uh oh!

roycaihwMay 21, 2026

Choose a reason for hiding this comment

Uh oh!

jmacekMay 21, 2026

Choose a reason for hiding this comment

Uh oh!

yliaog commented May 21, 2026

Uh oh!

k8s-ci-robot commented May 21, 2026

Uh oh!

Uh oh!

aojea commented May 22, 2026

Uh oh!

yliaog commented May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jmacek commented May 21, 2026•
edited
Loading

linux-foundation-easyclaBot commented May 21, 2026•
edited
Loading