Skip to content

Add use_krb5ccache option to SparkSubmitHook - #34386

Merged
eladkal merged 1 commit into
apache:mainfrom
zeotuan:main
Oct 21, 2023
Merged

Add use_krb5ccache option to SparkSubmitHook#34386
eladkal merged 1 commit into
apache:mainfrom
zeotuan:main

Conversation

@zeotuan

@zeotuanzeotuan commented Sep 15, 2023•
edited by eladkal
Loading

Copy link
Copy Markdown
Contributor

closes: #11246 - Add use_krb5ccache option to tell spark to ticket cache instead of relying on keytab for Kerberos login
https://spark.apache.org/docs/latest/security.html#using-a-ticket-cache

@potiuk

potiuk commented Sep 15, 2023

Copy link
Copy Markdown
Member

You need to make it compatible with Airflow 2.4. Seems that there is no airflow.security.kerberos in 2.4 (see the error in CI) so you need to see what was there and make conditional code based on airflow.__version__ and comment to remove it after the compatibility is not needed - i.e. when we go to 2.5+ or 2.6+ support - depending when it was aded (you can search for similar examples in other providers)

@Lee-W

Lee-W commented Oct 4, 2023

Copy link
Copy Markdown
Member

I believe the CI failure in this branch has been resolved by #34701. Could you please rebase the branch and see whether it passes the CI? Thanks!

Lee-W
Lee-W reviewed Oct 4, 2023
View reviewed changes
Comment threadairflow/providers/apache/spark/hooks/spark_submit.py Outdated
Comment threadtests/providers/apache/spark/hooks/test_spark_submit.py Outdated
@zeotuan
zeotuan requested a review from Lee-WOctober 9, 2023 23:05
eladkal
eladkal reviewed Oct 13, 2023
View reviewed changes
Comment threadairflow/security/kerberos.py Outdated
@zeotuanzeotuan mentioned this pull request Oct 14, 2023
Merged
@zeotuan
zeotuan requested a review from eladkalOctober 20, 2023 02:26
@eladkal
eladkal merged commit 645d52f into apache:mainOct 21, 2023
@eladkal

eladkal commented Oct 24, 2023

Copy link
Copy Markdown
Contributor

@zeotuan do we need to expose this parameter in SparkSubmitOperator as well?

@zeotuan

zeotuan commented Oct 25, 2023

Copy link
Copy Markdown
ContributorAuthor

Oh Yes we do. I will submit a PR for that

@eladkaleladkal mentioned this pull request Oct 28, 2023
Closed
27 tasks
@zeotuanzeotuan mentioned this pull request Nov 1, 2023
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TaragolisTaragolisTaragolis approved these changes
@Lee-WLee-WLee-W approved these changes
@eladkaleladkaleladkal approved these changes

Assignees

No one assigned

Labels

area:providersprovider:apache-spark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SparkSubmitHook Kerberos ccache support

5 participants

@zeotuan@potiuk@Lee-W@eladkal@Taragolis