Use pure-sasl in python 3.11 by mdeshmu · Pull Request #454 · dropbox/PyHive

mdeshmu · 2023-05-20T01:24:22Z

There is a known issue with installing sasl lib in Python 3.11.

Maintainers of the sasl repo haven't contributed to it since 2021.

This PR adds additional extras pyhive[hive_pure_sasl] which uses pure-sasl allowing end user to choose what sasl library they want to use.

Existing pyhive[hive] extras will continue to use sasl library.

With these code changes, pyhive can work with both python-sasl and pure-sasl with sasl being preferred first.

Please refer below links for the CI test logs from apache/superset repository:
3.11 (pure-sasl) -> https://github.com/apache/superset/actions/runs/5302603180/jobs/9597648049
3.10 (pure-sasl) -> https://github.com/apache/superset/actions/runs/5302917493/jobs/9598184738
3.9 (pure-sasl) -> https://github.com/apache/superset/actions/runs/5303030643/jobs/9598385720

3.10 (sasl) -> https://github.com/apache/superset/actions/runs/5302756384/jobs/9597916446
3.9 (sasl) -> https://github.com/apache/superset/actions/runs/5302964104/jobs/9598269981?pr=24157

JimNero009 · 2023-05-25T08:12:50Z

Stumbled upon this while doing some general py3.11 upgrades across some projects (transitive dep here via dbt-spark hive extras). Very keen for an installation path to be available, anything I can do to help?

bkyryliuk

LG%nit

tyong920 · 2023-06-21T10:01:32Z

Awesome work! Thanks you guys! Looking forward to a release ASAP!

colin-k-rogers · 2023-06-28T18:22:01Z

Any idea when this is going to be released?

mdeshmu · 2023-07-14T15:03:55Z

@JimNero009 @tyong920 @colin-rogers-dbt @mb-m

This is released in 0.7.1.dev0. You are requested to test with the dev version and report any bugs in the PyHive GitHub repository before 0.7.1 is released in a month or so.

PyHive now supports pure-sasl via additional extras 'pyhive[hive_pure_sasl]' which supports Python 3.11 in addition to previous Python versions. See #454

colin-k-rogers · 2023-08-03T19:31:59Z

@mdeshmu still planning on releasing in the next week or two?

mdeshmu · 2023-08-22T00:13:23Z

@colin-rogers-dbt this is released in 0.7.0

* feat: add HTTP and HTTPS to hive (dropbox#385) * feat: add https protocol * support HTTP * fix: make hive https py2 compat (dropbox#389) * fix: make hive https py2 compat * fix lint * Update README.rst (dropbox#423) * chore: rename Trino entry point (dropbox#428) * Support for Presto decimals (dropbox#430) * Support for Presto decimals * lower * Use str type for driver and name in HiveDialect (dropbox#450) PyHive's HiveDialect usage of bytes for the name and driver fields is not the norm is causing issues upstream: apache/superset#22316 Even other dialects within PyHive use strings. SQLAlchemy does not strictly require a string, but all the stock dialects return a string, so I figure it is heavily implied. I think the risk of breaking something upstream with this change is low (but it is there ofc). I figure in most cases we just make someone's `str(dialect.driver)` expression redundant. Examples for some of the other stock sqlalchemy dialects (name and driver fields using str): https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/sqlite/pysqlite.py#L501 https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/sqlite/base.py#L1891 https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/base.py#L2383 https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/mysqldb.py#L113 https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/pymysql.py#L59 * Correcting Iterable import for python 3.10 (dropbox#451) * changing drivers to support hive, presto and trino with sqlalchemy>=2.0 (dropbox#448) * Revert "changing drivers to support hive, presto and trino with sqlalchemy>=2.0 (dropbox#448)" (dropbox#452) This reverts commit b0206d3. * Update __init__.py (dropbox#453) dropbox@1c1da8bdropbox@1f99552 * use pure-sasl with python 3.11 (dropbox#454) * minimal changes for sqlalchemy 2.0 support (dropbox#457) * update readme to reflect recent changes (dropbox#459) * Update README.rst (dropbox#475) * Update README.rst (dropbox#476) * feat: JWT support * Add CI to build package --------- Co-authored-by: Daniel Vaz Gaspar <danielvazgaspar@gmail.com> Co-authored-by: Bogdan <b.kyryliuk@gmail.com> Co-authored-by: serenajiang <serena.jiang@airbnb.com> Co-authored-by: Usiel Riedl <usiel.riedl@gmail.com> Co-authored-by: Multazim Deshmukh <57723564+mdeshmu@users.noreply.github.com> Co-authored-by: nicholas-miles <nicholas.miles6@gmail.com>

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch 3 times, most recently from 5bd0564 to 0421cdaCompare May 22, 2023 14:30

This was referenced May 22, 2023
Test pure sasl with hive python 3.11 apache/superset#24155
Closed
Test pure sasl with hive python 3.10 apache/superset#24156
Closed
Test pure sasl with hive python 3.9 apache/superset#24157
Closed
Support Python 3.11 cloudera/python-sasl#30
Open

bkyryliuk reviewed May 24, 2023
View reviewed changes

Comment threadpyhive/hive.py Outdated

bkyryliuk reviewed May 24, 2023
View reviewed changes

Comment threadpyhive/sasl_compat.py

bkyryliuk reviewed May 24, 2023
View reviewed changes

Comment threadpyhive/tests/test_hive.py Outdated

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch 2 times, most recently from 52f9340 to dd62269Compare May 29, 2023 14:34

mdeshmu requested a review from bkyryliuk May 29, 2023 16:37

bkyryliuk reviewed May 30, 2023
View reviewed changes

bkyryliuk reviewed May 31, 2023
View reviewed changes

Comment threadpyhive/tests/test_sasl_compat.py

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch 7 times, most recently from 1e48903 to 36e6364Compare June 4, 2023 15:08

mdeshmu closed this Jun 4, 2023

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch from 36e6364 to 0bd6f5bCompare June 4, 2023 15:33

mdeshmu reopened this Jun 5, 2023

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch from 9539992 to 2c9a6feCompare June 5, 2023 10:27

mdeshmu requested a review from bkyryliuk June 5, 2023 12:05

mdeshmu commented Jun 6, 2023
View reviewed changes

Comment threadpyhive/hive.py Outdated

hsheth2 mentioned this pull request Jun 7, 2023
ci(ingest): test with python 3.11 datahub-project/datahub#6485
Closed
5 tasks

bkyryliuk reviewed Jun 13, 2023
View reviewed changes

Comment threadpyhive/tests/test_hive.py Outdated

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch from f6b2dce to e38eb23Compare June 18, 2023 07:09

mdeshmu requested a review from bkyryliuk June 18, 2023 09:40

bkyryliuk reviewed Jun 20, 2023
View reviewed changes

Comment threadpyhive/hive.py Outdated

bkyryliuk reviewed Jun 20, 2023
View reviewed changes

Comment threadpyhive/hive.py

bkyryliuk approved these changes Jun 20, 2023
View reviewed changes

use pure-sasl with python 3.11
1eef88b

mdeshmu force-pushed the use-pure-sasl-in-python-3.11 branch from 78691ce to 1eef88bCompare June 20, 2023 10:35

bkyryliuk merged commit 4367cc5 into dropbox:masterJun 20, 2023

hsheth2 pushed a commit to acryldata/PyHive that referenced this pull request Jun 29, 2023
use pure-sasl with python 3.11 (dropbox#454)
c6085e0

hsheth2 mentioned this pull request Jun 29, 2023
use pure-sasl with python 3.11 acryldata/PyHive#7
Merged

mdeshmu mentioned this pull request Jul 14, 2023
Run test for Python-3.11 astronomer/astronomer-providers#1237
Merged

This was referenced Aug 7, 2023
Bumping version to 1.3.3 and generate changelog dbt-labs/dbt-spark#853
Merged
Bumping version to 1.4.3 and generate changelog dbt-labs/dbt-spark#855
Merged
Bumping version to 1.5.2 and generate changelog dbt-labs/dbt-spark#854
Merged

JCZuurmond mentioned this pull request Aug 9, 2023
[ADAP-789] [Feature] Upgrade PyHive to support Python 3.11 dbt-labs/dbt-spark#864
Closed
3 tasks

mdeshmu mentioned this pull request Aug 22, 2023
build(deps): Bump PyHive apache/superset#25030
Merged
9 tasks

mdeshmu deleted the use-pure-sasl-in-python-3.11 branch August 23, 2023 00:35

hussein-awala mentioned this pull request Feb 10, 2024
Fix pyhive hive_pure_sasl extra name apache/airflow#37323
Merged

Uh oh!

Conversation

mdeshmu commented May 20, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JimNero009 commented May 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkyryliuk left a comment

Choose a reason for hiding this comment

Uh oh!

tyong920 commented Jun 21, 2023

Uh oh!

colin-k-rogers commented Jun 28, 2023

Uh oh!

mdeshmu commented Jul 14, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colin-k-rogers commented Aug 3, 2023

Uh oh!

mdeshmu commented Aug 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mdeshmu commented May 20, 2023•
edited
Loading

mdeshmu commented Jul 14, 2023•
edited
Loading