Skip to content

Replace port_to_expose with port as testcontainers upgrade - #35501

Merged
shunping merged 4 commits into
apache:masterfrom
shunping:fix-port-to-expose
Jul 2, 2025
Merged

Replace port_to_expose with port as testcontainers upgrade#35501
shunping merged 4 commits into
apache:masterfrom
shunping:fix-port-to-expose

Conversation

@shunping

@shunpingshunping commented Jul 1, 2025

Copy link
Copy Markdown
Collaborator

A few tests are failing (https://github.com/apache/beam/actions/runs/15985918137) due to testcontainers upgrade in (#35216).

It seems that after 4.x, testcontainers deprecated port_to_expose (testcontainers/testcontainers-python#296).

This PR attempts to forward fix this by replacing port_to_expose with port in our tests.

{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 11
"modification": 12

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this test suite is necessary, it runs on dataflow.

@shunping

shunping commented Jul 1, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Getting the following error:

E ModuleNotFoundError: No module named 'MySQLdb'
../../build/gradleenv/1398941893/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/mysqldb.py:149: ModuleNotFoundError

After searching on the web, it seems we will need to install mysqlclient.

@shunping

shunping commented Jul 2, 2025

Copy link
Copy Markdown
CollaboratorAuthor

I am trying to find out the root cause of why we need MySQLdb after testcontainers upgrade.

I followed the failed test code(

'mysql': DbData(
lambda: MySqlContainer(), ['mysql:mysql-connector-java:8.0.28'],
'mysql',
'com.mysql.cj.jdbc.Driver')
}
@classmethod
defsetUpClass(cls):
cls.containers= {}
cls.engines= {}
cls.jdbc_configs= {}
fordb_type, db_dataincls.DB_CONTAINER_CLASSPATH_STRING.items():
container=cls.start_container(db_data.container_fn)
cls.containers[db_type] =container
cls.engines[db_type] =sqlalchemy.create_engine(
container.get_connection_url())
), extracted the following code block and ran it on version 3.7.1 and 4.10.0.

fromtestcontainers.mysqlimportMySqlContainercontainer=MySqlContainer()
container.start()
print(container.get_connection_url())

Connection url from version 3.7.1

mysql+pymysql://test:test@localhost:32768/test

Connection url from version 4.10.0

mysql://test:test@localhost:32770/test

In version 3.7.1, it uses pymysql driver:
https://github.com/testcontainers/testcontainers-python/blob/28e3a471c32c1036dd5e37df13cdde3b1ba91000/testcontainers/mysql.py#L67

In version 4.10.0, if we do not specify dialect, it will not contain a driver in the url.

When we call sqlalchemy.create_engine(), the two urls are routed to different code paths

@shunping

shunping commented Jul 2, 2025

Copy link
Copy Markdown
CollaboratorAuthor

By adding dialect when initializing MySqlContainer, i.e. container = MySqlContainer(dialect='pymysql'), I am able to get the same connection url in 4.10.0 as 3.7.1.

@shunping
shunpingforce-pushed the fix-port-to-expose branch from 240df60 to e340e50CompareJuly 2, 2025 02:14
@shunping

Copy link
Copy Markdown
CollaboratorAuthor

Looks like the previous failed test is green now.
https://github.com/apache/beam/actions/runs/16014666896/job/45178934425?pr=35501

@shunping
shunping marked this pull request as ready for review July 2, 2025 03:12
@shunping

shunping commented Jul 2, 2025

Copy link
Copy Markdown
CollaboratorAuthor

cc'ed @Amar3tto@damccorm

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@damccormdamccorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shunping
shunping merged commit 02c587b into apache:masterJul 2, 2025
@shunping
shunping deleted the fix-port-to-expose branch July 2, 2025 13:27
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@shunping@claudevdm@damccorm