GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018

Merged
kou merged 5 commits into
apache:mainfrom
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged

GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom
danepitkin:danepitkin/gh-36642-warnings

Conversation

@danepitkin

@danepitkindanepitkin commented Aug 3, 2023•
edited by jorisvandenbossche
Loading

Copy link
Copy Markdown
Member

Rationale for this change

Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.

What changes are included in this PR?

  • Fixed existing warnings
  • Set warnings as errors in CI

Are these changes tested?

Yes, ran pytests locally w/o warnings.

Are there any user-facing changes?

No

@github-actions

github-actionsBot commented Aug 3, 2023

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@danepitkin

danepitkin commented Aug 3, 2023

Copy link
Copy Markdown
MemberAuthor

I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).

@danepitkin

danepitkin commented Aug 3, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-python*

@github-actions

github-actionsBot commented Aug 3, 2023

Copy link
Copy Markdown

Revision: 9195826

Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c

TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions

@danepitkin

danepitkin commented Aug 3, 2023

Copy link
Copy Markdown
MemberAuthor

For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).

____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin

danepitkin commented Aug 3, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-python*

@github-actions

github-actionsBot commented Aug 3, 2023

Copy link
Copy Markdown

Revision: 6db95f6

Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67

TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions

@danepitkin

danepitkin commented Aug 4, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-python*

@github-actions

github-actionsBot commented Aug 4, 2023

Copy link
Copy Markdown

Revision: fca7663

Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0

TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions

@danepitkindanepitkin changed the title WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions

github-actionsBot commented Aug 4, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.

kou
kou approved these changes Aug 7, 2023
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

BTW, are you interested in taking over #36745 from me...?

@kou
kou merged commit 2b36521 into apache:mainAug 7, 2023
@koukou removed the awaiting review Awaiting review label Aug 7, 2023
@github-actionsgithub-actionsBot added the awaiting merge Awaiting merge label Aug 7, 2023
@danepitkin

danepitkin commented Aug 7, 2023

Copy link
Copy Markdown
MemberAuthor

Hey @kou, sure I can take a look!

@kou

kou commented Aug 8, 2023

Copy link
Copy Markdown
Member

Thanks!

@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 8, 2023
Closed
@conbench-apache-arrow

conbench-apache-arrowBot commented Aug 11, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

@AlenkaFAlenkaF mentioned this pull request Aug 22, 2023
Closed
@raulcdraulcd mentioned this pull request Oct 12, 2023
Closed
loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees

No one assigned

Labels

awaiting mergeAwaiting mergeComponent: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test [Python] ResourceWarnings raised by s3 related tests

2 participants

@danepitkin@kou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-36642: [Python][CI] Configure warnings as errors during pytest - #37018
Merged
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Aug 7, 2023
Merged
GH-36642: [Python][CI] Configure warnings as errors during pytest#37018
kou merged 5 commits into
apache:mainfrom 
danepitkin:danepitkin/gh-36642-warnings
Conversation
@danepitkin
@danepitkindanepitkin
commented
Aug 3, 2023•

edited by jorisvandenbossche

Loading

Copy link


Copy Markdown

Member
Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
What changes are included in this PR?
  • Fixed existing warnings
  • Set warnings as errors in CI
Are these changes tested?
Yes, ran pytests locally w/o warnings.
Are there any user-facing changes?
No
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Thanks for opening a pull request!
If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or
MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
I enabled warnings as errors for conda python 3.8 through 3.11 and for conda python pandas integration to see what happens. Realistically, we probably want to limit warnings to just the latest python version (currently 3.11).
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 9195826
Submitted crossbow builds: ursacomputing/crossbow @ actions-1e510b906c
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
For python 3.8 through 3.11 there is only 1 warning (repeated in 5 test cases).
____________________ ERROR at setup of test_read_s3fs[True] ____________________
___________________ ERROR at setup of test_read_s3fs[False] ____________________
_______________ ERROR at setup of test_read_directory_s3fs[True] _______________
______________ ERROR at setup of test_read_directory_s3fs[False] _______________
__________ ERROR at setup of test_write_metadata_fs_file_combinations __________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fd5884620f0>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:341: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/runner.py:262: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_hooks.py:433: in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/pluggy/_manager.py:112: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:83: in pytest_runtest_setup
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
E E Traceback (most recent call last):
E File "/opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
E fixture_result = fixturefunc(**kwargs)
E ResourceWarning: unclosed <socket.socket fd=189, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35258), raddr=('127.0.0.1', 56633)>
opt/conda/envs/arrow/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

@danepitkin
danepitkin
commented
Aug 3, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 3, 2023

Copy link


Copy Markdown

Revision: 6db95f6
Submitted crossbow builds: ursacomputing/crossbow @ actions-2a9b540c67
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkin
danepitkin
commented
Aug 4, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit test-conda-python*
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

Revision: fca7663
Submitted crossbow builds: ursacomputing/crossbow @ actions-45800058d0
TaskStatus
test-conda-python-3.10Github Actions
test-conda-python-3.10-hdfs-2.9.2Github Actions
test-conda-python-3.10-hdfs-3.2.1Github Actions
test-conda-python-3.10-pandas-latestGithub Actions
test-conda-python-3.10-pandas-nightlyGithub Actions
test-conda-python-3.10-spark-v3.4.1Github Actions
test-conda-python-3.10-substraitGithub Actions
test-conda-python-3.11Github Actions
test-conda-python-3.11-dask-latestGithub Actions
test-conda-python-3.11-dask-upstream_develGithub Actions
test-conda-python-3.11-hypothesisGithub Actions
test-conda-python-3.11-pandas-upstream_develGithub Actions
test-conda-python-3.11-spark-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-pandas-1.0Github Actions
test-conda-python-3.8-spark-v3.4.1Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-latestGithub Actions
@danepitkindanepitkin
changed the title
WIP: [Python][CI] Configure warnings as errors during pytestGH-36642: [Python][CI] Configure warnings as errors during pytestAug 4, 2023
@github-actions
github-actionsBot
commented
Aug 4, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #36642has been automatically assigned in GitHub to PR creator.
kou
kou
approved these changes
Aug 7, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
BTW, are you interested in taking over #36745 from me...?
@kou
kou
merged commit 2b36521
into
apache:mainAug 7, 2023
@koukou
removed
the 
awaiting review
Awaiting review
label
Aug 7, 2023
@github-actionsgithub-actionsBot
added
the 
awaiting merge
Awaiting merge
label
Aug 7, 2023
@danepitkin
danepitkin
commented
Aug 7, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @kou, sure I can take a look!
@kou
kou
commented
Aug 8, 2023

Copy link


Copy Markdown

Member
Thanks!
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Aug 8, 2023

Closed

@conbench-apache-arrow
conbench-apache-arrowBot
commented
Aug 11, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2b36521.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
@AlenkaFAlenkaF
mentioned this pull request
Aug 22, 2023

Closed

@raulcdraulcd
mentioned this pull request
Oct 12, 2023

Closed

loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@danepitkin@loicalleyne
apacheGH-36642: [Python][CI] Configure warnings as errors during pyte…
b6e8eb1
…st (apache#37018)
### Rationale for this change
Warnings are constantly being introduced into the pyarrow tests. Let's try enforcing them as errors in an effort to keep the codebase healthy.
### What changes are included in this PR?
* Fixed existing warnings
* Set warnings as errors in CI
### Are these changes tested?
Yes, ran pytests locally w/o warnings.
### Are there any user-facing changes?
No
* Closes: apache#36642
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@raulcdraulcdAwaiting requested review from raulcdraulcd is a code owner

Assignees


No one assigned


Labels

awaiting mergeAwaiting mergeComponent: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python][CI] Configure warnings as errors during build/test

[Python] ResourceWarnings raised by s3 related tests


2 participants

@danepitkin@kou