Skip to content

ARROW-2799: [Python] Add safe option to Table.from_pandas to avoid unsafe casts - #2504

Closed
kszucs wants to merge 8 commits into
apache:masterfrom
kszucs:ARROW-2799
Closed

ARROW-2799: [Python] Add safe option to Table.from_pandas to avoid unsafe casts#2504
kszucs wants to merge 8 commits into
apache:masterfrom
kszucs:ARROW-2799

Conversation

@kszucs

Copy link
Copy Markdown
Member

Depends on both #2497 and #2503

@kszucs

Copy link
Copy Markdown
MemberAuthor

There is a spurious but pertinacious conda error:

CondaHTTPError: HTTP 500 INTERNAL SERVER ERROR for url <https://conda.anaconda.org/conda-forge/linux-64/repodata.json>
Elapsed: 00:00.413982
CF-RAY: 4541d1fc0b3e54f8-ORD
A remote server error occurred when trying to retrieve this URL.
A 500-type error (e.g. 500, 501, 502, 503, etc.) indicates the server failed to
fulfill a valid request. The problem may be spurious, and will resolve itself if you
try your request again. If the problem persists, consider notifying the maintainer
of the remote server.

@wesm

wesm commented Sep 2, 2018

Copy link
Copy Markdown
Member

This happens quite often. Is there a way to configure conda to retry some number of times, or wait and retry?

@xhochy

Copy link
Copy Markdown
Member

The 500 errors were a general outage of conda which was resolved. conda in the newest version should do some retries.

@xhochy

Copy link
Copy Markdown
Member

@kszucs The dependencies were merged. Can you rebase?

@kszucs
kszucsforce-pushed the ARROW-2799 branch 4 times, most recently from 6dfd903 to ff1c4e5CompareSeptember 6, 2018 11:00
@kszucs

Copy link
Copy Markdown
MemberAuthor

@xhochy rebased, CI fails because mesonbuild

@xhochyxhochy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

@xhochyxhochy closed this Sep 8, 2018
@wesm

wesm commented Sep 8, 2018

Copy link
Copy Markdown
Member

The parquet-cpp merge has caused our merge script to do something pretty wacky. I'm going to try to figure it out

@wesmwesm reopened this Sep 8, 2018
@wesm

wesm commented Sep 8, 2018

Copy link
Copy Markdown
Member

Rebased

jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
…ted by PyArrow
## What changes were proposed in this pull request?
Since 0.11.0, PyArrow supports to raise an error for unsafe cast ([PR](apache/arrow#2504)). We should use it to raise a proper error for pandas udf users when such cast is detected.
Added a SQL config `spark.sql.execution.pandas.arrowSafeTypeConversion` to disable Arrow safe type check.
## How was this patch tested?
Added test and manually test.
Closesapache#22807 from viirya/SPARK-25811.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@wesm@xhochy