Uh oh!
There was an error while loading. Please reload this page.
ARROW-5480: [Python] Add unit test asserting specifically that pandas.Categorical roundtrips to Parquet format without special options - #5110
Conversation
wesm
commented
Aug 16, 2019
@jorisvandenbossche@jreback the pandas test suite will probably need some changes or expansions now that Categorical (for strings at least) can be faithfully roundtripped using |
jorisvandenbossche
commented
Aug 16, 2019
@wesm indeed, created pandas-dev/pandas#27955 to track that. |
There was a problem hiding this comment.
this comment reads a bit unclear to me (it now reads back categorical?)
There was a problem hiding this comment.
Sorry I copy-pasted this from the bug report
jorisvandenbossche
commented
Aug 16, 2019
@wesm to what extent is this "fully" faithful for corner cases? (if not, might need to mention that as caveats in the pandas docs) For example for a categorical with values in the "categories" which are not present in the data, is this preserved on reading back? (I suppose we use the categories when creating a DictionaryArray, but are its dictionary's values exactly preserved in the parquet roundtrip?) |
wesm
commented
Aug 16, 2019
The category values will be exactly preserved whether or not they occur in the data. I can expand the unit test to exhibit this if it helps |
wesm
commented
Aug 16, 2019
Done |
wesm
commented
Aug 19, 2019
Rebased |
| assert result.schema.metadata is None | ||
| @pytest.mark.pandas |
There was a problem hiding this comment.
in the rest of the file, test functions using pandas are marked as such?
There was a problem hiding this comment.
Yes, this is a rebase artifact, fixing
jorisvandenbossche
commented
Aug 19, 2019
Updated test looks good, thanks for the clarification! |
wesm
commented
Aug 19, 2019
codecov-io
commented
Aug 19, 2019
Codecov Report
@@ Coverage Diff @@## master #5110 +/- ##
===========================================
- Coverage 87.62% 65.02% -22.61%
===========================================
Files 1014 495 -519 Lines 145828 67082 -78746 Branches 1437 0 -1437 ===========================================
- Hits 127788 43619 -84169 - Misses 17678 23463 +5785 + Partials 362 0 -362
Continue to review full report at Codecov.
|
This only works for string types for the moment. Once ARROW-6277 is addressed we can expand to other types.