Uh oh!
There was an error while loading. Please reload this page.
Add nullValue being respected when parsing CSVs - #224
Conversation
This change makes it so that we look for a user specified nullValue through the CSV parsing. This allows for handling CSVs that might use something else other than an empty string to represent nulls. It reuses the same flag as CSV saving, `nullValue`. This change should be non-breaking. This also pushes this behavior into inferSchema so that inferred schemas will properly reflect the user given null value.
codecov-io
commented
Dec 29, 2015
Current coverage is |
addisonj
commented
Dec 29, 2015
Just now noticed #76 does the same thing 🤦 It seems like that is pretty stale and might mess with inferSchema for those cases... But does accept a range of null values (not sure how likely it is that a single CSV will have multiple types of null?) But as is mentioned in that PR and in other issues, it really seems like null handling should be a part of CSV handling as we have large CSV files where it would be too expensive to remap just for remapping nulls. |
falaki
commented
Jan 6, 2016
Thanks @addisonj this looks good. I am going to merge it. |
addisonj
commented
Jan 6, 2016
Thanks! Currently running from a fork, any ETA on when you will be cutting a new release? |
ankurmitujjain
commented
May 14, 2016
Is this part of release? |
ankurmitujjain
commented
May 14, 2016
Not working on Databricks notebook.. I tried to load csv data with ? as null values. |
rjurney
commented
Sep 25, 2016
So this seems to still not be working in 2.0.0? |
This change makes it so that we look for a user specified nullValue
through the CSV parsing. This allows for handling CSVs that might use
something else other than an empty string to represent nulls.
It reuses the same flag as CSV saving,
nullValue. This change shouldbe non-breaking.
This also pushes this behavior into inferSchema so that inferred schemas
will properly reflect the user given null value.