ARROW-11119: [Rust] Expose functions to parse a single CSV column / StringRecord into an array / recordBatch

[jorgecarleitao]

This PR exposes two new functions:

• to parse a single array out of a CSV column of a [StringRecord]) and
• to parse a RecordBatch out of a StringRecord.

The motivation for the first function is that parsing arrays is trivially parallelizable. Thus, people may want to use e.g rayon to iterate in parallel over fields to build each array. IMO arrow crate should not make any assumption about how people want to parallelize this, and only offer the functionality to do it, in the same way we do it with kernels.

The motivation for the second function stems from the fact that parsing (not the IO reading) is the slowest operation in reading a CSV and people may want to iterate over the CSV differently. The main use-case here is to split the read of a single CSV file in multiple parts (using seek), and returning record batches (DataFusion is the example here) in parallel. Again, IMO the arrow crate should not make assumptions about how to perform this work, and instead offer the necessary CPU-blocking core functionality for users to build on top of.

The latter function is just a utility of the former function on which no parallelism is used (arrays are built in sequence).

[github-actions]

https://issues.apache.org/jira/browse/ARROW-11119

[codecov-io]

Codecov Report

Merging #9084 (f764311) into master (dfef236) will decrease coverage by 0.00%.
The diff coverage is 92.59%.

@@ Coverage Diff @@
## master #9084 +/- ##
==========================================
- Coverage 82.60% 82.60% -0.01% ==========================================
Files 204 204 Lines 50189 50174 -15 ==========================================
- Hits 41459 41446 -13 + Misses 8730 8728 -2 

Impacted Files	Coverage Δ
rust/arrow/src/csv/reader.rs	93.14% <92.59%> (-0.02%)	⬇️
rust/arrow/src/record_batch.rs	73.83% <0.00%> (-2.24%)	⬇️
rust/arrow/src/array/array_struct.rs	88.43% <0.00%> (-0.18%)	⬇️
rust/arrow/src/array/equal/mod.rs	92.69% <0.00%> (+0.37%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfef236...f764311. Read the comment docs.

[Dandandan]

This looks cool @jorgecarleitao !

Some thoughts for the future of csv /other parsers:

• It might be worth exploring if we can use the cast (or a similar) kernel of Arrow to parse the data. The benefit of this would be that we can just load the data (as bytes / string) into arrays and utilize the existing parsing logic in Arrow. I think this is interesting because the code can be vectorized/use SIMD, parallelized, etc. more easily from that point, will reduce code duplication, and creates more incentive to improve the cast kernels, which benefits more than "only" one parser.
• For further optimization it might be worth to stop using the StringRecords at some point (and use csv_core), as there is quite some overhead associated with them compared to "just" loading the bytes from the file. How would this fit into your suggestion?
• For a user like DataFusion, it might often not make sense to have parallelism on the file level (if there are many files), so I think it makes sense to not make the parser slower / consume more resources for one thread. This is more a general thing we should keep in mind.

[andygrove]

Great idea @jorgecarleitao 🚀

[Dandandan]

The downside is that this creates a dependency on StringRecord in the public API, making it harder to remove it when we want?

[Dandandan]

👍 I think the v.clone() could maybe even be removed?

[alamb]

Given the age of this PR I think we should rebase it against latest master prior to merging it in

[Dandandan]

Do you still want to move this in @jorgecarleitao ?
I had a few thoughts that followed from this PR:

• Making StringRecord public now would mean that we introduce a breaking change when changing the implementation of the csv parser.
• We can spend some more time optimizing the parsers themselves, e.g. for csv parsing we can remove the overhead of the csv crate (which seems a big chunk of reading csv according to profiling results), avoiding copies, utf-8 validation in some cases, etc.
• The parsers could use a cast kernel, this requires some more changes to cast. From there, additional parallelization on parsing might be easier / generic / reusable as we already have a number of Arrays. This might even enable more advanced things like using SIMD for parsing some data types.

[jorgecarleitao]

@Dandandan , you have a good point, and since you are working on the cast for this, I would also prefer to wait and see the result of that before committing to the StringRecord.

[alamb]

@jorgecarleitao

What is the status of this PR?

As part of trying to clean up the backlog of Rust PRs in this repo, I am going through seemingly stale PRs and pinging the authors to see if there are any plans to continue the work or conversation.

[jorgecarleitao]

Thanks @alamb . Let's wait for #8781 to see if this is still needed.