Uh oh!
There was an error while loading. Please reload this page.
ARROW-6360: [R] Update support for compression - #5329
Conversation
pitrou
commented
Sep 9, 2019
I don't want to opiniate too much on this (not a R developer), but I'm curious why the choice of snappy (which AFAIK is little used nowadays, as there are better options). Also, why would gzip fail on Windows? Gzip is so universal that hopefully it should be supported on Windows as well. |
nealrichardson
commented
Sep 9, 2019
It all depends on what the C++ library is built with on Windows. According to https://github.com/apache/arrow/blob/master/ci/PKGBUILD#L88, gzip should be there. Snappy support was added specifically because it was (reported to be) common with Parquet files. Speaking of Parquet, as the ticket said it would be nice to support writing to parquet with compression (cf. #5192, which further added support to the C++ library for specifying a compression level). |
pitrou
commented
Sep 9, 2019
Right, so personally I would make gzip the default everywhere, if you do need a default. |
nealrichardson
commented
Sep 9, 2019
Agreed. |
wesm
commented
Sep 9, 2019
Snappy should be the default compression when writing Parquet files. Make sure it isn't gzip or benchmarks will suffer (almost every library out there defaults to Snappy AFAIK) But anywhere else where there is a need for a compression gzip seems like the reasonable default |
3fb788b to
c8021a5Compare
nealrichardson
left a comment
There was a problem hiding this comment.
I rebased and pushed a few changes. I guess this R code is covered by test-compression.R, and the real business is covered in the C++ library elsewhere.
If it makes sense, you could separate the parquet encryption support to a different ticket and merge this.
| # more concisely | ||
| # rap( ~ decor:::parse_cpp_function(context)) | ||
| mutate(functions = map(context, decor:::parse_cpp_function)) %>% | ||
| { bind_cols(., bind_rows(pull(., functions))) } %>% |
There was a problem hiding this comment.
Is this necessary? If so, please add vctrs to the README where the codegen script dependencies are discussed.
There was a problem hiding this comment.
Nevermind, I see we already Suggestvctrs.
codecov-io
commented
Sep 11, 2019
Codecov Report
@@ Coverage Diff @@## master #5329 +/- ##
=========================================
Coverage ? 76.09% =========================================
Files ? 56 Lines ? 3572 Branches ? 0 =========================================
Hits ? 2718 Misses ? 854 Partials ? 0Continue to review full report at Codecov.
|
nealrichardson
commented
Sep 11, 2019
+1. I created https://issues.apache.org/jira/browse/ARROW-6532 and https://issues.apache.org/jira/browse/ARROW-6533 for followups. |
codec = NULL means choose the right default for the current platform. Default is set for `SNAPPY` on windows and `GZIP` otherwise, instead of universally using `GZIP` as the default and abort on windows. Closesapache#5329 from romainfrancois/ARROW-6360-compression-update and squashes the following commits: 88fbf00 <Neal Richardson> Remove windows test skip 7d83d2b <Neal Richardson> Move compression_codec to Codec; add docs; allow lowercase compression types c8021a5 <Romain Francois> Making GZIP default everywhere 9cc2b43 <Romain Francois> using vctrs::vec_bind instead of dplyr::bind_ 3727946 <Romain Francois> codec = NULL means choose the right default for the current platform. Lead-authored-by: Romain Francois <romain@rstudio.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338#5198ARROW-6364#5201ARROW-6323#5169ARROW-6278#5141ARROW-6360#5329ARROW-6533#5450ARROW-6348#5223ARROW-6337#5399ARROW-10850#9128ARROW-10624#9092ARROW-10386#8549ARROW-6994#23308ARROW-12774#10320ARROW-12670#10287ARROW-16828#13484ARROW-14989#13482ARROW-16977#13514ARROW-13404#10999ARROW-16887#13601ARROW-15906#13206ARROW-15280#13171ARROW-16144#13183ARROW-16511#13105ARROW-16085#13088ARROW-16715#13555ARROW-16268#13550ARROW-16700#13518ARROW-16807#13583ARROW-16871#13517ARROW-16415#13190ARROW-14821#12154ARROW-16439#13174ARROW-16394#13118ARROW-16516#13163ARROW-16395#13627ARROW-14848#12589ARROW-16407#13196ARROW-16653#13506ARROW-14575#13160ARROW-15271#13170ARROW-16703#13650ARROW-16444#13397ARROW-15016#13541ARROW-16776#13563ARROW-15622#13090ARROW-18131#14484ARROW-18305#14581ARROW-18285#14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
codecov-commenter
commented
Aug 1, 2025
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@## master #5329 +/- ##
===========================================
- Coverage 88.72% 76.09% -12.64%
===========================================
Files 945 56 -889 Lines 124147 3572 -120575 Branches 1437 0 -1437 ===========================================
- Hits 110146 2718 -107428 + Misses 13639 854 -12785 + Partials 362 0 -362 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
codec = NULL means choose the right default for the current platform. Default is set for
SNAPPYon windows andGZIPotherwise, instead of universally usingGZIPas the default and abort on windows.