Uh oh!
There was an error while loading. Please reload this page.
ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS - #13601
Conversation
There was a problem hiding this comment.
IIRC we recently changed how the Linux installation works. Do these lines still apply?
There was a problem hiding this comment.
Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.
There was a problem hiding this comment.
Is there also a reason for removing the partitioning option?
There was a problem hiding this comment.
This is a newer version of the dataset that is Hive partitioned, so doesn't require explicitly passing partitioning.
There was a problem hiding this comment.
What happens on Windows exactly? These bare FIXMEs are not explanatory.
There was a problem hiding this comment.
This was copied over from the S3 tests. I'm not sure what happens, but I can try and see.
There was a problem hiding this comment.
I deleted the skips and seems to run fine locally and in CI.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
I added this because it got a lot of files in it while building the R doc site; lmk if there's a good reason to not add it.
There was a problem hiding this comment.
No objection; what creates this?
There was a problem hiding this comment.
When I run pkgdown::build_site() within RStudio, during the "Installing package into temporary library" it creates this directory.
There was a problem hiding this comment.
This isn't intentional. I just ran make doc locally. Is something about my R out of date, or does that command need an update?
There was a problem hiding this comment.
Upgrading to roxygen2 7.2 fixed this.
96f7897 to
176a7f5Comparewjones127
commented
Jul 21, 2022
cc @thisisnic would you be wiling to review? |
pitrou
left a comment
There was a problem hiding this comment.
I haven't read in detail, but this looks fine to me overall.
nealrichardson
left a comment
There was a problem hiding this comment.
A few suggestions here and there but overall LGTM, thanks for taking this on.
There was a problem hiding this comment.
No objection; what creates this?
Uh oh!
There was an error while loading. Please reload this page.
| $ tip_amount : num [1:10918] 3 0 4 1 1 6 0 1.35 0 5.9 ... | ||
| $ total_amount : num [1:10918] 18.8 13.3 20.3 15.8 13.3 ... | ||
| $ passenger_count: int [1:10918] 3 2 1 1 1 1 1 1 1 3 ... | ||
| $ tip_pct : num [1:10918] 0.1596 0 0.197 0.0633 0.0752 ... |
There was a problem hiding this comment.
Not in scope here (and probably not relevant for this exact line, this just made me think of it): we should show an example of glimpse() in this vignette somewhere.
There was a problem hiding this comment.
Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.
Uh oh!
There was an error while loading. Please reload this page.
| ``` | ||
| tells the `S3FileSystem` that it should allow the creation of new buckets and to | ||
| talk to Google Storage instead of S3. The latter works because GCS implements an |
There was a problem hiding this comment.
! Why didn't we do this before then?!
| ``` | ||
| gs://anonymous@voltrondata-labs-datasets/nyc-taxi/?retry_limit_seconds=10 | ||
| ``` | ||
There was a problem hiding this comment.
It might be worth showing how this is equivalent to GcsFileSystem$create(anonymous = TRUE, retry_limit_seconds = 10)$path("voltrondata-labs-datasets/nyc-taxi/") or whatever it is. As in, URI query params are useful but not the only way to provide these options.
There was a problem hiding this comment.
Oh, since the FS objects are introduced below, maybe this note about equivalence should be introduced down there.
There was a problem hiding this comment.
Yeah I wasn't sure I wanted to do this refactor, but it did feel like it would almost be easier to introduce GcsFileSystem$create() and then explain options in URIs.
There was a problem hiding this comment.
I've reordered them so we introduce the filesystems and that made it more natural to make the statement you suggested.
thisisnic
commented
Jul 22, 2022
With combined vacation + rstudio::conf I won't have time to look at this til 2nd August, so probably best not wait for me! |
kszucs
commented
Jul 25, 2022
@jonkeane@nealrichardson can we go ahead and merge it? |
nealrichardson
commented
Jul 25, 2022
I already approved and don't feel the need to re-review, I'm sure @wjones127 took care of everything, please merge. |
ursabot
commented
Jul 26, 2022
Benchmark runs are scheduled for baseline = ab8c92c and contender = 42647dc. 42647dc is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
ursabot
commented
Jul 26, 2022
['Python', 'R'] benchmarks have high level of regressions. |
This PR: * Moves minio integration tests into a generic suite that is now run on minio (S3 emulator) and GCS testbench (GCS emulator). This is run in CI. * Move Minio and GCS test server initialization to within the tests. This makes it easier to setup the background processes in a cross-platform way. * MinIO and GCS tests are now run on R Ubuntu CI. MinIO is now run on Windows CI. I couldn't get GCS to run on Windows CI yet, due to some issue where the tests hang (I believe this is an issue with the test setup and not the functionality). See follow up at: [ARROW-17149: [R] Enable GCS tests for Windows](https://issues.apache.org/jira/browse/ARROW-17149) * Sets the default retry timeout to 15 seconds to mitigate issue described by ARROW-17020. This affects explicitly-created fs with `GcsFileSystem$create()` (and `gs_bucket()` introducted in #13601), but not URIs. Authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338#5198ARROW-6364#5201ARROW-6323#5169ARROW-6278#5141ARROW-6360#5329ARROW-6533#5450ARROW-6348#5223ARROW-6337#5399ARROW-10850#9128ARROW-10624#9092ARROW-10386#8549ARROW-6994#23308ARROW-12774#10320ARROW-12670#10287ARROW-16828#13484ARROW-14989#13482ARROW-16977#13514ARROW-13404#10999ARROW-16887#13601ARROW-15906#13206ARROW-15280#13171ARROW-16144#13183ARROW-16511#13105ARROW-16085#13088ARROW-16715#13555ARROW-16268#13550ARROW-16700#13518ARROW-16807#13583ARROW-16871#13517ARROW-16415#13190ARROW-14821#12154ARROW-16439#13174ARROW-16394#13118ARROW-16516#13163ARROW-16395#13627ARROW-14848#12589ARROW-16407#13196ARROW-16653#13506ARROW-14575#13160ARROW-15271#13170ARROW-16703#13650ARROW-16444#13397ARROW-15016#13541ARROW-16776#13563ARROW-15622#13090ARROW-18131#14484ARROW-18305#14581ARROW-18285#14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR:
voltrondata-labs-datasetsbucket.gs_bucket()function to R, which parallels the design of thes3_bucket()function.fs.Rmdvignette to also discuss GCS. Left discussion about authentication for a follow-up, partly because I am still confused how GCP auth works for non-application use.