ARROW-17252: [R] Intermittent valgrind failure by paleolimbot · Pull Request #13773 · apache/arrow

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

paleolimbot · 2022-08-02T01:14:05Z

This PR fixes intermittent leaks that occur after one of the changes from ARROW-16444: when we drain the RecordBatchReader that is emitted from the plan too quickly, it seems, some parts of the plan can leak (I don't know why this happens).

I tried removing various pieces of the RunWithCapturedR() changes (see #13746) but the only thing that removes the errors completely is draining the resulting RecordBatchReader from R (i.e., reader$read_table()) instead of in C++ (i.e., reader->ToTable()). Unfortunately, for user-defined functions to work in a plan we need a C++ level reader->ToTable(). I took the approach here of disabling the C++ level read by default, requiring a user to opt in to the version of collect() that works with a UDF. It's not ideal, but definitely safer (and more clearly marks the user-defined function behaviour as experimental).

I was able to replicate the original leaks but they are few and far between...our tests just happen to create and destroy many, many exec plans and something about the CI environment seems to trigger these more reliably (although the errors don't always occur there, either). Most of the leaks are small but there were some instances where an entire Table leaked.

github-actions · 2022-08-02T01:14:28Z

https://issues.apache.org/jira/browse/ARROW-17252

github-actions · 2022-08-02T01:14:30Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

paleolimbot · 2022-08-02T01:35:37Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-02T01:36:44Z

Revision: 3647aa3

Submitted crossbow builds: ursacomputing/crossbow @ actions-de212b2c40

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-02T12:28:29Z

 #' head()
+#' Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 #'
 register_scalar_function <- function(name, fun, in_type, out_type,


What if you put Sys.setenv(R_ARROW_COLLECT_WITH_UDF = "true") inside of register_scalar_function()? You're already opting-in to UDFs by calling this function, and there's no reason you'd want to call this but not have COLLECT_WITH_UDF working.

nealrichardson · 2022-08-02T13:34:39Z

- dplyr::mutate(fun_result = times_32(value)) %>%
- dplyr::collect() %>%
- dplyr::arrange(row_num)
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


You should be able to remove this now

nealrichardson · 2022-08-02T13:35:15Z

 skip_if_not(CanRunWithCapturedR())
 skip_if_not_available("dataset")
 # Snappy has a UBSan issue: https://github.com/google/snappy/pull/148
+ # TODO(ARROW-17178): remove when user-defined function execution is stabilized


Note that the skip is for snappy ubsan too, so both need to be resolved before we can unskip

(See also ARROW-17283)

nealrichardson · 2022-08-02T13:35:25Z

- dplyr::collect(),
- tibble::tibble(a = 1L, b = 32.0)
- )
+ withr::with_envvar(list(R_ARROW_COLLECT_WITH_UDF = "true"), {


Can revert this

Can we? To avoid the leaks for any subsequent tests we need the variable to not be "true".

I was thinking you can revert this because you've already called register_scalar_function() so the env var will be set.

I see! with_envvar() was definitely not doing the right thing here (which is hopefully why the last valgrind check failed). I fixed it (I think) and requested another check!

paleolimbot · 2022-08-02T14:48:47Z

(See #13779 and #13780 for some other potential fixes, both have their checks pending)

Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

westonpace

On the one hand, I can see why there might be some cases an extra collect is needed, to avoid valgrind errors. On the other hand:

The valgrind errors you are receiving are not the errors I would expect (I would expect direct leaks of scan-related objects).
I'm not entirely sure how this would be related to UDFs.

If this workaround fixes things for a little while then so be it. However, I'd really like to know exactly what is going on. You mentioned you had gotten better at reproducing. Could you share some kind of minimal(ish?) reproducer?

paleolimbot · 2022-08-03T01:12:59Z

If this workaround fixes things for a little while then so be it.

I agree that this is a temporary workaround...I don't think anything all that insidious is going on that affects normal usage, but I also don't want us to get an angry CRAN note about valgrind that attracts attention to anything else we're doing.

I'd really like to know exactly what is going on.

Slight progress there: I tried waiting for the thread pools to finish before unloading the package (#13779) and that seems to remove the errors as well, although it's a bit of a hack in its own way (getting the IO thread pool is not exported in the public headers). That seems consistent with the shutting down of the thread pools leaking something?

Could you share some kind of minimal(ish?) reproducer?

From the arrow/r directory, echo 'devtools::test()' | R --no-save -d "valgrind --tool=memcheck --leak-check=full". Not very minimal, but an improvement over the 5 hours it takes the crossbow job. You can run fewer test files using devtools::test(filter = "some_regex") but I was never able to get an error using a few obvious filters ("dplyr", for example). I wonder if the CI just runs threads really really slowly which is why leaks show up more frequently then.

I'm not entirely sure how this would be related to UDFs.

The error isn't, I don't think, but was exposed by a change that we needed to make UDFs work (we need to evaluate the entire exec plan within one RunWithCapturedR(), which meant calling reader->ToTable() instead of sending the RecordBatchReader to R, then converting it to a table there). Before this PR, all exec plan results were routed through a C++-level reader->ToTable() unless an R-level RecordBatchReader was explicitly requested; after this PR, all exec plans go through an R-level RecordBatchReader unless somebody actually registers a user-defined function.

paleolimbot · 2022-08-04T12:04:45Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-04T12:10:37Z

Revision: 3b71868

Submitted crossbow builds: ursacomputing/crossbow @ actions-511c82c07f

Task	Status
test-r-linux-valgrind

paleolimbot · 2022-08-05T00:48:16Z

The last failure here is disappointing and leaves me stumped as to which change introduced the problem. I will spend some time tomorrow trying to reproduce this so that it's maybe possible to diagnose.

paleolimbot · 2022-08-07T01:01:44Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T01:02:42Z

Revision: 2efade7

Submitted crossbow builds: ursacomputing/crossbow @ actions-d22fe5d8c4

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-07T10:29:01Z

+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")
 })

 test_that("register_user_defined_function() can register multiple kernels", {


Suggested change
test_that("register_user_defined_function() can register multiple kernels", {
test_that("register_scalar_function() can register multiple kernels", {

nealrichardson · 2022-08-07T10:32:33Z

 tibble::tibble(a = 1L, b = 32.0)
 )
+
+ Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF")


NBD but should this more properly go inside of on.exit above? like
on.exit({ unregister_binding("times_32", update_cache = TRUE) # TODO(ARROW-17178) remove the need for this! Sys.unsetenv("R_ARROW_COLLECT_WITH_UDF") })

… previous function name

paleolimbot · 2022-08-07T16:29:08Z

@github-actions crossbow submit test-r-linux-valgrind

github-actions · 2022-08-07T16:30:07Z

Revision: 37e6e2e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d72df52d33

Task	Status
test-r-linux-valgrind

nealrichardson · 2022-08-08T18:32:20Z

@paleolimbot is this good to merge now?

paleolimbot · 2022-08-08T18:43:20Z

Yes! Sorry, forgot to check the valgrind result.

ursabot · 2022-08-09T18:52:30Z

Benchmark runs are scheduled for baseline = 3148884 and contender = 7448322. 7448322 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.54% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.09% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 7448322e ec2-t3-xlarge-us-east-2
[Finished] 7448322e test-mac-arm
[Finished] 7448322e ursa-i9-9960x
[Finished] 7448322e ursa-thinkcentre-m75q
[Finished] 3148884e ec2-t3-xlarge-us-east-2
[Failed] 3148884e test-mac-arm
[Finished] 3148884e ursa-i9-9960x
[Finished] 3148884e ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-09T18:52:49Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

don't collect with an event loop unless requested
73e68de

github-actionsBot added the Component: R label Aug 2, 2022

modify example that needs special handling, skip a test on linux devel
3647aa3

paleolimbot marked this pull request as ready for review August 2, 2022 11:43

paleolimbot requested review from nealrichardson and westonpace August 2, 2022 12:03

nealrichardson reviewed Aug 2, 2022
View reviewed changes

auto opt-in to user-defined function-friendly collect
46d7ef3

nealrichardson reviewed Aug 2, 2022
View reviewed changes

better documentation of test skipping
f9f9e85

paleolimbotand others added 2 commits August 2, 2022 14:40

Update r/R/compute.R
10a9815
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>

document
3b71868

westonpace reviewed Aug 2, 2022
View reviewed changes

more explicit and more correct cleaning up of env var
2efade7

nealrichardson reviewed Aug 7, 2022
View reviewed changes

nealrichardson added a commit to nealrichardson/arrow that referenced this pull request Aug 7, 2022
Applyapache#13773, drop badges, add placeholder write_arrow
e976565

more reliable unsetting of environment variable, remove references to…
37e6e2e
… previous function name

paleolimbot mentioned this pull request Aug 8, 2022
ARROW-17252: [R] Intermittent valgrind failure (explicit StopProducing approach) #13780
Closed

paleolimbot mentioned this pull request Aug 8, 2022
ARROW-17252: [R] Intermittent valgrind failure (wait for thread pool approach) #13779
Closed

nealrichardson merged commit 7448322 into apache:masterAug 9, 2022

paleolimbot deleted the r-valgrind-2 branch August 9, 2022 12:56

asfimport mentioned this pull request Aug 11, 2022
[R] Intermittent valgrind failure #32535
Closed