Uh oh!
There was an error while loading. Please reload this page.
Update Arrow to the latest release tag (2.0.0) - #2
Merged
Conversation
tempdir is deprecated (https://crates.io/crates/tempdir) Closesapache#8157 from nevi-me/ARROW-9957 Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
This PR speeds up some of the aggregations in arrow by 10-60% by simplifying their logic and overall allowing the optimizer to do its work. The first 3 commits (up to 29754d7) simply improve the benchmark itself by: * not taking the creation of the arrays into account, only the computation, * moving it to another file * adding randomness to the data to reduce spurious results due to speculative execution and others * add case for data with nulls, since the kernels branch out on that condition The last 3 commits are the optimizations themselves. ``` sum 512 time: [535.66 ns 536.11 ns 536.57 ns] change: [-58.421% -58.222% -57.957%] (p = 0.00 < 0.05) Performance has improved. Found 10 outliers among 100 measurements (10.00%) 4 (4.00%) high mild 6 (6.00%) high severe min 512 time: [766.77 ns 775.85 ns 788.35 ns] change: [-41.555% -41.017% -40.388%] (p = 0.00 < 0.05) Performance has improved. Found 10 outliers among 100 measurements (10.00%) 4 (4.00%) high mild 6 (6.00%) high severe sum nulls 512 time: [1.0968 us 1.1000 us 1.1038 us] change: [-8.9918% -7.6232% -5.7130%] (p = 0.00 < 0.05) Performance has improved. Found 15 outliers among 100 measurements (15.00%) 3 (3.00%) high mild 12 (12.00%) high severe min nulls 512 time: [1.3208 us 1.3242 us 1.3286 us] change: [-11.028% -10.240% -9.4581%] (p = 0.00 < 0.05) Performance has improved. Found 11 outliers among 100 measurements (11.00%) 3 (3.00%) high mild 8 (8.00%) high severe ``` Command: ``` git checkout 29754d7 && cargo bench --bench aggregate_kernels && git checkout agg_arrow && cargo bench --bench aggregate_kernels ``` Closesapache#8165 from jorgecarleitao/agg_arrow Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
…Systems Closesapache#8101 from bkietz/9868-Provide-utility-for-copyi Authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
… WriteRecordBatch `ArrowStreamWriter` and `ArrowFileWriter` currently only supports an `WriteRecordBatchAsync`. This PR implements a synced version of it. All the logic has been reused and all `async` calls have been replaced with `sync` counterparts. Closesapache#8146 from suhsteve/writerecordbatch Authored-by: Steve Suh <suhsteve@gmail.com> Signed-off-by: Eric Erhardt <eric.erhardt@microsoft.com>
Simplified the internal code and moved some things to the R layer. This should make it easier to alter which attribute is kept for various types. Closesapache#8150 from romainfrancois/ARROW-9271/metadata Lead-authored-by: Romain Francois <romain@rstudio.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Romain François <romain@rstudio.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
R follow up from apache#7272 The current `$select()` uses a more familiar (though more expensive) tidyselect interface: ``` r library(arrow, warn.conflicts = FALSE) tab <- Table$create(x1 = 1:2, x2 = 3:4, y = 5:6) # lower level 0-based indices tab$SelectColumns(0:1) #> Table #> 2 rows x 2 columns #> $x1 <int32> #> $x2 <int32> # higher level tidyselect based tab$select(starts_with("x")) #> Table #> 2 rows x 2 columns #> $x1 <int32> #> $x2 <int32> ``` <sup>Created on 2020-09-07 by the [reprex package](https://reprex.tidyverse.org) (v0.3.0.9001)</sup> Do we want both ? `$select()` is used e.g. by the `read_csv(col_select=)` argument: ```r tab <- reader$Read()$select(!!enquo(col_select)) ``` Closesapache#8125 from romainfrancois/ARROW-9387/Table_SelectColumns Lead-authored-by: Romain Francois <romain@rstudio.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
This fixes a lot of clippy errors and warnings. We currently have over 600 such, and they tend to cause a lot of noise while working. This PR only includes changes to arrow, one change to datafusion and integration testing. I've left parquet out to potentially open a separare PR. Thoughts @andygrove@alamb@jorgecarleitao@paddyhoran? Closesapache#8168 from nevi-me/clippy-fixes-arrow-sep2020 Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Andy Grove <andygrove73@gmail.com>
This functionality is relevant for the DataFrame API only.
Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.
This PR proposes that, given a UDF `pow: ScalarUDF` named `"pow"`, users can (logically) plan it directly:
```rust
// plan a call
let expr = pow.call(vec![col("a"), col("b")]);
```
or register it (as before)
```rust
// register it
ctx.register_udf(pow);
```
or plan it from the registry (as before):
```rust
// access it from the registry via its name, when `pow` is not in scope
let pow = df.registry().udf("pow")?;
// plan a call
let expr = pow.call(vec![col("a"), col("b")]);
```
I changed the signature of the registry from `.udf(name, args) -> Expr` to `.udf(name) -> ScalarUDF`, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly `call(args)`. IMO it also makes it a bit more expressive that we are calling the `udf`.
Closesapache#8144 from jorgecarleitao/udf_usage
Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>…atures as functions. This commit makes the creation of aggregate expressions similar to the creation of built-in expressions. This allows to share coercion rules and signatures between scalar and aggregate functions. Closesapache#8155 from jorgecarleitao/aggregates Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com> Signed-off-by: Andy Grove <andygrove73@gmail.com>
…stamp without timezone offset as local The TO_TIMESTAMP function added in apache#8142 supports parsing timestamps without a specified timezone, such as `2020-09-08T13:42:29.190855` Such timestamps are supposed to be interpreted as in the local timezone, but instead are interpreted as UTC. This PR corrects that logical error Here is a query before this PR showing the problem on my machine (in `America/New_York` timezone) - note the values of the timestamp are the same for both the `Z` and non `Z` timestamps: ``` > select to_timestamp('2020-09-08T13:42:29.190855'), to_timestamp('2020-09-08T13:42:29.190855Z') from ts_table limit 1; +-------------------------------------------------+--------------------------------------------------+ | totimestamp(Utf8("2020-09-08T13:42:29.190855")) | totimestamp(Utf8("2020-09-08T13:42:29.190855Z")) | +-------------------------------------------------+--------------------------------------------------+ | 1599572549000190855 | 1599572549190855000 | +-------------------------------------------------+--------------------------------------------------+ 1 rows in set. Query took 0 seconds. ``` Here is a query after this PR: ``` > select to_timestamp('2020-09-08T13:42:29.190855'), to_timestamp('2020-09-08T13:42:29.190855Z') from ts_table limit 1; +-------------------------------------------------+--------------------------------------------------+ | totimestamp(Utf8("2020-09-08T13:42:29.190855")) | totimestamp(Utf8("2020-09-08T13:42:29.190855Z")) | +-------------------------------------------------+--------------------------------------------------+ | 1599586949000190855 | 1599572549190855000 | +-------------------------------------------------+--------------------------------------------------+ 1 rows in set. Query took 0 seconds. ``` Closesapache#8161 from alamb/alamb/local_timezone Authored-by: alamb <andrew@nerdnetworks.org> Signed-off-by: Andy Grove <andygrove73@gmail.com>
I added two test cases in apache#8007, which increased coverage. However, upon further review, I noticed the choice of parameters to hit edge conditions didn't cover the string data types. Rather than adding a bunch more copies of basically the same test to add new parameters for different tests, I instead propose using the same set of parameters for all data types and drive the tests using a table in this PR. It makes the test logic slightly harder to follow, in my opinion, but it does increase coverage Closesapache#8009 from alamb/alamb/ARROW-9790-test-consolidation Authored-by: alamb <andrew@nerdnetworks.org> Signed-off-by: Andy Grove <andygrove73@gmail.com>
This is a followup of https://issues.apache.org/jira/browse/ARROW-9979Closesapache#8173 from nevi-me/clippy-fixes-parquet-sep2020 Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
https://issues.apache.org/jira/browse/ARROW-9737Closesapache#7965 from zhztheplayer/ARROW-9737 Authored-by: Hongze Zhang <hongze.zhang@intel.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
Closesapache#8176 from jorgecarleitao/clean Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
A rebase and significant rewrite of sunchao/parquet-rs#197 Big improvement: I now use a more natural nested enum style, it helps break out what patterns of data types are . The rest of the broad strokes still apply. Goal === Writing many columns to a file is a chore. If you can put your values in to a struct which mirrors the schema of your file, this `derive(ParquetRecordWriter)` will write out all the fields, in the order in which they are defined, to a row_group. How to Use === ``` extern crate parquet; #[macro_use] extern crate parquet_derive; #[derive(ParquetRecordWriter)] struct ACompleteRecord<'a> { pub a_bool: bool, pub a_str: &'a str, } ``` RecordWriter trait === This is the new trait which `parquet_derive` will implement for your structs. ``` use super::RowGroupWriter; pub trait RecordWriter<T> { fn write_to_row_group(&self, row_group_writer: &mut Box<RowGroupWriter>); } ``` How does it work? === The `parquet_derive` crate adds code generating functionality to the rust compiler. The code generation takes rust syntax and emits additional syntax. This macro expansion works on rust 1.15+ stable. This is a dynamic plugin, loaded by the machinery in cargo. Users don't have to do any special `build.rs` steps or anything like that, it's automatic by including `parquet_derive` in their project. The `parquet_derive/src/Cargo.toml` has a section saying as much: ``` [lib] proc-macro = true ``` The rust struct tagged with `#[derive(ParquetRecordWriter)]` is provided to the `parquet_record_writer` function in `parquet_derive/src/lib.rs`. The `syn` crate parses the struct from a string-representation to a AST (a recursive enum value). The AST contains all the values I care about when generating a `RecordWriter` impl: - the name of the struct - the lifetime variables of the struct - the fields of the struct The fields of the struct are translated from AST to a flat `FieldInfo` struct. It has the bits I care about for writing a column: `field_name`, `field_lifetime`, `field_type`, `is_option`, `column_writer_variant`. The code then does the equivalent of templating to build the `RecordWriter` implementation. The templating functionality is provided by the `quote` crate. At a high-level the template for `RecordWriter` looks like: ``` impl RecordWriter for $struct_name { fn write_row_group(..) { $({ $column_writer_snippet }) } } ``` this template is then added under the struct definition, ending up something like: ``` struct MyStruct { } impl RecordWriter for MyStruct { fn write_row_group(..) { { write_col_1(); }; { write_col_2(); } } } ``` and finally _THIS_ is the code passed to rustc. It's just code now, fully expanded and standalone. If a user ever changes their `struct MyValue` definition the `ParquetRecordWriter` will be regenerated. There's no intermediate values to version control or worry about. Viewing the Derived Code === To see the generated code before it's compiled, one very useful bit is to install `cargo expand` [more info on gh](https://github.com/dtolnay/cargo-expand), then you can do: ``` $WORK_DIR/parquet-rs/parquet_derive_test cargo expand --lib > ../temp.rs ``` then you can dump the contents: ``` struct DumbRecord { pub a_bool: bool, pub a2_bool: bool, } impl RecordWriter<DumbRecord> for &[DumbRecord] { fn write_to_row_group( &self, row_group_writer: &mut Box<parquet::file::writer::RowGroupWriter>, ) { let mut row_group_writer = row_group_writer; { let vals: Vec<bool> = self.iter().map(|x| x.a_bool).collect(); let mut column_writer = row_group_writer.next_column().unwrap().unwrap(); if let parquet::column::writer::ColumnWriter::BoolColumnWriter(ref mut typed) = column_writer { typed.write_batch(&vals[..], None, None).unwrap(); } row_group_writer.close_column(column_writer).unwrap(); }; { let vals: Vec<bool> = self.iter().map(|x| x.a2_bool).collect(); let mut column_writer = row_group_writer.next_column().unwrap().unwrap(); if let parquet::column::writer::ColumnWriter::BoolColumnWriter(ref mut typed) = column_writer { typed.write_batch(&vals[..], None, None).unwrap(); } row_group_writer.close_column(column_writer).unwrap(); } } } ``` now I need to write out all the combinations of types we support and make sure it writes out data. Procedural Macros === The `parquet_derive` crate can ONLY export the derivation functionality. No traits, nothing else. The derive crate can not host test cases. It's kind of like a "dummy" crate which is only used by the compiler, never the code. The parent crate cannot use the derivation functionality, which is important because it means test code cannot be in the parent crate. This forces us to have a third crate, `parquet_derive_test`. I'm open to being wrong on any one of these finer points. I had to bang on this for a while to get it to compile! Potentials For Better Design === - [x] Recursion could be limited by generating the code as "snippets" instead of one big `quote!` AST generator. Or so I think. It might be nicer to push generating each columns writing code to another loop. - [X] ~~It would be nicer if I didn't have to be so picky about data going in to the `write_batch` function. Is it possible we could make a version of the function which accept `Into<DataType>` or similar? This would greatly simplify this derivation code as it would not need to enumerate all the supported types. Something like `write_generic_batch(&[impl Into<DataType>])` would be neat.~~ (not tackling in this generation of the plugin) - [X] ~~Another idea to improving writing columns, could we have a write function for `Iterator`s? I already have a `Vec<DumbRecord>`, if I could just write a mapping for accessing the one value, we could skip the whole intermediate vec for `write_batch`. Should have some significant memory advantages.~~ (not tackling in this generation of the plugin, it's a bigger parquet-rs enhancement) - [X] ~~It might be worthwhile to derive a parquet schema directly from a struct definition. That should stamp out opportunities for type errors.~~ (moved to apache#203) Status === I have successfully integrated this work with my own data exporter (takes postgres/couchdb and outputs a single parquet file). I think this code is worth including in the project, with the caveat that it only generates simplistic `RecordWriter`s. As people start to use we can add code generation for more complex, nested structs. We can convert the nested matching style to a fancier looping style. But for now, this explicit nesting is easier to debug and understand (to me at least!). Closesapache#4140 from xrl/parquet_derive Lead-authored-by: Xavier Lange <xrlange@gmail.com> Co-authored-by: Neville Dipale <nevilledips@gmail.com> Co-authored-by: Bryant Biggs <bryantbiggs@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
Lots of assorted things here: * Automatically generate global wrappers for calling registered compute functions * Improve metadata of such wrappers (e.g. name, docstring) * Make it easier to pass options (for example via kwargs) * Type-check options * Add some docstrings * Expose more function attributes (e.g. arity) * Fix some crashes Closesapache#8163 from pitrou/ARROW-9465-py-compute-cosmetics Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Allow passing a %-encoded secret key in a S3 URI. Also, detect and report unrecognized options in S3 URIs. Closesapache#8185 from pitrou/ARROW-9859-s3-uri-decode-key Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Reported here: https://issues.apache.org/jira/browse/ARROW-9973 Looks like a pretty straightforward fix. Let me know if you have any comments - thanks! Closesapache#8167 from pwoody/pw/ARROW-9973 Authored-by: Patrick Woody <pwoody@branta.io> Signed-off-by: liyafan82 <fan_li_ya@foxmail.com>
This PR speeds-up arithmetic ops by leveraging vectorization of non-divide operations (in non-SIMD), as well as removing an un-needed operation in SIMD division. For non-SIMD, this yields about `[-30%,-45%]` for all operations (`+-*/`) For SIMD, this yields about `-30%` on division. The culprit in non-SIMD was that we required the operation to return `Result<T::Native>`, which was not allowing the compiler to vectorize the operation. Only the division requires `Result`. For divide, removing the operator further speed up the operation (I do not know the reason). The culprit in SIMD was primarily a `simd_load` too many that was not doing anything. ## Benchmarks The benchmark used: ``` set -e git checkout 0852869 cargo bench --bench arithmetic_kernels git checkout divide_simd_faster cargo bench --bench arithmetic_kernels echo "##################################" git checkout 0852869 cargo bench --bench arithmetic_kernels --features simd git checkout divide_simd_faster cargo bench --bench arithmetic_kernels --features simd ``` and below are the results for the execution of the second `bench`, which is the one that gives the differential, in my machine: ### Non-SIMD ``` Previous HEAD position was 0852869 Improved benches for arithmetic. Switched to branch 'divide_simd_faster' Compiling arrow v2.0.0-SNAPSHOT (/Users/jorgecarleitao/projects/arrow/rust/arrow) Finished bench [optimized] target(s) in 37.24s Running /Users/jorgecarleitao/projects/arrow/rust/target/release/deps/arithmetic_kernels-d281862a43faaf38 Gnuplot not found, using plotters backend add 512 time: [1.4714 us 1.4758 us 1.4803 us] change: [-44.446% -43.969% -43.522%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high severe subtract 512 time: [1.4825 us 1.4844 us 1.4866 us] change: [-45.351% -45.018% -44.686%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe multiply 512 time: [1.4895 us 1.4936 us 1.4990 us] change: [-44.822% -44.135% -43.479%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 4 (4.00%) high mild 5 (5.00%) high severe divide 512 time: [1.9742 us 1.9773 us 1.9810 us] change: [-33.273% -32.688% -32.052%] (p = 0.00 < 0.05) Performance has improved. Found 14 outliers among 100 measurements (14.00%) 7 (7.00%) high mild 7 (7.00%) high severe limit 512, 512 time: [374.66 ns 375.64 ns 376.53 ns] change: [-0.1000% +0.4442% +0.9503%] (p = 0.10 > 0.05) No change in performance detected. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low severe 2 (2.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe add_nulls_512 time: [1.4880 us 1.4982 us 1.5115 us] change: [-44.084% -43.116% -42.111%] (p = 0.00 < 0.05) Performance has improved. Found 16 outliers among 100 measurements (16.00%) 3 (3.00%) high mild 13 (13.00%) high severe divide_nulls_512 time: [1.9731 us 1.9758 us 1.9790 us] change: [-33.404% -32.570% -31.416%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) high mild 6 (6.00%) high severe ``` ### SIMD divide is the only relevant ``` Previous HEAD position was 0852869 Improved benches for arithmetic. Switched to branch 'divide_simd_faster' Compiling arrow v2.0.0-SNAPSHOT (/Users/jorgecarleitao/projects/arrow/rust/arrow) Finished bench [optimized] target(s) in 38.63s Running /Users/jorgecarleitao/projects/arrow/rust/target/release/deps/arithmetic_kernels-b8dc1739cfb5ae36 Gnuplot not found, using plotters backend add 512 time: [879.31 ns 883.95 ns 889.17 ns] change: [-0.2041% +0.6502% +1.5484%] (p = 0.15 > 0.05) No change in performance detected. Found 16 outliers among 100 measurements (16.00%) 5 (5.00%) high mild 11 (11.00%) high severe subtract 512 time: [864.99 ns 866.95 ns 868.95 ns] change: [-4.8531% -4.1561% -3.5163%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 2 (2.00%) high mild 5 (5.00%) high severe multiply 512 time: [862.85 ns 864.87 ns 867.71 ns] change: [-3.8532% -3.1774% -2.4459%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high severe divide 512 time: [1.9703 us 1.9771 us 1.9843 us] change: [-30.046% -29.457% -28.903%] (p = 0.00 < 0.05) Performance has improved. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high severe limit 512, 512 time: [368.89 ns 369.96 ns 370.96 ns] change: [-1.9574% -1.0063% -0.0347%] (p = 0.04 < 0.05) Change within noise threshold. Found 26 outliers among 100 measurements (26.00%) 5 (5.00%) low severe 6 (6.00%) low mild 9 (9.00%) high mild 6 (6.00%) high severe add_nulls_512 time: [871.97 ns 876.99 ns 883.57 ns] change: [-5.1106% -3.6889% -2.3080%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) high mild 6 (6.00%) high severe divide_nulls_512 time: [1.9582 us 1.9625 us 1.9678 us] change: [-34.188% -33.161% -32.136%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) high mild 6 (6.00%) high severe ``` Closesapache#8191 from jorgecarleitao/divide_simd_faster Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com> Signed-off-by: Neville Dipale <nevilledips@gmail.com>
Closesapache#8192 from xhochy/ARROW-10011 Authored-by: Uwe L. Korn <uwe.korn@quantco.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
…pe (int/string) Pandas dataframe to pyarrow Table
This PR homogenizes error messages for mixed-type `Pandas` inputs to `pa.Table`.
The message for `Pandas` column with `int` followed by `string` is now
```
In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')
```
the same as for `double` followed by `string`:
```
In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')
```
As a side effect, this snippet [xref apache#5866, ARROW-7168] now throws an `ArrowInvalid` (has been `FutureWarning` since 0.16):
```
In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object))
...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64())
...: result = pa.array(cat, type=typ)
(... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int
```
Finally, this *does* break a test [xref apache#4484, ARROW-4036] - see code comment
Closesapache#8044 from arw2019/ARROW-7663
Authored-by: arw2019 <andrew.r.wieteska@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>Fixesapache#8128Closesapache#8127 from xhochy/ARROW-8359 Lead-authored-by: Uwe L. Korn <uwe.korn@quantco.com> Co-authored-by: Uwe L. Korn <xhochy@users.noreply.github.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
- Fix a bug where writes would hang forever for DoExchange - Make FlightRuntimeException#toString easier to read - Have DoPut reliably clean up the FlightStream when the call ends (instead of potentially closing it after gRPC thinks the call ends - this will be important for [ARROW-9586](https://issues.apache.org/jira/browse/ARROW-9586)) Closesapache#8010 from lidavidm/arrow-9587 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>
Closesapache#7844 from domoritz/patch-2 Authored-by: Dominik Moritz <domoritz@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Closesapache#8193 from pitrou/ARROW-10012-mockfs-thread-safe Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
…or creating maintenance branches Refactored the release related utilities and also covered the main components with unit tests. Updated the cherry-picking command to calculate the patches required to pick but not yet picked to the maintenance branch, so the cherry-picking procedure can be restarted now. 1. `archery release cherry-pick 1.0.1 --execute --recreate`: re-creates `maint-1.0.x` and tries to apply all the required patches 2. `archery release cherry-pick 1.0.1 --execute --continue`: in case of a merge conflict the process can be continued Closesapache#7932 from kszucs/archery-patch-release Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
I haven't had the time to investigate the build logs, but it will make the build queue pretty slow. Closesapache#8198 from kszucs/ARROW-10018 Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…pressions. Also moved operators to their own module. Closesapache#8182 from jorgecarleitao/ops Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com> Signed-off-by: Andy Grove <andygrove@nvidia.com>
Changes introduced in 0.15.0 changed the buffer alignment by adding a continuation marker to messages. The previous behaviour was then marked as legacy, while both worked under V4 of the IPC metadata version. This change catches the Rust implementation up to other languages, and has the consequence that more integration tests now pass. The change is applied on top of clippy changes, and can be reviewed/merged after the relevant PRs. Closesapache#8174 from nevi-me/ARROW-9848-on-clippy Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Andy Grove <andygrove@nvidia.com>
…actional seconds As pointed out on @jhorstmann: apache#8161 (comment) > One (not directly related) issue I noticed while trying this out, is that the local patterns seem to require the millisecond part, while for utc timestamps with "Z" they are optional: ``` > select to_timestamp('2020-09-12T10:30:00') from test limit 1; ArrowError(ExternalError(General("Error parsing \'2020-09-12T10:30:00\' as timestamp"))) > select to_timestamp('2020-09-12T10:30:00Z') from test limit 1; +-------------------------------------------+ | totimestamp(Utf8("2020-09-12T10:30:00Z")) | +-------------------------------------------+ | 1599906600000000000 | +-------------------------------------------+ ``` This PR allows parsing local timestamp patterns without the fractional seconds and tests for same Closesapache#8179 from alamb/alamb/ARROW-9986-fractional-secs Authored-by: alamb <andrew@nerdnetworks.org> Signed-off-by: Andy Grove <andygrove@nvidia.com>
… feature is not available Closesapache#8427 from kszucs/flight-test Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…enchmark query Closesapache#8409 from jhorstmann/ARROW-10240-load-data-into-memory-for-tpch Authored-by: Jörn Horstmann <git@jhorstmann.net> Signed-off-by: Andy Grove <andygrove@nvidia.com>
…atically using new implementation In principle, the user can also simply pass `use_legacy_dataset=False`, but I think it is 1) a bit more user friendly to do this automatically (it's clear that you need this with a new filesystem anyway) and 2) also clearer if a user switches from legacy to new HadoopFileSystem to have this working (instead of an error message that seems to indicate it is not supported). Closesapache#8414 from jorisvandenbossche/ARROW-7957 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…ethod Closesapache#8301 from jorisvandenbossche/ARROW-10100-parquetfilefragment-subset Lead-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…dependency version Closesapache#8429 from kszucs/ARROW-9553 Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…d updating homebrew Closesapache#8431 from kszucs/macwhl Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…s in parallel Closesapache#8428 from andygrove/ARROW-10251 Authored-by: Andy Grove <andygrove73@gmail.com> Signed-off-by: Andy Grove <andygrove@nvidia.com>
Tested by running tests, benchmarks and examples. Did this with `simd` on and off The main one here is `packed_simd` > `packed_simd_2`, as the former is broken on recent nightlies Closesapache#8433 from nevi-me/ARROW-10271 Authored-by: Neville Dipale <nevilledips@gmail.com> Signed-off-by: Andy Grove <andygrove@nvidia.com>
…in Gandiva This patch attempts to resolve the bug introduced by the addition of round() in ARROW-9641 round() for floats is returning incorrect results for some edge cases, like round(cast(1.55 as float), 1) gives 1.6, but it should be 1.5, since the result after casting 1.55 to float comes to 1.5499999523162842, due to inaccurate representation of floating point numbers in memory. Removing an intermediate explicit cast to float statement for a double value, which is used in subsequent computations, minimises the error introduced due to the incorrect representation. Closesapache#8398 from sagnikc-dremio/round-double and squashes the following commits: b1ef543 <Sagnik Chakraborty> ARROW-10234: Fix logic of round() for floats/decimals in Gandiva Authored-by: Sagnik Chakraborty <sagnikc@dremio.com> Signed-off-by: Praveen <praveen@dremio.com>
Alias `TypeClass` in `BinaryScalar` and `LargeBinaryScalar` are seemingly typo-ed to be `BinaryScalar` and `LargeBinaryScalar`. This causes issues when using `ScalarType::TypeClass`, esp. with `TypeTrait` - i.e., compiler complains that there are no whatever members in specialized `TypeTrait<BinaryScalar>` and `TypeTrait<LargeBinaryScalar>`. Fixing them to `BinaryType` and `LargeBinaryType`. Closesapache#8423 from zanmato1984/arrow-10262 Authored-by: zanmato1984 <zanmato1984@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
- Check `DEFINED` to avoid re-running the check on every CMake invocation - Log compiler output at DEBUG level to avoid dumping it by default - Use CMAKE_MESSAGE_INDENT to make it clearer that this is the output of a subcommand Closesapache#8442 from lidavidm/arrow-10286 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
Closesapache#8445 from pitrou/ARROW-10288-compile-i386 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
std::random_device may block on some systems, even for long times. Closesapache#8444 from pitrou/ARROW-10287-avoid-dev-random Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Closesapache#8439 from jorisvandenbossche/ARROW-10281-test-warnings Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…tem on import Closesapache#8440 from jorisvandenbossche/ARROW-10284 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
….orc submodule Closesapache#8441 from jorisvandenbossche/ARROW-10285 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Follow-up of apache#8431Closesapache#8436 from kszucs/macos-artifact-patterns Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
It should be ran against installed formula. Closesapache#8432 from kou/ci-homebrew Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…the new macos wheel platform tags Closesapache#8435 from kszucs/verify-macos-wheels Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…ions Closesapache#8451 from kszucs/cmake-compat Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
…-arrow-2.0.0 [maven-release-plugin] copy for tag apache-arrow-2.0.0
qoega pushed a commit
that referenced
this pull request
Aug 2, 2021
Before change: ``` Direct leak of 65536 byte(s) in 1 object(s) allocated from: #0 0x522f09 in #1 0x7f28ae5826f4 in #2 0x7f28ae57fa5d in #3 0x7f28ae58cb0f in #4 0x7f28ae58bda0 in ... ``` After change: ``` Direct leak of 65536 byte(s) in 1 object(s) allocated from: #0 0x522f09 in posix_memalign (/build/cpp/debug/arrow-dataset-file-csv-test+0x522f09) #1 0x7f28ae5826f4 in arrow::(anonymous namespace)::SystemAllocator::AllocateAligned(long, unsigned char**) /arrow/cpp/src/arrow/memory_pool.cc:213:24 #2 0x7f28ae57fa5d in arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::SystemAllocator>::Allocate(long, unsigned char**) /arrow/cpp/src/arrow/memory_pool.cc:405:5 #3 0x7f28ae58cb0f in arrow::PoolBuffer::Reserve(long) /arrow/cpp/src/arrow/memory_pool.cc:717:9 #4 0x7f28ae58bda0 in arrow::PoolBuffer::Resize(long, bool) /arrow/cpp/src/arrow/memory_pool.cc:741:7 ... ``` Closesapache#10498 from westonpace/feature/ARROW-13027--c-fix-asan-stack-traces-in-ci Authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
This was referenced Oct 15, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.