ARROW-12657: [C++] Adding String hex to numeric conversion by wmalpica · Pull Request #11064 · apache/arrow

wmalpica · 2021-09-02T00:56:25Z

This PR adds the ability for the StringConverter to parse hex strings of the form:
0x123abc to an integer.
The strings must start with "0x" case sensitive.
The values ABCDEF are case insensitive.

Note that there was already a Hex Parsing function here:
https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/string.cc#L76
Which was not really flexible enough for what was needed.
Not sure it it would be best to replace it with the new functionality added by this PR. I am open to suggestions

github-actions · 2021-09-02T00:56:42Z

https://issues.apache.org/jira/browse/ARROW-12657

github-actions · 2021-09-02T00:56:43Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

edponce · 2021-09-02T01:50:03Z

Having a more general hex string to number parser is good, and I would recommend for it to substitute the current simple parser. Replacing the hex parser will involve changing files not necessarily associated directly with this PR so you may want to consider to open a JIRA for the replacement.

edponce · 2021-09-02T01:55:09Z

 #undef PARSE_UNSIGNED_ITERATION_LAST

+#define PARSE_HEX_ITERATION(C_TYPE) \
+ if (length > 0) { \


I suggest to make this a function. The use of macros is undesired in the general case.

I agree with this recommendation. I was following the pattern of how ParseUnsigned and PARSE_UNSIGNED_ITERATION is done in the same file. Since I am new to the codebase I decided to follow the patterns I was seeing.

edponce · 2021-09-02T02:04:04Z

+
+
+inline bool ParseHex(const char* s, size_t length, uint8_t* out) {
+ uint8_t result = 0;


Instead of having multiple functions for each type, I suggest to use templates, and, given that each function is conceptually doing the same operation but different number of times based on the output data type width, use a for-loop instead.
template <typename T> inlineboolParseHex(...) { ... for (int i = 0; i < sizeof(T) * 2; ++i) { ParseHexIteration<T>(result); } ... }

As you prefer, but I would be fine with keeping this a macro in this case. Note there is a control flow inside the macro so you cannot turn it into such a trivial loop.

I turned it into a loop, it works fine

edponce · 2021-09-02T02:12:35Z

 return false;
 }
+ // If its starts with 0x then its hex
+ if (*s == '0' && *(s + 1) == 'x'){


Add support for case-insensitive 0x.

edponce · 2021-09-02T02:17:34Z

+ return false;
+ }
+ return true; 
+ }


You can simplify these checks and boolean result as
returnARROW_PREDICT_TRUE((sizeof(value_type) * 2 >= length) && ParseHex(s, length, out));

edponce · 2021-09-02T02:27:33Z

+
+ // If its starts with 0x then its hex
+ if (*s == '0' && *(s + 1) == 'x'){
+ length -= 2;


If there is a negative sign, then it is not a well-formed hex string.

Good catch. Fixed this

edponce · 2021-09-02T02:31:08Z

+ if (*s == '0' && *(s + 1) == 'x'){
+ length -= 2;
+ s += 2;
+ // lets make sure that the length of the string is not too big


Nit: You probably not need to decrease length and consider the offset of s as the difference to consider.

I opted not to do this, since length is used both at StringToSignedIntConverterMixin and at ParseHex. Getting rid of length would then involve keeping track of the original pointer of s and the new pointer and comparing that to the original length. It seems more convoluted

…gned integer vectors When adding a byte `0xff` to a UInt1Vector, the toString method produces `[-1]`. Since the vector contains unsinged integers, the correct result should be `[255]`. Closesapache#11029 from liyafan82/fly_0830_uin Authored-by: liyafan82 <fan_li_ya@foxmail.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

…hanges to Vectors) See https://issues.apache.org/jira/browse/ARROW-13544 According to the discussion in apache#10864 (comment), we want to split the task into multiple parts. This PR is for the changes related to vectors Closesapache#10910 from liyafan82/fly_0810_depv Authored-by: liyafan82 <fan_li_ya@foxmail.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

Exclude .factorypath files generated by Eclipse IDE for configuring annotation processing from Git commit and RAT plugin scans. Closesapache#11042 from laurentgo/laurentgo/exclude-factorypath Authored-by: Laurent Goujon <laurent@apache.org> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

…hanges to JDBC) See https://issues.apache.org/jira/browse/ARROW-13544 According to the discussion in apache#10864 (comment), we want to split the task into multiple parts. This PR is for the changes related to the JDBC adapter Closesapache#10912 from liyafan82/fly_0811_depj Authored-by: liyafan82 <fan_li_ya@foxmail.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>

Essentially, this failure boils down to: when generating the array of uniques for booleans, we pack 8 bytes at a time into one byte. The bytes are packed from what turns out to be a scratch array allocated by TempVectorStack, which does not initialize its memory. So when we have a non-multiple-of-8 number of bytes, we may end up packing initialized bytes and uninitialized bytes together into a single garbage byte, resulting in Valgrind complaining. Closesapache#11041 from lidavidm/arrow-13812 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: Yibo Cai <yibo.cai@arm.com>

Closesapache#11045 from cyb70289/13067-int2dec Authored-by: Yibo Cai <yibo.cai@arm.com> Signed-off-by: Yibo Cai <yibo.cai@arm.com>

Should fix the following issues found by OSS-Fuzz: * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37927 * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37915 * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37888 Also add the IPC integration reference files to the fuzzing corpus, this may help find more issues. Closesapache#11059 from pitrou/ARROW-13846-ipc-fuzz-crashes Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

Add validation to detect invalid DELTA_BINARY_PACKED data. This should fix the following issues: * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37431 * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37432 * https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=37421Closesapache#11060 from pitrou/ARROW-13850-parquet-fuzz Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

Closesapache#10730 from romainfrancois/ARROW-13164_altrep_with_nulls Lead-authored-by: Romain Francois <romain@rstudio.com> Co-authored-by: Romain François <romain@rstudio.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

Signed-off-by: Junwang Zhao <zhjwpku@gmail.com> Closesapache#11056 from zhjwpku/docs/missing_param_for_setcolumn Authored-by: Junwang Zhao <zhjwpku@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

Closesapache#11055 from kou/ruby-table-save-dataset Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

I templated from ARROW-11735. Let's see how all the tests go! Closesapache#11046 from karldw/arrow-12981 Authored-by: karldw <karldw@users.noreply.github.com> Signed-off-by: Ian Cook <ianmcook@gmail.com>

Closesapache#11061 from lidavidm/arrow-13782 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

…ension types Closesapache#11071 from pitrou/ARROW-13855-export-extension Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: David Li <li.davidm96@gmail.com>

- [x] collect() uses ExecPlan - [x] arrange() uses an OrderBySink - [x] .data inside of arrow_dplyr_query can itself be arrow_dplyr_query - [x] can build more query after calling summarize() - [x] handle non-deterministic dataset collect() tests - [x] fix group_by-expression behavior - [x] make official collapse() method with more testing of faithful behavior after collapsing - [x] make sort after summarize be configurable by option (default FALSE, though local_options TRUE in the tests) - [x] add print method for collapsed query - [x] Skip 32-bit rtools35 dataset tests/examples ~~- [ ] should queries on in-memory data evaluate eagerly (like dplyr)?~~ Followups: * ARROW-13777: [R] mutate after group_by should be ok as long as there are only scalar functions * ARROW-13778: [R] Handle complex summarize expressions * ARROW-13779: [R] Disallow expressions that depend on order after arrange() * ARROW-13852: [R] Handle Dataset schema metadata in ExecPlan * ARROW-13854: [R] More accurately determine output type of an aggregation expression * ARROW-13893: [R] Improve head/tail/[ methods on Dataset and queries Closesapache#10992 from nealrichardson/subquery Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

Closesapache#11074 from thisisnic/ARROW-13874_trimpotions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

…functions Closesapache#11078 from nealrichardson/summarize-0 Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

Closesapache#11083 from kou/ruby-slicer-expression Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Closesapache#11092 from thisisnic/ARROW-13904_modeoptions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

Closesapache#11093 from thisisnic/ARROW-13905_replacesliceoptions Lead-authored-by: Nic Crane <thisisnic@gmail.com> Co-authored-by: Nic <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

Closesapache#11094 from thisisnic/ARROW-13905_partitionnthoptions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

…s kernels Closesapache#11102 from thisisnic/ARROW-13869_misc_compute Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

Closesapache#11098 from thisisnic/ARROW-13908_extract_regex_options Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

… wmalpica/ARROW-12657

wmalpica · 2021-09-14T18:55:52Z

Something strange happened when I rebased. I will close this PR and create a new one.

This PR adds the ability for the StringConverter to parse hex strings of the form: 0x123abc to an integer. The strings must start with "0x" case insensitive. The values ABCDEF are case insensitive. Note that there was already a Hex Parsing function here: https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/string.cc#L76 Which was not really flexible enough for what was needed. Not sure it it would be best to replace it with the new functionality added by this PR. I am open to suggestions This work was originally done here #11064 but had to create a new PR due to a rebase issue Closes#11161 from wmalpica/ARROW-12657_3 Lead-authored-by: William Malpica <16705032+wmalpica@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

This PR adds the ability for the StringConverter to parse hex strings of the form: 0x123abc to an integer. The strings must start with "0x" case insensitive. The values ABCDEF are case insensitive. Note that there was already a Hex Parsing function here: https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/string.cc#L76 Which was not really flexible enough for what was needed. Not sure it it would be best to replace it with the new functionality added by this PR. I am open to suggestions This work was originally done here apache#11064 but had to create a new PR due to a rebase issue Closesapache#11161 from wmalpica/ARROW-12657_3 Lead-authored-by: William Malpica <16705032+wmalpica@users.noreply.github.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

wmalpica added 2 commits September 1, 2021 08:48

working implementation but lacks case-insensitivity and more unit tests
59dcbde

different algorithm. Added more tests and benchmarks
4cb862b

github-actionsBot added the Component: C++ label Sep 2, 2021

uncommented tests
d03b7eb

edponce suggested changes Sep 2, 2021
View reviewed changes

liyafan82and others added 19 commits September 1, 2021 21:24

ARROW-13067: [C++][Compute] Implement integer to decimal cast
bbecb6a
Closesapache#11045 from cyb70289/13067-int2dec Authored-by: Yibo Cai <yibo.cai@arm.com> Signed-off-by: Yibo Cai <yibo.cai@arm.com>

ARROW-13459: [C++][Docs]Missing param docs for RecordBatch::SetColumn
8c70a5f
Signed-off-by: Junwang Zhao <zhjwpku@gmail.com> Closesapache#11056 from zhjwpku/docs/missing_param_for_setcolumn Authored-by: Junwang Zhao <zhjwpku@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

ARROW-13831: [GLib][Ruby] Add support for writing by Arrow Dataset
a1d207e
Closesapache#11055 from kou/ruby-table-save-dataset Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

ARROW-13768: [R] Allow JSON to be an optional component
1440d5a
I templated from ARROW-11735. Let's see how all the tests go! Closesapache#11046 from karldw/arrow-12981 Authored-by: karldw <karldw@users.noreply.github.com> Signed-off-by: Ian Cook <ianmcook@gmail.com>

ARROW-13782: [C++] Add skip_nulls/min_count to tdigest/mode/quantile
a45fc3f
Closesapache#11061 from lidavidm/arrow-13782 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

ARROW-13855: [C++][Python] Implement C data interface support for ext…
5ead375
…ension types Closesapache#11071 from pitrou/ARROW-13855-export-extension Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: David Li <li.davidm96@gmail.com>

ARROW-13874: [R] Implement TrimOptions
858ac57
Closesapache#11074 from thisisnic/ARROW-13874_trimpotions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

ARROW-13543: [R] Handle summarize() with 0 arguments or no aggregate …
a49048b
…functions Closesapache#11078 from nealrichardson/summarize-0 Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

ARROW-13899: [Ruby] Implement slicer by compute kernels
f12c18e
Closesapache#11083 from kou/ruby-slicer-expression Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

MINOR: [Doc][Python] Fix a typo (apache#11085)
882e8b4

thisisnicand others added 14 commits September 13, 2021 22:39

ARROW-13904: [R] Implement ModeOptions
1cbc4a2
Closesapache#11092 from thisisnic/ARROW-13904_modeoptions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

ARROW-13905: [R] Implement ReplaceSliceOptions
f3d3c68
Closesapache#11093 from thisisnic/ARROW-13905_replacesliceoptions Lead-authored-by: Nic Crane <thisisnic@gmail.com> Co-authored-by: Nic <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

ARROW-13906: [R] Implement PartitionNthOptions
0b6f531
Closesapache#11094 from thisisnic/ARROW-13905_partitionnthoptions Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

ARROW-13869: [R] Implement options for non-bound MatchSubstringOption…
672149b
…s kernels Closesapache#11102 from thisisnic/ARROW-13869_misc_compute Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

ARROW-13908: [R] Implement ExtractRegexOptions
8875d5c
Closesapache#11098 from thisisnic/ARROW-13908_extract_regex_options Authored-by: Nic Crane <thisisnic@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

working implementation but lacks case-insensitivity and more unit tests
f1d6811

different algorithm. Added more tests and benchmarks
925b2a7

Implemented review feedback and added more unit tests
68ec4db

checked for empty hex falues. added scalar tests
a538072

fixed style with clang-format
400d886

implemented some improvements
9cac060

fixed clang format
6053936

fixed unit test
68a6844

Merge branch 'wmalpica/ARROW-12657' of github.com:wmalpica/arrow into…
7aee4f0
… wmalpica/ARROW-12657

github-actionsBot added Component: GLib Component: Go Component: Java Component: Python Component: R Component: Ruby Component: Parquet labels Sep 14, 2021

wmalpica closed this Sep 14, 2021

wmalpica mentioned this pull request Sep 15, 2021
ARROW-12657: [C++] Adding String hex to numeric conversion #11160
Closed

wmalpica mentioned this pull request Sep 15, 2021
ARROW-12657: [C++] Adding String hex to numeric conversion #11161
Closed

pitrou pushed a commit to wmalpica/arrow that referenced this pull request Sep 16, 2021
Re-implementing ARROW-12657 which was originally done in PRapache#11064
597cc9f

asfimport mentioned this pull request Sep 16, 2021
[C++][Python][Compute] String hex to numeric conversion and bit shifting #18653
Closed



		inline bool ParseHex(const char* s, size_t length, uint8_t* out) {
		uint8_t result = 0;

Uh oh!

Conversation

wmalpica commented Sep 2, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 2, 2021

Uh oh!

github-actionsBot commented Sep 2, 2021

Uh oh!

edponce commented Sep 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edponceSep 2, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edponceSep 2, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wmalpica commented Sep 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

wmalpica commented Sep 2, 2021•
edited
Loading

edponceSep 2, 2021•
edited
Loading

edponceSep 2, 2021•
edited
Loading