ARROW-972: UnionArray in pyarrow - #1216

Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom
pcmoritz:pyarrow-union-array
Closed

ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom
pcmoritz:pyarrow-union-array

Conversation

@pcmoritz

@pcmoritzpcmoritz commented Oct 20, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:

  • Support UnionType::SPARSE
  • Add doc strings

@wesm

wesm commented Oct 23, 2017

Copy link
Copy Markdown
Member

Can you rebase? What more work needs to be done here?

@wesm

wesm commented Nov 4, 2017

Copy link
Copy Markdown
Member

@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0

@pcmoritz

pcmoritz commented Nov 4, 2017

Copy link
Copy Markdown
ContributorAuthor

@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.

@pcmoritz
pcmoritzforce-pushed the pyarrow-union-array branch 5 times, most recently from df611cb to fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz changed the title [WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz

pcmoritz commented Nov 5, 2017

Copy link
Copy Markdown
ContributorAuthor

This is now ready to review

wesm
wesm reviewed Nov 7, 2017
View reviewed changes

@wesmwesm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!

Comment threadcpp/src/arrow/array.cc Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)

Comment threadcpp/src/arrow/array.cc Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes

Comment threadcpp/src/arrow/array.cc Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count

@pcmoritzpcmoritzNov 8, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadcpp/src/arrow/array.h Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer

@pcmoritzpcmoritzNov 8, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadcpp/src/arrow/array.h Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?

@pcmoritzpcmoritzNov 8, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.

Comment threadpython/pyarrow/scalar.pxi Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)

Comment threadpython/pyarrow/scalar.pxi Outdated

@wesmwesmNov 7, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?

@pcmoritzpcmoritzNov 8, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.

pcmoritz
pcmoritz commented Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:

[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed the pyarrow-union-array branch 2 times, most recently from 2007736 to 02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz

pcmoritz commented Nov 9, 2017

Copy link
Copy Markdown
ContributorAuthor

@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.

@wesm

wesm commented Nov 9, 2017

Copy link
Copy Markdown
Member

enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly

@wesm
wesmforce-pushed the pyarrow-union-array branch from 02bdfc8 to 7f3ca31CompareNovember 9, 2017 05:17
@wesm

wesm commented Nov 9, 2017

Copy link
Copy Markdown
Member

Done. Will merge this once the build is green

@pcmoritz

pcmoritz commented Nov 9, 2017

Copy link
Copy Markdown
ContributorAuthor

Great, thanks :)

@wesmwesm closed this in 65a9055Nov 9, 2017
@wesm
wesm deleted the pyarrow-union-array branch November 9, 2017 16:32
@wesm

wesm commented Nov 9, 2017

Copy link
Copy Markdown
Member

Thanks @pcmoritz!

@asfimportasfimport mentioned this pull request Jul 12, 2020
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-972: UnionArray in pyarrow - #1216
Closed
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Closed
ARROW-972: UnionArray in pyarrow#1216
pcmoritz wants to merge 10 commits into
apache:masterfrom 
pcmoritz:pyarrow-union-array
Conversation
@pcmoritz
@pcmoritzpcmoritz
commented
Oct 20, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
This is taking a stab at exposing UnionArray to pyarrow. Tasks to be done:
  •  Support UnionType::SPARSE
  •  Add doc strings
@wesm
wesm
commented
Oct 23, 2017

Copy link


Copy Markdown

Member
Can you rebase? What more work needs to be done here?
@wesm
wesm
commented
Nov 4, 2017

Copy link


Copy Markdown

Member
@pcmoritz anything I can do to help on this? Would be great to get this into 0.8.0
@pcmoritz
pcmoritz
commented
Nov 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm: Agreed it should be part of 0.8.0. I'll take a stab at the remaining items now and let you know how things go.
@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
5 times, most recently
from
df611cb to
fc73439CompareNovember 5, 2017 18:31
@pcmoritzpcmoritz
changed the title
[WIP] ARROW-972: UnionArray in pyarrowARROW-972: UnionArray in pyarrowNov 5, 2017
@pcmoritz
pcmoritz
commented
Nov 5, 2017

Copy link


Copy Markdown

ContributorAuthor
This is now ready to review
wesm
wesm
reviewed
Nov 7, 2017
View reviewed changes
@wesmwesm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some minor comments about names and a few other things. Thank you for doing this!
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be better with the type on the LHS (std::vector<std::shared_ptr<Field>> types;)
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be useful as an alternate version of arrow::union_? Maybe with the same API (the mode as the last argument) but omitting the type codes
Comment threadcpp/src/arrow/array.cc
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also want to assert that value_offsets has 0 null count
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this is naming similar to ListArray::FromArrays, but because there are two versions, I wonder if calling this UnionArray::MakeDense (or FromArraysDense, more verbose) would be clearer
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadcpp/src/arrow/array.h
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't initially clear that this is the child type, maybe call this child_type instead (and add doxygen comment)?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now gone and using the Array's type to get this information. It's more consistent with how things are done for ListArray and StructArray.
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true that using an underscore saves you from declaring this method in the pxd file? If so I've been wasting my time a bunch in the past :)
Comment threadpython/pyarrow/scalar.pxi
Outdated

@wesmwesmNov 7, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite expensive on a per-value basis. Could these wrapped types be accessed from the parent pyarrow UnionArray?
@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good point! I'm constructing the Python objects for children types once per array in the UnionType.
pcmoritz
pcmoritz
commented
Nov 8, 2017
View reviewed changes
Comment threadpython/pyarrow/types.pxi
Outdated

@pcmoritzpcmoritzNov 8, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this and plug mode in directly below, clang complains:
[ 33%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100873:100: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0]));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100882:102: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100891:104: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100929:130: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)d...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100938:101: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[0])));
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100947:132: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100956:103: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((enum arrow::UnionMode)digits[2]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[1]) << PyLong_SHIFT) ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100965:134: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) (((enum arrow::UnionMode)-1)*(((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMo...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
/Users/pcmoritz/arrow/python/build/temp.macosx-10.7-x86_64-3.5/lib.cxx:100974:105: error: invalid operands to binary expression ('enum arrow::UnionMode' and 'int')
return (enum arrow::UnionMode) ((((((((((enum arrow::UnionMode)digits[3]) << PyLong_SHIFT) | (enum arrow::UnionMode)digits[2]) << PyLong_SHIFT...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
9 errors generated.

@pcmoritz
pcmoritzforce-pushed
the
pyarrow-union-array
branch
2 times, most recently
from
2007736 to
02bdfc8CompareNovember 8, 2017 23:08
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm This is ready except for the Windows compile error, do you think you could take a quick look at that? I'm not sure what is going on.
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
enum class is a huge pain in Cython. I'll go ahead and change UnionMode to be a scoped enumeration here quickly
@wesm
wesmforce-pushed
the
pyarrow-union-array
branch
from
02bdfc8 to
7f3ca31CompareNovember 9, 2017 05:17
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Done. Will merge this once the build is green
@pcmoritz
pcmoritz
commented
Nov 9, 2017

Copy link


Copy Markdown

ContributorAuthor
Great, thanks :)
@wesmwesm
closed this
in
65a9055Nov 9, 2017
@wesm
wesm
deleted the

pyarrow-union-array

branch
November 9, 2017 16:32
@wesm
wesm
commented
Nov 9, 2017

Copy link


Copy Markdown

Member
Thanks @pcmoritz!
@asfimportasfimport
mentioned this pull request
Jul 12, 2020

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@pcmoritz@wesm