ARROW-14705: [C++][Python] Handle more types in UnifySchemas

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.

[lidavidm]

This expands UnifySchemas to handle more types, e.g. int32 + float64 = float64, by adding flags for various types.

The flags here may be overly fine-grained.

Not implemented here:

• Confirming that all these various casts exist
• Adding the bindings to the Python datasets module (would need PR ARROW-8221: [Python][Dataset] Expose schema inference/validation factory options through the validate_schema keyword #8912 first)
• Binding individual flags in Python instead of all-or-nothing
• Adding the bindings to the R datasets bindings

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14705

[jorisvandenbossche]

Just some high level questions from a quick look. But this will be a nice improvement to have more flexible schema unification!
(I personally don't have a strong opinion on the need of exposing the fine-grained options or not. It seems fine to me to keep this)

On the python side, I think this can be exposed in concat_tables, which uses ConcatenateTables under the hood which already supports passing those field_merge_options.

[jorisvandenbossche]

Just to be clear, that would mean that int64 cannot be unified with float32? (since that bitwidth is not "equal or greater") Or that it would be unified to float64?

For reference, numpy promotes that specific case to float64.

[lidavidm]

As implemented, you need also promote_numeric_width to unify int64 and float32 (to float64).

[lidavidm]

Oh wait, I'm wrong (awkward). Yes, int64 + float32 = float64.

[jorisvandenbossche]

.. or greater bit width?

Eg in numpy uint32 + int64 -> int64

[jorisvandenbossche]

For reading clarity, can we list here the ones that are still left to False in a comment? (eg promote_dictionary_ordered is still False, which I think is good)

promote_decimal should probably be added as true?

[lidavidm]

Whoops, I didn't update this after adding some of the additional options. I'll document what's not enabled as you suggest, thanks.

[lidavidm]

Corrected the doc comments.

For concat_tables, this requires us to have access to Cast() inside table.cc, so we need ARROW-8891 or some #ifdef'ing to implement this.

[lidavidm]

I've used some #ifdefs so that ConcatenateTables can cast columns as necessary now. I still have some TODOs listed up top before I can undraft this.

[pitrou]

@lidavidm Do we still want to do this? If so, do you want to push it forward?

[lidavidm]

I think it's still useful to have, but I don't think I'll have time to push this forward anytime soon.