Skip to content

Add missing map functions - #1461

Merged
timsaucer merged 12 commits into
apache:mainfrom
timsaucer:add-map-functions
Apr 4, 2026
Merged

Add missing map functions#1461
timsaucer merged 12 commits into
apache:mainfrom
timsaucer:add-map-functions

Conversation

@timsaucer

@timsaucertimsaucer commented Mar 29, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes#1448

Rationale for this change

These functions were not exposed but are available upstream.

What changes are included in this PR?

Expose functions.
Add python wrappers.
Add unit tests.

Are there any user-facing changes?

New addition only.

Disclaimer

Most of this code was written with an AI agent, but every line has been reviewed by me and I am willing to discuss all design decisions.

@timsaucer
timsaucer marked this pull request as ready for review March 29, 2026 23:03
@timsaucer
timsaucer requested a review from CopilotMarch 30, 2026 16:22
CopilotAI reviewed Mar 30, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Exposes DataFusion’s map-related scalar functions to the Python API (wrapping new Rust bindings) and adds Python-level helpers plus unit tests.

Changes:

  • Add Rust pyo3 bindings for make_map and map accessors (map_keys, map_values, map_extract, map_entries).
  • Add Python wrappers (map, make_map, map_keys, map_values, map_extract, map_entries, element_at) and export them via __all__.
  • Add unit tests covering the new map construction and accessor functions.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

FileDescription
crates/core/src/functions.rsAdds make_map binding and registers map accessor functions in the internal Rust module.
python/datafusion/functions.pyAdds Python-facing wrappers for map functions and updates exported symbol list.
python/tests/test_functions.pyAdds unit tests validating map creation and map accessor behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadpython/tests/test_functions.py Outdated
Comment threadpython/datafusion/functions.py Outdated
Comment threadpython/datafusion/functions.py
timsaucerand others added 7 commits April 3, 2026 10:40
@timsaucer@claude
Add map functions (make_map, map_keys, map_values, map_extract, map_e…
b8a4bac
…ntries, element_at)
Closesapache#1448
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Add unit tests for map functions
e421f7d
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Remove redundant pyo3 element_at function
2cd3d94
element_at is already a Python-only alias for map_extract,
so the Rust binding is unnecessary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Change make_map to accept a Python dictionary
5467eda
make_map now takes a dict for the common case and also supports
separate keys/values lists for column expressions. Non-Expr keys
and values are automatically converted to literals.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Make map the primary function with make_map as alias
1688aef
map() now supports three calling conventions matching upstream:
- map({"a": 1, "b": 2}) — from a Python dictionary
- map([keys], [values]) — two lists that get zipped
- map(k1, v1, k2, v2, ...) — variadic key-value pairs
Non-Expr keys and values are automatically converted to literals.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Improve map function docstrings
fc76525
- Add examples for all three map() calling conventions
- Use clearer descriptions instead of jargon (no "zipped" or "variadic")
- Break map_keys/map_values/map_extract/map_entries examples into
two steps: create the map column first, then call the function
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Remove map() in favor of make_map(), fix docstrings, add validation
dbe4d2e
- Remove map() function that shadowed Python builtin; make_map() is now
the sole entry point for creating map expressions
- Fix map_extract/element_at docstrings: missing keys return [None],
not an empty list (matches actual upstream behavior)
- Add length validation for the two-list calling convention
- Update all tests and docstring examples accordingly
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucer@claude
Consolidate map function tests into parametrized groups
461acca
Reduce boilerplate by combining make_map construction tests and map
accessor function tests into two @pytest.mark.parametrize groups.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@timsaucertimsaucer changed the title Add map functionsAdd missing map functionsApr 3, 2026
@timsaucer
timsaucer requested a review from CopilotApril 3, 2026 16:58
@timsaucer

timsaucer commented Apr 3, 2026

Copy link
Copy Markdown
MemberAuthor

@nuno-faria I think this one is good to go if you have time.

CopilotAI reviewed Apr 3, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadpython/datafusion/functions.py
Comment threadcrates/core/src/functions.rs
Comment threadpython/tests/test_functions.py
nuno-faria
nuno-faria approved these changes Apr 4, 2026
View reviewed changes

@nuno-farianuno-faria left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @timsaucer, I leave some additional suggestions below.

Comment threadpython/datafusion/functions.py Outdated
Comment threadpython/datafusion/functions.py Outdated
Comment threadpython/tests/test_functions.py Outdated
Comment threadpython/tests/test_functions.py Outdated
@nuno-farianuno-faria mentioned this pull request Apr 4, 2026
Merged
timsaucerand others added 4 commits April 4, 2026 11:52
@timsaucer@nuno-faria
Docstring update
d390b9d
Co-authored-by: Nuno Faria <nunofpfaria@gmail.com>
@timsaucer@nuno-faria
Docstring update
63cf6ee
Co-authored-by: Nuno Faria <nunofpfaria@gmail.com>
@timsaucer@nuno-faria
Simplify test for readability
21584a0
Co-authored-by: Nuno Faria <nunofpfaria@gmail.com>
@timsaucer@nuno-faria
Simplify test for readability
79710af
Co-authored-by: Nuno Faria <nunofpfaria@gmail.com>
@timsaucer
timsaucer merged commit 8a35cae into apache:mainApr 4, 2026
21 checks passed
@timsaucer
timsaucer deleted the add-map-functions branch April 4, 2026 16:20
github-merge-queueBot pushed a commit to apache/datafusion that referenced this pull request Apr 16, 2026
@nuno-faria
docs: Updatemap_extractexamples (#21360)
dbe3395
## Which issue does this PR close?
N/A.
## Rationale for this change
While reviewing apache/datafusion-python#1461, I
noticed that an example in the `map_extract` function was wrong:
```sql
-- example
SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'y');
----
[]
-- datafusion
SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'y');
----
[NULL]
```
## What changes are included in this PR?
- Fixed the previous example.
- Also added a new example showing a `map_extract` on an empty key.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
Rich-T-kid pushed a commit to Rich-T-kid/datafusion that referenced this pull request Apr 21, 2026
@nuno-faria@Rich-T-kid
docs: Updatemap_extractexamples (apache#21360)
f253de8
## Which issue does this PR close?
N/A.
## Rationale for this change
While reviewing apache/datafusion-python#1461, I
noticed that an example in the `map_extract` function was wrong:
```sql
-- example
SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'y');
----
[]
-- datafusion
SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'y');
----
[NULL]
```
## What changes are included in this PR?
- Fixed the previous example.
- Also added a new example showing a `map_extract` on an empty key.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@nuno-farianuno-farianuno-faria approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add map functions (map, map_keys, map_values, map_extract, map_entries, element_at)

3 participants

@timsaucer@nuno-faria