GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613

Merged
kevingurney merged 9 commits into
apache:mainfrom
mathworks:GH-37597
Sep 7, 2023
Merged

GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom
mathworks:GH-37597

Conversation

@sgilmore10

@sgilmore10sgilmore10 commented Sep 7, 2023•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Member

Rationale for this change

Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.

It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.

What changes are included in this PR?

  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.

Are these changes tested?

Yes. Added unit tests to tChunkedArray.m.

Are there any user-facing changes?

Yes. Users can now call toMATLAB on ChunkedArrays.

Example

>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11

kevingurney
kevingurney requested changes Sep 7, 2023
View reviewed changes

@kevingurneykevingurney left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!

Comment threadmatlab/test/arrow/array/tChunkedArray.m Outdated
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Sep 7, 2023
@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Sep 7, 2023
@kevingurney

kevingurney commented Sep 7, 2023

Copy link
Copy Markdown
Member

+1

@kevingurney
kevingurney merged commit 65e2f22 into apache:mainSep 7, 2023
@kevingurney
kevingurney deleted the GH-37597 branch September 7, 2023 17:46
@kevingurneykevingurney removed the awaiting merge Awaiting merge label Sep 7, 2023
@conbench-apache-arrow

conbench-apache-arrowBot commented Sep 8, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees

No one assigned

Labels

Component: MATLAB

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class

2 participants

@sgilmore10@kevingurney
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class - #37613
Merged
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Sep 7, 2023
Merged
GH-37597: [MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class#37613
kevingurney merged 9 commits into
apache:mainfrom 
mathworks:GH-37597
Conversation
@sgilmore10
@sgilmore10sgilmore10
commented
Sep 7, 2023•

edited by github-actions
Bot
Loading

Copy link


Copy Markdown

Member
Rationale for this change
Currently, there is no way to easily convert an arrow.array.ChunkedArray into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling toMATLAB on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to arrow.array.ChunkedArray that abstracts away all of these steps.
What changes are included in this PR?
  1. Added toMATLAB method to arrow.array.ChunkedArray class
  2. Added preallocateMATLABArray abstract method to arrow.type.Type class. This method is used by the ChunkedArraytoMATLAB to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure toMATLAB returns the correct MATLAB array when the ChunkedArray has zero chunks. If toMATLAB stored the result of calling toMATLAB on each chunk in a cell array before concatenating the values, toMATLAB would return a 0x0 double array for zero-chunked arrays. The pre-allocation approach avoids this issue.
  3. Implement preallocateMATLABArray on all arrow.type.Type classes.
  4. Added an abstract class arrow.type.NumericType that all classes representing numeric data types inherit from. NumericType implements preallocateMATLABArray for its subclasses.
Are these changes tested?
Yes. Added unit tests to tChunkedArray.m.
Are there any user-facing changes?
Yes. Users can now call toMATLAB on ChunkedArrays.
Example
>> a=arrow.array([12NaN45]);
>> b=arrow.array([6789NaN11]);
>> c=arrow.array.ChunkedArray.fromArrays(a, b);
>> data= toMATLAB(c)
data =12NaN456789NaN11
kevingurney
kevingurney
requested changes
Sep 7, 2023
View reviewed changes
@kevingurneykevingurney
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you!
Comment threadmatlab/test/arrow/array/tChunkedArray.m
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
and removed

awaiting review
Awaiting review
awaiting changes
Awaiting changes
labels
Sep 7, 2023
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting change review
Awaiting change review
labels
Sep 7, 2023
@kevingurney
kevingurney
commented
Sep 7, 2023

Copy link


Copy Markdown

Member
+1
@kevingurney
kevingurney
merged commit 65e2f22
into
apache:mainSep 7, 2023
@kevingurney
kevingurney
deleted the

GH-37597

branch
September 7, 2023 17:46
@kevingurneykevingurney
removed
the 
awaiting merge
Awaiting merge
label
Sep 7, 2023
@conbench-apache-arrow
conbench-apache-arrowBot
commented
Sep 8, 2023

Copy link


Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 65e2f22.
There were no benchmark performance regressions. 🎉
The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
loicalleyne pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
@sgilmore10@loicalleyne
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
6268d77
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
dgreiss pushed a commit
to dgreiss/arrow
that referenced
this pull request
Feb 19, 2024
@sgilmore10@dgreiss
apacheGH-37597: [MATLAB] AddtoMATLABmethod to `arrow.array.Chunke…
d2a56a8
…dArray` class (apache#37613)
### Rationale for this change
Currently, there is no way to easily convert an `arrow.array.ChunkedArray` into a corresponding MATLAB array, other than (1) manually iterating chunk by chunk, (2) calling `toMATLAB` on each chunk, and then (3) concatenating all of the converted chunks together into one contiguous MATLAB array.
It would be helpful to add a toMATLAB method to `arrow.array.ChunkedArray` that abstracts away all of these steps.
### What changes are included in this PR?
1. Added `toMATLAB` method to `arrow.array.ChunkedArray` class
2. Added `preallocateMATLABArray` abstract method to `arrow.type.Type` class. This method is used by the `ChunkedArray` `toMATLAB` to pre-allocate a MATLAB array of the expected class type and shape. This is necessary to ensure `toMATLAB` returns the correct MATLAB array when the `ChunkedArray` has zero chunks. If `toMATLAB` stored the result of calling `toMATLAB` on each chunk in a `cell` array before concatenating the values, `toMATLAB` would return a 0x0 `double` array for zero-chunked arrays. The pre-allocation approach avoids this issue.
3. Implement `preallocateMATLABArray` on all `arrow.type.Type` classes.
4. Added an abstract class `arrow.type.NumericType` that all classes representing numeric data types inherit from. `NumericType` implements `preallocateMATLABArray` for its subclasses.
### Are these changes tested?
Yes. Added unit tests to `tChunkedArray.m`.
### Are there any user-facing changes?
Yes. Users can now call `toMATLAB` on `ChunkedArray`s.
**Example**
```matlab
>> a = arrow.array([1 2 NaN 4 5]);
>> b = arrow.array([6 7 8 9 NaN 11]);
>> c = arrow.array.ChunkedArray.fromArrays(a, b);
>> data = toMATLAB(c)
data =
1
2
NaN
4
5
6
7
8
9
NaN
11
```
* Closes: apache#37597
Authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Signed-off-by: Kevin Gurney <kgurney@mathworks.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kevingurneykevingurneykevingurney approved these changes
@koukouAwaiting requested review from koukou is a code owner

Assignees


No one assigned


Labels

Component: MATLAB

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[MATLAB] Add toMATLAB method to arrow.array.ChunkedArray class


2 participants

@sgilmore10@kevingurney