ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
trxcllnt wants to merge 3 commits into from
Closed

ARROW-5762: [JS] Align Map type impl with the spec #5371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58f244a
reimplement Map type
trxcllnt Sep 11, 2019
da2a1ed
Merge branch 'master' into ARROW-5762/map-type
trxcllnt Sep 13, 2019
3a0eca9
ensure generated strings are unique
trxcllnt Sep 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion integration/integration_test.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

batch_sizes = [7, 10]
skip = set()
skip.add('JS') # TODO(ARROW-1279)
skip.add('Go') # TODO(ARROW-3679)
return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down
58 changes: 25 additions & 33 deletions js/DEVELOP.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
```sh
cd $ARROW_HOME

flatc --ts -o ./js/src/format ./format/*.fbs

cd ./js/src/format

# Delete Tensor_generated.js (skip this when we support Tensors)
rm ./Tensor_generated.ts

# Remove "_generated" suffix from TS files
mv ./File_generated.ts .File.ts
mv ./Schema_generated.ts .Schema.ts
mv ./Message_generated.ts .Message.ts
```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
```ts
import { flatbuffers } from "./flatbuffers" // <-- change
import { flatbuffers } from "flatbuffers" // <-- to this
```
1. Remove `_generated` from the ES6 imports of the generated files
```ts
import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
import * as NS16187549871986683199 from "./Schema"; // <------- to this
```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down
1 change: 0 additions & 1 deletion js/src/Arrow.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
Row,
Vector,
BaseVector,
BinaryVector,
Expand Down
2 changes: 1 addition & 1 deletion js/src/bin/arrow2csv.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
if (rowId++ % 350 === 0) {
this.push(`${formatRow(header, maxColWidths, sep)}\n`);
}
this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
}
}
cb();
Expand Down
4 changes: 2 additions & 2 deletions js/src/builder.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
DataType, strideForType,
Float, Int, Decimal, FixedSizeBinary,
Date_, Time, Timestamp, Interval,
Utf8, Binary, List,
Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
protected _pendingLength: number = 0;
protected _offsets: OffsetsBufferBuilder;
protected _pending: Map<number, any> | undefined;
Expand Down
50 changes: 42 additions & 8 deletions js/src/builder/map.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
public addChild(child: Builder, name = `${this.numChildren}`) {
const { children, keysSorted } = this.type;
const childIndex = this.children.push(child);
this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

protected _pending: MapValues<K, V> | undefined;
public set(index: number, value: MapValueExt<K, V> | TNull) {
return super.set(index, value as MapValue<K, V> | TNull);
}

public setValue(index: number, value: MapValueExt<K, V>) {
value = value instanceof Map ? value : new Map(Object.entries(value));
const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
const current = pending.get(index);
current && (this._pendingLength -= current.size);
this._pendingLength += value.size;
pending.set(index, value);
}

public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
if (this.numChildren > 0) {
throw new Error('ListBuilder can only have one child.');
}
this.children[this.numChildren] = child;
this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
return this.numChildren - 1;
}

protected _flushPending(pending: MapValues<K, V>) {
const offsets = this._offsets;
const setValue = this._setValue;
pending.forEach((value, index) => {
if (value === undefined) {
offsets.set(index, 0);
} else {
offsets.set(index, value.size);
setValue(this, index, value);
}
});
}
}
8 changes: 4 additions & 4 deletions js/src/data.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
}
throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
}
/** @nocollapse */
public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
}
/** @nocollapse */
public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
}
/** @nocollapse */
public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
}
public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down
17 changes: 0 additions & 17 deletions js/src/fb/File.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @param flatbuffers.ByteBuffer bb
* @param Footer= obj
* @returns Footer
*/
static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @returns org.apache.arrow.flatbuf.MetadataVersion
*/
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
builder.finish(offset);
}

/**
* @param flatbuffers.Builder builder
* @param flatbuffers.Offset offset
*/
static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
builder.finish(offset, undefined);
}

static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
Footer.startFooter(builder);
Footer.addVersion(builder, version);
Expand Down
35 changes: 0 additions & 35 deletions js/src/fb/Message.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @param flatbuffers.ByteBuffer bb
* @param RecordBatch= obj
* @returns RecordBatch
*/
static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* number of records / rows. The arrays in the batch should all have this
* length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @param flatbuffers.ByteBuffer bb
* @param DictionaryBatch= obj
* @returns DictionaryBatch
*/
static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @returns flatbuffers.Long
*/
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @param flatbuffers.ByteBuffer bb
* @param Message= obj
* @returns Message
*/
static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
}

/**
* @returns org.apache.arrow.flatbuf.MetadataVersion
*/
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
builder.finish(offset);
}

/**
* @param flatbuffers.Builder builder
* @param flatbuffers.Offset offset
*/
static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
builder.finish(offset, undefined);
}

static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
Message.startMessage(builder);
Message.addVersion(builder, version);
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-5762: [JS] Align Map type impl with the spec#5371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

trxcllnt
wants to merge
3
commits into

from

 Closed


ARROW-5762: [JS] Align Map type impl with the spec
#5371
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
58f244a

reimplement Map type


trxcllnt Sep 11, 2019
da2a1ed

Merge branch 'master' into ARROW-5762/map-type


trxcllnt Sep 13, 2019
3a0eca9

ensure generated strings are unique


trxcllnt Sep 13, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 0 additions & 1 deletion
integration/integration_test.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1073,7 +1073,6 @@ def generate_map_case():

 batch_sizes = [7, 10]
 skip = set()
 skip.add('JS') # TODO(ARROW-1279)
 skip.add('Go') # TODO(ARROW-3679)
 return _generate_file("map", fields, batch_sizes, skip=skip)

Expand Down

58 changes: 25 additions & 33 deletions
js/DEVELOP.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -39,8 +39,6 @@ If you’d like to report a bug but don’t have time to fix it, you can still p
it on JIRA, or email the mailing list
[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/)



# The npm scripts

* `npm run clean` - cleans targets
Expand DownExpand Up@@ -68,37 +66,31 @@ Uses [lerna](https://github.com/lerna/lerna) to publish each build target to npm

# Updating the Arrow format flatbuffers generated code

Once generated, the flatbuffers format code needs to be adjusted for our build scripts.

1. Generate the flatbuffers TypeScript source from the Arrow project root directory:
 ```sh
 cd $ARROW_HOME

 flatc --ts -o ./js/src/format ./format/*.fbs

 cd ./js/src/format

 # Delete Tensor_generated.js (skip this when we support Tensors)
 rm ./Tensor_generated.ts

 # Remove "_generated" suffix from TS files
 mv ./File_generated.ts .File.ts
 mv ./Schema_generated.ts .Schema.ts
 mv ./Message_generated.ts .Message.ts
 ```
1. Remove Tensor import from `Schema.ts`
1. Fix all the `flatbuffers` imports
 ```ts
 import { flatbuffers } from "./flatbuffers" // <-- change
 import { flatbuffers } from "flatbuffers" // <-- to this
 ```
1. Remove `_generated` from the ES6 imports of the generated files
 ```ts
 import * as NS16187549871986683199 from "./Schema_generated"; // <-- change
 import * as NS16187549871986683199 from "./Schema"; // <------- to this
 ```
1. Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
1. Execute `npm run lint` to fix all the linting errors
1. Once generated, the flatbuffers format code needs to be adjusted for our build scripts (assumes `gnu-sed`):

```shell
cd $ARROW_HOME

flatc --ts -o ./js/src/fb ./format/{File,Schema,Message}.fbs

cd ./js/src/fb

# Rename the existing files to <filename>.bak.ts
mv File{,.bak}.ts && mv Schema{,.bak}.ts && mv Message{,.bak}.ts

# Remove `_generated` from the ES6 imports of the generated files
sed -i '+s+_generated\";+\";+ig' *_generated.ts
# Fix all the `flatbuffers` imports
sed -i '+s+./flatbuffers+flatbuffers+ig' *_generated.ts
# Fix the Union createTypeIdsVector typings
sed -i -r '+s+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Uint8Array+static createTypeIdsVector\(builder: flatbuffers.Builder, data: number\[\] \| Int32Array+ig' Schema_generated.ts
# Add `/* tslint:disable:class-name */` to the top of `Schema.ts`
echo -e '/* tslint:disable:class-name */\n' | cat - Schema_generated.ts > Schema1.ts && mv Schema1.ts Schema_generated.ts
# Remove "_generated" suffix from TS files
mv File{_generated,}.ts && mv Schema{_generated,}.ts && mv Message{_generated,}.ts
```
2. Manually remove `Tensor` and `SparseTensor` imports and exports
3. Execute `npm run lint` from the `js` directory to fix the linting errors

[1]: mailto:dev-subscribe@arrow.apache.org
[2]: https://github.com/apache/arrow/tree/master/format
Expand Down

1 change: 0 additions & 1 deletion
js/src/Arrow.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,6 @@ export { Column } from './column';
export { Visitor } from './visitor';
export { Schema, Field } from './schema';
export {
 Row,
 Vector,
 BaseVector,
 BinaryVector,
Expand Down

2 changes: 1 addition & 1 deletion
js/src/bin/arrow2csv.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -177,7 +177,7 @@ function batchesToString(state: ToStringState, schema: Schema) {
 if (rowId++ % 350 === 0) {
 this.push(`${formatRow(header, maxColWidths, sep)}\n`);
 }
 this.push(`${formatRow([rowId, ...row].map(valueToString), maxColWidths, sep)}\n`);
 this.push(`${formatRow([rowId, ...row.toArray()].map(valueToString), maxColWidths, sep)}\n`);
 }
 }
 cb();
Expand Down

4 changes: 2 additions & 2 deletions
js/src/builder.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -25,7 +25,7 @@ import {
 DataType, strideForType,
 Float, Int, Decimal, FixedSizeBinary,
 Date_, Time, Timestamp, Interval,
 Utf8, Binary, List,
 Utf8, Binary, List, Map_
} from './type';

/**
Expand DownExpand Up@@ -441,7 +441,7 @@ export abstract class FixedWidthBuilder<T extends Int | Float | FixedSizeBinary
}

/** @ignore */
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List, TNull = any> extends Builder<T, TNull> {
export abstract class VariableWidthBuilder<T extends Binary | Utf8 | List | Map_, TNull = any> extends Builder<T, TNull> {
 protected _pendingLength: number = 0;
 protected _offsets: OffsetsBufferBuilder;
 protected _pending: Map<number, any> | undefined;
Expand Down

50 changes: 42 additions & 8 deletions
js/src/builder/map.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,15 +16,49 @@
// under the License.

import { Field } from '../schema';
import { Builder } from '../builder';
import { DataType, Map_ } from '../type';
import { DataType, Map_, Struct } from '../type';
import { Builder, VariableWidthBuilder } from '../builder';

/** @ignore */ type MapValue<K extends DataType = any, V extends DataType = any> = Map_<K, V>['TValue'];
/** @ignore */ type MapValues<K extends DataType = any, V extends DataType = any> = Map<number, MapValue<K, V> | undefined>;
/** @ignore */ type MapValueExt<K extends DataType = any, V extends DataType = any> = MapValue<K, V> | { [key: string]: V } | { [key: number]: V } ;

/** @ignore */
export class MapBuilder<T extends { [key: string]: DataType } = any, TNull = any> extends Builder<Map_<T>, TNull> {
 public addChild(child: Builder, name = `${this.numChildren}`) {
 const { children, keysSorted } = this.type;
 const childIndex = this.children.push(child);
 this.type = new Map_([...children, new Field(name, child.type, true)], keysSorted);
 return childIndex;
export class MapBuilder<K extends DataType = any, V extends DataType = any, TNull = any> extends VariableWidthBuilder<Map_<K, V>, TNull> {

 protected _pending: MapValues<K, V> | undefined;
 public set(index: number, value: MapValueExt<K, V> | TNull) {
 return super.set(index, value as MapValue<K, V> | TNull);
 }

 public setValue(index: number, value: MapValueExt<K, V>) {
 value = value instanceof Map ? value : new Map(Object.entries(value));
 const pending = this._pending || (this._pending = new Map() as MapValues<K, V>);
 const current = pending.get(index);
 current && (this._pendingLength -= current.size);
 this._pendingLength += value.size;
 pending.set(index, value);
 }

 public addChild(child: Builder<Struct<{ key: K, value: V }>>, name = `${this.numChildren}`) {
 if (this.numChildren > 0) {
 throw new Error('ListBuilder can only have one child.');
 }
 this.children[this.numChildren] = child;
 this.type = new Map_<K, V>(new Field(name, child.type, true), this.type.keysSorted);
 return this.numChildren - 1;
 }

 protected _flushPending(pending: MapValues<K, V>) {
 const offsets = this._offsets;
 const setValue = this._setValue;
 pending.forEach((value, index) => {
 if (value === undefined) {
 offsets.set(index, 0);
 } else {
 offsets.set(index, value.size);
 setValue(this, index, value);
 }
 });
 }
}

8 changes: 4 additions & 4 deletions
js/src/data.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -199,7 +199,7 @@ export class Data<T extends DataType = DataType> {
 case Type.List: return <unknown> Data.List( <unknown> type as List, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.FixedSizeList: return <unknown> Data.FixedSizeList( <unknown> type as FixedSizeList, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], (childData || [])[0]) as Data<T>;
 case Type.Struct: return <unknown> Data.Struct( <unknown> type as Struct, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], childData || []) as Data<T>;
 case Type.Map: return <unknown> Data.Map( <unknown> type as Map_, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.OFFSET] || [], (childData || [])[0]) as Data<T>;
 case Type.Union: return <unknown> Data.Union( <unknown> type as Union, offset, length, nullCount || 0, buffers[BufferType.VALIDITY], buffers[BufferType.TYPE] || [], buffers[BufferType.OFFSET] || childData, childData) as Data<T>;
 }
 throw new Error(`Unrecognized typeId ${type.typeId}`);
Expand DownExpand Up@@ -262,16 +262,16 @@ export class Data<T extends DataType = DataType> {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data | Vector) {
 public static FixedSizeList<T extends FixedSizeList>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], [child]);
 }
 /** @nocollapse */
 public static Struct<T extends Struct>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 }
 /** @nocollapse */
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, children: (Data | Vector)[]) {
 return new Data(type, offset, length, nullCount, [undefined, undefined, toUint8Array(nullBitmap)], children);
 public static Map<T extends Map_>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: (Data | Vector)) {
 return new Data(type, offset, length, nullCount, [toInt32Array(valueOffsets), undefined, toUint8Array(nullBitmap)], [child]);
 }
 public static Union<T extends SparseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[], _?: any): Data<T>;
 public static Union<T extends DenseUnion>(type: T, offset: number, length: number, nullCount: number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]): Data<T>;
Expand Down

17 changes: 0 additions & 17 deletions
js/src/fb/File.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,15 +34,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Footer= obj
 * @returns Footer
 */
 static getSizePrefixedRootAsFooter(bb: flatbuffers.ByteBuffer, obj?: Footer): Footer {
 return (obj || new Footer).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -168,14 +159,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedFooterBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createFooter(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, schemaOffset: flatbuffers.Offset, dictionariesOffset: flatbuffers.Offset, recordBatchesOffset: flatbuffers.Offset): flatbuffers.Offset {
 Footer.startFooter(builder);
 Footer.addVersion(builder, version);
Expand Down

35 changes: 0 additions & 35 deletions
js/src/fb/Message.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -125,15 +125,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param RecordBatch= obj
 * @returns RecordBatch
 */
 static getSizePrefixedRootAsRecordBatch(bb: flatbuffers.ByteBuffer, obj?: RecordBatch): RecordBatch {
 return (obj || new RecordBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * number of records / rows. The arrays in the batch should all have this
 * length
Expand DownExpand Up@@ -290,15 +281,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param DictionaryBatch= obj
 * @returns DictionaryBatch
 */
 static getSizePrefixedRootAsDictionaryBatch(bb: flatbuffers.ByteBuffer, obj?: DictionaryBatch): DictionaryBatch {
 return (obj || new DictionaryBatch).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns flatbuffers.Long
 */
Expand DownExpand Up@@ -404,15 +386,6 @@ export namespace org.apache.arrow.flatbuf {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @param flatbuffers.ByteBuffer bb
 * @param Message= obj
 * @returns Message
 */
 static getSizePrefixedRootAsMessage(bb: flatbuffers.ByteBuffer, obj?: Message): Message {
 return (obj || new Message).__init(bb.readInt32(bb.position()) + bb.position(), bb);
 }

 /**
 * @returns org.apache.arrow.flatbuf.MetadataVersion
 */
Expand DownExpand Up@@ -549,14 +522,6 @@ export namespace org.apache.arrow.flatbuf {
 builder.finish(offset);
 }

 /**
 * @param flatbuffers.Builder builder
 * @param flatbuffers.Offset offset
 */
 static finishSizePrefixedMessageBuffer(builder: flatbuffers.Builder, offset: flatbuffers.Offset) {
 builder.finish(offset, undefined);
 }

 static createMessage(builder: flatbuffers.Builder, version: NS7624605610262437867.org.apache.arrow.flatbuf.MetadataVersion, headerType: org.apache.arrow.flatbuf.MessageHeader, headerOffset: flatbuffers.Offset, bodyLength: flatbuffers.Long, customMetadataOffset: flatbuffers.Offset): flatbuffers.Offset {
 Message.startMessage(builder);
 Message.addVersion(builder, version);
Expand Down
Loading