A result as numpy arrays

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.

[tamnd]

result.fetchnumpy(), which is a dict of numpy arrays keyed by column name, and it needs numpy and nothing else.

It exists because numpy is what a lot of code already holds: a model that takes arrays, a plotting call, a loop somebody wrote before DataFrames. Handing that code an Arrow table means it has to convert one, and converting one means pyarrow has to be installed to do work numpy could have done with the same bytes.

The same bytes is what this is. zudb::query::column already hands back one owned buffer per column, values end to end in the layout every columnar format uses, and a numpy array is a pointer, a length and a dtype over exactly that. So an integer, float, datetime or duration column becomes an array by moving the Vec into numpy and naming the type: no pass over the values, no allocation, no second copy of the result. There is a test for the claim, since it is the only one worth making: owndata is false and there is a base object, which is numpy's own way of saying the memory came from somewhere else.

Two columns cost a pass because the layouts differ rather than the names. A boolean is a bit per row here and a byte per row there. A date is 32 bits here and 64 in a datetime64. Both are one widening pass and neither has another way.

The mapping, and each line of it has a test:

column	numpy
INT, FLOAT, BOOL	int64, float64, bool
DATE	datetime64[D]
DATETIME, ZONED DATETIME	datetime64[ns], the instant in UTC
TIME, day-time DURATION	timedelta64[ns]
year-month DURATION	timedelta64[M]
STRING, node, rel, path, list, record	object array

A time of day is nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A datetime with an offset is the instant, because numpy has no zone to keep the offset in. A time with an offset is refused, which is the same refusal the Arrow path makes and for the same reason: dropping the offset would move the value.

Nulls are the other half of the shape. numpy has no missing integer, so a column with a null in it comes back as a numpy.ma.masked_array: the data array underneath is still the engine's buffer and the mask is built from the validity bitmap the engine already filled, so masking is a wrapper and not a copy. A column with nothing missing is a plain array, because a mask nothing is masked by is a second buffer nobody asked for. Object columns carry None in the cell instead, since they have somewhere to put one.

Two columns of the same name are refused rather than silently collapsed into one dict entry, and the message says to name them apart.

Over a million rows in two columns on this machine: fetchnumpy() 26 ms, against 47 ms for building the Arrow table and calling to_numpy on each of its columns, and 130 ms for the same rows as tuples.

The extension links numpy's C API through rust-numpy, which loads it at the first array rather than at module init, so import zudb still imports numpy no more than it imports pandas. There is a test that says so and a line in the clean-machine smoke test that checks the refusal reads pip install 'zudb[numpy]'.

27 tests in tests/test_numpy.py, the stub, the numpy extra in pyproject, and a paragraph in the columns section of the README. Green locally: ruff, ruff format, clippy, cargo fmt, and the suite.