Add codecs for dealing with pgsparse vector.

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though

[vpetrovykh]

Add codecs for converting to/from regular arrays to sparse vectors.

[msullivan]

Probably make this a uint32_t, and do an unpack_uint32 when reading it out. Then negative numbers will be IndexErrors instead of wrapping from the back

[vpetrovykh]

Maybe I don't quite understand some subtle interaction here, but pg_sparse code has this when receiving binary data:

	int16 n_elem;
int16 dim;
int16 unused;
n_elem = pq_getmsgint(buf, sizeof(int16));
dim = pq_getmsgint(buf, sizeof(int16));

Which, I think, will interpret the value as a signed int16, even though it will eventually assign that to an int32 n_elem in the Vector struct. Honestly I find this whole business of jumping between int16 and int32 a lot more odd than whether it's unsigned.

https://github.com/paradedb/paradedb/blob/585c5dae321fa99576f1399a9589887573e0c700/pg_sparse/src/svector.c#L423-L428

[msullivan]

We should declare a float[:] array_view and assign val to it, like we do in pgvector_decode. This will use the buffer interface and allow us to avoid boxing and unboxing the floats.
(This is a https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html)

[msullivan]

Actually maybe we want to do float[::1] array_view, which should be faster since it will rely on the backing array being contiguous (which it always will be).

[msullivan]

It might be worth having a typed memoryview fast path for array/ndarray, like we do for pgvector.

Don't know if there is a way to do that without the annoying code duplication, though