PARQUET-490: Add file of DELTA_BINARY_PACKED encoding

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)

[shanhuuang]

data/delta_binary_packed.parquet is generated with parquet-mr version 1.10.0
The file contents are in data/delta_binary_packed_expect.csv.

[pitrou]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

[pitrou]

Also, what is the "int_value" column at the end?

[shanhuuang]

I'm curious: how was the data generated? It seems some columns are always negative, some not.

I generated a SQL query like "insert overwrite table some_table values ..." using python script.
The type of previous 64 columns are bigint, and the type of "int_value" column is int.
The script is roughly like this:

foriinrange(0, 66):
ifi==65: # for int_valueforjinrange(0, 33):
values[j][i] =str(np.random.randint(-2147483648, 2147483647, dtype=np.int32))
elifi==0: # for bitwidth0x=np.random.randint(-9223372036854775808, 9223372036854775807, dtype=np.int64)
forjinrange(0, 33):
values[j][i] =str(x)
else: # for bitwidth1 to bitwidth64values[0][i] =str(0) # the first row is always 0min_delta=-pow(2,i-1) # generate min_delta in a block, min_delta = -2^(i-1)values[1][i] =str(min_delta) # the second row is always -2^(i-1)delta_in_miniblock=np.random.randint(0, pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [2^(i-1), 2^(i))values[2][i] =str(values[1][i]+min_delta+delta_in_miniblock) # make sure that max bitwidth of delta_in_miniblock is ipre_val=values[2][i]
forjinrange(3, 33):
delta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1) # delta_in_miniblock belongs to [0, 2^(i))new_val=pre_val+min_delta+delta_in_miniblockwhile (new_val>9223372036854775807): # int64 overflowdelta_in_miniblock=np.random.randint(-pow(2,i-1), pow(2,i-1), dtype=np.int64)
delta_in_miniblock+=pow(2,i-1)
new_val=pre_val+min_delta+delta_in_miniblockvalues[j][i] =str(new_val)
pre_val=new_val

[pitrou]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

[shanhuuang]

Thank you. A thought: there are 33 values in each column. This seems it is exactly the size of a miniblock + the first value encoded in the header. Shouldn't you arrange to have several miniblocks, and possibly several blocks even? (also, perhaps better if the last miniblock isn't full).

OK. I will generate a new file with 200 rows including 1 first value, 2 blocks and 3 miniblocks(the last miniblock has 7 values)