feat: batch address trees by sergeytimoshin · Pull Request #18 · Lightprotocol/photon

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin · 2025-04-03T01:25:48Z

Overview

This pr adds support for batch address updates and introduces a new API endpoint getBatchAddressUpdateInfo

Changes:

API

NEW:getBatchAddressUpdateInfo endpoint.
Retrieves addresses pending insertion from the new address queue, along with necessary proofs and context for batch updates.

Request:

tree: Hash
batch_size: u16

Response:

start_index: u64,
addresses: Vec<AddressQueueIndex>,
non_inclusion_proofs: Vec<MerkleContextWithNewAddressProof>,
subtrees: Vec<[u8; 32]>,

DB

Added address_queues table, corresponding models & migrations.
Table structure:

pub address: Vec<u8>,
pub tree: Vec<u8>,
pub queue_index: i64,

Events

Added PublicTransactionEventV2 structure and parsing logic which contains MerkleTreeSequenceNumberV2(tree_pubkey, queue_pubkey, tree_type, seq).

Tests

tests/integration_tests/batched_state_tree_tests.rs
Contains new test_batched_address_transactions test.

50 addresses appended to the queue
5 batch address append operations with batch_size=10 invoked

Asserts:

getBatchAddressUpdateInfo return non-empty queue after indexing of initial 50 txs and contain valid expected addresses.
Final root is checked against reference tree after applying batch address append txs.

TODO:

Add test for the case when we have 2 addresses in 1 ix.

ananas-block · 2025-04-10T22:33:50Z

+ for (new_address, seq) in event
+ .new_addresses
+ .iter()
+ .zip(event.address_sequence_numbers.iter())
+ {
+ let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
+ .ok_or(IngesterError::ParserError("Missing queue".to_string()))?
+ .clone();
+ state_update_event.addresses.push(AddressQueueUpdate {
+ tree: tree_info.tree.into(),
+ address: new_address.address,
+ queue_index: seq.seq,
+ });
+ }


Suggested change
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
let tree_info = TreeInfo::get(&new_address.mt_pubkey.to_string())
.ok_or(IngesterError::ParserError("Missing queue".to_string()))?
.clone();
state_update_event.addresses.push(AddressQueueUpdate{
tree: tree_info.tree.into(),
address: new_address.address,
queue_index: seq.seq,
});
}
for(new_address, seq)in event
.new_addresses
.iter()
.zip(event.address_sequence_numbers.iter())
{
state_update_event.addresses.push(AddressQueueUpdate{
tree: new_address.mt_pubkey.into(),
address: new_address.address,
queue_index: seq.seq,
});
}

ananas-block · 2025-04-11T00:09:09Z

 }

-pub const MAX_HEIGHT: usize = 32;
+pub async fn get_subtrees(


shouldn't this be in persisted_indexed_merkle_tree ?

I think it makes sense to leave it here, because persisted_state_tree stores nodes for indexed trees, and persisted_indexed_merkle_tree works more like indexed array.

ananas-block

Thanks!
TLDR on comments:

are we sure queue indices are set correctly if we insert 2 addresses in the same tree in the same tx?
found some dead code in parse_public_transaction_event
PublicTransactionEvent::V1 maybe we can get away with less diff by implementing From
Final root assert in test
couple smaller improvements

…blicTransactionEvent

…id-arg feat: add support for custom account compression program ID

sergeytimoshin · 2025-04-29T16:17:36Z

@@ -79,18 +77,15 @@ solana-program = "1.18.0"
 solana-sdk = "1.18.0"


Try to remove

…d address is in the AddressQueue but not in tree yet, we should return error. For V1 trees we still return non-inclusion proof, because we don't have information about V1 queue. * Add offset as request parameter to get_batch_address_update_info. * Various cleanups

…pping

… node persistence

ananas-block · 2025-05-02T15:42:49Z

 light-compressed-account = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
 light-concurrent-merkle-tree = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }
-light-hasher = "2.0.0"
+light-hasher = { git = "https://github.com/Lightprotocol/light-protocol", rev = "368f9f08272db78c74b2ade1a1c2fead27dd0a96" }


why was this necessary?
(I didn't want to release light-hasher again.)

light-merkle-tree-reference has light-hasher dependency as well => there are multiple different versions of crate light_hasher in the dependency graph

sergeytimoshin marked this pull request as ready for review April 3, 2025 01:35

sergeytimoshin force-pushed the sergey/batch-address-updates branch from b089102 to db382f7Compare April 6, 2025 12:07

feat: batch address update support
c24977f

sergeytimoshin force-pushed the sergey/batch-address-updates branch from ed8b8f0 to c24977fCompare April 8, 2025 18:55

sergeytimoshin added 4 commits April 8, 2025 23:51

feat: implement batch address tree tests
394ff1e

refactor: remove debug print statements from multiple modules
596514d

fix: update .gitignore to ignore all .env files
6e03e90

fix: use create_dir_all for creating temp directory
9519fce

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/parser/tx_event_parser.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/persist/persisted_indexed_merkle_tree.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/persist/persisted_indexed_merkle_tree.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/persist/leaf_node.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/parser/tx_event_parser_v2.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/parser/tx_event_parser_v2.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/persist/persisted_state_tree.rs

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/persist/mod.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/parser/tx_event_parser.rs Outdated

ananas-block reviewed Apr 10, 2025
View reviewed changes

Comment threadsrc/ingester/parser/tx_event_parser.rs Outdated

ananas-block reviewed Apr 11, 2025
View reviewed changes

Comment threadsrc/ingester/persist/persisted_indexed_merkle_tree.rs Outdated

ananas-block reviewed Apr 11, 2025
View reviewed changes

Comment threadtests/integration_tests/batched_state_tree_tests.rs Outdated

ananas-block reviewed Apr 11, 2025
View reviewed changes

Comment threadtests/integration_tests/batched_state_tree_tests.rs Outdated

ananas-block reviewed Apr 11, 2025
View reviewed changes

Comment threadtests/integration_tests/batched_state_tree_tests.rs Outdated

ananas-block reviewed Apr 11, 2025
View reviewed changes

sergeytimoshin added 5 commits April 11, 2025 09:54

feat: add legacy state and address trees to tree_info
b50a5db

feat: implement conversion from PublicTransactionEventV1 and V2 to Pu…
c1d7fa7
…blicTransactionEvent

refactor: remove PublicTransactionEvent enum and related conversions
5b93a9f

update tests data
a7f15aa

update tests data
e388997

sergeytimoshin added 11 commits April 12, 2025 11:45

remove leftover
6b2587a

remove leftover
c07793c

remove leftover
a7127c8

update batched address tree tests
9e5b9b2

batched address tree tests wip
358ff97

cleanup
1a41a6d

cleanup
083857a

cleanup
b88fc31

cleanup
f780aa3

add support for custom account compression program ID
d94ae11

Merge pull request#20from Lightprotocol/sergey/compression-program-…
0341197
…id-arg feat: add support for custom account compression program ID

sergeytimoshin commented Apr 29, 2025
View reviewed changes

Comment threadCargo.toml
@@ -79,18 +77,15 @@ solana-program = "1.18.0"
solana-sdk = "1.18.0"
sergeytimoshinApr 29, 2025
Copy link
Copy Markdown
Author
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try to remove

sergeytimoshin commented Apr 29, 2025
View reviewed changes

Comment threadCargo.toml Outdated

sergeytimoshin added 2 commits April 29, 2025 22:51

Switch to solana-pubkey and update dependencies.
2a0892c

sergeytimoshin changed the title ~~feat: batch address trees support~~feat: batch address treesApr 30, 2025

sergeytimoshin added 10 commits April 30, 2025 10:55

Switch to solana-pubkey and update dependencies.
e7cf9ad

Update import for Pubkey from solana_pubkey and adjust account key ma…
3e1db91
…pping

update tree height handling and clean up debug prints
75cf9dd

fix test environment comment and increment state tree height for leaf…
129adf1
… node persistence

test tx data
9f1a7f3

test tx data
89c3991

test tx data
caeb4be

test tx data
f050ed2

refactor address tree handling and improve logging for batch updates
18bc057

refactor batched address tests
3dd48b5

ananas-block reviewed May 2, 2025
View reviewed changes

sergeytimoshin merged commit 61627ae into jorrit/feat-add-testMay 3, 2025