Add PDP config/ops layer (PR #5)

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.

[anjor]

Summary

This PR adds the configuration and operations layer for the PDP tracker. This work is independent of the pdpclient.go integration (PR #612) and can proceed in parallel.

Changes

Configuration

• Added PDPConfig struct in service/pdptracker/config.go with:
  • BatchSize — max pieces per AddRoots transaction
  • GasCap — gas limit for PDP transactions
  • ConfirmationDepth — blocks to wait for confirmation
  • PollingInterval — tx status polling frequency
• Added config validation with comprehensive tests

Tracker Updates

• Updated PDPTracker to accept PDPConfig in constructor
• Replaced interval parameter with config-driven polling
• Added logging:
  • Start/stop events with config values
  • Proof set status changes (live status)
  • Count of proof sets tracked per run

CLI

• Added CLI flags in cmd/run/pdptracker.go:
  • --pdp-batch-size (default 100)
  • --pdp-confirmation-depth (default 5)
  • --pdp-poll-interval (default 30s)
• Removed old --interval flag
• Added config validation before tracker startup

Testing

All tests pass:

• Config validation tests (5/5)
• Existing PDPTracker tests updated and passing
• MySQL tests skipped (no MariaDB in test env)

[parkan]

hmm so the 30s loop made me think... are we correctly WithTimeout'ing eth rpc calls?

another issue that came to mind is consistency -- the single pass scan-and-pick is better than per wallet approach for this already but worth thinking about

both make me wonder if a getlogs/event driven approach out the gate will save us headaches down the road

[parkan]

so events + multicall to hydrate as needed + cursor in db, something like that? FEVM should support all of these features so replicating the approach from eth L2 indexers might save us a lot of pain

what do you think? aside from that the PR looks OK but the above questions do come to the fore at this point

even with 1000 proofsets (i.e. 1TiB of data) this is already a crazy number of rpc calls as it stands, single pass will take ~10 mins

side note: in a sense this is not our job (indexer needs to be core infra not on client) but the napkin math does not work out

[parkan]

approved with the understanding that this is V0 POC and an event-driven approach will follow 👍

[anjor]

Thanks all. Per discussion, we are treating the current scan loop as PoC behavior and will follow immediately with a scalable ingestion path.

Filed follow-up issue: #615

Plan is events + multicall hydration + durable DB cursor, while keeping current deal update semantics stable.