fix: add retry and timeout to create_new_task

[avilagaston9]

Add retry and timeout to create_new_task

Warning

This PR requires the deploy of new metrics/dashboards

Motivation

Currently, if the gas price rises abruptly, we don't have a mechanism to abort the task sent by the batcher, causing it to get stuck while waiting for the transaction Receipt.

Description

General Description

• This PR adds a timeout of 3 blocks for waiting on the created task receipt.
• If the timeout is reached, proceed as follows:
  1. Gets a bumped_gas_price based on the current gas price of the net, the previous used one and the iteration number.
  2. Sends a transaction with the same nonce and the bumped_gas_price to override the previous transaction, aborting the batch inclusion.
  3. Waits for another 3 blocks to get the Receipt of the cancel transaction.
  4. If the timeout is reached, increments the iteration number and retries from step 1 using exponential backoff with a usize::MAX number of retries until the Receipt is received.

Observation: We use an exponential backoff because the timeout is not the only thing that can fail. For example, the get_gas_price call or sending the transaction can fail too.

Bump Gas Price Mechanism

• The formula used to bump each gas price is as follows:
  • plus_per_iteration = GAS_PRICE_INCREMENT_PERCENTAGE_PER_ITERATION * iteration
  • override_gas_multiplier = OVERRIDE_GAS_PRICE_PERCENTAGE_MULTIPLIER + plus_per_iteration.
  • bumped_gas_price = gas_price * override_gas_multiplier / PERCENTAGE_DIVIDER.

Constants added

• transaction_wait_timeout: 36_000 milliseconds (3 blocks)
• GAS_PRICE_INCREMENT_PERCENTAGE_PER_ITERATION: 5
• OVERRIDE_GAS_PRICE_PERCENTAGE_MULTIPLIER: 120

Additional Changes

• Replaces the http_providers with batcher_signers because they are needed to sign the cancel transaction.
• Removes the GasEscalatorMiddleware from our SignerMiddlewareT type, as we don't want the Provider to bump the created tasks for us.
• Moves get_gas_price and get_current_nonce functions to batcher/aligned-batcher/src/eth/utils.rs.
• Adds the transaction_wait_timeout constant to the batcher config file. This is the configurable time we wait for a transaction to be included in Ethereum on each attempt.
• Adds metrics and a new Grafana panel to track the count of canceled batches.

How to test

Modify the create_new_task function in batcher/aligned-batcher/src/lib.rs to call create_new_task_retryable with a timeout of only one millisecond:

create_new_task_retryable(
batch_merkle_root,
batch_data_pointer.clone(),
proof_submitters.clone(),
fee_params.clone(),1,&self.payment_service,&self.payment_service_fallback,)

Then, run the following:

make anvil_start_with_block_time
make batcher_start_local
make batcher_send_burst_groth16

After some time, you should see the following logs in the batcher:

2024-10-25T00:56:17Z INFO aligned_batcher::retry::batcher_retryables] Creating task for: 0x32e14361580d28b131fc38ac0238e80428062f839629b650cb03e53234a9ff7b
[2024-10-25T00:56:17Z WARN aligned_batcher::retry::batcher_retryables] Error while waiting for batch inclusion: deadline has elapsed
[2024-10-25T00:56:17Z INFO aligned_batcher] Cancelling createNewTask transaction...
[2024-10-25T00:56:17Z INFO aligned_batcher] createNewTask transaction successfully canceled
[2024-10-25T00:56:17Z ERROR aligned_batcher] Failed to send batch to contract, batch will be lost: Receipt not found

You can also modify the call to cancel_create_new_task_retryablein the same way to see the retries while canceling the created_task:

cancel_create_new_task_retryable(&self.batcher_signer,&self.batcher_signer_fallback,
bumped_gas_price,1,)

Type of change

• New feature
• Bug fix
• Optimization
• Refactor

Checklist

• Linked to Github Issue
• This change depends on code or research by an external entity
  • Acknowledgements were updated to give credit
• Unit tests added
• This change requires new documentation.
  • Documentation has been added/updated.
• This change is an Optimization
  • Benchmarks added/run
• Has a known issue
  • Abort batch in Ethereum in case of timeout #1293

[MarcosNicolau]

I tested on macos and linux and everything went well. I left some comments for minor improvements.

[MauroToscano]

Don't bump fee at 0.5 s, nothing in Ethereum gets done in half a second. Use more reasonable constants

[MauroToscano]

Add the constants used in the PR, I want to check the mechanism

[avilagaston9]

Add the constants used in the PR, I want to check the mechanism

@MauroToscano

My bad. I’ve updated the description to include the changes made. Let me know if you’d like me to review anything