Wip durable nat

[Totodore]

Description

Implement a scalable (multi indexing pipeline) nats source and implement a drain mechanism in the indexing pipeline.

Rationale

Nats does not support partitioned consumers, for this reason it is impossible to name a partition/consumer in a consumer group to reference it as a partition in Quickwit (PartitionId). Consumer groups are entirely dynamic in nats (no rebalancing, no static mechanism) and there is no other way to have this feature.
From this first issue, two solutions were considered:

• Using a Nats Ordered consumer that is created dynamically and use the native checkpoints mechanism to re-start at the last position. However with this solution it is not possible to scale the number of pipelines without doing manual subject partitioning. So it is a no-go for us.
• Using a classic Nats consumer with an explicit AckPolicy. And acknowledge in suggest_truncate, but in order to strenghten the pipeline in case of rolling out indexer we also need a drain mechanism so a source can gracefully acknowledge even when the pipeline is shutting down.

Side notes

Other source implementation could benefit from this drain mechanism like gcp pubsub.