Skip to content

reset-offsets: add --delete option - #169

Closed
extemporalgenome wants to merge 4 commits into
masterfrom
delete-offsets
Closed

reset-offsets: add --delete option#169
extemporalgenome wants to merge 4 commits into
masterfrom
delete-offsets

Conversation

@extemporalgenome

@extemporalgenomeextemporalgenome commented Dec 11, 2023

Copy link
Copy Markdown
  • Support --delete option for reset-offsets command.
  • Support --before-earliest and --after-latest partition selection flags.
  • Simplifications and lint fixes.

@extemporalgenome
extemporalgenome requested a review from a team as a code ownerDecember 11, 2023 07:41
@extemporalgenome

Copy link
Copy Markdown
Author

Note, t is just a topicctl alias pointing at my own compilation of topicctl (from this branch) running in a kafka-topics container.

bash-5.0# t reset-offsets identified-messages kgillette-test --offset 0 --partitions 0,14,23
[2023-12-12 21:56:30] INFO Checking cluster ID against version in cluster
[2023-12-12 21:56:32] INFO This will reset the offsets for the following partitions in topic identified-messages for group kgillette-test:
------------+-------------
PARTITION | NEW OFFSET ------------+-------------
0 | 0 14 | 0 23 | 0 ------------+-------------
[2023-12-12 21:56:32] INFO Please ensure that all other consumers are stopped, otherwise the reset might be overridden.
OK to continue? (yes/no) yes
[2023-12-12 21:56:39] INFO Success
bash-5.0# t get lags identified-messages kgillette-test 2>&1 | head -n 30
time="2023-12-12 21:58:15" level=info msg="Checking cluster ID against version in cluster"
time="2023-12-12 21:58:17" level=info msg="Group member lags:
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
PARTITION | MEMBER ID | MEMBER OFFSET | MEMBER TIME | LATEST OFFSET | LATEST TIME | OFFSET LAG | TIME LAG ------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
0 | | 0 | | 27380405 | 2023-12-12T21:58:13Z | 27380405 | 1 | | 27412567 | 2023-12-12T21:55:51Z | 27413939 | 2023-12-12T21:58:15Z | 1372 | 144s 2 | | 27428998 | 2023-12-12T21:55:51Z | 27430375 | 2023-12-12T21:58:15Z | 1377 | 144s 3 | | 27424885 | 2023-12-12T21:55:50Z | 27426290 | 2023-12-12T21:58:16Z | 1405 | 145s 4 | | 27409576 | 2023-12-12T21:55:51Z | 27410979 | 2023-12-12T21:58:14Z | 1403 | 143s 5 | | 27413931 | 2023-12-12T21:55:51Z | 27415343 | 2023-12-12T21:58:15Z | 1412 | 144s 6 | | 27420568 | 2023-12-12T21:55:51Z | 27421977 | 2023-12-12T21:58:15Z | 1409 | 144s 7 | | 27378912 | 2023-12-12T21:55:51Z | 27380324 | 2023-12-12T21:58:16Z | 1412 | 144s 8 | | 27396003 | 2023-12-12T21:55:52Z | 27397415 | 2023-12-12T21:58:16Z | 1412 | 143s 9 | | 27370922 | 2023-12-12T21:55:52Z | 27372334 | 2023-12-12T21:58:16Z | 1412 | 143s 10 | | 27375087 | 2023-12-12T21:55:52Z | 27376499 | 2023-12-12T21:58:16Z | 1412 | 143s 11 | | 27371684 | 2023-12-12T21:55:52Z | 27373096 | 2023-12-12T21:58:16Z | 1412 | 143s 12 | | 27381118 | 2023-12-12T21:55:52Z | 27382531 | 2023-12-12T21:58:16Z | 1413 | 143s 13 | | 27385899 | 2023-12-12T21:55:52Z | 27387301 | 2023-12-12T21:58:13Z | 1402 | 140s 14 | | 0 | | 27396588 | 2023-12-12T21:58:13Z | 27396588 | 15 | | 27370625 | 2023-12-12T21:55:52Z | 27372027 | 2023-12-12T21:58:13Z | 1402 | 140s 16 | | 27373673 | 2023-12-12T21:55:53Z | 27375075 | 2023-12-12T21:58:13Z | 1402 | 140s 17 | | 27373152 | 2023-12-12T21:55:53Z | 27374554 | 2023-12-12T21:58:13Z | 1402 | 140s 18 | | 27378605 | 2023-12-12T21:55:53Z | 27380008 | 2023-12-12T21:58:13Z | 1403 | 140s 19 | | 27379616 | 2023-12-12T21:55:53Z | 27381028 | 2023-12-12T21:58:16Z | 1412 | 142s 20 | | 27382512 | 2023-12-12T21:55:45Z | 27383915 | 2023-12-12T21:58:13Z | 1403 | 148s 21 | | 27383199 | 2023-12-12T21:55:54Z | 27384601 | 2023-12-12T21:58:15Z | 1402 | 140s 22 | | 27358526 | 2023-12-12T21:55:54Z | 27359928 | 2023-12-12T21:58:15Z | 1402 | 141s 23 | | 0 | | 27371349 | 2023-12-12T21:58:15Z | 27371349 | 24 | | 27393145 | 2023-12-12T21:55:54Z | 27394537 | 2023-12-12T21:58:14Z | 1392 | 140s 
bash-5.0# t reset-offsets identified-messages kgillette-test --delete --before-earliest
[2023-12-12 21:58:51] INFO Checking cluster ID against version in cluster
[2023-12-12 21:58:53] INFO This will delete the offsets for the following partitions in topic identified-messages for group kgillette-test:
-------------
PARTITION -------------
0 14 23 -------------
OK to continue? (yes/no) yes
[2023-12-12 21:58:57] INFO Success
bash-5.0# t get lags identified-messages kgillette-test 2>&1 | head -n 30
time="2023-12-12 21:59:19" level=info msg="Checking cluster ID against version in cluster"
time="2023-12-12 21:59:22" level=info msg="Group member lags:
------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
PARTITION | MEMBER ID | MEMBER OFFSET | MEMBER TIME | LATEST OFFSET | LATEST TIME | OFFSET LAG | TIME LAG ------------+-----------+---------------+----------------------+---------------+----------------------+------------+-----------
0 | | -1 | | 27381327 | 2023-12-12T21:59:19Z | 27381328 | 1 | | 27412567 | 2023-12-12T21:55:51Z | 27414879 | 2023-12-12T21:59:20Z | 2312 | 209s 2 | | 27428998 | 2023-12-12T21:55:51Z | 27431300 | 2023-12-12T21:59:19Z | 2302 | 208s 3 | | 27424885 | 2023-12-12T21:55:50Z | 27427246 | 2023-12-12T21:59:21Z | 2361 | 210s 4 | | 27409576 | 2023-12-12T21:55:51Z | 27411911 | 2023-12-12T21:59:19Z | 2335 | 207s 5 | | 27413931 | 2023-12-12T21:55:51Z | 27416275 | 2023-12-12T21:59:19Z | 2344 | 207s 6 | | 27420568 | 2023-12-12T21:55:51Z | 27422913 | 2023-12-12T21:59:19Z | 2345 | 207s 7 | | 27378912 | 2023-12-12T21:55:51Z | 27381274 | 2023-12-12T21:59:20Z | 2362 | 208s 8 | | 27396003 | 2023-12-12T21:55:52Z | 27398371 | 2023-12-12T21:59:21Z | 2368 | 208s 9 | | 27370922 | 2023-12-12T21:55:52Z | 27373287 | 2023-12-12T21:59:20Z | 2365 | 208s 10 | | 27375087 | 2023-12-12T21:55:52Z | 27377449 | 2023-12-12T21:59:20Z | 2362 | 207s 11 | | 27371684 | 2023-12-12T21:55:52Z | 27374048 | 2023-12-12T21:59:20Z | 2364 | 208s 12 | | 27381118 | 2023-12-12T21:55:52Z | 27383471 | 2023-12-12T21:59:20Z | 2353 | 207s 13 | | 27385899 | 2023-12-12T21:55:52Z | 27388233 | 2023-12-12T21:59:19Z | 2334 | 206s 14 | | -1 | | 27397521 | 2023-12-12T21:59:19Z | 27397522 | 15 | | 27370625 | 2023-12-12T21:55:52Z | 27372959 | 2023-12-12T21:59:19Z | 2334 | 206s 16 | | 27373673 | 2023-12-12T21:55:53Z | 27376025 | 2023-12-12T21:59:20Z | 2352 | 207s 17 | | 27373152 | 2023-12-12T21:55:53Z | 27375485 | 2023-12-12T21:59:19Z | 2333 | 205s 18 | | 27378605 | 2023-12-12T21:55:53Z | 27380954 | 2023-12-12T21:59:19Z | 2349 | 206s 19 | | 27379616 | 2023-12-12T21:55:53Z | 27381974 | 2023-12-12T21:59:21Z | 2358 | 207s 20 | | 27382512 | 2023-12-12T21:55:45Z | 27384859 | 2023-12-12T21:59:19Z | 2347 | 214s 21 | | 27383199 | 2023-12-12T21:55:54Z | 27385533 | 2023-12-12T21:59:19Z | 2334 | 205s 22 | | 27358526 | 2023-12-12T21:55:54Z | 27360872 | 2023-12-12T21:59:19Z | 2346 | 205s 23 | | -1 | | 27372291 | 2023-12-12T21:59:19Z | 27372292 | 24 | | 27393145 | 2023-12-12T21:55:54Z | 27395479 | 2023-12-12T21:59:19Z | 2334 | 205s 

@extemporalgenome

Copy link
Copy Markdown
Author

This is superseded by #172, which is intended to be followed by other PRs as well.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@extemporalgenome