[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node by abhivermaaa · Pull Request #28 · ps-dev/hydra

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaa · 2024-06-06T07:55:36Z

[Adapt1-1545] | Publish lag on consumer group internal topic as per the configured time for each node

Description :
This PR contains changes for :

Publishing a logger which is having a lag w.r.t. to _"hydra.consumer-groups" topic which is a DVS internal topic.
Interval for publishing lag is configurable and default is set to 1 minute.
It will publish lag specific to each node.

… aand also added ConsumerGroupsOffsetFacade for holding all offset info

… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

[ADAPT1-1526] | Create an offset stream for consumer group internal topic

…s/ConsumerGroupsEndpoint.scala

…ConsumerGroupsAlgebra.scala

Adapt1 1533 | Use offset-streams to check lag in internal topic

…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

… for new case object

…for _hydra.consumer_groups internal topic

…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

…to 1.minutes as deafult | Added cascading changes to spec files

…ers/ConsumerGroupMarshallers.scala

abhivermaaaand others added 18 commits April 21, 2024 16:06

Added an offset stream which will be having all committed offset info…
d179477
… aand also added ConsumerGroupsOffsetFacade for holding all offset info

returning empty fs2.stream instead of no implementation
9e43eed

Added chnages for creating a new endpoint for returing hydra internal…
7f16c3b
… consumer topic partition offset map | Added changes for using offset stream and updating the consumer group offset facade

Merge pull request#22from ps-dev/adapt1-1526
4d6d7d3
[ADAPT1-1526] | Create an offset stream for consumer group internal topic

Added changes for ingestors/kafka/src/main/scala/hydra/kafka/endpoint…
d9a0390
…s/ConsumerGroupsEndpoint.scala

Added changes to ingestors/kafka/src/main/scala/hydra/kafka/algebras/…
5052eda
…ConsumerGroupsAlgebra.scala

Merge pull request#24from ps-dev/adapt1-1533
ef340e1
Adapt1 1533 | Use offset-streams to check lag in internal topic

Refactored the getOffsetsForInternalCGTopic which is getting offset d…
d67bd93
…ata from cache | optimized the method | Added code to remove the partitions having 0 latest offset | partition level lag is calucalted properly

Created a new response having consolidated lags and lag percentage | …
dba86d2
…Added marshalling logic for new case object in ingestors/kafka/src/main/scala/hydra/kafka/marshallers/ConsumerGroupMarshallers.scala

Added offset change code for starting with 0 | Add code to marshaller…
ad8a210
… for new case object

Added code for sending the slack notifictaion having all offset info …
07bd289
…for _hydra.consumer_groups internal topic

Added test cases for checking the lag
4319877

Added impl to test TestConsumerGroupsAlgebra
593acab

Adding changes
02efd85

Refactored code to have a single stream and adjusted updation of cach…
bc7c6f7
…e one after the other. First Consumer group storage facade is getting updated then just after that consumer group offset facade.

Added code in app config in order to pick lagPublishInterval and set …
cd03d73
…to 1.minutes as deafult | Added cascading changes to spec files

Removed code from ingestors/kafka/src/main/scala/hydra/kafka/marshall…
e183771
…ers/ConsumerGroupMarshallers.scala

removed earlier change
8212ae2

abhivermaaa requested review from RenjithSomasekharan, aadit-chugh, aman-minz and shreedhar-kc June 6, 2024 08:05

pavan-kumarkulkarni force-pushed the adapt1-1545 branch from 8212ae2 to bc7c6f7Compare June 7, 2024 06:25

abhivermaaa added 2 commits June 10, 2024 16:27

Added a comment and Added active partition count in the logger
74f962d

limitting decimal places in the lag logger
35eb28d