Bq proto abstractions - #3

Merged

lavkesh merged 8 commits into

mainfrom

bq-proto-abstractions

Apr 27, 2022

lavkesh commented Apr 25, 2022

Member

No description provided.

lavkesh added 5 commits

April 25, 2022 16:32

feat: adding getmapping method for proto message

479ab8f

chore: move functions to correct classes

80e1ac3

chore: added coverage

9c4c53b

Merge remote-tracking branch 'origin/main' into bq-proto-abstractions

e68ae67

tests: adding missing tests

d535968

lavkesh force-pushed the bq-proto-abstractions branch from af41a26 to d535968Compare

April 25, 2022 12:15

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/bigquery/BigqueryStencilUpdateListenerFactory.java

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/bigquery/BigqueryStencilUpdateListenerFactory.java Outdated

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/ParsedOdpfMessage.java

- Map<String, Object> getMapping();
-} No newline at end of file
+ void validate(OdpfSinkConfig config);

MeghajitApr 26, 2022

Member

I am assuming validate will be used to check the configs and then after we will call getRaw or getMapping. However, it neither returns a boolean nor throws any exceptions. How will this be used ?

lavkeshApr 26, 2022

MemberAuthor

yea we can think about the method signature. it can just log or throw runtime exception too, we don't have to declare RuntimeException.

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/json/JsonOdpfMessageParser.java Outdated

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/json/JsonOdpfMessageParser.java Outdated

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/json/JsonOdpfMessageParser.java

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/proto/ProtoMapper.java

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/proto/ProtoOdpfMessageParser.java

Meghajit reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/proto/ProtoOdpfParsedMessage.java

chore: review comments

2d77c4d

lavkesh force-pushed the bq-proto-abstractions branch from 5006196 to 2d77c4dCompare

April 26, 2022 08:42

lavkesh added 2 commits

April 27, 2022 14:31

chore: convert OdpfMessage to a class

d634cea

refactor: make smaller private function

7e5b7a6

lavkesh merged commit e62c8a4 into main

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/bigquery/BigQuerySinkFactory.java

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/expcetion/ConfigurationException.java

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/expcetion/ProtoNotFoundException.java

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/message/MessageUtils.java

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/config/OdpfSinkConfig.java

  @ConverterClass(InputSchemaDataTypeConverter.class)
  @DefaultValue("PROTOBUF")
- InputSchemaDataType getInputSchemaDataTye();
+ InputSchemaDataType getSinkConnectorSchemaDataTye();

MayurGubreleMay 11, 2022

Member

There's a typo in the getter name 'Tye'. Also, Do you want to keep Object and Converter class names like this or with prefix 'SinkConnector' ?
Unable to comment there so adding it in this thread. Let's keep the key and getter names in sync for line 41

lavkeshMay 12, 2022

MemberAuthor

Right. I will check the configs.

MayurGubrele reviewed

View reviewed changes

src/main/java/io/odpf/sink/connectors/config/OdpfSinkConfig.java

lavkesh deleted the bq-proto-abstractions branch

May 12, 2022 05:14

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment