Emit events for all known types by dmdashenkov · Pull Request #127 · SpineEventEngine/ProtoData

dmdashenkov · 2023-03-17T09:36:54Z

In this PR we enable ProtoData plugin authors to listen to Protobuf compiler events that describe types from dependencies.

Previously, events like TypeEntered were only emitted for Proto definitions from within the module that ProtoData is processing. This is good for many cases, since this removes the need to check whether or not any code has to be generated based on a given event. However, this limited the users to only the types defined in the module. There was no way to know about a Proto message type from outside the module.

Now, we introduce a new DepdendencyDiscovered event and a ProtobufDependencyFile view. The two new types allow ProtoData plugin authors to generate code with awareness of the module's dependencies.

codecov · 2023-03-17T09:59:32Z

Codecov Report

Merging #127 (ff3ab1e) into master (23108f9) will increase coverage by 2.38%.
The diff coverage is 94.17%.

Additional details and impacted files

@@ Coverage Diff @@## master #127 +/- ##
============================================
+ Coverage 79.35% 81.73% +2.38% - Complexity 236 240 +4 
============================================
Files 73 79 +6 Lines 1889 1993 +104 Branches 140 146 +6 ============================================
+ Hits 1499 1629 +130 + Misses 335 313 -22 + Partials 55 51 -4

dmdashenkov · 2023-03-17T11:26:50Z

@armiol, @alexander-yevsyukov, PTAL.

This is a small yet consequential change set. Let's further discuss it a video call if necessary.

alexander-yevsyukov

Please see my comments. In general, I think that the generation_requested flag approach should be discussed.

alexander-yevsyukov · 2023-03-17T12:07:56Z

 report_paths: '**/build/test-results/test/TEST-*.xml'
 require_tests: true # will fail workflow if test reports not found

- # See: https://github.com/marketplace/actions/junit-report-action


Can we revert these changes please? Separation of test reports was needed some time ago when investigating build failures at CI.

@dmdashenkov I think this comment still stands.

alexander-yevsyukov · 2023-03-17T12:12:58Z

@@ -68,13 +68,15 @@ internal class EnumCompilerEvents(
 EnumEntered.newBuilder()


Please consider using Kotlin DSL for Protobuf. It should be already available here. I've just tried it in another branch and it worked for me.

armiol

@dmdashenkov I am now requesting changes, so that you could re-request from me, once this PR is ready. Alexander is off for a while, so it will be just me.

…s produce the same output

dmdashenkov · 2023-03-24T15:59:18Z

@armiol, PTAL.

armiol

@dmdashenkov please see my comments.

armiol · 2023-03-24T16:21:55Z

 report_paths: '**/build/test-results/test/TEST-*.xml'
 require_tests: true # will fail workflow if test reports not found

- # See: https://github.com/marketplace/actions/junit-report-action


@dmdashenkov I think this comment still stands.

armiol · 2023-03-24T16:23:21Z

 RpcName rpc = 3;
 }
+
+// Emitted when a Protobuf dependency file is discovered.


Is this about discovering an import in our code? I am not sure what do you mean by "discovered".

armiol · 2023-03-24T16:24:06Z

+// Normally, no code should be generated for dependencies. However, they can be used for additional
+// info when generating code for the module's own types.
+//
+message DependencyDiscovered {


I would be more precise with this name, so that Java- and Proto-level dependencies could be distinguished.

I've updated the doc somewhat. I'd like to avoid adding the Proto prefix because:
we only work with Protobuf, no Java or JS or other dependency is ever considered in ProtoData (not talking about the Gradle compiler here, only ProtoData proper);
this is one of a series of events such as TypeEntered, OptionDiscovered, etc., which do not specify the language either.

armiol · 2023-03-24T16:27:02Z

+}
+
+/**
+ * Assigns the field type and cardinality (`map`/`list`/`oneof_name`/`single`) to the receiver


Is this, in fact, copying the field type and the cardinality? If so, maybe we could get rid of this "assign" term which feels a bit over-used in our domain?

armiol · 2023-03-24T16:28:32Z

+ * Assigns the field type and cardinality (`map`/`list`/`oneof_name`/`single`) to the receiver
+ * builder.
+ *
+ * @return the receiver for method chaining.


I think, in Kotlin we still don't put periods in @return statement descriptions.

armiol · 2023-03-24T16:32:46Z

+/**
+ * Produces a sequence by walking through all the nested message definitions staring with `type`.
+ *
+ * @param type the message definition which may contain nested message definition to walk though


armiol · 2023-03-24T16:33:35Z

+ *
+ * The event reflects all the definitions from the file.
+ */
+internal fun toDependencyEvent(fileDescriptor: Descriptors.FileDescriptor) =


How about discoverDependencies(fileDescriptor: Descriptors.FileDescriptor)?

How about discoverDependencies(fd: FileDescriptor)? :)

armiol · 2023-03-24T16:34:19Z

 import io.spine.util.interlaced

 /**
 * Documentation contained in a Protobuf file.


Given its description, I don't really understand why such a generic thing is now in .backend.

I'm not a fan of the word .backend either.
This is the top-level package name for the :compiler module. :compiler serves as main implementation module for ProtoData.
Previously, Documentation, CompilerEvents, and other types where placed in the :api module in ...event package. However, it's not clear that those components belong in the :api module. Moving them to :compiler is logical since this is clearly an implementation detail (hence the internal modifier). The change of the package is a mere consequence.
I could repackage said components into ...backend.event, but I don't see much value in that right now. Would love to hear what you think.

armiol · 2023-03-24T16:36:02Z

 fun fromMessage(descriptor: Descriptor): LocationPath {
 val numbers = mutableListOf<Int>()
- numbers.add(FileDescriptorProto.MESSAGE_TYPE_FIELD_NUMBER)
+ numbers.add(DescriptorProtos.FileDescriptorProto.MESSAGE_TYPE_FIELD_NUMBER)


There is a lot of such changes in this PR. Don't you think it's less convenient to address the types and their members by specifying a "full" (so to speak) name, rather than stripping some common parts by having them "statically" imported?

armiol · 2023-03-24T16:37:11Z

-import io.spine.protodata.MessageType
 import io.spine.protodata.OneofGroup
 import io.spine.protodata.TypeName
+import io.spine.protodata.event.FieldEntered


Given that the events apparently are still in io.spine.protodata.event package, I don't understand why MessageCompilerEvents isn't.
I have the same question regarding all the similar cases. Maybe, we should discuss the idea of re-packaging things into .backend as a whole. Because right now I don't understand the goal.

@dmdashenkov, we do need event package so that Javadocs, KDocs, and source code JARs are easier to understand. Please do have events coming under the event package.

alexander-yevsyukov · 2023-03-28T15:49:47Z

+ * A type URL contains the type URL prefix and the qualified name of the type separated by
+ * the slash (`/`) symbol. See the docs of `google.protobuf.Any.type_url` for more info.
+ */
+ public fun typeUrl(): String? {


If the return value can really be null, please document when such a case can occur.

ProtoDeclarationName has a similar method that returns non-nullable String. How these two types are different in that regards?

Oops, that was a mistake. Fixed it.

alexander-yevsyukov · 2023-03-28T15:56:29Z

+ doc = documentation.forField(this@buildField)
+ this@field.number = this@buildField.number
+ this@field.declaringType = declaringType
+ copyTypeAndCardinality(this@buildField)


Please extract this@buildField into a variable (e.g. fieldDescr, or thisField). This way you'd have fewer “dogs” around.

alexander-yevsyukov

Please see my additional comments.

alexander-yevsyukov · 2023-03-28T15:58:46Z

+ declaredIn = declaringType
+ this@enumConstant.number = this@buildConstant.number
+ orderOfDeclaration = index
+ doc = documentation.forEnumConstant(this@buildConstant)


Similarly to the comment above, extracting this@buildConstant into a val in this block could improve the readability.

dmdashenkov · 2023-03-30T12:09:18Z

@alexander-yevsyukov, @armiol, PTAL again.

alexander-yevsyukov

Please see my comments. Almost there.

alexander-yevsyukov · 2023-03-30T13:08:21Z

+ * A type URL contains the type URL prefix and the qualified name of the type separated by
+ * the slash (`/`) symbol. See the docs of `google.protobuf.Any.type_url` for more info.
+ */
+ public fun typeUrl(): String {


Why don't we have it as another property rather than a function? Just curious, how you see it.
One reason I can think of is to have shorter names in Java code. But we aim to Kotlin anyway, and we always can have @JvmName defined, should we want to.

alexander-yevsyukov · 2023-03-30T13:43:19Z

 event.option.value.unpackGuessingType() shouldBe enumValue {
- name = IdempotencyLevel.NO_SIDE_EFFECTS.name
- number = IdempotencyLevel.NO_SIDE_EFFECTS_VALUE
+ name = DescriptorProtos.MethodOptions.IdempotencyLevel.NO_SIDE_EFFECTS.name


Can we import MethodOptions, please?

dmdashenkov · 2023-03-30T15:21:58Z

@alexander-yevsyukov, PTAL again.

alexander-yevsyukov

LGTM

Since we've discussed this PR in a call already.

dmdashenkov added 3 commits March 16, 2023 13:31

Emit events about all types
fdc909b

Add a test for non-request files
a4f32df

Bump version
1534077

dmdashenkov self-assigned this Mar 17, 2023

Add doc
77a711d

dmdashenkov marked this pull request as ready for review March 17, 2023 11:25

dmdashenkov requested review from alexander-yevsyukov and armiol March 17, 2023 11:25

alexander-yevsyukov suggested changes Mar 17, 2023
View reviewed changes

dmdashenkov added 3 commits March 17, 2023 17:08

Add definitions for a dependency-based view
c551eb9

Implement a view collecting dependency file data
643ad58

Move some implementation details to :compiler
b5963fd

armiol suggested changes Mar 21, 2023
View reviewed changes

dmdashenkov added 10 commits March 22, 2023 16:26

Emit events for dependencies
f8ca578

More testing
9811ba2

Add file path in services and remove the shouldGenerate flag
a412d59

Split new model-building code into a separate file
ea7025a

Assemble separate functions into a context-class
47884bb

Add missing type details
b363bf2

Add a test for ensuring event-based and procedure-based model builder…
7a0617e
…s produce the same output

Clean up Proto definition
85c211e

Fix imports
b105840

More doc
e078f86

dmdashenkov requested a review from armiol March 24, 2023 15:59

armiol previously requested changes Mar 24, 2023
View reviewed changes

dmdashenkov added 2 commits March 27, 2023 11:39

Revert changes from config
281d9b1

More details on the new compiler event
2527f70

dmdashenkov added 2 commits March 28, 2023 18:14

Simplify proto definition construction
51ed614

Improve readability
07dea0f

alexander-yevsyukov reviewed Mar 28, 2023
View reviewed changes

alexander-yevsyukov suggested changes Mar 28, 2023
View reviewed changes

alexander-yevsyukov mentioned this pull request Mar 28, 2023
same_as validation option SpineEventEngine/validation#83
Open

dmdashenkov added 9 commits March 30, 2023 10:55

Fix type nullability
c477f1b

Refactor functions to avoid this@smth-s
c8b2a60

Extract duplicates
0fa23c7

Repackage event factories
d6133ec

Fix typo
af59295

Add doc
cd18964

Fix typo
113e558

Clear up imports and use Kotest where possible
518c8b2

Remove redundant Companion specifier
8e207c1

dmdashenkov requested review from alexander-yevsyukov and armiol March 30, 2023 12:09

alexander-yevsyukov suggested changes Mar 30, 2023
View reviewed changes

dmdashenkov added 2 commits March 30, 2023 17:45

Add static imports
5a11ddd

Use val instead of fun
ff3ab1e

dmdashenkov requested a review from alexander-yevsyukov March 30, 2023 15:21

alexander-yevsyukov approved these changes Mar 30, 2023
View reviewed changes

dmdashenkov enabled auto-merge March 31, 2023 08:26

dmdashenkov added this pull request to the merge queue Mar 31, 2023

Merged via the queue into master with commit 621a33aMar 31, 2023

armiol deleted the emit-events-for-all-known-types branch June 13, 2023 16:50

		@@ -68,13 +68,15 @@ internal class EnumCompilerEvents(
		EnumEntered.newBuilder()

Uh oh!

Conversation

dmdashenkov commented Mar 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecovBot commented Mar 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dmdashenkov commented Mar 17, 2023

Uh oh!

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

armiol left a comment

Choose a reason for hiding this comment

Uh oh!

dmdashenkov commented Mar 24, 2023

Uh oh!

armiol left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmdashenkovMar 27, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexander-yevsyukovMar 28, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dmdashenkov commented Mar 30, 2023

Uh oh!

alexander-yevsyukov left a comment

Choose a reason for hiding this comment

Uh oh!

dmdashenkov commented Mar 17, 2023•
edited
Loading

codecovBot commented Mar 17, 2023•
edited
Loading

dmdashenkovMar 27, 2023•
edited
Loading

alexander-yevsyukovMar 28, 2023•
edited
Loading