Skip to content

Add Data Catalog createEntry quickstart samples and tests. - #1731

Merged
tswast merged 32 commits into
GoogleCloudPlatform:masterfrom
mesmacosta:task/add-dc-fileset-quickstart-sample
Dec 10, 2019
Merged

Add Data Catalog createEntry quickstart samples and tests.#1731
tswast merged 32 commits into
GoogleCloudPlatform:masterfrom
mesmacosta:task/add-dc-fileset-quickstart-sample

Conversation

@mesmacosta

Copy link
Copy Markdown
Contributor

I would like to add samples for Data Catalog create_entry method, released on 0.28.0-alpha google-cloud-datacatalog version.

Since we already have samples for createEntry and createEntryGroup, I would like to submit this PR as an integrated version, which would be used here:
https://cloud.google.com/data-catalog/docs/how-to/filesets#data-catalog-fileset-java

@mesmacosta
mesmacosta requested a review from a teamNovember 13, 2019 18:58
@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 13, 2019
@mesmacosta

Copy link
Copy Markdown
ContributorAuthor

@tswast

@lesvlesv removed their assignment Nov 14, 2019

// Store names for clean up on teardown
String expectedEntryGroupName =
EntryGroupName.of(PROJECT_ID, LOCATION, entryGroupId).toString();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I think these helpers are being deprecated. At least, they were deprecated in Python (googleapis/gapic-generator#3012) and the same reasoning applies to the Java generator (it's not possible with new proto annotations for configuring the generator).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know the recommended way?
I see two alternatives...

  1. Use EntryGroupName.Builder....setProject(project).setLocation(location).setEntryGroup(entryGroup).build()
  2. Build the string manually

Thanks

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least in Python (2) is recommended, as the resource builders are going away there. I haven't seen any discussion about it outside of Python, though.

I don't know why, as the same configuration changes affect all languages.

@mesmacostamesmacostaNov 28, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, it makes sense going that way if this configuration will affect all languages...
@kurtisvg Could you point me the recommended way in Java ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, they aren't going away for Java. They are generally preferred to manually formatting the string.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tswast WDYT can we keep the helpers?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have doubts that they will actually stay in Java, based on my conversations with Luke on the ACtools team. I'll check with him if the resource helper deprecation is supposed to affect more than just Python.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will wait for your confirmation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can proceed with using the resource builders. I misunderstood the reasoning for removal in Python.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so AFAIK the PR is ready to be merged, @tswast are you able to review it and merge if everything is ok?

@tswasttswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 4, 2019
@kokoro-teamkokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 4, 2019
@tswast

Copy link
Copy Markdown
Contributor

I just added the label to run our CI. Once that passes, I think this is OK to merge.

@tswast

Copy link
Copy Markdown
Contributor

Test failures:

******************** TESTING PROJECTS ********************
------------------------------------------------------------
- testing datacatalog/cloud-client
------------------------------------------------------------
[ERROR] Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.966 s <<< FAILURE! - in com.example.datacatalog.CreateEntryTests
[ERROR] testCreateFilesetEntry(com.example.datacatalog.CreateEntryTests) Time elapsed: 0.55 s <<< ERROR!
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: "my_bucket" is an invalid value for CreateEntryRequest.entry.gcs_fileset_spec.file_patterns. Bucket name must contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). See https://cloud.google.com/storage/docs/naming for details.
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
at com.google.cloud.datacatalog.v1beta1.DataCatalogClient.createEntry(DataCatalogClient.java:1005)
at com.example.datacatalog.CreateFilesetEntry.createEntry(CreateFilesetEntry.java:108)
at com.example.datacatalog.CreateEntryTests.testCreateFilesetEntry(CreateEntryTests.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: "my_bucket" is an invalid value for CreateEntryRequest.entry.gcs_fileset_spec.file_patterns. Bucket name must contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). See https://cloud.google.com/storage/docs/naming for details.
at io.grpc.Status.asRuntimeException(Status.java:533)
... 24 more
[ERROR] com.example.datacatalog.CreateEntryTests Time elapsed: 0.555 s <<< FAILURE!
java.lang.AssertionError: Something went wrong, no entries were generated
at org.junit.Assert.fail(Assert.java:89)
at com.example.datacatalog.CreateEntryTests.tearDownClass(CreateEntryTests.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[ERROR] Failures:
[ERROR] CreateEntryTests.tearDownClass:67 Something went wrong, no entries were generated
[ERROR] Errors:
[ERROR] CreateEntryTests.testCreateFilesetEntry » InvalidArgument io.grpc.StatusRuntim...
[ERROR] Tests run: 6, Failures: 1, Errors: 1, Skipped: 0
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project datacatalog-google-cloud-samples: There are test failures.
[ERROR]
[ERROR] Please refer to /tmpfs/src/github/java-docs-samples/datacatalog/cloud-client/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Testing failed: Maven returned a non-zero exit code.

.setDisplayName("My Fileset")
.setDescription("This fileset consists of ....")
.setGcsFilesetSpec(
GcsFilesetSpec.newBuilder().addFilePatterns("gs://my_bucket/*").build())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to use a real bucket here, such as gs://cloud-samples-data

@mesmacostamesmacostaDec 5, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the docs, underscore should be accepted, I filled a bug to check this, but using a real bucket is better, so made the change, thanks.

@mesmacosta

Copy link
Copy Markdown
ContributorAuthor

ping @tswast

@tswasttswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 10, 2019
@kokoro-teamkokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 10, 2019
@tswast
tswast merged commit 43764f8 into GoogleCloudPlatform:masterDec 10, 2019
@mesmacosta
mesmacosta deleted the task/add-dc-fileset-quickstart-sample branch December 11, 2019 13:12
charlieyu1996 pushed a commit that referenced this pull request Nov 11, 2022
…1731)
* ADD sample for create fileset entry quickstart
* RAN google java format
* CHANGE exception comment
* FIX lint issues
* Split quickstart into a new folder
* Rename files since class name will appear on samples
* Ran java formatter
* revert region tags change
* remove extra space
* change to real bucket name
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@mesmacosta@tswast@kurtisvg@lesv@googlebot@kokoro-team