[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE by dongjoon-hyun · Pull Request #30418 · apache/spark

dongjoon-hyun · 2020-11-19T03:15:32Z

What changes were proposed in this pull request?

This PR fixes the RAT exclusion rule which was originated from SPARK-1144 (Apache Spark 1.0)

Why are the changes needed?

This prevents the situation like #30415.

Currently, it missed catalog directory due to .log rule.

$ dev/check-license
Could not find Apache license headers in the following files:
!????? /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataColumn.java
!????? /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsMetadataColumns.java

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CI with the new rule.

dongjoon-hyun · 2020-11-19T03:20:54Z

cc @rdblue , @HyukjinKwon , @viirya

dongjoon-hyun · 2020-11-19T03:24:26Z

cc @ScrapCodes for SPARK-1144, too

viirya · 2020-11-19T03:35:53Z

 .*txt
 .*json
 .*data


Does it also apply to above .txt, .json, .data?

To be safe, yes. Those are also looks suspicious.

sunchao · 2020-11-19T03:48:10Z

nice find @dongjoon-hyun !

SparkQA · 2020-11-19T04:00:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35916/

dongjoon-hyun · 2020-11-19T04:02:36Z

Thanks. @viirya , @HyukjinKwon , @sunchao .
I updated more according to @viirya 's comment.

dongjoon-hyun · 2020-11-19T04:04:18Z

+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#


I created a JIRA because I touch this file inevitabily.

For this change, we need to wait for the full test result of GitHub Action.

dongjoon-hyun · 2020-11-19T04:07:41Z

+.*\.avsc
+.*\.txt
+.*\.json
+.*\.data


This hides the directory data and metadata.

dongjoon-hyun · 2020-11-19T04:07:52Z

+.*\.txt
+.*\.json
+.*\.data
+.*\.log


This hides catalog directory.

HyukjinKwon · 2020-11-19T04:20:29Z

+.*\.res
+flights_tiny.txt.1
+over1k
+over10k


I see. I checked these are .txt

HyukjinKwon · 2020-11-19T04:20:34Z

+flights_tiny.txt.1
+over1k
+over10k
+exported_table/*


SparkQA · 2020-11-19T04:33:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35916/

SparkQA · 2020-11-19T04:51:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35920/

ScrapCodes · 2020-11-19T04:53:01Z

@dongjoon-hyun Thanks for fixing this. My bad, this bug have existed for years.

SparkQA · 2020-11-19T05:23:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35920/

SparkQA · 2020-11-19T05:33:02Z

Test build #131314 has finished for PR 30418 at commit b7aa6f9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-11-19T06:18:29Z

retest this please

SparkQA · 2020-11-19T06:49:31Z

Test build #131316 has finished for PR 30418 at commit 45658f0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-19T07:16:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35927/

SparkQA · 2020-11-19T07:43:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35927/

dongjoon-hyun · 2020-11-19T07:58:43Z

Thank you, @HyukjinKwon , @viirya , @ScrapCodes , @sunchao .
Merged to master.

SparkQA · 2020-11-19T08:05:02Z

Test build #131323 has finished for PR 30418 at commit 45658f0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

[MINOR][INFRA] Fix rat exclusion not to exclude catalog
1d35ad4

dongjoon-hyun mentioned this pull request Nov 19, 2020
[SPARK-31255][SQL] Add SupportsMetadataColumns to DSv2 #28027
Closed

viirya approved these changes Nov 19, 2020
View reviewed changes

viirya reviewed Nov 19, 2020
View reviewed changes

HyukjinKwon approved these changes Nov 19, 2020
View reviewed changes

Fix more
45658f0

github-actionsBot added the SQL label Nov 19, 2020

dongjoon-hyun changed the title ~~[MINOR][INFRA] Fix rat exclusion not to exclude catalog~~[MINOR][INFRA][TESTS] Fix rat exclusion not to exclude catalogNov 19, 2020

dongjoon-hyun commented Nov 19, 2020
View reviewed changes

dongjoon-hyun changed the title ~~[MINOR][INFRA][TESTS] Fix rat exclusion not to exclude catalog~~[SPARK-33483][INFRA][TESTS] Fix rat exclusion not to exclude catalogNov 19, 2020

dongjoon-hyun changed the title ~~[SPARK-33483][INFRA][TESTS] Fix rat exclusion not to exclude catalog~~[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSENov 19, 2020

dongjoon-hyun commented Nov 19, 2020
View reviewed changes

Comment threaddev/.rat-excludes
.*\.avsc
.*\.txt
.*\.json
.*\.data
dongjoon-hyunNov 19, 2020
Copy link
Copy Markdown
MemberAuthor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This hides the directory data and metadata.

dongjoon-hyun commented Nov 19, 2020
View reviewed changes

HyukjinKwon reviewed Nov 19, 2020
View reviewed changes

Comment threaddev/.rat-excludes
.*\.res
flights_tiny.txt.1
over1k
over10k
HyukjinKwonNov 19, 2020
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I checked these are .txt

HyukjinKwon reviewed Nov 19, 2020
View reviewed changes

Comment threaddev/.rat-excludes
flights_tiny.txt.1
over1k
over10k
exported_table/*
HyukjinKwonNov 19, 2020
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is .q

HyukjinKwon approved these changes Nov 19, 2020
View reviewed changes

viirya approved these changes Nov 19, 2020
View reviewed changes

ScrapCodes approved these changes Nov 19, 2020
View reviewed changes

dongjoon-hyun closed this in d5e7bd0Nov 19, 2020

Uh oh!

Conversation

dongjoon-hyun commented Nov 19, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunchao commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

ScrapCodes commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

HyukjinKwon commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dongjoon-hyun commented Nov 19, 2020•
edited
Loading