Add a workflow that builds the docs and deploys them at staged or production by martin-g · Pull Request #104 · apache/datafusion-python

martin-g · 2022-12-05T22:29:59Z

Which issue does this PR close?

Closes#39.

Rationale for this change

Publish the documentation for the Datafusion Python bindings.

When changes are pushed to master then it will copy the generated HTMLs to asf-staging branch and update https://arrow.staged.apache.org/datafusion-python, i.e. the staging site. See Add a workflow that builds the docs and deploys them at staged or production #104 (comment)
When changes are pushed to a non-RC tag, e.g. 0.8.0, then it will copy the generated HTMLs to asf-site branch and update https://arrow.apache.org/datafusion-python, i.e. the production site. See Issue #39 - Add .asf.yaml for asf-site #105

What changes are included in this PR?

A new GitHub Actions workflow is introduced that builds the docs and deploys them at the appropriate site

Are there any user-facing changes?

No API changes!

martin-g · 2022-12-05T22:32:27Z

@andygrove Before merging this PR please create a new branch named asf-staging from asf-site and update its .asf.yaml file contents to be the same as in https://github.com/martin-g/arrow-datafusion-python/blob/asf-staging/.asf.yaml.
It will be used for the staging deployment.

martin-g · 2022-12-05T22:32:51Z

@kou Please review!

…duction Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

kou · 2022-12-06T01:52:01Z

+ - name: Copy & push the generated HTML
+ run: |
+ cd docs-target
+ find ./ -name "*.html" -type f -print0 | xargs -0 /bin/rm -f


This may not work for other extensions such as *.css. How about using rsync?
rsync \ -a \ --delete \ --exclude '/.git/' \ ../docs/build/html/ \ ./

Right! There are .css/.js/images/... !
I have to test your approach!
The important thing is to preserve anything that is not autogenerated, especially .asf.yaml

How about copying .asf.yaml from master to asf-site?
apache/arrow-site uses this approach: apache/arrow-site#116

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:

find ./ -not -name ".asf.yaml" | grep -v ".git" | xargs rm -rf seems to do the trick.

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:
Can we write both staging: and publish: in master's .asf.yaml?
staging: whoami: asf-stagingsubdir: datafusion-pythonpublish: whoami: asf-sitesubdir: datafusion-python

I think this should work!
The only way to find out is to try it! :-)

I think so too. :-)

done with d5e32a2!
depends on #107

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

kou

+1

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

kou · 2022-12-07T20:22:35Z

+on:
+ push:
+ branches:
+ - docs-workflow


We need to change this to master when we merge this.

Right! I used it to be able to test on my fork!

kou · 2022-12-07T20:41:22Z

+ - name: Checkout docs target branch
+ uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+ ref: ${{ steps.target-branch.outputs.value }}
+ path: docs-target


It seems that we need to prepare asf-site and asf-staging branches for this: https://github.com/martin-g/arrow-datafusion-python/actions/runs/3633415170/jobs/6130403408
How about using git worktree instead?
diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index 806202a..bfafafa 100644 --- a/.github/workflows/docs.yaml+++ b/.github/workflows/docs.yaml@@ -26,12 +26,6 @@ jobs: fi - name: Checkout docs sources uses: actions/checkout@v3 - - name: Checkout docs target branch- uses: actions/checkout@v3- with:- fetch-depth: 0- ref: ${{ steps.target-branch.outputs.value }}- path: docs-target - name: Setup Python uses: actions/setup-python@v4 with: @@ -61,7 +55,12 @@ jobs: - name: Copy & push the generated HTML run: | set -x + git worktree add docs-target cd docs-target + if ! git checkout ${{ steps.target-branch.outputs.value }}; then+ git checkout --orphan ${{ steps.target-branch.outputs.value }}+ git rm -rf .+ fi # delete anything but: 1) '.'; 2) '..'; 3) .git/ find ./ | grep -vE "^./$|^../$|^./.git" | xargs rm -rf cp ../.asf.yaml .

IMO this is more complex/confusing than the current solution.

A task before merging this PR is to create asf-staging branch.

Created: https://github.com/apache/arrow-datafusion-python/tree/asf-staging

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

martin-g · 2022-12-07T21:00:22Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions
Those tags were created from docs-workflow branch, not from master.

kou · 2022-12-07T21:41:59Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)
Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions Those tags were created from docs-workflow branch, not from master.

Could you retry testN after you create asf-site branch on your fork?

martin-g · 2022-12-07T21:50:26Z

Could you retry testN after you create asf-site branch on your fork?

Done!
There is no Deploy DataFusion Python site for test-rc10 and there is such for test10.

kou · 2022-12-07T23:35:09Z

Thanks. Committed: martin-g@34ed8b8

I merge this.

martin-g force-pushed the docs-workflow branch from 6056d73 to a0e9bd6Compare December 5, 2022 22:33

Add a workflow that builds the docs and deploys them at staged or pro…
aea783e
…duction Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from a0e9bd6 to aea783eCompare December 5, 2022 22:36

martin-g mentioned this pull request Dec 5, 2022
Issue #39 - Add .asf.yaml for asf-site #105
Closed

kou reviewed Dec 6, 2022
View reviewed changes

martin-gand others added 2 commits December 6, 2022 09:19

Add a branch for unsupported branch/tag
91d97e6
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Improve the glob to ignored RCx tags
289f36e

martin-g force-pushed the docs-workflow branch 2 times, most recently from 52bcc20 to 64dc8d9Compare December 6, 2022 20:38

"git commit" only if there are changed files
e0a6910
Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from 64dc8d9 to e0a6910Compare December 6, 2022 20:49

martin-g added 4 commits December 6, 2022 23:00

Test building tags
617ead0
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Delete everything but .asf.yaml and .git/ when copying the new docs
832cc6d
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Remove the filtering by paths
8f46ef9
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Do not try to delete . and ..
822153e
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from d27c482 to 822153eCompare December 6, 2022 21:47

martin-g mentioned this pull request Dec 6, 2022
Add entries for publishing production (asf-site) and staging docs #107
Merged

Use .asf.yaml from master/tag
d5e32a2
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

kou approved these changes Dec 7, 2022
View reviewed changes

martin-gand others added 2 commits December 7, 2022 22:55

Remove empty line
2a65b5d
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Fix wording
8b92e71
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Change the tested branch back to master
e0692fa

kou merged commit a110049 into apache:masterDec 7, 2022

kou mentioned this pull request Dec 7, 2022
Add DataFusion Python docs apache/arrow-site#279
Closed

martin-g deleted the docs-workflow branch December 8, 2022 09:38

kou mentioned this pull request Dec 13, 2022
chore: set up release process apache/arrow-adbc#174
Merged

Uh oh!

Conversation

martin-g commented Dec 5, 2022• edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

martin-g commented Dec 5, 2022

Uh oh!

martin-g commented Dec 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-gDec 6, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

martin-g commented Dec 7, 2022

Uh oh!

kou commented Dec 7, 2022

Uh oh!

martin-g commented Dec 7, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Dec 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martin-g commented Dec 5, 2022•
edited by kou
Loading

martin-gDec 6, 2022•
edited
Loading

martin-g commented Dec 7, 2022•
edited
Loading