ARROW-14840: [R][CI] test-ubuntu-20.10-docs nightly build failing due to R install issue

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

[thisisnic]

I'm not 100% sure why this fixes this fail, but downgrading the Ubuntu version to 20.04 seems to fix it, and here's my reasoning.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

Reading the R docs for Ubuntu packages[4], it says there that "Package for the current R 4.1. release* are available for most stable Desktop releases of Ubuntu until their official end of life date" which maybe explains why the URL for R packages works fine for focal[5] but results in a 404 error for groovy [6].

The way of getting the signing key for updating packages changed [4] to no longer use apt-key [7] - I have no idea if this had any effect either, but to be on the safe side I have updated that too.

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html
[2] https://wiki.ubuntu.com/Releases
[3] https://news.itsfoss.com/ubuntu-20-10-end-of-life/
[4] https://cran.r-project.org/bin/linux/ubuntu/
[5] https://cran.r-project.org/bin/linux/ubuntu/focal-cran40/
[6] https://cran.r-project.org/bin/linux/ubuntu/groovy-cran40/
[7] http://web.archive.org/web/20210618164400/https://cloud.r-project.org/bin/linux/ubuntu/

[github-actions]

https://issues.apache.org/jira/browse/ARROW-14840

[thisisnic]

@github-actions crossbow submit test-ubuntu-21.04-docs

[github-actions]

Revision: 8804c56

Submitted crossbow builds: ursacomputing/crossbow @ actions-1218

Task	Status
test-ubuntu-21.04-docs

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 518cb8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1219

Task	Status
test-ubuntu-20.04-docs

[thisisnic]

Alternatively, we could move to 21.04 after #11786 has merged

[jonkeane]

Whoa fantastic debugging and puzzle-piece-putting together here.

So, apt-key, which is used as part of the R bit of the docs build is now deprecated[1] and Ubuntu 20.10 (Groovy Gorilla) is no longer a supported release as of 22nd July 2021 [2] which can apparently cause issues subsequently trying to do stuff via apt[3].

This now makes so much sense now. I'm fine with waiting for the Clang fix and then bumping to 21.04 (or 21.10??). Also, we could add CLANG_TOOLS: 9 or CLANG_TOOLS: 11 to the task and bump now like we do for a few others

FWIW, https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1219-azure-test-ubuntu-20.04-docs looks unrelated to the R package stuff (though I suspect the R package issues masked when this was introduced):

+ sphinx-build -b html -j 2 /arrow/docs/source /build/docs
Running Sphinx v3.5.4
Extension error:
Could not import extension sphinx_tabs.tabs (exception: No module named 'sphinx_tabs')
2
Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=7.0.0.dev219 ubuntu-docs` exited with a non-zero exit code 2, see the process log above.

[thisisnic]

@github-actions crossbow submit test-ubuntu-20.04-docs

[github-actions]

Revision: 9ab33be

Submitted crossbow builds: ursacomputing/crossbow @ actions-1222

Task	Status
test-ubuntu-20.04-docs

[kou]

test-ubuntu-default-docs already uses Ubuntu 20.04.

[thisisnic]

Good point! Given that this additional job was introduced in #10430 and things have moved on since then, I've removed the extra job. Or should I add it back in and pin it to a different version? It'd be good to know your thoughts too @jonkeane as you were the author of the PR I mentioned above and I don't fully understand everything that's going on there.

[jonkeane]

IIRC, I introduced test-ubuntu-default-docs because we started being unable to build docs on 20.04 (and that inability wasn't noticed cause we were only testing 20.10 before this)

It looks like the 20.10 build was changed from a much older one in #9933.

I'm not sure what the purpose of having a default build and then a separate build on a non-default version is. The one reason I can think of is that we want the non-default version to be a test of a new(ish) ubuntu version, to check for issues there. If that is the case (and we want to maintain that), we probably should build on 21.04 or 21.10. But I'm also totally fine removing the non-default docs build if it's more hassle than is helpful to maintain it.

[thisisnic]

OK, sounds like it made sense at the time but no longer serves a specific purpose, so unless there's any other reason to keep it in, let's leave it out for now (and add it back in if necessary later).

[kou]

We chose Ubuntu 20.10 to reduce build time by using system gRPC: #9933 (review)

I'm OK with removing the docs job with non default UBUNTU version.

Could you also remove UBUNTU=20.10 from dev/release/post-09-docs.sh and ci/docker/ubuntu-20.10-cpp.dockerfile file?

[thisisnic]

Aha, thanks for clarifying that @kou! Just to check my understanding, did you mean remove the reference to 20.10 in dev/release/post-09-docs.sh but entirely remove the file ci/docker/ubuntu-20.10-cpp.dockerfile as no other tasks depend on this now?

[kou]

Yes.

[jonkeane]

Looks good to me

[ursabot]

Benchmark runs are scheduled for baseline = 5227f24 and contender = 736a69e. 736a69e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True