ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nealrichardson wants to merge 18 commits into from
Closed

ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor #4538

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
af1852e
Throw this at appveyor and see how far off we are
nealrichardson Jun 12, 2019
94c9a42
Debug with fewer jobs in the matrix
nealrichardson Jun 12, 2019
39236f3
Put mine first and fast-finish
nealrichardson Jun 12, 2019
8c3e898
nm we already fast finish
nealrichardson Jun 12, 2019
f7327ed
Find shell script up a level
nealrichardson Jun 12, 2019
d5e8f4e
Bootstrap within the package dir
nealrichardson Jun 12, 2019
1ba4839
let the debugging begin
nealrichardson Jun 12, 2019
b6f6c8d
No popd?
nealrichardson Jun 12, 2019
18803c3
cd this way
nealrichardson Jun 12, 2019
15c0342
Try building first, see where that goes
nealrichardson Jun 12, 2019
fb19911
Try this name hack
nealrichardson Jun 12, 2019
19b60df
Enable R package to install on windows even if rwinlib isn't found
nealrichardson Jun 13, 2019
f4cf5c3
Some appveyor.yml cleanup
nealrichardson Jun 13, 2019
567f041
Cleanups
nealrichardson Jun 13, 2019
53c3a20
Makevars.win is dynamically generated now, no need to tweak it in the…
nealrichardson Jun 13, 2019
328bc37
Merge remote-tracking branch 'upstream/master' into r-appveyor
nealrichardson Jun 13, 2019
84eef1b
Merge upstream/master
nealrichardson Jun 17, 2019
6d6a74e
PR feedback
nealrichardson Jun 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 55 additions & 1 deletion appveyor.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
# Skip commits not related to Python, C++, C#, Go, Ruby or Rust
# Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
files:
- appveyor.yml
- c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
- format/
- go/
- python/
- r/
- ruby/
- rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
GO111MODULE: on
GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
- JOB: "R without libarrow"
USE_CLCACHE: false

MSVC_DEFAULT_OPTIONS: ON
APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
- call ci\appveyor-build.bat

for:
-
matrix:
only:
- JOB: "R without libarrow"
init:
ps: |
$ErrorActionPreference = "Stop"
Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.
Import-Module '..\appveyor-tool.ps1'

before_build:
- cd r
- ps: Bootstrap
- cd %APPVEYOR_BUILD_FOLDER%

build_script:
# TODO: for the version of the R tests that builds, probably can't use this
# ci script because we need to build with the Rtools toolchain
# - call ci\appveyor-build.bat
- pushd r
- travis-tool.sh install_deps

test_script:
- travis-tool.sh run_tests

on_failure:
- travis-tool.sh dump_logs

environment:
NOT_CRAN: true

artifacts:
- path: 'r\*.Rcheck\**\*.log'
name: Logs

- path: 'r\*.Rcheck\**\*.out'
name: Logs

- path: 'r\*.Rcheck\**\*.fail'
name: Logs

- path: 'r\*.Rcheck\**\*.Rout'
name: Logs

- path: 'r\*_*.tar.gz'
name: Bits

- path: 'r\*_*.zip'
name: Bits

# Disable test discovery
test: off

Expand Down
7 changes: 7 additions & 0 deletions ci/appveyor-filter-changes.bat
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
echo ===
appveyor exit
)
) else if "%JOB:~,2%" == "R " (
if "%ARROW_CI_R_AFFECTED%" == "0" (
echo ===
echo === No C++ or R changes, exiting job
echo ===
appveyor exit
)
) else (
if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
echo ===
Expand Down
14 changes: 0 additions & 14 deletions dev/release/00-prepare-test.rb
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
"+Version: #{@release_version}"],
],
},
{
path: "r/src/Makevars.win",
hunks: [
["-VERSION = #{@previous_version}.9000",
"+VERSION = #{@release_version}"],
],
},
{
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
"+Version: #{@release_version}.9000"],
],
},
{
path: "r/src/Makevars.win",
hunks: [
["-VERSION = #{@release_version}",
"+VERSION = #{@release_version}.9000"],
],
},

@wesmwesmJun 17, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?

@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.

@nealrichardsonnealrichardsonJun 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.

{
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
hunks: [
Expand Down
8 changes: 0 additions & 8 deletions dev/release/00-prepare.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
git add DESCRIPTION
cd -

cd "${SOURCE_DIR}/../../r/src"
sed -i.bak -E -e \
"s/^VERSION = .+/VERSION = ${r_version}/" \
Makevars.win
rm -f Makevars.win.bak
git add Makevars.win
cd -

cd "${SOURCE_DIR}/../../ruby"
sed -i.bak -E -e \
"s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down
47 changes: 47 additions & 0 deletions r/configure.win
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
echo "*** Generating code with data-raw/codegen.R"
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then

@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.

@nealrichardsonnealrichardsonJun 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).

I can add some comments to explain.

@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.

"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."

@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.

@nealrichardsonnealrichardsonJun 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments

# We'll have to install without libarrow. Note to the user how to rectify.
echo "------------------------- NOTE ---------------------------"
echo "After installation, please run arrow::install_arrow() to install"
echo "required runtime libraries"
echo "---------------------------------------------------------"
else
# Set the right flags to point to and enable arrow/parquet
RWINLIB="../windows/arrow-${VERSION}"
PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0

@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.

@nealrichardsonnealrichardsonJun 17, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions r/src/Makevars.win
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions r/tools/winlibs.R
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
if(getRversion() < "3.3.0") setInternet2()
download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
dir.create("../windows", showWarnings = FALSE)
unzip("lib.zip", exdir = "../windows")
dir.create("windows", showWarnings = FALSE)
unzip("lib.zip", exdir = "windows")
unlink("lib.zip")
}
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor#4538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

nealrichardson
wants to merge
18
commits into

from

 Closed


ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
#4538
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
af1852e

Throw this at appveyor and see how far off we are


nealrichardson Jun 12, 2019
94c9a42

Debug with fewer jobs in the matrix


nealrichardson Jun 12, 2019
39236f3

Put mine first and fast-finish


nealrichardson Jun 12, 2019
8c3e898

nm we already fast finish


nealrichardson Jun 12, 2019
f7327ed

Find shell script up a level


nealrichardson Jun 12, 2019
d5e8f4e

Bootstrap within the package dir


nealrichardson Jun 12, 2019
1ba4839

let the debugging begin


nealrichardson Jun 12, 2019
b6f6c8d

No popd?


nealrichardson Jun 12, 2019
18803c3

cd this way


nealrichardson Jun 12, 2019
15c0342

Try building first, see where that goes


nealrichardson Jun 12, 2019
fb19911

Try this name hack


nealrichardson Jun 12, 2019
19b60df

Enable R package to install on windows even if rwinlib isn't found


nealrichardson Jun 13, 2019
f4cf5c3

Some appveyor.yml cleanup


nealrichardson Jun 13, 2019
567f041

Cleanups


nealrichardson Jun 13, 2019
53c3a20

Makevars.win is dynamically generated now, no need to tweak it in the…


nealrichardson Jun 13, 2019
328bc37

Merge remote-tracking branch 'upstream/master' into r-appveyor


nealrichardson Jun 13, 2019
84eef1b

Merge upstream/master


nealrichardson Jun 17, 2019
6d6a74e

PR feedback


nealrichardson Jun 17, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


56 changes: 55 additions & 1 deletion
appveyor.yml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
os: Visual Studio 2015

only_commits:
 # Skip commits not related to Python, C++, C#, Go, Ruby or Rust
 # Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
 files:
 - appveyor.yml
 - c_glib/
Expand All@@ -29,6 +29,7 @@ only_commits:
 - format/
 - go/
 - python/
 - r/
 - ruby/
 - rust/

Expand DownExpand Up@@ -102,6 +103,8 @@ environment:
 GO111MODULE: on
 GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
 PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
 - JOB: "R without libarrow"
 USE_CLCACHE: false

 MSVC_DEFAULT_OPTIONS: ON
 APPVEYOR_SAVE_CACHE_ON_ERROR: true
Expand All@@ -119,6 +122,57 @@ before_build:
build_script:
 - call ci\appveyor-build.bat

for:
-
 matrix:
 only:
 - JOB: "R without libarrow"
 init:
 ps: |
 $ErrorActionPreference = "Stop"
 Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Comment thread
fsaintjacques marked this conversation as resolved.

 Import-Module '..\appveyor-tool.ps1'

 before_build:
 - cd r
 - ps: Bootstrap
 - cd %APPVEYOR_BUILD_FOLDER%

 build_script:
 # TODO: for the version of the R tests that builds, probably can't use this
 # ci script because we need to build with the Rtools toolchain
 # - call ci\appveyor-build.bat
 - pushd r
 - travis-tool.sh install_deps

 test_script:
 - travis-tool.sh run_tests

 on_failure:
 - travis-tool.sh dump_logs

 environment:
 NOT_CRAN: true

 artifacts:
 - path: 'r\*.Rcheck\**\*.log'
 name: Logs

 - path: 'r\*.Rcheck\**\*.out'
 name: Logs

 - path: 'r\*.Rcheck\**\*.fail'
 name: Logs

 - path: 'r\*.Rcheck\**\*.Rout'
 name: Logs

 - path: 'r\*_*.tar.gz'
 name: Bits

 - path: 'r\*_*.zip'
 name: Bits

# Disable test discovery
test: off

Expand Down

7 changes: 7 additions & 0 deletions
ci/appveyor-filter-changes.bat

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
 echo ===
 appveyor exit
 )
) else if "%JOB:~,2%" == "R " (
 if "%ARROW_CI_R_AFFECTED%" == "0" (
 echo ===
 echo === No C++ or R changes, exiting job
 echo ===
 appveyor exit
 )
) else (
 if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
 echo ===
Expand Down

14 changes: 0 additions & 14 deletions
dev/release/00-prepare-test.rb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -130,13 +130,6 @@ def test_update_version_pre_tag
 "+Version: #{@release_version}"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@previous_version}.9000",
 "+VERSION = #{@release_version}"],
 ],
 },
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand DownExpand Up@@ -279,13 +272,6 @@ def test_update_version_post_tag
 "+Version: #{@release_version}.9000"],
 ],
 },
 {
 path: "r/src/Makevars.win",
 hunks: [
 ["-VERSION = #{@release_version}",
 "+VERSION = #{@release_version}.9000"],
 ],
 },
@wesmwesmJun 17, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 00-prepare.sh does not modify Makevars.win anymore, the generation of this file is now deferred at installation (@nealrichardson please confirm) via the added configure.win.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch makes Makevars.win be generated dynamically, so there's no longer a file that needs to be updated in the release scripts.
 {
 path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
 hunks: [
Expand Down

8 changes: 0 additions & 8 deletions
dev/release/00-prepare.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -101,14 +101,6 @@ update_versions() {
 git add DESCRIPTION
 cd -

 cd "${SOURCE_DIR}/../../r/src"
 sed -i.bak -E -e \
 "s/^VERSION = .+/VERSION = ${r_version}/" \
 Makevars.win
 rm -f Makevars.win.bak
 git add Makevars.win
 cd -

 cd "${SOURCE_DIR}/../../ruby"
 sed -i.bak -E -e \
 "s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \
Expand Down

47 changes: 47 additions & 0 deletions
r/configure.win

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# generate code
if [ "$ARROW_R_DEV" == "TRUE" ]; then
 echo "*** Generating code with data-raw/codegen.R"
 "${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
fi

VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
# Try to find/download a C++ Arrow binary
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
# If binary not found, script exits nonzero
if [ $? -ne 0 ]; then
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected? And should you exit immediately if this failure is encountered.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tools/winlibs.R will exit nonzero if https://github.com/apache/arrow/pull/4538/files/328bc376f9adb2717d1505f96d8d7452ee9d8212#diff-48db18073973d5ea65a7f0023f920145R22 fails to find and download a prebuilt binary. This triggers the fallback build-package-without-libarrow behavior (see #4471).
I can add some comments to explain.
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now I understand more clearly. I think this warrants a comment or clearer message, e.g.
"Failed to download and extract libarrow runtime and dependencies. The R arrow package will compile with shim methods. Please run arrow::install_arrow() to install required runtime libraries and recompile this package with proper support."
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate this message in half, one for users, and a comment for maintainers.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed some comments
 # We'll have to install without libarrow. Note to the user how to rectify.
 echo "------------------------- NOTE ---------------------------"
 echo "After installation, please run arrow::install_arrow() to install"
 echo "required runtime libraries"
 echo "---------------------------------------------------------"
else
 # Set the right flags to point to and enable arrow/parquet
 RWINLIB="../windows/arrow-${VERSION}"
 PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
 PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
fi

echo "*** Writing Makevars.win"
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
# Success
exit 0
@fsaintjacquesfsaintjacquesJun 17, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small bash nit, exit 0 does not need to be explicited since it's implicit that a script returns 0.
@nealrichardsonnealrichardsonJun 17, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copypasta from r/configure 🤷‍♂

38 changes: 0 additions & 38 deletions
r/src/Makevars.win

View file


Open in desktop


This file was deleted.


6 changes: 3 additions & 3 deletions
r/tools/winlibs.R

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,10 +17,10 @@

# Download static arrow from rwinlib
VERSION <- commandArgs(TRUE)
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
 if(getRversion() < "3.3.0") setInternet2()
 download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
 dir.create("../windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "../windows")
 dir.create("windows", showWarnings = FALSE)
 unzip("lib.zip", exdir = "windows")
 unlink("lib.zip")
}