Skip to content

Commit 83c211c

Browse files
nealrichardsonfsaintjacques
authored andcommitted
ARROW-3759: [R][CI] Build and test (no libarrow) on Windows in Appveyor
This is very much a work-in-progress for running the R package tests on Appveyor. It's an attempt to bring in the "standard" R Appveyor build tooling (https://github.com/krlmlr/r-appveyor, Apache 2.0 licensed) into Arrow's configuration. I currently have the C++ library building and installing, and R and all R package dependencies installing, and R checks beginning, but they fail. This is because https://github.com/apache/arrow/blob/master/r/tools/winlibs.R is configured only to build with a released [sic] version of the arrow C++ binaries pre-built and hosted on rwinlib. Two immediate to-dos are: 1. get the package to install and check without libarrow at all (as #4471 now allows) 2. get the package to find the libarrow we built locally from source After that, I can reverse-engineer what appveyor does and document it for any Windows developers (https://issues.apache.org/jira/browse/ARROW-3758). There's obviously some mess I've made of appveyor.yml that I'll clean up before I'm done, and I'll do my best to document and refactor some of the non-obvious features of the setup that I've discovered (e.g. that `ci/appveyor-cpp-build-mingw.bat` also installs GLib and Ruby and runs their test suites too) while I'm at it so that the next person might be able to pick things up faster. cc @romainfrancois @javierluraschi Author: Neal Richardson <neal.p.richardson@gmail.com> Closes #4538 from nealrichardson/r-appveyor and squashes the following commits: 6d6a74e <Neal Richardson> PR feedback 84eef1b <Neal Richardson> Merge upstream/master 328bc37 <Neal Richardson> Merge remote-tracking branch 'upstream/master' into r-appveyor 53c3a20 <Neal Richardson> Makevars.win is dynamically generated now, no need to tweak it in the release script 567f041 <Neal Richardson> Cleanups f4cf5c3 <Neal Richardson> Some appveyor.yml cleanup 19b60df <Neal Richardson> Enable R package to install on windows even if rwinlib isn't found fb19911 <Neal Richardson> Try this name hack 15c0342 <Neal Richardson> Try building first, see where that goes 18803c3 <Neal Richardson> cd this way b6f6c8d <Neal Richardson> No popd? 1ba4839 <Neal Richardson> let the debugging begin d5e8f4e <Neal Richardson> Bootstrap within the package dir f7327ed <Neal Richardson> Find shell script up a level 8c3e898 <Neal Richardson> nm we already fast finish 39236f3 <Neal Richardson> Put mine first and fast-finish 94c9a42 <Neal Richardson> Debug with fewer jobs in the matrix af1852e <Neal Richardson> Throw this at appveyor and see how far off we are
1 parent 0631357 commit 83c211c

File tree

7 files changed

+112
-64
lines changed

7 files changed

+112
-64
lines changed

appveyor.yml

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
os: Visual Studio 2015
2020

2121
only_commits:
22-
# Skip commits not related to Python, C++, C#, Go, Ruby or Rust
22+
# Skip commits not related to Python, C++, C#, Go, R, Ruby, or Rust
2323
files:
2424
- appveyor.yml
2525
- c_glib/
@@ -29,6 +29,7 @@ only_commits:
2929
- format/
3030
- go/
3131
- python/
32+
- r/
3233
- ruby/
3334
- rust/
3435

@@ -102,6 +103,8 @@ environment:
102103
GO111MODULE: on
103104
GOTOOLDIR: '%GOROOT%\pkg\tool\windows_amd64'
104105
PATH: '%GOPATH%\bin;%GOROOT%\bin;%PATH%'
106+
- JOB: "R without libarrow"
107+
USE_CLCACHE: false
105108

106109
MSVC_DEFAULT_OPTIONS: ON
107110
APPVEYOR_SAVE_CACHE_ON_ERROR: true
@@ -119,6 +122,57 @@ before_build:
119122
build_script:
120123
- call ci\appveyor-build.bat
121124

125+
for:
126+
-
127+
matrix:
128+
only:
129+
- JOB: "R without libarrow"
130+
init:
131+
ps: |
132+
$ErrorActionPreference = "Stop"
133+
Invoke-WebRequest http://raw.githubusercontent.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
134+
Import-Module '..\appveyor-tool.ps1'
135+
136+
before_build:
137+
- cd r
138+
- ps: Bootstrap
139+
- cd %APPVEYOR_BUILD_FOLDER%
140+
141+
build_script:
142+
# TODO: for the version of the R tests that builds, probably can't use this
143+
# ci script because we need to build with the Rtools toolchain
144+
# - call ci\appveyor-build.bat
145+
- pushd r
146+
- travis-tool.sh install_deps
147+
148+
test_script:
149+
- travis-tool.sh run_tests
150+
151+
on_failure:
152+
- travis-tool.sh dump_logs
153+
154+
environment:
155+
NOT_CRAN: true
156+
157+
artifacts:
158+
- path: 'r\*.Rcheck\**\*.log'
159+
name: Logs
160+
161+
- path: 'r\*.Rcheck\**\*.out'
162+
name: Logs
163+
164+
- path: 'r\*.Rcheck\**\*.fail'
165+
name: Logs
166+
167+
- path: 'r\*.Rcheck\**\*.Rout'
168+
name: Logs
169+
170+
- path: 'r\*_*.tar.gz'
171+
name: Bits
172+
173+
- path: 'r\*_*.zip'
174+
name: Bits
175+
122176
# Disable test discovery
123177
test: off
124178

ci/appveyor-filter-changes.bat

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,13 @@ if "%JOB%" == "Rust" (
4343
echo ===
4444
appveyor exit
4545
)
46+
) else if "%JOB:~,2%" == "R " (
47+
if "%ARROW_CI_R_AFFECTED%" == "0" (
48+
echo ===
49+
echo === No C++ or R changes, exiting job
50+
echo ===
51+
appveyor exit
52+
)
4653
) else (
4754
if "%ARROW_CI_PYTHON_AFFECTED%" == "0" (
4855
echo ===

dev/release/00-prepare-test.rb

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -130,13 +130,6 @@ def test_update_version_pre_tag
130130
"+Version: #{@release_version}"],
131131
],
132132
},
133-
{
134-
path: "r/src/Makevars.win",
135-
hunks: [
136-
["-VERSION = #{@previous_version}.9000",
137-
"+VERSION = #{@release_version}"],
138-
],
139-
},
140133
{
141134
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
142135
hunks: [
@@ -279,13 +272,6 @@ def test_update_version_post_tag
279272
"+Version: #{@release_version}.9000"],
280273
],
281274
},
282-
{
283-
path: "r/src/Makevars.win",
284-
hunks: [
285-
["-VERSION = #{@release_version}",
286-
"+VERSION = #{@release_version}.9000"],
287-
],
288-
},
289275
{
290276
path: "ruby/red-arrow-cuda/lib/arrow-cuda/version.rb",
291277
hunks: [

dev/release/00-prepare.sh

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -101,14 +101,6 @@ update_versions() {
101101
git add DESCRIPTION
102102
cd -
103103

104-
cd "${SOURCE_DIR}/../../r/src"
105-
sed -i.bak -E -e \
106-
"s/^VERSION = .+/VERSION = ${r_version}/" \
107-
Makevars.win
108-
rm -f Makevars.win.bak
109-
git add Makevars.win
110-
cd -
111-
112104
cd "${SOURCE_DIR}/../../ruby"
113105
sed -i.bak -E -e \
114106
"s/^ VERSION = \".+\"/ VERSION = \"${version}\"/g" \

r/configure.win

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/bin/bash
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
21+
# generate code
22+
if [ "$ARROW_R_DEV" == "TRUE" ]; then
23+
echo "*** Generating code with data-raw/codegen.R"
24+
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" data-raw/codegen.R
25+
fi
26+
27+
VERSION=$(grep ^Version DESCRIPTION | sed s/Version:\ //)
28+
# Try to find/download a C++ Arrow binary
29+
"${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe" "tools/winlibs.R" $VERSION
30+
# If binary not found, script exits nonzero
31+
if [ $? -ne 0 ]; then
32+
# We'll have to install without libarrow. Note to the user how to rectify.
33+
echo "------------------------- NOTE ---------------------------"
34+
echo "After installation, please run arrow::install_arrow() to install"
35+
echo "required runtime libraries"
36+
echo "---------------------------------------------------------"
37+
else
38+
# Set the right flags to point to and enable arrow/parquet
39+
RWINLIB="../windows/arrow-${VERSION}"
40+
PKG_CFLAGS="-I${RWINLIB}/include -DARROW_STATIC -DPARQUET_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ARROW"
41+
PKG_LIBS="-L${RWINLIB}/lib"'$(subst gcc,,$(COMPILED_BY))$(R_ARCH) '"-L${RWINLIB}/lib"'$(R_ARCH) '"-lparquet -larrow -lthrift -lboost_regex-mt-s -ldouble-conversion -lz -lws2_32"
42+
fi
43+
44+
echo "*** Writing Makevars.win"
45+
sed -e "s|@cflags@|$PKG_CFLAGS|" -e "s|@libs@|$PKG_LIBS|" src/Makevars.in > src/Makevars.win
46+
# Success
47+
exit 0

r/src/Makevars.win

Lines changed: 0 additions & 38 deletions
This file was deleted.

r/tools/winlibs.R

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@
1717

1818
# Download static arrow from rwinlib
1919
VERSION <- commandArgs(TRUE)
20-
if(!file.exists(sprintf("../windows/arrow-%s/include/arrow/api.h", VERSION))){
20+
if(!file.exists(sprintf("windows/arrow-%s/include/arrow/api.h", VERSION))){
2121
if(getRversion() < "3.3.0") setInternet2()
2222
download.file(sprintf("https://github.com/rwinlib/arrow/archive/v%s.zip", VERSION), "lib.zip", quiet = TRUE)
23-
dir.create("../windows", showWarnings = FALSE)
24-
unzip("lib.zip", exdir = "../windows")
23+
dir.create("windows", showWarnings = FALSE)
24+
unzip("lib.zip", exdir = "windows")
2525
unlink("lib.zip")
2626
}

0 commit comments

Comments
 (0)