Skip to content

Conversation

@javierluraschi
Copy link
Contributor

Fix for https://issues.apache.org/jira/browse/ARROW-4995. winbuilder provides a test environment similar to the CRAN build infrastructure to validate packages before submitting to CRAN. This PR fixes the remaining warnings flagged by this sytem.

@codecov-io
Copy link

codecov-io commented Mar 22, 2019

Codecov Report

Merging #4011 into master will increase coverage by 0.02%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4011      +/-   ##
==========================================
+ Coverage   87.83%   87.85%   +0.02%     
==========================================
  Files         734      736       +2     
  Lines       90433    90595     +162     
  Branches     1252     1252              
==========================================
+ Hits        79431    79593     +162     
  Misses      10885    10885              
  Partials      117      117
Impacted Files Coverage Δ
cpp/src/arrow/adapters/orc/adapter.cc 73.27% <0%> (-2.65%) ⬇️
python/pyarrow/tests/test_parquet.py 95.93% <0%> (-0.45%) ⬇️
cpp/src/arrow/adapters/orc/adapter_util.cc 80.7% <0%> (ø)
cpp/src/arrow/adapters/orc/adapter-test.cc 97.61% <0%> (ø)
cpp/src/arrow/util/bpacking.h 99.81% <0%> (ø) ⬆️
js/src/ipc/reader.ts 88.13% <0%> (+0.06%) ⬆️
cpp/src/arrow/util/thread-pool-test.cc 98.59% <0%> (+0.93%) ⬆️
cpp/src/plasma/thirdparty/ae/ae.c 71.69% <0%> (+0.94%) ⬆️
js/src/schema.ts 89.79% <0%> (+1.02%) ⬆️
js/src/ipc/metadata/message.ts 94.26% <0%> (+1.27%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81c726f...e21f7b1. Read the comment docs.

@javierluraschi
Copy link
Contributor Author

@wesm mentioned in the dev mailing list he would want to see the R bindings build in Windows without rwinlibs, the previous commit adds support for setting ARROW_PATH while building in windows to configure against a local build of the Arrow binaries in Mingw. Windows dev documentation is meant to be added in confluence.

See: https://lists.apache.org/thread.html/2a3e265dfbeaebf88d3ecebeefa297cb550249ebf515a8b0c6cf6149@%3Cdev.arrow.apache.org%3E

@javierluraschi
Copy link
Contributor Author

Investigated further the print.nteger64 winbuilder error, this code was introduced due to https://issues.apache.org/jira/browse/ARROW-3647, I've verified that the fix in this PR does not regress the original Jira issue. Here are the winbuilder results: https://win-builder.r-project.org/4pQ07Z4pxxTQ/00check.log

I also added an error for the function using compress under windows and scoped the skip() test to windows only.

Hopefully this wraps up this PR. Thanks!

Copy link
Member

@wesm wesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, with concerns noted

#' @param codec a codec
#' @export
CompressedOutputStream <- function(stream, codec = compression_codec("GZIP")){
if (.Platform$OS.type == "windows") stop("'CompressedOutputStream' is unsupported in Windows.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because of missing compression libraries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. We can probably solve this by figuring out which are the missing dependencies and adding support to compile them in Mingw, but I much rather suggest we disable this functionality in the 0.13 in Windows.

dir.create("../windows", showWarnings = FALSE)
unzip("lib.zip", exdir = "../windows")
unlink("lib.zip")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely scary-looking and appears to introduce a dependency on binaries in a foreign repository, which is an attack vector. For example, a malicious actor could inject Arrow libraries into this repository with malicious code in them that sends them files from your %HOME% directory (this is what already happened to the NPM community). I don't know who has access to this repository and if anyone who does has their keys stolen or their GitHub account compromised (maybe they don't have 2FA enabled) then we are in trouble

If we're going to have dependencies on libraries such as these, at minimum we should have a SHA256/SHA512 hash for what we expect their contents to be to prevent malicious code from getting shipped.

Of course, a preferable solution is to build libraries locally based on the source code in the archive

Can you open some follow up JIRA issues about this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesm wesm closed this in c43a7f2 Mar 26, 2019
@javierluraschi
Copy link
Contributor Author

@wesm can we reopen this PR? Not sure why it got closed, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants