PolyFun can't find R dependencies

[bschilder]

I'm working on getting some additional functionality from PolyFun working in the echofinemap::POLYFUN wrapper.
The issue I'm running into is that rpy2 doesn't' seem to be able to find the required R packages, which are installed both on my desktop and within a dedicated conda env called "echoR_mini". In fact, I'm calling python from within the "echoR_mini" conda env.

Command

/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/bin/python /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py
   
    --compute-h2-L2
   
    --output-prefix /var/folders/zq/h7mtybc533b1qzkys_ttgpth0000gn/T//RtmpQlGIFj/PolyFun/output/dataset1/dataset1
   
    --sumstats /var/folders/zq/h7mtybc533b1qzkys_ttgpth0000gn/T//RtmpQlGIFj/PolyFun/file123db57c7004fsumstats.munged.parquet
   
    --ref-ld-chr /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/example_data/annotations.
   
    --w-ld-chr /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/example_data/weights.
   
    --num-bins 20
   
    --allow-missing

Output

********************************************************************
* PolyFun (POLYgenic FUNctionally-informed fine-mapping)
* Version 1.0.0
* (C) 2019-2021 Omer Weissbrod
*********************************************************************

[INFO]  Reading summary statistics from /var/folders/zq/h7mtybc533b1qzkys_ttgpth0000gn/T//RtmpQlGIFj/PolyFun/file123db57c7004fsumstats.munged.parquet ...
[INFO]  Read summary statistics for 182450 SNPs.
[INFO]  Reading reference panel LD Score from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/example_data/annotations.[1-22] ...
100%|██████████| 22/22 [00:00<00:00, 40.86it/s]
[INFO]  Read reference panel LD Scores for 182454 SNPs.
[INFO]  Reading regression weight LD Score from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/example_data/weights.[1-22] ...
100%|██████████| 22/22 [00:00<00:00, 43.36it/s]
[INFO]  Read regression weight LD Scores for 182454 SNPs.
[INFO]  After merging with reference panel LD, 182450 SNPs remain.
[INFO]  After merging with regression SNP LD, 182450 SNPs remain.
[WARNING]  number of SNPs is smaller than 200k; this is almost always bad.
[INFO]  Removed 6 SNPs with chi^2 > 383.29 (182444 SNPs remain)
[INFO]  iterating over chromosomes to compute XTX, XTy...
100%|██████████| 22/22 [00:00<00:00, 161.02it/s]
[INFO]  Evaluating Ridge lambdas...
100%|██████████| 100/100 [00:00<00:00, 201.49it/s]
[INFO]  Selected ridge lambda: 1.7475e-04 (43/100)  score: 8.7191e-02  score lstsq: 8.7190e-02
100%|██████████| 2/2 [00:00<00:00, 201.82it/s]
[INFO]  Estimating annotation coefficients for each chromosomes set
[INFO]  Computing per-SNP h^2 for each chromosome...
100%|██████████| 22/22 [00:00<00:00, 103.64it/s]
[INFO]  Saving constrained SNP variances to disk
100%|██████████| 22/22 [00:02<00:00, 10.57it/s]
[INFO]  Saving SNP variances to disk
100%|██████████| 22/22 [00:02<00:00, 10.81it/s]
During startup - Warning messages:
1: package "methods" in options("defaultPackages") was not found 
2: package ‘utils’ in options("defaultPackages") was not found 
3: package ‘grDevices’ in options("defaultPackages") was not found 
4: package ‘graphics’ in options("defaultPackages") was not found 
5: package ‘stats’ in options("defaultPackages") was not found 
6: package ‘methods’ in options("defaultPackages") was not found 
[INFO]  Clustering SNPs into bins using the R Ckmeans.1d.dp package
[INFO]  cffi mode is CFFI_MODE.ANY
[INFO]  R home found: /Library/Frameworks/R.framework/Resources
[DEBUG]  Looking for LD_LIBRARY_PATH with: /Library/Frameworks/R.framework/Resources/bin/Rscript -e cat(Sys.getenv("LD_LIBRARY_PATH"))
[INFO]  R library path: 
[INFO]  LD_LIBRARY_PATH: 
[DEBUG]  cffi mode is InterfaceType.API
[INFO]  Default options to initialize R: rpy2, --quiet, --no-save
[WARNING]  R[write to console]: Error in library.dynam(lib, package, package.lib) : 
  shared object ‘methods.dylib’ not found

[ERROR]  Could not load the R package Ckmeans.1d.dp. Either install it or rerun PolyFun with --skip-Ckmedian
[ERROR]  
Traceback (most recent call last):
  File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py", line 848, in <module>
    polyfun_obj.polyfun_main(args)
  File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py", line 772, in polyfun_main
    self.polyfun_h2_L2(args)
  File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py", line 605, in polyfun_h2_L2
    self.partition_snps_to_bins(args, use_ridge=True)
  File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py", line 516, in partition_snps_to_bins
    self.df_bins = self.partition_snps_Ckmedian(args, use_ridge=use_ridge)
  File "/Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py", line 432, in partition_snps_Ckmedian
    import rpy2.robjects.numpy2ri as numpy2ri
  File "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/python3.9/site-packages/rpy2/robjects/__init__.py", line 18, in <module>
    from rpy2.robjects.robject import RObjectMixin, RObject
  File "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/python3.9/site-packages/rpy2/robjects/robject.py", line 86, in <module>
    class RObjectMixin(abc.ABC):
  File "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/python3.9/site-packages/rpy2/robjects/robject.py", line 98, in RObjectMixin
    __show = _get_exported_value('methods', 'show')
  File "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/python3.9/site-packages/rpy2/rinterface_lib/conversion.py", line 45, in _
    cdata = function(*args, **kwargs)
  File "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/python3.9/site-packages/rpy2/rinterface.py", line 810, in __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error in library.dynam(lib, package, package.lib) : 
  shared object ‘methods.dylib’ not found

conda list

Details

# packages in environment at /Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini:
#
# Name                    Version                   Build  Channel
_r-mutex                  1.0.1               anacondar_1    conda-forge
arrow-cpp                 7.0.1           py39h04a14be_6_cpu    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
aws-c-cal                 0.5.11               hd2e2f4b_0    conda-forge
aws-c-common              0.6.2                h0d85af4_0    conda-forge
aws-c-event-stream        0.2.7               hb9330a7_13    conda-forge
aws-c-io                  0.10.5               h35aa462_0    conda-forge
aws-checksums             0.1.11               h0010a65_7    conda-forge
aws-sdk-cpp               1.8.186              h7c85d8e_4    conda-forge
axel                      2.17.11              h815e4d9_0    conda-forge
bitarray                  2.6.0            py39ha30fb19_1    conda-forge
bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
brotlipy                  0.7.0           py39ha30fb19_1005    conda-forge
bwidget                   1.9.14               h694c41f_1    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.18.1               h0d85af4_0    conda-forge
ca-certificates           2022.9.24            h033912b_0    conda-forge
cairo                     1.16.0            h904041c_1014    conda-forge
cctools_osx-64            973.0.1             h2b95895_10    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py39h131948b_2    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
clang                     14.0.4               h694c41f_0    conda-forge
clang-14                  14.0.4          default_h55ffa42_0    conda-forge
clang_osx-64              14.0.4               h3a95cd4_2    conda-forge
clangxx                   14.0.4          default_h55ffa42_0    conda-forge
clangxx_osx-64            14.0.4               he1dbc44_2    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
compiler-rt               14.0.4               h7fcd477_0    conda-forge
compiler-rt_osx-64        14.0.4               h6df654d_0    conda-forge
cramjam                   2.6.0            py39hd4bc93a_0    conda-forge
cryptography              38.0.2           py39h7eb6a14_2    conda-forge
curl                      7.86.0               h57eb407_0    conda-forge
cytoolz                   0.12.0           py39ha30fb19_1    conda-forge
dask                      2022.10.1          pyhd8ed1ab_0    conda-forge
dask-core                 2022.10.1          pyhd8ed1ab_0    conda-forge
deprecated                1.2.13             pyh6c4a22f_0    conda-forge
distributed               2022.10.1          pyhd8ed1ab_0    conda-forge
exceptiongroup            1.0.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hf0c8a7f_0    conda-forge
fastparquet               0.8.3            py39h7cc1f47_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               h5bb23bf_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               h3f81eb7_0    conda-forge
fribidi                   1.0.10               hbcb3906_0    conda-forge
fsspec                    2022.10.0          pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h8a4c099_0    conda-forge
gflags                    2.2.2             hb1e8313_1004    conda-forge
gfortran_impl_osx-64      11.3.0              h4c39eb8_25    conda-forge
gfortran_osx-64           11.3.0               h18f7dce_0    conda-forge
glog                      0.6.0                h8ac2a54_0    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
graphite2                 1.3.13            h2e338ed_1001    conda-forge
grpc-cpp                  1.47.1               h834a566_7    conda-forge
gsl                       2.7                  h93259b0_0    conda-forge
gzip                      1.12                 h5eb16cf_0    conda-forge
harfbuzz                  5.3.0                h08f8713_0    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
htslib                    1.16                 h567f53e_0    bioconda
icu                       70.1                 h96cf925_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
isl                       0.22.1               hb1e8313_2    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hac89ed1_2    conda-forge
krb5                      1.19.3               hb49756b_0    conda-forge
lcms2                     2.13.1               h90f4b2a_0    conda-forge
ld64_osx-64               609                 h1e06c2b_10    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libabseil                 20220623.0      cxx17_h844d122_4    conda-forge
libblas                   3.9.0           16_osx64_openblas    conda-forge
libbrotlicommon           1.0.9                hb7f2c08_8    conda-forge
libbrotlidec              1.0.9                hb7f2c08_8    conda-forge
libbrotlienc              1.0.9                hb7f2c08_8    conda-forge
libcblas                  3.9.0           16_osx64_openblas    conda-forge
libclang-cpp14            14.0.4          default_h55ffa42_0    conda-forge
libcrc32c                 1.1.2                he49afe7_0    conda-forge
libcurl                   7.86.0               h57eb407_0    conda-forge
libcxx                    14.0.6               hccf4f1f_0    conda-forge
libdeflate                1.13                 h775f41a_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libevent                  2.1.10               h815e4d9_4    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           10_4_0_h97931a8_25    conda-forge
libgfortran-devel_osx-64  11.3.0              h824d247_25    conda-forge
libgfortran5              11.3.0              h082f757_25    conda-forge
libglib                   2.74.1               h3ba3332_0    conda-forge
libgoogle-cloud           2.2.0                hb0fe3b0_1    conda-forge
libiconv                  1.17                 hac89ed1_0    conda-forge
libidn2                   2.3.4                hb7f2c08_0    conda-forge
liblapack                 3.9.0           16_osx64_openblas    conda-forge
libllvm14                 14.0.4               h41df66c_0    conda-forge
libnghttp2                1.47.0               h7cbc4dc_1    conda-forge
libopenblas               0.3.21          openmp_h429af6e_3    conda-forge
libpng                    1.6.38               ha978bb4_0    conda-forge
libprotobuf               3.21.9               hbc0c0cd_0    conda-forge
libsqlite                 3.39.4               ha978bb4_0    conda-forge
libssh2                   1.10.0               h7535e13_3    conda-forge
libthrift                 0.16.0               h08c06f4_2    conda-forge
libtiff                   4.4.0                h5e0c7b4_3    conda-forge
libunistring              0.9.10               h0d85af4_0    conda-forge
libutf8proc               2.7.0                h0d85af4_0    conda-forge
libwebp-base              1.2.4                h775f41a_0    conda-forge
libxcb                    1.13              h0d85af4_1004    conda-forge
libxml2                   2.10.3               hb9e07b5_0    conda-forge
libzlib                   1.2.13               hfd90126_4    conda-forge
llvm-openmp               14.0.4               ha654fa7_0    conda-forge
llvm-tools                14.0.4               h41df66c_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.0.2            py39hd0af75a_0    conda-forge
lz4-c                     1.9.3                he49afe7_1    conda-forge
make                      4.3                  h22f3db7_1    conda-forge
markupsafe                2.1.1            py39ha30fb19_2    conda-forge
mpc                       1.2.1                hbb51d92_0    conda-forge
mpfr                      4.1.0                h0f52abe_1    conda-forge
msgpack-python            1.0.4            py39h92daf61_1    conda-forge
ncurses                   6.3                  h96cf925_1    conda-forge
networkx                  2.8.7              pyhd8ed1ab_0    conda-forge
numpy                     1.23.4           py39hdfa1d0c_1    conda-forge
openjpeg                  2.5.0                h5d0d7b0_1    conda-forge
openssl                   1.1.1q               hfd90126_1    conda-forge
orc                       1.8.0                ha9d861c_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.5.1            py39hecff1ad_1    conda-forge
pandas-plink              2.2.9            py39h89e85a6_0    conda-forge
pango                     1.50.11              h7fca291_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
partd                     1.3.0              pyhd8ed1ab_0    conda-forge
pcre2                     10.37                h3f55489_1    conda-forge
pillow                    9.2.0            py39h35d4919_3    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               hbcb3906_0    conda-forge
pluggy                    1.0.0            py39h6e9494a_4    conda-forge
psutil                    5.9.3            py39ha30fb19_1    conda-forge
pthread-stubs             0.4               hc929b4f_1001    conda-forge
pyarrow                   7.0.1           py39hb941c77_6_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.2.0            py39h6e9494a_1    conda-forge
python                    3.9.13          h57e37ff_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2022.5             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.5             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0      py39h6e9494a_3    conda-forge
pyyaml                    6.0              py39ha30fb19_5    conda-forge
r-backports               1.4.1             r42h815d134_1    conda-forge
r-base                    4.2.1                he54549f_2    conda-forge
r-brio                    1.1.3             r42h815d134_1    conda-forge
r-callr                   3.7.2             r42hc72bb7e_1    conda-forge
r-ckmeans.1d.dp           4.3.4             r42h49197e3_1    conda-forge
r-cli                     3.4.1             r42h49197e3_1    conda-forge
r-colorspace              2.0_3             r42h815d134_1    conda-forge
r-crayon                  1.5.2             r42hc72bb7e_1    conda-forge
r-desc                    1.4.2             r42hc72bb7e_1    conda-forge
r-diffobj                 0.3.5             r42h815d134_1    conda-forge
r-digest                  0.6.30            r42h49197e3_0    conda-forge
r-ellipsis                0.3.2             r42h815d134_1    conda-forge
r-evaluate                0.17              r42hc72bb7e_1    conda-forge
r-expm                    0.999_6           r42h63324e3_1    conda-forge
r-fansi                   1.0.3             r42h815d134_1    conda-forge
r-farver                  2.1.1             r42h49197e3_1    conda-forge
r-fs                      1.5.2             r42h49197e3_2    conda-forge
r-gbrd                    0.4_11          r42hc72bb7e_1004    conda-forge
r-ggplot2                 3.3.6             r42hc72bb7e_1    conda-forge
r-glue                    1.6.2             r42h815d134_1    conda-forge
r-gtable                  0.3.1             r42hc72bb7e_1    conda-forge
r-irlba                   2.3.5.1           r42hce01bf1_0    conda-forge
r-isoband                 0.2.6             r42h49197e3_1    conda-forge
r-jsonlite                1.8.3             r42h815d134_0    conda-forge
r-labeling                0.4.2             r42hc72bb7e_2    conda-forge
r-lattice                 0.20_45           r42h815d134_1    conda-forge
r-lifecycle               1.0.3             r42hc72bb7e_1    conda-forge
r-magrittr                2.0.3             r42h815d134_1    conda-forge
r-mass                    7.3_58.1          r42h815d134_1    conda-forge
r-matrix                  1.5_1             r42hce01bf1_0    conda-forge
r-matrixstats             0.62.0            r42h815d134_1    conda-forge
r-mgcv                    1.8_41            r42h40f944a_0    conda-forge
r-mixsqp                  0.3_43            r42hf5e6a41_2    conda-forge
r-munsell                 0.5.0           r42hc72bb7e_1005    conda-forge
r-nlme                    3.1_160           r42h1e4e481_0    conda-forge
r-pillar                  1.8.1             r42hc72bb7e_1    conda-forge
r-pkgconfig               2.0.3             r42hc72bb7e_2    conda-forge
r-pkgload                 1.3.1             r42hc72bb7e_0    conda-forge
r-plyr                    1.8.7             r42h49197e3_1    conda-forge
r-praise                  1.0.0           r42hc72bb7e_1006    conda-forge
r-processx                3.8.0             r42h815d134_0    conda-forge
r-ps                      1.7.2             r42h815d134_0    conda-forge
r-r6                      2.5.1             r42hc72bb7e_1    conda-forge
r-rbibutils               2.2.9             r42h815d134_1    conda-forge
r-rcolorbrewer            1.1_3             r42h785f33e_1    conda-forge
r-rcpp                    1.0.9             r42h49197e3_2    conda-forge
r-rcpparmadillo           0.11.4.0.1        r42hf5e6a41_0    conda-forge
r-rdpack                  2.4               r42hc72bb7e_1    conda-forge
r-rematch2                2.1.2             r42hc72bb7e_2    conda-forge
r-reshape                 0.8.9             r42hbe3e9c8_1    conda-forge
r-rlang                   1.0.6             r42h49197e3_1    conda-forge
r-rprojroot               2.0.3             r42hc72bb7e_1    conda-forge
r-scales                  1.2.1             r42hc72bb7e_1    conda-forge
r-stringi                 1.7.8             r42h7183e51_1    conda-forge
r-susier                  0.12.27           r42hc72bb7e_1    conda-forge
r-testthat                3.1.5             r42h49197e3_1    conda-forge
r-tibble                  3.1.8             r42h815d134_1    conda-forge
r-utf8                    1.2.2             r42h815d134_1    conda-forge
r-vctrs                   0.5.0             r42h49197e3_0    conda-forge
r-viridislite             0.4.1             r42hc72bb7e_1    conda-forge
r-waldo                   0.4.0             r42hc72bb7e_1    conda-forge
r-withr                   2.5.0             r42hc72bb7e_1    conda-forge
r-xml2                    1.3.3             r42h3576887_2    conda-forge
re2                       2022.06.01           hb486fe8_0    conda-forge
readline                  8.1.2                h3899abd_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
rpy2                      3.5.1           py39r42h7cc1f47_1    conda-forge
scikit-learn              1.1.3            py39hdbdcc14_1    conda-forge
scipy                     1.9.3            py39h8a15683_1    conda-forge
setuptools                65.5.0             pyhd8ed1ab_0    conda-forge
sigtool                   0.1.3                h57ddcff_0    conda-forge
simplegeneric             0.8.1                      py_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                h6e38e02_1    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.39.4               h9ae0607_0    conda-forge
tabix                     1.11                 hdfd78af_0    bioconda
tapi                      1100.0.11            h9ce4665_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h5dbffcc_0    conda-forge
tktable                   2.10                 h49f0cf7_3    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39h63b48b0_3    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022e                h191b570_0    conda-forge
tzlocal                   4.2              py39h6e9494a_2    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wget                      1.20.3               h52ee1ee_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py39ha30fb19_1    conda-forge
xarray                    2022.10.0          pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.9                h35c211d_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hfd90126_4    conda-forge
zstandard                 0.18.0           py39ha30fb19_1    conda-forge
zstd                      1.5.2                hfa58983_4    conda-forge

rpy2 version

rpy2                      3.5.1           py39r42h7cc1f47_1    conda-forge

Conda env yaml

You can use the following yaml to create the env I'm using:
echoR_mini.yml.txt

[bschilder]

I think a clue is at this part:

[INFO]  R home found: /Library/Frameworks/R.framework/Resources
[DEBUG]  Looking for LD_LIBRARY_PATH with: /Library/Frameworks/R.framework/Resources/bin/Rscript -e cat(Sys.getenv("LD_LIBRARY_PATH"))
[INFO]  R library path: 
[INFO]  LD_LIBRARY_PATH: 

If I can tell rpy2 to look for R or my R library at a specific location, then maybe it will find the packages.

From within Rstudio (outside conda env):

> .libPaths()
[1] "/Library/Frameworks/R.framework/Versions/4.2/Resources/library"

From within conda env:

> .libPaths()
[1] "/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/R/library"

[bschilder]

@omerwe let me know if you have any ideas about this, or if it's something you've encountered before. Thanks!

[bschilder]

@jdblischak I think your input here would also be super valuable if you wouldn't mind! Thanks!

[bschilder]

via command line (within conda env)

From command line, with the conda env activated, this command works fine and is able to load.

import rpy2
import rpy2.robjects.numpy2ri as numpy2ri
from rpy2.robjects.packages import importr
importr('Ckmeans.1d.dp')
median_seg_func = ro.r('Ckmedian.1d.dp')
mean_seg_func = ro.r('Ckmeans.1d.dp')

via Rstudio python interpreter (outside conda env)

However, when I run the same command through a system() wrapper of the polyfun.py python script, I get:

import rpy2
import rpy2.robjects.numpy2ri as numpy2ri

R[write to console]: Error in .Primitive("as.environment")("package:utils") : 
  no item called "package:utils" on the search list

RRuntimeError: Error in .Primitive("as.environment")("package:utils") : 
  no item called "package:utils" on the search list

The above error message persists even when I explicitly set the R_HOME global:

import os
os.environ['R_HOME'] = '/Users/schilder/Library/Caches/org.Rproject.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/lib/R'

[jdblischak]

This isn't a PolyFun-specific issue. I run PolyFun inside of a conda env all the time. The official PolyFun conda env spec includes R and its dependencies, and Python/rpy2 thus have no trouble finding them.

run the same command through a system() wrapper of the polyfun.py python script

This is where I suspect the awkwardness is arising. Are you somehow activating the conda env in the system() call prior to running the Python script? The Python parts are working because you are pointing directly to the python executable, so python "knows" where to find its installed packages. However, it has no idea that it is inside a conda env, nor that R is available inside that same conda env.

I think it's probably the same situation with the command-line example from the original post. You need to activate the conda env first.

# Replace this
/Users/schilder/Library/Caches/org.R-project.R/R/basilisk/1.9.11/echoconda/0.99.8/echoR_mini/bin/python /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py
# with
conda activate echoR_mini
python /Library/Frameworks/R.framework/Versions/4.2/Resources/library/echofinemap/tools/polyfun/polyfun.py

[bschilder]

Yes indeed @jdblischak, this is not specific to PolyFun. PolyFun as controlled by echofinemap via Rstudio is just an example of a scenario in which this occurs. The reason I'm doing it this way is bc my main focus is getting echofinemap (and all its dependencies) to fully work from within Rstudio.

Running PolyFun directly in the conda env from the terminal is no problem, it's interfacing with the dependencies inside the conda env from outside it that I'm having trouble with. This occurs with both the "polyfun" and "echoR_mini" (which is partially based on "polyfun") conda envs.

Activating the conda env via basilisk or reticulate doesn't seem to help finding the R packages within the env. This might be due to the disconnect between using either of these to activate the env from within R, and the system function I'm using to execute the python script. Or it might simply be that both basilisk and reticulate only handle making python packages available to R, not R packages that are also installed in the same env.

Despite all these constraints, I do think there should be a way to get polyfun (or any python script) to find R packages in a given coda env, even when it's being called from outside it (e.g. from Rstudio). Setting os.environ['R_HOME'] seems like it should work, but nevertheless gives the error I mentioned above.

[bschilder]

@LTLA do you happen to know if there's a way to activate a conda environment from within Rstudio such that the R packages within that conda env become available to rpy2 scripts?

I realize this phrasing is rather confusing.
If it helps, the control hierarchy would look something like this:

- Rstudio 
-> Basilisk/reticulate 
--> python 
---> rpy2 
----> R packages installed in conda env

[LTLA]

Not to my knowledge. basilisk.utils::activateEnvironment should set the appropriate environment variables inside the current R session, which should then be inherited by any R subprocess that is kicked off somewhere. I'm guessing that the activation process would try to set R_LIBS, R_LIBS_SITE or R_LIBS_USER; I would further speculate that these variables only have an effect during R start-up, so one would have to make sure that they are set before the subprocess starts.

[bschilder]

Thanks for the feedback @LTLA ! These are some helpful clues. I'll keep trying and keep everyone posted if I figure out a solution.

[jdblischak]

I put together a minimal example to see if I could figure this out. Unfortunately I couldn't get it to work. rpy2 always identifies the outer R process (in the original post, this would be R running in RStudio), instead of the intended R installed in a conda env (the env to run PolyFun).

# Outer env to run R
mamba create --yes -n outer r-base=4.2 r-reticulate

# Inner env that calls R from Python via rpy2 (simulate PolyFun)
mamba create --yes -n inner python rpy2 r-base=4.0 r-susier

# Start from outer R process
mamba activate outer

# Python script that uses rpy2 to find R
cat > rpy2-code.py << EOL
import sys
print(f"Python executable: {sys.executable}")
import rpy2.situation
print(f"R_HOME: {rpy2.situation.get_r_home()}")
EOL

# R script that uses reticulate to activate conda env
cat > run-code-in-conda-env.R << EOL
library("reticulate")
use_condaenv("inner")
reticulate::source_python("rpy2-code.py")
EOL

# reticulate finds the inner env python, but rpy2 still detects outer R
Rscript run-code-in-conda-env.R
## Python executable: ~/.conda/envs/inner/bin/python
## R_HOME: ~/.conda/envs/outer/lib/R