Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Apple Accelerate #74

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

Conversation

mgates3
Copy link
Collaborator

@mgates3 mgates3 commented Dec 20, 2023

Add support for Apple's new Accelerate library. Unfortunately it takes a rather different approach to updating than any other library, requiring that we use Apple's headers to get the decorated prototypes.

@TeachRaccooon
Copy link

Hi @mgates3, just wanted to ask about whether you are still working on this PR.

@mgates3
Copy link
Collaborator Author

mgates3 commented Feb 12, 2024

Yes, but not a high priority for me right now. If you want to take it over, we can meet to discuss. If so, let's email.

@mgates3
Copy link
Collaborator Author

mgates3 commented Feb 17, 2024

@TeachRaccooon I rebased this onto the latest master. As discussed, it needs the CMake config. Possibly everything else is done, but I haven't looked at it in a couple months. Thanks for taking a look at it.

@rileyjmurray
Copy link

Hi @mgates3. Is there anything blocking merging this PR?

@mgates3
Copy link
Collaborator Author

mgates3 commented Sep 23, 2024

@rileyjmurray I updated the Python config and CMake. It's a little tricky detecting whether we have the new or old Accelerate — I think it has a reliable test now. Can you test to see if it works?

@rileyjmurray
Copy link

rileyjmurray commented Sep 26, 2024

@mgates3 the CMake route worked perfectly. I checked that I could hook it into RandBLAS just fine.

The configure.py route couldn't get anywhere when I was using my spack-managed compiler (LLVM 17, with OpenMP). It seemed to work when I used my system compiler (LLVM 15, no OpenMP), in the sense that make -j install ran without errors. I wasn't able to test if RandBLAS could use this, since that would require some kind of manual configuration.

Edit: here's the blas/defines.h file of the installation produced with configure.py. Looks good to me!

// Copyright (c) 2017-2023, University of Tennessee. All rights reserved.
// SPDX-License-Identifier: BSD-3-Clause
// This program is free software: you can redistribute it and/or modify it under
// the terms of the BSD 3-Clause license. See the accompanying LICENSE file.

#ifndef BLAS_DEFINES_H
#define BLAS_DEFINES_H

// auto-generated by: configure.py prefix=/Users/rjmurr/Documents/tempblas/blaspp/../install blas=Accelerate blas_int=int64
//  
// host: 
//
// Definitions for:
// CXX   = g++
//         
// LIBS  = -framework Accelerate
//         

#define BLAS_HAVE_ACCELERATE
#define ACCELERATE_NEW_LAPACK
#define BLAS_FORTRAN_ADD_
#define BLAS_ILP64
#define BLAS_COMPLEX_RETURN_ARGUMENT
#define BLAS_HAVE_CBLAS
#define BLAS_HAVE_LAPACK


#endif        //  #ifndef BLAS_DEFINES_H

@TeachRaccooon
Copy link

Update on my side:

  1. Using an M1 mac with CMake 3.27.9 and macOS 15.0, I run into errors of SYCL headers missing, unless the -dgpu_backend=none option is in use.
  2. After BLAS++ was installed successfully, I tried to install LAPACK++. At the configuration step, CMake failed to find LAPACK. Using just the -Duse_cmake_find_blas=yes did not help; adding -DBLA_VENDOR=Apple resulted in -Duse_cmake_find_blas=yes getting ignored and CMake finding a nonsense version of LAPACK: LAPACK version 4294967299.4294967298.4294967297 (429496729942949672984294967297). This naturally prevents LAPACK++ compilation.

@mgates3 is there any chance there exists an obvious solution to my issue?

@mgates3
Copy link
Collaborator Author

mgates3 commented Oct 26, 2024

@TeachRaccooon

  1. Does it find a GPU backend? Otherwise, it should act the same as gpu_backend=none. Logs would help.
  2. LAPACK++ hasn't been updated for Accelerate. I need to finish up these changes to BLAS++, then propagate to LAPACK++.

@TeachRaccooon
Copy link

TeachRaccooon commented Oct 28, 2024

@mgates3

  1. Yes, looks like it finds SYCL, but then fails to find the right headers. I'll paste the output log below, but regardless, the issue is resolvable gpu_backend=none.

maksim$ cmake -DCMAKE_BUILD_TYPE=Release -Dblas_int=int64 -DCMAKE_INSTALL_PREFIX=pwd/../../../install/blaspp-install -DCMAKE_BINARY_DIR=pwd -Dbuild_tests=OFF ../../randlibs/blaspp;
-- The CXX compiler identification is AppleClang 15.0.0.15000309
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using CMAKE_INSTALL_PREFIX = /Users/maksim/BALLISTIC_RNLA/install/blaspp-install

-- Looking for CUDA (gpu_backend = auto)
-- No CUDA support: CUDA not found

-- Looking for HIP/ROCm (gpu_backend = auto)
-- No HIP/ROCm support: ROCm not found

-- Looking for oneMKL-SYCL (gpu_backend = auto)
-- Performing Test FSYCL_SUPPORT
-- Performing Test FSYCL_SUPPORT - Success
-- Building oneMKL-SYCL device support

-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
-- Could NOT find OpenMP (missing: OpenMP_CXX_FOUND)
-- blaspp_id = 61259aa

-- Looking for BLAS libraries and options (blas = auto)
default (no library)
libs:
defs:
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 no (didn't link: routine not found)
-DBLAS_FORTRAN_LOWER -DBLAS_ILP64 no (didn't link: routine not found)
-DBLAS_FORTRAN_UPPER -DBLAS_ILP64 no (didn't link: routine not found)

Intel MKL ilp64, sequential, gfortran
libs: -lmkl_gf_ilp64 -lmkl_sequential -lmkl_core
defs:
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 no (library not found)

Intel MKL ilp64, sequential, ifort
libs: -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core
defs:
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 no (library not found)

IBM ESSL int64 (ilp64), sequential
libs: -lessl6464
defs:
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 no (library not found)

OpenBLAS
libs: -lopenblas
defs:
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 no (library not found)

Apple Accelerate (new)
libs: -framework Accelerate
defs: -DBLAS_HAVE_ACCELERATE -DACCELERATE_NEW_LAPACK
-DBLAS_FORTRAN_ADD_ -DBLAS_ILP64 yes
Found BLAS library: -framework Accelerate

-- Checking BLAS library version
Accelerate framework
-- Checking BLAS complex return type
BLAS (zdotc) returns complex as hidden argument (Intel ifort convention)
-- Checking BLAS float return type
BLAS (sdot) returns float as float (standard)
-- Checking for CBLAS library
Found CBLAS library

-- Looking for LAPACK libraries and options (lapack = auto)
In BLAS library yes
Found LAPACK library in BLAS library

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/maksim/BALLISTIC_RNLA/build/blaspp-build
rileys-mbp-2:blaspp-build maksim$ make -j20
[ 4%] Building CXX object CMakeFiles/blaspp.dir/src/batch_hemm.cc.o
[ 4%] Building CXX object CMakeFiles/blaspp.dir/src/axpy.cc.o
[ 4%] Building CXX object CMakeFiles/blaspp.dir/src/asum.cc.o
[ 6%] Building CXX object CMakeFiles/blaspp.dir/src/batch_her2k.cc.o
[ 8%] Building CXX object CMakeFiles/blaspp.dir/src/batch_gemm.cc.o
[ 8%] Building CXX object CMakeFiles/blaspp.dir/src/batch_herk.cc.o
[ 9%] Building CXX object CMakeFiles/blaspp.dir/src/batch_trsm.cc.o
[ 10%] Building CXX object CMakeFiles/blaspp.dir/src/batch_syrk.cc.o
[ 16%] Building CXX object CMakeFiles/blaspp.dir/src/copy.cc.o
[ 16%] Building CXX object CMakeFiles/blaspp.dir/src/batch_syr2k.cc.o
[ 16%] Building CXX object CMakeFiles/blaspp.dir/src/dot.cc.o
[ 16%] Building CXX object CMakeFiles/blaspp.dir/src/batch_symm.cc.o
[ 17%] Building CXX object CMakeFiles/blaspp.dir/src/batch_trmm.cc.o
[ 18%] Building CXX object CMakeFiles/blaspp.dir/src/ger.cc.o
[ 20%] Building CXX object CMakeFiles/blaspp.dir/src/gemm.cc.o
[ 21%] Building CXX object CMakeFiles/blaspp.dir/src/gemv.cc.o
[ 22%] Building CXX object CMakeFiles/blaspp.dir/src/hemm.cc.o
[ 24%] Building CXX object CMakeFiles/blaspp.dir/src/her2.cc.o
[ 27%] Building CXX object CMakeFiles/blaspp.dir/src/her.cc.o
[ 27%] Building CXX object CMakeFiles/blaspp.dir/src/hemv.cc.o
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_syrk.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_syrk.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_trsm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_her2k.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
1 error generated.
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_herk.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_hemm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.hIn file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_trmm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~

// For CL version
^~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_herk.cc.o] Error 1
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_her2k.cc.o] Error 1
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_trsm.cc.o] Error 1
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_trmm.cc.o] Error 1
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_hemm.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_gemm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_gemm.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_syr2k.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_syr2k.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/batch_symm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/batch_symm.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/hemm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/copy.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/hemm.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/dot.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/blaspp.dir/src/copy.cc.o] Error 1
1 error generated.
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/ger.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/blaspp.dir/src/dot.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/gemm.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/ger.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/axpy.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/blaspp.dir/src/gemm.cc.o] Error 1
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/axpy.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/her2.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/asum.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/blaspp.dir/src/her2.cc.o] Error 1
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/asum.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/her.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/hemv.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/her.cc.o] Error 1
make[2]: *** [CMakeFiles/blaspp.dir/src/hemv.cc.o] Error 1
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/src/gemv.cc:7:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas.hh:74:
In file included from /Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device_blas.hh:6:
/Users/maksim/BALLISTIC_RNLA/randlibs/blaspp/include/blas/device.hh:45:14: fatal error: 'sycl/detail/cl.h' file not found
#include <sycl/detail/cl.h> // For CL version
^~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/blaspp.dir/src/gemv.cc.o] Error 1
make[1]: *** [CMakeFiles/blaspp.dir/all] Error 2
make: *** [all] Error 2

  1. Understood regarding LAPACK++. So then, do we just need to test configure.py to finalize the changes? I personally had no issues with it, so all looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants