-
Notifications
You must be signed in to change notification settings - Fork 219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial input for backend selection #2396
Initial input for backend selection #2396
Conversation
Co-authored-by: KulikovNikita <nikita.kulikov@intel.com>
Co-authored-by: KulikovNikita <nikita.kulikov@intel.com>
…amgrigoriev/daal into dev/agrigorev-backend-selection
Co-authored-by: KulikovNikita <nikita.kulikov@intel.com>
Co-authored-by: KulikovNikita <nikita.kulikov@intel.com>
…amgrigoriev/daal into dev/agrigorev-backend-selection
Co-authored-by: KulikovNikita <nikita.kulikov@intel.com>
This PR fails when
Command used:
|
COMPILER.mac.clang = clang++ -m64 -fgnu-runtime -stdlib=libc++ -mmacosx-version-min=10.15 -fwrapv \ | ||
-DDAAL_REF -Werror -Wreturn-type | ||
COMPILER.lnx.clang = clang++ -m64 \ | ||
-DDAAL_REF -Werror -Wreturn-type |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far the only difference between mkl and ref mk files is -DDAAL_REF definition - would it be worth putting it's definition to main makefile instead of copying mkl and ref makefiles? So this would reduce code duplication here
Even with future architectures march targets, they can be put within same files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem is that X86 is the simplest case where it is possible to use simple solutions. In general (having cross-compilation in mind), it is necessary to add the following:
- --target=...
- --gcc-toolchain=...
- --sysroot=...
- -march=...
- -mabi=...
- -mtune=...
I guess it is better to separate every such compiler settings in its own file
/intelci: run |
There are also formatting hits EOF ==> not exactly one newline before EOF |
Performance issues, this PR with |
It can probably be related to this lines of the code: int old_threads = openblas_get_num_threads();
openblas_set_num_threads(1);
...
openblas_set_num_threads(old_threads); I suspect that it changes the global state of OpenMP and for some reason forces it to run on a specific thread despite what TBB thread assigns it to. I suspect that we should be able to build it without affinity: https://github.com/xianyi/OpenBLAS/blob/develop/USAGE.md#how-could-i-disable-openblas-threading-affinity-on-runtime |
/intelci: run |
1 similar comment
/intelci: run |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just blocking merge as safety measure until win build warnings would be fixed.
So, on warning fixes |
/intelci: run |
1 similar comment
/intelci: run |
/intelci: run |
FPK update didn't help - duplication happens even without IPP lib |
Could you expose the warning? |
Is this that one: daal_vmlipp_core.lib(vml_dLog1p_L9HAynn.obj) : warning LNK4006: fpk_vml_dLog1pI_L9HAynn already defined in daal_mkl_thread.lib(vml_dLog1p_L9HAynn.obj)? I am wondering if another library contains duplicated symbols after IPP removal? |
daal_vmlipp_core.lib(mkl_support.obj) : warning LNK4006: fpk_serv_strcat2 already defined in daal_mkl_thread.lib(mkl_support.obj); second definition ignored It's same obj as it still contain vml/vsl part although ipp was removed. Now there is 2100 warning lines vs 2500 with previous FPK. All warnings are only happen for vml/vsl functions now. |
/intelci: run |
@amgrigoriev - thank you for fix. Warnings are gone now. Waiting for CI checks to finish |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks to everyone involved!
* Initial input for backend selection * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Changed default backend to 'mkl' * Buildable onedal_c * Added clang support * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Compiler fixes for icc, icx, vc plus clang-format * Update dev/make/cmplr.clang.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * adding support for selecting different math/rng/service backends in case of build with bazel * fixed build issue with kmeans serialization * Introducing backend_config param selection to build.sh * Create openblas.sh * Introduce CI build for BLAS backend * adding execute permission on openblas.sh * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Addressed part of the comments * Clang-format * Turned off hyperthreadig for ref config in order to use TBB default number of threads * Addressed more comments * Macro fixed (APPLE) * More changes in REF RNG * Update build.sh * Removed 'sed' from Makefile * Update openblas.sh * Removed config_template from BAZEL * Removed backend_config_header from BAZEL * Update openblas.sh * Replaced safe function not supported by GNU * Fixed bugs in ref backend for OpenBLAS build * Fixed bugs in ref backend for OpenBLAS build #2 * Reduced header file dependencies in REF backend; removed << operator for RNG (compiling issue) * Update ci.yml * Fixed export on symbols for OpenBLAS build * export.def handling in bazel * Update cpp/daal/src/externals/service_math_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Added libfgortran to REF build * removed config_template auxiliary func * Removed libgfortran for REF backend * Fixed BACKEND incdirs for oneAPI; addressed some comments * Removed unnecessary includes; fixed ifdef in _DECLAR_ files * fixed omitted ifdef in _DECLAR_ files * Clang-format * Clang-format fix * Added an option to move compression to exclude list for examples * Excluded compression examples for all configurations * revert some macros to __intel_compiler * replase some more macros * add NO_FORTRAN=1 to openblas.sh script * Update cpp/daal/src/externals/service_service_ref.h * Fixed missed fpk symbols for oneapi examples (REF backend) * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Fixed error message in oneapi * exclude failed examples with ref backend * exclude examples for oneapi/cpp ifaces * exclude mpi examples for ref backend * Apply suggestions from code review * AVX512_MIC cleanup * Fixed CI pipeline * Added more includes for ONEAPI * Update .ci/pipeline/ci.yml * Switch to core count for blas build * Update .ci/env/openblas.sh * Adding _MKL suffix for job name * Fixing daal4py job dependency * Attempt to fix warnings --------- Co-authored-by: Aleksey Grigorev <a.grigorev@yadro.com> Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> Co-authored-by: a.maklaev <a.maklae@yadro.com> Co-authored-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> Co-authored-by: Pavel Yakovlev <p.yakovlev@yadro.com>
@amgrigoriev, this PR reverted #2392. Are you sure there are no additional rebasing mistakes in PR? |
|
* Initial input for backend selection * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Changed default backend to 'mkl' * Buildable onedal_c * Added clang support * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Compiler fixes for icc, icx, vc plus clang-format * Update dev/make/cmplr.clang.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * adding support for selecting different math/rng/service backends in case of build with bazel * fixed build issue with kmeans serialization * Introducing backend_config param selection to build.sh * Create openblas.sh * Introduce CI build for BLAS backend * adding execute permission on openblas.sh * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Addressed part of the comments * Clang-format * Turned off hyperthreadig for ref config in order to use TBB default number of threads * Addressed more comments * Macro fixed (APPLE) * More changes in REF RNG * Update build.sh * Removed 'sed' from Makefile * Update openblas.sh * Removed config_template from BAZEL * Removed backend_config_header from BAZEL * Update openblas.sh * Replaced safe function not supported by GNU * Fixed bugs in ref backend for OpenBLAS build * Fixed bugs in ref backend for OpenBLAS build uxlfoundation#2 * Reduced header file dependencies in REF backend; removed << operator for RNG (compiling issue) * Update ci.yml * Fixed export on symbols for OpenBLAS build * export.def handling in bazel * Update cpp/daal/src/externals/service_math_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Added libfgortran to REF build * removed config_template auxiliary func * Removed libgfortran for REF backend * Fixed BACKEND incdirs for oneAPI; addressed some comments * Removed unnecessary includes; fixed ifdef in _DECLAR_ files * fixed omitted ifdef in _DECLAR_ files * Clang-format * Clang-format fix * Added an option to move compression to exclude list for examples * Excluded compression examples for all configurations * revert some macros to __intel_compiler * replase some more macros * add NO_FORTRAN=1 to openblas.sh script * Update cpp/daal/src/externals/service_service_ref.h * Fixed missed fpk symbols for oneapi examples (REF backend) * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Fixed error message in oneapi * exclude failed examples with ref backend * exclude examples for oneapi/cpp ifaces * exclude mpi examples for ref backend * Apply suggestions from code review * AVX512_MIC cleanup * Fixed CI pipeline * Added more includes for ONEAPI * Update .ci/pipeline/ci.yml * Switch to core count for blas build * Update .ci/env/openblas.sh * Adding _MKL suffix for job name * Fixing daal4py job dependency * Attempt to fix warnings --------- Co-authored-by: Aleksey Grigorev <a.grigorev@yadro.com> Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> Co-authored-by: a.maklaev <a.maklae@yadro.com> Co-authored-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> Co-authored-by: Pavel Yakovlev <p.yakovlev@yadro.com>
* Fix incorrect include in service_hash_table.h file * Add initial implementation of the base algorithm::Hyperparameter class * clang-format * Add Hyperparameter* _hpar field into base Algorithm and AlgorithmContainer classes * Implement initial version of Hyperparameters class for Covariance algorithm in DAAL * Extend covariance::BatchImpl (and thus covariance::Batch) interface with the Hyperparameter support * Remove outdated verionst of setArgument methods * Add ability to pass hyperparameters from DAAL API to kernels in Covariance algorithm * 1. Make the hyperparameter field of DAAL Algorithm class constant. 2. Implement passing hyperparameters from oneDAL to DAAL on CPU * Fix * Fix Java build * Move covariance::Hyperparameter class to covariance::internal namespace * Removal of compression functionality (#2462) * Temporary disabling gemv test (#2463) * AVX/SSSE3/AVX512_MIC ISA cleanup (#2465) * cleanup avx512_mic * cleanup ssse3 * cleanup avx * fixing typos * fixing / typos * fixing make typos * clang format * Updating ISA in install.md * replace AVX with AVX2 for intersect * Remove mention of compression in docs (#2467) * Init addition of online interfaces in Covariance (#2455) * Add the link to oneAPI samples browser (#2470) * infra: update binary version (#2468) * update binary version * more binary version updates * Initial input for backend selection (#2396) * Initial input for backend selection * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Changed default backend to 'mkl' * Buildable onedal_c * Added clang support * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Compiler fixes for icc, icx, vc plus clang-format * Update dev/make/cmplr.clang.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * adding support for selecting different math/rng/service backends in case of build with bazel * fixed build issue with kmeans serialization * Introducing backend_config param selection to build.sh * Create openblas.sh * Introduce CI build for BLAS backend * adding execute permission on openblas.sh * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Addressed part of the comments * Clang-format * Turned off hyperthreadig for ref config in order to use TBB default number of threads * Addressed more comments * Macro fixed (APPLE) * More changes in REF RNG * Update build.sh * Removed 'sed' from Makefile * Update openblas.sh * Removed config_template from BAZEL * Removed backend_config_header from BAZEL * Update openblas.sh * Replaced safe function not supported by GNU * Fixed bugs in ref backend for OpenBLAS build * Fixed bugs in ref backend for OpenBLAS build #2 * Reduced header file dependencies in REF backend; removed << operator for RNG (compiling issue) * Update ci.yml * Fixed export on symbols for OpenBLAS build * export.def handling in bazel * Update cpp/daal/src/externals/service_math_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Added libfgortran to REF build * removed config_template auxiliary func * Removed libgfortran for REF backend * Fixed BACKEND incdirs for oneAPI; addressed some comments * Removed unnecessary includes; fixed ifdef in _DECLAR_ files * fixed omitted ifdef in _DECLAR_ files * Clang-format * Clang-format fix * Added an option to move compression to exclude list for examples * Excluded compression examples for all configurations * revert some macros to __intel_compiler * replase some more macros * add NO_FORTRAN=1 to openblas.sh script * Update cpp/daal/src/externals/service_service_ref.h * Fixed missed fpk symbols for oneapi examples (REF backend) * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Fixed error message in oneapi * exclude failed examples with ref backend * exclude examples for oneapi/cpp ifaces * exclude mpi examples for ref backend * Apply suggestions from code review * AVX512_MIC cleanup * Fixed CI pipeline * Added more includes for ONEAPI * Update .ci/pipeline/ci.yml * Switch to core count for blas build * Update .ci/env/openblas.sh * Adding _MKL suffix for job name * Fixing daal4py job dependency * Attempt to fix warnings --------- Co-authored-by: Aleksey Grigorev <a.grigorev@yadro.com> Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> Co-authored-by: a.maklaev <a.maklae@yadro.com> Co-authored-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> Co-authored-by: Pavel Yakovlev <p.yakovlev@yadro.com> * resolve merge conflict while rebasing in Fix incorrect include in service_hash_table.h file * Add ability to pass hyperparameters from DAAL API to kernels in Covariance algorithm * minor * update examples * clang-format * minor --------- Co-authored-by: Alexander Andreev <alexander.andreev@intel.com> Co-authored-by: ethanglaser <42726565+ethanglaser@users.noreply.github.com> Co-authored-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Co-authored-by: Aleksandr Solovev <aleksandr.solovev@intel.com> Co-authored-by: Alexandra <alexandra.epanchinzeva@intel.com> Co-authored-by: amgrigoriev <majagalz@mail.ru> Co-authored-by: Aleksey Grigorev <a.grigorev@yadro.com> Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> Co-authored-by: a.maklaev <a.maklae@yadro.com> Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> Co-authored-by: Pavel Yakovlev <p.yakovlev@yadro.com> Co-authored-by: Md Shafiul Alam <md.shafiul.alam@intel.com> Co-authored-by: msa <111298646+md-shafiul-alam@users.noreply.github.com>
* Initial input for backend selection * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Changed default backend to 'mkl' * Buildable onedal_c * Added clang support * Update dev/make/cmplr.gnu.mkl.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Update dev/make/cmplr.gnu.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * Compiler fixes for icc, icx, vc plus clang-format * Update dev/make/cmplr.clang.ref.mk Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> * adding support for selecting different math/rng/service backends in case of build with bazel * fixed build issue with kmeans serialization * Introducing backend_config param selection to build.sh * Create openblas.sh * Introduce CI build for BLAS backend * adding execute permission on openblas.sh * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Update cpp/daal/src/externals/config_ref.h Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> * Addressed part of the comments * Clang-format * Turned off hyperthreadig for ref config in order to use TBB default number of threads * Addressed more comments * Macro fixed (APPLE) * More changes in REF RNG * Update build.sh * Removed 'sed' from Makefile * Update openblas.sh * Removed config_template from BAZEL * Removed backend_config_header from BAZEL * Update openblas.sh * Replaced safe function not supported by GNU * Fixed bugs in ref backend for OpenBLAS build * Fixed bugs in ref backend for OpenBLAS build uxlfoundation#2 * Reduced header file dependencies in REF backend; removed << operator for RNG (compiling issue) * Update ci.yml * Fixed export on symbols for OpenBLAS build * export.def handling in bazel * Update cpp/daal/src/externals/service_math_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Added libfgortran to REF build * removed config_template auxiliary func * Removed libgfortran for REF backend * Fixed BACKEND incdirs for oneAPI; addressed some comments * Removed unnecessary includes; fixed ifdef in _DECLAR_ files * fixed omitted ifdef in _DECLAR_ files * Clang-format * Clang-format fix * Added an option to move compression to exclude list for examples * Excluded compression examples for all configurations * revert some macros to __intel_compiler * replase some more macros * add NO_FORTRAN=1 to openblas.sh script * Update cpp/daal/src/externals/service_service_ref.h * Fixed missed fpk symbols for oneapi examples (REF backend) * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Update cpp/daal/src/externals/service_service_ref.h Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> * Fixed error message in oneapi * exclude failed examples with ref backend * exclude examples for oneapi/cpp ifaces * exclude mpi examples for ref backend * Apply suggestions from code review * AVX512_MIC cleanup * Fixed CI pipeline * Added more includes for ONEAPI * Update .ci/pipeline/ci.yml * Switch to core count for blas build * Update .ci/env/openblas.sh * Adding _MKL suffix for job name * Fixing daal4py job dependency * Attempt to fix warnings --------- Co-authored-by: Aleksey Grigorev <a.grigorev@yadro.com> Co-authored-by: KulikovNikita <nikita.kulikov@intel.com> Co-authored-by: a.maklaev <a.maklae@yadro.com> Co-authored-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Co-authored-by: Victoriya Fedotova <viktoria.nn@gmail.com> Co-authored-by: Pavel Yakovlev <p_yak_11@mail.ru> Co-authored-by: Pavel Yakovlev <p.yakovlev@yadro.com>
Description