Repository navigation
How to build cv2 in debug non-optimized mode? #424
Description
Activity
Alright I came up with an extremely hacky solution that does work. Someone more used to
cmakewould probably laugh at this solution, but it does work (and is faster than my trying to fight a build system I don't understand). I'll leave it here for the record so that I can move on, but I'm sure someone here can pop in and explain the right way to do this.I ran the build as follows choosing to keep all temporary build files:
build.shexport ENABLE_HEADLESS=1 export VERBOSE=1 mkdir build TMPDIR=$PWD/build pip wheel --no-clean --verbose .Then I swallowed up the output into
build.log:$ bash my_build.sh 2>&1 | tee build.logThen I extracted and transformed all commands involving the build:
hack_build.py#!/usr/bin/env python3 import sys for line in sys.stdin: if (line.startswith(' cd') or line.startswith(' /usr/bin/ar') or line.startswith(' /usr/bin/c++') or line.startswith(' /usr/bin/ranlib')): newline = line.strip(' ').replace('-O3', '-O0 -g') sys.stdout.write(newline)$ python3 hack_build.py < build.log > hacked_build.shFinally executing
$ bash hacked_build.shre-built the shared library without any optimizations. On my computer it produced the file:./build/pip-req-build-ro720y4z/_skbuild/linux-x86_64-3.9/cmake-build/lib/python3/cv2.cpython-39-x86_64-linux-gnu.soAfter copying that file into the directory
path/to/lib/python3.9/site-packages/cv2/I replaced the other version with mine and it works.Reacted by Jiacong FangI believe
scikit-build(CMake wrapper library) overwrites the-DCMAKE_BUILD_TYPE=Debugflag here: https://github.com/scikit-build/scikit-build/blob/c0741f9a1de9dd97e0a1e6f0b3b02fcd76221418/skbuild/setuptools_wrap.py#L105.Because of the
pip wheelcommand, the parameters toscikit-builtcannot be passed directly as command line parameters as documented here (due to this there is theCMAKE_ARGSvariable in this repository): https://scikit-build.readthedocs.io/en/latest/usage.html?highlight=debug#usage-scikit-build-optionsHowever, you could try running:
pip wheel . --verbose --build-option="--build-type=Debug"(or--global-option, I did not test either one)but then
pipbuilds the whole dependency chain from source which is slow and not needed: pypa/pip#2677pip wheel command docs: https://pip.pypa.io/en/stable/reference/pip_wheel/#customising-the-build
OpenCV build information: https://docs.opencv.org/master/db/d05/tutorial_config_reference.html
Reacted by Jiacong FangThanks for the help, but when I tried
pip wheel . --verbose --build-option="--build-type=Debug"I got the following error:37361 Building wheels for collected packages: opencv-python-headless, numpy 37362 Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-nljd2_ld 37363 ERROR: Cannot build wheel for opencv-python-headless using PEP 517 when --build-option is present 37364 Created temporary directory: /home/user/src/opencv-python/build/pip-wheel-mj3bd_7i 37365 ERROR: Cannot build wheel for numpy using PEP 517 when --build-option is present 37366 Failed to build opencv-python-headless numpy 37367 ERROR: Failed to build one or more wheels 37368 Exception information: 37369 Traceback (most recent call last): 37370 File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 210, in _main 37371 status = self.run(options, args) 37372 File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper 37373 return func(self, options, args) 37374 File "/home/user/folder/repos/vision/sandbox/lib/python3.9/site-packages/pip/_internal/commands/wheel.py", line 184, in run 37375 raise CommandError( 37376 pip._internal.exceptions.CommandError: Failed to build one or more wheelsGiven that I only want to do this for investigative/understanding work, my hack seems like the easiest approach for the moment since it already works. I'm guessing there aren't that many people who find themselves in my boat.
But thanks for the response!
There's one additional way of doing this via the
python setup.py bdist_wheelcommand. I tested this on Windows and it works:python setup.py bdist_wheel --build-type=DebugPlease note that you must install at least
scikit-buildandnumpybefore that command. Might need also something else, since the tooling does not get installed automatically viapyproject.toml. Due to that, I recommend running the build in a virtual environment which has all needed dependencies installed.Reacted by Kris Stern and Jiacong FangI just checked and this worked for me on Linux too. Might be worth adding something in the main readme about this (which I can try to remember to do), but other than that seems good. Thanks!
Reacted by Jiacong FangReacted by Jiacong FangC:\temp\opencv-python>python setup.py bdist_wheel --build-type=Debug
C:\Program Files\Python39\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
[1/2] Linking CXX shared module lib\python3\cv2.cp39-win_amd64.pyd
FAILED: lib/python3/cv2.cp39-win_amd64.pyd
cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=modules\python3\CMakeFiles\opencv_python3.dir --rc=C:\PROGRA2\WI3CF21\10\bin\1001831.0\x64\rc.exe --mt=C:\PROGRA2\WI3CF21\10\bin\1001831.0\x64\mt.exe --manifests -- C:\PROGRA2\MICROS2\2019\COMMUN1\VC\Tools\MSVC\14281.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG "C:\Program Files\Python39\libs\python39.lib" lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK Pass 1: command "C:\PROGRA2\MICROS2\2019\COMMUN1\VC\Tools\MSVC\14281.293\bin\Hostx64\x64\link.exe /nologo modules\python3\CMakeFiles\opencv_python3.dir_\src2\cv2.cpp.obj /out:lib\python3\cv2.cp39-win_amd64.pyd /implib:lib\opencv_python3.lib /pdb:lib\python3\cv2.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /NODEFAULTLIB:atlthunk.lib /NODEFAULTLIB:atlsd.lib /DEBUG C:\Program Files\Python39\libs\python39.lib lib\opencv_core451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_ml451d.lib lib\opencv_photo451d.lib lib\opencv_dnn451d.lib lib\opencv_features2d451d.lib lib\opencv_imgcodecs451d.lib lib\opencv_videoio451d.lib lib\opencv_calib3d451d.lib lib\opencv_highgui451d.lib lib\opencv_objdetect451d.lib lib\opencv_stitching451d.lib lib\opencv_video451d.lib lib\opencv_gapi451d.lib lib\opencv_videoio451d.lib lib\opencv_imgcodecs451d.lib 3rdparty\lib\libjpeg-turbod.lib 3rdparty\lib\libwebpd.lib 3rdparty\lib\libpngd.lib 3rdparty\lib\libtiffd.lib 3rdparty\lib\libopenjp2d.lib 3rdparty\lib\IlmImfd.lib comctl32.lib gdi32.lib ole32.lib setupapi.lib 3rdparty\lib\quircd.lib lib\opencv_video451d.lib lib\opencv_dnn451d.lib 3rdparty\lib\libprotobufd.lib lib\opencv_calib3d451d.lib lib\opencv_features2d451d.lib lib\opencv_flann451d.lib lib\opencv_imgproc451d.lib lib\opencv_core451d.lib 3rdparty\lib\zlibd.lib 3rdparty\lib\ittnotifyd.lib 3rdparty\lib\ippiwd.lib 3rdparty\ippicv\ippicv_win\icv\lib\intel64\ippicvmt.lib lib\ade.lib wsock32.lib ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:modules\python3\CMakeFiles\opencv_python3.dir/intermediate.manifest modules\python3\CMakeFiles\opencv_python3.dir/manifest.res" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'python39_d.lib'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\setuptools_wrap.py", line 589, in setup
cmkr.make(make_args, env=env)
File "C:\Users\jjosburn\AppData\Roaming\Python\Python39\site-packages\skbuild\cmaker.py", line 496, in make
raise SKBuildError(An error occurred while building with CMake.
Command:
cmake --build . --target install --config Debug --
Source directory:
C:\temp\opencv-python
Working directory:
C:\temp\opencv-python_skbuild\win-amd64-3.9\cmake-build
Please see CMake's output for more information.Having the same issue as @black13. I just need the debug output from https://github.com/opencv/opencv/blob/1363496c1106606684d40447f5d1149b2c66a9f8/modules/videoio/src/cap_dshow.cpp#L302 but looks like to get this I need to get a debug build of python including debug builds of opencv-python dependencies. Is this true?
You should really only need a debug build of the library you're interested in, but it can be a bit tricky. The method I used in my post should work for Linux, but if you're on Windows like black13, I can't really help you with the details.
@letmaik @black13 The error you are seeing is caused by the missing Python debug binaries. We can see clearly from the output that the linker is trying to link against the Python debug binary (note the _d):
LINK : fatal error LNK1104: cannot open file 'python39_d.lib'See for example this issue: opencv/opencv#7395
You must install the debug binaries. This can be done via the official Python installer as documented here (customize installation): https://docs.python.org/3/using/windows.html#installation-steps
Selecting “Customize installation” will allow you to select the features to install, the installation location and other options or post-install actions. To install debugging symbols or binaries, you will need to use this option.
Edit: You can run the installer also without UI if needed: https://docs.python.org/3/using/windows.html#installing-without-ui
Reacted by Alexander Alekhin@skvark Thanks for the pointer. I tried to do that here by adding
Include_debug=1: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38298354 But I'm getting the same error (see end of log).It seems_d.dllfiles are installed but no.libfiles.
EDIT: The.libfiles are correctly installed, e.g. python39_d.lib. I think the issue is in the CMake config of opencv somehow:-DPYTHON3_LIBRARY=C:/mypython/libs/python39.libthis should be the debug library. I'm not sure if this is really the root issue however.Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41
Scikit-build picks up the wrong path, you can change it here: https://github.com/opencv/opencv-python/blob/master/setup.py#L41
I hacked in the right path (upstream issue: scikit-build/scikit-build#533) and got a bit closer. It now fails during copying of the files from the CMake install folder:
Traceback (most recent call last): File "C:\projects\opencv-python\setup.py", line 458, in <module> main() File "C:\projects\opencv-python\setup.py", line 218, in main skbuild.setup( File "C:\mypython\lib\site-packages\skbuild\setuptools_wrap.py", line 621, in setup _classify_installed_files(cmake_manifest, package_data, package_prefixes, File "C:\projects\opencv-python\setup.py", line 371, in _classify_installed_files_override raise Exception("Not found: '%s'" % relpath_re) Exception: Not found: 'python/cv2[^/]*\.cp39\-win_amd64\.pyd'Here's the log: https://ci.appveyor.com/project/letmaik/opencv-python/builds/38432602
What's weird is that the "-- Installing:" line from CMake for the .pyd is missing, compared to a regular build like https://ci.appveyor.com/project/skvark/opencv-python/builds/36078624/job/vd96v0pmj7ai9bno?fullLog=true.
EDIT: Looks like it's related to this code: https://github.com/opencv/opencv/blob/fc1a15626226609babd128e043cf7c4e32f567ca/modules/python/common.cmake#L125-L129
For those who just want a "python39_d.lib" (DLL) what is the option? [No desire to actually BUILD Python]
Edit: I am working with latest commit on the opencv repo and with the tag 4.5.5
I installed a python 3.10 version with the corresponding python310_d.lib, and also attempted to create a python debug build, though it seemed to be a bigger hassle than it was worth. I couldn't get the installer debug libs to work even when opencv found the python310_d.lib, perhaps due to the problem described above with scikit?
Anyways, prior to this I did something like:
https://stackoverflow.com/questions/50422351/compile-and-link-opencv-to-my-project-in-my-main-cmakelists-txt
and it has worked using two sample programs, and my own so far, so I'll stick to that.
I keep opencv as a git submodule in my project tree as./third_party/opencv.In case the stackoverflow topic is lost, this is the minimum CMakeLists.txt for the Hough Transform example:
# Example using the hough transform sample code project(MyProject) add_subdirectory(third_party) # which then does add_subdirectory(opencv) add_executable(MyExe MyHoughTransform.cpp ) target_link_libraries(MyExe PRIVATE opencv_core opencv_imgcodecs opencv_highgui opencv_imgproc ) target_include_directories(MyExe PRIVATE third_party/opencv/modules/calib3d/include third_party/opencv/modules/core/include third_party/opencv/modules/cudaarithm/include third_party/opencv/modules/cudabgsegm/include third_party/opencv/modules/cudacodec/include third_party/opencv/modules/cudafeatures2d/include third_party/opencv/modules/cudafilters/include third_party/opencv/modules/cudaimgproc/include third_party/opencv/modules/cudalegacy/include third_party/opencv/modules/cudaobjdetect/include third_party/opencv/modules/cudaoptflow/include third_party/opencv/modules/cudastereo/include third_party/opencv/modules/cudawarping/include third_party/opencv/modules/cudev/include third_party/opencv/modules/dnn/include third_party/opencv/modules/features2d/include third_party/opencv/modules/flann/include third_party/opencv/modules/highgui/include third_party/opencv/modules/imgcodecs/include third_party/opencv/modules/imgproc/include third_party/opencv/modules/ml/include third_party/opencv/modules/objdetect/include third_party/opencv/modules/photo/include third_party/opencv/modules/shape/include third_party/opencv/modules/stitching/include third_party/opencv/modules/superres/include third_party/opencv/modules/ts/include third_party/opencv/modules/video/include third_party/opencv/modules/videoio/include third_party/opencv/modules/videostab/include third_party/opencv/modules/viz/include third_party/opencv/modules/world/include build )The only downside I see is that in addition to the target includes in above link, I have to include my
./buildfolder since./build/opencv2/opencv_modules.hppis referred to in the opencv code insidemodules/core/include/opencv2/core/base.hpp.I have no concrete idea why it really works, I get the same errors and warnings as when I set up for building opencv inside Visual Studio, but the fact that it does work in both
DebugandReleaseis more than enough for me at the moment.
I want to build the final cv2 shared library (e.g.
cv2.cpython-39-x86_64-linux-gnu.so) with debug symbols and without optimizations. I have tried various strategies like editing theopencv/CMakeLists.txtas well as running a build with something like this:No matter what I do, the final library seems to be built in Release mode and ignores my optimizations. How do I do this?