Added 3d asynchronous functionality and example

doc/modules/3d_reconstruction_module.md

@@ -109,7 +109,7 @@ It should be similar to the following image.
 You can copy and modify the OpenPose 3-D demo to use any camera brand by:
 
 1. You can optionally turn off the `WITH_FLIR_CAMERA` while compiling CMake.
-2. Copy `examples/tutorial_api_cpp/13_synchronous_custom_input.cpp` (or `17_synchronous_custom_all_and_datum.cpp`).
+2. Copy `examples/tutorial_api_cpp/14_synchronous_custom_input.cpp` (or `18_synchronous_custom_all_and_datum.cpp`).
 3. Modify `WUserInput` and add your custom code there. Your code should fill `Datum::name`, `Datum::cameraMatrix`, `Datum::cvInputData`, and `Datum::cvOutputData` (fill cvOutputData = cvInputData).
 4. Remove `WUserPostProcessing` and `WUserOutput` (unless you want to have your custom post-processing and/or output).
 

doc/release_notes.md

@@ -385,19 +385,26 @@ OpenPose Library - Release Notes
 
 ## Current version - Future OpenPose 1.6.0
 1. Main improvements:
-    1. Created Matrix as container of cv::Mat, and String as container of std::string.
-    2. After replacing cv::Mat by Matrix, headers do not contain any 3rd-party library includes nor functions. This way, OpenPose can be exported without needing 3rd-party includes nor static library files (e.g., lib files in Windows), allowing people to use their own versions of OpenCV, Eigen, etc. without conflicting with OpenPose. Dynamic library files (e.g., `dll` files in Windows, `so` in Ubuntu) are still required.
-    3. Created the `openpose_private` directory with some internal headers that, if exported with OpenPose, would require including 3rd-party headers and static library files.
-    4. Default OpenCV version for Windows upgraded to version 4.2.0, extracted from their oficial website: section `Releases`, subsection `OpenCV - 4.2.0`, `Windows` version.
-    5. In all `*.cpp` files, their include of their analog `*.hpp` file has been moved to the first line of those `*.cpp` files to slightly speed up compiling time.
-    6. String is used in `include/openpose/wrapper/` to avoid std::string to cause errors for using diferent std DLLs.
-    7. Added `ScaleMode::ZeroToOneFixedAspect` and `ScaleMode::PlusMinusOneFixedAspect`. Compared to `ZeroToOne` and `PlusMinusOne`, the new ones also preserve the aspect ratio of each axis.
+    1. Multi-camera (3D) working on Asynchronous mode.
+        1. Functions `WrapperT::waitAndEmplace()` and `WrapperT::tryEmplace()` improved, allowing multi-camera/3-D (`TDatums` of size > 1).
+        2. Added `createMultiviewTDatum()` to auto-generate a `TDatums` for multi-camera/3-D from a single cv::Mat (that is splitted) and the desired camera parameter matrices.
+        3. Added `examples/tutorial_api_cpp/11_asynchronous_custom_input_multi_camera.cpp` for a test example.
+    2. Created Matrix as container of cv::Mat, and String as container of std::string.
+    3. After replacing cv::Mat by Matrix, headers do not contain any 3rd-party library includes nor functions. This way, OpenPose can be exported without needing 3rd-party includes nor static library files (e.g., lib files in Windows), allowing people to use their own versions of OpenCV, Eigen, etc. without conflicting with OpenPose. Dynamic library files (e.g., `dll` files in Windows, `so` in Ubuntu) are still required.
+    4. Created the `openpose_private` directory with some internal headers that, if exported with OpenPose, would require including 3rd-party headers and static library files.
+    5. Default OpenCV version for Windows upgraded to version 4.2.0, extracted from their oficial website: section `Releases`, subsection `OpenCV - 4.2.0`, `Windows` version.
+    6. In all `*.cpp` files, their include of their analog `*.hpp` file has been moved to the first line of those `*.cpp` files to slightly speed up compiling time.
+    7. String is used in `include/openpose/wrapper/` to avoid std::string to cause errors for using diferent std DLLs.
+    8. Added `ScaleMode::ZeroToOneFixedAspect` and `ScaleMode::PlusMinusOneFixedAspect`. Compared to `ZeroToOne` and `PlusMinusOne`, the new ones also preserve the aspect ratio of each axis.
 2. Functions or parameters renamed:
     1. All headers moved into `openpose_private`, all 3rd-party library calls in headers, and std::string calls in `include/openpose/wrapper/`.
     2. Renamed `dLog()` as `opLogIfDebug()`, `log()` as `opLog()`, `check()` as `checkBool()`, and also renamed all the `checkX()` functions in `include/openpose/utilities/check.hpp`. This avoids compiling crashes when exporting OpenPose to other projects which contain other 3rd-party libraries that define functions with the same popular names with `#define`.
 3. Main bugs fixed:
-    1. Removed many Visual Studio (Windows) warnings.
-    2. Natural sort now works properly with filenames containining numbers longer than the limit of an int.
+    1. Debug version of OpenPose actually targets debug lib/DLL files of 3rd-party libraries.
+    2. Debug version no longer prints on console a huge log message from Caffe with the network when starting OpenPose (fixed by using the right debug libraries).
+    3. Removed many Visual Studio (Windows) warnings.
+    4. Natural sort now works properly with filenames containining numbers longer than the limit of an int.
+    5. Optionally auto-generated bin folder only contains the required DLLs (depending on the CMake configuration), instead of all of them.
 4. Changes/additions that affect the compatibility with the OpenPose Unity Plugin:
 
 

examples/tutorial_add_module/1_custom_post_processing.cpp

@@ -20,7 +20,7 @@
 // 4. If extra classes and files are required, add those extra files inside the OpenPose include and src folders,
 // under a new folder (i.e., `include/newMethod/` and `src/newMethod/`), including `namespace op` on those files.
 
-// This example is a sub-case of `tutorial_api_cpp/15_synchronous_custom_postprocessing.cpp`, where only custom post-processing is
+// This example is a sub-case of `tutorial_api_cpp/16_synchronous_custom_postprocessing.cpp`, where only custom post-processing is
 // considered.
 
 // Command-line user interface

examples/tutorial_api_cpp/11_asynchronous_custom_input_multi_camera.cpp

@@ -0,0 +1,240 @@
+// ------------------------- OpenPose C++ API Tutorial - Example 10 - Custom Input -------------------------
+// Asynchronous mode: ideal for fast prototyping when performance is not an issue.
+// In this function, the user can implement its own way to create frames (e.g., reading his own folder of images)
+// and emplaces/pushes the frames to OpenPose.
+
+// Third-party dependencies
+#include <opencv2/opencv.hpp>
+// Command-line user interface
+#define OPENPOSE_FLAGS_DISABLE_PRODUCER
+#include <openpose/flags.hpp>
+// OpenPose dependencies
+#include <openpose/headers.hpp>
+
+// Custom OpenPose flags
+// Producer
+DEFINE_string(video,                "3d_4camera_video.avi",
+    "Use a video file instead of the camera. Use `examples/media/video.avi` for our default example video.");
+DEFINE_string(camera_parameter_path,    "models/cameraParameters/flir/",
+    "String with the folder where the camera parameters are located. If there is only 1 XML file (for single"
+    " video, webcam, or images from the same camera), you must specify the whole XML file path (ending in .xml).");
+
+// This worker will just read and return all the basic image file formats in a directory
+class UserInputClass
+{
+public:
+    UserInputClass(const std::string& videoPath, const std::string& cameraParameterPath) :
+        mClosed{false},
+        mFrameCounter{0ull},
+        mVideoCapture{videoPath}
+    {
+        if (!mVideoCapture.isOpened())
+        {
+            mClosed = true;
+            op::error("No video " + videoPath + " opened.", __LINE__, __FUNCTION__, __FILE__);
+        }
+
+        // Create CameraParameterReader
+        mCameraParameterReader.readParameters(cameraParameterPath);
+    }
+
+    std::shared_ptr<std::vector<std::shared_ptr<op::Datum>>> createDatum()
+    {
+        if (mClosed)
+        {
+            op::opLog("Video already closed, nullptr returned.", op::Priority::High);
+            return nullptr;
+        }
+
+        // Read cv::Mat
+        cv::Mat cvInputData;
+        mVideoCapture >> cvInputData;
+        // If empty frame -> return nullptr
+        if (cvInputData.empty())
+        {
+            // Close program when empty frame
+            op::opLog("Empty frame detected, closing program.", op::Priority::High);
+            mClosed = true;
+            return nullptr;
+        }
+
+        // Create new datum and add 3D information (cv::Mat splitted and camera parameters)
+        auto datumsPtr = std::make_shared<std::vector<std::shared_ptr<op::Datum>>>();
+        op::createMultiviewTDatum<op::Datum>(datumsPtr, mFrameCounter, mCameraParameterReader, (void*)&cvInputData);
+
+        return datumsPtr;
+    }
+
+    bool isFinished() const
+    {
+        return mClosed;
+    }
+
+private:
+    bool mClosed;
+    unsigned long long mFrameCounter;
+    cv::VideoCapture mVideoCapture;
+    op::CameraParameterReader mCameraParameterReader;
+};
+
+void configureWrapper(op::Wrapper& opWrapper)
+{
+    try
+    {
+        // Configuring OpenPose
+
+        // logging_level
+        op::checkBool(
+            0 <= FLAGS_logging_level && FLAGS_logging_level <= 255, "Wrong logging_level value.",
+            __LINE__, __FUNCTION__, __FILE__);
+        op::ConfigureLog::setPriorityThreshold((op::Priority)FLAGS_logging_level);
+        op::Profiler::setDefaultX(FLAGS_profile_speed);
+
+        // Applying user defined configuration - GFlags to program variables
+        // outputSize
+        const auto outputSize = op::flagsToPoint(op::String(FLAGS_output_resolution), "-1x-1");
+        // netInputSize
+        const auto netInputSize = op::flagsToPoint(op::String(FLAGS_net_resolution), "-1x368");
+        // faceNetInputSize
+        const auto faceNetInputSize = op::flagsToPoint(op::String(FLAGS_face_net_resolution), "368x368 (multiples of 16)");
+        // handNetInputSize
+        const auto handNetInputSize = op::flagsToPoint(op::String(FLAGS_hand_net_resolution), "368x368 (multiples of 16)");
+        // poseMode
+        const auto poseMode = op::flagsToPoseMode(FLAGS_body);
+        // poseModel
+        const auto poseModel = op::flagsToPoseModel(op::String(FLAGS_model_pose));
+        // JSON saving
+        if (!FLAGS_write_keypoint.empty())
+            op::opLog(
+                "Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json`"
+                " instead.", op::Priority::Max);
+        // keypointScaleMode
+        const auto keypointScaleMode = op::flagsToScaleMode(FLAGS_keypoint_scale);
+        // heatmaps to add
+        const auto heatMapTypes = op::flagsToHeatMaps(FLAGS_heatmaps_add_parts, FLAGS_heatmaps_add_bkg,
+                                                      FLAGS_heatmaps_add_PAFs);
+        const auto heatMapScaleMode = op::flagsToHeatMapScaleMode(FLAGS_heatmaps_scale);
+        // >1 camera view?
+        const auto multipleView = (FLAGS_3d || FLAGS_3d_views > 1);
+        // Face and hand detectors
+        const auto faceDetector = op::flagsToDetector(FLAGS_face_detector);
+        const auto handDetector = op::flagsToDetector(FLAGS_hand_detector);
+        // Enabling Google Logging
+        const bool enableGoogleLogging = true;
+
+        // Pose configuration (use WrapperStructPose{} for default and recommended configuration)
+        const op::WrapperStructPose wrapperStructPose{
+            poseMode, netInputSize, outputSize, keypointScaleMode, FLAGS_num_gpu, FLAGS_num_gpu_start,
+            FLAGS_scale_number, (float)FLAGS_scale_gap, op::flagsToRenderMode(FLAGS_render_pose, multipleView),
+            poseModel, !FLAGS_disable_blending, (float)FLAGS_alpha_pose, (float)FLAGS_alpha_heatmap,
+            FLAGS_part_to_show, op::String(FLAGS_model_folder), heatMapTypes, heatMapScaleMode, FLAGS_part_candidates,
+            (float)FLAGS_render_threshold, FLAGS_number_people_max, FLAGS_maximize_positives, FLAGS_fps_max,
+            op::String(FLAGS_prototxt_path), op::String(FLAGS_caffemodel_path),
+            (float)FLAGS_upsampling_ratio, enableGoogleLogging};
+        opWrapper.configure(wrapperStructPose);
+        // Face configuration (use op::WrapperStructFace{} to disable it)
+        const op::WrapperStructFace wrapperStructFace{
+            FLAGS_face, faceDetector, faceNetInputSize,
+            op::flagsToRenderMode(FLAGS_face_render, multipleView, FLAGS_render_pose),
+            (float)FLAGS_face_alpha_pose, (float)FLAGS_face_alpha_heatmap, (float)FLAGS_face_render_threshold};
+        opWrapper.configure(wrapperStructFace);
+        // Hand configuration (use op::WrapperStructHand{} to disable it)
+        const op::WrapperStructHand wrapperStructHand{
+            FLAGS_hand, handDetector, handNetInputSize, FLAGS_hand_scale_number, (float)FLAGS_hand_scale_range,
+            op::flagsToRenderMode(FLAGS_hand_render, multipleView, FLAGS_render_pose), (float)FLAGS_hand_alpha_pose,
+            (float)FLAGS_hand_alpha_heatmap, (float)FLAGS_hand_render_threshold};
+        opWrapper.configure(wrapperStructHand);
+        // Extra functionality configuration (use op::WrapperStructExtra{} to disable it)
+        const op::WrapperStructExtra wrapperStructExtra{
+            FLAGS_3d, FLAGS_3d_min_views, FLAGS_identification, FLAGS_tracking, FLAGS_ik_threads};
+        opWrapper.configure(wrapperStructExtra);
+        // Output (comment or use default argument to disable any output)
+        const op::WrapperStructOutput wrapperStructOutput{
+            FLAGS_cli_verbose, op::String(FLAGS_write_keypoint), op::stringToDataFormat(FLAGS_write_keypoint_format),
+            op::String(FLAGS_write_json), op::String(FLAGS_write_coco_json), FLAGS_write_coco_json_variants,
+            FLAGS_write_coco_json_variant, op::String(FLAGS_write_images), op::String(FLAGS_write_images_format),
+            op::String(FLAGS_write_video), FLAGS_write_video_fps, FLAGS_write_video_with_audio,
+            op::String(FLAGS_write_heatmaps), op::String(FLAGS_write_heatmaps_format), op::String(FLAGS_write_video_3d),
+            op::String(FLAGS_write_video_adam), op::String(FLAGS_write_bvh), op::String(FLAGS_udp_host),
+            op::String(FLAGS_udp_port)};
+        opWrapper.configure(wrapperStructOutput);
+        // GUI (comment or use default argument to disable any visual output)
+        const op::WrapperStructGui wrapperStructGui{
+            op::flagsToDisplayMode(FLAGS_display, FLAGS_3d), !FLAGS_no_gui_verbose, FLAGS_fullscreen};
+        opWrapper.configure(wrapperStructGui);
+        // Set to single-thread (for sequential processing and/or debugging and/or reducing latency)
+        if (FLAGS_disable_multi_thread)
+            opWrapper.disableMultiThreading();
+    }
+    catch (const std::exception& e)
+    {
+        op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
+    }
+}
+
+int tutorialApiCpp()
+{
+    try
+    {
+        op::opLog("Starting OpenPose demo...", op::Priority::High);
+        const auto opTimer = op::getTimerInit();
+
+        // Required flags to enable 3-D
+        FLAGS_3d = true;
+        FLAGS_number_people_max = 1;
+        FLAGS_3d_min_views = 3;
+        FLAGS_output_resolution = "320x256"; // Optional, but otherwise it gets too big to render in real time
+        // FLAGS_3d_views = X; // Not required because it only affects OpenPose producers (rather than custom ones)
+
+        // Configuring OpenPose
+        op::opLog("Configuring OpenPose...", op::Priority::High);
+        op::Wrapper opWrapper{op::ThreadManagerMode::AsynchronousIn};
+        configureWrapper(opWrapper);
+
+        // Start, run, and stop processing - exec() blocks this thread until OpenPose wrapper has finished
+        op::opLog("Starting thread(s)...", op::Priority::High);
+        opWrapper.start();
+
+        // User processing
+        UserInputClass userInputClass(FLAGS_video, FLAGS_camera_parameter_path);
+        bool userWantsToExit = false;
+        while (!userWantsToExit && !userInputClass.isFinished())
+        {
+            if (!opWrapper.isRunning())
+            {
+                op::opLog("OpenPose wrapper is no longer running, exiting video.", op::Priority::High);
+                break;
+            }
+            // Push frame
+            auto datumToProcess = userInputClass.createDatum();
+            if (datumToProcess != nullptr)
+            {
+                auto successfullyEmplaced = opWrapper.waitAndEmplace(datumToProcess);
+                if (!successfullyEmplaced)
+                    op::opLog("Processed datum could not be emplaced.", op::Priority::High);
+            }
+        }
+
+        op::opLog("Stopping thread(s)", op::Priority::High);
+        opWrapper.stop();
+
+        // Measuring total time
+        op::printTime(opTimer, "OpenPose demo successfully finished. Total time: ", " seconds.", op::Priority::High);
+
+        // Return
+        return 0;
+    }
+    catch (const std::exception&)
+    {
+        return -1;
+    }
+}
+
+int main(int argc, char *argv[])
+{
+    // Parsing command line flags
+    gflags::ParseCommandLineFlags(&argc, &argv, true);
+
+    // Running tutorialApiCpp
+    return tutorialApiCpp();
+}

examples/tutorial_api_cpp/CMakeLists.txt

@@ -9,13 +9,14 @@ set(EXAMPLE_FILES
     08_heatmaps_from_image.cpp
     09_keypoints_from_heatmaps.cpp
     10_asynchronous_custom_input.cpp
-    11_asynchronous_custom_output.cpp
-    12_asynchronous_custom_input_output_and_datum.cpp
-    13_synchronous_custom_input.cpp
-    14_synchronous_custom_preprocessing.cpp
-    15_synchronous_custom_postprocessing.cpp
-    16_synchronous_custom_output.cpp
-    17_synchronous_custom_all_and_datum.cpp)
+    11_asynchronous_custom_input_multi_camera.cpp
+    12_asynchronous_custom_output.cpp
+    13_asynchronous_custom_input_output_and_datum.cpp
+    14_synchronous_custom_input.cpp
+    15_synchronous_custom_preprocessing.cpp
+    16_synchronous_custom_postprocessing.cpp
+    17_synchronous_custom_output.cpp
+    18_synchronous_custom_all_and_datum.cpp)
 
 include(${CMAKE_SOURCE_DIR}/cmake/Utils.cmake)
 

include/openpose/core/matrix.hpp

@@ -95,6 +95,15 @@ namespace op
     class OP_API Matrix
     {
     public:
+        /**
+         * @param matrixesResized For 3-D OpenPose, if >1, it will assume the image is composed of
+         * numberImagesStackedHorizontally horizontally stacked images. It must be already resized to avoid
+         * internally allocating/removing elements of std::vector (to avoid errors if using different std DLLs)
+         * @param cvMatPtr should be a cv::Mat element or it will provoke a core dumped. Done to
+         * avoid explicitly exposing 3rdparty libraries on the headers.
+         */
+        static void splitCvMatIntoVectorMatrix(std::vector<Matrix>& matrixesResized, const void* const cvMatPtr);
+
         Matrix();
 
         /**

include/openpose/producer/wDatumProducer.hpp

@@ -81,7 +81,7 @@ namespace op
                 opLogIfDebug("", Priority::Low, __LINE__, __FUNCTION__, __FILE__);
             }
             // Equivalent to WQueueSplitter
-            // Queued elements - Multiple views --> Split views into different share pointers
+            // Queued elements - Multiple views --> Split views into different shared pointers
             if (tDatums != nullptr && tDatums->size() > 1)
             {
                 // Add tDatums to mQueuedElements