Added tracking id in JSON + 2x faster resize

doc/contributors.md

@@ -15,4 +15,5 @@ We would also like to thank the following people who have highly contributed to
 2. [Bikramjot Hanzra](https://www.linkedin.com/in/bikz05): Former OpenPose maintainer, CMake (Ubuntu and Windows) version, and initial Travis Build version for Ubuntu.
 3. [Donglai Xiang](https://xiangdonglai.github.io): Camera calibration toolbox improvement, including the implementation of its bundle adjustment algorithm.
 4. [Luis Fernando Fraga](https://github.com/fragalfernando): Implementation of Lukas-Kanade algorith and person ID extractor.
-5. [Helen Medina](https://github.com/helen-medina): Initial Windows version.
+5. [Akash Patwal](https://www.linkedin.com/in/akash-patwal-63a12012a): Speedup of the CUDA image resize and visual skeleton rendering, as well as extension that allows OpenPose to speedup linearly to more than 4 GPUs.
+6. [Helen Medina](https://github.com/helen-medina): Initial Windows version.

doc/release_notes.md

@@ -264,7 +264,9 @@ OpenPose Library - Release Notes
     2. Speed up of the CUDA functions of OpenPose:
         1. Greedy body part connector implemented in CUDA: +~30% speedup in Nvidia (CUDA) version with default flags and +~10% in maximum accuracy configuration. In addition, it provides a small 0.5% boost in accuracy (default flags).
         2. +5-30% additional speedup for the body part connector of point 1.
-        3. 2-4x speedup for NMS.
+        3. ~2-4x speedup for NMS.
+        4. ~2x speedup for image resize.
+        5. +25-30% speedup for rendering.
     3. Unity binding of OpenPose released. OpenPose adds the flag `BUILD_UNITY_SUPPORT` on CMake, which enables special Unity code so it can be built as a Unity plugin.
     4. If camera is unplugged, OpenPose GUI and command line will display a warning and try to reconnect it.
     5. Wrapper classes simplified and renamed. Wrapper renamed as WrapperT, and created Wrapper as the non-templated class equivalent.
@@ -330,6 +332,7 @@ OpenPose Library - Release Notes
     35. Windows updates:
         1. Upgraded OpenCV version for Windows from 3.1 to 4.0.1, which provides stable 30 FPS for webcams (vs. 10 FPS that OpenCV 3.1 provides by default on Windows).
         2. Upgrade VS2015 to VS2017, allowing CUDA 10 and 20XX Nvidia cards.
+    36. Output JSON updated to version 1.3, which now includes the person IDs (if any).
 2. Functions or parameters renamed:
     1. By default, python example `tutorial_developer/python_2_pose_from_heatmaps.py` was using 2 scales starting at -1x736, changed to 1 scale at -1x368.
     2. WrapperStructPose default parameters changed to match those of the OpenPose demo binary.

include/openpose/core/array.hpp

@@ -5,6 +5,7 @@
 #include <vector>
 #include <opencv2/core/core.hpp> // cv::Mat
 #include <openpose/core/macros.hpp>
+#include <openpose/utilities/errorAndLog.hpp>
 
 namespace op
 {
@@ -87,6 +88,26 @@ namespace op
          */
         Array(const Array<T>& array, const int index, const bool noCopy = false);
 
+        /**
+         * Array constructor. It manually copies the Array<T2> into the new Array<T>
+         * @param array Array<T2> with a format T2 different to the current Array type T.
+         */
+        template<typename T2>
+        Array(const Array<T2>& array) :
+            Array{array.getSize()}
+        {
+            try
+            {
+                // Copy
+                for (auto i = 0u ; i < array.getVolume() ; i++)
+                    pData[i] = array[i];
+            }
+            catch (const std::exception& e)
+            {
+                error(e.what(), __LINE__, __FUNCTION__, __FILE__);
+            }
+        }
+
         /**
          * Copy constructor.
          * It performs `fast copy`: For performance purpose, copying a Array<T> or Datum or cv::Mat just copies the

include/openpose/filestream/fileStream.hpp

@@ -25,33 +25,37 @@ namespace op
     OP_API void saveFloatArray(const Array<float>& array, const std::string& fullFilePath);
 
     // Save/load json, xml, yaml, yml
-    OP_API void saveData(const std::vector<cv::Mat>& cvMats, const std::vector<std::string>& cvMatNames,
-                         const std::string& fileNameNoExtension, const DataFormat dataFormat);
+    OP_API void saveData(
+        const std::vector<cv::Mat>& cvMats, const std::vector<std::string>& cvMatNames,
+        const std::string& fileNameNoExtension, const DataFormat dataFormat);
 
-    OP_API void saveData(const cv::Mat& cvMat, const std::string cvMatName, const std::string& fileNameNoExtension,
-                         const DataFormat dataFormat);
+    OP_API void saveData(
+        const cv::Mat& cvMat, const std::string cvMatName, const std::string& fileNameNoExtension,
+        const DataFormat dataFormat);
 
-    OP_API std::vector<cv::Mat> loadData(const std::vector<std::string>& cvMatNames,
-                                         const std::string& fileNameNoExtension, const DataFormat dataFormat);
+    OP_API std::vector<cv::Mat> loadData(
+        const std::vector<std::string>& cvMatNames, const std::string& fileNameNoExtension,
+        const DataFormat dataFormat);
 
-    OP_API cv::Mat loadData(const std::string& cvMatName, const std::string& fileNameNoExtension,
-                            const DataFormat dataFormat);
+    OP_API cv::Mat loadData(
+        const std::string& cvMatName, const std::string& fileNameNoExtension, const DataFormat dataFormat);
 
     // Json - Saving as *.json not available in OpenCV verions < 3.0, this function is a quick fix
-    OP_API void savePeopleJson(const Array<float>& keypoints,
-                               const std::vector<std::vector<std::array<float,3>>>& candidates,
-                               const std::string& keypointName, const std::string& fileName,
-                               const bool humanReadable);
+    OP_API void savePeopleJson(
+        const Array<float>& keypoints, const std::vector<std::vector<std::array<float,3>>>& candidates,
+        const std::string& keypointName, const std::string& fileName, const bool humanReadable);
 
     // It will save a bunch of Array<float> elements
-    OP_API void savePeopleJson(const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
-                               const std::vector<std::vector<std::array<float,3>>>& candidates,
-                               const std::string& fileName, const bool humanReadable);
+    OP_API void savePeopleJson(
+        const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
+        const std::vector<std::vector<std::array<float,3>>>& candidates, const std::string& fileName,
+        const bool humanReadable);
 
     // Save/load image
-    OP_API void saveImage(const cv::Mat& cvMat, const std::string& fullFilePath,
-                          const std::vector<int>& openCvCompressionParams
-                            = {CV_IMWRITE_JPEG_QUALITY, 100, CV_IMWRITE_PNG_COMPRESSION, 9});
+    OP_API void saveImage(
+        const cv::Mat& cvMat, const std::string& fullFilePath,
+        const std::vector<int>& openCvCompressionParams
+            = {CV_IMWRITE_JPEG_QUALITY, 100, CV_IMWRITE_PNG_COMPRESSION, 9});
 
     OP_API cv::Mat loadImage(const std::string& fullFilePath, const int openCvFlags = CV_LOAD_IMAGE_ANYDEPTH);
 

include/openpose/filestream/peopleJsonSaver.hpp

@@ -13,9 +13,10 @@ namespace op
 
         virtual ~PeopleJsonSaver();
 
-        void save(const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
-                  const std::vector<std::vector<std::array<float,3>>>& candidates,
-                  const std::string& fileName, const bool humanReadable = true) const;
+        void save(
+            const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
+            const std::vector<std::vector<std::array<float,3>>>& candidates, const std::string& fileName,
+            const bool humanReadable = true) const;
     };
 }
 

include/openpose/filestream/wPeopleJsonSaver.hpp

@@ -72,7 +72,12 @@ namespace op
                     // const auto fileName = baseFileName;
                     const auto fileName = baseFileName + (i != 0 ? "_" + std::to_string(i) : "");
 
+                    // Pose IDs from long long to float
+                    Array<float> poseIds{tDatumPtr->poseIds};
+
                     const std::vector<std::pair<Array<float>, std::string>> keypointVector{
+                        // Pose IDs
+                        std::make_pair(poseIds, "person_id"),
                         // 2D
                         std::make_pair(tDatumPtr->poseKeypoints, "pose_keypoints_2d"),
                         std::make_pair(tDatumPtr->faceKeypoints, "face_keypoints_2d"),
@@ -85,7 +90,8 @@ namespace op
                         std::make_pair(tDatumPtr->handKeypoints3D[1], "hand_right_keypoints_3d")
                     };
                     // Save keypoints
-                    spPeopleJsonSaver->save(keypointVector, tDatumPtr->poseCandidates, fileName, humanReadable);
+                    spPeopleJsonSaver->save(
+                        keypointVector, tDatumPtr->poseCandidates, fileName, humanReadable);
                 }
                 // Profiling speed
                 Profiler::timerEnd(profilerKey);

include/openpose/gpu/cuda.hu

@@ -89,19 +89,20 @@ namespace op
 
     // Cubic interpolation
     template <typename T>
-    inline __device__ void cubicSequentialData(int* xIntArray, int* yIntArray, T& dx, T& dy, const T xSource, const T ySource,
-                                               const int width, const int height)
+    inline __device__ void cubicSequentialData(
+        int* xIntArray, int* yIntArray, T& dx, T& dy, const T xSource, const T ySource, const int widthSource,
+        const int heightSource)
     {
-        xIntArray[1] = fastTruncateCuda(int(xSource + 1e-5), 0, width - 1);
+        xIntArray[1] = fastTruncateCuda(int(floor(xSource)), 0, widthSource - 1);
         xIntArray[0] = fastMaxCuda(0, xIntArray[1] - 1);
-        xIntArray[2] = fastMinCuda(width - 1, xIntArray[1] + 1);
-        xIntArray[3] = fastMinCuda(width - 1, xIntArray[2] + 1);
+        xIntArray[2] = fastMinCuda(widthSource - 1, xIntArray[1] + 1);
+        xIntArray[3] = fastMinCuda(widthSource - 1, xIntArray[2] + 1);
         dx = xSource - xIntArray[1];
 
-        yIntArray[1] = fastTruncateCuda(int(ySource + 1e-5), 0, height - 1);
+        yIntArray[1] = fastTruncateCuda(int(floor(ySource)), 0, heightSource - 1);
         yIntArray[0] = fastMaxCuda(0, yIntArray[1] - 1);
-        yIntArray[2] = fastMinCuda(height - 1, yIntArray[1] + 1);
-        yIntArray[3] = fastMinCuda(height - 1, yIntArray[2] + 1);
+        yIntArray[2] = fastMinCuda(heightSource - 1, yIntArray[1] + 1);
+        yIntArray[3] = fastMinCuda(heightSource - 1, yIntArray[2] + 1);
         dy = ySource - yIntArray[1];
     }
 
@@ -135,27 +136,29 @@ namespace op
         for (unsigned char i = 0; i < 4; i++)
         {
             const auto offset = yIntArray[i]*widthSourcePtr;
-            temp[i] = cubicInterpolate(sourcePtr[offset + xIntArray[0]], sourcePtr[offset + xIntArray[1]],
-                                       sourcePtr[offset + xIntArray[2]], sourcePtr[offset + xIntArray[3]], dx);
+            temp[i] = cubicInterpolate(
+                sourcePtr[offset + xIntArray[0]], sourcePtr[offset + xIntArray[1]], sourcePtr[offset + xIntArray[2]],
+                sourcePtr[offset + xIntArray[3]], dx);
         }
         return cubicInterpolate(temp[0], temp[1], temp[2], temp[3], dy);
     }
 
     template <typename T>
     inline __device__ T bicubicInterpolate8Times(
         const T* const sourcePtr, const T xSource, const T ySource, const int widthSource, const int heightSource,
-        const int widthSourcePtr, const int threadIdxX, const int threadIdxY)
+        const int threadIdxX, const int threadIdxY)
     {
         // Now we only need dx and dy
-        const T dx = xSource - fastTruncateCuda(int(xSource + 1e-5), 0, widthSource - 1);
-        const T dy = ySource - fastTruncateCuda(int(ySource + 1e-5), 0, heightSource - 1);
+        const T dx = xSource - fastTruncateCuda(int(floor(xSource)), 0, widthSource - 1);
+        const T dy = ySource - fastTruncateCuda(int(floor(ySource)), 0, heightSource - 1);
 
         T temp[4];
         for (unsigned char i = 0; i < 4; i++)
         {
             const auto offset = 5 * (i + (threadIdxY > 3 ? 1 : 0)) + (threadIdxX > 3 ? 1 : 0);
-            temp[i] = cubicInterpolate(sourcePtr[offset + 0], sourcePtr[offset + 1],
-                                       sourcePtr[offset + 2], sourcePtr[offset + 3], dx);
+            temp[i] = cubicInterpolate(
+                sourcePtr[offset], sourcePtr[offset+1], sourcePtr[offset+2],
+                sourcePtr[offset+3], dx);
         }
         return cubicInterpolate(temp[0], temp[1], temp[2], temp[3], dy);
     }
@@ -167,7 +170,8 @@ namespace op
     }
 
     template <typename T>
-    inline __device__ void addColorWeighted(T& colorR, T& colorG, T& colorB, const T* const colorToAdd, const T alphaColorToAdd)
+    inline __device__ void addColorWeighted(
+        T& colorR, T& colorG, T& colorB, const T* const colorToAdd, const T alphaColorToAdd)
     {
         colorR = addWeighted(colorR, colorToAdd[0], alphaColorToAdd);
         colorG = addWeighted(colorG, colorToAdd[1], alphaColorToAdd);

src/openpose/core/array.cpp

@@ -1,7 +1,6 @@
 #include <typeinfo> // typeid
 #include <numeric> // std::accumulate
 #include <openpose/utilities/avx.hpp>
-#include <openpose/utilities/errorAndLog.hpp>
 #include <openpose/core/array.hpp>
 
 // Note: std::shared_ptr not (fully) supported for array pointers:

src/openpose/filestream/fileStream.cpp

@@ -17,15 +17,16 @@ namespace op
         return fileNameNoExtension + "." + dataFormatToString(dataFormat);
     }
 
-    void addKeypointsToJson(JsonOfstream& jsonOfstream,
-                            const std::vector<std::pair<Array<float>, std::string>>& keypointVector)
+    void addKeypointsToJson(
+        JsonOfstream& jsonOfstream, const std::vector<std::pair<Array<float>, std::string>>& keypointVector)
     {
         try
         {
             // Sanity check
             for (const auto& keypointPair : keypointVector)
-                if (!keypointPair.first.empty() && keypointPair.first.getNumberDimensions() != 3 )
-                    error("keypointVector.getNumberDimensions() != 3.", __LINE__, __FUNCTION__, __FILE__);
+                if (!keypointPair.first.empty() && keypointPair.first.getNumberDimensions() != 3
+                    && keypointPair.first.getNumberDimensions() != 1)
+                    error("keypointVector.getNumberDimensions() != 1 && != 3.", __LINE__, __FUNCTION__, __FILE__);
             // Add people keypoints
             jsonOfstream.key("people");
             jsonOfstream.arrayOpen();
@@ -76,16 +77,16 @@ namespace op
         }
     }
 
-    void addCandidatesToJson(JsonOfstream& jsonOfstream,
-                             const std::vector<std::vector<std::array<float,3>>>& candidates)
+    void addCandidatesToJson(
+        JsonOfstream& jsonOfstream, const std::vector<std::vector<std::array<float,3>>>& candidates)
     {
         try
         {
             // Add body part candidates
             jsonOfstream.key("part_candidates");
             jsonOfstream.arrayOpen();
             // Ger max numberParts
-            auto numberParts = candidates.size();
+            const auto numberParts = candidates.size();
             jsonOfstream.objectOpen();
             for (auto part = 0u ; part < numberParts ; part++)
             {
@@ -283,16 +284,15 @@ namespace op
         }
     }
 
-    void savePeopleJson(const Array<float>& keypoints,
-                        const std::vector<std::vector<std::array<float,3>>>& candidates,
-                        const std::string& keypointName, const std::string& fileName,
-                        const bool humanReadable)
+    void savePeopleJson(
+        const Array<float>& keypoints, const std::vector<std::vector<std::array<float,3>>>& candidates,
+        const std::string& keypointName, const std::string& fileName, const bool humanReadable)
     {
         try
         {
             savePeopleJson(
-                std::vector<std::pair<Array<float>, std::string>>{std::make_pair(keypoints, keypointName)}, candidates,
-                fileName, humanReadable
+                std::vector<std::pair<Array<float>, std::string>>{std::make_pair(keypoints, keypointName)},
+                candidates, fileName, humanReadable
             );
         }
         catch (const std::exception& e)
@@ -301,16 +301,18 @@ namespace op
         }
     }
 
-    void savePeopleJson(const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
-                        const std::vector<std::vector<std::array<float,3>>>& candidates,
-                        const std::string& fileName, const bool humanReadable)
+    void savePeopleJson(
+        const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
+        const std::vector<std::vector<std::array<float,3>>>& candidates, const std::string& fileName,
+        const bool humanReadable)
     {
         try
         {
             // Sanity check
             for (const auto& keypointPair : keypointVector)
-                if (!keypointPair.first.empty() && keypointPair.first.getNumberDimensions() != 3 )
-                    error("keypointVector.getNumberDimensions() != 3.", __LINE__, __FUNCTION__, __FILE__);
+                if (!keypointPair.first.empty() && keypointPair.first.getNumberDimensions() != 3
+                    && keypointPair.first.getNumberDimensions() != 1)
+                    error("keypointVector.getNumberDimensions() != 1 && != 3.", __LINE__, __FUNCTION__, __FILE__);
             // Record frame on desired path
             JsonOfstream jsonOfstream{fileName, humanReadable};
             jsonOfstream.objectOpen();
@@ -319,7 +321,8 @@ namespace op
             // Version 1.0: Added face and hands (2-D)
             // Version 1.1: Added candidates
             // Version 1.2: Added body, face, and hands (3-D)
-            jsonOfstream.version("1.2");
+            // Version 1.3: Added person ID (for temporal consistency)
+            jsonOfstream.version("1.3");
             jsonOfstream.comma();
             // Add people keypoints
             addKeypointsToJson(jsonOfstream, keypointVector);

src/openpose/filestream/peopleJsonSaver.cpp

@@ -12,9 +12,10 @@ namespace op
     {
     }
 
-    void PeopleJsonSaver::save(const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
-                               const std::vector<std::vector<std::array<float,3>>>& candidates,
-                               const std::string& fileName, const bool humanReadable) const
+    void PeopleJsonSaver::save(
+        const std::vector<std::pair<Array<float>, std::string>>& keypointVector,
+        const std::vector<std::vector<std::array<float,3>>>& candidates, const std::string& fileName,
+        const bool humanReadable) const
     {
         try
         {

src/openpose/net/nmsBase.cu

@@ -251,7 +251,6 @@ namespace op
     {
         try
         {
-            //Forward_cpu(bottom, top);
             const auto num = sourceSize[0];
             const auto height = sourceSize[2];
             const auto width = sourceSize[3];