Change Log automation

[SergioRAgostinho]

Just opening this to report that I started working on automating the change log generation.

So far the idea is a Python 3 script which queries the GitHub API for all merged PR from a given tag up till now. It looks at the labels from each PR to assign the PR to the appropriate modules and flags the special cases like ABI, API and behaviour changes, as long as the PR is labelled.

Predictable pitfalls from this strategy: we usually branch for each tag release and there are certain PRs which get merged to master but not to the release branch. With the strategy I mentioned those won't be properly parsed. Basically I would need to find the parent commit when the branch occurred.

[taketwo]

Have a look at this project, perhaps it's flexible enough to meet our demands.

Regarding the pitfall, maybe the script can accept start and end commit ids (or timestamps) and create a changelog for everything in between. This will be a bit less automated than specifying a release tag, but way simpler to implement.

[SergioRAgostinho]

Have a look at this project, perhaps it's flexible enough to meet our demands.

It looked like overkill.

Regarding the pitfall, maybe the script can accept start and end commit ids (or timestamps) and create a changelog for everything in between. This will be a bit less automated than specifying a release tag, but way simpler to implement.

I've offloaded this task to git. We can specify just the tag now as originally intended.

I'm halfway through this: I'm now fetching PR info and later building the change log based on it.

[SergioRAgostinho]

It's already generating something like the snippet shown below. I still need to add some authentication because there's a 60 requests per hour limit to access the GitHub API otherwise, plus highlight PRs which broke ABI/API/Behavor.

It's probably important to revisit the uncategorized PRs and label them properly, etc...

= master (20.07.2018) =

• Correct setting of is_dense flag in SegmentDifferences [#2380]
• Unbreak build on BSDs [#2374]
• Unbreak build against Boost 1.68 [#2373]
• [PARSE] Another 'parse_file_extension_argument ()' [#2347]
• addition of declaration header file [#2356]
• Fix for ::posix_fallocate on Mac OSX (PR Add header with low level I/O helpers. #2341) [#2354]
• Improve QHull finder script [#2344]
• outofcore: Explictly use mt19937 random generator for boost 1.67. [#2338]
• Improve speed of transformPointCloud/WithNormals() functions [#2247]
• Add RGB constructor that takes R, G, and B components [#2329]
• Second change VTK version check in addTextureMesh [#2322]
• Fix cmake warning: Logical block closes with mis-matching arguments [#2320]
• fix missing vtkTexture.h, follow up on Remove depreciated VTK function MapDataArrayToMultiTextureAttribute #2291 [#2316]
• Change VTK version check in addTextureMesh [#2311]
• minor fixes. [#2300]
• Dereference shared_ptr, fix for GCC8 (Closes: Build fail on fedora 28 #2282) [#2299]
• Tiny fixes: Typo + function type [#2297]
• (Issue GASD signature not available for nearest neighbor search using pcl::search::KdTree #2290) Add GASDSignatures to PCL_POINT_TYPES and PCL_FEATURE_P… [#2295]
• Added missing 8 bytes to compressed binary pcd length. [#2281]
• Setting the resolution of the occupancy grid [#2273]
• Make MomentOfInertia instantiations consistent with the rest of the library [#2266]
• [CMAKE] Fix apps compilation [#2251]
• fix link order issue with boost [#2236]
• Mark found PCL component libraries and include dirs as advanced [#2235]
• Prevent search for disabled optional dependencies in targets. [#2229]
• Fix access specifier in PointCloudColorHandlerRGBAField [#2226]
• [Ensenso grabber] uint is undefined in Visual studio. [#2223]
• Correct testPoint for organized nearest neighbor search [#2198]
• Fix two "unreachable code" warnings in pca.hpp [#2219]
• Check if color handler returns false [#2209]
• Public access to VoxelGrid boost pointer. [#2205]
• Fix compilation error in gpu_people code [#2199]
• Fix some CUDA 9 related errors [#2181]
• Fixed typo and rearragend items in the issue template [#2197]
• Add protection from invalid WIDTH values in PCD reader [#2195]
• [TEST] OCTREE Iterator begin/end method and add test [#2174]
• Split (yet again) Travis test job into two and tweak timings in build… [#2182]
• Add const qualifiers to getters in filter_indices.h [#2193]
• Fix installation rules for ml module [#2192]
• opennurbs: fix ON_Curve::EvaluatePoint calculation [#2185]
• Add TrajkovicKeypoint2D/3D to CMake build [#2179]
• Update message about legacy point cloud types in tutorial [#2175]
• [TYPO] Fix some comment in octree test [#2173]
• Enable Global Tests on Windows CI [#2137]
• Change Stale daysTillClose to 100 years [#2166]
• handle VTK legacy function (follow up to Small optimizations and fixes in PCLVisualizer #2112) [#2165]
• Travis merge test jobs [#2163]
• Avoid huge index jumps in RandomSample [#2141]
• set stale daysUntilClose to a really big number [#2162]
• Cast cloud point step as 64-bit integer [#2161]
• fixed issue failed to run pcl_openni2_viewer with Kinect Xbox 360(OpenNI2 + libfreenect) #1348 by adding device sensor check for IR and depth modesetting [#2150]
• [VISU] Add another method 'getPointCloudRenderingProperties()' [#2142]
• Remove unnecessary PPAs and packages from Travis [#2153]
• Stale set up [#2101]
• changed toff to size_t in writeBinaryCompressed [#2144]
• fix small typo (reagion -> region) [#2143]
• Prevent POINTS field parsing before point_step is specified [#2131]
• Check COUNT value specified in PCD files [#2126]
• Restrict accepted types in PCLVisualizer::addLine signature [#2134]
• Change to simple style of specify triplet [#2135]
• Fix covariance calculation in PCA [#2130]
• Prevent mmapping more than the original PCD file size [#2125]
• [VISU] Typo fix in error message. [#2132]
• Initial Appveyor CI integration [#2083]
• Fix convert to Eigen::Map from Normal of pcl::PointXYZINormal [#2128]
• Change Travis to use pip3 for installing sphinx [#2124]
• Fix conditional branch of Visual C++ 2017 [#2121]
• [TEST] Fix the compilation of 'test_visualisation' [#2115]
• Improve Doxygen comments for HistogramInterpolationMethod [#2111]
• Fix pcd_viewer color handling when invalid fields are present in pcd [#2105]
• Add pcl visualizer interactor null guards [#2104]
• Add remove text3d [#2088]
• Add *_USE_STATIC options to PCLConfig [#2086]
• Suppress the final set of warnings in pcl apps [#2082]
• Add search path suffixes for Vcpkg [#2085]
• fix header names [#2079]
• Suppress unused parameter warning [#2074]
• Suppress (maybe) uninitialized warning [#2073]
• Remove unused variable from octree_viewer [#2069]
• Suppress sign compare warnings [#2068]
• Suppress unused parameter warnings [#2067]
• Update finder scripts for Ensenso, OpenNI, and OpenNI2 [#2061]
• Suppress control reaches end of non-void function in io.h [#2057]
• Consistent ptr typedefs for kd tree flann [#1607]
• Fix incorrect Ptr/ConstPtr typedefs in MovingLeastSquaresOMP [#2055]
• Fix PACKAGE to include cmake/Modules directory [#2053]
• Unifies Find scripts in PCLConfig [#1421]
• CUDA 9 Arch Flags [#2047]
• Modify STRICT_ALIGN because macro expansion w/defined is undefined [#2043]
• [gcc] fixes -Wimplicit-fallthrough: common/io.h [#2041]
• Support both RGB and RGBA colors in mesh loading [#2036]
• Misleading indentation [#2034]
• fix /MP option not generated for Visual Studio. [#2031]
• Fix the incorrect include directory. [#2024]
• Fix several documentation typos [#2020]
• Replace float indices with Eigen Index [#2017]
• Silence compile warning by removing superfluous call to std::max() [#2014]
• [TRAVIS] Enable the build of apps. [#2012]
• [TRAVIS] Enable the build of tools. [#2007]
• Disable tools build in CI. [#2003]
• Adds in-memory PCD serialization/deserialization; de-duplicates PCDReader::readHeader(). (take Fix compilation #2) [#1986]
• [OCTREE] Compute accurately the centroids of octree in 'pcl_octree_viewer' [#1981]
• Fix bug in PointCloudGeometryHandlerCustom [#2001]
• Add new mls projection method [#1960]
• Fixed queryBBIncludes_subsample [#1988]
• Fixes remove_indices in UniformSampling [#1902]
• Replace literal include of wrong CMakeLists file with correct script [#1971]
• Add accessor for current rendering framerate in PCLVisualizer [#1974]
• [OCTREE] Fix pcl_octree_viewer [#1973]
• [OCTREE] Add bounding box checks in isVoxelOccupiedAtPoint() and deleteVoxelAtPoint() [#1976]
• Generate pkgconfig for 2d module [#1979]
• [OCTREE] Remove a useless field in octree_viewer. [#1980]
• Allow changing LUT properties of a shape actor [#1668]
• Redo: Simulation: enable returning of organized point clouds [#1687]
• Simulation: more access to camera parameters [#1650]
• Update Find Boost [#1972]
• Typo [#1968]
• Include pcl/point_cloud.h and pcl/point_types.h headers. [#1962]
• Add ability to cache mls results [#1952]
• Add test for macro _USE_MATH_DEFINES. [#1956]
• Fixed spelling and grammar errors [#1959]
• Fixed error in documentation. [#1957]
• need to include instantiate.hpp to use PCL_INSTANTIATE [#1943]
• Update Ensenso tutorial for Ensenso X devices [#1933]
• Added CUDA compute capability 5.3 [#1929]
• Fix issue PCL-git build fail with CUDA? #1674 [#1926]
• instantiate: remove duplicate macro definition. Fixes duplicate macro definition in instantiate.hpp header #1924. [#1925]
• Add new gtest path [#1920]

libpcl_common:

• [PARSE] Constness of the API [#2224]

libpcl_cuda:

• Fix compilation issues with CUDA 9.1 [#2212]

libpcl_features:

• Update the OpenMP implementations of normal and FPFH estimation [#2278]

libpcl_gpu:

• Fix compilation issues with CUDA 9.1 [#2212]

libpcl_io:

• Allocate disk space with posix_fallocate before mmapping. [#2325]
• Remove useless size check in PLYReader::endHeaderCallback() [#2246]
• Fix PLYReader is_dense behavior [#2133]

libpcl_octree:

• [OCTREE] Implementation of the iterator 'OctreeLeafNodeBreadthIterator'. [#2204]

libpcl_recognition:

• Inline helper function gcCorrespSorter() [#2248]

libpcl_simulation:

• Use GLuint rather than size_t to represent OpenGL indices. [#2238]

PCL Apps:

• add windows.h for includes GL/gl.h; handle cancel for denoiseWidget. [#2267]

PCL Tests:

• Cleanup and improve unit test coverage for transformPointCloud functions [#2245]
• Fixes and new assertion macro in "pcl_tests.h" [#2237]

[taketwo]

Oh, looks great! So if I understand correctly, all the PRs in the first section are unlabeled, but as soon as we mark them with some "module" label, they will move to the corresponding section?

Regarding the breaking changes, I think it would be nice to have both a) highlight or icon in the log entries b) separate section in the beginning of the changelog with a summary of breaking changes.

[SergioRAgostinho]

Oh, looks great! So if I understand correctly, all the PRs in the first section are unlabeled, but as soon as we mark them with some "module" label, they will move to the corresponding section?

Yes and they can have more than one module label if needed. A PR which affects multiple modules will appear in each module's section. I will still need to go over the currently available module labels since a bunch of them are still missing.

The general idea is that now, once we merge a PR, we need to ensure that the title is descriptive enough and that the module, changes and platform labels are properly set. A minor maintenance overhead penalty for an improved release experience.

a) highlight or icon in the log entries

Since we have more than one special status: abi change, api change, behavior change and new feature (?); it might be worth to adopt some icons for each.

b) separate section in the beginning of the changelog with a summary of breaking changes.

By summary you mean an actual summary or just an aggregation of all PRs with breaking changes?

[SergioRAgostinho]

Back at this again 👍

[taketwo]

Since we have more than one special status: abi change, api change, behavior change and new feature (?); it might be worth to adopt some icons for each.

On the other hand, easily understandable icons may be hard to find (how do you visualize ABI change?), and also they won't be rendered when viewing changelog not on GitHub. Thus maybe literal tags like [api] or [behavior] are more comprehensible.

By summary you mean an actual summary or just an aggregation of all PRs with breaking changes?

I meant simple aggregation. Then we can manually expand the items with suggestions what to do about these changes, effectively ending up with a rudimentary "migration guide".

[SergioRAgostinho]

The latest flavor. I'm going to open the PR and write some instructions on how use the tool.

= HEAD (20.07.2018) =

New Features:

• [octree] [OCTREE] Implementation of the iterator 'OctreeLeafNodeBreadthIterator'. [#2204]
• [OCTREE] Implementation of the iterator 'OctreeFixedDepthIterator'. [#1983]

API Changes:

• [octree] [OCTREE] Implementation of the iterator 'OctreeLeafNodeBreadthIterator'. [#2204]
• Const-qualify most of the methods in SAC model classes [#2270]
• [simulation] Use GLuint rather than size_t to represent OpenGL indices. [#2238]
• OctreeIterators special member revision [#2108]
• Add support pcl::PointXYZRGBA to pcl::VLPGrabber [#2102]
• The remaining deprecation removals [#2077]
• Remove deprecated ros headers [#2075]
• Remove registration module deprecated methods [#2076]
• Remove deprecated functions and variables from SAC module [#2071]
• Removal of PCA deprecated constructor [#2070]
• Removed normal related accessors and types from EuclideanClusterComparator [#1542]

ABI Changes:

• [OCTREE] Implementation of the iterator 'OctreeFixedDepthIterator'. [#1983]
• Provide proper EuclideanClusterComparator method depreciation [#2096]
• Add support pcl::PointXYZRGBA to pcl::VLPGrabber [#2102]
• Allow specifying decimation step in convertToTsdfCloud [#2099]
• More warning suppression in pcl apps [#2080]
• Removed unused member from ply_parser [#2066]
• Removed normal related accessors and types from EuclideanClusterComparator [#1542]

Behavior Changes:

• [io] Fix PLYReader is_dense behavior [#2133]

Modules:

Uncategorized:

• Correct setting of is_dense flag in SegmentDifferences [#2380]
• Unbreak build on BSDs [#2374]
• Unbreak build against Boost 1.68 [#2373]
• [PARSE] Another 'parse_file_extension_argument ()' [#2347]
• addition of declaration header file [#2356]
• Fix for ::posix_fallocate on Mac OSX (PR Add header with low level I/O helpers. #2341) [#2354]
• Improve QHull finder script [#2344]
• outofcore: Explictly use mt19937 random generator for boost 1.67. [#2338]
• Improve speed of transformPointCloud/WithNormals() functions [#2247]
• Add RGB constructor that takes R, G, and B components [#2329]
• Second change VTK version check in addTextureMesh [#2322]
• Fix cmake warning: Logical block closes with mis-matching arguments [#2320]
• fix missing vtkTexture.h, follow up on Remove depreciated VTK function MapDataArrayToMultiTextureAttribute #2291 [#2316]
• Change VTK version check in addTextureMesh [#2311]
• minor fixes. [#2300]
• Dereference shared_ptr, fix for GCC8 (Closes: Build fail on fedora 28 #2282) [#2299]
• Tiny fixes: Typo + function type [#2297]
• [api] Const-qualify most of the methods in SAC model classes [#2270]
• (Issue GASD signature not available for nearest neighbor search using pcl::search::KdTree #2290) Add GASDSignatures to PCL_POINT_TYPES and PCL_FEATURE_P… [#2295]
• Added missing 8 bytes to compressed binary pcd length. [#2281]
• Setting the resolution of the occupancy grid [#2273]
• Make MomentOfInertia instantiations consistent with the rest of the library [#2266]
• [CMAKE] Fix apps compilation [#2251]
• fix link order issue with boost [#2236]
• Mark found PCL component libraries and include dirs as advanced [#2235]
• [abi][new-feature] [OCTREE] Implementation of the iterator 'OctreeFixedDepthIterator'. [#1983]
• Prevent search for disabled optional dependencies in targets. [#2229]
• Fix access specifier in PointCloudColorHandlerRGBAField [#2226]
• [Ensenso grabber] uint is undefined in Visual studio. [#2223]
• Correct testPoint for organized nearest neighbor search [#2198]
• Fix two "unreachable code" warnings in pca.hpp [#2219]
• Check if color handler returns false [#2209]
• Public access to VoxelGrid boost pointer. [#2205]
• fix error in Line 11 in example code for estimate set of surface for a subset of points in the input dataset [#2203]
• Fix compilation error in gpu_people code [#2199]
• Fix some CUDA 9 related errors [#2181]
• Fixed typo and rearragend items in the issue template [#2197]
• Add protection from invalid WIDTH values in PCD reader [#2195]
• [TEST] OCTREE Iterator begin/end method and add test [#2174]
• Split (yet again) Travis test job into two and tweak timings in build… [#2182]
• Add const qualifiers to getters in filter_indices.h [#2193]
• Fix installation rules for ml module [#2192]
• opennurbs: fix ON_Curve::EvaluatePoint calculation [#2185]
• Add TrajkovicKeypoint2D/3D to CMake build [#2179]
• Update message about legacy point cloud types in tutorial [#2175]
• [TYPO] Fix some comment in octree test [#2173]
• Enable Global Tests on Windows CI [#2137]
• Change Stale daysTillClose to 100 years [#2166]
• handle VTK legacy function (follow up to Small optimizations and fixes in PCLVisualizer #2112) [#2165]
• Travis merge test jobs [#2163]
• Avoid huge index jumps in RandomSample [#2141]
• set stale daysUntilClose to a really big number [#2162]
• Cast cloud point step as 64-bit integer [#2161]
• fixed issue failed to run pcl_openni2_viewer with Kinect Xbox 360(OpenNI2 + libfreenect) #1348 by adding device sensor check for IR and depth modesetting [#2150]
• [VISU] Add another method 'getPointCloudRenderingProperties()' [#2142]
• Remove unnecessary PPAs and packages from Travis [#2153]
• Stale set up [#2101]
• changed toff to size_t in writeBinaryCompressed [#2144]
• fix small typo (reagion -> region) [#2143]
• Prevent POINTS field parsing before point_step is specified [#2131]
• Check COUNT value specified in PCD files [#2126]
• [api] OctreeIterators special member revision [#2108]
• Restrict accepted types in PCLVisualizer::addLine signature [#2134]
• Change to simple style of specify triplet [#2135]
• Fix covariance calculation in PCA [#2130]
• Prevent mmapping more than the original PCD file size [#2125]
• [VISU] Typo fix in error message. [#2132]
• Initial Appveyor CI integration [#2083]
• Add url to API/ABI compatibity report [#2116]
• Fix convert to Eigen::Map from Normal of pcl::PointXYZINormal [#2128]
• Change Travis to use pip3 for installing sphinx [#2124]
• Fix conditional branch of Visual C++ 2017 [#2121]
• [TEST] Fix the compilation of 'test_visualisation' [#2115]
• Improve Doxygen comments for HistogramInterpolationMethod [#2111]
• Fix pcd_viewer color handling when invalid fields are present in pcd [#2105]
• [abi] Provide proper EuclideanClusterComparator method depreciation [#2096]
• Add pcl visualizer interactor null guards [#2104]
• Add remove text3d [#2088]
• Add *_USE_STATIC options to PCLConfig [#2086]
• [abi][api] Add support pcl::PointXYZRGBA to pcl::VLPGrabber [#2102]
• [abi] Allow specifying decimation step in convertToTsdfCloud [#2099]
• Suppress the final set of warnings in pcl apps [#2082]
• Add search path suffixes for Vcpkg [#2085]
• [abi] More warning suppression in pcl apps [#2080]
• fix header names [#2079]
• [api] The remaining deprecation removals [#2077]
• [api] Remove deprecated ros headers [#2075]
• [api] Remove registration module deprecated methods [#2076]
• Suppress unused parameter warning [#2074]
• Suppress (maybe) uninitialized warning [#2073]
• [api] Remove deprecated functions and variables from SAC module [#2071]
• [api] Removal of PCA deprecated constructor [#2070]
• Remove unused variable from octree_viewer [#2069]
• Suppress sign compare warnings [#2068]
• Suppress unused parameter warnings [#2067]
• [abi] Removed unused member from ply_parser [#2066]
• Update finder scripts for Ensenso, OpenNI, and OpenNI2 [#2061]
• Suppress control reaches end of non-void function in io.h [#2057]
• Consistent ptr typedefs for kd tree flann [#1607]
• Fix incorrect Ptr/ConstPtr typedefs in MovingLeastSquaresOMP [#2055]
• Fix PACKAGE to include cmake/Modules directory [#2053]
• [abi][api] Removed normal related accessors and types from EuclideanClusterComparator [#1542]
• Unifies Find scripts in PCLConfig [#1421]
• CUDA 9 Arch Flags [#2047]
• Modify STRICT_ALIGN because macro expansion w/defined is undefined [#2043]
• [gcc] fixes -Wimplicit-fallthrough: common/io.h [#2041]
• Support both RGB and RGBA colors in mesh loading [#2036]
• Misleading indentation [#2034]
• fix /MP option not generated for Visual Studio. [#2031]
• Fix the incorrect include directory. [#2024]
• Fix several documentation typos [#2020]
• Replace float indices with Eigen Index [#2017]
• Silence compile warning by removing superfluous call to std::max() [#2014]
• [TRAVIS] Enable the build of apps. [#2012]
• [TRAVIS] Enable the build of tools. [#2007]
• Disable tools build in CI. [#2003]
• Adds in-memory PCD serialization/deserialization; de-duplicates PCDReader::readHeader(). (take Fix compilation #2) [#1986]
• [OCTREE] Compute accurately the centroids of octree in 'pcl_octree_viewer' [#1981]
• Fix bug in PointCloudGeometryHandlerCustom [#2001]
• Add new mls projection method [#1960]
• Fixed queryBBIncludes_subsample [#1988]
• Fixes remove_indices in UniformSampling [#1902]
• Replace literal include of wrong CMakeLists file with correct script [#1971]
• Add accessor for current rendering framerate in PCLVisualizer [#1974]
• [OCTREE] Fix pcl_octree_viewer [#1973]
• [OCTREE] Add bounding box checks in isVoxelOccupiedAtPoint() and deleteVoxelAtPoint() [#1976]
• Generate pkgconfig for 2d module [#1979]
• [OCTREE] Remove a useless field in octree_viewer. [#1980]
• Allow changing LUT properties of a shape actor [#1668]
• Redo: Simulation: enable returning of organized point clouds [#1687]
• Simulation: more access to camera parameters [#1650]
• Update Find Boost [#1972]
• Typo [#1968]
• Include pcl/point_cloud.h and pcl/point_types.h headers. [#1962]
• Add ability to cache mls results [#1952]
• Add test for macro _USE_MATH_DEFINES. [#1956]
• Fixed spelling and grammar errors [#1959]
• Fixed error in documentation. [#1957]
• need to include instantiate.hpp to use PCL_INSTANTIATE [#1943]
• Update Ensenso tutorial for Ensenso X devices [#1933]
• Added option to specify translation and rotation convergence deltas in ICP and NDT algorithms. [#1724]
• Added CUDA compute capability 5.3 [#1929]
• Fix issue PCL-git build fail with CUDA? #1674 [#1926]
• instantiate: remove duplicate macro definition. Fixes duplicate macro definition in instantiate.hpp header #1924. [#1925]
• Add new gtest path [#1920]

libpcl_common:

• [PARSE] Constness of the API [#2224]

libpcl_cuda:

• Fix compilation issues with CUDA 9.1 [#2212]

libpcl_features:

• Update the OpenMP implementations of normal and FPFH estimation [#2278]

libpcl_gpu:

• Fix compilation issues with CUDA 9.1 [#2212]

libpcl_io:

• Allocate disk space with posix_fallocate before mmapping. [#2325]
• Remove useless size check in PLYReader::endHeaderCallback() [#2246]
• [behavior] Fix PLYReader is_dense behavior [#2133]

libpcl_octree:

• [api][new-feature] [OCTREE] Implementation of the iterator 'OctreeLeafNodeBreadthIterator'. [#2204]

libpcl_recognition:

• Inline helper function gcCorrespSorter() [#2248]

libpcl_simulation:

• [api] Use GLuint rather than size_t to represent OpenGL indices. [#2238]

PCL Apps:

• add windows.h for includes GL/gl.h; handle cancel for denoiseWidget. [#2267]

PCL Tests:

• Cleanup and improve unit test coverage for transformPointCloud functions [#2245]
• Fixes and new assertion macro in "pcl_tests.h" [#2237]