Reworked directory_iterator on Windows to add support for O_NOFOLLOW.

directory_iterator implementation now explicitly opens a directory
handle and relies on GetFileInformationByHandleEx or NtQueryDirectoryFile
to query information of the files in the directory. With
GetFileInformationByHandleEx, there are at least three ways to request
the information, each supported on different Windows versions and
different filesystems and providing different sets of information.
We support all three (FILE_ID_BOTH_DIR_INFO, FILE_FULL_DIR_INFO and
FILE_ID_EXTD_DIR_INFO), and we fall back to older ones if the newer
ones are not supported. GetFileInformationByHandleEx is available
since Windows Vista.

NtQueryDirectoryFile is an NT API that provides similar information to
GetFileInformationByHandleEx, but exists since at least Windows XP,
but some sources state it has existed since NT 4.0. This API is now
used when GetFileInformationByHandleEx is not available and it replaces
FindFirstFileW/FindNextFileW/FindClose based implementation (which
supposedly uses NtQueryDirectoryFile internally). Being an NT API,
we have to handle NTSTATUS error codes returned from it, so we had to
refactor unique path code that also used NTSTATUS previously.

Opening a directory handle explicitly allows to support O_NOFOLLOW
semantics on Windows, as we are able to prevent CreateFile from
following to the reparse point target. This is needed for remove_all
implementation to fix CVE-2022-21658. Similar to POSIX, remove_all_impl
on Windows now requests the directory iterator to not follow symlinks
when it is used to dive into a directory.

FindFirstFileW/FindNextFileW/FindClose based implementation is still
preserved for now for Windows CE, as it is unlikely to support neither
GetFileInformationByHandleEx nor NtQueryDirectoryFile. Given that
Windows CE has been untested for many years and is probably broken
anyway, its support is now declared deprecated. The related code,
including Find*-based directory iterator, will be removed in a
future release.

Closes boostorg#224.

And a couple cleanup changes:

 - Since we now may use GetFileInformationByHandleEx, we are now using
   it to query FILE_ATTRIBUTE_TAG_INFO for testing whether a reparse
   point is a symlink.

 - In resize_file_impl, we now specify all sharing flags to allow the
   operation to succeed if the file is already opened by someone.

Author: Lastique

doc/release_history.html

@@ -42,9 +42,12 @@
 <h2>1.79.0</h2>
 <ul>
   <li>Fixed compilation of path appending and concatenation operators with arguments of types convertible to <code>path</code> or compatible string type. (<a href="https://github.com/boostorg/filesystem/issues/223">#223</a>)</li>
-  <li>On POSIX systems that support <a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/fdopendir.html"><code>fdopendir</code></a> and <a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html"><code>O_NOFOLLOW</code></a>, <code>remove_all</code> is now protected against <a href="https://www.cve.org/CVERecord?id=CVE-2022-21658">CVE-2022-21658</a>. The vulnerability is a race condition that allows a third party process to replace a directory that is being concurrently processed by <code>remove_all</code> with a directory symlink and cause <code>remove_all</code> to follow the symlink and remove files in the linked directory instead of removing the symlink itself. (<a href="https://github.com/boostorg/filesystem/issues/224">#224</a>)</li>
+  <li>On POSIX systems that support <a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/fdopendir.html"><code>fdopendir</code></a> and <a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html"><code>O_NOFOLLOW</code></a> and on Windows, <code>remove_all</code> is now protected against <a href="https://www.cve.org/CVERecord?id=CVE-2022-21658">CVE-2022-21658</a>. The vulnerability is a race condition that allows a third party process to replace a directory that is being concurrently processed by <code>remove_all</code> with a directory symlink and cause <code>remove_all</code> to follow the symlink and remove files in the linked directory instead of removing the symlink itself. (<a href="https://github.com/boostorg/filesystem/issues/224">#224</a>)</li>
+  <li>On Windows, <code>directory_iterator</code> internal implementation has been reworked to better utilize modern Windows APIs, which may improve performance while handling symlinks.</li>
   <li>On Windows, initialize internal WinAPI function pointers early, if possible, to allow Boost.Filesystem operations to be invoked in global constructors. This is only supported on MSVC, GCC, Clang and compatible compilers.</li>
+  <li>On Windows, <code>resize_file</code> should no longer fail with an error if the file to be resized is opened.</li>
   <li><b>Deprecated:</b> <code>boost/filesystem/string_file.hpp</code> header is deprecated and will be removed in a future release. The header is no longer included by <code>boost/filesystem.hpp</code> by default. Users are advised to implement the functionality themselves or migrate to other implementations.</li>
+  <li><b>Deprecated:</b> Windows CE support is deprecated and will be removed in a future release. Windows CE has been untested for many years and is likely non-functional.</li>
 </ul>
 
 <h2>1.78.0</h2>

include/boost/filesystem/directory.hpp

@@ -20,6 +20,7 @@
 #include <boost/filesystem/path.hpp>
 #include <boost/filesystem/file_status.hpp>
 
+#include <cstddef>
 #include <string>
 #include <vector>
 #include <utility> // std::move
@@ -258,40 +259,29 @@ class directory_iterator;
 
 namespace detail {
 
-BOOST_FILESYSTEM_DECL
-system::error_code dir_itr_close( // never throws()
-    void*& handle
-#if defined(BOOST_POSIX_API)
-    , void*& buffer
-#endif
-    ) BOOST_NOEXCEPT;
-
 struct dir_itr_imp :
     public boost::intrusive_ref_counter< dir_itr_imp >
 {
+#ifdef BOOST_WINDOWS_API
+    unsigned char extra_data_format;
+    std::size_t current_offset;
+#endif
     directory_entry dir_entry;
     void* handle;
 
-#if defined(BOOST_POSIX_API)
-    void* buffer; // see dir_itr_increment implementation
-#endif
-
     dir_itr_imp() BOOST_NOEXCEPT :
-        handle(0)
-#if defined(BOOST_POSIX_API)
-        , buffer(0)
+#ifdef BOOST_WINDOWS_API
+        extra_data_format(0u),
+        current_offset(0u),
 #endif
+        handle(NULL)
     {
     }
+    BOOST_FILESYSTEM_DECL ~dir_itr_imp() BOOST_NOEXCEPT;
 
-    ~dir_itr_imp() BOOST_NOEXCEPT
-    {
-        dir_itr_close(handle
-#if defined(BOOST_POSIX_API)
-            , buffer
-#endif
-        );
-    }
+    BOOST_FILESYSTEM_DECL static void* operator new(std::size_t class_size, std::size_t extra_size) BOOST_NOEXCEPT;
+    BOOST_FILESYSTEM_DECL static void operator delete(void* p, std::size_t extra_size) BOOST_NOEXCEPT;
+    BOOST_FILESYSTEM_DECL static void operator delete(void* p) BOOST_NOEXCEPT;
 };
 
 BOOST_FILESYSTEM_DECL void directory_iterator_construct(directory_iterator& it, path const& p, unsigned int opts, system::error_code* ec);