ARROW-12730: [MATLAB] Update featherreadmex and featherwritemex to build against latest Arrow C++ APIs

**Overview**
* The MEX functions ``featherreadmex`` and ``featherwritemex`` fail to build against the latest Arrow C++ APIs. These changes allow them to successfully build.
* These changes require CMake version 3.20 or later in order to access the latest functionality exposed by [FindMatlab.cmake](https://cmake.org/cmake/help/latest/module/FindMatlab.html). We noticed that some Arrow project components, such as [Gandiva](https://arrow.apache.org/docs/developers/cpp/building.html?highlight=gandiva#cmake-version-requirements), require newer versions of CMake than the core Arrow C++ libraries.  If version 3.20 is too new, we're happy to find an alternative.
* We couldn't find a way to read and write a table description for feather V1 files using the latest APIs. It looks like support for reading and writing descriptions was modified in pull request #6694. For now, we've removed support for table descriptions.

**Testing**
* Built ``featherreadmex`` and ``featherwritemex`` on Windows 10 with Visual Studio 2019
* Built ``featherreadmex`` and ``featherwritemex`` on macOS Big Sur (11.2.3) with GNU Make 3.81
* Built ``featherreadmex`` and ``featherwritemex`` on Debian 10 with GNU Make GNU 4.2.1
* Ran all tests in ``tfeather`` and ``tfeathermex`` on all platforms in MATLAB R2021a

**Future Directions**
* We did not detect the build failures due to the lack of CI integration. We hope to add CI support soon and will follow up with a mailing list discussion to talk through the details.
* These changes are temporary to allow us to have a clean slate to start developing the  [MATLAB Interface to Apache Arrow](https://github.com/apache/arrow/blob/master/matlab/doc/matlab_interface_for_apache_arrow_design.md).
* Eventually we would like to support the full ranges of data types for feather V1 and feather V2.
* In order to modernize the code, we plan to migrate to the [C++ MEX](https://www.mathworks.com/help/matlab/cpp-mex-file-applications.html) and [MATLAB Data Array](https://www.mathworks.com/help/matlab/matlab-data-array.html) APIs.
* We are going to follow up with another pull request to update the README.md to provide more detailed platform-specific development instructions.
* The MATLAB based build system inside of the ``build_support`` folder is out of date.  We are not sure if we want to maintain a separate MATLAB based build system along side the CMake based one. We will follow up on this in the future via the mailing list or Jira.

We acknowledge there is a lot of information in this pull request. In the future, we will work in smaller increments. We felt a bigger pull request was necessary to get back to a working state.

Thanks,
Sarah

Closes #10305 from sgilmore10/ARROW_12730

Lead-authored-by: sgilmore <sgilmore@mathworks.com>
Co-authored-by: sgilmore10 <74676073+sgilmore10@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>

Author: sgilmore10

matlab/CMakeLists.txt

@@ -15,7 +15,8 @@
 # specific language governing permissions and limitations
 # under the License.
 
-cmake_minimum_required(VERSION 3.2)
+cmake_minimum_required(VERSION 3.20)
+
 set(CMAKE_CXX_STANDARD 11)
 
 set(MLARROW_VERSION "5.0.0-SNAPSHOT")
@@ -29,22 +30,45 @@ if(EXISTS "${CPP_CMAKE_MODULES}")
   set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CPP_CMAKE_MODULES})
 endif()
 
-## Arrow is Required
+set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_SOURCE_DIR}/cmake_modules)
+
+# Arrow is Required
 find_package(Arrow REQUIRED)
 
-## MATLAB is required to be installed to build MEX interfaces
-set(MATLAB_ADDITIONAL_VERSIONS "R2018a=9.4")
-find_package(Matlab REQUIRED MX_LIBRARY)
-
-# Build featherread mex file based on the arrow shared library
-matlab_add_mex(NAME featherreadmex
-               SRC src/featherreadmex.cc src/feather_reader.cc src/util/handle_status.cc
-                   src/util/unicode_conversion.cc
-               LINK_TO ${ARROW_SHARED_LIB})
-target_include_directories(featherreadmex PRIVATE ${ARROW_INCLUDE_DIR})
-
-# Build featherwrite mex file based on the arrow shared library
-matlab_add_mex(NAME featherwritemex
-               SRC src/featherwritemex.cc src/feather_writer.cc src/util/handle_status.cc
-               LINK_TO ${ARROW_SHARED_LIB})
-target_include_directories(featherwritemex PRIVATE ${ARROW_INCLUDE_DIR})
+# MATLAB is Required
+find_package(Matlab REQUIRED)
+
+# Construct the absolute path to featherread's source files
+set(featherread_sources featherreadmex.cc feather_reader.cc util/handle_status.cc
+                        util/unicode_conversion.cc)
+list(TRANSFORM featherread_sources PREPEND ${CMAKE_SOURCE_DIR}/src/)
+
+# Build featherreadmex MEX binary
+matlab_add_mex(R2018a
+               NAME featherreadmex
+               SRC ${featherread_sources}
+               LINK_TO arrow_shared)
+
+# Construct the absolute path to featherwrite's source files
+set(featherwrite_sources featherwritemex.cc feather_writer.cc util/handle_status.cc
+                         util/unicode_conversion.cc)
+list(TRANSFORM featherwrite_sources PREPEND ${CMAKE_SOURCE_DIR}/src/)
+
+# Build featherwritemex MEX binary
+matlab_add_mex(R2018a
+               NAME featherwritemex
+               SRC ${featherwrite_sources}
+               LINK_TO arrow_shared)
+
+# Ensure the MEX binaries are placed in the src directory on all platforms
+if(WIN32)
+  set_target_properties(featherreadmex PROPERTIES RUNTIME_OUTPUT_DIRECTORY
+                                                  $<1:${CMAKE_SOURCE_DIR}/src>)
+  set_target_properties(featherwritemex PROPERTIES RUNTIME_OUTPUT_DIRECTORY
+                                                   $<1:${CMAKE_SOURCE_DIR}/src>)
+else()
+  set_target_properties(featherreadmex PROPERTIES LIBRARY_OUTPUT_DIRECTORY
+                                                  $<1:${CMAKE_SOURCE_DIR}/src>)
+  set_target_properties(featherwritemex PROPERTIES LIBRARY_OUTPUT_DIRECTORY
+                                                   $<1:${CMAKE_SOURCE_DIR}/src>)
+endif()

matlab/src/+mlarrow/+util/createMetadataStruct.m

@@ -1,4 +1,4 @@
-function metadata = createMetadataStruct(description, numRows, numVariables)
+function metadata = createMetadataStruct(numRows, numVariables)
 % CREATEMETADATASTRUCT Helper function for creating Feather MEX metadata
 % struct.
 
@@ -17,8 +17,7 @@
 % implied.  See the License for the specific language governing
 % permissions and limitations under the License.
 
-metadata = struct('Description', description, ...
-                  'NumRows', numRows, ...
+metadata = struct('NumRows', numRows, ...
                   'NumVariables', numVariables);
 end
 

matlab/src/+mlarrow/+util/table2mlarrow.m

@@ -23,7 +23,6 @@
 %
 %   Field Name    Class         Description
 %   ------------  -------       ----------------------------------------------
-%   Description   char          Table description (T.Properties.Description)
 %   NumRows       double        Number of table rows (height(T))
 %   NumVariables  double        Number of table variables (width(T))
 %
@@ -51,7 +50,7 @@
 variables = repmat(createVariableStruct('', [], [], ''), 1, width(t));
 
 % Struct representing table-level metadata.
-metadata = createMetadataStruct(t.Properties.Description, height(t), width(t));
+metadata = createMetadataStruct(height(t), width(t));
 
 % Iterate over each variable in the given table,
 % extracting the underlying array data.

matlab/src/feather_reader.cc

@@ -18,16 +18,21 @@
 #include <algorithm>
 #include <cmath>
 
+#include "feather_reader.h"
+
+#include <arrow/array/array_base.h>
+#include <arrow/array/builder_base.h>
+#include <arrow/array/builder_primitive.h>
 #include <arrow/io/file.h>
 #include <arrow/ipc/feather.h>
+#include <arrow/result.h>
 #include <arrow/status.h>
 #include <arrow/table.h>
 #include <arrow/type.h>
-#include <arrow/util/bit-util.h>
-
+#include <arrow/type_traits.h>
+#include <arrow/util/bitmap_visit.h>
 #include <mex.h>
 
-#include "feather_reader.h"
 #include "matlab_traits.h"
 #include "util/handle_status.h"
 #include "util/unicode_conversion.h"
@@ -52,11 +57,11 @@ mxArray* ReadNumericVariableData(const std::shared_ptr<Array>& column) {
   mxArray* variable_data =
       mxCreateNumericMatrix(column->length(), 1, matlab_class_id, mxREAL);
 
-  std::shared_ptr<ArrowArrayType> integer_array =
+  auto arrow_numeric_array =
       std::static_pointer_cast<ArrowArrayType>(column);
 
   // Get a raw pointer to the Arrow array data.
-  const MatlabType* source = integer_array->raw_values();
+  const MatlabType* source = arrow_numeric_array->raw_values();
 
   // Get a mutable pointer to the MATLAB array data and std::copy the
   // Arrow array data into it.
@@ -121,8 +126,7 @@ void BitUnpackBuffer(const std::shared_ptr<Buffer>& source, int64_t length,
 // writes to a zero-initialized destination buffer.
 // Implements a fast path for the fully-valid and fully-invalid cases.
 // Returns true if the destination buffer was successfully populated.
-bool TryBitUnpackFastPath(const std::shared_ptr<Array>& array,
-                          mxLogical* destination) {
+bool TryBitUnpackFastPath(const std::shared_ptr<Array>& array, mxLogical* destination) {
   const int64_t null_count = array->null_count();
   const int64_t length = array->length();
 
@@ -177,32 +181,24 @@ Status FeatherReader::Open(const std::string& filename,
   *feather_reader = std::shared_ptr<FeatherReader>(new FeatherReader());
 
   // Open file with given filename as a ReadableFile.
-  std::shared_ptr<io::ReadableFile> readable_file(nullptr);
-
-  RETURN_NOT_OK(io::ReadableFile::Open(filename, &readable_file));
-
-  // TableReader expects a RandomAccessFile.
-  std::shared_ptr<io::RandomAccessFile> random_access_file(readable_file);
-
+  ARROW_ASSIGN_OR_RAISE(auto readable_file, io::ReadableFile::Open(filename));
+ 
   // Open the Feather file for reading with a TableReader.
-  RETURN_NOT_OK(ipc::feather::TableReader::Open(random_access_file,
-                                                &(*feather_reader)->table_reader_));
-
-  // Read the table metadata from the Feather file.
-  (*feather_reader)->num_rows_ = (*feather_reader)->table_reader_->num_rows();
-  (*feather_reader)->num_variables_ = (*feather_reader)->table_reader_->num_columns();
-  (*feather_reader)->description_ =
-      (*feather_reader)->table_reader_->HasDescription()
-          ? (*feather_reader)->table_reader_->GetDescription()
-          : "";
-
-  if ((*feather_reader)->num_rows_ > internal::MAX_MATLAB_SIZE ||
-      (*feather_reader)->num_variables_ > internal::MAX_MATLAB_SIZE) {
-    mexErrMsgIdAndTxt("MATLAB:arrow:SizeTooLarge",
-                      "The table size exceeds MATLAB limits: %u x %u",
-                      (*feather_reader)->num_rows_, (*feather_reader)->num_variables_);
+  ARROW_ASSIGN_OR_RAISE(auto reader, ipc::feather::Reader::Open(readable_file));
+ 
+  // Set the internal reader_ object.
+  (*feather_reader)->reader_ = reader;
+
+  // Check the feather file version
+  auto version = reader->version();
+  if (version == ipc::feather::kFeatherV2Version) {
+    return Status::NotImplemented("Support for Feather V2 has not been implemented.");
+  } else if (version != ipc::feather::kFeatherV1Version) {
+    return Status::Invalid("Unknown Feather format version.");
   }
 
+  // read the table metadata from the Feather file
+  (*feather_reader)->num_variables_ = reader->schema()->num_fields();
   return Status::OK();
 }
 
@@ -225,15 +221,11 @@ mxArray* FeatherReader::ReadMetadata() const {
   mxSetField(metadata, 0, "NumVariables",
              mxCreateDoubleScalar(static_cast<double>(num_variables_)));
 
-  // Set the description.
-  mxSetField(metadata, 0, "Description",
-             util::ConvertUTF8StringToUTF16CharMatrix(description_));
-
   return metadata;
 }
 
 // Read the table variables from the Feather file as a mxArray*.
-mxArray* FeatherReader::ReadVariables() const {
+mxArray* FeatherReader::ReadVariables() {
   const int32_t num_variable_fields = 4;
   const char* fieldnames[] = {"Name", "Type", "Data", "Valid"};
 
@@ -242,16 +234,34 @@ mxArray* FeatherReader::ReadVariables() const {
   mxArray* variables =
       mxCreateStructMatrix(1, num_variables_, num_variable_fields, fieldnames);
 
-  // Read all the table variables in the Feather file into memory.
+  std::shared_ptr<arrow::Table> table;
+  auto status = reader_->Read(&table);
+  if (!status.ok()) {
+    mexErrMsgIdAndTxt("MATLAB:arrow:FeatherReader::FailedToReadTable",
+                      "Failed to read arrow::Table from Feather file. Reason: %s",
+                      status.message().c_str());
+  }
+
+  // Set the number of rows
+  num_rows_ = table->num_rows();
+
+  if (num_rows_ > internal::MAX_MATLAB_SIZE ||
+      num_variables_ > internal::MAX_MATLAB_SIZE) {
+    mexErrMsgIdAndTxt("MATLAB:arrow:SizeTooLarge",
+                      "The table size exceeds MATLAB limits: %u x %u", num_rows_,
+                      num_variables_);
+  }
+
+  auto column_names = table->ColumnNames();
+
   for (int64_t i = 0; i < num_variables_; ++i) {
-    std::shared_ptr<ChunkedArray> column;
-    util::HandleStatus(table_reader_->GetColumn(i, &column));
+    auto column = table->column(i);
     if (column->num_chunks() != 1) {
       mexErrMsgIdAndTxt("MATLAB:arrow:FeatherReader::ReadVariables",
                         "Chunked columns not yet supported");
     }
     std::shared_ptr<Array> chunk = column->chunk(0);
-    const std::string column_name = table_reader_->GetColumnName(i);
+    const std::string column_name = column_names[i];
 
     // set the struct fields data
     mxSetField(variables, i, "Name", internal::ReadVariableName(column_name));

matlab/src/feather_reader.h

@@ -23,7 +23,6 @@
 #include <arrow/ipc/feather.h>
 #include <arrow/status.h>
 #include <arrow/type.h>
-
 #include <matrix.h>
 
 namespace arrow {
@@ -56,7 +55,7 @@ class FeatherReader {
   ///        Clients are responsible for freeing the returned mxArray memory
   ///        when it is no longer needed, or passing it to MATLAB to be managed.
   /// \return variables mxArray* struct array containing table variable data
-  mxArray* ReadVariables() const;
+  mxArray* ReadVariables();
 
   /// \brief Initialize a FeatherReader object from a given Feather file.
   /// \param[in] filename path to a Feather file
@@ -66,12 +65,11 @@ class FeatherReader {
 
  private:
   FeatherReader() = default;
-  std::unique_ptr<ipc::feather::TableReader> table_reader_;
+  std::shared_ptr<ipc::feather::Reader> reader_;
   int64_t num_rows_;
   int64_t num_variables_;
   std::string description_;
 };
 
 }  // namespace matlab
 }  // namespace arrow
-