Upgrades flat buffers to most recent version (deeplearning4j#10189)

* update flatbuffers to new 64 bit version

* Add ADR

* fix namespace usage after new generation

* Add rest of updates for flatc generated classes.

Author: agibsonccc

ADRs/0034 - FlatBuffers upgrade.md

@@ -0,0 +1,128 @@
+# ADR 0034: FlatBuffers Modernization
+
+## Status
+
+Implemented
+
+Proposed by: Assistant (20-02-2025)
+Discussed with: Adam Gibson
+
+## Context
+
+The libnd4j library uses FlatBuffers for serialization of neural network graphs and related data structures. The current implementation uses FlatBuffers 1.12.0 syntax and conventions, particularly for handling sequences and arrays. With the upgrade to newer versions of FlatBuffers, we need to modernize our schema definitions and code generation.
+
+The primary challenges include:
+
+1. Migration from legacy Sequence/SequenceItem patterns to modern vector types
+2. Proper generation of Java files with correct package structures
+3. Integration with build system for consistent compilation
+4. Maintaining backward compatibility where possible
+5. Ensuring proper environment variable passing for code generation
+6. Proper schema namespace management
+
+## Decision
+
+We implement a modernized FlatBuffers integration that uses current vector syntax and build processes. This includes:
+
+### Key Components
+- Schema files using modern vector syntax (`[Type]` instead of Sequence)
+- Direct build process integration in CMake
+- Explicit environment variable handling for flatc compiler
+- Namespace standardization across schema files
+
+### Implementation Details
+1. Schemas use vector syntax for array types (e.g., `[FlatArray]` instead of `Sequence<FlatArray>`)
+2. CMake executes flatc compilation directly instead of using custom targets
+3. Build process ensures flatc is compiled before schema generation
+4. Java package structure matches expected nd4j-api layout
+5. Schema namespaces standardized to `graph` without `sd` prefix
+
+### Schema Example
+```fbs
+namespace graph;
+
+table SequenceItem {
+  name:string;
+  associated_variable:[FlatArray];
+}
+
+table SequenceItemRoot {
+  sequence_items:[SequenceItem];
+}
+
+root_type SequenceItemRoot;
+```
+
+### Build Integration
+```cmake
+execute_process(
+    COMMAND ${CMAKE_COMMAND} -E env "FLATC_PATH=${FLATC_EXECUTABLE}" 
+    bash ${CMAKE_CURRENT_SOURCE_DIR}/flatc-generate.sh
+    RESULT_VARIABLE FLATC_RESULT
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+)
+```
+
+## Consequences
+
+### Advantages
+1. Modern FlatBuffers Integration:
+   - Cleaner schema definitions
+   - Better type safety through vector syntax
+   - More maintainable code generation
+   - Consistent with current FlatBuffers best practices
+
+2. Build System Integration:
+   - Reliable flatc compilation
+   - Proper environment variable handling
+   - Direct process execution instead of custom targets
+   - Better error handling and reporting
+
+3. Code Organization:
+   - Correct Java package structure
+   - Standardized namespaces
+   - Clear separation of generated code
+   - Better integration with existing nd4j structure
+
+### Disadvantages
+1. Implementation Requirements:
+   - Need to update existing schema files
+   - Must ensure build process compatibility
+   - Potential for temporary build issues during transition
+
+2. Migration Effort:
+   - Updates needed for existing code using legacy patterns
+   - Need to verify all schema files
+   - Testing required for all serialization paths
+
+## Technical Details
+
+### Schema Location
+```
+/libnd4j/include/graph/scheme/*.fbs
+```
+
+### Generated Code Location
+```
+/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/graph/
+```
+
+### Build Process
+1. CMake configures build
+2. flatc compiler is built
+3. Schema generation script runs
+4. Java files are copied to appropriate location
+
+## Alternatives Considered
+
+1. Custom Target Approach:
+   - Pros: More traditional CMake integration
+   - Cons: Less direct control over process, harder to debug
+
+2. Manual File Generation:
+   - Pros: Simpler build process
+   - Cons: Error-prone, harder to maintain
+
+3. Separate Build Step:
+   - Pros: Cleaner separation of concerns
+   - Cons: More complex build process, potential for synchronization issues

deeplearning4j/deeplearning4j-ui-parent/deeplearning4j-vertx/pom.xml

@@ -40,10 +40,10 @@
 
     <build>
         <plugins>
-        <!--    <plugin>
+            <plugin>
                 <groupId>org.moditect</groupId>
                 <artifactId>moditect-maven-plugin</artifactId>
-            </plugin>-->
+            </plugin>
         </plugins>
     </build>
 
@@ -309,9 +309,9 @@
         </dependency>
         <!-- For SameDiff UI -->
         <dependency>
-            <groupId>org.webjars.npm</groupId>
-            <artifactId>flatbuffers</artifactId>
-            <version>1.9.0</version>
+            <groupId>org.nd4j</groupId>
+            <artifactId>nd4j-web</artifactId>
+            <version>${project.version}</version>
         </dependency>
     </dependencies>
 

libnd4j/CMakeLists.txt

@@ -641,7 +641,14 @@ if (${HELPERS_cudnn})
     endif()
 endif()
 
-# Download and unpack flatbuffers at configure time
+# Before the flatbuffers download section
+if(DEFINED ENV{GENERATE_FLATC} OR DEFINED GENERATE_FLATC)
+    set(FLATBUFFERS_BUILD_FLATC "ON" CACHE STRING "Enable flatc build" FORCE)
+else()
+    set(FLATBUFFERS_BUILD_FLATC "OFF" CACHE STRING "Disable flatc build" FORCE)
+endif()
+
+# Existing flatbuffers download section
 configure_file(CMakeLists.txt.in flatbuffers-download/CMakeLists.txt)
 execute_process(COMMAND ${CMAKE_COMMAND}  "${CMAKE_GENERATOR}" .
         RESULT_VARIABLE result
@@ -656,15 +663,54 @@ if(result)
     message(FATAL_ERROR "Build step for flatbuffers failed: ${result}")
 endif()
 
-# Add flatbuffers directly to our build.
+# Add flatbuffers directly to build
 add_subdirectory(${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src
         ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build
         EXCLUDE_FROM_ALL)
 
-set(HAVE_FLATBUFFERS 1)
-set(FLATBUFFERS_PATH ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src)
-include_directories(${FLATBUFFERS_PATH}/include)
+# After flatbuffers is added, handle generation if enabled
+if(DEFINED ENV{GENERATE_FLATC} OR DEFINED GENERATE_FLATC)
+    # First, ensure flatc is built
+    execute_process(
+            COMMAND ${CMAKE_COMMAND} --build ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build --target flatc
+            RESULT_VARIABLE FLATC_BUILD_RESULT
+    )
+    if(FLATC_BUILD_RESULT)
+        message(FATAL_ERROR "Failed to build flatc: ${FLATC_BUILD_RESULT}")
+    endif()
+
 
+    set(FLATC_EXECUTABLE "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build/flatc")
+    message("Using flatc from: ${FLATC_EXECUTABLE}")
+
+    # Create required directories
+    execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory
+            ${CMAKE_CURRENT_SOURCE_DIR}/include/graph/generated
+            ${CMAKE_CURRENT_SOURCE_DIR}/include/graph/generated/sd
+    )
+
+    # Run flatc generation
+    execute_process(
+            COMMAND ${CMAKE_COMMAND} -E env "FLATC_PATH=${FLATC_EXECUTABLE}"
+            bash ${CMAKE_CURRENT_SOURCE_DIR}/flatc-generate.sh
+            RESULT_VARIABLE FLATC_RESULT
+            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+    )
+    if(FLATC_RESULT)
+        message(FATAL_ERROR "Flatbuffer generation failed: ${FLATC_RESULT}")
+    endif()
+
+    # Copy Java files
+    execute_process(
+            COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/copy-flatc-java.sh
+            RESULT_VARIABLE COPY_RESULT
+            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+    )
+    if(COPY_RESULT)
+        message(FATAL_ERROR "Java file copying failed: ${COPY_RESULT}")
+    endif()
+endif()
+include_directories(${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src/include)
 
 configure_file(include/config.h.in include/config.h)
 include_directories(${CMAKE_CURRENT_BINARY_DIR}/include)

libnd4j/CMakeLists.txt.in

@@ -2,15 +2,22 @@ cmake_minimum_required(VERSION 2.8.2)
 
 project(flatbuffers-download NONE)
 
+# Force clean everything first
+file(REMOVE_RECURSE ${CMAKE_CURRENT_BINARY_DIR}/*)
+
 include(ExternalProject)
 ExternalProject_Add(flatbuffers
-  GIT_REPOSITORY    https://github.com/KonduitAI/flatbuffers.git
-  GIT_TAG           main
-  SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src"
-  BINARY_DIR        "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build"
-  CONFIGURE_COMMAND ""
-  CMAKE_ARGS        "-DFLATBUFFERS_BUILD_FLATC=OFF"
-  BUILD_COMMAND     ""
-  INSTALL_COMMAND   ""
-  TEST_COMMAND      ""
-)
+    GIT_REPOSITORY    https://github.com/google/flatbuffers/
+    GIT_TAG           v25.2.10
+    SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src"
+    BINARY_DIR        "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-build"
+    CMAKE_ARGS
+        -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
+        -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
+        -DFLATBUFFERS_BUILD_FLATC=ON
+        -DCMAKE_BUILD_TYPE=Release
+    DOWNLOAD_COMMAND  ${CMAKE_COMMAND} -E rm -rf ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src && git clone https://github.com/google/flatbuffers/ ${CMAKE_CURRENT_BINARY_DIR}/flatbuffers-src
+    UPDATE_COMMAND    ""
+    INSTALL_COMMAND   ""
+    TEST_COMMAND      ""
+)

libnd4j/CMakePresets.json

@@ -12,6 +12,7 @@
                 "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}",
                 "SD_LIBRARY_NAME": "nd4jcpu",
                 "SD_CPU": true,
+                "GENERATE_FLATC": "ON",
                 "SD_ARCH": "x86-64",
                 "SD_GCC_FUNCTRACE": "ON",
                 "PRINT_MATH": "ON",
@@ -36,7 +37,7 @@
                 "PRINT_INDICES": "ON",
                 "SD_BUILD_TESTS": "OFF",
                 "PRINT_MATH": "ON",
-
+                "GENERATE_FLATC": "ON",
                 "SD_ALL_OPS": true,
                 "CMAKE_BUILD_TYPE" : "Debug",
                 "OPENBLAS_PATH": "$env{HOME}/.javacpp/cache/openblas-0.3.19-1.5.7-linux-x86_64.jar/org/bytedeco/openblas/linux-x86_64"
@@ -62,6 +63,7 @@
                 "SD_GCC_FUNCTRACE": "ON",
                 "CMAKE_CUDA_ARCHITECTURES": "86",
                 "SD_BUILD_TESTS": "OFF",
+                "GENERATE_FLATC": "ON",
                 "CUDA_TOOLKIT_ROOT_DIR": "/usr/local/cuda",
                 "CMAKE_CUDA_COMPILER": "/usr/local/cuda/bin/nvcc"
             }
@@ -80,6 +82,7 @@
                 "SD_LIBRARY_NAME": "nd4jcuda",
                 "SD_ALL_OPS": true,
                 "SD_CUDA": true,
+                "GENERATE_FLATC": "ON",
                 "BLAS":true,
                 "SD_GCC_FUNCTRACE": "ON",
                 "__CUDACC__" : "ON",

libnd4j/buildnativeoperations.sh

@@ -93,13 +93,19 @@ LOG_OUTPUT="none"
 PRINT_MATH="OFF"
 KEEP_NVCC="OFF"
 PREPROCESS="ON"  # Initialize PREPROCESS variable
+CMAKE_ARGUMENTS=""
 
 while [[ $# -gt 0 ]]
 do
     key="$1"
     value="${2:-}"
     # Build type (release/debug), packaging type, chip: cpu,cuda, lib type (static/dynamic)
     case $key in
+      --generate-flatc)
+                export GENERATE_FLATC="ON"
+                CMAKE_ARGUMENTS="$CMAKE_ARGUMENTS -DGENERATE_FLATC=ON"
+                shift # past argument
+                ;;
         -ol|--optimization-level)
             OPTIMIZATION_LEVEL="$value"
             shift # past argument