java dataset jar HDFS not supported.

[zinking]

Describe the bug, including details regarding any error messages, version, and platform.

error:

^M[Stage 0:>                                                          (0 + 1) / 1]^M23/08/23 11:27:07 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (VM-73-99-centos executor 13): java.lang.RuntimeException: Got HDFS URI but Arrow compiled without HDFS support
        at org.apache.arrow.dataset.file.JniWrapper.makeFileSystemDatasetFactory(Native Method)

I'ved tried to rebuild cpp with -DARROW_HDFS=ON, and then rebuild dataset with achery, but the resulting 74609088 Aug 23 04:24 x86_64/libarrow_dataset_jni.so seeems not changed.

how shall I build the dataset jar so that it can support HDFS remote read ?

Component(s)

Java

[davisusanibar]

Hi @zinking ,

You could try with this:

1. Go to arrow/java/dataset
2. Get your absolute path of your new dataset native library: i.e $PWD/lib/x86_64/libarrow_dataset_jni.so
3. Build new Java Dataset module:
   mvn clean install -Darrow.cpp.build.dir=$PWD/lib
4. Test your new dataset library with:

import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.Schema;

public class ReadHdfsParquet {
    public static void main(String[] args) {
        //declare JVM environment variable: HADOOP_HOME = /Users/dsusanibar/hadoop-3.3.2
        //where to search for: lib/native/libhdfs.dylib
        String uri = "hdfs://localhost:9000/Users/dsusanibar/data4_2rg_gzip.parquet";
        ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
        try (
            BufferAllocator allocator = new RootAllocator();
            DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
            Dataset dataset = datasetFactory.finish();
            Scanner scanner = dataset.newScan(options);
            ArrowReader reader = scanner.scanBatches()
        ) {
            Schema schema = scanner.schema();
            System.out.println(schema);
            while (reader.loadNextBatch()) {
                System.out.println(reader.getVectorSchemaRoot().contentToTSVString());
                System.out.println("RowCount: " + reader.getVectorSchemaRoot().getRowCount());
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

I was able to read HDFS parquet files, but the program will not shut down for some reason.

Please let me know if you obtain the same result on your side.

[zinking]

@davisusanibar I didn't run the above function. but following patch worked for me.

diff --git a/ci/scripts/java_jni_manylinux_build.sh b/ci/scripts/java_jni_manylinux_build.sh
index 03939715e..fff3c33f3 100755
--- a/ci/scripts/java_jni_manylinux_build.sh
+++ b/ci/scripts/java_jni_manylinux_build.sh
@@ -84,6 +84,7 @@ cmake \
   -DARROW_GANDIVA_PC_CXX_FLAGS=${GANDIVA_CXX_FLAGS} \
   -DARROW_GANDIVA=${ARROW_GANDIVA} \
   -DARROW_GCS=${ARROW_GCS} \
+  -DARROW_HDFS=${ARROW_DATASET} \
   -DARROW_JEMALLOC=${ARROW_JEMALLOC} \
   -DARROW_ORC=${ARROW_ORC} \
   -DARROW_PARQUET=${ARROW_PARQUET} \
@@ -106,7 +107,7 @@ cmake \
   ${arrow_dir}/cpp
 ninja install

able to read remote hdfs file now.