Skip to content

Commit 40d59ff

Browse files
HyukjinKwonsrowen
authored andcommitted
[SPARK-18422][CORE] Fix wholeTextFiles test to pass on Windows in JavaAPISuite
## What changes were proposed in this pull request? This PR fixes the test `wholeTextFiles` in `JavaAPISuite.java`. This is failed due to the different path format on Windows. For example, the path in `container` was ``` C:\projects\spark\target\tmp\1478967560189-0/part-00000 ``` whereas `new URI(res._1()).getPath()` was as below: ``` /C:/projects/spark/target/tmp/1478967560189-0/part-00000 ``` ## How was this patch tested? Tests in `JavaAPISuite.java`. Tested via AppVeyor. **Before** Build: https://ci.appveyor.com/project/spark-test/spark/build/63-JavaAPISuite-1 Diff: master...spark-test:JavaAPISuite-1 ``` [info] Test org.apache.spark.JavaAPISuite.wholeTextFiles started [error] Test org.apache.spark.JavaAPISuite.wholeTextFiles failed: java.lang.AssertionError: expected:<spark is easy to use. [error] > but was:<null>, took 0.578 sec [error] at org.apache.spark.JavaAPISuite.wholeTextFiles(JavaAPISuite.java:1089) ... ``` **After** Build started: [CORE] `org.apache.spark.JavaAPISuite` [![PR-15866](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=198DDA52-F201-4D2B-BE2F-244E0C1725B2&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/198DDA52-F201-4D2B-BE2F-244E0C1725B2) Diff: master...spark-test:198DDA52-F201-4D2B-BE2F-244E0C1725B2 ``` [info] Test org.apache.spark.JavaAPISuite.wholeTextFiles started ... ``` Author: hyukjinkwon <gurwls223@gmail.com> Closes #15866 from HyukjinKwon/SPARK-18422.
1 parent 795e9fc commit 40d59ff

File tree

1 file changed

+11
-6
lines changed

1 file changed

+11
-6
lines changed

core/src/test/java/org/apache/spark/JavaAPISuite.java

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import java.io.*;
2121
import java.nio.channels.FileChannel;
2222
import java.nio.ByteBuffer;
23-
import java.net.URI;
2423
import java.nio.charset.StandardCharsets;
2524
import java.util.ArrayList;
2625
import java.util.Arrays;
@@ -46,6 +45,7 @@
4645
import com.google.common.collect.Lists;
4746
import com.google.common.base.Throwables;
4847
import com.google.common.io.Files;
48+
import org.apache.hadoop.fs.Path;
4949
import org.apache.hadoop.io.IntWritable;
5050
import org.apache.hadoop.io.Text;
5151
import org.apache.hadoop.io.compress.DefaultCodec;
@@ -1075,18 +1075,23 @@ public void wholeTextFiles() throws Exception {
10751075
byte[] content2 = "spark is also easy to use.\n".getBytes(StandardCharsets.UTF_8);
10761076

10771077
String tempDirName = tempDir.getAbsolutePath();
1078-
Files.write(content1, new File(tempDirName + "/part-00000"));
1079-
Files.write(content2, new File(tempDirName + "/part-00001"));
1078+
String path1 = new Path(tempDirName, "part-00000").toUri().getPath();
1079+
String path2 = new Path(tempDirName, "part-00001").toUri().getPath();
1080+
1081+
Files.write(content1, new File(path1));
1082+
Files.write(content2, new File(path2));
10801083

10811084
Map<String, String> container = new HashMap<>();
1082-
container.put(tempDirName+"/part-00000", new Text(content1).toString());
1083-
container.put(tempDirName+"/part-00001", new Text(content2).toString());
1085+
container.put(path1, new Text(content1).toString());
1086+
container.put(path2, new Text(content2).toString());
10841087

10851088
JavaPairRDD<String, String> readRDD = sc.wholeTextFiles(tempDirName, 3);
10861089
List<Tuple2<String, String>> result = readRDD.collect();
10871090

10881091
for (Tuple2<String, String> res : result) {
1089-
assertEquals(res._2(), container.get(new URI(res._1()).getPath()));
1092+
// Note that the paths from `wholeTextFiles` are in URI format on Windows,
1093+
// for example, file:/C:/a/b/c.
1094+
assertEquals(res._2(), container.get(new Path(res._1()).toUri().getPath()));
10901095
}
10911096
}
10921097

0 commit comments

Comments
 (0)