Skip to content

Commit 80d5356

Browse files
author
Andrew Or
committed
Revert "[SPARK-8437] [DOCS] Using directory path without wildcard for filename slow for large number of files with wholeTextFiles and binaryFiles"
This reverts commit b268455.
1 parent c0fbd67 commit 80d5356

File tree

1 file changed

+2
-6
lines changed

1 file changed

+2
-6
lines changed

core/src/main/scala/org/apache/spark/SparkContext.scala

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -824,8 +824,6 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
824824
* }}}
825825
*
826826
* @note Small files are preferred, large file is also allowable, but may cause bad performance.
827-
* @note On some filesystems, `.../path/*` can be a more efficient way to read all files in a directory
828-
* rather than `.../path/` or `.../path`
829827
*
830828
* @param minPartitions A suggestion value of the minimal splitting number for input data.
831829
*/
@@ -873,11 +871,9 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
873871
* (a-hdfs-path/part-nnnnn, its content)
874872
* }}}
875873
*
876-
* @note Small files are preferred; very large files may cause bad performance.
877-
* @note On some filesystems, `.../path/*` can be a more efficient way to read all files in a directory
878-
* rather than `.../path/` or `.../path`
879-
*
880874
* @param minPartitions A suggestion value of the minimal splitting number for input data.
875+
*
876+
* @note Small files are preferred; very large files may cause bad performance.
881877
*/
882878
@Experimental
883879
def binaryFiles(

0 commit comments

Comments
 (0)