[SPARK-30653][INFRA][SQL] EOL character enforcement for java/scala/xml/py/R files

[HeartSaVioR]

What changes were proposed in this pull request?

This patch converts CR/LF into LF in 3 source files, which most files are only using LF. This patch also add rules to enforce EOL as LF for all java, scala, xml, py, R files.

Why are the changes needed?

The majority of source code files are using LF and only three files are CR/LF. While using IDE would let us don't bother with the difference, it still has a chance to make unnecessary diff if the file is modified with the editor which doesn't handle it automatically.

Does this PR introduce any user-facing change?

No

How was this patch tested?

grep -IUrl --color "^M" . | grep "\.java\|\.scala\|\.xml\|\.py\|\.R" | grep -v "/target/" | grep -v "/build/" | grep -v "/dist/" | grep -v "dependency-reduced-pom.xml" | grep -v ".pyc"

(Please note you'll need to type CTRL+V -> CTRL+M in bash shell to get ^M because it's representing CR/LF, not a combination of ^ and M.)

Before the patch, the result is:

./sql/core/src/main/java/org/apache/spark/sql/execution/columnar/ColumnDictionary.java
./sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala

and after the patch, the result is None.

And git shows WARNING message if EOL of any of source files in given types are modified to CR/LF, like below:

warning: CRLF will be replaced by LF in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala.
The file will have its original line endings in your working directory.

[HeartSaVioR]

I think it's ideal to define the rules in .gitattributes, but would like to hear the voices before moving forward.

[srowen]

If it's just a few files, OK. I think it's also fine to have git enforce it. Is there any downside to that?

[SparkQA]

Test build #117425 has finished for PR 27365 at commit 25d3e61.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public final class ColumnDictionary implements Dictionary

[dongjoon-hyun]

@HeartSaVioR . When I follow the direction in the PR description at the master branch, the result is different. Did I miss something?

$ git log --oneline -n1
43d9c7e7e5 (HEAD -> master, apache/master, apache/HEAD) [SPARK-30640][PYTHON][SQL] Prevent unnecessary copies of data during Arrow to Pandas conversion

$ grep -IUrl --color "^M" . | grep "\.java\|\.scala\|\.xml\|\.py\|\.R" | grep -v "/target/" | grep -v "/build/" | grep -v "/dist/" | grep -v "dependency-reduced-pom.xml" | grep -v ".pyc"
./python/pyspark/_globals.py
./python/pyspark/heapq3.py
./python/pyspark/mllib/linalg/__init__.py
./python/pyspark/shuffle.py
./python/pyspark/ml/linalg/__init__.py
./R/pkg/vignettes/sparkr-vignettes.Rmd
./examples/src/main/python/logistic_regression.py
./dev/github_jira_sync.py

[HeartSaVioR]

@dongjoon-hyun ^M in the PR description is CR/LF, so you may want to type CTRL+V -> CTRL+M in bash shell to get it. I'll update the PR description.

[HeartSaVioR]

I think it's also fine to have git enforce it. Is there any downside to that?

I don't think there's outstanding downside, as if it does have some considerable downsides someone should have been complained. Only 3 files were CR/LF and others have been LF. (cmd/bat files are enforced to have CR/LF as EOL, as they're only used in Windows OS.)

[SparkQA]

Test build #117430 has finished for PR 27365 at commit a5d975c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Merged to master.
Thank you, @HeartSaVioR and @srowen .

[HeartSaVioR]

Thanks all for reviewing amd merging! Thanks again @dongjoon-hyun to deal with filing JIRA and connecting this.