Skip to content

[SPARK-46804][DOCS][TESTS] Recover the generated documents #44843

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jan 23, 2024

What changes were proposed in this pull request?

This PR regenerated the documents with the following.

SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "core/testOnly *SparkThrowableSuite -- -t \"Error classes match with document\""

Why are the changes needed?

The following PR broke CIs by manually fixing the generated docs.

Currently, CI is broken like the following.

[info] - Error classes match with document *** FAILED *** (24 milliseconds)
[info]   "...lstates.html#class-0[A]-feature-not-support..." did not equal "...lstates.html#class-0[a]-feature-not-support..." The error class document is not up to date. Please regenerate it. (SparkThrowableSuite.scala:322)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually check.

$ build/sbt "core/testOnly *.SparkThrowableSuite"
...
[info] SparkThrowableSuite:
[info] - No duplicate error classes (31 milliseconds)
[info] - Error classes are correctly formatted (47 milliseconds)
[info] - SQLSTATE is mandatory (2 milliseconds)
[info] - SQLSTATE invariants (26 milliseconds)
[info] - Message invariants (8 milliseconds)
[info] - Message format invariants (7 milliseconds)
[info] - Error classes match with document (65 milliseconds)
[info] - Round trip (28 milliseconds)
[info] - Error class names should contain only capital letters, numbers and underscores (7 milliseconds)
[info] - Check if error class is missing (15 milliseconds)
[info] - Check if message parameters match message format (4 milliseconds)
[info] - Error message is formatted (1 millisecond)
[info] - Error message does not do substitution on values (1 millisecond)
[info] - Try catching legacy SparkError (0 milliseconds)
[info] - Try catching SparkError with error class (1 millisecond)
[info] - Try catching internal SparkError (0 milliseconds)
[info] - Get message in the specified format (6 milliseconds)
[info] - overwrite error classes (61 milliseconds)
[info] - prohibit dots in error class names (23 milliseconds)
[info] Run completed in 1 second, 357 milliseconds.
[info] Total number of tests run: 19
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

cc @HyukjinKwon

@HyukjinKwon
Copy link
Member

oops

@dongjoon-hyun
Copy link
Member Author

Thank you!

@dongjoon-hyun
Copy link
Member Author

Merged to master to recover the CI.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-46804 branch January 23, 2024 01:57
@nchammas
Copy link
Contributor

nchammas commented Jan 23, 2024

I suppose if we need to do this to get the build passing again, then so be it.

But the anchor links being generated here do not work, at least not on Safari.

I see a few follow-up tasks to this PR:

  • Fix how the links are generated so that the anchor links work correctly.
  • Fix the build so it catches problems like the one introduced in [MINOR][DOCS] Fix SQL Error links and link anchors #44825.
  • Better document this process of generating the error documentation (if it's not documented already somewhere), as it seems several people (myself included, of course) were not aware that it was required, or simply forgot.

@nchammas
Copy link
Contributor

nchammas commented Jan 23, 2024

  • Better document this process of generating the error documentation (if it's not documented already somewhere), as it seems several people (myself included, of course) were not aware that it was required, or simply forgot.

One thing we could do as part of the generation process is insert a comment into each generated file warning that it should not be manually edited. (Edit: Done in #44847.)

Another thing we could do is generate the necessary error documentation files dynamically as part of the documentation build and not have them checked in to git at all. Perhaps this idea was discussed already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants