Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark 3.5: Adjust repeated INFO logs to DEBUG in SparkWrite and SparkPositionDeltaWrite #12404

Merged
merged 1 commit into from
Mar 24, 2025
Merged

Spark 3.5: Adjust repeated INFO logs to DEBUG in SparkWrite and SparkPositionDeltaWrite #12404

merged 1 commit into from
Mar 24, 2025

Conversation

manuzhang
Copy link
Collaborator

@manuzhang manuzhang commented Feb 26, 2025 •
edited
Loading

I see repeated logs as follows from SparkWrite in Spark driver.

25/02/26 01:48:56 INFO SparkWrite: Requesting 0 bytes advisory partition size for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting 0 bytes advisory partition size for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting 0 bytes advisory partition size for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting UnspecifiedDistribution as write distribution for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting UnspecifiedDistribution as write distribution for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting [] as write ordering for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting [] as write ordering for table ***
25/02/26 01:48:56 INFO SparkWrite: Requesting [] as write ordering for table ***

I think they are more appropriate for debugging purpose, so this PR adjusts the corresponding log levels to DEBUG.

@github-actions github-actions bot added the spark label Feb 26, 2025
@manuzhang manuzhang mentioned this pull request Feb 28, 2025
Open
3 tasks
@manuzhang
Copy link
Collaborator Author

manuzhang commented Mar 24, 2025 •
edited
Loading

@nastra @dramaticlly any more comments?

@nastra nastra merged commit cb7d5a8 into apache:main Mar 24, 2025
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nastra nastra nastra approved these changes

@dramaticlly dramaticlly dramaticlly approved these changes

@aokolnychyi aokolnychyi Awaiting requested review from aokolnychyi

Assignees
No one assigned
Labels
spark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@manuzhang @nastra @dramaticlly