Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-30891][CORE][DOC] Add version information to the configuration of History #27751

Closed

Conversation

beliefer
Copy link
Contributor

@beliefer beliefer commented Mar 1, 2020

What changes were proposed in this pull request?

1.Add version information to the configuration of History.
2.Update the docs of History.

I sorted out some information show below.

Item name Since version JIRA ID Commit ID Note
spark.history.fs.logDirectory 1.1.0 SPARK-1768 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.fs.safemodeCheck.interval 1.6.0 SPARK-11020 cf04fdf#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.fs.update.interval 1.4.0 SPARK-6046 4527761#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.fs.cleaner.enabled 1.3.0 SPARK-3562 8942b52#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e Branch branch-1.3 does not exist, exists in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
spark.history.fs.cleaner.interval 1.4.0 SPARK-5933 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
spark.history.fs.cleaner.maxAge 1.4.0 SPARK-5933 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
spark.history.fs.cleaner.maxNum 3.0.0 SPARK-28294 bbc2be4#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.store.path 2.3.0 SPARK-20642 74daf62#diff-19f35f981fdc5b0a46f070b879a9a9fc  
spark.history.store.maxDiskUsage 2.3.0 SPARK-20654 8b49704#diff-19f35f981fdc5b0a46f070b879a9a9fc  
spark.history.ui.port 1.0.0 SPARK-1276 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058  
spark.history.fs.inProgressOptimization.enabled 2.4.0 SPARK-6951 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc  
spark.history.fs.endEventReparseChunkSize 2.4.0 SPARK-6951 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc  
spark.history.fs.eventLog.rolling.maxFilesToRetain 3.0.0 SPARK-30481 a2fe73b#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.fs.eventLog.rolling.compaction.score.threshold 3.0.0 SPARK-30481 a2fe73b#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.fs.driverlog.cleaner.enabled 3.0.0 SPARK-25118 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.fs.driverlog.cleaner.interval 3.0.0 SPARK-25118 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.fs.driverlog.cleaner.maxAge 3.0.0 SPARK-25118 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.ui.acls.enable 1.0.1 Spark 1489 c8dd132#diff-b49b5b9c31ddb36a9061004b5b723058  
spark.history.ui.admin.acls 2.1.1 SPARK-19033 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.ui.admin.acls.groups 2.1.1 SPARK-19033 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.fs.numReplayThreads 2.0.0 SPARK-13988 6fdd0e3#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.retainedApplications 1.0.0 SPARK-1276 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058
spark.history.provider 1.1.0 SPARK-1768 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e  
spark.history.kerberos.enabled 1.0.1 Spark-1490 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058  
spark.history.kerberos.principal 1.0.1 Spark-1490 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058  
spark.history.kerberos.keytab 1.0.1 Spark-1490 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058  
spark.history.custom.executor.log.url 3.0.0 SPARK-26311 ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b  
spark.history.custom.executor.log.url.applyIncompleteApplication 3.0.0 SPARK-26311 ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b  

Why are the changes needed?

Supplemental configuration version information.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exists UT

@@ -26,53 +26,64 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"

val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
.version("1.1.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1768,commit ID: 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.stringConf
.createWithDefault(DEFAULT_LOG_DIR)

val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
.version("1.6.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-11020, commit ID: cf04fdf#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")

val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
.version("1.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-6046, commit ID: 4527761#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")

val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
.version("1.3.0")
Copy link
Contributor Author

@beliefer beliefer Mar 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-3562, commit ID: 8942b52#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
This item does not in branch-1.3, in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml

.booleanConf
.createWithDefault(false)

val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
.version("1.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-5933, commit ID: 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")

val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
.version("1.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-5933, commit ID: 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")

val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-28294, commit ID: bbc2be4#diff-6bddeb5e25239974fc13db66266b167b

.intConf
.createWithDefault(Int.MaxValue)

val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
.version("2.3.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-20642, commit ID: 74daf62#diff-19f35f981fdc5b0a46f070b879a9a9fc

.stringConf
.createOptional

val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
.version("2.3.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-20654, commit ID: 8b49704#diff-19f35f981fdc5b0a46f070b879a9a9fc

.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")

val HISTORY_SERVER_UI_PORT = ConfigBuilder("spark.history.ui.port")
.doc("Web UI port to bind Spark History Server")
.version("1.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1276, commit ID: 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058

.intConf
.createWithDefault(18080)

val FAST_IN_PROGRESS_PARSING =
ConfigBuilder("spark.history.fs.inProgressOptimization.enabled")
.doc("Enable optimized handling of in-progress logs. This option may leave finished " +
"applications that fail to rename their event logs listed as in-progress.")
.version("2.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-6951, commit ID: 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc

@@ -81,6 +92,7 @@ private[spark] object History {
.doc("How many bytes to parse at the end of log files looking for the end event. " +
"This is used to speed up generation of application listings by skipping unnecessary " +
"parts of event log files. It can be disabled by setting this config to 0.")
.version("2.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-6951, commit ID: 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc

@@ -90,6 +102,7 @@ private[spark] object History {
"By default, all event log files will be retained. Please set the configuration " +
s"and ${EVENT_LOG_ROLLING_MAX_FILE_SIZE.key} accordingly if you want to control " +
"the overall size of event log files.")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-30481, commit ID: a2fe73b#diff-6bddeb5e25239974fc13db66266b167b

@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-30481, commit ID: a2fe73b#diff-6bddeb5e25239974fc13db66266b167b

.internal()
.doubleConf
.createWithDefault(0.7d)

val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-25118, commit ID: 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b

.fallbackConf(CLEANER_ENABLED)

val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-25118, commit ID: 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b

.fallbackConf(CLEANER_INTERVAL_S)

val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-25118, commit ID: 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b

.fallbackConf(MAX_LOG_AGE_S)

val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
.version("1.0.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark 1489, commit ID: c8dd132#diff-b49b5b9c31ddb36a9061004b5b723058

.booleanConf
.createWithDefault(false)

val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
.version("2.1.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-19033, commit ID: 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.stringConf
.toSequence
.createWithDefault(Nil)

val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
.version("2.1.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-19033, commit ID: 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.stringConf
.toSequence
.createWithDefault(Nil)

val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
.version("2.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-13988, commit ID: 6fdd0e3#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)

val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
.version("1.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1276, commit ID: 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058

.intConf
.createWithDefault(50)

val PROVIDER = ConfigBuilder("spark.history.provider")
.version("1.1.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1768, commit ID: 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e

.stringConf
.createOptional

val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
.version("1.0.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark-1490, commit ID: 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058

.booleanConf
.createWithDefault(false)

val KERBEROS_PRINCIPAL = ConfigBuilder("spark.history.kerberos.principal")
.version("1.0.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark-1490, commit ID: 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058

.stringConf
.createOptional

val KERBEROS_KEYTAB = ConfigBuilder("spark.history.kerberos.keytab")
.version("1.0.1")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark-1490, commit ID: 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058

@@ -156,6 +182,7 @@ private[spark] object History {
"some path variables via patterns which can vary on cluster manager. Please check the " +
"documentation for your cluster manager to see which patterns are supported, if any. " +
"This configuration has no effect on a live application, it only affects the history server.")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-26311, commit ID: ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b

@@ -165,6 +192,7 @@ private[spark] object History {
s"${CUSTOM_EXECUTOR_LOG_URL.key}, to incomplete application as well. " +
"Even if this is true, this still only affects the behavior of the history server, " +
"not running spark applications.")
.version("3.0.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-26311, commit ID: ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b

.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")

val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
.version("1.4.0")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-3562, commit ID: 8942b52#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e
This item does not in branch-1.3, in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml

@SparkQA
Copy link

SparkQA commented Mar 1, 2020

Test build #119139 has finished for PR 27751 at commit 68a0aa1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 1, 2020

Test build #119141 has finished for PR 27751 at commit 52c7e28.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

retest this please

@SparkQA
Copy link

SparkQA commented Mar 2, 2020

Test build #119145 has finished for PR 27751 at commit 52c7e28.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

HyukjinKwon pushed a commit that referenced this pull request Apr 5, 2020
… of History

### What changes were proposed in this pull request?
1.Add version information to the configuration of `History`.
2.Update the docs of `History`.

I sorted out some information show below.

Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.history.fs.logDirectory | 1.1.0 | SPARK-1768 | 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.safemodeCheck.interval | 1.6.0 | SPARK-11020 | cf04fdf#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.update.interval | 1.4.0 | SPARK-6046 | 4527761#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.cleaner.enabled | 1.3.0 | SPARK-3562 | 8942b52#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e | Branch branch-1.3 does not exist, exists in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
spark.history.fs.cleaner.interval | 1.4.0 | SPARK-5933 | 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxAge | 1.4.0 | SPARK-5933 | 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxNum | 3.0.0 | SPARK-28294 | bbc2be4#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.store.path | 2.3.0 | SPARK-20642 | 74daf62#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.store.maxDiskUsage | 2.3.0 | SPARK-20654 | 8b49704#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.ui.port | 1.0.0 | SPARK-1276 | 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.fs.inProgressOptimization.enabled | 2.4.0 | SPARK-6951 | 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.fs.endEventReparseChunkSize | 2.4.0 | SPARK-6951 | 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.fs.eventLog.rolling.maxFilesToRetain | 3.0.0 | SPARK-30481 | a2fe73b#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.eventLog.rolling.compaction.score.threshold | 3.0.0 | SPARK-30481 | a2fe73b#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.enabled | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.interval | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.maxAge | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.ui.acls.enable | 1.0.1 | Spark 1489 | c8dd132#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.ui.admin.acls | 2.1.1 | SPARK-19033 | 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.ui.admin.acls.groups | 2.1.1 | SPARK-19033 | 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.numReplayThreads | 2.0.0 | SPARK-13988 | 6fdd0e3#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.retainedApplications | 1.0.0 | SPARK-1276 | 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.provider | 1.1.0 | SPARK-1768 | 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.kerberos.enabled | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.kerberos.principal | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.kerberos.keytab | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.custom.executor.log.url | 3.0.0 | SPARK-26311 | ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.custom.executor.log.url.applyIncompleteApplication | 3.0.0 | SPARK-26311 | ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b |  

### Why are the changes needed?
Supplemental configuration version information.

### Does this PR introduce any user-facing change?
No

### How was this patch tested?
Exists UT

Closes #27751 from beliefer/add-version-to-history-config.

Authored-by: beliefer <beliefer@163.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
@HyukjinKwon
Copy link
Member

Merged to branch-3.0 too

sjincho pushed a commit to sjincho/spark that referenced this pull request Apr 15, 2020
… of History

### What changes were proposed in this pull request?
1.Add version information to the configuration of `History`.
2.Update the docs of `History`.

I sorted out some information show below.

Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.history.fs.logDirectory | 1.1.0 | SPARK-1768 | 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.safemodeCheck.interval | 1.6.0 | SPARK-11020 | cf04fdf#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.update.interval | 1.4.0 | SPARK-6046 | 4527761#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.cleaner.enabled | 1.3.0 | SPARK-3562 | 8942b52#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e | Branch branch-1.3 does not exist, exists in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
spark.history.fs.cleaner.interval | 1.4.0 | SPARK-5933 | 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxAge | 1.4.0 | SPARK-5933 | 1991337#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxNum | 3.0.0 | SPARK-28294 | bbc2be4#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.store.path | 2.3.0 | SPARK-20642 | 74daf62#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.store.maxDiskUsage | 2.3.0 | SPARK-20654 | 8b49704#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.ui.port | 1.0.0 | SPARK-1276 | 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.fs.inProgressOptimization.enabled | 2.4.0 | SPARK-6951 | 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.fs.endEventReparseChunkSize | 2.4.0 | SPARK-6951 | 653fe02#diff-19f35f981fdc5b0a46f070b879a9a9fc |  
spark.history.fs.eventLog.rolling.maxFilesToRetain | 3.0.0 | SPARK-30481 | a2fe73b#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.eventLog.rolling.compaction.score.threshold | 3.0.0 | SPARK-30481 | a2fe73b#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.enabled | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.interval | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.fs.driverlog.cleaner.maxAge | 3.0.0 | SPARK-25118 | 5f11e8c#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.ui.acls.enable | 1.0.1 | Spark 1489 | c8dd132#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.ui.admin.acls | 2.1.1 | SPARK-19033 | 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.ui.admin.acls.groups | 2.1.1 | SPARK-19033 | 4ca1788#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.fs.numReplayThreads | 2.0.0 | SPARK-13988 | 6fdd0e3#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.retainedApplications | 1.0.0 | SPARK-1276 | 9ae80bf#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.provider | 1.1.0 | SPARK-1768 | 21ddd7d#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |  
spark.history.kerberos.enabled | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.kerberos.principal | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.kerberos.keytab | 1.0.1 | Spark-1490 | 866b03e#diff-b49b5b9c31ddb36a9061004b5b723058 |  
spark.history.custom.executor.log.url | 3.0.0 | SPARK-26311 | ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b |  
spark.history.custom.executor.log.url.applyIncompleteApplication | 3.0.0 | SPARK-26311 | ae5b2a6#diff-6bddeb5e25239974fc13db66266b167b |  

### Why are the changes needed?
Supplemental configuration version information.

### Does this PR introduce any user-facing change?
No

### How was this patch tested?
Exists UT

Closes apache#27751 from beliefer/add-version-to-history-config.

Authored-by: beliefer <beliefer@163.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
@beliefer beliefer deleted the add-version-to-history-config branch April 23, 2024 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants