[Feature][history server] support endpoint `/api/v0/logs/file` #4411

machichima · 2026-01-18T02:33:20Z

Why are these changes needed?

Enable getting logs of a specific file through /api/v0/logs/file in history server from either live or dead cluster.

Manual test: #4411 (comment)

Related issue number

Closes #4387

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: machichima <nary12321@gmail.com>

machichima · 2026-01-18T02:36:56Z

Manual test:

Following setup guide here, but skip "7. Delete Ray Cluster (Trigger Log Upload)" step
Get logs from the live cluster

❯ cat ~/cookies.txt
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

localhost       FALSE   /       FALSE   1768701371      session_name    live
localhost       FALSE   /       FALSE   1768701371      cluster_namespace       default
localhost       FALSE   /       FALSE   1768701371      cluster_name    raycluster-historyserver

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl -b ~/cookies.txt "http://localhost:8080/api/v0/logs?node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30"
{"result": true, "msg": "", "data": {"result": {"internal": ["old/", "events/", "export_events/", "ray_client_server.out", "ray_client_server.err", "dashboard_TrainHead.log", "dashboard_TrainHead.out", "dashboard_TrainHead.err", "dashboard_MetricsHead.log", "dashboard_MetricsHead.out", "dashboard_MetricsHead.err", "dashboard_JobHead.log", "dashboard_JobHead.out", "dashboard_JobHead.err", "dashboard_ServeHead.log", "dashboard_ServeHead.out", "dashboard_ServeHead.err", "dashboard_DataHead.log", "dashboard_DataHead.out", "dashboard_DataHead.err", "dashboard_EventHead.log", "dashboard_EventHead.out", "dashboard_EventHead.err", "dashboard_ReportHead.log", "dashboard_ReportHead.out", "dashboard_ReportHead.err", "dashboard_NodeHead.log", "dashboard_NodeHead.out", "dashboard_NodeHead.err", "dashboard_StateHead.log", "dashboard_StateHead.out", "dashboard_StateHead.err", "debug_state.txt", "ray_process_exit.log", "log_monitor.log", "log_monitor.out", "log_monitor.err", "nsight/", "rocprof_sys/", "profiles/", "job-driver-rayjob-9p6gf.log", "runtime_env_setup-01000000.log", "jobs/"], "gcs_server": ["gcs_server.out", "gcs_server.err"], "autoscaler": ["monitor.log", "monitor.out", "monitor.err"], "dashboard": ["dashboard.log", "dashboard.out", "dashboard.err"], "raylet": ["raylet.out", "raylet.err"], "agent": ["dashboard_agent.log", "dashboard_agent.out", "dashboard_agent.err", "runtime_env_agent.out", "runtime_env_agent.err", "runtime_env_agent.log"], "driver": ["python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_525.log", "python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_779.log"], "core_worker": ["python-core-worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90_730.log"], "worker_out": ["worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90-01000000-730.out"], "worker_err": ["worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90-01000000-730.err"]}}}%

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl -b ~/cookies.txt "http://localhost:8080/api/v0/logs/file?node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30&filename=raylet.out&lines=10"
[state-dump]    ray::rpc::JobInfoGcsService.grpc_client.MarkJobFinished.OnReplyReceived - 1 total (0 active), Execution time: mean = 0.01ms, total = 0.01ms, Queueing time: mean = 0.01ms, max = 0.01ms, min = 0.01ms, total = 0.01ms
[state-dump]    ray::rpc::JobInfoGcsService.grpc_client.GetAllJobInfo - 1 total (0 active), Execution time: mean = 0.14ms, total = 0.14ms, Queueing time: mean = 0.00ms, max = -0.00ms, min = 9223372036854.78ms, total = 0.00ms
[state-dump]    NodeManager.GCTaskFailureReason - 1 total (1 active), Execution time: mean = 0.00ms, total = 0.00ms, Queueing time: mean = 0.00ms, max = -0.00ms, min = 9223372036854.78ms, total = 0.00ms
[state-dump]    Subscriber.HandlePublishedMessage_GCS_NODE_ADDRESS_AND_LIVENESS_CHANNEL - 1 total (0 active), Execution time: mean = 0.02ms, total = 0.02ms, Queueing time: mean = 0.05ms, max = 0.05ms, min = 0.05ms, total = 0.05ms
[state-dump]    ray::rpc::InternalKVGcsService.grpc_client.GetInternalConfig.OnReplyReceived - 1 total (0 active), Execution time: mean = 8.80ms, total = 8.80ms, Queueing time: mean = 0.02ms, max = 0.02ms, min = 0.02ms, total = 0.02ms
[state-dump]    ray::rpc::JobInfoGcsService.grpc_client.GetAllJobInfo.OnReplyReceived - 1 total (0 active), Execution time: mean = 0.00ms, total = 0.00ms, Queueing time: mean = 0.00ms, max = 0.00ms, min = 0.00ms, total = 0.00ms
[state-dump]    ray::rpc::InternalKVGcsService.grpc_client.GetInternalConfig - 1 total (0 active), Execution time: mean = 0.24ms, total = 0.24ms, Queueing time: mean = 0.00ms, max = -0.00ms, min = 9223372036854.78ms, total = 0.00ms
[state-dump] DebugString() time ms: 1
[state-dump]
[state-dump]

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl "http://localhost:8080/clusters"

[
 {
  "name": "raycluster-historyserver",
  "namespace": "default",
  "sessionName": "live",
  "createTime": "2026-01-18 01:45:46 +0000 UTC",
  "createTimeStamp": 1768700746
 }
]%

Delete the RayCluster to trigger the upload
Get logs from the dead cluster

❯ SESSION="session_2026-01-17_17-46-19_978420_1"

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-historyserver/$SESSION"

{
 "name": "raycluster-historyserver",
 "namespace": "default",
 "result": "success",
 "session": "session_2026-01-17_17-46-19_978420_1"
}%


~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl -b ~/cookies.txt "http://localhost:8080/api/v0/logs?node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30"
{"data":{"result":{"padding":["dashboard.err","dashboard.log","dashboard.out","dashboard_DataHead.err","dashboard_DataHead.log","dashboard_DataHead.out","dashboard_EventHead.err","dashboard_EventHead.log","dashboard_EventHead.out","dashboard_JobHead.err","dashboard_JobHead.log","dashboard_JobHead.out","dashboard_MetricsHead.err","dashboard_MetricsHead.log","dashboard_MetricsHead.out","dashboard_NodeHead.err","dashboard_NodeHead.log","dashboard_NodeHead.out","dashboard_ReportHead.err","dashboard_ReportHead.log","dashboard_ReportHead.out","dashboard_ServeHead.err","dashboard_ServeHead.log","dashboard_ServeHead.out","dashboard_StateHead.err","dashboard_StateHead.log","dashboard_StateHead.out","dashboard_TrainHead.err","dashboard_TrainHead.log","dashboard_TrainHead.out","dashboard_agent.err","dashboard_agent.log","dashboard_agent.out","debug_state.txt","gcs_server.err","gcs_server.out","job-driver-rayjob-9p6gf.log","log_monitor.err","log_monitor.log","log_monitor.out","monitor.err","monitor.log","monitor.out","python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_525.log","python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff_779.log","python-core-worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90_730.log","ray_client_server.err","ray_client_server.out","ray_process_exit.log","raylet.err","raylet.out","runtime_env_agent.err","runtime_env_agent.log","runtime_env_agent.out","runtime_env_setup-01000000.log","worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90-01000000-730.err","worker-441560b06fac01ae9a1abbf311306319307ce56a12f43fcaa5c9cc90-01000000-730.out","events/","export_events/","jobs/"]}}}%

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl -b ~/cookies.txt "http://localhost:8080/api/v0/logs/file?node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30&filename=raylet.out&lines=10"
reason_message: "received SIGTERM"

[2026-01-17 17:52:04,876 I 495 495] (raylet) accessor.cc:186: Unregistering node node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30
[2026-01-17 17:52:04,878 I 495 495] (raylet) accessor.cc:194: Finished unregistering node info, status = OK node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30
[2026-01-17 17:52:04,878 W 495 510] (raylet) store.cc:365: Disconnecting client due to connection error with code 2: End of file
[2026-01-17 17:52:04,881 I 495 495] (raylet) agent_manager.cc:116: Killing agent dashboard_agent, pid 525.
[2026-01-17 17:52:04,885 I 495 526] (raylet) agent_manager.cc:83: Agent process with name dashboard_agent exited, exit code 0.
[2026-01-17 17:52:04,885 I 495 495] (raylet) agent_manager.cc:116: Killing agent runtime_env_agent, pid 527.
[2026-01-17 17:52:04,886 I 495 528] (raylet) agent_manager.cc:83: Agent process with name runtime_env_agent exited, exit code 0.
[2026-01-17 17:52:04,887 I 495 495] (raylet) stats.h:149: Stats module has shutdown.%

~/workData/open-source/kuberay history-server-logs-file *4 ?1
❯ curl "http://localhost:8080/clusters"
[
 {
  "name": "raycluster-historyserver",
  "namespace": "default",
  "sessionName": "session_2026-01-17_17-46-19_978420_1",
  "createTime": "2026-01-17T17:46:19Z",
  "createTimeStamp": 1768671979
 }
]%

machichima · 2026-01-18T02:37:49Z

historyserver/pkg/historyserver/router.go

 		Doc("get logfile").Param(ws.QueryParameter("node_id", "node_id")).
 		Param(ws.QueryParameter("filename", "filename")).
 		Param(ws.QueryParameter("lines", "lines")).
-		Param(ws.QueryParameter("format", "format")).


There's no "format" query string in ray dashboard /api/logs/file endpoint

https://github.com/ray-project/ray/blob/0781f3c2edc45f897f64086e92980a0a79277855/python/ray/dashboard/modules/state/state_head.py#L220-L239

Should we support other queries (like actor_id, task_id..), or will this be handled in a follow-up?

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

historyserver/pkg/historyserver/router.go

historyserver/pkg/historyserver/reader.go

machichima · 2026-01-18T02:38:50Z

historyserver/pkg/historyserver/reader.go

+	index := 0
+	totalLines := 0
+
+	// Get the last N lines following Ray Dashboard API behavior with circular buffer


ray dashboard /api/logs/file endpoint get the "last N lines" when the line is set:

https://github.com/ray-project/ray/blob/9a8321f9d92d1aed8225256362fb5af74481b458/python/ray/dashboard/modules/log/log_agent.py#L372-L377

Signed-off-by: machichima <nary12321@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

historyserver/pkg/historyserver/reader.go

Signed-off-by: machichima <nary12321@gmail.com>

machichima · 2026-01-18T03:45:27Z

When trying to get the log file with path containing ../, we will raise error directly to prevent path traversal attacks

❯ curl -b ~/cookies.txt "http://localhost:8080/api/v0/logs/file?node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30&filename=../../raylet.out&lines=0"
invalid path: ../ not allowed in the path (node_id=15755942e81843fd6a8ef2a788fb9d9b7605d16643cb03526c52ab30, filename=../../raylet.out)%

machichima · 2026-01-18T03:46:12Z

@Future-Outlier PTAL, thank you!

Future-Outlier

cc @win5923 @AndySung320 @justinyeh1995 to help review

win5923 · 2026-01-20T13:57:46Z

historyserver/pkg/historyserver/router.go

+	// Convert lines parameter to int
+	maxLines := 0
+	if lines != "" {
+		if parsedLines, err := strconv.Atoi(lines); err == nil {
+			maxLines = parsedLines
+		}
+	}


Suggested change

// Convert lines parameter to int

maxLines := 0

if lines != "" {

if parsedLines, err := strconv.Atoi(lines); err == nil {

maxLines = parsedLines

}

}

// Convert lines parameter to int

maxLines := 0

if lines != "" {

parsedLines, err := strconv.Atoi(lines)

if err != nil {

resp.WriteErrorString(http.StatusInternalServerError,fmt.Sprintf("invalid lines parameter: %s", lines))

return

}

}

Fixed in a092446

win5923 · 2026-01-20T14:16:30Z

historyserver/pkg/historyserver/router.go

+		resp.WriteError(400, err)
+		return
+	}
+	resp.Write(content)


Suggested change

resp.Write(content)

resp.Header().Set("Content-Type", "text/plain")

resp.Write(content)

https://github.com/ray-project/ray/blob/22f7f7d85cdfe3b628b3a9e9aa37cf2ae3954820/python/ray/dashboard/modules/state/state_head.py#L264

Fixed in 9fc90c3

win5923 · 2026-01-20T14:26:23Z

historyserver/pkg/historyserver/router.go

+	content, err := s._getNodeLogFile(clusterNameID+"_"+clusterNamespace, sessionName, nodeID, filename, maxLines)
+	if err != nil {
+		logrus.Errorf("Error getting node log file: %v", err)
+		resp.WriteError(400, err)


Suggested change

resp.WriteError(400, err)

resp.WriteError(http.StatusInternalServerError, err)

https://github.com/ray-project/ray/blob/22f7f7d85cdfe3b628b3a9e9aa37cf2ae3954820/python/ray/dashboard/modules/state/state_head.py#L282-L284

Fixed in c986529

win5923 · 2026-01-20T14:59:54Z

historyserver/pkg/historyserver/reader.go

+func (s *ServerHandler) _getNodeLogFile(rayClusterNameID, sessionID, nodeID, filename string, maxLines int) ([]byte, error) {
+	logPath := path.Join(sessionID, "logs", nodeID, filename)
+
+	reader := s.reader.GetContent(rayClusterNameID, logPath)


No need to implement in this PR. For StorageReader, but i think we can introduce a GetContentStream method that returns an io.ReadCloser instead of loading the entire file into memory. This would allow callers to stream content directly and manage resource cleanup explicitly, which is essential for handling large log files efficiently.

Nice suggestions! Will do it in follow-up

win5923 · 2026-01-20T15:46:35Z

historyserver/pkg/historyserver/router.go

+	// Parse query parameters
+	nodeID := req.QueryParameter("node_id")
+	filename := req.QueryParameter("filename")
+	lines := req.QueryParameter("lines")


Maybe we can define an struct, similar to Ray’s GetLogOptions?

While there's only 3 parameters here, I think it's not needed for now. If in the future we want to add support for those options, we can add it. WDYT?

Signed-off-by: machichima <nary12321@gmail.com>

machichima added 3 commits January 17, 2026 10:11

feat: support logs/file endpoint

3180714

Signed-off-by: machichima <nary12321@gmail.com>

refactor: redundant log & var name

7fc3c87

Signed-off-by: machichima <nary12321@gmail.com>

feat: get the last N lines logs

2a75aef

Signed-off-by: machichima <nary12321@gmail.com>

machichima commented Jan 18, 2026

View reviewed changes

cursor bot reviewed Jan 18, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Show resolved Hide resolved

historyserver/pkg/historyserver/router.go Show resolved Hide resolved

historyserver/pkg/historyserver/reader.go Show resolved Hide resolved

machichima commented Jan 18, 2026

View reviewed changes

machichima added 3 commits January 17, 2026 18:45

refactor: typo & unused comment

3f280d2

Signed-off-by: machichima <nary12321@gmail.com>

feat: set default and max log lines

f06a62a

Signed-off-by: machichima <nary12321@gmail.com>

feat: prevent path traversal attacks

ac166a3

Signed-off-by: machichima <nary12321@gmail.com>

cursor bot reviewed Jan 18, 2026

View reviewed changes

historyserver/pkg/historyserver/reader.go Outdated Show resolved Hide resolved

feat: check path traversal attacks for each ele

be433ac

Signed-off-by: machichima <nary12321@gmail.com>

Future-Outlier self-assigned this Jan 19, 2026

Future-Outlier reviewed Jan 20, 2026

View reviewed changes

win5923 self-requested a review January 20, 2026 11:58

win5923 reviewed Jan 20, 2026

View reviewed changes

JiangJiaWei1103 added this to My Kuberay & Ray Jan 20, 2026

JiangJiaWei1103 moved this to To Review in My Kuberay & Ray Jan 20, 2026

machichima added 3 commits January 21, 2026 07:08

fix: raise err if cannot parse lines

a092446

Signed-off-by: machichima <nary12321@gmail.com>

fix: add content type for resp as text

9fc90c3

Signed-off-by: machichima <nary12321@gmail.com>

fix: return internal server error

c986529

Signed-off-by: machichima <nary12321@gmail.com>

	resp.Write(content)
	resp.Header().Set("Content-Type", "text/plain")
	resp.Write(content)

	resp.WriteError(400, err)
	resp.WriteError(http.StatusInternalServerError, err)

[Feature][history server] support endpoint /api/v0/logs/file #4411

Are you sure you want to change the base?

[Feature][history server] support endpoint /api/v0/logs/file #4411

Conversation

machichima commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

machichima commented Jan 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

machichima commented Jan 18, 2026

Uh oh!

machichima commented Jan 18, 2026

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Feature][history server] support endpoint `/api/v0/logs/file` #4411

[Feature][history server] support endpoint `/api/v0/logs/file` #4411

machichima commented Jan 18, 2026 •

edited

Loading

win5923 Jan 20, 2026 •

edited

Loading

win5923 Jan 20, 2026 •

edited

Loading

machichima Jan 20, 2026 •

edited

Loading