feat(sdk): live logs for jobs

[tejapulagam]

Summary

Currently the SDK has no way to stream a job's logs. Job.logs only returned the finished log and raised an error while the job was running. Live logs existed in lightning deployment logs only.

This PR adds a way to stream a job's logs live from the Python SDK.
Job.stream_logs(follow=True, tail=..., rank=..., idle_timeout=...) yields log lines as a generator, backed by JobApiV2.stream_logs that follows the controlplane log websocket.

demo: https://www.loom.com/share/2017cb29696542bc96d20c91d03cb940

I will follow up with a lightning job logs CLI command.

Usage

from lightning_sdk import Job
job = Job(name="my-job")
# Follow logs live until the job finishes (Ctrl-C to stop).
for line in job.stream_logs():
    print(line)
# Only the last 100 lines, then follow.
for line in job.stream_logs(tail=100):
    print(line)
# Stop automatically after 30s with no new output (good for scripts).
for line in job.stream_logs(idle_timeout=30):
    print(line)
# A specific rank of a distributed job.
for line in job.stream_logs(rank=1):
    print(line)

[lianakoleva]

awesome! 3 comments. please add testing as well.

[justusschock]

how does job.stream_logs() behave when calling it twice? will it only return the logs emitted after the first one or all of them?

Also should we unify this with .logs? so depending on the state, it will either do the streaming or all of them? all the flags (tail, rank & follow) should work with both of them

[tejapulagam]

how does job.stream_logs() behave when calling it twice? will it only return the logs emitted after the first one or all of them?

Also should we unify this with .logs? so depending on the state, it will either do the streaming or all of them? all the flags (tail, rank & follow) should work with both of them

On calling twice: it's stateless, each call will open a fresh stream starting from the job's creation time (or the last tail N lines), so you get the full logs again, not just the delta since the last call. We don't persist a cursor between calls.

stream_logs is no longer exposed, everything is under job.logs, which dispatches on state:
finished -> saved logs
running -> live snapshot
running + follow=True → live stream
print(job.logs) gives a snapshot string; job.logs(follow=True, tail=...) returns an iterator of lines