Skip to content

ateom: detect actor process exits and expose GetWorkloadHealth - #1202

Open
Zoe Zhao (zoez7) wants to merge 3 commits into
agent-substrate:mainfrom
zoez7:crash-detect-1
Open

ateom: detect actor process exits and expose GetWorkloadHealth#1202
Zoe Zhao (zoez7) wants to merge 3 commits into
agent-substrate:mainfrom
zoez7:crash-detect-1

Conversation

@zoez7

Copy link
Copy Markdown
Collaborator

Part of #292

  • Added a new RPC, GetWorkloadHealth(actor_uid) in ateom.proto so atelet can probe each ateom to get the status.
  • Both gVisor and microVM ateom will monitor the health of the containers inside.

@sfunkenhauser sfunkenhauser left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple small review comments.

A larger overall question though. Do we need to re-implement this logic, or can we instead use the pod informer to get the container status?

Comment thread cmd/ateom-gvisor/main.go Outdated
// wait on during graceful shutdown. The sandbox runs one workload at a time.
type workloadSession struct {
rcmd *runsc
prober containerProber

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like containerProber just exposes a subset of runsc methods. Is that correct? Can we just use the existing rcmd variable instead? If the reason for the interface was to enable testing, let's still combine them and create an interface for all runsc methods.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was using the interface for testing. Created interface for all runsc methods.

Comment thread cmd/ateom-gvisor/main.go
var containersToDelete []string
defer func() {
if retErr != nil {
s.activeActor.Store(nil)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a little weird that we have multiple defer func() that both unset s.activeActor. Are they both necessary?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes both are needed.

  1. If we remove the first defer, any error between line 759 and 786 does not register the cleanup of s.activeActor.
  2. If we remove the second one, during deactivateActorNetworking etc, the activeActor would not be unset yet (due to LIFO of defer functions)

Comment thread counter Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't push the counter binary. Maybe we are missing this from .gitignore?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like our .gitignore already ignores all bin/, I updated my GOBIN env var locally to point there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants