Skip to content

Conversation

@WH77
Copy link
Contributor

@WH77 WH77 commented Jan 16, 2020

Though all Singularity tasks should be able to handle bounces, some can't. Even if they can, it's understandable that users may want some high-importance tasks to avoid bouncing if possible. Therefore, we should allow users to indicate they don't want their tasks to be shuffled without having to edit the configuration and redeploy Singularity.

Concerns:

  • Currently, it's possible for no tasks to be considered for shuffling on overloaded hosts. This is fine for a static configuration, but may change when we switch to a dynamic shuffle blacklist. Though leaving tasks off the blacklist by default is probably enough to keep Singularity in the clear, we should also refactor the shuffling logic to override the blacklist if absolutely necessary.
  • Might be painful to add expiration to shuffle opt outs later.
  • This PR adds a field to SingularityRequestParent.

TODO:

  • Unbreak SingularityUI.
  • Support shuffle opt out as an optional deployment flag.
  • Unit testing of the new manager.
  • Unit testing that task shuffling respects the dynamic shuffle blacklist.
  • Testing on staging.

cc - @ssalinas

William Hou added 28 commits January 9, 2020 16:36
@WH77 WH77 added the WIP label Jan 16, 2020
@WH77 WH77 removed the WIP label Jan 21, 2020
/>
);

const avoidShuffle = (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One note on placement of this. We will likely want to make a button+modal for this in the top bar of the request detail page (example below) rather than putting it on the request form, for two reasons:

  1. We hide the request form from developers in most cases, since, internally, the request json is managed by our deploy api
  2. Given that the request form is meant to essentially represent the request json, and shuffle is under a separate root/api/object, it makes sense to separate shuffle a bit

side note, a nice to have, but maybe we could actually show the button on the task detail page as well when the task in question was killed due to shuffling? Would have to identify that based on the cleanup message in the status updates

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented the side note, but to avoid complicating the redux flow for the TaskDetail page the button will always prompt the user to disable shuffling for the request instead of being a toggle - no technical issues (safe to post the same request id to the blacklist multiple times), but maybe a slight source of confusion.

@WH77 WH77 changed the title Self Service Shuffle Opt Out [WIP] Self Service Shuffle Opt Out Jan 23, 2020
buttonStyle="primary"
formElements={[]}>
<p>{this.renderActionText()} shuffling for {requestIds.length > 1 ? 'these' : 'this'} request{requestIds.length > 1 && 's'}?</p>
<p>Note that Singularity may still bounce your tasks if necessary, such as during host failures.</p>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things, we can probably add a first note that this is specifically 'Disable task shuffling due to memory pressure or cpu overusage', with a follow up of 'Note that this does not prevent all restarts of your task. A task may still be bounced or shuffled for events such as host decommissions'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, let's give this a go in QA 👍

@ssalinas
Copy link
Member

🚢

import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.HEAD;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from a merge conflict?

import java.util.UUID;
import java.util.stream.Collectors;

public class SingularityTaskShuffler {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for splitting this out

@WH77 WH77 merged commit 65138cd into master Jan 28, 2020
@WH77 WH77 deleted the self-service-shuffle-opt-out branch January 28, 2020 17:58
@ssalinas ssalinas added this to the 1.2.0 milestone Jan 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants