Create a new Agent scheduler for Agent workload fast scheduling #4804

qi-min · 2025-12-11T08:02:20Z

What type of PR is this?

feature

What this PR does / why we need it:

Add a new scheduler to execute fast path scheduling for workload like AI agent. The Scheduler is focus fast scheduling and scheduling based on Sharding. This PR include basic function to schedule Agent.
For more details please refer to: proposal | design doc

Which issue(s) this PR fixes:

Fixes #
#4722

Special notes for your reviewer:

Does this PR introduce a user-facing change?

gemini-code-assist · 2025-12-11T08:04:06Z

Summary of Changes

Hello @qi-min, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a specialized vc-agent-scheduler to Volcano, aiming to optimize the scheduling of high-throughput AI agent workloads through a fast-path, sharding-based approach. It integrates a new NodeShard Custom Resource Definition for managing dedicated node groups and refines the scheduling queue with advanced priority handling. The changes span across build configurations, Helm chart integration, and a new plugin-based architecture, significantly extending Volcano's capabilities for diverse scheduling demands.

Highlights

New Agent Scheduler Introduction: A new vc-agent-scheduler component has been introduced, specifically designed for fast-path scheduling of workloads like AI agents, focusing on sharding-based scheduling.
Build System and Deployment Integration: The Makefile has been updated to include targets for building the new vc-agent-scheduler binary and its Docker image. The Volcano Helm chart now includes templates and configurations to deploy this new scheduler.
New Custom Resource Definition (CRD): A NodeShard CRD (nodeshards.shard.volcano.sh) has been added to manage collections of nodes dedicated to specific schedulers, enabling more granular control over resource allocation.
Enhanced Scheduling Queue: The agent scheduler incorporates a modified Kubernetes-derived scheduling queue, featuring priority handling for binding conflicts and removal of NominatedNode logic for streamlined operation.
Plugin Framework for Agent Scheduler: A dedicated plugin framework has been established for the agent scheduler, including predicates and nodeorder plugins, allowing for extensible and customizable scheduling policies.
Node Information and Snapshot Improvements: The NodeInfo structure in pkg/scheduler/api now includes Generation and BindGeneration fields to facilitate conflict checking during binding. The snapshot mechanism has been enhanced to include Volcano's NodeInfo and support incremental updates.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new agent scheduler designed for fast-path scheduling of workloads like AI agents. This is a significant feature addition that includes a new scheduler binary, associated CRDs like NodeShard, and a conflict-aware binding mechanism to improve scheduling reliability. The implementation thoughtfully reuses and adapts components from the existing Volcano scheduler framework. Additionally, the PR refactors the predicates and nodeorder plugins to better align with the upstream Kubernetes scheduler framework, which is a commendable improvement for long-term maintainability. My review identified a couple of minor issues, including an ungraceful panic on initialization error and a minor copy-paste error in a file header. Overall, this is a well-structured and substantial contribution towards enhancing Volcano's scheduling capabilities.

cmd/agent-scheduler/app/server.go

pkg/agentscheduler/actions/factory.go

hzxuzhonghu · 2025-12-15T08:28:29Z

cmd/agent-scheduler/app/server.go

+	"volcano.sh/apis/pkg/apis/helpers"
+	"volcano.sh/volcano/cmd/agent-scheduler/app/options"
+	scheduler "volcano.sh/volcano/pkg/agentscheduler"
+	"volcano.sh/volcano/pkg/agentscheduler/metrics"
+	"volcano.sh/volcano/pkg/kube"
+	"volcano.sh/volcano/pkg/signals"
+	commonutil "volcano.sh/volcano/pkg/util"
+
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/util/uuid"
+	clientset "k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/kubernetes/scheme"
+	corev1 "k8s.io/client-go/kubernetes/typed/core/v1"
+	"k8s.io/klog/v2"
+
+	utilfeature "k8s.io/apiserver/pkg/util/feature"
+
+	restclient "k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/leaderelection"
+	"k8s.io/client-go/tools/leaderelection/resourcelock"
+	"k8s.io/client-go/tools/record"
+	basecompatibility "k8s.io/component-base/compatibility"
+
+	// Register rest client metrics
+	_ "k8s.io/component-base/metrics/prometheus/restclient"


nit: first go std lib and seconde group thirdparty pkg together, and at last group volcano pkg together

sure, I changed the order

hzxuzhonghu · 2025-12-15T08:57:53Z

pkg/scheduler/plugins/nodeorder/nodeorder.go

 )

-type nodeOrderPlugin struct {
+type NodeOrderPlugin struct {


is this just a refactor or any other changes, can you point it

For changes under scheduler/plugins are just refactor, like abstract functions, so that it can be used in both batch scheduler and agent scheduler.
Modification on line 69 is to allow this plugin resued in agent-scheduler package

hzxuzhonghu · 2025-12-15T09:12:51Z

pkg/scheduler/plugins/predicates/predicates.go

this seems a refactor, if there are any other change, please point it to simplify review

change in predicate is redactor, to allow the functions can be used in both agent-scheduler and batch scheduler

hzxuzhonghu · 2025-12-15T09:21:02Z

pkg/scheduler/plugins/predicates/predicates.go

-			node, found := nodeMap[nodeName]
-			if !found {
-				klog.Errorf("predicates, update pod %s/%s allocate from NOT EXIST node [%s]", pod.Namespace, pod.Name, nodeName)
+			node, err := pp.Handle.SnapshotSharedLister().NodeInfos().Get(nodeName)


I am concerned about this change, previously it get nodes from the session, now it get from the informer list.

Will it bring some caveats?

https://github.com/qi-min/volcano/blob/5e134b9d039065acea682f15132a49f3f05d16fe/pkg/scheduler/plugins/predicates/predicates.go#L190-L191 Actually we still use the nodemap in session here, there's a judgment here in NewFramework:
https://github.com/qi-min/volcano/blob/5e134b9d039065acea682f15132a49f3f05d16fe/pkg/scheduler/plugins/util/k8s/framework.go#L223-L227
so the logic of getting the nodemap from the session hasn't changed

hzxuzhonghu · 2025-12-15T09:31:22Z

pkg/agentscheduler/scheduler.go

+		}
+	}
+
+	cache := schedcache.New(config, opt.SchedulerNames, opt.NodeSelector, opt.NodeWorkerThreads, opt.ResyncPeriod)


we donot need multiple SchedulerNames

ok, changed it to use single name

hzxuzhonghu · 2025-12-15T09:39:18Z

pkg/agentscheduler/scheduler.go

+		worker := &Worker{}
+		worker.framework = framework.NewFramework(sched.actions, sched.tiers, sched.cache, sched.configurations)
+		index := i
+		go wait.Until(func() { worker.runOnce(index) }, 0, stopCh)


When there are no any pods to schedule, wouldn't it run into deadloop?

When no pod to schedule, runOnce will be blocked at Popping pod from queue until new Pending pod enqueue.

You mean the queue is blocking read, that's ok

hzxuzhonghu · 2025-12-15T09:49:34Z

pkg/agentscheduler/cache/binder.go

+	task := scheduleResult.SchedCtx.Task
+	// 1. Check conflict
+	node := binder.FindNonConflictingNode(scheduleResult)
+	if node == nil {


how many pods are scheduled once, when could node == nil？

One pod is scheduled in each worker at once. The scheduleResult(candicate nodes, default is 3) are sent to binder from worker. FindNonConflictingNode check whether theses nodes confilct with previous binding. If the version of node is not used in previsous binding, node will be used for binding. Otherwise node will be rejected. Nil is returned if all candidates nodes are rejected.

E.g. candidate nodes are n1(v1), n2(v1), n3(v2). If nodes with these versions are used in previous binding, then all candidates will be dropped and nil returned. If previsous binding on n3 is at version v1, then n3 will be used.

hzxuzhonghu · 2025-12-15T10:37:50Z

pkg/agentscheduler/cache/binder.go

+		default:
+		}
+		if len(binder.BindCheckChannel) == 0 {
+			break


I am also wondering the cpu usage of not any pod need to be binded

the function is updated, it will be blocked if no result sent to channel

hzxuzhonghu · 2025-12-15T10:47:49Z

pkg/agentscheduler/cache/binder.go

+		return
+	}
+
+	binder.schedulingQueue.Done(task.Pod.UID) // Mark previous processing as done to clean in-flight records, avoid memory leak


Done should be called in pair with Pop, otherwise it is very easy to miss calling Done

Although retries based on events are not yet implemented, if we call Done immediately after Pop,if scheduling fails, the pod will no longer be in the inflight list:

volcano/third_party/kubernetes/pkg/scheduler/backend/queue/active_queue.go

Lines 392 to 406 in 76c3253

inFlightPod, ok := aq.inFlightPods[pInfo.Pod.UID]

if !ok {

return nil, fmt.Errorf("in flight Pod isn't found in the scheduling queue. If you see this error log, it's likely a bug in the scheduler")

}

var events []*clusterEvent

for event := inFlightPod.Next(); event != nil; event = event.Next() {

e, ok := event.Value.(*clusterEvent)

if !ok {

// Must be another in-flight Pod (*v1.Pod). Can be ignored.

continue

}

events = append(events, e)

}

return events, nil

Event changes that occur during the scheduling of this pod will prevent the pod from retrying quickly.
Currently, after Pop, Done is handled without omission. When scheduling fails, Done is called in AddUnschedulableIfNotPresent, and Done is also called when CheckAndBind is reached after successful scheduling, so we didn't miss it

hzxuzhonghu · 2025-12-15T11:11:29Z

pkg/agentscheduler/cache/cache.go

what is the different with pkg/scheduler/cache/cache.go. ducoment it and add a TODO: can we merge them or anstract the common parts

In agent scheduler, we used informers, nodeinfo, nodeList, Binder from scheduler cache and also add/modify several fields and function like confictAwareBinder, scheduling queue, etc. I will add a TODO to abstract common parts from agent/batch scheduler

hzxuzhonghu · 2025-12-16T08:01:27Z

cmd/agent-scheduler/main.go

+	_ "volcano.sh/volcano/pkg/agentscheduler/plugins"
+
+	// init assert
+	_ "volcano.sh/volcano/pkg/scheduler/util/assert"


remove, this should only occur in test

hzxuzhonghu · 2025-12-16T08:08:43Z

cmd/agent-scheduler/app/options/options.go

+)
+
+// ServerOption is the main context object for the controller manager.
+type ServerOption struct {


This is mostly copied from scheduler and RegisterOptions has some coupling with scheduler. set ServerOpts.EnableCSIStorage

Could we eliminate that

the plugins in batch scheduler depend on options.ServerOpts, but ServerOpts is a global variable at cmd/scheduler /app/options level. So it is not inited with flag values when agent scheduler startup. RegisterOptions pass options to options.ServerOpts so that it can pick up value from flags of agent scheduler.
In next pr for sharing, the logic is changed to pass all options of agented scheduler to batch scheduler, no need to specially specify EnableCSIStorage
voptions.ServerOpts = ServerOpts.ServerOption

hzxuzhonghu · 2025-12-16T08:09:53Z

cmd/agent-scheduler/main.go

+
+	commonutil.LeaderElectionDefault(&s.LeaderElection)
+	s.LeaderElection.ResourceName = commonutil.GenerateComponentName(s.SchedulerNames)
+	componentbaseoptions.BindLeaderElectionFlags(&s.LeaderElection, fs)


I would prefer moving the leader elector related to app.Run

hzxuzhonghu · 2025-12-16T08:12:03Z

cmd/agent-scheduler/app/server.go

+	}
+
+	// Align default feature-gates with the connected cluster's version.
+	if err := setupComponentGlobals(config); err != nil {


what does this do

seems it is used to discover kube server version and set feature gate defaults to match the cluster

hzxuzhonghu · 2025-12-16T08:16:21Z

cmd/agent-scheduler/app/server.go

+		return err
+	}
+
+	metrics.InitKubeSchedulerRelatedMetrics()


backoffqueue depend on the metrics, but i donot think we ever update them.

yes, the scheduling queue we imported in third-party path depend on the metrics, nil error is reported if metrics are not inited. I think we need a TODO decouple the kube metrics and record metrics.

just found batch scheduler also call this InitKubeSchedulerRelatedMetrics to init metrics few weeks ago for #4731

hzxuzhonghu · 2025-12-16T09:23:27Z

pkg/agentscheduler/cache/factory.go

+// Register registers or updates a binder for the given plugin name . The plugin can be such as preBinder or postBinder.
+// It always overwrites the existing binder map to support plugin configuration updates
+// during runtime, as plugins may be reconfigured without restarting the scheduler.
+func (r *BinderRegistry) Register(name string, binder interface{}) {


Note: this is not called, so volume binding check is not supported

it is called in OnPluginInit in predicate plugin

hzxuzhonghu · 2025-12-16T09:25:49Z

pkg/agentscheduler/framework/interface.go

+	OnActionInit(configurations []conf.Configuration)
+
+	// Initialize initializes the allocator plugins.
+	Initialize()


@hzxuzhonghu what need be merged?

I mean OnActionInit and OnActionInit can be merged

hzxuzhonghu · 2025-12-16T09:30:23Z

pkg/agentscheduler/scheduler.go

+		conf.EnabledActionMap[action.Name()] = true
+	}
+
+	schedCtx, err := worker.generateNextSchedulingContext()


You can add a for loop here instead of depending on wait.Until(func() { worker.runOnce(index) }, 0, stopCh) to do that in function granularity, which could be less efficient and produce confusing logs

good suggestion, I will change it

hzxuzhonghu · 2025-12-16T09:32:22Z

pkg/agentscheduler/scheduler.go

+		worker := &Worker{}
+		worker.framework = framework.NewFramework(sched.actions, sched.tiers, sched.cache, sched.configurations)
+		index := i
+		go wait.Until(func() { worker.runOnce(index) }, 0, stopCh)


You mean the queue is blocking read, that's ok

hzxuzhonghu · 2025-12-16T09:45:45Z

pkg/agentscheduler/actions/allocate/allocate.go

+		SchedCtx:       schedCtx,
+		BindContext:    alloc.CreateBindContext(schedCtx),
+	}
+	alloc.EnqueueSchedulerResultForTask(result)


seems we need a more clear name, like sendToBinder

wangyang0616 · 2025-12-17T08:19:38Z

pkg/agentscheduler/actions/allocate/allocate.go

+	agentapi "volcano.sh/volcano/pkg/agentscheduler/api"
+	"volcano.sh/volcano/pkg/agentscheduler/framework"
+	"volcano.sh/volcano/pkg/scheduler/api"
+	vcache "volcano.sh/volcano/pkg/scheduler/cache"


The agentscheduler replicates the implementations of cache and framework, but it also references the cache and framework under the scheduler directory here. What are the principles for distinguishing between the cache and framework components in these two directories?

Some structure definition is reused from batch scheduler, that's why scheduler/cache is also imported. We need to abstract the common part of cache for batch/agent scheduler, so only common cache will be referenced in future. I will add a TODO

It is recommended to create an independent issue tracking system for optimization.

wangyang0616 · 2025-12-17T08:41:33Z

pkg/agentscheduler/framework/framework.go

+	PrePredicateFns   map[string]api.PrePredicateFn
+	NodeOrderFns      map[string]api.NodeOrderFn
+	BatchNodeOrderFns map[string]api.BatchNodeOrderFn
+	NodeMapFns        map[string]api.NodeMapFn


Do the two callback functions NodeMapFns and NodeReduceFns have specific scenario requirements in the agent-scheduler context, or are they merely intended to align with the existing scheduling logic?

Just to align with current scheduling logic so that the existing plugins can be reused

wangyang0616 · 2025-12-17T08:47:04Z

pkg/agentscheduler/framework/framework.go

+
+// Framework manages the scheduler plugins and their execution points.
+type Framework struct {
+	*k8sutil.Framework // Embedding Framework to implement k8sframework.Handle interface


The k8sutil package belongs to the common directory under scheduler/plugin, and it is currently intended to be referenced by multiple algorithm plugins. Is it appropriate for the current agentscheduler to reference it directly?

*k8sutil.Framework defined some new fields and functions that are also need in agent scheduler, that's why it is referenced
snapshot *Snapshot

This is not a good way to depend on each other. Normally, plugins should be able to reference the cache and framework packages, but the framework should not depend on the plugin packages in return. It is recommended to register a separate issue for rectification and optimization.

wangyang0616 · 2025-12-17T09:33:58Z

pkg/agentscheduler/scheduler.go

+	sched.tiers = tiers
+	sched.configurations = configurations
+	sched.metricsConf = metricsConf
+	sched.mutex.Unlock()


Would it be better to use defer sched.mutex.Unlock() here, to avoid the risk of deadlock caused by the continuous addition of code in the future?

Signed-off-by: Haoran <97868579@qq.com>

Signed-off-by: handan-yxh <yxh953167434@163.com>

… deploy Signed-off-by: qi-min <qim_34@163.com>

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

Signed-off-by: handan-yxh <yxh953167434@163.com>

Signed-off-by: qi-min <qim_34@163.com>

qi-min · 2025-12-18T08:55:59Z

@hzxuzhonghu @wangyang0616 review comments have been handled, would you please review again, thanks.

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

wangyang0616 · 2025-12-19T13:55:49Z

/lgtm

wangyang0616 · 2025-12-19T13:56:08Z

Please confirm the closure status of the review comments. @hzxuzhonghu

volcano-sh-bot · 2025-12-22T02:26:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

volcano-sh-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2025

volcano-sh-bot requested review from k82cn, merryzhou and wpeng102 December 11, 2025 08:02

volcano-sh-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 11, 2025

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

cmd/agent-scheduler/app/server.go Show resolved Hide resolved

pkg/agentscheduler/actions/factory.go Outdated Show resolved Hide resolved

qi-min force-pushed the agent-scheduler-development branch from 91044dc to 4506d5e Compare December 11, 2025 08:07

qi-min mentioned this pull request Dec 15, 2025

Add a new Scheduler to support Agent workload scheduling #4787

Closed

hzxuzhonghu reviewed Dec 15, 2025

View reviewed changes

Copilot AI mentioned this pull request Dec 16, 2025

Review summary for PR #4804: Agent scheduler implementation issues #4820

Closed

hzxuzhonghu reviewed Dec 16, 2025

View reviewed changes

qi-min changed the title ~~[WIP]Create a new Agent scheduler for Agent workload fast scheduling~~ Create a new Agent scheduler for Agent workload fast scheduling Dec 17, 2025

volcano-sh-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 17, 2025

wangyang0616 reviewed Dec 17, 2025

View reviewed changes

0YHR0 and others added 7 commits December 18, 2025 16:52

agent scheduler: create nodeshard CRD

4e771a9

Signed-off-by: Haoran <97868579@qq.com>

agent scheduler: setup agent scheduler application

c2a01b1

Signed-off-by: handan-yxh <yxh953167434@163.com>

agent scheduler: update build and deploy scripts to support build and…

2535030

… deploy Signed-off-by: qi-min <qim_34@163.com>

agent scheduler: setup scheduler framework

e541a0f

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

agent scheduler: support clone nodes from cache to snapshot in framework

eb01ce6

Signed-off-by: handan-yxh <yxh953167434@163.com>

agent scheduler: add action and support multi worker scheduling

f353059

Signed-off-by: qi-min <qim_34@163.com>

agent-scheduler: updated based on comments

603f68e

Signed-off-by: qi-min <qim_34@163.com>

qi-min force-pushed the agent-scheduler-development branch from 394f15d to f14eca0 Compare December 18, 2025 08:54

qi-min force-pushed the agent-scheduler-development branch from f14eca0 to 74efa74 Compare December 19, 2025 01:51

agent-scheduler: updated based on 2nd round review comments

6363d12

Signed-off-by: JesseStutler <chenzicong4@huawei.com>

JesseStutler force-pushed the agent-scheduler-development branch from 74efa74 to 6363d12 Compare December 19, 2025 08:35

volcano-sh-bot assigned wangyang0616 Dec 19, 2025

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Dec 19, 2025

hzxuzhonghu approved these changes Dec 22, 2025

View reviewed changes

volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 22, 2025

volcano-sh-bot merged commit 49be4a2 into volcano-sh:master Dec 22, 2025
22 of 23 checks passed

	inFlightPod, ok := aq.inFlightPods[pInfo.Pod.UID]
	if !ok {
	return nil, fmt.Errorf("in flight Pod isn't found in the scheduling queue. If you see this error log, it's likely a bug in the scheduler")
	}

	var events []*clusterEvent
	for event := inFlightPod.Next(); event != nil; event = event.Next() {
	e, ok := event.Value.(*clusterEvent)
	if !ok {
	// Must be another in-flight Pod (*v1.Pod). Can be ignored.
	continue
	}
	events = append(events, e)
	}
	return events, nil

Create a new Agent scheduler for Agent workload fast scheduling #4804

Create a new Agent scheduler for Agent workload fast scheduling #4804

Conversation

qi-min commented Dec 11, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

gemini-code-assist bot commented Dec 11, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment