Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Caching for FlowDefExecution #1581

Merged
merged 1 commit into from
Jul 21, 2016
Merged

Use Caching for FlowDefExecution #1581

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions scalding-core/src/main/scala/com/twitter/scalding/Execution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,14 +591,17 @@ object Execution {
* This allows you to run any cascading flowDef as an Execution.
*/
private case class FlowDefExecution(result: (Config, Mode) => FlowDef) extends Execution[Unit] {
protected def runStats(conf: Config, mode: Mode, cache: EvalCache)(implicit cec: ConcurrentExecutionContext) =
Trampoline(
protected def runStats(conf: Config, mode: Mode, cache: EvalCache)(implicit cec: ConcurrentExecutionContext) = {
lazy val future =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this go outside the method? Or does this live across method calls?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it depends on conf and mode so I don't think it can go outside the method. Being lazy just means we don't actually fire this future unless the cache does not already have a run of this Execution with the same Config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, right. I missed the getOrElse part below.

for {
flowDef <- toFuture(Try(result(conf, mode)))
_ = FlowStateMap.validateSources(flowDef, mode)
(id, jobStats) <- cache.runFlowDef(conf, mode, flowDef)
_ = FlowStateMap.clear(flowDef)
} yield ((), Map(id -> ExecutionCounters.fromJobStats(jobStats))))
} yield ((), Map(id -> ExecutionCounters.fromJobStats(jobStats)))

Trampoline(cache.getOrElseInsert(conf, this, future))
}
}

/*
Expand Down