Add make(dest) to TypedPipe

[avibryant]

This allows (very) simple make-like behavior with Execution, where a writeThrough will just turn into a read if the destination already exists.

Note that since validateTaps isn't defined for local mode, this won't work at all in that context. It may be reasonable to require that to be fixed as part of this PR.

Also, it obviously needs a couple of tests before it can be merged.

[johnynek]

This idea is really nice. People have reimplemented this exact function at least twice internally at Twitter.

I agree that shipping this for Local mode but it being broken is not okay. We could either always compute in Local mode (lame) or we could make Local mode properly support validateTaps everywhere.

Actually, I didn't know that validateTaps is broken in that case. That deserves its own issue. Do you have more details?

[avibryant]

Yeah, in Local mode validateTaps is always a no-op, which means it will never compute in Local mode, but presumably it will then crash down the line. See:
https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/FileSource.scala#L183

@johnynek what do you think of the name, and do we need a variation that returns Execution[Unit]? (I kinda think we don't, and that people can just chain .unit if they want).

[johnynek]

@avibryant I think there is no need for an Execution[Unit] version (agree about .unit). I like the name make since it reminds us of the make program.

[reconditesea]

I also like make program :) But will .readOrMake be more informative about what does it do?

[avibryant]

Updated FileSource to do the right thing in validateTaps in Local mode.
Also changed make to require a FileSource because for now it's very unlikely the semantics will be right for any other Source, and I'd rather protect people from confusion.

[joshualande]

@avibryant, thanks for working on this. This will be a great feature.

fwiw, it seems like the main use-case for this tool is ad-hoc work in the Scalding REPL(?).

The name that would make the most sense to me is .writeOrLoad (or .saveOrLoad) so that the API looks similar to the .write/.save functions.

[avibryant]

@joshualande actually, I hadn't been thinking of REPL use-cases (though that makes sense now that you mention it). I was thinking more about restartable data pipelines with complex dependency graphs - what you might use luigi or oozie for currently.

[johnynek]

you want Local(true) here right? strict = true?

[johnynek]

I guess not. The previous (perhaps questionable) logic is that strict means something is there, but maybe not all the paths. Nothing being present still won't run. Since currently Local only supports one path (even though it could support more), I suppose this is correct.

[DanielleSucher]

Ping! Anything else y'all want before this feels ready to merge? ^^

[johnynek]

Let's shipit!

[johnynek]

closes #1126

[coveralls]

Changes Unknown when pulling 1314460 on avibryant:avi-make into * on twitter:develop*.