Simplified dockerized runs

[beczkowb]

Simplified dockerized runs

The problem

Running dockerized Dataflow jobs locally, using an image built from the BigFlow project Dockerfile, requires
building the project package and image, and deploying the image to the Google Container registry, so
Dataflow can reach it. It requires understanding what is going on under the hood, from BigFlow users.
It's just too complex and inconvenient in the current form, based on the observations inside Allegro.

Proposed solutions

Adding a new command to the CLI

Adding another command to BigFlow CLI could be a solution for the problem. We can add the run-dockerized
command (or a flag to the standard run command), which is basically an alias for:

bigflow build-package --skip-tests # we have to implement the skip-tests flag btw.
bigflow build-image
bigflow deploy-image
bigflow run ...

Pros:

• super easy to implement
• generic – potentially any other dockerized job can benefit from that command

Cons:

• still, requires some understanding from our users

Extending the Job contract

We can extend the bigflow.Job contract, with an additional field use_docker_image: Union[True, False, DockerImageId].
Then, the run command can automatically build and deploy the image, if use_docker_image==True and bigflow.is_running_locally().

Pros:

• generic
• requires zero understanding from our users

Cons:

• not that easy to implement, especially that it introduces a new is_running_locally concept for the whole framework

Handling the problem inside the BeamJob

We can run build and deploy inside the BeamJob.execute method if use_docker_image==True and bigflow.is_running_locally().

Pros:

• requires zero understanding from our users
• very specific solution for (currently) very specific problem, not leaking to the whole framework

Cons:

• non-generic
• it's weird that the BeamJob.execute runs build and deploy methods under the hood
• requires the is_running_locally() thing

[iwko]

We can extend the bigflow.Job contract, with an additional field use_docker_image: Union[True, False, DockerImageId].
Then, the run command can automatically build and deploy the image, if use_docker_image==True and bigflow.is_running_locally().

What's the difference between True and real DockerImageId?

[beczkowb]

DockerImageId is just a string with a full image id.

[iwko]

For me, the first solution is the best. It is not adding a new field to Job/BeamJob APIs and it is the cheapest to implement, probably more elastic for future changes if we have to add some other special logic for local runs. My only caveat is to the name. I suggest run-local or run --local or complete-run

[beczkowb]

The problem is that it's not automatic. These little non-automatic things accumulate.

[anjensan]

IMO last option (3rd) is no-go, as it will require too much "hackery" and breaking of internal abstraction layers (eg. it will be hard to support).

For option 1 I suppose we can just put such script into documentation (3 command), or even put to documentation how to create an alias for this (instruction how to add oneliner to .bashrc). But custom command might be also ok, but personally I'd prefer just an alias.

Second option is doable, less hacky than 3rd, but IMO still a bit too complex for a problem we are trying to solve.