Create a lightweight "masking engine" FastAPI application

[ThomasLaPiana]

Is your feature request related to a specific problem?

It adds complexity for, in my opinion, very little value that the fides webserver can be run without redis and a database. However I also assume I'm missing some background info here as to why those features were added in the first place.

Describe the solution you'd like

Remove enabled fields for redis and the database. Remove all checks for those components. Fides should have a database and redis as hard requirements to run. For context, currently fidesops is unable to process DSRs without redis.

Describe alternatives you've considered, if any

Leaving as-is, or setting up the masking-engine as its own service within the codebase that can be run individually.

Additional context

From a slack conversation with @adamsachs it sounds like this was added as a POC for a client to use the masking engine as a stand-alone service, but this isn't actually currently being deployed or utilized. I guess an important question here is, how many users would use the masking engine in this way, without any other part of fidesops running?

[ThomasLaPiana]

@NevilleS do you have more context here? Curious if there were any other clients this might be important for?

[NevilleS]

That's the exact reason, yup. We wanted a way to run fides as a stateless masking engine with the dependencies removed, and exclusively use the utility APIs like /api/v1/masking/mask.

[ThomasLaPiana]

@NevilleS what do you think about starting up the server in a specific masking engine mode?

The startup/running of the unified webserver is inherently complex, and by adding conditional checks everywhere we are increasing that complexity. Fides, as a webserver, can't actually run without redis and a database so to me it feels wrong that they share a code path.

Instead, we could define a separate FastAPI application that has a really streamlined and light startup/feature-set that matches exactly what the masking engine needs, and add a fides masking-engine command to the CLI so that users can spin that up specifically in a very lightweight manner. This feels much much better to me than our current solution

[ThomasLaPiana]

@adamsachs would be curious to hear your opinion on my above suggestion as well ^^

[adamsachs]

@ThomasLaPiana i like the idea - the masking engine-only usage pattern is so different from the general fides webserver/app functionality that splitting it out into its own application does seem cleaner than trying to stuff it into the same app. i say this with the caveat that i don't know exactly how this would be done, but i can imagine it :)

looping in @pattisdr as well, who has picked up work on the next iteration of the masking engine

[ThomasLaPiana]

@adamsachs Thanks for confirming! I think the solution would be a masking_engine.py module that defines its own FastAPI application (app = FastAPI(title="masking_engine")) and then only adds the routers/middleware that it needs to run

on the CLI side it looks like from fides.api.masking_engine import start_masking_engine and 'start_masking_engine()`

[ThomasLaPiana]

this is fixed directly in https://github.com/ethyca/fidesops-plus/pull/99

[seanpreston]

Hey @ThomasLaPiana — what's the best way to test this?