3042: Refactor client applications into a monorepo with turborepo, extract fides-js into a standalone package, and improve privacy-center to load configuration at runtime

[eastandwestwind]

Closes #3042

❗ Potential area of risk in deployment- https://github.com/ethyca/fides/pull/3105/files/faab9e7d25dc92c4463b2d3facde7a1858cc64ac#diff-0f3c0f74b23a0b3e4798f66a9b7ffd7f5d850165a10793f9a101235e75f1b61bR36, let's be sure to watch for this at time of deployment to ensure admin-ui is deployed appropriately to pypi

Code Changes

• Integrate Turborepo in /clients so that we can share code between all clients.
• Extracted out fides-consent from privacy-center into a first-class package in clients so that 1) we can share code from fides-consent between all clients, and 2) we're no longer reliant on having to build/run the privacy-center to use fides-consent.
• Renamed fides-consent to fides-js to clarify that it can support other, non-consent functionality in the future (like proxying API calls to Fides, being used for DSR requests, etc.). See Refactor fides-consent and privacy-center to load all configuration at run-time (instead of build-time), and rename fides-consent -> fides-js #3135
• Refactored privacy-center and fides-js to load configuration at run-time, instead of build-time. This allows editing the config.json file provided to the Privacy Center and no longer requiring a npm run build to update the changes. See Refactor fides-consent and privacy-center to load all configuration at run-time (instead of build-time), and rename fides-consent -> fides-js #3135
• Restructured Docker builds across all environments (dev, test, prod) to accommodate the new infra to install/build/run clients together and separately.
• Updated nox commands, gitlab ci commands, and front-end dev docs to call necessary turbo commands.
• Change bundle output of fides-js to fides.js instead of fides.global.js
• Remove duplicate fides-demo.html from fides-js
• Address TODOs for fides-js: document src/fides.ts, cleanup Fides.init, etc.
• Add (empty?) Jest test suite to fides-js
• Fix Cypress & Jest tests for all clients
• Rollback to Rollup for bundling the fides-js package

Follow-up issues:

• Optimize Privacy Center /fides.js download with cache headers and a local public/assets/fides.js file #3170
• Add error-handling and versioned logs to Privacy Center to help admins troubleshoot configuration issues #3171
• Update Privacy Center configuration documentation with new configuration settings, Javascript module, and asset hosting  #3172
• Update main client dependencies to next major versions: React 18, NextJS 13, Chakra 2 #3173
• Optimize all client Docker builds (admin-ui, privacy-center, etc.) to improve build times and reduce image size #3174
• Support fetching privacy-center configuration from external sources (S3, HTTPS API) instead of just file: URLs #3175
• Share the same config files for all client apps: tsconfig, .eslintrc.json, jest.config.js, etc. #3209
• Fix Privacy Center "config" Redux state to not fallback to unsafe defaults, reset on error, etc. #3212
• Add automated Cypress tests for Privacy Center configuration #3216

Dev Changes

If you only use nox commands, continue as before, nothing needed.

If you wish to run any/all clients outside of the nox environment, with Turborepo as part of the codebase, you'll need to do 3 things:

1. Install Turborepo globally so that you can use it on the command line npm install turbo --global
2. In clients folder, run npm install. This will install the new dev dependency turbo, which is basically a wrapper around npm that helps manage shared dependencies. This will also install the appropriate dependencies in clients and in each client within clients folder. You may need to first remove all node_modules in each of the clients (admin-ui, pc, etc)
3. In short, use turbo instead of npm. E.g. instead of running npm run dev, use turbo run dev, etc. All commands that existed before should still exist, given you're using turbo. Let me know if something is broken for you!

Also note that to install packages in any package in clients:

npm install <package> --workspace=<workspace>

Example:

npm install react --workspace=admin-ui

See https://turbo.build/repo/docs/handbook/package-installation#addingremovingupgrading-packages for more details

Steps to Confirm

• All commands used to develop against the project should still work as before

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: -0.01 ⚠️

Comparison is base (8d9ed30) 87.37% compared to head (9149f2f) 87.36%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3105      +/-   ##
==========================================
- Coverage   87.37%   87.36%   -0.01%     
==========================================
  Files         313      313              
  Lines       18243    18248       +5     
  Branches     2366     2368       +2     
==========================================
+ Hits        15940    15943       +3     
- Misses       1871     1872       +1     
- Partials      432      433       +1     

see 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[cypress]

Passing run #1752 ↗︎

0

4

0

0

0

Details:

Merge 9149f2f into 422040d...
Project: fides	Commit: 0711bd8b34 ℹ️
Status: Passed	Duration: 00:57 💡
Started: May 4, 2023 1:39 PM	Ended: May 4, 2023 1:40 PM

_{This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.}

[allisonking]

woo turbo!! nice job working through setting this up!

[TheAndrewJackson]

Things seemed to working well! I agree with what @allisonking said. It's probably worth shipping it and ironing out the kinks this week.

I'm looking forward to using this

[NevilleS]

OK, I think this is ready to go. Lots of follow-up issues to tackle though, so before we merge we'll just need to figure out a safe release plan!