Another big upstream merge (#1111)

* Prevent modifying default taxonomy fields (#990)

* Add is_default field to taxonomy sql models (#976)

* Add is_default field to taxonomy sql models

* Update changelog

* Autoformat alembic migration file

* Update dataset.yml with is_default

* Bump fideslang version

Update changelog

Autoformat alembic migration file

Update dataset.yml with is_default

Bump fideslang version

* Prevent modifying default taxonomy on update

Add docstring

* Move is_default check to routes/crud.py

Revert database/crud.py changes

* Handle forbidding when given a list of fides keys

* Add test

* Handle attempting to modify is_default field

* Update changelog

* Debug failing test

* Add another print debugging stmt

* Revert "Add another print debugging stmt"

This reverts commit 527bd03.

* Revert "Debug failing test"

This reverts commit daf3ca8.

* Scope resources_dict to function

* Clean up changelog

* Allow modifying defaults but not is_default

* Handle case where checking for new upsert

* Refactor to put tests in a class

* Handle upserting is_default

* Delete taxonomy UI (#1006)

* Add delete call to slices

* Hook up delete button

* Add result handling

* Only show delete on nodes without children

* Add tests for delete

* Update changelog

* Render action buttons as a prop to AccordionTree

* Fix import consistency

* Rename onEdit and onDelete

* Clear edit entity on delete

* Use TreeNode type

* Update cypress fixtures (#1022)

* Update fixtures

* Update tests based on updated fixtures

* Add taxonomy entity form (#1019)

* Rename data-categories.slice --> taxonomy.slice

* Add active taxonomy type to store

* Add create mutation to slices

* Hook up create to form

* Fixup form UX

* Derive parent key from fides key

* Conditionally render parent key field

* Add tests for adding taxonomy entities

* Clean up

* Update changelog

* Simplify setting add state

* Add test for showing either add or create form

* Derive isCreate from status of fides key

* Add check for is_default before rendering delete button (#1023)

* Add check for is_default before rendering delete button

* Update changelog

* Update test data to include is_default

* Add cypress tests

* Fix mypy error (#1030)

* Fix mypy error

* Update CHANGELOG

* Remove unused import

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Custom label for user defined taxonomy fields (#1027)

* Add renderTag prop

* Add cypress tests

* Update changelog

* Add boolean fields and use them in taxonomy forms (#1028)

* Add CustomRadioGroup as a form input

* Use new radio group input for taxonomy forms

* Update extra form fields prop to be a function

* Cast string boolean back to real boolean

* Tests

* Update changelog

* Clean up some type comparisons

* Revert === undefined since the fields can actually be null

* remove exclude_unset=True to return clean diff (#1026)

* remove exclude_unset=True to return clean diff

* changelog

* don't stop the test suite when there is a failure

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Set pydantic < 1.10.0 to fix CI issues with fideslang functions (#1045)

Should be able to revert after fideslang figures out why the latest
pydantic causes failures

* [fidesctl-plus #78] cross app navigation (#1037)

* fctl/nav: Move NavBar into nav directory

* fctl/nav: Decouple Header and NavBar

* fctl/nav: Extract NavButton presentational component

* fctl/nav: Extract NavLink component

* fctl/zones: Generic utility for configuring app zones

This change brings in some zone handling code from fidesctl-plus and makes it more generic so that
we can use the same pattern in both apps.

* fctl/nav: Use zone-aware nav links

* fctl/nav: Replace Jest nav tests with Cypress tests

This change switches moves our nav routing tests into Cypress instead of Jest.
The React test renderer was having trouble integrating with Next's dynamic import,
which was going to require a complex solution. Instead, by moving the nav tests into
Cypress we can test the real router state without relying on mocks.

This will prove even more useful in a later commit when we update what links are available
based on whether the API says we are in fidesctl-plus.

* fctl/features: Query /plus/health API to determine if Plus features should be shown

* Update changelog

* 1.8.3 (#1050)

* Prepare changelog for 1.8.3 release

* Fixup misattributed changelog items

* Add fix for pydantic version

* noxfiles: Session for building fidesctl python package (#1047)

* noxfiles: Session for building fctl python package

* fix a pylint error

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* [942] fctl/api: Serve static files using route maps (#1046)

* fctl/api: Adapt generate_route_file_map from Ops and unit test it

* fctl/api: Serve static files using route maps

This includes both packaged (pip installed) and local build files, depending on
what is available.

* Update changelog

* Fix truncated evaluation error messages (#1053)

* Fix truncated evaluation error messages

* add a test for the evaluations output

* fix mypy error

* update changelog

* Bump next-auth from 4.9.0 to 4.10.3 in /clients/ctl/admin-ui (#1025)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.9.0 to 4.10.3.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.9.0...next-auth@v4.10.3)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Allison King <allisonjuliaking@gmail.com>

* Cascade delete taxonomy children (#1042)

* Configure pytest for async functions

* Refactor FixtureRequest so it can be shared across tests

* Add test for deleting children taxonomy

* Add logic to cascade delete

* Update changelog

* Update UI to allow deleting parent taxonomy fields with a warning

* Clean up

* Update cypress tests

* Initial systems management page (#1054)

* Rename SystemsTable to SystemsGrid

* Add BorderGrid component to abstract out grid

* Build out SystemCard

* Add search feature

* Pull SystemCard out into its own component

* Add cypress tests for system management page

* Update changelog

* Fix nav bar test

* 1.8.4 (#1061)

* update changelog

* remove phantom bullet point

* Alter taxonomy upsert behavior (#1040)

* Configure pytest for async functions

* Add tests for changing updating default taxonomy

* Change upsert behavior to append

* Update changelog

* Address PR comments

* Replace upsert with create

* Remove unused import

* Remove disabled Nav Buttons (#1067)

* Update NavBar.tsx

* remove "more" from the navbar

* fix a linting error

* remove cypress tests for disabled links

* Form to add a system via yaml (#1062)

* Add new system page

* Refactor YamlForm so it can be reused

* Fixup some types in existing system slice

* Add SystemYamlForm

* Add cypress test for system

* Update changelog

* Fix empty state and remove ellipsis for now

* Remove tests on more actions button

* Move changelog items to Unreleased

* ui/dataset: "Classify" toggle for fidesctl-plus (#1057)

* ui/dataset: Spacing improvements for generate form

* ui/inputs: CustomSwitch component for switches

* ui/dataset: "Classify" toggle for fidesctl-plus

* ui/dataset: Cypress test when classify is available

* Update changelog

* [1058] ui/dataset: Confirmation modal to kick off classify (#1069)

* ui/featuers: Extract Plus API into its own slice

* ui/dataset: Refactor dataset creation chain of mutations

This should make it easier to extend the sequence of API calls to add Classify.
Before, the structure made it hard to identify the orders in which mutations were called,
and which object was the generated (temporary) vs persisted dataset. Now the generate
and create functions return their results or an error string.

I also made a change to how the error message is shown: instead of always assigning
the message to the form's URL field, it's shown in a error toast. Now that we have
multiple fields on this form, it wasn't clear the URL is going to be responsible
for any errors.

* ui/plus: Mock implementation of the classify API

* ui/dataset: Request classify if toggled

* ui/dataset: Confirmation modal to kick off classify

* ui/dataset: Cypress test for starting classify

* ui/dataset: Clear active dataset on un-mount instead of mount

This change makes it possible for the classify flow to highlight the newly-created
dataset when we navigate back to the table. Previously, the active dataset was cleared
by index page, which made preserving the active set between routes impossible. Now
we only clear when we leave a datset's view, which gives us the same experience and
the new feature.

* Update changelog

* Scaffold manual system flow (#1068)

* Refactor generate type to boolean

* Add ManualSystemFlow

* Add ConfigureSteps

* Make button bigger

* Update changelog

* Remove UI features for WIP elements

* [1073] ui/dataset: Show status badge for datasets using classification results (#1074)

* ui/plus: Classifications grouped by dataset fides key

* ui/dataset: Show status badge for datasets using classification results

* ui/dataset: Cypress tests for classified table

* Update changelog

* ui/state: Deduplicate query results that were being stored in state slices (#1083)

* ui/dedup-state: Systems unused

* ui/dedup-state: Datasets unused

* ui/dedup-state: Organization hydrate unused

* ui/dedup-state: Taxonomy DataCategories query

* ui/dedup-state: DataSubjects query

* ui/dedup-state: DataQualifiers query

* ui/dedup-state: DataUses query

* Fix header help link (#1078)

* Fix header help link

* update changelog

* Reuse config wizard forms for adding a system (#1072)

* Consolidate tooltips in DescribeSystemsForm

* Add form to manual system flow

* Add validation schema for DescribeSystemsForm

* Fix onBlur handlers of select fields

* Refactor DescribeSystemsForm

* Refactor PrivacyDeclarationForm

* Continue refactoring PrivacyDeclarationForm

* Refactor ReviewSystemForm

* Refactor SuccessPage

* Update changelog

* Refactor ReviewSystemForm to use a grid ReviewItem

* Add cypress test for flow

* Try to fix flaky test

* Ensure we stub taxonomy items

* Refactor PrivacyDeclarationAccordion to match designs more

* Fix adding another declaration when name field is blank

* Rename SuccessPage --> SystemRegisterSuccess

* Pass system object through props

* Update tests

* UI to delete a system (#1085)

* Implement delete system feature

* Add NotFoundError

* Add cypress tests

* Update changelog

* Refactor errors to ErrorDetails

* Separate 'next' and 'add' logic in PrivacyDeclarationForm (#1086)

* Separate continue and add logic

* Update tests

* Update changelog

* New fields on system forms (#1082)

* Extend DescribeSystemsForm

* Extend PrivacyDeclarationForm

* Add joint controller and data protection impact assessment

* Move system dependency field to abridged form

* Prepare data protection impact assessment for payload

* Add cypress tests

* Extend ReviewSystemForm

* Add tests for extended review form

* Fix adding another declaration when name field is blank

* Rename SuccessPage --> SystemRegisterSuccess

* Pass system object through props

* Update tests

* Update changelog

* Refactor ReviewSystemForm

* Move config wizard system forms to system directory (#1097)

* Move system forms to system directory

* Refactor form layout components into its own file

* Rename files from form --> step

* Update changelog

* ui/datasets: Refactor selectors (#1087)

* ui/dataset: Track dataset by fides key instead of object copy and nulls

Previously we've stored a copy of the dataset returned by the get-by-key query,
but only every use its key property. With this change we only store the key and
use the object from the query directly.

As part of this, I also converted a lot of `null` to `undefined`. There different
opinions on this, but undefined is generally more useful in a TS codebase because:

1. It plays nicely with optional function arguments.
2. It can be represented by a single question mark in interfaces. (See the State
   changes in this commit.)
3. It doesn't have null's `typeof null === "object"` confusion.

* ui/dataset: Single state for edit drawer

* ui/dataset: Extract collection lookups to selector

* ui/dataset: Extract field lookups to selector

* ui/dataset: Extract field types into Cell component

* ui/cypress: Consistent test fides key

* Bump fideslang to 1.3.0 (#1103)

* Bump fideslang to 1.3.0

* Add `egress` and `ingress` to `ctl_systems`

* Update `CHANGELOG.md`

* Prepare 1.8.5 release (#1107)

* Bump fideslang to 1.3.0

* Add `egress` and `ingress` to `ctl_systems`

* Update `CHANGELOG.md`

* Prepare `CHANGELOG.md` for v1.8.5

* upgrade pymysql to version 1.0.2 (#1094)

* Edit system UI (#1096)

* Set activeSystem in system.slice

* Allow form fields to take undefined

* Extend DescribeSystemForm to be able to edit

* Allow editing system from its card

* Update changelog

* Pull the form part of privacy declaration into its own component

* Further refactor PrivacyDeclarationForm to support editing

* Pull common intercepts out into a stubs file

* Add cypress test for editing

* Reserve dataset_references for unabridged forms

* Make sure not to override privacy declaration

* Explain allowing undefined for form inputs

* Change routing behavior to go back on cancel

* Fix error message

* Update CHANGELOG.md

* included more changes from fidesctl

* fix static checks

* manual updates from ctl UI

* formatting

* change line endings to fix prettier error

* fix docker build issue

* merge alembic heads for fidesctl merge and get tests parsing

* don't include the worker as a dependency for docker-compose fides

* remove worker flag from default fides service

* fix static checks

* get webserver running again

* reenable the worker for the docker-compose service

* enable redis for the compose service

* clean command is absolute in its destruction

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Allison King <allisonjuliaking@gmail.com>
Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Steve Murphy <steven.d.murphy@gmail.com>
Co-authored-by: Sebastian Sangervasi <2236777+ssangervasi@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Phil Salant <PSalant726@users.noreply.github.com>

.dockerignore

@@ -5,6 +5,11 @@ src/fides.egg-info/
 src/fides/ui-build/
 src/ui-build/
 
+# Frontend
+**/node_modules
+**/.next
+
+
 # Ignore Python-Specific Files
 .mypy_cache/
 .nox/
@@ -34,12 +39,8 @@ docs/
 # Ignore dev files
 .github/
 .devcontainer/
-**/node_modules/
 
 # Ignore cypress artifacts
 **/videos/
 **/screenshots/
-clients/ops/admin-ui/node_modules
-clients/ops/admin-ui/.next
-clients/privacy-center/.next
 .DS_Store

.fides/db_dataset.yml

@@ -654,6 +654,15 @@ dataset:
       data_categories:
       - system.operations
       data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
+      retention: null
+      fields: null
+    - name: egress
+      description: null
+      data_categories:
+      - system.operations
+      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
+      retention: null
+      fields: null
     - name: fides_key
       data_categories:
       - system.operations
@@ -666,6 +675,15 @@ dataset:
       data_categories:
       - system.operations
       data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
+      retention: null
+      fields: null
+    - name: ingress
+      description: null
+      data_categories:
+      - system.operations
+      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
+      retention: null
+      fields: null
     - name: joint_controller
       description: Encrypted contact information for a joint controller (name, address,
         email, phone)

.fides/fides.toml

@@ -27,7 +27,6 @@ port = 6379
 charset = "utf8"
 default_ttl_seconds = 604800
 db_index = 0
-enabled = true
 ssl = false
 ssl_cert_reqs = "required"
 
@@ -48,7 +47,6 @@ task_retry_backoff = 1
 subject_identity_verification_required = false
 task_retry_count = 0
 task_retry_delay = 1
-worker_enabled = false
 
 [admin_ui]
 enabled = true

CHANGELOG.md

@@ -15,11 +15,77 @@ The types of changes are:
 * `Fixed` for any bug fixes.
 * `Security` in case of vulnerabilities.
 
-## [Unreleased](https://github.com/ethyca/fides/compare/1.8.2...main)
+## [Unreleased](https://github.com/ethyca/fides/compare/1.8.5...main)
 
 ### Added
 
-* Added more taxonomy fields that can be edited via the UI [#1000](https://github.com/ethyca/fides/pull/1000)
+* Dataset generation enhancements using Fides Classify for Plus users:
+  * Added toggle for enabling classify during generation. [#1057](https://github.com/ethyca/fides/pull/1057)
+  * Initial implementation of API request to kick off classify, with confirmation modal. [#1069](https://github.com/ethyca/fides/pull/1069)
+  * Initial Classification & Review status for generated datasets. [#1074](https://github.com/ethyca/fides/pull/1074)
+* System management UI:
+  * New page to add a system via yaml [#1062](https://github.com/ethyca/fides/pull/1062)
+  * Skeleton of page to add a system manually [#1068](https://github.com/ethyca/fides/pull/1068)
+  * Refactor config wizard system forms to be reused for system management [#1072](https://github.com/ethyca/fides/pull/1072)
+  * Add additional optional fields to system management forms [#1082](https://github.com/ethyca/fides/pull/1082)
+  * Delete a system through the UI [#1085](https://github.com/ethyca/fides/pull/1085)
+  * Edit a system through the UI [#1096](https://github.com/ethyca/fides/pull/1096)
+
+### Changed
+
+* Changed behavior of `load_default_taxonomy` to append instead of upsert [#1040](https://github.com/ethyca/fides/pull/1040)
+* Changed behavior of adding privacy declarations to decouple the actions of the "add" and "next" buttons [#1086](https://github.com/ethyca/fides/pull/1086)
+* Moved system related UI components from the `config-wizard` directory to the `system` directory [#1097](https://github.com/ethyca/fides/pull/1097)
+
+### Fixed
+
+* Fixed the "help" link in the UI header [#1078](https://github.com/ethyca/fides/pull/1078)
+
+### Security
+
+* Upgraded pymysql to version `1.0.2` [#1094](https://github.com/ethyca/fides/pull/1094)
+
+## [1.8.5](https://github.com/ethyca/fides/compare/1.8.4...1.8.5)
+
+### Changed
+
+* Update fideslang to v1.3.0 [#1103](https://github.com/ethyca/fides/pull/1103)
+
+## [1.8.4](https://github.com/ethyca/fides/compare/1.8.3...1.8.4) - 2022-09-09
+
+### Added
+
+* Initial system management page [#1054](https://github.com/ethyca/fides/pull/1054)
+
+### Changed
+
+* Deleting a taxonomy field with children will now cascade delete all of its children as well. [#1042](https://github.com/ethyca/fides/pull/1042)
+
+### Fixed
+
+* Fixed navigating directly to frontend routes loading index page instead of the correct static page for the route.
+* Fix truncated evaluation error messages [#1053](https://github.com/ethyca/fides/pull/1053)
+
+## [1.8.3](https://github.com/ethyca/fides/compare/1.8.2...1.8.3) - 2022-09-06
+
+### Added
+
+* Added more taxonomy fields that can be edited via the UI [#1000](https://github.com/ethyca/fides/pull/1000) [#1028](https://github.com/ethyca/fides/pull/1028)
+* Added the ability to add taxonomy fields via the UI [#1019](https://github.com/ethyca/fides/pull/1019)
+* Added the ability to delete taxonomy fields via the UI [#1006](https://github.com/ethyca/fides/pull/1006)
+  * Only non-default taxonomy entities can be deleted [#1023](https://github.com/ethyca/fides/pull/1023)
+* Prevent deleting taxonomy `is_default` fields and from adding `is_default=True` fields via the API [#990](https://github.com/ethyca/fides/pull/990).
+* Added a "Custom" tag to distinguish user defined taxonomy fields from default taxonomy fields in the UI [#1027](https://github.com/ethyca/fides/pull/1027)
+* Added initial support for enabling Fides Plus [#1037](https://github.com/ethyca/fides/pull/1037)
+  * The `useFeatures` hook can be used to check if `plus` is enabled.
+  * Navigating to/from the Data Map page is gated behind this feature.
+  * Plus endpoints are served from the private Plus image.
+
+### Fixed
+
+* Fixed failing mypy tests [#1030](https://github.com/ethyca/fides/pull/1030)
+* Fixed an issue where `fides push --diff` would return a false positive diff [#1026](https://github.com/ethyca/fides/pull/1026)
+* Pinned pydantic version to < 1.10.0 to fix an error in finding referenced fides keys [#1045](https://github.com/ethyca/fides/pull/1045)
 
 ### Fixed
 

Dockerfile

@@ -9,13 +9,11 @@ ARG PYTHON_VERSION=3.10.6
 FROM node:16 as frontend
 
 # Build the admin-io frontend
-WORKDIR /fidesops/clients/admin-ui
+WORKDIR /fides/clients/admin-ui
 COPY clients/admin-ui/ .
 RUN npm install
 RUN npm run export
 
-
-
 #############
 ## Backend ##
 #############
@@ -99,5 +97,4 @@ RUN python setup.py sdist
 RUN pip install dist/ethyca-fides-*.tar.gz
 
 # Copy frontend build over
-COPY --from=frontend /fidesops/clients/admin-ui/out/ /fidesops/src/fidesops/build/static/
-
+COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/fides/ui-build/static/admin

clients/admin-ui/.prettierrc.json

@@ -1,3 +1,4 @@
 {
-  "tabWidth": 2
+  "tabWidth": 2,
+  "endOfLine": "lf"
 }

clients/admin-ui/__tests__/features/common/zones/config.test.ts

@@ -0,0 +1,103 @@
+import { configureZones, resolveZone } from "~/features/common/zones";
+
+describe("resolveZone", () => {
+  // Allow mocking and resetting the node env in this test suite.
+  const env = process.env as { NODE_ENV: string };
+  afterEach(() => {
+    env.NODE_ENV = "test";
+  });
+
+  const config = configureZones({
+    development: {
+      host: "localhost:3000",
+    },
+
+    zones: [
+      {
+        basePath: "/datamap",
+        development: {
+          host: "localhost:4000",
+        },
+      },
+      {
+        basePath: "/no/dev",
+      },
+    ],
+  });
+
+  describe("Within the root zone", () => {
+    const basePath = "/";
+
+    it("Does not modify a link within the root zone", () => {
+      env.NODE_ENV = "development";
+
+      const link = resolveZone({ config, basePath, href: "/root/route" });
+      expect(link).toMatchObject({
+        href: "/root/route",
+        basePath: "/",
+      });
+    });
+
+    it("Links to the dev host of another zone", () => {
+      env.NODE_ENV = "development";
+
+      const link = resolveZone({ config, basePath, href: "/datamap" });
+      expect(link).toMatchObject({
+        href: "http://localhost:4000/datamap",
+        basePath: "/datamap",
+      });
+    });
+
+    it("Does not link to the dev host in production", () => {
+      env.NODE_ENV = "production";
+
+      const link = resolveZone({ config, basePath, href: "/datamap" });
+      expect(link).toMatchObject({
+        href: "/datamap",
+        basePath: "/datamap",
+      });
+    });
+
+    it("Does not link to the dev host if none is configured", () => {
+      env.NODE_ENV = "development";
+
+      const link = resolveZone({ config, basePath, href: "/no/dev" });
+      expect(link).toMatchObject({
+        href: "/no/dev",
+        basePath: "/no/dev",
+      });
+    });
+  });
+
+  describe("Within a sub zone", () => {
+    const basePath = "/datamap";
+
+    it("Strips the base path of the sub-zone", () => {
+      const link = resolveZone({ config, basePath, href: "/datamap/route" });
+      expect(link).toMatchObject({
+        href: "/route",
+        basePath: "/datamap",
+      });
+    });
+
+    it("Links to the dev host of the root zone", () => {
+      env.NODE_ENV = "development";
+
+      const link = resolveZone({ config, basePath, href: "/root/route" });
+      expect(link).toMatchObject({
+        href: "http://localhost:3000/root/route",
+        basePath: "/",
+      });
+    });
+
+    it("Does not link to the dev host in production", () => {
+      env.NODE_ENV = "production";
+
+      const link = resolveZone({ config, basePath, href: "/root/route" });
+      expect(link).toMatchObject({
+        href: "/root/route",
+        basePath: "/",
+      });
+    });
+  });
+});