Sm 104 project Database #2192

cd-bitwarden · 2022-08-15T19:20:20Z

Type of change

We should create a database table for projects + repository + entity using EF.

Projects

Id uniqueidentifier
OrganizationId fk organization
Name nvarchar(max)
CreationDate DATETIME2 (7)
RevisionDate DATETIME2 (7)
DeletedDate DATETIME2 (7)

The repository should have ways to, GetAll (by organization), GetById, Add, Edit and SoftDeleteMany(array).

- [ ] Bug fix
- [ x ] New feature development
- [ ] Tech debt (refactoring, code cleanup, dependency upgrades, etc)
- [ ] Build/deploy pipeline (DevOps)
- [ ] Other

Objective

Adding tables to hold the Project information, and procedures to retrieve and soft delete Projects.

Code changes

-- Added the following procedures, the Project table create script and the migration script

Project_Create
Project_GetById
Project_GetByOrganizationId
Project_SoftDelete
Project_Update

Before you submit

- [ ] I have checked for formatting errors (`dotnet format --verify-no-changes`) (required)
- [ x ] If making database changes - I have also updated Entity Framework queries and/or migrations
- [ ] I have added **unit tests** where it makes sense to do so (encouraged but not required)
- [ ] This change requires a **documentation update** (notify the documentation team)
- [ ] This change has particular **deployment requirements** (notify the DevOps team)

coltonhurst

Everything looks good, just left one comment.

If everything has been tested locally and works, I think we are good to go.

util/Migrator/DbScripts/2022-08-10_00_Project.sql

Thomas-Avery · 2022-08-16T21:44:01Z

If we are doing everything in entity framework, we won't need the stored procedures for this.

addisonbeck · 2022-08-17T11:29:16Z

Is SM going to be EF-only? The way the Jira ticket is written implies to me that we'll need to support SQL and Entity Framework.

Regardless, please also create an Entity for this and generate an Entity Framework migration.

Thomas-Avery · 2022-08-17T14:18:15Z

The direction I was given for creating the Secrets table was EF-only from Oscar. I would assume we want to do the same for other SM tables. As part of that work, we set up EF for Microsoft SQL Server as a target EF database provider.

addisonbeck · 2022-08-17T17:28:44Z

That's great news, Thomas. Thanks for clarifying.

… framework code

Hinton

Seems we accidentally deleted the provider create sproc. We'll need to bring it back.

I don't believe we need the SqlServerEfScaffold/Migrations directory, we skipped it for the secrets table, thoughts @Thomas-Avery?

src/Sql/dbo/Stored Procedures/Provider_Create.sql

Thomas-Avery · 2022-08-22T14:06:35Z

I agree SqlServerEfScaffold is intended to help engineers validate C# code changes against MSSQL. We shouldn't put actual migrations in the folder.

src/Api/Controllers/OrganizationExportController.cs

src/Sql/dbo/Tables/Project.sql

Hinton

Some additional things I noticed.

src/Core/Entities/Project.cs

src/Sql/dbo/Tables/Project.sql

src/Infrastructure.EntityFramework/Configurations/ProjectEntityTypeConfiguration.cs

Hinton · 2022-09-01T08:54:57Z

@Thomas-Avery mind throwing a quick glance at this PR as well?

Thomas-Avery

Overall looks great, just a few things I noticed.

src/Sql/dbo/Tables/Project.sql

...rden_license/src/Commercial.Infrastructure.EntityFramework/Repositories/ProjectRepository.cs

util/Migrator/DbScripts/2022-08-10_00_Project.sql

util/PostgresMigrations/Migrations/DatabaseContextModelSnapshot.cs

Thomas-Avery

LGTM

* [SM-66] Create Secret Database Table (#2144) Objective The purpose of this PR is to create a database table, entity, and repository for the new Secret database table. The new Secret table will use entity framework for all database providers. * [SM-67] Get all secrets by org ID (#2163) Add a controller to fetch secrets associated with an organization ID. To note, the [SecretsManager] attribute makes this controller only available for local development. * [SM-68] Add API endpoints for getting, creating, and editing secrets (#2201) The purpose of this PR is to add API endpoints for getting, creating, and editing secrets for the Secrets Manager project. * Move interfaces to core (#2211) * [SM-63] Read UTC DateTimes from databases via EF and order by revision date (#2206) * Read UTC DateTimes from db and order by revision * Move orderby to repo layer * [SM-185] Add EE_Testing_env to server (#2222) * Sm 104 project Database (#2192) * Project DB addition and sprocs * Adding spaces to the end of each file, fixing minor issues * removing useless comments * Adding soft delete proc to migration * Project EF Scaffold * Additional changes to use EF instead of procedures * Adding dependency injection * Fixing lint errors * Bug fixes * Adding migration scripts, removing sproc files, and setting up Entity framework code * Adding back accidentally deleted sproc * Removing files that shouldn't have been created * Lint * Small changes based on Oscar's rec (#2215) * Migrations for making CreateDate not null * adding space to end of file * Making Revision date not null * dotnet format * Adding nonclustered indexes to SQL * SM-104: Update PR with changes Thomas proposed Co-authored-by: CarleyDiaz-Bitwarden <103955722+CarleyDiaz-Bitwarden@users.noreply.github.com> Co-authored-by: Thomas Avery <tavery@bitwarden.com> Co-authored-by: Colton Hurst <colton@coltonhurst.com> * Removing org ID from create request body (#2243) * SM-114: Add create & update project endpoints (#2251) * SM-114: Initial commit with create project endpoint (for SM) * SM-114: Add Update Project route (for SM) * SM-114: Fix file encodings * Fix DI issue for SM Project Create/Update commands * Fix import ordering for linter * SM-114: Remove unneeded lines setting DeletedDate, as it should already be null * SM-114: Only have OrgId in route for CreateProject * Remove unneeded using * SM-114: Initial commit with create project endpoint (for SM) * SM-114: Add Update Project route (for SM) * SM-114: Fix file encodings * Fix DI issue for SM Project Create/Update commands * Fix import ordering for linter * SM-114: Remove unneeded lines setting DeletedDate, as it should already be null * SM-114: Only have OrgId in route for CreateProject * Remove unneeded using * Fully remove OrgId from ProjectCreateRequestModel * [SM-64] Soft Delete Secrets (#2253) * Bulk delete secrets with command unit tests * Controller unit tests * Optimize conditionals * SM-64 bulk delete integration test * fix test * SM-64 code review updated * [SM-65] Fix return empty secrets list (#2281) * Secrets return empty list * [SM-246] Use repository in integration test (#2285) * [SM-190] Add integration tests to Secrets (#2292) * Adding integration tests for the SecretsController Co-authored-by: Hinton <hinton@users.noreply.github.com> * Sm 95 - Adding GetProjects endpoint (#2295) * SM-114: Initial commit with create project endpoint (for SM) * SM-114: Add Update Project route (for SM) * SM-114: Fix file encodings * Fix DI issue for SM Project Create/Update commands * Adding GetProjectsByOrg * fixing merge conflicts * fix * Updating to return empty list * removing null check Co-authored-by: Colton Hurst <colton@coltonhurst.com> Co-authored-by: CarleyDiaz-Bitwarden <103955722+CarleyDiaz-Bitwarden@users.noreply.github.com> * [SM-191] Create ServiceAccount Table (#2301) * SM-191 Create ServiceAccount Table * [SM-207] API for listing service accounts by organization (#2307) * SM-207 list service accounts by org * SM-96: Add ability to get project by id (#2314) * SM-96: Small change to allow getting project by id * Fix whitespace issue * Add first integration test and fix date bug * Ensure tests are consistent * Add more project controller integration tests * Remove commented delete for now * [SM-187] Create ServiceAccounts (#2323) * SM-187 Create & Update ServiceAccounts * Remove extra new line src/Api/Controllers/ServiceAccountsController.cs Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * [SM-218] [SM-219] SM Auth flow (#2297) * SM-282 Delete Projects (#2335) * SM-282 delete & bulk delete projects * Have delete commands return tuple with object * Fix admin project not working after secrets manager changes (#2339) * [SM-150] proj and secrets mapping (#2286) * Beggining of changes for Project Secrets mapping * Beggining of changes for project and secrets mapping * Inital changes to add Mapping table for Project Secrets * Resolve migration not working properly * Indent sql * Changes to try and return projects in the GetManyByOrganizaationIDAsync on SecretRepository. * Changes made with Oscar * Add reversemap * running lint and removing comments * Lint fixes * fixing merge issues * Trying to fix the DB issue * DB fixes * fixes * removing unused space * fixing lint issue * final lint fix I hope * removing manually added sql.sqlproj * Lint changes and fixing the sql proj issues * adding ServiceAccount to sql proj * Removing ON DELETE CASCADE * remove On delete cascade * changes for deleting project and secret inside of the Organization_DeleteById procedure. * changes for deleting project and secret inside of the Organization_DeleteById procedure. * migration changes * Updating constraints * removing void * remove spaces * updating cipherRepo tests to be task instead of void * fixing * fixing * test * fix * fix * changes to remove circular dependency * fixes * sending guid and string name of the project over * Update src/Sql/dbo/Tables/Secret.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * Update src/Sql/dbo/Tables/Project.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * removing unused code * Potential refactor (#2340) * migrations * Postgres migraiton * Update src/Api/SecretManagerFeatures/Models/Response/SecretResponseModel.cs Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * rename file * Update util/Migrator/DbScripts/2022-09-19_00_ProjectSecret.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * Lint fixes * removing extra semi colon * removing circular references with projects and secrets * adding back projects * Add ProjectFixture * Update util/Migrator/DbScripts/2022-09-19_00_ProjectSecret.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * Update util/Migrator/DbScripts/2022-09-19_00_ProjectSecret.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> Co-authored-by: CarleyDiaz-Bitwarden <103955722+CarleyDiaz-Bitwarden@users.noreply.github.com> Co-authored-by: Hinton <hinton@users.noreply.github.com> * [SM-300] Access token endpoint (#2377) * [SM-324] Add Organization to JWT claim (#2379) * [SM-259] Add create access token endpoint for service accounts (#2411) * Add create access token for service accounts * [SM-259] Fix create access token scope initialization (#2418) * Fix namespace for ServiceAccount command tests * Remove "this" from SecretsManager requests * Fix have scope be assigned a JSON list * SM-99: Individual Project / Secrets Tab (#2399) Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * [SM-361] Add Support for never expiring ApiKeys (#2450) * Update database to support never expiring ApiKey * Update Api to support never expiring ApiKeys * Fix unit test variable naming * Remove required from model * Fix spacing * Add EF migrations * Run dotnet format * Update util/Migrator/DbScripts/2022-11-29_00_ApiKey_Never_Expire.sql Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> * [SM-359] Fix project secrets migration (#2443) * [SM-299] Add UseSecretsManager flag (#2413) * [SM-193] Access Policy (#2359) * [SM-371] Fix and re-enable parallel integration tests (#2460) * Fix and re-enable parallel integration tests * Fix package lock files * Move fix to ApiApplicationFactory * Run dotnet restore --force * Run dotnet format * Reset packages.lock.json files * Add project access checks for listing * SM-99: Add CreateSecretWithProject Integration Test (#2452) * Add GetSecretsByProjectAsync endpoint * Add GetManyByProjectIdAsync endpoint * Update response model for GetSecretsByProjectAsync * Include projects when returning secrets by project id * SM-99: Add ability to specify projectId when creating a secret * SM-99: Update tests to accomodate for new create secret parameter * Fix failing test * SM-99: Handle optional projectId for new secret in ToSecret() * SM-99: Filter out deleted secrets on GetManyByProjectIdAsync() and small refactorings * SM-99: make CreateAsync for secret more clear * Add CreateSecretWithProject integration test * Fix CreateSecretWithProject integration test for SM-99 * Run dotnet format * Undo added space * Refactor test * Refactor CreateSecretWithProject API Integration test again * Change to boolean flag * [SM-379] Add SDK device type (#2486) * Add support for service accounts * Improve logic for project repository * Add remaining client types * Experiment with separate enum for access control * Add access checks to update project * Rework AccessClientType * Add access checks to fetching project * Add checks to delete project command (untested) * Remove some service account stuff * Add ServiceAccount to AccessClientType * Change CS8509 to error and 8424 to ignore * Remove unused utcNow * Fix delete tests * SM-73 changes (#2422) * testing * test2 * testing * trying to save the projects associated with the secret * changes * more changes * Fix EF error * Second attempt * Replace AddIfNotExists with Add. * changes * fixing await issue * lint * lint fixes * suggested changes * suggested changes * updating tests * fixing tests 2 * fixing tests * fixing test * fixing test * fixing tests * test * testing * fixing tests for the millionth time * fixing tests * allowing nulls for projectIds, fixing lint * fixing tests Co-authored-by: Hinton <hinton@users.noreply.github.com> * fixing tests * fixing tests * [SM-222] [SM-357] Squash Secrets Manager migrations (#2540) * Fix tables not being cleaned up * Fix migration * Squash secrets manager migrations * Reset EF to pre SM state * Add EF migrations * Fix unified docker * Add missed copy * Fix all unit tests * draft changes to add access checks to secrets * updating code * more changes * fixing issues * updating logic for access checks * updating secrets controller * changes * changes * merging more * changes * updateS * removing unused comment * changes requested by Thomas * more changes suggested by Thomas * making thomas's suggested changes * final changes * Run dotnet format * fixes * run dotnet format * Updating tests * Suggested changes * lint fixes * Test updates * Changes * Fixes for tests, and dotnet format * Fixes * test fixes * changes * fix * fix * test fix * removing duplicate * Removing dupe --------- Co-authored-by: Thomas Avery <43214426+Thomas-Avery@users.noreply.github.com> Co-authored-by: Oscar Hinton <Hinton@users.noreply.github.com> Co-authored-by: CarleyDiaz-Bitwarden <103955722+CarleyDiaz-Bitwarden@users.noreply.github.com> Co-authored-by: Thomas Avery <tavery@bitwarden.com> Co-authored-by: Colton Hurst <colton@coltonhurst.com>

CarleyDiaz-Bitwarden-zz added 2 commits August 15, 2022 15:08

Project DB addition and sprocs

6925ea1

Adding spaces to the end of each file, fixing minor issues

b315170

coltonhurst reviewed Aug 15, 2022

View reviewed changes

util/Migrator/DbScripts/2022-08-10_00_Project.sql Outdated Show resolved Hide resolved

CarleyDiaz-Bitwarden-zz added 2 commits August 15, 2022 16:12

removing useless comments

6ff9bff

Adding soft delete proc to migration

5833e6f

cd-bitwarden marked this pull request as ready for review August 16, 2022 17:57

cd-bitwarden requested review from addisonbeck, eliykat and justindbaur August 16, 2022 19:04

Thomas-Avery and others added 6 commits August 17, 2022 14:07

Project EF Scaffold

8ee53e3

Additional changes to use EF instead of procedures

06b0e6c

Adding dependency injection

7136b89

Fixing lint errors

c166a40

Bug fixes

2d7a273

Adding migration scripts, removing sproc files, and setting up Entity…

261f315

… framework code

Hinton reviewed Aug 22, 2022

View reviewed changes

src/Sql/dbo/Stored Procedures/Provider_Create.sql Outdated Show resolved Hide resolved

CarleyDiaz-Bitwarden-zz added 3 commits August 22, 2022 15:51

Adding back accidentally deleted sproc

eb4e164

Removing files that shouldn't have been created

04d4aa2

Lint

b6cce75

Hinton reviewed Aug 23, 2022

View reviewed changes

src/Api/Controllers/OrganizationExportController.cs Outdated Show resolved Hide resolved

Hinton reviewed Aug 23, 2022

View reviewed changes

src/Sql/dbo/Tables/Project.sql Outdated Show resolved Hide resolved

coltonhurst mentioned this pull request Aug 25, 2022

Small changes based on Oscar's rec #2215

Merged

Small changes based on Oscar's rec (#2215)

ac7bd1e

Hinton self-requested a review August 26, 2022 16:25

Hinton reviewed Aug 29, 2022

View reviewed changes

src/Core/Entities/Project.cs Outdated Show resolved Hide resolved

src/Core/Entities/Project.cs Outdated Show resolved Hide resolved

src/Sql/dbo/Tables/Project.sql Outdated Show resolved Hide resolved

CarleyDiaz-Bitwarden-zz added 5 commits August 30, 2022 09:42

Merge branch 'sm/secrets-feature' into SM-104-Project

0350f67

Migrations for making CreateDate not null

50ec647

adding space to end of file

5a0b00a

Making Revision date not null

8a04c48

dotnet format

c6ba598

Hinton reviewed Sep 1, 2022

View reviewed changes

src/Infrastructure.EntityFramework/Configurations/ProjectEntityTypeConfiguration.cs Show resolved Hide resolved

Hinton requested a review from Thomas-Avery September 1, 2022 08:54

Adding nonclustered indexes to SQL

9b235dd

Thomas-Avery requested changes Sep 1, 2022

View reviewed changes

SM-104: Update PR with changes Thomas proposed

dd2baa7

Thomas-Avery approved these changes Sep 2, 2022

View reviewed changes

Hinton approved these changes Sep 5, 2022

View reviewed changes

coltonhurst merged commit de08ad5 into sm/secrets-feature Sep 5, 2022

coltonhurst deleted the SM-104-Project branch September 5, 2022 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sm 104 project Database #2192

Sm 104 project Database #2192

cd-bitwarden commented Aug 15, 2022

coltonhurst left a comment

Thomas-Avery commented Aug 16, 2022

addisonbeck commented Aug 17, 2022

Thomas-Avery commented Aug 17, 2022 •

edited

Loading

addisonbeck commented Aug 17, 2022 •

edited

Loading

Hinton left a comment

Thomas-Avery commented Aug 22, 2022

Hinton left a comment

Hinton commented Sep 1, 2022

Thomas-Avery left a comment

Thomas-Avery left a comment

Sm 104 project Database #2192

Sm 104 project Database #2192

Conversation

cd-bitwarden commented Aug 15, 2022

Type of change

Projects

Objective

Code changes

Before you submit

coltonhurst left a comment

Choose a reason for hiding this comment

Thomas-Avery commented Aug 16, 2022

addisonbeck commented Aug 17, 2022

Thomas-Avery commented Aug 17, 2022 • edited Loading

addisonbeck commented Aug 17, 2022 • edited Loading

Hinton left a comment

Choose a reason for hiding this comment

Thomas-Avery commented Aug 22, 2022

Hinton left a comment

Choose a reason for hiding this comment

Hinton commented Sep 1, 2022

Thomas-Avery left a comment

Choose a reason for hiding this comment

Thomas-Avery left a comment

Choose a reason for hiding this comment

Thomas-Avery commented Aug 17, 2022 •

edited

Loading

addisonbeck commented Aug 17, 2022 •

edited

Loading