Refactor CI/CD (microsoft#1064)

* re-factor ci/cd

* re-factor ci/cd

* fix links

* Update docs/CI-CD/dev-sec-ops/secrets-management/README.md

Co-authored-by: Mark Jervelund <Doommius@users.noreply.github.com>

* Update docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md

Co-authored-by: Mark Jervelund <Doommius@users.noreply.github.com>

---------

Co-authored-by: Mark Jervelund <Doommius@users.noreply.github.com>

docs/.pages

@@ -0,0 +1,4 @@
+nav:
+  - ...
+  - CI/CD: ci-cd
+  - ...

docs/CI-CD/.pages

@@ -0,0 +1,6 @@
+nav:
+  - Continuous Delivery: continuous-delivery
+  - Continuous Integration: continuous-integration
+  - DevSecOps: dev-sec-ops
+  - GitOps: gitops
+  - ...

docs/continuous-integration/CICD.md → docs/CI-CD/README.md

@@ -1,10 +1,10 @@
-# Continuous Integration and Delivery
+# Continuous Integration and Continuous Delivery
 
-Continuous Integration is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed.
+[**Continuous Integration (CI)**](continuous-integration.md) is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed.
 
 Paired with an automated testing approach, continuous integration also allows us to also test the integrated build such that we can verify that not only does the code base still build correctly, but also is still functionally correct. This is also a best practice for building robust and flexible software systems.
 
-Continuous Delivery takes the Continuous Integration concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage.
+[**Continuous Delivery (CD)**](continuous-delivery.md) takes the **Continuous Integration (CI)** concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage.
 
 The goal of all of this is to ensure that the main branch is always shippable, meaning that we could, if we needed to, take a build from the main branch of our code base and ship it on production.
 
@@ -14,6 +14,23 @@ Our expectation is that CI/CD should be used in all the engineering projects tha
 
 For a much deeper understanding of all of these concepts, the books [Continuous Integration](https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380) and [Continuous Delivery](https://www.amazon.com/gp/product/0321601912) provide a comprehensive background.
 
+## Why CI/CD
+
+- We want to have an automated build and deployment of our software
+- We want automated configuration of all components
+- We want to be able to quickly re-build the environment from scratch in case of disaster
+- We want the latest version of the code to always be deployed to our dev/test environments
+- We want a reliable release strategy, where the policies for release are well understood by all
+
+## The fundamentals
+
+- We run a quality pipeline (with linting, unit tests etc.) on each PR/update of the main branch
+- All cloud resources (including secrets and permissions) are provisioned through infrastructure as code templates – ex. Terraform, Bicep (ARM), Pulumi etc.
+- All release candidates are deployed to a non-production environment through an automated process (ex Azure DevOps or Github pipelines)
+- Releases are deployed to the production environment through an automated process
+- Release rollbacks are carried out through a repeatable process
+- Our release pipeline runs automated tests, validating all release candidate artifact(s) end-to-end against a non-production environment
+
 ## Tools
 
 ### Azure Pipelines

docs/continuous-delivery/README.md → docs/CI-CD/continuous-delivery.md

@@ -139,7 +139,7 @@ Try to limit the number of versions of your application running parallel in prod
 
 Low code solutions have increased their participation in the applications and processes and because of that it is required that a proper conjunction of disciplines improve their development.
 
-Here is a guide for [continuous deployment for Low Code Solutions](low-code-solutions/README.md).
+Here is a guide for [continuous deployment for Low Code Solutions](recipes/cd-on-low-code-solutions.md).
 
 ## References
 

docs/continuous-integration/README.md → docs/CI-CD/continuous-integration.md

@@ -45,7 +45,7 @@ An automated build should encompass the following principles:
 
 - Code across an engineering team must be formatted to agreed coding standards. Such standards keep code consistent, and most importantly easy for the team and customer(s) to read and refactor. Code styling consistency encourages collective ownership for project scrum teams and our partners.
 - There are several open source code style validation tools available to choose from ([code style checks](https://github.com/checkstyle/checkstyle), [StyleCop](https://en.wikipedia.org/wiki/StyleCop)). The [Code Review recipes section](../code-reviews/recipes/README.md) of the playbook has suggestions for linters and preferred styles for a number of languages.
-- Your code and documentation should avoid the use of non-inclusive language wherever possible. Follow the [Inclusive Linting section](inclusive-linting.md) to ensure your project promotes an inclusive work environment for both the team and for customers.
+- Your code and documentation should avoid the use of non-inclusive language wherever possible. Follow the [Inclusive Linting section](recipes/inclusive-linting.md) to ensure your project promotes an inclusive work environment for both the team and for customers.
 - We recommend incorporating security analysis tools within the build stage of your pipeline such as: code credential scanner, security risk detection, static analysis, etc. For Azure DevOps, you can add a security scan task to your pipeline by installing the [Microsoft Security Code Analysis Extension](https://secdevtools.azurewebsites.net/#pills-onboard). GitHub Actions supports a similar extension with the [RIPS security scan solution](https://github.com/marketplace/actions/rips-security-scan).
 - Code standards are maintained within a single configuration file. There should be a step in your build pipeline that asserts code in the latest commit conforms to the known style definition.
 

docs/CI-CD/dev-sec-ops/.pages

@@ -0,0 +1,3 @@
+nav:
+  - Azure DevOps service connection security: azure-devops-service-connection-security
+  - ...

docs/CI-CD/dev-sec-ops/README.md

@@ -0,0 +1,21 @@
+# DevSecOps
+
+## The concept of DevSecOps
+
+DevSecOps or DevOps security is about introducing security earlier in the life cycle of application development (a.k.a shift-left), thus minimizing the impact of vulnerabilities and bringing security closer to development team.
+
+## Why
+
+By embracing shift-left mentality, DevSecOps encourages organizations to bridge the gap that often exists between development and security teams to the point where many of the security processes are automated and are effectively handled by the development team.
+
+## DevSecOps Practices
+
+This section covers different tools, frameworks and resources allowing introduction of DevSecOps best practices to your project at early stages of development.
+Topics covered:
+
+1. [Credential Scanning](./secrets-management/credential_scanning.md) - automatically inspecting a project to ensure that no secrets are included in the project's source code.
+1. [Secrets Rotation](./secrets-management/secrets_rotation.md) - automated process by which the secret, used by the application, is refreshed and replaced by a new secret.
+1. [Static Code Analysis](./secrets-management/static-code-analysis.md) - analyze source code or compiled versions of code to help find security flaws.
+1. [Penetration Testing](./penetration-testing.md) - a simulated attack against your application to check for exploitable vulnerabilities.
+1. [Container Dependencies Scanning](./dependency-and-container-scanning.md) - search for vulnerabilities in container operating systems, language packages and application dependencies.
+1. [Evaluation of Open Source Libraries](./evaluate-open-source-software.md) - make it harder to apply open source supply chain attacks by evaluating the libraries you use.

docs/continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md → docs/CI-CD/dev-sec-ops/azure-devops-service-connection-security.md

@@ -13,10 +13,11 @@ Securing Service Connections can be achieved by using several methods.
 - [Project permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints#project-permissions---cross-project-sharing-of-service-connections) can be configured to ensure only certain Azure DevOps projects are able to use the Service Connection.
 
 After using the above methods, what is secured is **who** can use the Service Connections.
-What still *isn't* secured however, is **what** can be done with the Service Connections.  
+What still *isn't* secured however, is **what** can be done with the Service Connections.
+
+Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users.
+An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources.  Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done.
 
-Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users.  
-An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources.  Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done.  
 ```yaml
 pool:
   vmImage: ubuntu-latest
@@ -36,12 +37,12 @@ steps:
 
 ## Pipeline Security caveat
 
-YAML pipelines can be triggered without the need for a pull request, this introduces a security risk.  
+YAML pipelines can be triggered without the need for a pull request, this introduces a security risk.
 
-In good practice, [Pull Requests](../../../code-reviews/pull-requests.md) and [Code Reviews](../../../code-reviews/README.md) should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues.  
-However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer.  
+In good practice, [Pull Requests](../../code-reviews/pull-requests.md) and Code Reviews should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues.
+However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer.
 
-The configuration of *when* a pipeline should be triggered is specified in the YAML Pipeline itself and therefore a pipeline can be configured to execute on changes in a temporary branch. In this temporary branch, any changes made to the pipeline itself will be executed without being reviewed.  
+The configuration of *when* a pipeline should be triggered is specified in the YAML Pipeline itself and therefore a pipeline can be configured to execute on changes in a temporary branch. In this temporary branch, any changes made to the pipeline itself will be executed without being reviewed.
 
 If the given pipeline has been granted [Pipeline-level permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints#pipeline-permissions) to use a specific Service Connection, any command can be executed using that Service Connection, without anyone reviewing the command.
 Since Service Connections can have a lot of permissions in the external service, executing any pipeline without review could potentially have big consequences.
@@ -54,13 +55,13 @@ Configuration can be done in the Approvals and Checks menu on the Service Connec
 
 ### Branch Control
 
-By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch.  
+By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch.
 
 By configuring Branch Control to only allow the main branch (and potentially release branches) you can ensure a YAML Pipeline can only use the Service Connection after any changes to that pipeline have been merged into the main branch, and therefore has passed any Pull Requests checks and Code Reviews.
 As an additional check, Branch Control can verify if Branch Protections (like required Pull Requests and Code Reviews) are actually configured on the allowed branches.
 
-With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore.  
+With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore.
 
-**Note: When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.*
+> **Note:** When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.*
 
 ![BranchControl](images/branch-control.png)

docs/continuous-delivery/secrets-management/README.md → docs/CI-CD/dev-sec-ops/secrets-management/README.md

@@ -1,12 +1,24 @@
-# Secrets Management
+# Secrets management
 
-Secrets Management refers to the way in which we protect configuration settings and other sensitive data which, if
-made public, would allow unauthorized access to resources. Examples of secrets are usernames, passwords, api keys, SAS
-tokens etc.
+Secret management refers to the tools and practices used to manage digital authentication credentials (like API keys, tokens, passwords, and certificates). These secrets are used to protect access to sensitive data and services, making their management critical for security.
 
 We should assume any repo we work on may go public at any time and protect our secrets, even if
 the repo is initially private.
 
+## Importance of Secrets Management
+
+In modern software development, applications often need to interact with other software components, APIs, and services. These interactions often require authentication, which is typically handled using secrets. If these secrets are not managed properly, they can be exposed, leading to potential security breaches.
+
+## Best Practices for Secrets Management
+
+1. **Centralized Secret Storage:** Store all secrets in a centralized, encrypted location. This reduces the risk of secrets being lost or exposed.
+1. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets.
+1. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised.
+1. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities.
+1. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error.
+
+Remember, the goal of secret management is to protect sensitive information from unauthorized access and potential security threats.
+
 ## General Approach
 
 The general approach is to keep secrets in separate configuration files that are not checked in
@@ -19,7 +31,7 @@ the Azure CLI to do the same is a useful time-saving utility. See [az webapp con
 
 It's best practice to maintain separate secrets configurations for each environment that you run. e.g. dev, test, prod, local etc
 
-The [secrets-per-branch recipe](../gitops/secret-management/azure-devops-secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment.
+The [secrets-per-branch recipe](../../gitops/secret-management/azure-devops-secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment.
 
 > Note: even if the secret was only pushed to a feature branch and never merged, it's still a part of the git history. Follow [these instructions](https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository) to remove any sensitive data and/or regenerate any keys and other sensitive information added to the repo. If a key or secret made it into the code base, rotate the key/secret so that it's no longer active
 
@@ -53,7 +65,7 @@ These techniques make the loading of secrets  transparent to the developer.
 For .NET SDK (version 2.0 or higher) we have `dotnet secrets`, a tool provided by the .NET SDK that allows you to manage and protect sensitive information, such as API keys, connection strings, and other secrets, during development. The secrets are stored securely on your machine and can be accessed by your .NET applications.
 
 ```shell
-# Initialize dotnet secret 
+# Initialize dotnet secret
 dotnet user-secrets init
 
 # Adding secret
@@ -176,4 +188,4 @@ The following steps lay out a clear pathway to creating new secrets and then uti
 
 ### Validation
 
-Automated credential scanning can be performed on the code regardless of the programming language. Read more about it [here](../../continuous-integration/dev-sec-ops/secret-management/credential_scanning.md)
+Automated [credential scanning](credential_scanning.md) can be performed on the code regardless of the programming language.