Skip to content

Commit

Permalink
Docs: change terminology from CI/CD to WAP (#8356)
Browse files Browse the repository at this point in the history
  • Loading branch information
ozkatz authored Nov 11, 2024
1 parent 4e395f3 commit 3c5ab6f
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,16 @@ Comply with data audits.

In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward.

### CI/CD for Data
### Write-Audit-Publish

Data pipelines feed processed data from data lakes to downstream consumers like business dashboards and machine learning models. As more and more organizations rely on data to enable business critical decisions, data reliability and trust are of paramount concern. Thus, it’s important to ensure that production data adheres to the data governance policies of businesses. These data governance requirements can be as simple as a file format validation, schema check, or an exhaustive PII(Personally Identifiable Information) data removal from all of organization’s data.

Thus, to ensure the quality and reliability at each stage of the data lifecycle, data quality gates need to be implemented. That is, we need to run Continuous Integration(CI) tests on the data, and only if data governance requirements are met can the data can be promoted to production for business use.
Thus, to ensure the quality and reliability at each stage of the data lifecycle, data quality gates need to be implemented. That is, we need to run quality and correctness tests on the data, and only if data governance requirements are met can the data can be published to production for business use.

Everytime there is an update to production data, the best practice would be to run CI tests and then promote(deploy) the data to production. With lakeFS you can create hooks that make sure that only data that passed these tests will become part of production.
Everytime there is an update to production data, the best practice would be to run tests and then publish (deploy) the data to production. With lakeFS you can create hooks that make sure that only data that passed these tests will become part of production.

### Rollback

A rollback operation is used to to fix critical data errors immediately.

What is a critical data error? Think of a situation where erroneous or misformatted data causes a signficant issue with an important service or function. In such situations, the first thing to do is stop the bleeding.
Expand Down

0 comments on commit 3c5ab6f

Please sign in to comment.