Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync original #1

Merged
merged 63 commits into from
Nov 10, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
0a9cc9b
Bump version to 3.0.0 (#1744)
Ekrekr Jun 21, 2024
41e05de
Enable File-Specific Formatting in dataform format (#1754)
hiracky16 Jun 21, 2024
c91ec38
Assertions configs as protos (#1731)
Ekrekr Jun 24, 2024
4e13354
Fix vscode extension crashes on windows (#1751)
moker-spaghetti Jun 24, 2024
0634d3a
Update the version of the vscode extension (#1759)
GJMcGowan Jun 25, 2024
e9238a4
Fix empty object internal error (#1763)
Ekrekr Jun 26, 2024
b787f18
Split table tests (#1761)
Ekrekr Jun 26, 2024
c52cabb
Fix protected default, add a unit test for SQLX config parameters (#1…
Ekrekr Jun 26, 2024
9698b76
Declaration configs as protos (#1767)
Ekrekr Jun 26, 2024
da1d7bd
Add support for action configs table assertions and hermeticity, add …
Ekrekr Jun 27, 2024
3614c9e
Rethrow useful errors when reading workflow_settings.yaml (#1772)
Ekrekr Jun 28, 2024
b3638a8
View config as protos, via a new class (#1760)
Ekrekr Jun 28, 2024
461faf8
Incremental tables as protos, via a new class (#1773)
Ekrekr Jul 7, 2024
f2f6f88
Operation config as protos (#1766)
Ekrekr Jul 7, 2024
b322655
Fix vscode ref issue #1768 (#1769)
moker-spaghetti Jul 8, 2024
4015b05
Add workflow_settings.yaml validation/completion VSCode extension (#1…
moker-spaghetti Jul 8, 2024
d3d0194
fix: dry run of assertions failing (#1778)
ashish10alex Jul 8, 2024
cd61c3e
feat: vscode extension, more intutive compile errors (#1775)
ashish10alex Jul 9, 2024
97123fa
Update vscode version to incorporate recent changes (#1784)
GJMcGowan Jul 9, 2024
6635abc
Added proto definitions for Data Preparations (#1788)
fernst Jul 18, 2024
85f47d4
Handling for Data Preparation actions (#1789)
fernst Jul 29, 2024
2469634
support building on arm mac, linux environments (#1790)
bmagyarkuti Jul 30, 2024
c4fc417
fix(1781): go to definition when namePrefix is specified in workflow_…
ashish10alex Jul 30, 2024
1ef847f
Fix vscode server linting (#1797)
Ekrekr Aug 5, 2024
9959bcb
Make the stateless CLI package installation error clearer (#1795)
Ekrekr Aug 5, 2024
8dcccd8
Regenerate configs reference with camelCase fields (#1796)
Ekrekr Aug 7, 2024
1c188e4
Fix notebook dependency targets (#1792)
Ekrekr Aug 7, 2024
e5316d5
Bumped version to 3.0.1 (#1798)
fernst Aug 7, 2024
a918c0c
Remove some outdated configs references, fix generated formatting (#1…
Ekrekr Aug 7, 2024
54dfa9c
Fix catchall errors (#1802)
Ekrekr Aug 8, 2024
dc5a001
Update WORKSPACE (#1512)
a2wd Aug 15, 2024
ae85965
Refresh test credentials (#1810)
Ekrekr Aug 15, 2024
2531b12
Update output for Data Preparation processing (#1808)
fernst Aug 15, 2024
7f69306
Sync proto structure and message names for data preparation. (#1811)
fernst Aug 16, 2024
bd7d6fb
Temporarily disable docs script during publish (#1817)
Ekrekr Aug 22, 2024
674b1e6
Added support for compilation overrides in Data Preparations (#1818)
fernst Sep 4, 2024
1868d37
Bump dataprep proto to latest config (#1825)
Ekrekr Sep 4, 2024
104c840
fix: vscode extension crash on windows due to *.cmd spawn (#1814)
moker-spaghetti Sep 5, 2024
b309aa0
Update unit test's result messages (#1812)
spider-man-tm Sep 5, 2024
1f48a0e
Restore ability to unit test views (#1824)
bmagyarkuti Sep 5, 2024
4b71bbf
Bump elliptic from 6.5.4 to 6.5.7 (#1831)
dependabot[bot] Sep 5, 2024
4d20929
fix: dry run fails when table/view/assertion already does not exsist …
ashish10alex Sep 5, 2024
bb36a53
Revert "fix: dry run fails when table/view/assertion already does not…
Ekrekr Sep 9, 2024
86254eb
Update Data Preparation processor to remove proto dependency. (#1839)
fernst Sep 11, 2024
840bb5b
MOSS Cap1 (#1833)
Tuseeq1 Sep 13, 2024
05341b7
Correct permission and branch name (#1845)
Tuseeq1 Sep 14, 2024
bb061c4
Remove cloudbuild.yaml (#1847)
Tuseeq1 Sep 16, 2024
f36a513
Make published packages include the version.bzl file (#1848)
Ekrekr Sep 17, 2024
09f606a
Remove data preparation proto definition. (#1849)
fernst Sep 19, 2024
338262b
Update the npm package version (#1852)
Tuseeq1 Sep 30, 2024
83afdbe
Branch name from master to main (#1853)
Tuseeq1 Sep 30, 2024
cb4cec4
Local test fix with npm install on cloudbuild (#1850)
Tuseeq1 Sep 30, 2024
00462d7
Fixing the name for automaticallty created PR (#1856)
Tuseeq1 Oct 1, 2024
cf9bfee
Update the npm package version (#1855)
Tuseeq1 Oct 1, 2024
6fcdab1
Creating release notes as part of publishing process (#1857)
Tuseeq1 Oct 4, 2024
5f1b9a0
fix `uniqueKey` assertions for views, incremental_tables (#1836)
bmagyarkuti Oct 7, 2024
9b6a2ce
Bump version to 3.5 to rollup uniquekeys fix (#1858)
Ekrekr Oct 11, 2024
ecd62d0
Bump the patch version to 3.0.6 (#1859)
Tuseeq1 Oct 14, 2024
fcfd46e
Change variable name in cloudbuild.yaml (#1860)
Tuseeq1 Oct 14, 2024
e16717f
Bump the pacth version (#1861)
Tuseeq1 Oct 15, 2024
136a990
Added logic to fully resolve targets (with prefixes/suffixes) when up…
fernst Nov 1, 2024
031fb9b
Update the npm package version (#1867)
Tuseeq1 Nov 4, 2024
0dff7a8
Bump rollup from 2.8.0 to 2.79.2 (#1854)
dependabot[bot] Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added logic to fully resolve targets (with prefixes/suffixes) when up…
…dating the Data Preparation YAML. (dataform-co#1866)

* Added logic to fully resolve targets (with prefixes/suffixes) when updating the Data Preparation YAML.

* Added logic for target finalization.
  • Loading branch information
fernst authored Nov 1, 2024
commit 136a9904aa51797d941a5d68375bf29100432bd4
22 changes: 16 additions & 6 deletions core/actions/data_preparation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,12 @@ export class DataPreparation extends ActionBuilder<dataform.DataPreparation> {
const targets = this.getTargets(dataPreparationAsJson as {
[key: string]: any;
});
this.proto.targets = targets.map(target =>
this.applySessionToTarget(target, session.projectConfig, config.filename, true)
const resolvedTargets = targets.map(target =>
this.applySessionToTarget(target, session.projectConfig, config.filename, true)
)
// Finalize list of targets.
this.proto.targets = resolvedTargets.map(target =>
this.finalizeTarget(target)
);
this.proto.canonicalTargets = targets.map(target =>
this.applySessionToTarget(target, session.canonicalProjectConfig)
Expand All @@ -54,7 +58,7 @@ export class DataPreparation extends ActionBuilder<dataform.DataPreparation> {

// Set the unique target key as the first target defined.
// TODO: Remove once multiple targets are supported.
this.proto.target = this.proto.targets[0];
this.proto.target = resolvedTargets[0];
this.proto.canonicalTarget = this.proto.canonicalTargets[0];

this.proto.tags = config.tags;
Expand Down Expand Up @@ -161,15 +165,21 @@ export class DataPreparation extends ActionBuilder<dataform.DataPreparation> {
this.session.projectConfig)
// Convert resolved target into a Data Preparation Table Reference
let resolvedTableReference : {[key: string]: string} = {
table: resolvedTarget.name,
table: this.session.finalizeName(resolvedTarget.name),
}

// Ensure project and dataset field are added in order
if (resolvedTarget.schema) {
resolvedTableReference = { dataset: resolvedTarget.schema, ...resolvedTableReference }
resolvedTableReference = {
dataset: this.session.finalizeSchema(resolvedTarget.schema),
...resolvedTableReference
}
}
if (resolvedTarget.database) {
resolvedTableReference = { project: resolvedTarget.database, ...resolvedTableReference }
resolvedTableReference = {
project: this.session.finalizeDatabase(resolvedTarget.database),
...resolvedTableReference
}
}
return resolvedTableReference;
}
Expand Down
12 changes: 12 additions & 0 deletions core/actions/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,18 @@ export abstract class ActionBuilder<T> {
return target;
}

public finalizeTarget(
targetFromConfig: dataform.Target
): dataform.Target {
return dataform.Target.create({
name: this.session.finalizeName(targetFromConfig.name),
schema: targetFromConfig.schema ?
this.session.finalizeSchema(targetFromConfig.schema) : undefined,
database: targetFromConfig.database ?
this.session.finalizeDatabase(targetFromConfig.database) : undefined
})
}

/** Retrieves the filename from the config. */
public abstract getFileName(): string;

Expand Down
39 changes: 23 additions & 16 deletions core/main_test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -955,7 +955,14 @@ nodes:
});

test(`data preparations resolves compilation overrides before encoding`, () => {
const projectDir = createSimpleDataPreparationProject();
const projectDir = createSimpleDataPreparationProject(`
defaultProject: defaultProject
defaultDataset: defaultDataset
defaultLocation: US
projectSuffix: projectSuffix
datasetSuffix: datasetSuffix
namePrefix: tablePrefix
`);
const dataPreparationYaml = `
configuration:
errorTable:
Expand Down Expand Up @@ -1013,16 +1020,16 @@ nodes:
const resolvedYaml = `
configuration:
errorTable:
project: defaultProject
dataset: defaultDataset
table: error
project: defaultProject_projectSuffix
dataset: defaultDataset_datasetSuffix
table: tablePrefix_error
nodes:
- id: node1
source:
table:
project: defaultProject
dataset: defaultDataset
table: src
project: defaultProject_projectSuffix
dataset: defaultDataset_datasetSuffix
table: tablePrefix_src
generated:
sourceGenerated:
sourceSchema:
Expand All @@ -1041,9 +1048,9 @@ nodes:
nodeId: node1
destination:
table:
project: defaultProject
dataset: defaultDataset
table: dest
project: defaultProject_projectSuffix
dataset: defaultDataset_datasetSuffix
table: tablePrefix_dest
generated:
sourceGenerated:
sourceSchema:
Expand Down Expand Up @@ -1072,9 +1079,9 @@ nodes:
asPlainObject([
{
target: {
database: "defaultProject",
schema: "defaultDataset",
name: "dest"
database: "defaultProject_projectSuffix",
schema: "defaultDataset_datasetSuffix",
name: "tablePrefix_dest"
},
canonicalTarget: {
database: "defaultProject",
Expand All @@ -1083,9 +1090,9 @@ nodes:
},
targets: [
{
database: "defaultProject",
schema: "defaultDataset",
name: "dest"
database: "defaultProject_projectSuffix",
schema: "defaultDataset_datasetSuffix",
name: "tablePrefix_dest"
}
],
canonicalTargets: [
Expand Down