You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# We want to run CI processes that can run independent of databricks as branch rules so that we dont # deploy at cost code that we already should know needs changing
2
-
# such as linting, and unit test for python, maybe dab? verify
3
-
# we run these on all pull request because if there is a hot fix it may not have passed through
4
-
# staging for example
5
-
# qqqq check this is up to date
6
-
name: CI - Pull Request Checks
7
-
8
-
# Run CI on all pull requests
9
-
on:
10
-
pull_request:
11
-
branches:
12
-
- '**'# all branches
13
-
14
-
jobs:
15
-
ci_checks:
16
-
name: "Linting, Unit Tests, DAB Verify"
17
-
runs-on: ubuntu-latest
18
-
19
-
steps:
20
-
# Checkout code
21
-
- name: Checkout repository
22
-
uses: actions/checkout@v4
23
-
24
-
# Set up Python
25
-
- name: Setup Python
26
-
uses: actions/setup-python@v5
27
-
with:
28
-
python-version: "3.x"
29
-
30
-
# Install dependencies used for linting and unit tests
31
-
- name: Install dependencies
32
-
run: pip install -r requirements-dev.txt
33
-
34
-
# Run python unit tests
35
-
- name: Run Unit Tests
36
-
run: pytest tests/unit
37
-
38
-
# Run python lint
39
-
# qqqq on example used flake8 instead
40
-
# pyproject.toml will need configuring
41
-
- name: Run Linting
42
-
run: pylint src
43
-
44
-
# qqqq to do run commit lint step and put in commit lint config
45
-
# see TELBlazor
46
-
- name: Commit lint
47
-
run: |
48
-
echo "Commit lint not implemented"
49
-
exit 1
50
-
51
-
# qqqq to do run version generation step and put in commit lint config
52
-
# see TELBlazor
53
-
- name: Version Generation Test Run
54
-
run: |
55
-
echo "Version test run not implemented"
56
-
exit 1
1
+
# We want to run CI processes that can run independent of databricks as branch rules so that we dont # deploy at cost code that we already should know needs changing
2
+
# such as linting, and unit test for python, maybe dab? verify
3
+
# we run these on all pull request because if there is a hot fix it may not have passed through
4
+
# staging for example
5
+
# qqqq check this is up to date
6
+
name: CI - Pull Request Checks
7
+
8
+
# Run CI on all pull requests
9
+
on:
10
+
pull_request:
11
+
branches:
12
+
- '**'# all branches
13
+
14
+
jobs:
15
+
ci_checks:
16
+
name: "Linting, Unit Tests, DAB Verify"
17
+
runs-on: ubuntu-latest
18
+
19
+
steps:
20
+
# Checkout code
21
+
- name: Checkout repository
22
+
uses: actions/checkout@v4
23
+
24
+
# Set up Python
25
+
- name: Setup Python
26
+
uses: actions/setup-python@v5
27
+
with:
28
+
python-version: "3.x"
29
+
30
+
# Install dependencies used for linting and unit tests
31
+
- name: Install dependencies
32
+
run: pip install -r requirements-dev.txt
33
+
34
+
# Run python unit tests
35
+
- name: Run Unit Tests
36
+
run: pytest tests/unit
37
+
38
+
# Run python lint
39
+
# qqqq on example used flake8 instead
40
+
# pyproject.toml will need configuring
41
+
- name: Run Linting
42
+
run: pylint src
43
+
44
+
# qqqq to do run commit lint step and put in commit lint config
45
+
# see TELBlazor
46
+
- name: Commit lint
47
+
run: |
48
+
echo "Commit lint not implemented"
49
+
exit 1
50
+
51
+
# qqqq to do run version generation step and put in commit lint config
It would be nice without the terminal and without needing to push to github to trigger unit tests, bundle validation, and bundle deployment for the local development user areas.
4
+
5
+
This doesnt seem do-able with a notebook, and enabling the terminal is an option, so using the databrick.yml ui deploy, and remembering to triggered any unit tests seems like it will be the process for now.
## We should think about these resource files I think potentially a .yml per layer bronze.yml may make sense
5
-
## We will not define schemas here
6
-
## We use this file to expose from databricks.yml the variables we need to set up the pipeline
7
-
## We will define too variables just for the set of pipelines here too if we start running layer based .ymls then we would have layer level variables here
8
-
###############################
9
-
## If we want specific pipeline resource file per .py file we should use this i think
10
-
# libraries:
11
-
# - notebook:
12
-
# path: ../../src/ingestion/ods_ingest.py
13
-
## if we want per layer maybe
14
-
# libraries:
15
-
# - glob:
16
-
# # if doing a pipeline per layer would do something like
17
-
# include: ../../src/ingestion/**.py
18
-
## if we want per domain maybe
19
-
# libraries:
20
-
# - glob:
21
-
# # if doing a pipeline per layer would do something like
22
-
# include: ../../src/ingestion/ods_*.py
23
-
###############################
24
-
25
-
# qqqq discus where want these things to live if it was using a wheel then the python file could be literally a table and a foreach
26
-
#####
27
-
# If we are running multlipe pipelines we may define all their vars at the top
28
-
#####
29
-
30
-
31
-
# qqqq
32
-
## im thinking var for in script var <-- also no because i cand get bundle.xyz and no all vars seem accessible everywhere i get catalog from databricks.yml
33
-
## bundle for vars originating from databricks.ymly
34
-
### i get vars from databricks
35
-
## pipeline. from pipeline files
36
-
## but files run, it shouldnt be bundle and pipeline it should represent the scope they come from
37
-
38
-
## qqqq i like the top level config value to pass i do not like construction vars in a yml instead of python but
39
-
# Error: cannot create pipeline: The target schema field is required for UC pipelines. Reason: DLT requires specifying a target schema for UC pipelines. Please use the TEMPORARY keyword in the CREATE MATERIALIZED VIEW or CREATE STREAMING TABLE statement if you do not wish to publish your dataset..
40
-
# Error: cannot update pipeline: Specified 'schema' field in the pipeline settings is illegal. Reason: Cannot unset 'schema' field once it's defined in the pipeline spec. Please create a new DLT pipeline. For more information about publishing modes, see https://docs.databricks.com/en/dlt/migrate-to-dpm.html.
41
-
variables:
42
-
layer:
43
-
default: bronze
44
-
description: bronze, silver, transfrormations etc
45
-
46
-
47
-
x-bronze-config: &bronze-config
48
-
bundle.env_name: ${var.env_name}
49
-
bundle.storage_account: ${var.storage_account} #storage is environment specific so defined in databricks.yml
50
-
pipeline.layer: ${var.layer} # if we are doing layer based resource files qqqq get from var
<<: *bronze-config #config anchor point for bronze layer so all pipelines in this file will have this set of configs
1
+
###############################
2
+
## POC notes - DELETE LATER
3
+
###############################
4
+
## We should think about these resource files I think potentially a .yml per layer bronze.yml may make sense
5
+
## We will not define schemas here
6
+
## We use this file to expose from databricks.yml the variables we need to set up the pipeline
7
+
## We will define too variables just for the set of pipelines here too if we start running layer based .ymls then we would have layer level variables here
8
+
###############################
9
+
## If we want specific pipeline resource file per .py file we should use this i think
10
+
# libraries:
11
+
# - notebook:
12
+
# path: ../../src/ingestion/ods_ingest.py
13
+
## if we want per layer maybe
14
+
# libraries:
15
+
# - glob:
16
+
# # if doing a pipeline per layer would do something like
17
+
# include: ../../src/ingestion/**.py
18
+
## if we want per domain maybe
19
+
# libraries:
20
+
# - glob:
21
+
# # if doing a pipeline per layer would do something like
22
+
# include: ../../src/ingestion/ods_*.py
23
+
###############################
24
+
25
+
# qqqq discus where want these things to live if it was using a wheel then the python file could be literally a table and a foreach
26
+
#####
27
+
# If we are running multlipe pipelines we may define all their vars at the top
28
+
#####
29
+
30
+
31
+
# qqqq
32
+
## im thinking var for in script var <-- also no because i cand get bundle.xyz and no all vars seem accessible everywhere i get catalog from databricks.yml
33
+
## bundle for vars originating from databricks.ymly
34
+
### i get vars from databricks
35
+
## pipeline. from pipeline files
36
+
## but files run, it shouldnt be bundle and pipeline it should represent the scope they come from
37
+
38
+
## qqqq i like the top level config value to pass i do not like construction vars in a yml instead of python but
39
+
# Error: cannot create pipeline: The target schema field is required for UC pipelines. Reason: DLT requires specifying a target schema for UC pipelines. Please use the TEMPORARY keyword in the CREATE MATERIALIZED VIEW or CREATE STREAMING TABLE statement if you do not wish to publish your dataset..
40
+
# Error: cannot update pipeline: Specified 'schema' field in the pipeline settings is illegal. Reason: Cannot unset 'schema' field once it's defined in the pipeline spec. Please create a new DLT pipeline. For more information about publishing modes, see https://docs.databricks.com/en/dlt/migrate-to-dpm.html.
41
+
variables:
42
+
layer:
43
+
default: bronze
44
+
description: bronze, silver, transfrormations etc
45
+
46
+
47
+
x-bronze-config: &bronze-config
48
+
bundle.env_name: ${var.env_name}
49
+
bundle.storage_account: ${var.storage_account} #storage is environment specific so defined in databricks.yml
50
+
pipeline.layer: ${var.layer} # if we are doing layer based resource files qqqq get from var
0 commit comments