Cloud functions ingest: backfill CLI tool + minor improvements#190

Merged

danieldeleo merged 71 commits intoGoogleCloudPlatform:masterfrom

jaketf:cloud-functions-ingest

Nov 18, 2020

Commits on Oct 23, 2020

gcs ingest cloud function
Jacob Ferriero
committed

Commits on Oct 27, 2020

address review feedback
Jacob Ferriero
committed
remove extra newline
Jacob Ferriero
committed
remove project files
Jacob Ferriero
committed

Commits on Oct 28, 2020

regex destination details env var
Jacob Ferriero
committed
ignore intellij .idea/ configs
Jacob Ferriero
committed
incremental loads docs
Jacob Ferriero
committed
update TODOs
Jacob Ferriero
committed
Merge branch 'master' into cloud-functions-ingest
Jacob Ferriero
authored
remove bad labels
Jacob Ferriero
committed
Merge branch 'cloud-functions-ingest' of github.com:jaketf/bigquery-utils into cloud-functions-ingest
Jacob Ferriero
committed
move external query recursive search todo
Jacob Ferriero
committed

Commits on Oct 30, 2020

passing integration tests
Jacob Ferriero
committed

Commits on Oct 31, 2020

add ci for gcs_event_based_ingest cloud function
Jacob Ferriero
committed
handle duplicate pubsub notification
Jacob Ferriero
committed

Commits on Nov 2, 2020

Merge branch 'master' into cloud-functions-ingest
ryanmcdowell
authored
fixup ci dockerfile
Jacob Ferriero
committed
more ci
Jacob Ferriero
committed
Merge branch 'cloud-functions-ingest' of github.com:jaketf/bigquery-utils into cloud-functions-ingest
Jacob Ferriero
committed

Commits on Nov 3, 2020

better defaults, faster ci
Jacob Ferriero
committed
add integration test for partitioned tables / data
Jacob Ferriero
committed

Commits on Nov 4, 2020

parallelize tests 4x+ speed up in ci
Jacob Ferriero
committed
fixup docs
Jacob Ferriero
committed

Commits on Nov 5, 2020

add terraform module for deployment
Jacob Ferriero
committed
remove useless gitignore
Jacob Ferriero
committed
add pytest.mark.IT
Jacob Ferriero
committed

Commits on Nov 9, 2020

add note on alternatives
Jacob Ferriero
committed

Commits on Nov 11, 2020

wip
Jacob Ferriero
committed
move cloud functions to tools
Jacob Ferriero
committed
Merge branch 'chore/move-gcf-to-tools' into cloud-functions-ingest
Jacob Ferriero
committed
wip backfill cli
Jacob Ferriero
committed
backfill cli
Jacob Ferriero
committed
update docs
Jacob Ferriero
committed
mock function name
Jacob Ferriero
committed
Merge branch 'master' into cloud-functions-ingest
Jacob Ferriero
authored
restructure tests
Jacob Ferriero
committed
add backfill cli test
Jacob Ferriero
committed
Merge branch 'cloud-functions-ingest' of github.com:jaketf/bigquery-utils into cloud-functions-ingest
Jacob Ferriero
committed
update test docstring
Jacob Ferriero
committed

Commits on Nov 12, 2020

dry up pytest fixtures
Jacob Ferriero
committed
docs on tests fixtures
Jacob Ferriero
committed
fixup docs formatting
Jacob Ferriero
committed
fix default regex
Jacob Ferriero
committed
fixup regex docs
Jacob Ferriero
committed

Commits on Nov 13, 2020

Update tools/cloud_functions/gcs_event_based_ingest/backfill.py

Jacob Ferriero
and
mik-laj
authored
Update tools/cloud_functions/gcs_event_based_ingest/backfill.py

Jacob Ferriero
and
mik-laj
authored
fixup! tests
Jacob Ferriero
committed
add docs example / guidance for overriding destination regex
Jacob Ferriero
committed
fixup backfill usage docs
Jacob Ferriero
committed
fixup pylint
Jacob Ferriero
committed
fixup regex docs
Jacob Ferriero
committed
fixup document and support more flexible deployment scenarios
Jacob Ferriero
committed

Commits on Nov 17, 2020

add support for Cloud Functions direct triggering
Jacob Ferriero
committed
Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest/main.py

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/README.md

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/README.md

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/tests/conftest.py

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/tests/cli/test_backfill.py

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/README.md

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/README.md

Jacob Ferriero
and
danieldeleo
authored
Update tools/cloud_functions/gcs_event_based_ingest/README.md

Jacob Ferriero
and
danieldeleo
authored
Fixup address review feedback and fix linters
Jacob Ferriero
committed
Merge branch 'cloud-functions-ingest' of github.com:jaketf/bigquery-utils into cloud-functions-ingest
Jacob Ferriero
committed
fixup google style imports everywhere
Jacob Ferriero
committed
set pytest workers to auto
Jacob Ferriero
committed
don't alias use full import paths
Jacob Ferriero
committed
fixup! move --workers=auto to pytest.ini
Jacob Ferriero
committed
fixup don't reverse implement storage.Client.get_bucket
Jacob Ferriero
committed

Commits on Nov 18, 2020

fixup import google style
Jacob Ferriero
committed
fixup ci cloudbuild
Jacob Ferriero
committed
remove trailing whitespace
Jacob Ferriero
committed