Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create guide for Machine Learning Engine operators #8207 #8968

Closed
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
155 commits
Select commit Hold shift + click to select a range
a28c66f
[AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#…
oxymor0n Apr 30, 2020
4a1d71d
Fix the process of requirements generations (#8648)
potiuk Apr 30, 2020
b185b36
Reduce response payload size of /dag_stats and /task_stats (#8633)
XD-DENG Apr 30, 2020
4421f01
Improve template capabilities of EMR job and step operators (#8572)
oripwk May 1, 2020
6560f29
Enhanced documentation around Cluster Policy (#8661)
vardancse May 1, 2020
511d98e
[AIRFLOW-4363] Fix JSON encoding error (#8287)
retornam May 1, 2020
ce50538
Add check for pre-2.0 style hostname_callable config value (#8637)
dimberman May 1, 2020
0a7b500
Fix displaying Executor Class Name in "Base Job" table (#8679)
kaxil May 2, 2020
d92e848
Persist start/end date and duration for DummyOperator Task Instance (…
XD-DENG May 2, 2020
0954140
Ensure "started"/"ended" in tooltips are not shown if job not started…
XD-DENG May 2, 2020
19ac45a
Add support for fetching logs from running pods (#8626)
msumit May 3, 2020
1100cea
Remove _get_pretty_exception_message in PrestoHook
fusuiyi123 May 3, 2020
62796b9
Improve tutorial - Include all imports statements (#8670)
Lyalpha May 3, 2020
dd6a7bc
Group Google services in one section (#8623)
mik-laj May 3, 2020
ac59735
Refactor test_variable_command.py (#8535)
May 3, 2020
bc45fa6
Add system test and docs for Facebook Ads operators (#8503)
randr97 May 3, 2020
0b598a2
Fix connection add/edit for spark (#8685)
XD-DENG May 3, 2020
ffbbbfc
Sort connection type list in add/edit page alphabetically (#8692)
XD-DENG May 3, 2020
d8cb0b5
Support k8s auth method in Vault Secrets provider (#8640)
May 3, 2020
67caae0
Add system test for gcs_to_bigquery (#8556)
joppevos May 4, 2020
aec768b
[AIRFLOW-7008] Add perf kit with common used decorators/contexts (#7650)
mik-laj May 4, 2020
c3a46b9
Invalid output in test_variable assertion (#8698)
mik-laj May 4, 2020
5ddc458
Change provider:GCP to provider:Google for Labeler Bot (#8697)
mik-laj May 4, 2020
caa60b1
Remove config side effects from tests (#8607)
turbaszek May 4, 2020
923f423
Check consistency between the reference list and howto directory (#8690)
mik-laj May 4, 2020
b31ad51
Prevent clickable sorting on non sortable columns in TI view (#8681)
Acehaidrey May 4, 2020
6600e47
Import Connection directly from multiprocessing.connection. (#8711)
jhtimmins May 4, 2020
2c92a29
Fix typo in Google Display & Video 360 guide
michalslowikowski00 May 5, 2020
41b4c27
Carefully parse warning messages when building documentation (#8693)
mik-laj May 5, 2020
8d6f1aa
Support num_retries field in env var for GCP connection (#8700)
mik-laj May 5, 2020
c717d12
Add __repr__ for DagTag so tags display properly in /dagmodel/show (#…
XD-DENG May 5, 2020
487b5cc
Add guide for Apache Spark operators (#8305)
May 5, 2020
520aeed
Fix pickling failure when spawning processes (#8671)
jhtimmins May 6, 2020
25ee421
Support all RuntimeEnvironment parameters in DataflowTemplatedJobStar…
mik-laj May 6, 2020
d923b5b
Add jinja template test for AirflowVersion (#8505)
mik-laj May 6, 2020
e673413
Avoid loading executors in jobs (#7888)
mik-laj May 6, 2020
3437bea
Optimize count query on /home (#8729)
mik-laj May 6, 2020
336aa27
Correctly deserialize dagrun_timeout field on DAGs (#8735)
ashb May 6, 2020
2e9ef45
Stop Stalebot on Github issues (#8738)
kaxil May 6, 2020
fd6e057
Make loading plugins from entrypoint fault-tolerant (#8732)
kaxil May 6, 2020
bd29ee3
Ensure test_logging_config.test_reload_module works in spawn mode. (#…
jhtimmins May 6, 2020
d15839d
Latest debian-buster release broke image build (#8758)
potiuk May 7, 2020
ff5b701
Add google_api_to_s3_transfer example dags and system tests (#8581)
feluelle May 7, 2020
7c04604
Add google_api_to_s3_transfer docs howto link (#8761)
feluelle May 7, 2020
723c52c
Add documentation for SpannerDeployInstanceOperator (#8750)
ephraimbuddy May 7, 2020
6e4f5fa
[AIRFLOW-4568]The ExternalTaskSensor should be configurable to raise …
lokeshlal May 7, 2020
b7566e1
Add SQL query tracking for pytest (#8754)
mik-laj May 8, 2020
58aefb2
Added SDFtoGCSOperator (#8740)
michalslowikowski00 May 8, 2020
b37ce29
Patch Pool.DEFAULT_POOL_NAME in BaseOperator (#8587)
vshshjn7 May 8, 2020
2bd3e76
Support same short flags for `create user` as 1.10 did for `user_crea…
ashb May 8, 2020
09770e4
Add WorldRemit as Airflow user (#8786)
May 8, 2020
a091c1f
fix typing errors reported by dmypy (#8773)
May 8, 2020
42c5975
Update example SingularityOperator DAG (#8790)
ashb May 8, 2020
791d1a7
Backport packages are renamed to include backport in their name (#8767)
potiuk May 9, 2020
100f530
Fixed test-target command (#8795)
potiuk May 9, 2020
db1b51d
Make celery worker_prefetch_multiplier configurable (#8695)
nadflinn May 9, 2020
bc19778
[AIP-31] Implement XComArg to pass output from one operator to the ne…
jonathanshir May 9, 2020
7506c73
Add default `conf` parameter to Spark JDBC Hook (#8787)
May 9, 2020
5e1c33a
Fix docs on creating CustomOperator (#8678)
JonnyWaffles May 10, 2020
21cc7d7
Document default timeout value for SSHOperator (#8744)
abhilash1in May 10, 2020
cd635dd
[AIRFLOW-5906] Add authenticator parameter to snowflake_hook (#8642)
koszti May 10, 2020
c7788a6
Add imap_attachment_to_s3 example dag and system test (#8669)
feluelle May 10, 2020
a715aa6
Correctly store non-default Nones in serialized tasks/dags (#8772)
ashb May 10, 2020
280f1f0
Correctly restore upstream_task_ids when deserializing Operators (#8775)
ashb May 10, 2020
cbebed2
Allow passing backend_kwargs to AWS SSM client (#8802)
kaxil May 10, 2020
79ef8be
Added Upload Multiple Entity Read Files to specified big query datase…
michalslowikowski00 May 10, 2020
e1cc17e
Remove old airflow logger causing side effects in tests (#8746)
kaxil May 10, 2020
9bb91ef
Add comments to breeze scripts (#8797)
potiuk May 10, 2020
493b685
Add separate example DAGs and system tests for google cloud speech (#…
ephraimbuddy May 10, 2020
bed1995
Avoid color info in response of /dag_stats & /task_stats (#8742)
XD-DENG May 11, 2020
b59adab
Support cron presets in date_range function (#7777)
Rcharriol May 11, 2020
5f3774a
[AIRFLOW-6921] Fetch celery states in bulk (#7542)
mik-laj May 11, 2020
d5c4001
Useful help information in test-target and docker-compose commands (#…
potiuk May 11, 2020
a6434a5
Fix bash command in performance test dag (#8812)
ashb May 11, 2020
0c3db84
[AIRFLOW-7068] Create EC2 Hook, Operator and Sensor (#7731)
mustafagok May 11, 2020
5ae76d8
Option to set end_date for performance testing dag. (#8817)
ashb May 11, 2020
2ec0130
[AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream (#7…
teddyhartanto May 11, 2020
1fb9f07
Synchronize extras between airflow and providers (#8819)
potiuk May 11, 2020
d590e5e
Add option to propagate tags in ECSOperator (#8811)
JPonte May 11, 2020
f410d64
Use fork when test relies on mock.patch in parent process. (#8794)
jhtimmins May 11, 2020
3ad4f96
[AIRFLOW-1156] BugFix: Unpausing a DAG with catchup=False creates an …
kaxil May 11, 2020
4375607
Fix typo. 'zobmies' => 'zombies'. (#8832)
jhtimmins May 12, 2020
78a48db
Add support for non-default orientation in `dag show` command (#8834)
klsnreddy May 12, 2020
7533378
Access function to be pickled as attribute, not method, to avoid erro…
jhtimmins May 12, 2020
1d12c34
Refactor BigQuery check operators (#8813)
turbaszek May 12, 2020
4b06fde
Fix Flake8 errors (#8841)
kaxil May 12, 2020
6911dfe
Fix template fields in Google operators (#8840)
turbaszek May 12, 2020
01db738
Azure storage 0.37.0 is not installable any more (#8833)
potiuk May 12, 2020
578fc51
[AIRFLOW-4543] Update slack operator to support slackclient v2 (#5519)
serkef May 12, 2020
7236862
[AIRFLOW-2310] Enable AWS Glue Job Integration (#6007)
abdulbasitds May 12, 2020
8b54919
Refactor BigQuery hook methods to use python library (#8631)
turbaszek May 12, 2020
7d69987
Remove duplicate code from perf_kit (#8843)
kaxil May 12, 2020
e1e833b
Update GoogleBaseHook to not follow 308 and use 60s timeout (#8816)
waiyan1612 May 13, 2020
8a94d18
Fix Environment Variable in perf/scheduler_dag_execution_timing.py (#…
kaxil May 13, 2020
ed3f513
Correctly pass sleep time from AWSAthenaOperator down to the hook. (#…
ashb May 13, 2020
f1dc2e0
The librabbitmq library stopped installing for python3.7 (#8853)
potiuk May 13, 2020
c3af681
Convert tests/jobs/test_base_job.py to pytest (#8856)
ashb May 13, 2020
81fb9d6
Add metric for monitoring email notification failures (#8771)
May 13, 2020
2878f17
Relax Flask-Appbuilder version to ~=2.3.4 (#8857)
feluelle May 13, 2020
e61b9bb
Add AWS EMR System tests (#8618)
xinbinhuang May 13, 2020
fc862a3
Do not create a separate process for one task in CeleryExecutor (#8855)
mik-laj May 14, 2020
961c710
Make Custom XCom backend a subsection of XCom docs (#8869)
turbaszek May 14, 2020
fe42191
Don't use ProcessorAgent to test ProcessorManager (#8871)
ashb May 14, 2020
4813b94
Create log file w/abs path so tests pass on MacOS (#8820)
jhtimmins May 14, 2020
35c523f
Fix list formatting of plugins doc. (#8873)
ashb May 15, 2020
85bbab2
Add EMR operators howto docs (#8863)
xinbinhuang May 15, 2020
f82ad45
Fix KubernetesPodOperator pod name length validation (#8829)
dsaiztc May 15, 2020
92585ca
Added automated release notes generation for backport operators (#8807)
potiuk May 15, 2020
82de6f7
Spend less time waiting for DagFileProcessor processes to complete (#…
ashb May 15, 2020
a3a4bac
JIRA and Github issues explanation (#8539)
mschickensoup May 16, 2020
f4edd90
Speed up TestAwsLambdaHook by not actually running a function (#8882)
ashb May 16, 2020
15273f0
Check for same task instead of Equality to detect Duplicate Tasks (#8…
kaxil May 16, 2020
a3a3411
Fix master failing on generating requirements (#8885)
potiuk May 16, 2020
f3521fb
Regenerate readme files for backport package release (#8886)
potiuk May 16, 2020
f6d5917
Updated docs for experimental API /dags/<DAG_ID>/dag_runs (#8800)
randr97 May 16, 2020
707bb0c
[AIRFLOW-6535] Add AirflowFailException to fail without any retry (#7…
jstern May 16, 2020
a546a10
Add Snowflake system test (#8422)
dhuang May 16, 2020
8985df0
Monitor pods by labels instead of names (#6377)
dimberman May 16, 2020
ff342fc
Added SalesforceHook missing method to return only dataframe (#8565) …
pranjalmittal May 17, 2020
12c5e5d
Prepare release candidate for backport packages (#8891)
potiuk May 17, 2020
2121f49
Avoid failure on transient requirements in CI image (#8892)
potiuk May 17, 2020
841d816
Allow setting the pooling time in DLPHook (#8824)
xuan616 May 19, 2020
dd57ec9
Fix task and dag stats on home page (#8865)
May 19, 2020
375d1ca
Release candidate 2 for backport packages 2020.05.20 (#8898)
potiuk May 19, 2020
bae5cc2
Fix race in Celery tests by pre-creating result tables (#8909)
ashb May 19, 2020
499493c
[AIRFLOW-6586] Improvements to gcs sensor (#7197)
May 19, 2020
ce7fdea
UX Fix: Prevent undesired text selection with DAG title selection in …
ryanahamilton May 19, 2020
fef00e5
Use Debian's provided JRE from Buster (#8919)
ashb May 20, 2020
5360045
Fix incorrect Env Var to stop Scheduler from creating DagRuns (#8920)
kaxil May 20, 2020
51d9557
Re-run all tests when Dockerfile or Github worflow change (#8924)
ashb May 20, 2020
c6224e2
Remove unused self.max_threads argument in SchedulerJob (#8935)
kaxil May 21, 2020
12c22e0
Added Greytip to Airflow Users list (#8887)
Sarankrishna May 21, 2020
8476c1e
Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work (#8938)
ashb May 21, 2020
f17b4bb
Fix DagRun Prefix for Performance script (#8934)
kaxil May 21, 2020
a9dfd7d
Remove side-effect of session in FAB (#8940)
mik-laj May 21, 2020
f3f74c7
Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#…
harrisjoseph May 21, 2020
8d3acd7
Fix docstring in DagFileProcessor._schedule_task_instances (#8948)
kaxil May 21, 2020
47413d9
Remove singularity from CI images (#8945)
ashb May 21, 2020
16206cd
Update example webserver_config.py to show correct CSRF config (#8944)
ashb May 21, 2020
97b6cc7
Add note in Updating.md about the removel of DagRun.ID_PREFIX (#8949)
kaxil May 21, 2020
41481bb
Python base images are stored in cache (#8943)
potiuk May 21, 2020
b26b3ca
Don't hard-code constants in scheduler_dag_execution_timing (#8950)
ashb May 21, 2020
113982b
Make scheduler_dag_execution_timing grok dynamic start date of elasti…
ashb May 21, 2020
90a07d8
Cache 1 10 ci images (#8955)
dimberman May 21, 2020
dd72040
Pin Version of Azure Cosmos to <4 (#8956)
kaxil May 21, 2020
94a7673
Pin google-cloud-datacatalog to <0.8 (#8957)
kaxil May 21, 2020
9a4a2d1
[AIRFLOW-5262] Update timeout exception to include dag (#8466)
curiousjazz77 May 22, 2020
b055151
Add context to execution_date_fn in ExternalTaskSensor (#8702)
Acehaidrey May 22, 2020
f107338
Add support for spark python and submit tasks in Databricks operator(…
siddartha-ravichandran May 22, 2020
e742ef7
Fix typo in test_project_structure (#8978)
mik-laj May 23, 2020
4d67704
Remove duplicate line from CONTRIBUTING.rst (#8981)
kaxil May 23, 2020
db70da2
Flush pending Sentry exceptions before exiting (#7232)
mikeclarke May 23, 2020
cf5cf45
Support YAML input for CloudBuildCreateOperator (#8808)
joppevos May 23, 2020
bdb8369
Add secrets to test_deprecated_packages (#8979)
mik-laj May 23, 2020
f3456b1
Fix formatting code block in TESTING.rst (#8985)
ad-m May 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Pin Version of Azure Cosmos to <4 (#8956)
Old Repo: https://github.com/Azure/azure-cosmos-python
New Repo: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/cosmos/azure-cosmos

azure-cosmos==4.0.0 was released on 20 May 2020 that breaks Airflow
  • Loading branch information
kaxil authored May 21, 2020
commit dd7204066fb3fbf6c47df4c101b99a44502c7e8e
64 changes: 32 additions & 32 deletions requirements/requirements-python3.6.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Editable install with no version control (apache-airflow==2.0.0.dev0)
Authlib==0.14.2
Authlib==0.14.3
Babel==2.8.0
Flask-AppBuilder==2.3.4
Flask-Babel==1.0.0
Expand All @@ -14,7 +14,7 @@ Flask-WTF==0.14.3
Flask==1.1.2
GitPython==3.1.2
HeapDict==1.0.1
JPype1==0.7.4
JPype1==0.7.5
JayDeBeApi==1.1.1
Jinja2==2.10.3
Mako==1.1.2
Expand Down Expand Up @@ -45,7 +45,7 @@ apispec==1.3.3
appdirs==1.4.4
argcomplete==1.11.1
asn1crypto==1.3.0
astroid==2.4.1
astroid==2.3.3
async-generator==1.10
async-timeout==3.0.1
atlasclient==1.0.0
Expand All @@ -72,9 +72,9 @@ beautifulsoup4==4.7.1
billiard==3.6.3.0
black==19.10b0
blinker==1.4
boto3==1.13.9
boto3==1.13.14
boto==2.49.0
botocore==1.16.9
botocore==1.16.14
bowler==0.8.0
cached-property==1.5.1
cachetools==4.1.0
Expand All @@ -84,7 +84,7 @@ celery==4.4.2
certifi==2020.4.5.1
cffi==1.14.0
cfgv==3.1.0
cfn-lint==0.31.1
cfn-lint==0.32.1
cgroupspy==0.1.6
chardet==3.0.4
click==6.7
Expand Down Expand Up @@ -114,13 +114,13 @@ ecdsa==0.15
elasticsearch-dbapi==0.1.0
elasticsearch-dsl==7.2.0
elasticsearch==7.5.1
email-validator==1.1.0
email-validator==1.1.1
entrypoints==0.3
enum34==1.1.10
eventlet==0.25.2
execnet==1.7.1
facebook-business==7.0.0
fastavro==0.23.3
facebook-business==7.0.1
fastavro==0.23.4
filelock==3.0.12
fissix==20.5.1
flake8-colors==0.1.6
Expand All @@ -129,7 +129,7 @@ flaky==3.6.1
flask-swagger==0.2.13
flower==0.9.4
freezegun==0.3.15
fsspec==0.7.3
fsspec==0.7.4
funcsigs==1.0.2
future-fstrings==1.2.0
future==0.18.2
Expand All @@ -141,23 +141,23 @@ google-api-core==1.17.0
google-api-python-client==1.8.3
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-auth==1.14.3
google-auth==1.15.0
google-cloud-automl==0.10.0
google-cloud-bigquery-datatransfer==1.0.0
google-cloud-bigquery==1.24.0
google-cloud-bigtable==1.2.1
google-cloud-container==0.5.0
google-cloud-core==1.3.0
google-cloud-datacatalog==0.7.0
google-cloud-dataproc==0.7.0
google-cloud-datacatalog==0.8.0
google-cloud-dataproc==0.8.0
google-cloud-dlp==0.15.0
google-cloud-kms==1.4.0
google-cloud-language==1.3.0
google-cloud-logging==1.15.0
google-cloud-monitoring==0.36.0
google-cloud-pubsub==1.5.0
google-cloud-redis==1.0.0
google-cloud-secret-manager==0.2.0
google-cloud-secret-manager==1.0.0
google-cloud-spanner==1.16.0
google-cloud-speech==1.3.2
google-cloud-storage==1.28.1
Expand All @@ -172,19 +172,19 @@ graphviz==0.14
greenlet==0.4.15
grpc-google-iam-v1==0.12.3
grpcio-gcp==0.2.2
grpcio==1.28.1
grpcio==1.29.0
gunicorn==19.10.0
hdfs==2.5.8
hmsclient==0.1.1
httplib2==0.17.3
httplib2==0.18.1
humanize==0.5.1
hvac==0.10.1
hvac==0.10.2
identify==1.4.15
idna-ssl==1.1.0
idna==2.9
ijson==2.6.1
imagesize==1.2.0
immutables==0.13
immutables==0.14
importlib-metadata==1.6.0
importlib-resources==1.5.0
inflection==0.4.0
Expand Down Expand Up @@ -219,18 +219,18 @@ mccabe==0.6.1
mock==4.0.2
mongomock==3.19.0
monotonic==1.5
more-itertools==8.2.0
more-itertools==8.3.0
moto==1.3.14
msgpack==1.0.0
msrest==0.6.13
msrest==0.6.14
msrestazure==0.6.3
multi-key-dict==2.0.3
multidict==4.7.5
multidict==4.7.6
mypy-extensions==0.4.3
mypy==0.770
mysql-connector-python==8.0.18
mysqlclient==1.3.14
nbclient==0.2.0
nbclient==0.3.1
nbformat==5.0.6
nest-asyncio==1.3.3
networkx==2.4
Expand All @@ -240,7 +240,7 @@ ntlm-auth==1.4.0
numpy==1.18.4
oauthlib==2.1.0
oscrypto==1.2.0
packaging==20.3
packaging==20.4
pandas-gbq==0.13.2
pandas==1.0.3
papermill==2.1.1
Expand All @@ -260,14 +260,14 @@ pre-commit==2.4.0
presto-python-client==0.7.0
prison==0.1.3
prompt-toolkit==3.0.5
protobuf==3.11.3
protobuf==3.12.1
psutil==5.7.0
psycopg2-binary==2.8.5
ptyprocess==0.6.0
py4j==0.10.7
py==1.8.1
pyOpenSSL==19.1.0
pyarrow==0.17.0
pyarrow==0.17.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
pycodestyle==2.6.0
Expand Down Expand Up @@ -307,7 +307,7 @@ pytzdata==2019.3
pywinrm==0.4.1
pyzmq==19.0.1
qds-sdk==1.15.2
redis==3.5.1
redis==3.5.2
regex==2020.5.14
requests-kerberos==0.12.0
requests-mock==1.8.0
Expand All @@ -326,14 +326,14 @@ sentry-sdk==0.14.4
setproctitle==1.1.10
sh==1.13.1
simple-salesforce==1.0.0
six==1.14.0
six==1.15.0
slackclient==2.5.0
smmap==3.0.4
snowballstemmer==2.0.0
snowflake-connector-python==2.2.6
snowflake-sqlalchemy==1.2.3
sortedcontainers==2.1.0
soupsieve==2.0
soupsieve==2.0.1
sphinx-argparse==0.2.5
sphinx-autoapi==1.0.0
sphinx-jinja==1.1.1
Expand Down Expand Up @@ -372,16 +372,16 @@ tzlocal==1.5.1
unicodecsv==0.14.1
uritemplate==3.0.1
urllib3==1.25.9
vertica-python==0.10.3
vertica-python==0.10.4
vine==1.3.0
virtualenv==20.0.20
virtualenv==20.0.21
watchtower==0.7.3
wcwidth==0.1.9
websocket-client==0.57.0
wrapt==1.12.1
wrapt==1.11.2
xmltodict==0.12.0
yamllint==1.23.0
yandexcloud==0.37.0
yandexcloud==0.38.0
yarl==1.4.2
zdesk==2.7.1
zict==2.0.0
Expand Down
Loading