Sync extensions for JEDI #1418
Sync extensions for JEDI #1418
Conversation
…extensions into sync_extensions
scripts/ci/sync_extensions.py
Outdated
|
|
||
|
|
||
| def _get_updated_extension_names(): | ||
| cmd = 'git --no-pager diff --diff-filter=ACMRT HEAD~1 -- src/index.json' |
There was a problem hiding this comment.
I'm still little concern on replying on change on index.json. what if the pipeline run failed? new run need to catch up extension blob changes in last several index.json change.
There was a problem hiding this comment.
Added environment variable to control commit numbers to diff. So in case of failure, we can manually trigger the pipeline and pick up all the changes for the last n commits.
scripts/ci/sync_extensions.py
Outdated
| def _get_updated_extension_names(): | ||
| cmd = 'git --no-pager diff --diff-filter=ACMRT HEAD~1 -- src/index.json' | ||
| updated_content = check_output(cmd.split()).decode('utf-8') | ||
| updated_urls = [line.replace('+', '') for line in updated_content.splitlines() if line.startswith('+') and not line.startswith('+++') and 'downloadUrl' in line] |
There was a problem hiding this comment.
why not load the json content and get from key directly?
There was a problem hiding this comment.
Here we only extract updated downloadUrl from git diff for incremental sync.
scripts/ci/sync_extensions.py
Outdated
| the_ex = None | ||
| while count > 0: | ||
| try: | ||
| response = requests.get(url, allow_redirects=True) |
There was a problem hiding this comment.
since whl package size may be very big, is this lib supporting resume from break-point? if there any more reliable/dedicated package downloading library?
There was a problem hiding this comment.
Changed to stream downloading so we can handle big files well.
|
Can't we move this inside ADO? |
code too long, ADO inline script will truncate it. |
I was wondering whether there is a better place to place this script instead of exposing it to the public because user don't need it. |
| def main(): | ||
| import shutil | ||
| import tempfile | ||
| from azure.storage.blob import BlockBlobService |
There was a problem hiding this comment.
install azure.storage.blob in Bash task?
There was a problem hiding this comment.
yes, as a prerequisite.
scripts/ci/sync_extensions.py
Outdated
| if not updated_urls: | ||
| return updated_exts | ||
| for line in updated_urls: | ||
| search_result = re.search(r'"downloadUrl":\s+"(.*?)"', line) |
There was a problem hiding this comment.
What if user manually update src/index.json and enter 'downloadUrl'?
There was a problem hiding this comment.
This also detects such case and is the same process as update by bot.
There was a problem hiding this comment.
I guess the re expression r'"downloadUrl":\s+"(.*?)" won't be ablt to search 'download'
…extensions into sync_extensions
This script will be used in an internal pipeline to sync up new extensions, triggered by new commits in master branch.
It also supports cleaning up and syncing all extensions from the master branch index.json file.
This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally?For new extensions:
About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your PR is merged into master branch, a new PR will be created to update
src/index.jsonautomatically.The precondition is to put your code inside this repo and upgrade the version in the PR but do not modify
src/index.json.