[Exporters] Download of CMSIS packs

[ohagendorf]

Description

• Type: Bug
• Related issue: PR [Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708
• Priority: Major

The succesfull download seems to depend on the PyCurl version. The behaviour differs at two different PCs and with different PyCurl versions:

PC1 and current PyCurl version 7.43.0: a complete failure, no downloads possible - no way to get the packs
PC1 and older PyCurl version 7.19.5.3: download was possible

PC2 and current PyCurl version 7.43.0: 27 errors
PC2 and older PyCurl version 7.19.5.3: only 11 errors

The idea to use an older version was found here: http://stackoverflow.com/questions/38644845/pycurl-timing-out-with-working-network-connection-on-windows-10

Maybe the problem is not restricted the these two PC.

An error handling would be nice. But beside this It would be necessary to get proper error messages, to see the list of downloaded packs and have a possibility to repeat the download.

One error message is:

(28, 'Operation too slow. 

PC1 is connected over 50/10Mbit/s VDSL and PC2 is within an University network. A speed test says 24/26Mbit/s.
Maybe the timeout behaviour within the pack download could be improved

Another error message:

(28, 'Connection timed out after

seems to be a redirection problem. Also with curl a correct download isn't possible. Using the redirected address everything is OK. Here is an example address:

http://www.keil.com/pack/Keil.TMPM4_DFP.pdsc

Downloading over mbed export ... gives the above error message, downloading with curl gives:

....
<p>The document has moved <a href="http://sadevicepacksprod.blob.core.windows.net/pdsc/Keil.TMPM4_DFP.pdsc">here</a>
...

Maybe a possibility to handle redirections would be good. I've seen the curl option in the code to handle redirection. But it seems to fail sometimes.

---

Bug

Target
all

mbed-cli version:
0.9.9.

meed-os sha:
mbed-os 5.2 rc3
873686c Merge pull request #3033 from ARMmbed/release-candidate
and current master
7bcc63b Merge pull request #2977 from andreaslarssonublox/ublox_drivers
DAPLink version:

Expected behavior
download of all CMSIS packs

Actual behavior
all/several download attempts fails

Steps to reproduce
After the first call of

mbed export ...

or after deleting

C:\Users\xxx\.local\share\arm-pack-manager

and calling the mbed export.

[ohagendorf]

@sarahmarshy, please have a look at the above problem description.

[bridadan]

@theotherjimmy Would it be possible to remove the dependency on pyCurl altogether? This creates a dependency on libcurl in Linux that you have to install manually (messed up CI for a bit before manually added the package on all machines). Is there an all Python solution we could use instead?

[ohagendorf]

Just a question, maybe a solution.

The reason to use the offline tools is to work offline. Currently, the first time I use mbed export (and maybe more) a huge amount of files are downloaded to collect the necessary infos to create the project files.

Is it really necessary to do that in every offline tool installation?

The result of the download are some json files, as far as I found. So maybe these files can be part of the repository. Together with a possibility to update these when necessary.

This would solve another problem. As far as I see, a huge amount of tests are executed before release, among other with the current version of CMSIS packs. But when I take this release and use it in a year or so, than the new CMSIS packs are used not the state during testing. Maybe this can lead to an unusable package.

[theotherjimmy]

@ohagendorf Now that #3067 is merged, is all of this resolved?

[theotherjimmy]

--update-packs is available to force a re-download of the packs if you need to test the bleeding edge. Otherwise, we don't download and use the checked-in version of the index and aliases files.

Ergo:

• No errors on export
• Offline tools don't touch the internet.

[ohagendorf]

@theotherjimmy, I think the changes will improve the behaviour incredible.

I checked --update-packs at the system, that didn't download at all the packs with the current pycurl version. It is still not possible. The old pycurl version runs through. So maybe changing to another download library should be a future task.

[theotherjimmy]

Yup. I'm going to open another issue for the change from pycurl to something better.