passing the insecure packages list from safetydb results in "Argument list too long"

[fesieg]

I'm using a constrained index on my devpi server with a customized DEVPI_SERVER_ROOT. To get a sensible list of constraints, I converted the insecure.json list from safetydb into a requirements.txt-style file. This results in the following message and the constraints on my index not being updated:

bash: /usr/local/bin/devpi: Argument list too long

Does this mean that I'm passing too many constraints generally, (the file has 4000+ lines) or is the list of version ranges specified in any single of the constraints too long? For reference, this is one of the longest ones:

tensorflow<1.10.0,<1.12.2,<1.15,<1.15.0rc0,<1.15.2,<1.15.3,<1.15.4,<1.15.5,<1.6.0a1,<1.7.0,<1.7.0a1,<1.7.1,<2.11.1,<2.12.1,<2.14.1,<2.4.0,<2.4.4,<2.5.3,<2.6.4,<2.7.2,<2.7.4,<2.8.4,<2.9.3,<=1.7,==2.7.0,>=0,<1.12.2,>=0,<2.0.0,>=0,<2.6.4,>=0,<2.8.4,>=1.15.0rc0,<1.15.4,>=2.0.0a0,<2.0.1,>=2.0.0a0,<2.0.2,>=2.0.0a0,<2.0.3,>=2.0.0a0,<2.0.4,>=2.1.0a0,<2.1.2,>=2.1.0rc0,<2.1.1,>=2.1.0rc0,<2.1.2,>=2.1.0rc0,<2.1.3,>=2.1.0rc0,<2.1.4,>=2.10.0,<2.10.1,>=2.10.0rc0,<2.10.1,>=2.11.0rc0,<2.11.0,>=2.12.0rc0,<2.12.0,>=2.13.0rc0,<2.13.0,>=2.2.0a0,<2.2.1,>=2.2.0rc0,<2.2.1,>=2.2.0rc0,<2.2.2,>=2.2.0rc0,<2.2.3,>=2.3.0a0,<2.3.1,>=2.3.0rc0,<2.3.1,>=2.3.0rc0,<2.3.2,>=2.3.0rc0,<2.3.3,>=2.3.0rc0,<2.3.4,>=2.3.0rc0,<2.3.4rc0,>=2.4.0rc0,<2.4.0,>=2.4.0rc0,<2.4.2,>=2.4.0rc0,<2.4.3,>=2.4.0rc0,<2.4.3rc0,>=2.5.0rc0,<2.5.0,>=2.5.0rc0,<2.5.1,>=2.5.0rc0,<2.5.2,>=2.5.0rc0,<=2.5.0,>=2.6.0,<2.6.3,>=2.6.0a0,<2.6.3,>=2.6.0a1,<2.6.0,>=2.6.0rc0,<2.6.0,>=2.6.0rc0,<2.6.1,>=2.6.0rc0,<2.6.3,>=2.7.0,<2.7.2,>=2.7.0a0,<2.7.1,>=2.7.0rc0,<2.7.0,>=2.7.0rc0,<2.7.1,>=2.7.0rc0,<2.7.2,>=2.8.0,<2.8.1,>=2.8.0a0,<2.8.0,>=2.8.0rc0,<2.8.0,>=2.8.0rc0,<2.8.1,>=2.8.0rc0,<2.8.3,>=2.9.0,<2.9.1,>=2.9.0,<2.9.3,>=2.9.0rc0,<2.9.0,>=2.9.0rc0,<2.9.2,>=2.9.0rc0,<2.9.3

What is the limit here?

[fschulze]

That is from your shell. Are you using devpi index constraints="$(cat constraints.txt)" or how do you pass the data? But I think with the current design of devpi-server and devpi-constrained a list of 4000 constraints might cause performance issues anyway. There are plans to fix that for the next major release, but there is no ETA for that yet.

[fesieg]

Yeah, I'm setting / attempting to set via devpi index constraints="$(cat constraints.txt)"

Good to know you have a fix in mind. I'm unsure if I would be of any help, but depending on the scope I could try to contribute towards that version.

For the moment, how many constraints could I approximately pass without impacting the performance too much (rough estimate)?

EDIT: Just for reference, passing just the tensorflow constraints from above didn't cause any trouble. It seems to be the amount of constraints in general, not the amount of version ranges

[fschulze]

I just remembered: you can get around the shell size limitation by using a json file:

["constraints=..."]

where ... is the content of you constraints. You might have to escape newlines somehow.

Use devpi patchjson [path to index] [yourfile.json] to send the json to devpi-server.

In regard to performance you have to try it out.

[fesieg]

That's a great tip, thanks

[fesieg]

Would the key:value pairs in my JSON just be "packagename":"versionRanges"

Or "packagename":[ "versionRange1", "versionRange2", "versionRangeN" ]

Or what would the shape of the JSON be?

[fschulze]

A single string like on the command line. This is what devpi client sends to the server, one string per command line argument. The server does all the parsing and validation.