Feature Planning: Advanced crate handling

[Feliix42]

Looking through the issue list, it appears that we have a number of different feature requests in the backlog that affect the URL handling and/or the way the application interacts with repositories/selects crates. These issues are:

• Support rust code in folder within repo root #95 -> allow handling of crates living in sub-directories
• Support a badge per sub-crate in a single repo #19 -> badges for said sub-directory crates
• Support for private git servers / self-hosting. #84 -> analyze repositories on self-hosted git-servers
• branch selector #12 -> allow branch selection
• Check Cargo.lock for Bin Crates #26 -> check the lockfile instead of the Cargo.toml

I think that these issues should be solved separately. But we should probably think about how general solutions could look like (and what of these should be implemented in our public instance or if some of these things require a private instance).
All these issues will probably alter the routing scheme so it would make sense to think about things like these in a central issue so we don't end up with several different (or conflicting) solutions.

[Enet4]

I believe #26 should also be considered alongside those issues. Like #95 and #12, it's something that users may wish to opt in for a particular project.

• Support for private git servers / self-hosting. #84 -> analyze public(?) repositories on self-hosted git-servers

My interpretation is that this one should also enable the analysis of any repository within premises and with the right credentials, and not just public repos.

[Feliix42]

You're right, I didn't notice this when browsing the issue list. Dutifully added.

Ah, hence the (?). This sounds like a good idea.

[Feliix42]

Ok so after giving this a bit of thought, I would propose this first idea, let me know what you think. We could add GET parameters for handling most of the things mentioned above:

• Self-hosted repositories could be specified using this URL scheme: repo/external?url=<URL>. I'm currently not 100% certain if slashes etc would have to be escaped in the URL, but if so we should probably offer a input field on the deps.rs landing page for users to enter the web address pointing to the repository.
• Sub-directory selection should work for all repo paths using a root GET parameter. (this would apply to both badges and status pages.
• Branch-selection could also be a handled using a parameter like branch. Selectors would be needed on both the Status page itself and the front page where aforementioned URL input can take place. On the status page this could be a dropdown if we manage to extract that information while scanning the repo, but on the front page it should be a free text field.
• Scanning the lockfile is enabled simply by adding lock to the parameter list.

All these changes would of course require documentation, not just for us internally, but also on the deployed page. I could imagine either a Documentation section on the page, or even better, a nice dialogue for generating the link on the landing page, combining all the features mentioned above in a sort of lookup mask you can fill to scan a repository. Apart from the Lockfile, nothing of the things above targets crates.io, but it should still be available as dialogue (maybe selectable via a panel or something).

This is just my first idea on how to solve this. Let me know what you think. The only thing that would be missing here is handling credentials to private repositories of any sort, not really sure how to handle that since every platform has a homegrown solution to authenticating these requests.

[Enet4]

Self-hosted repositories could be specified using this URL scheme: repo/external?url=<URL>. I'm currently not 100% certain if slashes etc would have to be escaped in the URL, but if so we should probably offer a input field on the deps.rs landing page for users to enter the web address pointing to the repository.

Self-hosted environments will (I think) usually be limited to a known set of repositories¹. Through a local configuration file, these repositories could be given a short alias, and so not require specifying a full URL. This would also enable specifying credentials or other means of granting access to the repository.

¹ Granted, there's usually a repository central system, such as Phabricator's Diffusion. But right now I don't picture an integration at this level in deps.rs.

[Feliix42]

Ah, that's an interesting take, bearing another question: Should the deployed version on deps.rs support self-hosted repositories or would that require a deployment on your own server? Because from your suggestion it sounds to me as if you would have to prepare your own deployment for that.

[robjtede]

It seems to me that a local configuration based deployment could work for deps.rs as well as self-hosted deployments. Aliases we already have: github, gitlab, bitbucket. They could be described with the same configuration as other internal aliases; just without authentication needed.

I think it would be useful to support the API key auth systems used by self hosted versions of github, gitlab, bitbucket as well as HTTP basic auth to support a vast array of setups. I do not thing deps.rs should try itself to support reaching into internal systems.