Is it possible to allow remote projectreference from git repo?

[Thaina]

In addition to building project and publish into package, sometimes it was more convenient to let the project reference another project directly and compile it along with the main project

But current method of projectreference require project to present in machine and reference it only locally. Which is inconvenient and require setup on each machine

So I think msbuild should allow referencing git repo in csproj. Internally you could pull git project into temp folder and do the same process as projectreference. But it should be standardize and allow us to build project with CI/CD by projectreference

<Project Sdk="Microsoft.NET.Sdk.BlazorWebAssembly" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <ProjectReference
      Include="src/Blazor.Extensions.Canvas/Blazor.Extensions.Canvas.csproj"
      Repo="https://github.com/BlazorExtensions/Canvas.git"      <!-- the source repo -->
      Commit="3d9b5e6eccb0a66d34172f07ceeb8b7f4d82aaec"      <!-- commit ID for versioning -->
    />
  </ItemGroup>
</Project>

Is it possible?

[Forgind]

It's a nice idea, but I don't think this would work well for any but the most basic projects.

Specifically, if the only part of the git repo you want is src/Blazor.Extensions.Canvas/Blazor.Extensions.Canvas.csproj, then it would be quick to pull down just that and continue your build—but you could also do that beforehand or in a separate build step—there's probably a way to do it with the DownloadFile task.

In the more normal case, you'd want a csproj plus everything it depends on, which is unknown before attempting to download it, so we would have to eagerly download the whole repo, which would take a lot of time. If you do want to go this route, it might be reasonable to clone the repo locally and just update it (git pull) as part of each build. Still slower than if you had it locally, but better.

Does that sound reasonable to you?

[Thaina]

@Forgind I wish that we could have the process you describe be standard automatic process of msbuild along with reference package from nuget

As I said, I wish we could let the msbuild clone project into temp folder of local machine. And by temp folder I means it could cache and share between project, only delete when space is required, and so it kept the shallow copy of specific version for rebuild, the same way we kept download package from nuget in local machine

And the main point is, it then allow me to share my project to other machine, and do CI/CD job on the CI/CD server like github or bitbucket, with just an element of projectreference, in the same manner as nuget denpendency

[Forgind]

So to see if I understand correctly, you're proposing an MSBuild command that downloads just one file from a GitHub repo and puts it in the %temp% folder (in a folder like %TEMP%\Canvas\Blazor.Extensions.Canvas.csproj or %TEMP%\Canvas\src\Blazor.Extensions.Canvas\Blazor.Extensions.Canvas.csproj or on its own?) and save it there between builds, only deleting it if the computer is running out of memory?

One extra problem to consider is when we should assume it's out-of-date. If we aren't maintaining the full repo between builds, we'd essentially have to download it every time if we wanted to ensure that it's still up-to-date (or at least download a timestamp for it) and at that point, we may as well be using the DownloadFile task.

[Thaina]

@Forgind The repo should be considered outdated manually, like a specific version of nuget package, the commit ID is like the version itself. I consider this as constraint for this method of referencing, unless we have an ability to specified semver query with git commit

But I also have something in mind, maybe we could specified Commit attribute as {anybranch}/HEAD or {anybranch}/{anytag} and that would request to check remote server on every build and pull every change, or maybe on dotnet restore that will update repo to that version

downloads just one file

Well, maybe not. I think we must load the shallow copy of the repo, which is all of the files in that commit. But in that repo it could contain any folder structure so we also need to specified where and which csproj we want to include
Then again, if we have this kind of feature in msbuild. We then have ability to create repo and project separate from each other as one repo per project. And reference all dependency by its repo directly

or on its own?

I think we should download into %TEMP%/msbuildreference/github.com/BlazorExtensions/Canvas/{Commit}. It would be like local centralized mirror that every project will be reused the same project of the same commit

[Forgind]

Using commit ID as a version could work as far as telling the build system when it's out-of-date, but it's clunky. Also, unless I'm following the repo closely, I presumably wouldn't know I should change the commit sha, nor would I know what to change it to. At that point, is it easier to have a separate task or just to run git checkout <sha> from the relevant repo whenever you need to?

I'm starting to worry about the security aspects of this. If I specify that I want whatever code happens to be in, say, dotnet/msbuild:master, and the owner of msbuild:master is malicious, that owner could put whatever code they wanted there, and you would automatically download it and run it even if you don't change your code at all. Looking at specific commits sounds safer to me.

Having all the files in a commit doesn't necessarily mean you have all the files the files you need rely on. Like I can update Microsoft.Common.CurrentVersion.targets without touching any of the tasks it relies on, which would mean I would just be relying on the previous versions. Pulling in just the commit would miss that.

I do like the \Canvas{Commit} plan as far as preventing wrong version-type errors, but it would also make invalidating (and deleting) pseudo-repos hard. Git has an incredible branching structure so it only has to remember diffs when switching between commits. If we were to have a separate folder for each commit we asked for, that could be several almost-identical versions of the same repo side-by-side, which would waste a lot of memory. Deleting them is made difficult because they're shared—you'd have to verify that no project references a particular commit before you could delete its folder. Otherwise, you'd risk downloading the same commit of the same repo every time you switched what you were working on.

[Thaina]

Using commit ID as a version could work as far as telling the build system when it's out-of-date, but it's clunky. Also, unless I'm following the repo closely, I presumably wouldn't know I should change the commit sha, nor would I know what to change it to. At that point, is it easier to have a separate task or just to run git checkout from the relevant repo whenever you need to?

I'm starting to worry about the security aspects of this. If I specify that I want whatever code happens to be in, say, dotnet/msbuild:master, and the owner of msbuild:master is malicious, that owner could put whatever code they wanted there, and you would automatically download it and run it even if you don't change your code at all. Looking at specific commits sounds safer to me.

This two problem is the same as current nuget anyway isn't it?
When you just specified strong version number. It then only pull that version even there is an update
When you just specified semver. It then allow project to update to newest version conveniently, but it also allow repo owner to inject malicious code into their package and publish malicious new version that would be included in your project on build
Using commit ID or tag name is exactly the same tradeoff, at your own risk and trust

Having all the files in a commit doesn't necessarily mean you have all the files the files you need rely on. Like I can update Microsoft.Common.CurrentVersion.targets without touching any of the tasks it relies on, which would mean I would just be relying on the previous versions. Pulling in just the commit would miss that.

You then couldn't use that repo as ProjectReference. You might fork it into another repo of your own and cut the reliance on previous version. I think shallow copy have more size advantage for building on CI/CD and should be main priority than very specific project structure like that

Actually I am confused, did you think that shallow clone will only download a file that changed in one commit? No, it download the whole repo, every file and folder as you see in github. It just not download the whole history of that repo like normal clone

it would also make invalidating (and deleting) pseudo-repos hard

We should just use cache clean that clear all cached repo. We might have cache clean unused to list all project in every subfolders of current folder and let that function determine what repo that don't require anymore

Deleting them is made difficult

Nope, I just think caching and sharing is for convenient in daily or hourly development. but it not that important. We can clean it and redownload it in the same manner as using nuget. Clean all the cache and redownload it 2-3 times a week shouldn't hurt. Git repo should be strong enough to accept recloning eventually

[AraHaan]

In addition to building project and publish into package, sometimes it was more convenient to let the project reference another project directly and compile it along with the main project

But current method of projectreference require project to present in machine and reference it only locally. Which is inconvenient and require setup on each machine

So I think msbuild should allow referencing git repo in csproj. Internally you could pull git project into temp folder and do the same process as projectreference. But it should be standardize and allow us to build project with CI/CD by projectreference

<Project Sdk="Microsoft.NET.Sdk.BlazorWebAssembly" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <ProjectReference
      Include="src/Blazor.Extensions.Canvas/Blazor.Extensions.Canvas.csproj"
      Repo="https://github.com/BlazorExtensions/Canvas.git"      <!-- the source repo -->
      Commit="3d9b5e6eccb0a66d34172f07ceeb8b7f4d82aaec"      <!-- commit ID for versioning -->
    />
  </ItemGroup>
</Project>

Is it possible?

I would prefer to specify branch instead of commit hash as commit hashes always change.

[Thaina]

@AraHaan commit hashes is for specific version that you want to target it permanently. It it a history record that will not changed unless the owner decide to force delete it

[AraHaan]

@AraHaan commit hashes is for specific version that you want to target it permanently. It it a history record that will not changed unless the owner decide to force delete it

Not everyone wants to fix it to commit hashes however like me (I might pull in and have an AI update an zlib submodule for this very thing), infact why not JUST use submodules instead and then on the CI before it clones your repository to build it would have to set to clone recursively by default (which would also clone the submodule) and avoid having to ask for a feature like this. Even I locally pull the submodule as well before building too.

Even in git having subprojects is a normal thing so that was why they invented submodules to begin with.

Now it could be possible to make an msbuild task that looks inside of the .gitmodules file to look for submodules then run git submodule update to clone them if they are not already cloned (or update them if the .gitmodules targets a specific branch).

[Thaina]

@AraHaan The branch is on my mind, not that it wouldn't possible, it just that hash would be default and safest way, there would be no chance of breaking change when pull from specific commit

Think about this, you might target master/head and do some developing for hours. But when you decide to build it then restore another version, which the owner was also change many of his API, and your project require change again

That's why I think specific commit version should be default way, like nuget reference

submodules

It was a complicate setup to have both submodule and then reference from that submodule. If every of your project use the same submodule then it also redundant