NuGet should refresh its HTTP cache if it can't find the package requested

[ericstj]

We see restore failures when we update a project to consume a package that was just published. We can use --no-cache to workaround it, but that shouldn't be necessary.

It seems like NuGet should realize it is running from a cache, and attempt to refresh that cache, before failing the restore because it cannot find the package.

This would still let the common case be fast while not failing in the case that the cache was stale.

[emgarten]

Are you using floating versions, or are you specifying the version exactly in your project.json?

[ericstj]

Version is specified exactly, not floating.

Workflow we see is:

1. Produce and publish package Foo version 1.0.2 in repo 1. Wait for that to appear on myget.
2. Update references to Foo in repo 2 to version 1.0.2. Submit a build.
3. Some machines will fail to restore, some succeed. Failing machines claim that 1.0.2 doesn't exist.

[rrelyea]

Today the best solutions for this are to:

1. clear the http cache before build; or
2. call restore with --no-cache

We've fixed this kind of thing in the UI scenario for install.
Not committing to doing this now...but keeping it around to consider.
Thanks for raising the issue.

[NickCraver]

We're still hitting this often. When we have geo-distributed NuGet feeds and one doesn't have the package, we need to go clear %localappdata%\NuGet\v3-cache manually on the build server. This is the same for any user having the same issue. Indefinitely caching a 404 is just against all reason when dealing with HTTP. Why are we caching 404s? I'm not saying there's no merit to it, but caching it for more than a minute or so only serves to cause this maddening behavior.

If you click restore packages over and over in VS by right clicking the solution, a user is just shown that the page isn't in the feed. But it is. It's lying. There's an invalid cache in play. I've had to explain this to several of our developers lately and none of them find this intuitive.

Can we please not cache 404s indefinitely? Fixing a build server isn't something every dev should have to learn how to do, nor do they necessarily have access to do so.

[martincostello]

I've had this issue myself with NuGet packages just published to a ProGet feed from a TeamCity job, which aren't quite indexed fast enough when I try to reference them elsewhere.

As annoying as it is, another workaround I've found is to add/remove as appropriate a trailing slash onto the feed URL (in my case, as the path isn't one ending in .json), which cache-busts the 404.

[NickCraver]

Still hitting this several times a week. Then I have to go clear the cache folder on dozens of build servers. Please change this behavior. It's not right. It doesn't follow HTTP standards. It's an invalid cache, straight up.

[llehn]

I'm too hitting this bug frequently. Please fix this, caching 404s indeed doesn't seem like a good idea

[NickCraver]

I just spent an hour clearing out the HTTP "cache" folder on a whole series of build agents, instructing developers how to do it locally, and getting builds back in order due to a cached 404 yet again.

PLEASE stop caching a 404 forever. This behavior is nuts. I cannot find a single other product or platform that does this...and I looked, really hard.

[genscape-agodfrey]

After a few days of troubleshooting I found this github issue. This started with our teamcity build server and all builds failing after a lock. Here are what I consider to be related issues:
#7060
#7020

[NickCraver]

Is this behavior going to change in 5.0? We're still hitting it several times a month. While cleaning out build servers due to race conditions is super fun, but I'd rather spend my time on things that matter.

Please stop caching 404s. No sane platform does this. And they certainly don't do it indefinitely.

[kilasuit]

The last time an iteraction with this from the team was #3116 (comment) - which happens to have been November 2017

This is pretty bad in managing this as an incident report and whilst this does not currently affect me, I can sympathise with the others on this thread that it does affect.

This effect is undesirable at best and therefore should be prioritised to be changed in a future release.

The fact that this has been hitting @NickCraver and team enough that he has come and replied 3 times since his original comment April 2018 should show that this requires better prioritisation than this is currently getting

[Kryptos-FR]

Why hasn't this been fixed already?
I don't see any technical difficulties and it is driving people crazy.
Our project (Xenko) relies on nuget and I'm pretty sure we hit that exact issue all the time.

[rrelyea]

Appreciate the feedback. Will try to get this fixed early in 5.x

[vatsan-madhavan]

@rrelyea, Setting RestoreNoCache=true prior to RestoreTask doesn't seem to work reliably for me. Is this not quite the same as --no-cache ?

[Kryptos-FR]

@rrelyea fixing it for a new version is not acceptable, it should be fixed in minor version on all version currently deployed. Caching HTTP (esp. 404) requests is just a bad practice and a design flaw. Fixing it ASAP should be considered CRITICAL.

[anangaur]

@nkolev92 whats the best solution here? IMO, not caching 404s seems like the right option. Shouldn’t have adverse side effects. If a package is not found, build breaks and hence not something folks run repeatedly unless they fix the availability of package or change the dependency graph to an available version.

Or we could expire 404 cache in a min or some small time period.

I would still vote for first option.

[nkolev92]

I guess the misconception is that we cache 404s.
We do not do that.
We cache the versions list, which is not a 404 if the package exists on said server.

We could consider retrying if we get a cache response that doesn't contain the requested version, as suggested by @ericstj , but that could have an effect on performance, which we should consider.
Of course that'd be driven by the distribution of the packages/versions across the different sources that a project/solution has, so it might be barely noticeable. Unfortunately there's no low hanging fruit here :)

[NinoFloris]

You could do a version list retry if the requested version is higher than any in the cached list, right?

[nkolev92]

@NinoFloris

Yeah the special refresh logic could be, the requested version is higher than the one in the cached list, or it could be that the version is missing altogether like the OP suggested.

[pfeigl]

@karann-msft - Would you expect the environment variable workaround to also work within visual studio?

I could make it work when running msbuild from the developer command line, but within visual studio doing a rebuild-all did not work as expected to fetch transitive packages right after they are available on the internal nuget server.

I also tried settings the properties via Directory.Build.props file which also didn't work in VS :-(

<Project>
    <PropertyGroup>
        <RestoreForce>true</RestoreForce>
        <RestoreNoCache>true</RestoreNoCache>
    </PropertyGroup>
</Project>

[mmitche]

@nkolev92 What's the status of a 'real' fix here? It's pretty clear that there is a workaround, but I'd like a fix that doesn't involve knowledge of the underpinnings of feed infrastructure (e.g. what's the package publish time of my input feeds?).

.NET has been hitting this pretty regularly lately (we removed our http cache clear a bit back).

I understand that there are some complexities here. In .NET's case, we look for exact version matches in all cases, so the error path (package version not found in http cache, fallback should be to re-request) is correct and should not affect performance. On the other hand, when * versioning or version ranges are in use, then things get complicated. I think in those cases, you either need to get more info from the package feeds (e.g. a simple bit that indicates whether a new version of X package has been published), or you cannot use the http cache altogether from a correctness standpoint. I lean more towards the second option in those cases.

[vatsan-madhavan]

@mmitche Are the problems in .NET build still related to MSBuild SDK's? If so, then does MSBuild's SDK resolver honor RestoreNoCache yet (I don't believe it did last I checked over a year ago and that made this workaround unreliable).

[mmitche]

Not sure, @riarenas is investigating. We had a cache clearing step in the builds before (I think it was an explicit initial clear), but it was removed at some point.

[aortiz-msft]

@mmitche - Can you use the RestoreNoCache env variable mentioned in #3116 (comment)?

[mmitche]

@mmitche - Can you use the RestoreNoCache env variable mentioned in #3116 (comment)?

Yes, we believe we can and we're adding it back into our builds. But I'm more interested in a fix for the underlying issue.

[vatsan-madhavan]

We had a cache clearing step in the builds before (I think it was an explicit initial clear), but it was removed at some point.

The old nuget-cache clearing in Arcade's build targets/scripts weren't helpful for MSBuild SDK's even in the past, unfortunately.

The MSBuild SDK resolver did its own NuGet calls I believe (probably still does), and didn't respect RestoreNoCache env etc (which means that these workarounds - however helpful in general - weren't helpful enough for .NET builds).

[mmitche]

@vatsan-madhavan I don't think that we're seeing those problems in these cases. All of these are post msbuild SDK-install restore issues.

[tig]

I think I'm encountering this. It is not clear from the massive number of comments above what a good work around is.

[tig]

I think I'm encountering this. It is not clear from the massive number of comments above what a good work around is.

Yes, I was.

dotnet nuget locals all --clear

worked.

[DaveSenn]

will there ever be a fix for this issue?
we still have the exact same problem as reported in 2016.... today in the time of. NET 6.

[mungojam]

Similarly if the feed sends back no-cache type headers for a particular item, then these should be respected as well. My workaround has been to change the HTTP cache folder to point at the workspace for each build.

[BasTossings]

When will this be fixed? This bug is really disruptive to my nuget workflow; after I push a new package version to GitHub packages, visual studio will claim the new version does not exist for a good 15 minutes (presumably until the http cache for the nuget server /query request expires).

Manually clearing all nuget caches works but this feels like killing a mosquito with a howizer. Plus, it requires me to restart Visual Studio every single time I update a package.

All in all a huge pain in the behind. This feels like it should be very easy to fix! Come on Microsoft!

[stefankaufmann]

We are facing the same issue. Really disturbing for CI Pipelines which use a nuget repository for exchanging artifacts

[jt-pt-dev]

Still bumping in to this issue... Cost me a few hours to get to the bottom of.

[JoseMarcenaro]

Still facing the same issue.

[isikdos]

Still facing the same issue, although the --no-cache appears to work it is confusing why this isn't default behavior. Can't find the version and refuses to take a look.