Mysterious "Some packages have new versions but cannot be upgraded" message

[fingolfin]

I really like the new status --outdated feature coming in Julia 1.8! It relieves a serious pain point that's been bothering some of us for years. Thanks!

There is a minor weirdness about it, though: in some cases it keeps telling me

Info Some packages have new versions but cannot be upgraded. To see why use status --outdated

But entering that command shows no packages:

$ julia-master
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-DEV.1558 (2022-02-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 9542577c88* (0 days old master)
|__/                   |

(@v1.8) pkg> update
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`
        Info Some packages have new versions but cannot be upgraded. To see why use `status --outdated`

(@v1.8) pkg> status --outdated
Status `~/.julia/environments/v1.8/Project.toml`

(@v1.8) pkg> status
Status `~/.julia/environments/v1.8/Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.1
  [c3a54625] JET v0.5.6
  [f1435218] Oscar v0.8.1-DEV `~/.julia/dev/Oscar`
  [295af30f] Revise v3.3.1
  [aa65fe97] SnoopCompile v2.9.0
  [e2b509da] SnoopCompileCore v2.9.0

(@v1.8) pkg>

The answer, "of course" (well... as I realize while typing this, but after having been puzzled by it for several days) is that there is a package in my manifest that is outdated, but it is only a dependency of one of my packages:

(v1.8) pkg> status --outdated -m
Status `/tmpbig/horn/julia_depot/environments/v1.8/Manifest.toml`
⌅ [4fe8b98c] Mongoc v0.6.2 (<v0.7.1): Polymake

I see two solutions for this:

1. don't show that message about outdated packages in this situation, i.e., if it only affects things which dependencies of the packages I actually care about.

2. detect this situation, and then change the recommendation shown to the user, and suggest to enter status --outdated --manifest

Personally I prefer 1: I don't care about those indirect dependencies, I have to assume that the packages I use "know what they are doing".

[IanButterworth]

Yeah, 1 does sound better

[DilumAluthge]

Personally I disagree. I want to know about outdated indirect dependencies.

[KristofferC]

You do if you pass the -m flag to status though?

[DilumAluthge]

Sure, but it's nice to get the message even if I just do Pkg.status().

I think option 2 is the best choice here.

[DilumAluthge]

I personally think that option 1 is a step backwards, and that option 2 would solve this issue just fine.

[DilumAluthge]

If an indirect dependency is being held back to an older version, this can definitely have an impact on the behavior and/or performance of the direct dependencies.

So I disagree with the idea that most users can ignore it if their indirect dependencies are being held back.

[IanButterworth]

Both options:
#2990
#2991

[IanButterworth]

I'm ok with either. But I get @DilumAluthge point. Perhaps let's go with #2991 and see if it grinds people's gears

[KristofferC]

status has always shown information about direct dependencies, status -m shows it for manifest. This is completely consistent.

[IanButterworth]

Which do you mean by "this"?

To be clear, this isn't about pkg> status it's about the combined mode status print at the end of pkg> update.

If there were hidden outdated deps in both the project and manifest you'd get the warning twice, in each.

[IanButterworth]

i.e. option 2 could result in

(@v1.8) pkg> update
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
  No Changes to `~/.julia/environments/v1.8/Project.toml`
        Info Some packages have new versions but cannot be upgraded. To see why use `status --outdated`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`
        Info Some packages have new versions but cannot be upgraded. To see why use `status --outdated --manifest`

[DilumAluthge]

That seems perfectly fine to me. There are outdated direct dependencies, and I can use status --outdated to list them. There are also outdated indirect dependencies, and I can use status --outdated --manifest to list those.

The important part, in my opinion, is to make sure we show these warnings when people do ] update, because we have seen plenty of users on Discourse that believe that ] update will always update all packages, and are surprised when this does not happen.

The original problem reported in this issue was that ] update told the user to do status --outdated, but then status --outdated did not show any packages. The only thing we need to do, then, is to tell the user to do status --outdated --manifest, which completely solves the problem.