Update stats are weird

[michaelblyons]

@braver notes

I must say that I find the upgrade stats a bit hard to believe. A package that has been upgraded halve a year ago that sees on average 70 upgrades per week? As I understand it that means every week 70 users start ST for the first time since May, or have auto_upgrade off and manually request upgrades (who does that?)? Even at > 100k total installs that number feels high to me. 🤷🏻‍♂️ Not saying that it's wrong or broken, but it's not what I'd expect.

And I think I've heard similar from Kaste. I know I've had my own questions about it, too.

What is going on? Helpful data and rampant speculation are both welcome.

[michaelblyons]

I do know someone who used one of my packages as their primary language syntax. When shipped packages released a new version, my package broke (and I didn't notice for months 😢 ). This person pinned their main ST version until I fixed my package.

That kind of behavior could time-shift some of the upgrade counts, but only for packages whose ST version compatibility changed, too. It seems unlikely to explain the large numbers.

[michaelblyons]

One wild theory is that people have development environments in VM snapshots. You restore a snapshot, load it up to do some work, and ST immediately notices that packages are out-of-date and grabs all the new ones.

I've never done this myself, but I've heard of it. If it's popular with even a relatively small fraction of the install base, their packages will show a lot of upgrades.

[braver]

What events are triggering these pings exactly? I even doubt the uninstalls now. Perhaps some internal behavior of the PC client is misunderstood and the pings don’t exactly mean what we think they mean?

[kaste]

We see automated installs and upgrades. Also compare the stats on pc.io and the new one. Same numbers for new packages (e.g. LSP), for old packages (e.g. A File Icon) pc.io had twice (!) the installs last time I checked. (= client did not update Package Control)

The PC client code is correct, I made the patch/PR. I would think I would have noticed...

[braver]

This is one with a recent update is actually not weird. It's what you expect and hope to see and learn from this data. 13k total installs, but looking at the graph like 1k actual active users, and perhaps a lot of distribution across users that keep ST as a secondary editor if at all. I see a lot of people switch to perhaps an IDE or something with strong AI features for their day-to-day, and keep ST around for log files, notes, bits-n-bobs. Maybe they open it once a month (and less during summer, our August-heavy slice of data might also just be weird), and those upgrades trickle in over time. Solarized has 10 times as many installs and is probably mostly popular among developers old enough to even know what that is... maybe there is a base of users for whom ST is a "once-every-few-months for niche use cases" editor nowadays, that is around ~50 a week... IIRC ST is still a top-10 IDE choice among devs, and even a single digit percentage could mean a million devs. And non-professional devs also use ST. And people who don't self-identify as devs (and thus won't fill in SO surveys) use ST, like data researchers (who love python apparently), and people write books and recipes in ST as well. Maybe that's not their day job and that's why upgrades of Solarized and some other packages come in late for them.

.... Just thoughts :) I guess this goes to show that data is not information is not knowledge is not understanding 🙂

[braver]

I'm no longer convinced anything weird is going on, perhaps a lot of users do manually update, or just use ST sporadically. The volume of users is there to allow that. And I trust the people who wrote the code that manages this stuff.

But... This just still looks really weird to me.

... And then I look in "other releases" and that explains a lot probably:

So perhaps not all releases are marked on the chart? Maybe that's another bit of information that explains the update stats?

[michaelblyons]

Only the most recent release (for the most recent ST version) is on the chart. It moves when you tag a new one.

[kaste]

That is WIP.

• We need a better design for the releases/versions section.
• We ship with real .sublime-package download links
• We show not the most recent but e.g. the releases of the last 52 weeks
• ...

[michaelblyons]

I thought the thing that was weird was the "long tail" of upgrades after a new release. See PowerShell getting ~60 hits a week, ten months after its last release.

[braver]

Well it's a consistently long tail for all packages with substantial install base. So that must just be "normal", like the base line. You can disable automatic updates on packages, and with a few million ST users, or even just 1 million or whatever the number is, maybe 50 per week of manual updates or, or twice-per-year users, is normal?

I'm just saying I'm over the idea that this is weird. It's probably just the "noise floor". Which might mean we should visualize differently, but we've only been at this a few months, we need to learn some more and then iterate.

[michaelblyons]

I pushed a Powershell release, and (despite the week not being over yet), the new uptake just in this week is >100x the top of the noise floor... and 200x the after-August¹ numbers. The new data make the old jitter all look like zeroes.

Footnotes

1. Maybe university students loading up ST for the first time since the summer are having a mild effect on the update stats? ↩

[braver]

Sass seeing hundreds of upgrades per week a year after its last update is a bit weird. But then packages you expect to be kind of dead, are indeed also dead in terms of upgrades. So apparently there really is a very long tail of upgrades. Incidental ST users, users that manually upgrade... there must be thousands of them out there.

I'm inclined to say case closed.

[michaelblyons]

I don't mind the discussion closure, but I'm not sure we learned very much about the noise floor of popular packages' upgrade tail.

[braver]

but I'm not sure we learned very much about the noise floor of popular packages' upgrade tail.

For sure. I still have questions, but I no longer doubt the system. Perhaps SublimeHQ has more telemetry to figure out more precisely what ST usage looks like?