Enhancement: installed custom modules statistics, include installed modules in the latest-version check

[gmarcon]

Currently, the latest-version.txt request for version check transmits basic data for statistics like PHP version, database type, and webtrees version. I propose to include a list of installed modules.

This would help to track the popularity of custom modules-ons and ultimately to improve the understanding of the ecosystem.

I will propose a simple pull request for the change in webtrees, but there is also change needed in the script generating the page https://dev.webtrees.net/statistics.html that I do not think is released in github and I think is managed directly by @fisharebest .

Thank you for considering this enhancement!

[BertKoor]

I'm in favour of no tracking, no metrics or data gathering. If usually leads to no good and pissing contests. None of the 8000+ admins has agreed to this.

If you want tracking of your own module, then go ahead. You seem to know how things work.

[gmarcon]

Hi @BertKoor, thanks for sharing your perspective. I completely understand and respect the caution around data gathering and privacy.

My goal with this proposal is not to track individual installations/site administrator choices, but rather to look at ecosystem-wide health and quality. To clarify the intent, the data would be aggregated and treated anonymously exactly as it is now for the webtrees version, PHP version, database type, location. Statistical information on custom module usage could bring a few key benefits to the community:

• Breaking changes planning: It gives visibility into which third-party custom modules are particularly relevant, allowing for better planning with their developers before making breaking structural changes in webtrees.
• Quality and longevity: It helps custom module developers understand which modules are most actively used, allowing them to prioritize maintenance, optimization, and security updates for the community.
• Discovery: It helps site administrators discover which modules are widely trusted and useful, enhancing the overall platform experience.

Regarding implementing this independently within a specific custom module: that would actually require a dedicated external server to collect the data. Personally, I would have much higher trust in a centralized, transparent function managed directly by @fisharebest than in individual, fragmented "phone home" mechanisms built into separate third-party modules. Anyway, I could explore with @Jefferson49 if we could implement it in "Custom Module Manager", although we would still need a server to collect the data and we would be collecting data only for users who have installed this module.

That said, I absolutely agree that transparency is vital. If you think webtrees site administrators should explicitly consent to this kind of telemetry, an opt-in/opt-out configuration for the latest version check should probably be included, but this goes beyond my proposal for statistics on installed custom modules.

[fisharebest]

If you know me at all, you will know that I am also very much against tracking.
Wherever there is a design decision in webtrees, I prefer the most private option.

I try to follow the best-practices and follow data protection rules.

• I collect only the minimum data that is needed to provide the "check for update" service.
• I delete it as soon as it is no longer needed (30 days)
• I use anonymous/random site-ids instead of IP addresses to aggregate the data.

Previously, I also collected OS type (linux/windows) and mysql-version - these were needed for older versions of webtrees.
As soon as they were no longer needed, I stopped collecting them.

For openness, I also share the aggregated data.

Knowing more about how sites are used would be enormously useful. For example, if 90% of users change a config setting, I could make it the default. If 90% of users only have one tree, I could optimise the GUI for one-tree sites. etc.

But I have resisted collecting this data. It's not my data, and collecting it unnecessarily would affect my reputation, and that of webtrees.

So anything that is not essential would need to be opt-in.

Also, this sort of telemetry may need a different cadence to the check-for-updates. Perhaps weekly or monthly?
It will be sending more data, which won't change frequently. It will need a dedicated server/service to process the requests.

the script generating the page https://dev.webtrees.net/statistics.html that I do not think is released in github

FYI, the code for dev.webtrees.net is in a private repository. There is nothing secret here, but the script evolved from a proof-of-concept, and isn't the prettiest code I have ever written ;-)

[LarsRabe]

I also think this is a very bad Idea. Accordingly the PR for this Issue should not be accepted.

[gmarcon]

Thank you for your feedback @fisharebest , it is very much appreciated.

Your answer perfectly reflects why the webtrees community would trust you with this data, and why an official, centralized approach feels safer than a third-party alternative.

Given your position, would you be open to (without necessarily endorsing) this feature being implemented entirely within a third-party extension like the "Custom Module Manager", provided it includes a clear disclosure and an opt-out mechanism? If you don't object to that direction, I will reach out to @Jefferson49 to see what he thinks about it.

Feel free to refuse the pull request 5378 and to close this issue whenever you see fit.

Thanks again for considering this and the discussion!

[gmarcon]

Update: I have implemented this feature as an enhancement directly inside the Custom Module Manager using a dedicated serverless Supabase backend. This keeps the core webtrees framework completely independent while still providing the community statistics, which should make the architectural idea clearer for when you share your feedback!

[fisharebest]

Apart from module installations, what other telemetry might be useful?

[fisharebest]

What, if any, protections have you made against malicious submissions? e.g. a malicious actor could submit a large number of repeated and/or bogus submissions, which could DDoS the receiving server?

[gmarcon]

What, if any, protections have you made against malicious submissions? e.g. a malicious actor could submit a large number of repeated and/or bogus submissions, which could DDoS the receiving server?

In my proposed architecture:

• DDoS protection is handled by Supabase, which sits behind Cloudflare. It includes built-in edge protection, payload size limits, and automatic API rate-limiting by default to prevent basic DDoS attacks.
• Spoofing requires sustained effort for zero real reward. An attacker would have to go to the lengths of generating thousands of fake site UUIDs and submit them continuously over time (since the SQL query only counts check-ins from the last 60 days). All this effort would just be for the benefit of artificially inflating a module's popularity, which would likely be spotted quite easily. The worst-case scenario is that the statistics are (temporarily) wrong, but no functionality is impacted. This seems like a very reasonable risk to me, but I am open to any suggestion! Do you already have a way to manage this by chance in the check for the latest update?
• The client fails gracefully. Both the telemetry and statistics calls have a strict 3-second timeout. If the server is unresponsive or overloaded, the module catches the error, logs it (flash message), and the UI degrades gracefully by simply not showing the installation counts.

[fisharebest]

I'm not too bothered by incorrect statistics. The daily numbers are unimportant. The trends are more useful.

In my set up, responses are generated by nginx configuration, so are very lightweight. Processing the logs is a daily task.

If I needed to store data from each response, I would need a setup similar to the one you have described.

[gmarcon]

Apart from module installations, what other telemetry might be useful?

You are probably the best person to answer this question! ;)

As you suggested previously, this kind of data:

• would reveal real-world configuration patterns to help optimize defaults and GUI layouts;
• could be collected on a much longer timeframe (monthly/yearly/upon upgrade/manually triggered by the site admin).

Technically, we could use a flexible JSON object and store it as-is (for instance, in a standard Postgres JSONB column). This way, the database schema never needs to change when you decide to add or drop a metric, and you just collect what is strictly actionable at that moment in time.

Some examples of metrics that could make sense:

• number of individuals
• number of media files
• technical information: OS type, database version, server RAM, CPUs, etc.
• number of active users
• default theme / active themes
• number of trees
• tree preferences (for instance, usage of text vs. markdown, surname tradition, watermarks, etc.)

For most of these, we could consider clustering the data before sending it instead of the raw number (for instance, for individuals: 0-500, 501-5000, etc.). However, since this would require knowing the relevant clusters beforehand, it is probably better to just collect the raw data and cluster it during the analysis phase.

[fisharebest]

For most of these, we could consider clustering the data before sending it instead of the raw number

I like this idea.

In my head, I am visualising the system showing the user what data will be shared before they agree to share it.

Clustering before sending would help reassure the user that we aren't collecting any unnecessary data.