Implement opt-in telemetry

[strugee]

It would be extremely useful to have some mechanism wherein a server admin could opt-in to sending (limited) data back to the pump.io project, in order to answer various questions.

Some examples of things that would be extremely useful to know:

• Content Security Policy violations
• How many users are registered and how many are "active"
• Whether the server supports TLS and/or how many incoming posts reference non-TLS resources
• What percentage of incoming posts are AS2 vs AS1, etc. (when we start transitioning to newer protocols)
• Host system information (OS version, Node.js version, npm version)
• Pump.io version and installation source (source, npm registry, npm git)
• Databank driver
• What percentage of traffic comes from which clients (including the web interface) - this one's more for fun

I should also point out that the key here is that the server admin would opt in to this, not opt out. The data would also be submitted anonymously.

[strugee]

Add to the list:

• 500 Internal Server Error responses and associated stack traces

[strugee]

I've been thinking about this a lot and I think it would be valuable to include in this list a simple ping that basically says "hey, I'm a sever and I exist." I'm imagining that server admins would be able to opt in to reporting only a subset of the list presented here, so admins uncomfortable with reporting anything specific (but OK with reporting the existence of their node) could enable only the ping reporter, giving us a sense of how many servers chose not to opt in to data reporting.

However, in order for the ping to have real meaning, we'd have to make it opt-out, not opt-in. My gut feeling is that as long as we document the hell out of this and make it clear and obvious what's happening, turning on a ping by default would be OK. However, obviously this has serious privacy/security implications, so I'd welcome feedback from the community. Thoughts?

[larjona]

I thought this was already the case when you "register your node with ofirehose" (a setting in the options file).
Sorry I cannot give more details now, I'm far from a decent screen.

El 18 de septiembre de 2016 09:22:25 CEST, Alex Jordan notifications@github.com escribió:

I've been thinking about this a lot and I think it would be valuable to
include in this list a simple ping that basically says "hey, I'm a
sever and I exist." I'm imagining that server admins would be able to
opt in to reporting only a subset of the list presented here, so admins
uncomfortable with reporting anything specific (but OK with reporting
the existence of their node) could enable only the ping reporter,
giving us a sense of how many servers chose not to opt in to data
reporting.

However, in order for the ping to have real meaning, we'd have to make
it opt-out, not opt-in. My gut feeling is that as long as we document
the hell out of this and make it clear and obvious what's happening,
turning on a ping by default would be OK. However, obviously this has
serious privacy/security implications, so I'd welcome feedback from the
community. Thoughts?

Laura Arjona Reina
https://wiki.debian.org/LauraArjona

[strugee]

@larjona that's actually a great point, hadn't thought of that. Certainly that would be one way to get an idea of how many nodes exist (vs how many are reporting). Thanks!

[strugee]

(Of course ofirehose.com is down but we should fix that anyway...)

[strugee]

More for the list:

• Probing whether specific usernames exist - this is useful when adding new endpoints to a domain, as we probably shouldn't do that if they would override a particular username's profile page
• Various performance metrics (timings, mostly)

[strugee]

Adding this to our next meeting agenda. We decided last year(!) that this data would go through OFirehose, but I don't really think that's a good idea anymore, mostly because OFirehose doesn't retain anything. So we would either have to a) have OFirehose retain stuff, which would significantly increase the operational cost, or we'd just have to build an external service to retain stuff anyway. Even if we did change OFirehose we'd still have to build out tooling to do reports and stuff. It just really seems like these should be separate.