Allow pods to register to tags, to have federated tag following

[Flaburgan]

I'm opening an issue here to have feedback on a suggestion I made a while ago. I'd love to see tag following working across pods in the federation and I'm not satisfied with the relay solution which have several flaws in its design imo.

Btw @annando how is friendica dealing with that problem?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[cmrd-senya]

So basically in terms of the federation protocol we need a message that one pod sends to another telling "subscribe me to #tag". Maybe we can also save some bandwidth if we allow to put more than one tag in one message.

But there is another problem in terms of design. Even for public posts message sender is always a person, not a pod. So maybe we should also allow pod being a sender on its own. We also need this for subscription refactoring. I'm sure @SuperTux88 has thought about the issue already and has opinion about it.

[annando]

@cmrd-senya technically it would be no problem to send these items. Technically it is the same like with the relay functionality.

Friendica does it differently. We are using an advanced relay functionality without the need of a dedicated relay server. Since Friendica already scans other servers on a regular base, we now are scanning these servers for their relay subscriptions. Then we are sending the content to them.

[SuperTux88]

Even for public posts message sender is always a person, not a pod. So maybe we should also allow pod being a sender on its own. We also need this for subscription refactoring. I'm sure @SuperTux88 has thought about the issue already and has opinion about it.

For subscription-refactoring (by the way: that part is already done and in production), I just use a "random" user when a pod sends a subscribe-message to another pod, because it's only needed to verify the sender pod, the exact sender person doesn't matter (and is not stored anymore in the final version, but that's still todo). So it only needs to be a valid person from that pod.

The "random" is not so random: If the pod has a podmin configured (in diaspora.yml), the podmin is used, if not, the first admin-user is used (with admin role), if there aren't any podmins/admins, it just uses the first user that isn't closed yet. But for the receiving pod it just needs to be a valid signature from that pod, the exact person doesn't matter.

[cmrd-senya]

Great, then we can reuse this random person method for tag subscriptions.

Then we need to come up with the message format and name.

Can we do something like

class TagSubscription < Entity
  property :tags, [:string]
end

?

[SuperTux88]

We currently only support array of entities, not arrays of strings. But for the profile tag_string we just use space separated tags, maybe we can use the same here (which would have the advantage that we can use the same validator).

And there needs to be a method to unsubscribe tags, we could use a second entity for that, or use a boolean properties that says if the current message is subscribe or unsubscribe, or just have two properties here which are both optional. So you can have something like this:

<tag_subscription>
  <subscribe>#foo #bar</subscribe>
  <unsubscribe>#baz</unsubscribe>
</tag_subscription>

(Which is the format I would prefer at the moment, because it can be done without the need of inventing new possibilities which would add more complexity to the already pretty complex generating/parsing logic)

[annando]

@Flaburgan I do see two problems of your concept: You had to know all servers and had to subscribe there. And when the tag subscription has changed, you have to subscribe/unsubscribe on each server.

And this here isn't true (anymore): "As you can only search for tags and not for content, this is not a real limitation."

Diaspora (and Friendica) do have a possibility to show the whole public feed.

[Flaburgan]

And this here isn't true (anymore): "As you can only search for tags and not for content, this is not a real limitation."

At least for diaspora*, this is still true: you can't search for every posts containing a specific word / sentence, only for tags.

[Flaburgan]

About the problems you pointed, for the first one well yeah, discovery still is a problem, but it's like users search. I kept the same philosophy than the rest of the protocol. Maybe we can improve pods and users discovery one day but that's another topic imo. And about subscribe / unsubscribe, that's also how user relationship is working, I don't see what's the problem. Especially, you would unsubscribe to a followed tag only if every users of your pod following it stop following, so it's not something that will happen everyday.

[annando]

You do have /public on Diaspora servers where you can see the whole public feed.

Concerning server and user discovery: Friendica already has got data exchange for this. After some time most servers will know most other servers and most of the active users. But you are right, this is another topic.

What do you think about the decentralized relay functionality that Friendica uses? We don't need a central server and it is already multi protocol able.

[DeusFigendi]

I like the general approach @Flaburgan it keeps diaspora* more decentral (while the relays create some kind of central hub).
I have on concern and one question/idea:

concern: If I get this right this is not about a user subscribing #hashtag@pod.example.org this is about a user subscribing #hashtag and the pod subscribes at all the pods it knows. So this might be or become a huge bunch of posts which might be heavy to handle for pods on small hardware (like single board computers). So I'd like to suggest to give the podmin the option to somehow limit this so if an user on a 5-user-pod subscribes some user somewhere it is way less load but if an user on the same pod subscribes #linux, it's getting a lot of posts :D

question/idea: The proposal and this discussion suggests to have a subscribe/unsubscribe -model. But this needs some idea of the state of the other pods... which one knows about which subscriptions (the proposal handles this).
But how about if the pod has just a list of tags it is interested in and this list is public available or can be pulled by the other pods or gets propagated when it changes or if a whatever amount of time has passed.
So instead of saying "hey okay, I'd like to have additional this and that tag and another one but forget about those:…" the pod would say "hey okay, forget about my old subscriptions, here are the current".
This way the whole thing becomes stateless and it doesn't matter if a pod misses a message or accidentially forgets about the subscriptions of other pods or similar.

Plus: I see no downsides having this list public, so there could be a list of tags a pod is interested in and every pod (or whoever is interested) could poll it whenever it wants. Additional there could be a message in the protocol that says "hey my list has changed, would you bother to pull it?". This could also be helpful in migration (to offer the list doesn't break anything and to request a page that doesn't exist too).

[annando]

@DeusFigendi this public list of all hash tags already exists: https://pirati.ca/.well-known/x-social-relay

[DeusFigendi]

Ah cool cool, so why not use this? Update this like... twice a day and start pushing fitting public posts to those pods.

The "please Update my subscriptions"-message seems very unimportant to me and should be understood soon but could be start to be sent in a far future. (I dont know how robust diaspora compatible server software is when it comes to unknown message types)

[Flaburgan]

For the record, the discussion about relays and content spreading on the mastodon side: mastodon/mastodon#7399

[cmrd-senya]

I think that it doesn't contradict the original subscription proposal to have a possibility to fetch the state of the tag subscriptions. We have fetching for public posts so we can implement fetching for tag subscription list in a similar manner. The main use for it as I see it is when pods discover each other for the first time they can fetch each others tag subscription lists in order to have up-to-date information. This feels better than pushing full tag lists unconditionally on new pod discovery. Also this might be useful when one pod wants to fix the state that was broken for some reason (e.g. offline for a long period of time). At the same time I think that for normal operation we still should use push model for tag subscription/unsibscription.

With these two features in the protocol we can actually implement a relay as a special case.

[DeusFigendi]

For the record, the discussion about relays and content spreading on the mastodon side: mastodon/mastodon#7399

Ah thanks @Flaburgan , @annando wrote there what I meant:

#issuecomment-387995637

Friendica has got some "direct relay" mechanism in the current develop branch. We enhanced the social relay protocol according to this issue: jaywink/social-relay#64
When a Friendica server had activated the "direct relay" it regularly checks all known servers for their relay subscriptions (Tags or all content). When a new public item is created, the server then sends the content additionally to these servers according to their relay settings.

This sounds very logical to me and might get fast compatible to friendicas implementation.

[annando]

This is some "minimal effort" solution, since it just extends the existing functionality.

[SuperTux88]

We could allow to fetch tags just with /fetch/tag_subscription (without any guid, because there is none), which would then include all current tags in subscribe. I don't think the existing x-social-relay matches, because the new solution wouldn't be a relay anymore, pods would send the posts directly to the target. But it maybe can be used as an initial workaround, as friendica does it now.

[Flaburgan]

So, do we have an agreement there? @SuperTux88 or @cmrd-senya, as you are the most familiar with the code base, could you please list what should be done to make it clear in case someone wants to tackle this issue?