Tags performance issue Forum::tags

[luceos]

Bug Report

Current Behavior

    public function loadTagsRelationship(WillSerializeData $event)
    {
        // Expose the complete tag list to clients by adding it as a
        // relationship to the /api endpoint. Since the Forum model
        // doesn't actually have a tags relationship, we will manually load and
        // assign the tags data to it using an event listener.
        if ($event->isController(ShowForumController::class)) {
            $event->data['tags'] = Tag::whereVisibleTo($event->actor)
                ->withStateFor($event->actor)
                ->with(['parent', 'lastPostedDiscussion'])
                ->get();
        }
    }

https://github.com/flarum/tags/blob/fa83f496b0004e31ef2f4fc854bfcdf6a4cfa3c3/src/Listener/AddForumTagsRelationship.php#L46-L58

This piece of code is responsible for a performance penalty with forums that have many tags. There are a few things at fault here:

• Loading all tags into the js app on load, even though only 5 (?) are initially shown on the side bar and the index can load them through the api as an include.
• This specific piece of code seems to cause 2 queries per tag. For a forum with 100+ tags, this boils down to an additional two seconds loading time (first byte that is).

Suggested solution

What needs to be done is:

• Load only those tags on page load that are actually shown on the sidebar
• Load the additional tags on request
• Include the tags in the discussion list request

or:

• Identify and solve the issue with the Relation listener

Environment

• Flarum version: all of them, but tested on beta 13

[askvortsov1]

This PR: flarum/tags#84 should already reduce strain substantially. From there:

• It would be nice to figure out why ShowForumController is running twice: could that be a rogue extension? It doesn't seem to be that way in core.
• We could look into loading only the necessary tags (without the lastDiscussion relation too), and then loading the full set (with lastPostedDiscussion) from /api/tags the first time that the Tags page is visited.

[rafaucau]

After migrating tags from the old forum to Flarum, I wondered why the forum was loading indefinitely. Well, we have the answer - I have 2000+ tags 😄

I like the first solution:

What needs to be done is:

• Load only those tags on page load that are actually shown on the sidebar
• Load the additional tags on request
• Include the tags in the discussion list request

[askvortsov1]

@rafaucau we're now investigating another possible cause: by any chance, do you have the fof follow tags extension installed?

[rafaucau]

@askvortsov1 Yes

[askvortsov1]

Then that's likely it: an issue with FoF follow tags means that multiple SQL queries are made per tag whenever loading a page. I'm gonna close this for now unless someone can reproduce this on an install with only core extensions.

[rafaucau]

@askvortsov1 After disabling FoF follow-tags I don't see any difference. It still loads very slowly

[askvortsov1]

Nevermind then... Could you open a new issue and share the output of your php flarum infoand link to this one

[simon1222]

The problem is generated in TagSerializer by:

'canStartDiscussion' => $this->actor->can('startDiscussion', $tag),
'canAddToDiscussion' => $this->actor->can('addToDiscussion', $tag),

and

protected function lastPostedDiscussion($tag)
    {
        return $this->hasOne($tag, DiscussionSerializer::class);
    }

[askvortsov1]

@simon1222 is this something you've confirmed via testing? We were able to determine that disabling FoF Follow Tags reduces the number of SQL queries sent to the database on initial load from 411 to 54 for a forum with hundreds of tags.

[simon1222]

@askvortsov1 yes, I've confimed via testing. I have 320k discussions and 2k tags, so lastPostedDiscussion was one of the problem. I was waiting about 60s, so I've commented return $this->hasOne($tag, DiscussionSerializer::class); and reduced to 30s.
I was testing line by line and this piece of code is working fine:

$event->data['tags'] = Tag::whereVisibleTo($event->actor)
                ->withStateFor($event->actor)
                ->with(['parent', 'lastPostedDiscussion'])
                ->get();

but methods, which are executed in Serializer, are working too slow. I think these methods generate too many queries to the database.