Experiment/Draft: Search engine

[SmallCoccinelle]

This branch implements a search engine in the stash backend. It is currently very much a draft, as there are numerous unclosed cases which needs tracking. But In the spirit of making the experiment a bit more visible, I'll make a draft PR for it.

Status

Currently we implement:

• A new GraphQL field, Query.search(..).
• Dataloaders for scene, performer, tag
• Full reindexing of the sqlite database into a single search index
• On-the-fly reindexing of changes, piggybacking on plugin change hooks
• Limited documents for Scene, Performer, and Tag. These are deliberately kept small until the main design has settled.

We can query for scenes and performers. Scoring is TF-IDF.

Query string support is what Bleve supports. I.e., a query kitty redhead searches for documents with kitty OR redhead, but scores objects where both match higher than documents with a partial match. You can do either "kitty redhead" for an exact phrase match, +kitty +redhead for an conjunctive match (i.e., AND), or +kitty /red(head)?/ for the corresponding regex match and so on.

We also support dates. I.e., +date:>=2013 +date:<2020 searches for objects in that date range.

We support a simple date range facet tracking recent stuff (less than 30 days old) from older stuff. But clearly, this needs to be a GraphQL input type so the front-end can manipulate the facets.

Performance

Currently, indexing 20k scenes and 300 performers takes about 5-6 seconds. This is fast, but as we flesh out the documents, it's going to take a lot longer because we need to analyze more and more data. I have a ~4TB stash and the current index size is 7 megabyte. Again, this is going to climb. My Sqlite db is 148 megabyte.

Things missing

• We are currently indexing a small subset of what is in sqlite. This is deliberate. It's relatively easy to flesh out once the other parts fall into place. That's where the hard parts are.
• Index Management. We want to occasionally reindex everything to remove staleness. We also want to reindex if the index version changes. Doing so should happen in the background with an index swap once the reindex has caught up to the current state. The tool here is index aliases.
• Handling of tags. With performers and tags in the experiment, we would know way more about data interlinkage.
• Proper deletion. If a tag/performer is being deleted, it must also be removed from all scenes. This is currently not implemented, but can be done by batched index queries. We want to track a deletion list for a batch and handle it accordingly.
• Proper configuration. I sorta hacked in a new config-knob, but it does require some more attention, and I'm not sure how to handle this correctly so it can eventually show in the UI.
• Galleries. To be handled later.
• Movies. To be handled later.
• Studios. To be handled later.
• Images. Large document set.
• Reindexing is currently a mess. Plan is to fix this once we have tags indexed as well because there's enough working stuff going on. It's slowly being improved, but there's some repeated code in there I don't like.
• There's a lot of batching in the indexing process to avoid using too many resources. This is currently untuned, and somewhat requires indexing not being the mess it currently is.
• A lot of this code should be fairly easy to test. But there are way too few test cases at the moment.

Rough code overview

The code is mostly orthogonal to the rest of stash. Our interfacing points are:

• Some transactions using repositories.
• The event bus, for tracking on-the-fly changes.
• Dataloaders. I added them to pkg/models.

The go files are as follows:

• documents/documents.go - Implements the documents which get stored in the search index, together with the index mappings of those documents.
• loaders.go a collection of data loaders used by the search engine. Can be pushed into the resolver code as well.
• changeset.go - implements changesets. Changesets can be turned into index batches and applied to the index.
• rollup.go - A rollup goroutine tracks event changes into a changeset and hands them off to the search engine upon request.
• search.go - The main API. Implements search.Search(..) the main API entry.
• engine.go - Implements the meat of the search engine backend. An engine is governed by a managing goroutine, communicating with a rollup goroutine to maintain the index. The engine handles reindexing.
• engine_indexing.go - Indexing code for the engine
• engine_preprocess.go - Preprocessing code for the index. This analyzes a changeset to figure out collateral updates.

Playground

Add

search: /path/to/index/directory

Into your stash config. Start stash. It should generate an index and start a full reindex in 5s. Every 15s it should write out stats for the reindexing. If you trash the index.bleve folder, it will reindex. Then go to your GraphQL playground. A typical query would look something like

query Q {
  search(query: "kitty redhead") {
    took
    total

    edges {
      score
      node {
        id
        __typename
        ... on Performer {
          name
        }
        ... on Scene {
          title
          details
          date
          updated_at
          performers {
            name
            eye_color
          }
        }
      }
    }
  }
}

[SmallCoccinelle]

Things which are easily salvaged out of this:

• Dataloaders
• SearchResultItem. This interface is usually called Node in GraphQL parlance, but unfortunately, our IDs don't contain the type, so supporting

interface Node {
    id: ID!
}

type Scene implements Node {
    ...
}

type Query {
    nodes(ids: [ID]): [Node]
    ...
}

isn't really possible.

[SmallCoccinelle]

• Properly handle Tag deletion. It seems quite within reach now we properly delete performers from the index.

[SmallCoccinelle]

• Handle tag merging. No event is being caught, so is one being generated?

[SmallCoccinelle]

• Studios should be linked to Scenes

[skier233]

@SmallCoccinelle Is there any interest in finishing this or would it need started from scratch?