feat: realtime API and Payload ORM

[GermanJablo]

PayloadQuery

Currently you can stream DB updates to the client using a polling strategy (refetching at intervals). For example, to get the number of posts with react-query:

const { data, error, isLoading } = useQuery({
  queryKey: ['postsCount'],
  queryFn: () => fetch('/api/posts/count').then((res) => res.json()),
  refetchInterval: 5000, // refetch every 5 seconds
})

This has some problems:

1. It's not really "real-time". You have to wait for the refetch interval to be met.
2. It is not efficient, since even if nothing changed, it will make unnecessary requests.
3. It is not type-safe.

To solve these problems, we are introducing payloadQuery.

To use it in React, you need to wrap your app with PayloadQueryClientProvider:

import { PayloadQueryClientProvider } from '@payloadcms/plugin-realtime'

export function App({ children }: { children: React.ReactNode }) {
  return (
    <PayloadQueryClientProvider>
      {children}
    </PayloadQueryClientProvider>
  )
}

Now you can use the usePayloadQuery hook anywhere in your app:

import { usePayloadQuery } from '@payloadcms/plugin-realtime'

const { data, error, isLoading } = usePayloadQuery('count', { collection: 'posts' })

This will automatically update the data when the DB changes!

You can use all 3 Local API reading methods: find, findById, and count (with all their possible parameters: where clauses, pagination, etc.).

How it works

Under the hood, PayloadQueryClientProvider opens a single centralized connection that is kept alive, and is used to listen and receive updates via Server Sent Events (SSE). The initial request is made with a normal HTTP request that does not require keep-alive.

The same queries are cached on the client and server, so as not to repeat them unnecessarily.

On the server, the afterChange and afterDelete hooks are used to loop through all registered queries and fire them if they have been invalidated. Invalidation logic allows for incremental improvements. Instead of naively invalidating all queries for any change in the database, we can analyze the type of operation, the collection, or options such as the where clause.

What if I don't use React?

This plugin was intentionally made framework-agnostic, in vanilla javascript:

import { createPayloadClient } from '@payloadcms/plugin-realtime'

const { payloadQuery } = createPayloadClient()

const initialCount = await payloadQuery(
  'count',
  { collection: 'posts' },
  {
    onChange: (result) => {
      if (result.data) {
        console.log(result.data.totalDocs) // do something with the result
      }
    },
  },
)

The React version is just a small wrapper around these functions. Feel free to bind it to your UI framework of choice!

What if I use Vercel / Serverless?

Serverless platforms like Vercel don't allow indefinitely alive connections (either HTTP or WebSockets).

This API has a reconnection logic that can solve some problems, but even then when the server reconnects the queries stored in memory would be lost, so the client might miss some updates.

We want to explore later how to solve this, either by storing it in the payload database or in some external service or server.

TODO

• Discuss overall strategy and API (this is a very primitive PoC yet)
• Add tests
• Add docs
• Make it fully type-safe. Currently, the parameter options are type-safe depending on the method chosen (find, findById or count). But I would like to make the responses type-safe as well, like in an ORM. My idea is to pass an auto-generated parser to the ClientProvider that follows the collections interface.
  EDIT: Luckily I didn't start doing this, I wasn't aware that this work was already being done with the SDK. Using that package could simplify the development of this one.
• To be discussed: Currently, this is a plugin. I think it would make sense to move the vanilla implementation to core and the react hook to the /ui package.
• Provide similar methods or hooks for type-safe mutations
• Reliability in serverless. Store in our own DB, or an external service or server?
• Our intention is to do much more in the real-time domain, such as showing who is editing a document in the admin panel. We would have to decide what name we want to give to this API that is a part of that broader umbrella. I am currently using the term PayloadQuery because of the similarity to react-query, but it actually does quite a bit more than react-query. There are many related terms that come to mind: RCP, ORM, real-time, reactive, etc.
• Options could be added to disable real-time updates and/or caching. This can be useful for example in development when you don't feel like configuring the client provider, or if you're on serverless, or you just don't care about real-time for another reason.

[HarleySalas]

would it be feasible to combine this idea with ritsu's fully typed sdk?

I think it'd be a pretty nice DX to just be able to pass realtime: true or something along those lines to a query and it just "work."
I have to admit, I don't like the name "payloadQuery" at all for this, as it does not imply that it is focused on realtime. By the name, I would expect it to be a query client, without the whole realtime thing going on.

Just thinking out loud. I haven't looked in to the code itself yet, but I think it'd be genuinely game changing if we could provide initial state via the local api/server, use the rest API for things like infinite scrolling and interaction that must take place on the client and then the icing on the cake, to simply be able to tell the data to always be in realtime would be one of the greatest features of payload overall.

[GermanJablo]

would it be feasible to combine this idea with ritsu's fully typed sdk?

Definitely! This is precisely why we agreed to review that PR soon.

---

About the name, thanks for the feedback! I agree that we need something better.

As far as the vanilla option goes, we could just use the sdk's third argument object to configure this behavior:

const [initialPostsCount, onChangePostsCount] = await sdk.count(
  { collection: 'posts' },
  {
    // option 1: `realtime` prop. If true, we return a tuple of [initialResult, onChange]
    // I think the first element of the tuple can be confusing (people might think it's reactive)
    realtime: true,
    // option 2: onChange callback. if defined, realtime is assumed. No tuple returned
    onChange: (result) => { /** ... */ }
  },
)

As far as the react hook goes, I like the idea of ​​it being reactive/real-time by default and that being opt-out. There is also the option of not having a default and making the realtime property required. In the vanilla version I can see it making more sense to have it be false by default, as the mechanism can be a bit more verbose. On the other hand, having different defaults in react and vanilla might be a bit weird.

I'd love to hear your thoughts on the default value for real-time both in vanilla and react, as well as possible names for the react hook ☺️

---

I also thought about infinite scrolling and it deserves further thought... could it be part of the react hook somehow? 🤔