How to Implement Redis Caching in my API to Retrieve New and Different Posts Every Time?

[xts-bit]

Description

I am trying to retrieve some posts from my MongoDB database but it was taking a lot of time to get posts to I implemented Redis caching in my API but now I am facing a problem that it's returning the same posts from the cache every time I hit the API How to get new and different posts from redis cache whenever I hit the API? Instagram uses Redis and they give me new and different posts every time I scroll so fast How to balance things in my case?

What can be a better approach to solve this problem with the API?

router.get('/posts', async (req, res) => {
    try {
        const cacheKey = `posts:${req.query.page || 1}`;
        const cacheTTL = 60;

        const cachedData = await redisClient.json.get(cacheKey, '$');

        if (cachedData) {
            console.log('Data fetched from Redis cache');
            return res.json(cachedData);
        }

        const page = Number(req.query.page) || 1;
        const limit = Number(req.query.limit) || 50;
        const skip = (page - 1) * limit;

        const result = await User.aggregate([
            { $project: { posts: 1 } },
            { $unwind: '$posts' },
            { $project: { postImage: '$posts.post', date: '$posts.date' } },
            { $sort: { date: -1 } },
            { $skip: skip },
            { $limit: limit },
        ]);

        await redisClient.json.set(cacheKey, '$', result, 'EX', cacheTTL);

        console.log('Data fetched from MongoDB and cached in Redis');
        res.json(result);
    } catch (err) {
        console.error(err);
        res.status(500).json({ message: 'Internal server error' });
    }
});

[itamarhaber]

Hello @xts-bit

As far as I know, the JSON.SET doesn't support the EX option, so this line just sets the value without an expiry:

await redisClient.json.set(cacheKey, '$', result, 'EX', cacheTTL);

You should replace it with something like:

await Promise.all([
  redisClient.json.set(cacheKey, '$', result),
  redisClient.expire(cacheKey, cacheTTL)
]);

[xts-bit]

@itamarhaber Yep I changed it and now deletes the cache after 60 seconds,

but any idea "How to get new and different posts from the Redis cache whenever I hit the API? Instagram uses Redis and they give me new and different posts every time I scroll so fast How to balance things in my case?"

It takes 700ms on average to get a response when i implemented Redis Cache

[itamarhaber]

I don't understand what "new and different posts" means

[xts-bit]

@itamarhaber i mean when I currently hit the API I get the same post every time for 60 seconds after 60 seconds i get a new post because its fetched from MongoDB

But when i scroll Instagram posts it gives me new and different posts every time and with a very fast time

How can i also get new and different posts from Redis cache very fast like Instagram

[leibale]

@xts-bit This question has no "right" answer, there are many ways to do it, each with its advantages and drawbacks...

The simplest way I can think of is to have more than one "posts bucket" for each user, each bucket sorts the posts in a different order, and each time the user opens the app or refresh, use a different bucket.

This "trick" can be implemented using "normal" keys just like you did, but instead of having 1 key per user - create multiple of them, or you can store the posts directly in Redis using RedisJSON + RediSearch, add a random number to each document, and use a formula to create random but consistent buckets of posts using FT.SEARCH with APPLY and SORT.

I hope that this (kind of) answers your question.. :)

[xts-bit]

@leibale Thank you for telling but my code still returns me same posts every time i generate a random bucket index based on the user ID

router.get('/posts', async (req, res) => {
    try {
        const userId = req.body;
        const numBuckets = 5; 
        const cacheTTL = 60;

        const bucketIndex = userId % numBuckets;
        const cacheKey = `posts:${bucketIndex}:${req.query.page || 1}`;

        const cachedData = await redisClient.json.get(cacheKey, '$');

        if (cachedData) {
            console.log('Data fetched from Redis cache');
            return res.json(cachedData);
        }

        const page = Number(req.query.page) || 1;
        const limit = Number(req.query.limit) || 50;
        const skip = (page - 1) * limit;

        const result = await User.aggregate([
            { $project: { posts: 1 } },
            { $unwind: '$posts' },
            { $project: { postImage: '$posts.post', date: '$posts.date' } },
            { $sort: { date: -1 } },
            { $skip: skip },
            { $limit: limit },
        ]);

        await redisClient.json.set(cacheKey, '$', result, 'EX', cacheTTL);

        console.log('Data fetched from MongoDB and cached in Redis');
        res.json(result);
    } catch (err) {
        console.error(err);
        res.status(500).json({ message: 'Internal server error' });
    }
});

[leibale]

@xts-bit because you are using the same query in all of them.. you need to use the "bucket id" to sort the posts differently (you can use the trick I suggested last comment to do it "randomly").

[xts-bit]

@leibale Is this way correct, generating a random bucket index using Math.random() and Math.floor()?

router.get('/posts', async (req, res) => {
  try {
    const userId = req.body;
    const numBuckets = 5;
    const cacheTTL = 60;
    const bucketIndex = Math.floor(Math.random() * numBuckets);
    const cacheKey = `posts:${bucketIndex}:${req.query.page || 1}`;
    const cachedData = await redisClient.json.get(cacheKey, '$');
    if (cachedData) {
      console.log('Data fetched from Redis cache');
      return res.json(cachedData);
    }
    const page = Number(req.query.page) || 1;
    const limit = Number(req.query.limit) || 50;
    const skip = (page - 1) * limit;
    const result = await User.aggregate([
      { $project: { posts: 1 } },
      { $unwind: '$posts' },
      { $project: { postImage: '$posts.post', date: '$posts.date' } },
      { $sort: { [bucketIndex % 2 === 0 ? 'date' : 'postImage']: bucketIndex % 2 === 0 ? -1 : 1 } },
      { $skip: skip },
      { $limit: limit },
    ]);
    await redisClient.json.set(cacheKey, '$', result, 'EX', cacheTTL);
    console.log('Data fetched from MongoDB and cached in Redis');
    res.json(result);
  } catch (err) {
    console.error(err);
    res.status(500).json({ message: 'Internal server error' });
  }
});