Parse-server V6.3.1 query.distinct("score") throw ParseError: Invalid aggregate stage 'hint'

[zurmokeeper]

New Issue Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.
• I can reproduce the issue with the latest version of Parse Server.

Issue Description

const distinctResult = await query.distinct("score");

parse-server v6.3.1 This code will give you an error.

ParseError: Invalid aggregate stage 'hint'.

Steps to reproduce

src/cloud/main.js

Parse.Cloud.define('getObject', async (request) => {
    try {
      const className = 'GameScoreXXX';
      const objectId = "3JrNlj8Wkf";
      const query = new Parse.Query(className);
      const distinctResult = await query.distinct("score");   // throw ParseError: Invalid aggregate stage 'hint'.
      const object = await query.get(objectId);
      return object.toJSON();
    } catch (error) {
     console.log('error-->', error)
      throw new Parse.Error(500, 'Failed to update object: ' + error.message);
    }
});


// src/index.js
var express = require('express');
var ParseServer = require('parse-server').ParseServer;
const PORT = 1337;
...........................................
var api = new ParseServer({
    databaseURI: database.uri,
    cloud: server.cloud,
    appId: server.appId,
    masterKey: server.masterKey
});

var app = express();
(async ()=>{
    await api.start()    
    app.use("/parse", api.app)  

    var httpServer = require('http').createServer(app)
    httpServer.listen(PORT, function() {
        console.error('parse-server-mojitest running on port ' + PORT + '.')
    })
})()

Actual Outcome

ParseError: Invalid aggregate stage 'hint'.

Expected Outcome

no error message

Environment

Server

• Parse Server version: V6.3.1
• Operating system: win11
• Local or remote host (AWS, Azure, Google Cloud, Heroku, Digital Ocean, etc): local

Database

• System (MongoDB or Postgres): MongoDB
• Database version: 5.0
• Local or remote host (MongoDB Atlas, mLab, AWS, Azure, Google Cloud, etc): Local

Just use the third release tool to request it directly

Logs

Some Detail

// Parse-server V5.6.1  src/Routers/AggregateRouter.js  

export class AggregateRouter extends ClassesRouter {
  handleFind(req) {
    const body = Object.assign(req.body, ClassesRouter.JSONFromQuery(req.query));
    const options = {};
    if (body.distinct) {
      options.distinct = String(body.distinct);
    }
     //  Here body.hint : underfined, so it doesn't delete to the
    if (body.hint) {
      options.hint = body.hint;
      delete body.hint;
    }

   // so body.hint : underfined
}

So when stageName=hint   here, it's a direct error

  static transformStage(stageName, stage) {
    const skipKeys = ['distinct', 'where'];
    if (skipKeys.includes(stageName)) {
      return;
    }
    if (stageName[0] !== '$') {
      throw new Parse.Error(Parse.Error.INVALID_QUERY, `Invalid aggregate stage '${stageName}'.`);
    }
}

}

[parse-github-assistant]

Thanks for opening this issue!

• 🚀 You can help us to fix this issue faster by opening a pull request with a failing test. See our Contribution Guide for how to make a pull request, or read our New Contributor's Guide if this is your first time contributing.

[theolundqvist]

I am having the same problem

[chadpav]

I'm updating an older Parse 5.2.x server running against MongoDB 4.4. I noticed this deprecation warning in my logs:

DeprecationWarning: Using aggregation stages without a leading '$' is deprecated and will be removed in a future version. Try $distinct instead.

Could your error be related to this? If you are running MongoDB 5 or higher maybe they no longer support distinct queries without the leading '$'.

I came here to validate that a new parse server implemented this change. BTW, your description says you are running Parse Server 6.3.1 but the extra details say Parse Server 5.6.1. Can you verify that?

Are you in a position to test this against MongoDB 4.4 and/or against the latest Parse Server?

[chadpav]

Following up on my comment above, I can verify that these issues are related. I updated my Parse Server 5.2.x server all the way to MongoDB 7.x and still received the DeprecationWarning. Everything worked though.

Next, I upgraded to the latest 5.6.x Parse Server and everything was working.

Finally, I upgraded to latest Parse Server 6.4.x and now I get the same error as described in the initial bug report. I rolled back to 6.0.x and confirmed the bug was introduced between 5.6 -> 6.0.

I am using a distinct query on an object pointer in Cloud Code if that helps narrow it down. I'll keep digging as well.

[chadpav]

Update

• This was a planned deprecation which started 2-3 years ago
• I have a workaround... but I believe this is a bug in the Parse Server distinct method not converting to mongo pipelines in the AggregateRouter correctly. I tried to locate the bug but gave up after a while.
• You can experiment with using the mongo db aggregate pipeline syntax directly (I got that to work too but it's not as friendly as the Parse api's)
• The workaround that I'm using for now is supplying a query hint. I'm not exactly sure how it fixes the issue but I tried it on a hunch from something the original poster said. Boom! I'm back in business.
• This is a still a bug, hopefully @mtrezza will prioritize. 👍

My pseudocode:

...
  var games = Parse.Object.extend('Games')
  var query = new Parse.Query(games)
  query.equalTo('stadiumRef', stadiumRef); // stadiumRef is an object pointer to a record in the stadiums collection

  query.hint('teamRef'); // <== Adding this fixes my distinct query in Parse Server >=6.0

  // Get count of distinct team's that have played a game in the stadium
  query.distinct('teamRef', {useMasterKey: true})
    .then(function(results) {
....

[mtrezza]

@chadpav Would you want to open a PR with a failing test, and do you have a suggestion for a bug fix? Once we have a failing test we can also put a bounty to expedite the fixing.

[chadpav]

I have a work around and I did try to dig into the server code but it's a little beyond my skillset to do so. Hopefully I've narrowed down the reproduction steps for you guys.

[Chilldev]

It happens because ParseQuery.distinct,aggregate in Parse-SDK-JS usually sends hint: undefined to AggregateRouter.handleFind which does not remove the hint from the request body unless it holds a value then it's passed to AggregateRouter.getPipeline(body);.
It has been modified to make sure aggregate operators are preceded by a $ which is not the case for hint: undefiend hence the invalid aggregate stage error is thrown.

Original code:

handleFind(req) {
    const body = Object.assign(req.body, ClassesRouter.JSONFromQuery(req.query));
    const options = {};
    if (body.distinct) {
      options.distinct = String(body.distinct);
    }
    if (body.hint) {
      options.hint = body.hint;
      delete body.hint;
    }
    if (body.explain) {
      options.explain = body.explain;
      delete body.explain;
    }
    if (body.comment) {
      options.comment = body.comment;
      delete body.comment;
    }
    if (body.readPreference) {
      options.readPreference = body.readPreference;
      delete body.readPreference;
    }
    options.pipeline = AggregateRouter.getPipeline(body);
    if (typeof body.where === 'string') {
      body.where = JSON.parse(body.where);
    }
    return rest
      .find(
        req.config,
        req.auth,
        this.className(req),
        body.where,
        options,
        req.info.clientSDK,
        req.info.context
      )
      .then(response => {
        for (const result of response.results) {
          if (typeof result === 'object') {
            UsersRouter.removeHiddenProperties(result);
          }
        }
        return { response };
      });
  }

A fix:

handleFind(req) {
    const body = Object.assign(req.body, ClassesRouter.JSONFromQuery(req.query));
    const options = {};
    if (body.distinct) {
      options.distinct = String(body.distinct);
    }
    ['hint', 'explain', 'comment', 'readPreference'].forEach(key => {
      if (body[key]) options[key] = body[key];
      delete body[key];
    });
    options.pipeline = AggregateRouter.getPipeline(body);
    if (typeof body.where === 'string') {
      body.where = JSON.parse(body.where);
    }
    return rest
      .find(
        req.config,
        req.auth,
        this.className(req),
        body.where,
        options,
        req.info.clientSDK,
        req.info.context
      )
      .then(response => {
        for (const result of response.results) {
          if (typeof result === 'object') {
            UsersRouter.removeHiddenProperties(result);
          }
        }
        return { response };
      });
  }

A test:

const Config = require('../lib/Config');
const ClientSDK = require('../lib/ClientSDK');

  it('"handleFind" properly handles "req.body.hint" and invalid aggregate stage error should not be thrown', async () => {
   const config = Config.get('test');
   const clientSDK = ClientSDK.fromString('i0.0.0');
   const req = {
     config,
     params: { className: '_User' },
     body: {
       distinct: 'first_name',
       where: {
         objectId: 'someId',
       },
       hint: undefined,
     },
     info: {
       clientSDK,
       context: '1',
     },
     auth: { readOnly: false },
   };

   const expected = { response: { results: [] } };

   const aggregateRouter = new AggregateRouter();
   const result = await aggregateRouter.handleFind(req);

   expect(result).toEqual(expected);
 });

[mtrezza]

@Chilldev would you want to submit a PR with a failing test?

[Chilldev]

@mtrezza Sure. Let me check the contribution guide.

[Chilldev]

The issue occures when directAccess is set to true and the distinct query is wrapped within an eachBatch.

I wrote the test but for it to work I have to comment this line:

spec/helper.js:190

Parse.CoreManager.setRESTController(RESTController);

The error occures because this is what naturally happens when parse server started withdirectAccess set to true:

src/ParseServer:323

if (process.env.PARSE_SERVER_ENABLE_EXPERIMENTAL_DIRECT_ACCESS === '1' || directAccess) { Parse.CoreManager.setRESTController(ParseServerRESTController(appId, appRouter)); }

ParseServer rest controller is diffrent from Parse-JS-SDK I suppose which makes it access the AggregateController directly instead of SchemaController.