Horrible response times

[gateway]

Please read the following instructions carefully.

Check out #1271 for an ideal bug report.
The closer your issue report is to that one, the more likely we are to be able to help, and the more likely we will be to fix the issue quickly!

Many members of the community use Stack Overflow and Server Fault to ask questions.
Read through the existing questions or ask your own!

• Stack Overflow: http://stackoverflow.com/questions/tagged/parse.com
• Server Fault: https://serverfault.com/tags/parse

For database migration help, please file a bug report at https://parse.com/help#report

Make sure these boxes are checked before submitting your issue -- thanks for reporting issues back to Parse Server!

• [ x] You've met the prerequisites: https://github.com/ParsePlatform/parse-server/wiki/Parse-Server-Guide#prerequisites.
• [x ] You're running the latest version of Parse Server: https://github.com/ParsePlatform/parse-server/releases
• [x ] You've searched through existing issues: https://github.com/ParsePlatform/Parse-Server/issues?utf8=%E2%9C%93&q=is%3Aissue Chances are that your issue has been reported or resolved before.
• [x ] You have filled out every section below. Issues without sufficient information are more likely to be closed.

Issue Description

We are testing out a heroku standard server along with mlab m1 cluster, both servers are in the same region. I installed new relic monitoring tools to see how the server would handle just a few users, maybe 5-9 and deff not at the same time. Looking at response times returns its pretty crazy to see 1 to 2 second response times ..

Here is a transaction trace from NR

Describe your issue in as much detail as possible.

No idea how to debug this or see whats causing this huge hang in response times.

Steps to reproduce

Would need our data set and app, but can provide access to a parse dev.

Expected Results

better response times.

Environment Setup

• Server
  • parse-server version (Be specific! Don't say 'latest'.) : 2.2.17
  • Operating System: Heroku Standar Dyno
  • Hardware: Heroku Standar Dyno
  • Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): Heroku
• Database
  • MongoDB version:3.2
  • Storage engine: wiredtirger
  • Hardware: Mlab M1 server 1.7 gig of ram clustered.
  • Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): mLab

Logs/Trace

You can turn on additional logging by configuring VERBOSE=1 in your environment.

nothing really in here

[nirco123]

I am having the same issues, all started when from version from 2.2.23, I suspect that the issue somehow relate to the user side of each request.

[flovilmart]

@nirco123 Are you seeing it as well on you local development machine with a local mongodb?

[jeveloper]

I see periodical spikes , regular latency against my cloud code is about 200-300ms and then once in a while 1.9 seconds or so

[acinader]

Not exactly on topic, but is there a recommended profiling tool? I'd like to start sampling but not sure what the state of the art is on this stack.

[gateway]

I going to roll back to 2.2.15, some people said anything > than that are seeing potential issues. I havent had time to debug it but have some profiling tools running over the weekend, I agree to seeing the spikes that @jeveloper is seeing as well.

[jeveloper]

I assumed that its the cached object tree that gets flushed x amount of seconds, not entirely sure, but i suspect that its flushing EVERYTHING regardless of update frequency or TTYL.

if anyone is using custom cache adaptor, id love to hear

[gateway]

So Im using app dynamics to dig though some of these long requests.. Here is an image of a call that took 868ms

those calls alone add up to 250ms.. stringify seems expensive, then 2 calls to SchemaController ?

[gateway]

Here is an hour snapshot , you can see quite a bit of slow responses happening.. :(

[gateway]

So to provide more details for example.

We have a banner class which has the following:

Start and End Date for the banner (string)
Name of the Banner (string)
Banner ID (link to s3 image)
LocationID (our location data ID - string)
Radius (string)
Disabled (Number)

Our iOS client does a query around a geoPoint to see if the user is in an area where a banner would show up.

iOS Swift Function.

class func queryForAdsToDisplay(userGeo:PFGeoPoint, complete: (result: [Banners]) -> Void) {

      let query = PFQuery(className: "Banners")
      query.whereKey("disabled", notEqualTo: 1)
      query.whereKey("geoPoint", nearGeoPoint: userGeo, withinMiles: 20)

      query.findObjectsInBackgroundWithBlock { (objects, error) in

         if error == nil {

            var dataToExtract = [Banners]()

            if objects != nil {

               for aBanner in objects! {

                  dataToExtract.append(aBanner as! Banners)
               }

               complete(result: dataToExtract)
            }

         } else {

            // Log details of the failure
            print("Error@@@@: \(error!) \(error!.userInfo)")
         }
      }
   }

We have no cloud code for this function..

Here is an example of what we are seeing under the hood from the tools we are using.

For this specific call it was 1,535 ms ..

The call graph for this class seems to have spent 61% in the auth/scram.jj line 75

This is the index for the db, please understand im getting used to figuring out how to set indexe's up since parse.com did it all for us.

I should also make a note its not just this class.

[flovilmart]

The auth/scram look to be part of the mongo driver. Not sure what we can do about it now. Maybe you should ask the guys at MongoDB about that module.

[gateway]

@flovilmart let me call them right up :) this is not the only issue, again as stated 50% of the classes are having high response times :(

[jeveloper]

I read a bit about this @gateway , it might be that we're sending different salts and scram has to do its check, that could be the problem. Are you running mongodb 3.0 (with or without rocksdb) or mongo 3.2? I wonder if its possible to tune this.

I suspect parse server is not reusing connections to mongodb and establishing new ones every time with user/pwd for every call

[gateway]

@jeveloper running 3.2 , I might fire up a 3.0 but I prefer to get 3.2 with WiredTiger running for the performance increase and space reduction.

[flovilmart]

Should we close that one as it seems this has been resolved offline?

[jeveloper]

Hey @flovilmart @gateway , please don't close, its still an issue for many of us.
As i mentioned : "I suspect parse server is not reusing connections to mongodb and establishing new ones every time with user/pwd for every call" , shouldnt the server (parse) contain a connection pool to avoid auth every time an api is called?
each time we use mongodb connect() , thats a new connection and scram is used, cpu spike and delays.

Any ideas how to mitigate this ? That would be great.