Some Push Notifications stuck in SENDING state

[UnlikelySassou]

Issue Description

I've recently moved my Parse Server instance to Heroku. I've put my push logic in Cloud Code instead of using client push and I can send and receive push notifications.

My issue is that the status of the push object is stuck at SENDING in Parse Dashboard ("status": "running" on the mongo object) when one of the pushes can't be delivered to an installation.

Example: I send a push to a list of users, but some of them have an installation with no deviceToken configured. Those with a deviceToken receive the push, but the others don't.

That's an expected result, but the push status will stay forever in the SENDING state and I'm not sure if I should worry about it or not.

Note: The push status is changed correctly to SENT if all the Installations have a deviceToken and the number of push sent matches the number of push delivered. Also this behaviour did not occurred when I was using client push.

Steps to reproduce

Send a push to one or more installations with some correctly configured and some with no deviceToken.

Expected Results

Those with a deviceToken receive the push, but the others don't.
The push status change to something other than "running" even if some failed.

Actual Outcome

The push object is stuck forever at SENDING in Parse Dashboard ("status": "running" on the mongo object) even is some notifications were delivered.

I'm not sure if this is a big issue and if I should worry about the server trying to send pushes indefinitely.

Environment Setup

• Server

  • parse-server version : 2.3.5
  • Hardware: Standard-1x (512 MB)
  • Localhost or remote server?: Heroku

• Database

  • MongoDB version: 3.0.14
  • Hardware: Shared Cluster
  • Localhost or remote server? mLab

Logs/Trace

Example: 3 pushes sent, 1 device received it, 2 failed because of missing deviceToken. Push object now in sending state forever.

2017-02-24T13:59:29.004277+00:00 app[web.1]: �[32minfo�[39m: Ran cloud function sendPushToRecipients for user xVOydE7ugH with:
2017-02-24T13:59:29.004293+00:00 app[web.1]:   Input: {"recipients":["dpVQ9ZQypH","Vs3OADhHnS"],"data":{"alert":"test.","NTYPE":"NEW","badge":1,"sound":"default"},"appVersion":"1"}
2017-02-24T13:59:29.004295+00:00 app[web.1]:   Result: "Push Sent!" functionName=sendPushToRecipients, recipients=[dpVQ9ZQypH, Vs3OADhHnS], alert=test., NTYPE=NEW, badge=1, sound=default, appVersion=519, user=xVOydE7ugH

_PushStatus object

{
    "_id": "ZieiF3w31X",
    "pushTime": "2017-02-24T13:59:28.981Z",
    "query": "{\"currentProfile\":{\"$inQuery\":{\"where\":{\"objectId\":{\"$in\":[\"dpVQ9ZQypH\",\"Vs3OADhHnS\"]}},\"className\":\"Profile\"}},\"appVersion\":\"1\"}",
    "payload": "{\"alert\":\"test",\"NTYPE\":\"NEW\",\"badge\":1,\"sound\":\"default\"}",
    "source": "rest",
    "status": "running",
    "numSent": 1,
    "pushHash": "882c51079dfab452aed1f146fedb37a2",
    "_wperm": [],
    "_rperm": [],
    "_acl": {},
    "_created_at": {
        "$date": "2017-02-24T13:59:28.981Z"
    },
    "_updated_at": {
        "$date": "2017-02-24T13:59:29.066Z"
    },
    "count": 3,
    "sentPerType": {
        "ios": 1
    }
}

Cloud Code

Parse.Cloud.define("sendPushToRecipients", function(request, response) {
  var profileQuery = new Parse.Query("Profile");
  profileQuery.containedIn("objectId", request.params.recipients);
  var installationQuery = new Parse.Query(Parse.Installation);
  installationQuery.matchesQuery("currentProfile", profileQuery);
  installationQuery.equalTo("appVersion", request.params.appVersion);
  
  Parse.Push.send({
    expiration_interval:600,
    where: installationQuery,
     data: request.params.data
    }, {
      success: function() {
        response.success("Push Sent!");
      },
      error: function(error) {
        response.error("Push failed: " + error);
      },
      useMasterKey: true
    });
});

[adirgan]

@flovilmart It's the same problem that I raised in #3080

[UnlikelySassou]

@adirgan Yes, it seems to be the same issue. My Parse Dashboard push page is very similar: the push status is changed correctly to SENT only if all the _Installation objects targeted by the push have a deviceToken. No matter if it's sent to a channel or specific recipients.

From what I've read, is it safe to assume that all the push notifications that have an audience (with valid deviceToken) are actually sent anyway and that it is just an issue with Parse Server not updating _PushStatus correctly because of the faulty _Installation objects?

I'm worried that this SENDING state means some performance issue or something bad with the server. If it is just a state update issue, I can live with that for now.

[adirgan]

@iOSassou Do not worry that the push messages arrive without problem and does not affect the performance at all, it is only an error to update the status.

[UnlikelySassou]

@adirgan Thank you

[flovilmart]

@adirgan sorry, I believe the ball was dropped on my side on that... 🙇 I need to investigate further. But the state reporting has proven to be quite unreliable in general :/

[adirgan]

@flovilmart Do not worry, while the problem is only visual and not performance is fine.

[UnlikelySassou]

@flovilmart @adirgan Yes. As long as it is not really affecting the app, I think it's fine (maybe just a bit confusing).

[flovilmart]

Yeah that should be fixed however :/ that's clearly a regression

[felipeandradebezerra]

@flovilmart Have found the same issue here after upgrading to the latest version.
Any clues in how to solve that?

[flovilmart]

Need to add a small check to exclude installations without deviceToken from the query

[jeacott1]

upgrade to apn 2.x is really the only solution for proper tracking.

[flovilmart]

Also there is a bug that I uncovered that improperly forwards the deviceType on successful pushes.

[felipeandradebezerra]

@flovilmart All iOS pushes are not being send, need to downgrade to Parse-Server 2.3.1 to get it working properly.

[flovilmart]

@felipemobile are they not sent or just marked 'sending'?

[felipeandradebezerra]

Not sent @flovilmart

[felipeandradebezerra]

Maybe a conflict after the #3264, #3080 improvements?

[flovilmart]

Uhm, not sent is more problematic than just not marked as Sent, can you provide more details, logs etc... to debug that issue? There was an underlying bug in the push adapter that prevented the push status to be properly reported.

[flovilmart]

Sending a single push don't seem to cause the issue

[felipeandradebezerra]

@flovilmart do you need any special type of log to debug that issue?

[flovilmart]

Setting VERBOSE=1 should log all see https://github.com/parse-server-modules/parse-server-push-adapter/blob/master/src/index.js

[felipeandradebezerra]

@flovilmart I've confirmed! This bug was introduced in the parse-server 2.3.3, maybe it's related to #3264 improvement.

[flovilmart]

@felipemobile I know the bug is here, however I'm not sure the push are not delivered

[felipeandradebezerra]

I've done a very simple and basic test:

Works (Push sent 1)

$ npm install parse-server@2.3.2
$ npm start index.js

Not working (Push sent 0)

$ npm install parse-server@2.3.3
$ npm start index.js

[flovilmart]

yeah but are they delivered on the device? that doesn't really help here.

[felipeandradebezerra]

Yeap, they are delivered only when using parse-server 2.3.2 or lower.

[felipeandradebezerra]

Please, check this log:

/app/push: {
  "data": {
    "alert": "Teste"
  },
  "where": {
    "deviceToken": "3ac4dbf3bc1c73e2ec2542cb183f3b12e516e38cfccc2a607c64dbe05b45a413",
    "deviceType": {
      "$in": [
        "ios"
      ]
    }
  }
} method=POST, url=/app/push, x-real-ip=172.31.33.54, x-forwarded-for=177.159.215.149, 172.31.33.54, x-nginx-proxy=true, host=sandbox.app.com.br, connection=close, content-length=331, accept=*/*, accept-encoding=gzip, deflate, br, accept-language=pt,en-US;q=0.8,en;q=0.6, content-type=text/plain, dnt=1, origin=https://dashboard.app.com.br, referer=https://dashboard.app.com.br/apps/Di%C3%A1rio%20do%20Par%C3%A1/push/new, user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36, x-forwarded-port=443, x-forwarded-proto=https, alert=Teste, deviceToken=3ac4dbf3bc1c73e2ec2542cb183f3b12e516e38cfccc2a607c64dbe05b45a413, $in=[ios]
verbose: RESPONSE from [POST] /app/push: {
  "headers": {
    "X-Parse-Push-Status-Id": "CvtA31WMJq"
  },
  "response": {
    "result": true
  }
} X-Parse-Push-Status-Id=CvtA31WMJq, result=true
verbose: _PushStatus CvtA31WMJq: sending push to 1 installations

ERR! parse-server-push-adapter APNS no qualified connections for com.app 3ac4dbf3bc1c73e2ec2542cb183f3b12e516e38cfccc2a607c64dbe05b45a413
verbose: _PushStatus CvtA31WMJq: sent push! 0 success, 1 failures

[flovilmart]

So it reaches correctly the adapter but the connection is not found, and he status is not reported as failed. Can you check with a valid connection?

[felipeandradebezerra]

@flovilmart This is a valid connection, changing to parse-server 2.3.2 works perfectly.

$ npm install parse-server@2.3.2

verbose: REQUEST for [POST] /app/push: {
  "data": {
    "alert": "Teste"
  },
  "where": {
    "deviceToken": "3ac4dbf3bc1c73e2ec2542cb183f3b12e516e38cfccc2a607c64dbe05b45a413",
    "deviceType": {
      "$in": [
        "ios"
      ]
    }
  }
} method=POST, url=/app/push, x-real-ip=172.31.18.105, x-forwarded-for=177.159.215.149, 172.31.18.105, x-nginx-proxy=true, host=sandbox.app.com.br, connection=close, content-length=331, accept=*/*, accept-encoding=gzip, deflate, br, accept-language=pt,en-US;q=0.8,en;q=0.6, content-type=text/plain, dnt=1, origin=https://dashboard.app.com.br, referer=https://dashboard.app.com.br/apps/Di%C3%A1rio%20do%20Par%C3%A1/push/new, user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36, x-forwarded-port=443, x-forwarded-proto=https, alert=Teste, deviceToken=3ac4dbf3bc1c73e2ec2542cb183f3b12e516e38cfccc2a607c64dbe05b45a413, $in=[ios]
verbose: RESPONSE from [POST] /app/push: {
  "headers": {
    "X-Parse-Push-Status-Id": "cK2bS1vwCx"
  },
  "response": {
    "result": true
  }
} X-Parse-Push-Status-Id=cK2bS1vwCx, result=true
verbose: sending push to 1 installations
verbose: sent push! 1 success, 0 failures

[flovilmart]

The adapter says the opposite, which is odd.

[felipeandradebezerra]

In some way it's not clustering the devices per connections

parse-server-push-adapter/APNS.js

110     let qualifiedConnIndexs = chooseConns(this.conns, device);
111	  if (qualifiedConnIndexs.length == 0) {

Maybe a issue in the adapter?

src/Adapters/Push/PushAdapter.js

export class PushAdapter {

 -  send(devices, installations, pushStatus) { }
 +  send(body: any, installations: any[], pushStatus: any): ?Promise<*> {}

[flovilmart]

I plan to change he way the state is changed from sending to sent in the next release. Perhaps aggressively right after all batches are scheduled. Then, you can check number sent that should properly reflect the counts.

[felipeandradebezerra]

@bjorno You can downgrade your Parse-server to version 2.3.1 and it will work except for a high volume of push messages (80k installations seems to work fine). Please, check the change log for more info.

You can downgrade your Parse-server by running the code:
$ npm install parse-server@2.3.1

@flovilmart I'm running my own version of the parse-server-example. To give you an example of what I've done you can check the forked version where I've updated the parse-server-example to include a config file in json format and a way to run multiple instances of Parse-Server on different ports.

Regarding the push issue, I think the setup function is not being called on the adapter.

[flovilmart]

I really discourage running multiple parse-server on different ports unless you start multiple node process. the Parse SDK will only be initialized with the last provided AppId.

[felipeandradebezerra]

@flovilmart It's running really fine for more than 20 apps! ;-)

In the parse-server-push-adapter@1.1.0 you were handling all connection callbacks in the APNS.js but in the newest version I can see a huge difference. I will take some time this weekend to check line by line. Unfortunately I did not get a way to debug it yet.

this.conns = [];
...
for (let apnsArgs of apnsArgsList) {
 let conn = new apn.Connection(apnsArgs);
 ...
 // Set apns client callbacks
 conn.on('connected', () => {
   log.verbose(LOG_PREFIX, 'APNS Connection %d Connected', conn.index);
 });
...
 this.conns.push(conn);
}

[flovilmart]

As long as you don't use cloud code......... we know it can somehow look like it work, but it's really dangerous, you may end up saving data in the wrong app, and that I know it for a fact, as a large rejected PR shows

[bjorno]

@felipemobile That command does not seem to work, maybe because I installed the parse-server by downloading the entire parse-server-example zip file, and then pushed the entire thing to heroku. On my dashboard it just says server version 2.3.7.

[felipeandradebezerra]

@bjorno Maybe you should install by using -g flag, so it will install the package into the global node_modules folder.

$ npm install -g parse-server@2.3.1

You can try to clear the cache or remove the folder (rm -rf node_modules):

$ npm cache clean

[bjorno]

@felipemobile Thanks for the suggestions! I managed to fix the problem when I realised that the real issue was that my .p12 cert had expired, and I never uploaded the renewed one. I appreciate you taking the time to help me though!

[felipeandradebezerra]

@flovilmart running one instance of Parse-server in a single node process seems to solve the push issue however it has a downside when running multiple node process (20x) in a single amazon EC2 t2.micro instance.

Can you give me an advice? Have you ever implemented a similar architecture?

[flovilmart]

You should probably scale up this instance, 20 process would require about 4Gb of RAM+. We're running on kubernetes on multiple clusters and AZ's so we have quite a different setup, with different costs too

[felipeandradebezerra]

@flovilmart I scale up to a t2.medium instance and it works however I've found something weird - Pushes are sent to all installations even when I define an audience.

I've created a shell script that reads the config file and run 'forever' for each instance then I will be able to start or stop each app individually.

[felipeandradebezerra]

@flovilmart Android messages are not counting towards the total so the status is stuck in SENDING state.

[flovilmart]

uhh that's odd, need to check on the adapter why it doesn't count.

[felipeandradebezerra]

@flovilmart Push Notifications stuck in SENDING state when an installation's deviceToken is set to null. I'm not sure if all push messages after this row are sent or if this bug is happening due to an unhandled exception:

parse-server/src/RestWrite.js

Line 649 in 2533a8c

throw new Parse.Error(135,

I've found some old installations in my _Installation collection without deviceToken and pushType values (March/2015) for both Android and iOS devices.

PS 1: the STATUS changes to SENT when I delete these buggy rows.
PS 2: the push subscriber count shows the right value (1).

[flovilmart]

We can force exclusion in the initial query so we don't count them nor fail

[felipeandradebezerra]

Nice! I've confirmed that all other push messages are sent and status changes to SENT after deleting all rows with null value set on deviceToken column.

@flovilmart What do you think if we force the exclusion here?

[felipeandradebezerra]

Something like this:
updateWhere["deviceToken"] = {"$exists":true}

[flovilmart]

That would be on the where directly, if not set :) updateWhere is only used for the badge updates.
With the according tests obviously :)

[felipeandradebezerra]

Right, I need a little sleep.

Maybe this code below should help here. If you agree I can try to write the according tests.

var hasDeviceToken = where.hasOwnProperty('deviceToken');
if (!hasDeviceToken) {
  where["deviceToken"] = {"$exists":true}
}

[flovilmart]

Yep, that should do, only if the property is not set :)

[felipeandradebezerra]

@flovilmart the push stuck in sending state only if the property is not set :)

[flovilmart]

Yep!

[felipeandradebezerra]

I hope the problem will be solved with this pull request :)
#3764

[cinder92]

is this fixed? i just have same issue, trying sending ios and android push notifications, not sendind to anyone 😢

[flovilmart]

@cinder92 did you check your configuration? can you provide logs? This issue, involving _PushStatus stuck in sending state would actually get stuck AFTER sending and delivering the pushes.

[myusuf3]

@flovilmart I am in the state as @cinder92 I recently added android and the issue of thing being stuck in send state for several users.

It was working fine for a couple of days then broke for a certain chunk of users, possibly users who also use android as well. Ideas on what the issue is?

my configuration was working now it just seems to have stopped.

[LassassinX]

I can confirm that pushes stuck at the sending state never reach the device.