UnifiedPush (ntfy) not working

[MrRulf]

Steps to reproduce

1. Set up/start Element from f-droid
2. Set notification method to unifiedPush (ntfy in my case)
3. Close Element
4. Ask somebody to write u a message
5. Check if there is a notification (in my case, none)

My guess is that the problem is on Elements side because I'm only having this issue since the last update (1.4.34)

Outcome

What did you expect?

To get notifications

What happened instead?

I get no notifications

Your phone model

Fairphone 4

Operating system version

Android 12

Application version and app store

Element - Secure Messenger version 1.4.34 from f-droid

Homeserver

matrix.org

Will you send logs?

No

Are you willing to provide a PR?

No

[MrRulf]

Forgot to add, using the background sync of Element still works.

[MrRulf]

If somebody explains me how I can send useful logs I'm totally willing to send them, I just picked No because I think they won't help much, since the notification isn't broken or anything, just delayed until I open the app.
About the PR, if I can help in any way, same as with the logs, I just don't think I'd be able to find the error in the code and fix it, why I picked No to give no wrong expectations.

[blacklight]

I can confirm the issue. ntfy push notifications used to work before the latest update, they stopped working after updating, and background sync is still working.

Are these things properly tested before pushing out a new stable release?

[p1gp1g]

176.126.240.158 got banned by ntfy.sh after too many requests, and looks like a matrix.org IP (https://github.com/matrix-org/synapse/issues?q=is%3Aissue+176.126.240.158).

Is there a list of matrix.org IP to whitelist them on ntfy.sh ?

[p1gp1g]

We need the ones for push requests (outgoing connections)

[blacklight]

Hmm this is weird. I use my own ntfy server for push notifications, and that's what my Android app is registered to. Does the IP blacklist on ntfy.sh impact also personal servers?

[p1gp1g]

It is possible: check the doc https://ntfy.sh/docs/config/#rate-limiting

[binwiederhier]

This should be resolved now. I whitelisted the matrix.org IP address using this binwiederhier/ntfy@ed4cc86. Thanks @p1gp1g for the pointers.

[p1gp1g]

If you want to do the same setup than ntfy.sh:

/etc/ntfy/server.yml

visitor-request-limit-exempt-hosts: "176.126.240.158"

/etc/nginx/nginx.conf

        geo $limited {
                default 1;
                176.126.240.158/32 0;
        }
        map $limited $limitkey {
                1 $binary_remote_addr;
                0 "";
        }
        limit_req_zone $limitkey zone=one:10m rate=1r/s;

You should add ipv6 too your server supports it (2a00:1098:84:1c8::158)

Please not these IPs were discovered with the requests from the matrix.org server

[huguesdk]

i have the same problem. i’m using my own ntfy server. could it be a problem related to rate limiting when fluffychat notifications (that use the same ntfy server) continue to work?

i’ve noticed quite slow sync times recently. i don’t know whether it’s a client or server problem, though. i run my own synapse homeserver.

to be sure to get notifications on android, i use element, fluffychat and schildichat at the same time. i usually get 1 or 2 notifications, and not always from the same clients. it’s sadly funny.

people trying to use element instead of whatsapp or signal keep complaining to me that they often don’t receive notifications. would it be possible to have a safety measure that would ensure to at least perform a sync every hour or so, to catch messages for which notifications failed? i know android is quite aggressive regarding battery optimizations, but maybe it’s nevertheless possible?

[binwiederhier]

@huguesdk I recommend enabling debug (or trace mode) in ntfy for a while to see where the issue is. It's really verbose so you should be able to spot a problem if there is one in ntfy. Don't forget to turn it off though, because otherwise your logs will fill up.

[kojid0]

Any news on this? Element is not an instant messenger anymore... This should get maximum priority.

[bahur142]

I've never had any problems since I get using ntfy. I self host both - synapse and ntfy. Using schildichat fork instead element. Non of my friends complains about notifications.
Are you sure your ntfy app shows silent notification "Listening for incoming notifications"? This should be enabled. Because if this silent notification is suppressed, you could miss some notifications on element. Also you should set ntfy app in android settings "battery optimization" to "Not optimized" to ensure the app will stay awake and not killed by the os.

[DeepBlueBubble]

Mine is also not working.
I am using the FCM distributor one. When the App is in the background the notification will never arrive. It can receive when in the foreground.
However when I switch to the Nextpush it works nomral.

[dessalines]

Same, I just switched from google's fcm to ntfy. The element subscription shows up in ntfy android, but no notifications come through until I manually open element.

[binwiederhier]

For anyone using the ntfy.sh server and not a selfhosted instance: I am seeing lots and lots of rate limited (aka rejected) requests from various matrix instances. There are very few whitelisted Matrix servers (matrix.org being one of them), so it may very well be the case that you are just a victim of rate limiting on the ntfy.sh server.

This only affects the public ntfy.sh instance. If you are selfhosting ntfy, i suggest you turn on debugging/tracing and look at the logs.

(I am the ntfy maintainer)

[dessalines]

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says ntfy.sh/[MY_ID_HERE] .... im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

[binwiederhier]

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

You can enable logging in ntfy, and see if ntfy receives and forwards the notifications. Go to Settings -> Record logs, and then after you should have received a notification, you can Copy or Upload the logs. That should tell you more.

[MelBourbon]

Since a few days ntfy method is also not working anymore with Android 13 (graphene OS). Google Service also not working. So currently the only way to get reliable notifications for Element is using background sync. Other users of my server have the same issue.

[dessalines]

I tried this again, and can verify its still broken. @casta is correct that:

After digging a bit more, it seems that the notification is broken as soon as Element is not running.

While element is in the foreground / not closed, I get ntfy notifs. As soon as its killed in the background, the notifs stop.

If the ntfy is wired to "wake up" element when it receives a notif, its not doing so.

[ismay]

I've been encountering this problem as well. Strange enough after some fiddling around it's working now. I'm not sure why it's working now, but I get the idea that it wasn't working because I also had other element instances signed in to the same server on other devices.

It's a long shot, and I'll keep an eye on it, but it seemed like notifications for element on my phone with ntfy (without element running in the background) started working again when I closed everything else that was signed into the same account on other devices. Also, it doesn't seem to be ntfy specific, because I was observing the same failure to notify on mobile when element was connecting directly to the server via polling. So it seems to be an element bug, not ntfy.

[FrederikSchack]

I still have issues with notifications and calling, running on Lineage OS for MicroG with NTFY. My first experience with Lineage OS was good in January and a bit into february, calling could be slow, but it worked. Since then I could rarely get a call through and messages can be very slow at arriving, like several minutes. Sometimes I see a notification about a message and I can see the first bit of the message, but I can't see the message inside Element and it can take yet more minutes before the message appears inside Element.

I also have Element installed on a Windows PC.

[FrederikSchack]

"Matrix largest home servers are incredibly slow to use":
https://www.youtube.com/watch?v=aVwl892hqb4

[kinslayer1982]

Noticed the same problem with Element and ntfy.

I'm running a matrix server (synapse), ntfy server and fmd server behind a nginx reverse proxy. Element, ntfy and fmd from F-Droid on a Galaxy S21 FE (Android 13).
After setting up ntfy I got notifications from Element for a short time, then they stopped working while for fmd they were still working. Now that I've re-registered ntfy with Element again, they started working again. I will observe if they keep working or stop again.

Edit: 4 days now and still seems to be working. Don't know why notifications in Element stopped working while others didn't.

[cal-andrew]

It started working for me when I disabled battery optimisations for Element, as well as ntfy

I am using the F-Droid version of Element on CalyxOS with microg.
The Google Play version of Element could also get push notifications through microg, without disabling battery optimisations.

microg, being a system app, may have permissions to wake up apps with battery optimisations, while ntfy and other user-installed apps don't.
If so, custom ROMs could fix this by including ntfy (or some other UP distributor) by default.

[luk6173]

Have same issue with ntfy. Running Element F-Droid Version (latest), also latest NTFY on my Android Smartphone and on my Server.

Is there any fix planned next time ?

[p1gp1g]

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

[luk6173]

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

@p1gp1g adding my NTFY IP from my Server or from my Smartphone (Android App) ?

[mpeter50]

It seems this issue has reports about a lot of issues for which the cause cant be determined. Matrix notifications through Ntfy are not working for me either, but I'm sure that the Ntfy app itself is working fine, below I'll tell you why.

I think I have found a few ways to test whether a few very specific parts of the notification pipeline are working or not.
An important tool in these tests is the Ntfy webapp: https://ntfy.sh/app
This is hosted by binwiederhier (lets hope that if I dont use an @ sign you wont get unnecessarily mentioned), who is the developer of Ntfy, and who runs the main Ntfy instance that is used by the matrix.org homeserver, so this webapp is safe to use.

I have written the below guide

• because Ntfy Matrix notifications do not work for me either, other than the test notification from the troubleshoot menu, and I think I have never seen them working
• and so that the devs are seeing my testing methodology, so they can have a better idea on what is failing if others follow it too to tell their findings

It got to be a bit long, so instead of dumping it here you can read it in this gist: https://gist.github.com/mpeter50/9220dac1056c9a66c313d049838c1ab2

[scottwallacesh]

I think this is "simply" a problem of Element/Schildichat not waking when ntfy receives the push notification.

If you disable battery optimisation on Element/SC and allow it to run in the background, everything works as you would expect.

[mpeter50]

I have it disabled for SC already. Also, I didn't receive the notification in the Ntfy webapp either, when I was watching the topic of the push key while sending messages to my account from an other one.

[p1gp1g]

@luk6173 if you run your own synapse server, you have to whitelist your ntfy ip address: https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#ip_range_whitelist

@p1gp1g adding my NTFY IP from my Server or from my Smartphone (Android App) ?

This should be your ntfy IP as seen by your synapse server. If you look at your synapse logs

[p1gp1g]

For helping many users, especially for users self-hosting their services, the problem is most of the time a not working configuration.

To troubleshoot UnifiedPush for self-hosters:

First, if you self-host your homeserver and your push server: there is a high probability you have not set the ip_range_whitelist. Check your synapse's logs.

Then, you should run the notifications troubleshooter feature on element.

If you don't receive the notification, the problem comes from your push provider:

• If you use ntfy, double check you have correctly set the authorizations: UnifiedPush topics (they all start with “up”) world-writeable (https://unifiedpush.org/users/distributors/ntfy/#limit-access-to-some-users-acl) *
• Test UP-Example.

If you receive the test notification:

• Have you check the ip_range_whitelist ? Once again, this is a very common error *
• Check your synapse logs
• Make sure your synapse server can reach your push server: open a shell on your synapse server, or container if you use it, and try to send a request with curl curl https://mypushdomain.tld/foo *
• Make sure you have disable battery optimization for your distributor and Element.
• Check https://dontkillmyapp.com/ and follow the suggested steps if needed

* Most common issues

[luk6173]

This should be your ntfy IP as seen by your synapse server. If you look at your synapse logs

I assume then I have to whitelist 0.0.0.0 because my ntfy IP from my Android Smartphone App changes often, but should not be a problem because of user based access

[p1gp1g]

Your ntfy server IP as seen by your synapse server. It may be a private IP depending on your setup. You should check the logs

[luk6173]

Your ntfy server IP as seen by your synapse server. It may be a private IP depending on your setup. You should check the logs

thank you for your clarification, I go now and setup it

[luk6173]

For helping many users, especially for users self-hosting their services, the problem is most of the time a not working configuration.

Thank you very much this helped me to get it running and to understand how it works!

[p1gp1g]

You're welcome

[Saturn745]

I'm facing this same issue but I am not self hosting my home server or ntfy server. Notifications are only coming through once I open Element. I've allowed background usage in settings for both ntfy and Element and other ntfy topics I get notifications just fine.

[Akselmo]

For helping many users, especially for users self-hosting their services, the problem is most of the time a not working configuration.

To troubleshoot UnifiedPush for self-hosters:

First, if you self-host your homeserver and your push server: there is a high probability you have not set the ip_range_whitelist. Check your synapse's logs.

Then, you should run the notifications troubleshooter feature on element.

If you don't receive the notification, the problem comes from your push provider:

* If you use ntfy, double check you have correctly set the authorizations: UnifiedPush topics (they all start with “up”) world-writeable (https://unifiedpush.org/users/distributors/ntfy/#limit-access-to-some-users-acl) *

* Test [UP-Example](https://f-droid.org/en/packages/org.unifiedpush.example/).

If you receive the test notification:

* Have you check the ip_range_whitelist ? Once again, this is a very common error *

* Check your synapse logs

* Make sure your synapse server can reach your push server: open a shell on your synapse server, or container if you use it, and try to send a request with curl `curl https://mypushdomain.tld/foo` *

* Make sure you have disable battery optimization for your distributor and Element.

* Check https://dontkillmyapp.com/ and follow the suggested steps if needed

• Most common issues

Oh thank you so much.

To add, my problem was that in ntfy server.yaml, my baseurl started http instead of https. Setting it to https made everything work.

Phew!

[rei-zero246]

This still doesn't work, correct?

[FrederikSchack]

I didn't selfhost and gave up using Matrix/Element because of this bug.

[mpeter50]

Not with the matrix.org homeserver. With an other one it does, at least when using the SchildiChat app, which is a fork of Element Android. As I recall Element X also got support for it a few months ago, so if your phone is supported you can try it out, but when it does not work thats mostly not because of the app, but because of server misconfiguration.

[teohhanhui]

FWIW actual users can tell you it's also a problem when using matrix.org home server. Notifications do not work when using ntfy, even though the test reveals no issues.

[mpeter50]

Thats exactly what I have meant. If you choose an other one, one that is even faster than matrix.org, it should work.

I can imagine that if the misconfiguration was by accident, maybe they are intentionally keeping it that way. They have said time and time again that the matrix.org server is too huge, costs a lot of resources to host, and everyone using it goes against decentralization. Maybe they keep this as a reason to let users choose an other one.

Personally I have stopped using my matrix.org account and registered a new one at a friendly local homeserver. While obviously this alone does not solve everything, its been much better since then.