v1.29.0 Tesla Fleet API via MyTeslaMate stopped working - unsure if local TeslaMate or MyTeslaMate proxy issue

[lits101]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Yesterday I configured a new local docker-compose instance of TeslaMate 1.29.0, using the FleetAPI available via the free tier of MyTeslaMate. This worked perfectly for a few hours, then while I was out driving it seemed to just stop. I'm getting some non-descript errors relating to the API: unknown, offline, Vehicle_1_api_error, etc...

I've tried refeshing my tokens via MyTeslaMate, reconfigured my local TeslaMate with the new token code, to no avail.

Initially upon failure, my local TeslaMate instance was just showing a failed health check for the car and not reporting its position or any telemetry. After leaving it running for a few hours, it seems to be reporting the cars location at the time of API failure yesterday (this is incorrect, obviously since the car is home now) and it's occasionally grabbing its asleep state and time correctly. All other telemetry is stuck at the point of failure yesterday (SoC, tyre pressure, heading, etc).

I'm a bit lost as to what to do. I've tried creating a new local docker instance of TeslaMate on another host, and I'm having the exact same issue. So maybe it's not anything I've done?

Expected Behavior

No response

Steps To Reproduce

No response

Relevant log output

2024-05-30 08:46:38.853 [notice]     :alarm_handler: {:clear, TeslaMate.Vehicles.Vehicle_1_api_error}
2024-05-30 08:40:28.887 [info] GET https://api.myteslamate.com/api/1/vehicles/xxxx/vehicle_data?token=xxxx -> 429 (6293.800 ms)
2024-05-30 08:40:28.887 [warning] TeslaApi.Error / ""
2024-05-30 08:40:28.888 car_id=1 [error] Error / :unknown
2024-05-30 08:41:11.593 car_id=1 [info] Start / :offline
2024-05-30 08:43:30.386 [info] GET /
2024-05-30 08:43:30.404 [info] Sent 200 in 17ms
2024-05-30 08:43:39.858 [info] GET /settings
2024-05-30 08:43:39.885 [info] Sent 200 in 27ms
2024-05-30 08:46:38.853 [notice]     :alarm_handler: {:clear, TeslaMate.Vehicles.Vehicle_1_api_error}

Screenshots

Additional data

No response

Type of installation

Docker

Version

v1.29.0

[lits101]

The MyTeslaMate API Proxy appears to be communicating with my car correctly. It can see when it's online. I've double-checked and I've given it the appropriate access via the Tesla account page.

[jlestel]

The MyTeslaMate API Proxy appears to be communicating with my car correctly. It can see when it's online. I've double-checked and I've given it the appropriate access via the Tesla account page.

Hi MyTeslaMate API owner here, this is a 429 response (Too many requests) on the vehicle_data from Tesla API. It seems there's less limitation on the vehicle endpoint you mention after.

Now my tests have led me to the same result as you.

The official API mentions a limit of 1 call every 5 minutes: that's actually 300 calls per day on vehicle_data then gets a 429 response with a retry-after header of 86400 seconds.

With TeslaMate, I think that when the vehicle is driving there are API calls every 15 seconds => maximum 75 minutes of drives and there's also hits when the vehicle is awake.

Yesterday, I just saw that Fleet streaming also brings me events with a later date, which causes "Discarded stale fetch result:" every second until I arrive at the date of the event from the streaming. Did you see "Discarded stale fetch result" on your side?

I am preparing a PR to be able to modify these default timers in order to drive by relying more on the streaming API than on the Fleet API.

In the meantime, I just modified CloudFlare to cache these error and certain hits, that should improve the experience.

You should keep an instance on the old API to be sure you have the data and/or continue to help me test it live. Thanks for testing!

[brianmay]

Ideally, teslamate should support parsing the retry-after header, print it to the logs, and wait until then before polling again.

Although 86400 seconds == 24 hours - if that is happening is somewhat extreme and unusable.

[jlestel]

Better not to exceed the limit because it is harsh.

[lits101]

The MyTeslaMate API Proxy appears to be communicating with my car correctly. It can see when it's online. I've double-checked and I've given it the appropriate access via the Tesla account page.

Hi MyTeslaMate API owner here, this is a 429 response (Too many requests) on the vehicle_data from Tesla API. It seems there's less limitation on the vehicle endpoint you mention after.

Now my tests have led me to the same result as you.

Ok cool, so it's a known issue / limitation of the API. Glad I'm not going crazy!

Yesterday, I just saw that Fleet streaming also brings me events with a later date, which causes "Discarded stale fetch result:" every second until I arrive at the date of the event from the streaming. Did you see "Discarded stale fetch result" on your side?

I don't think I've seen a 'stale fetch result' error. My log is just full of 429 responses as far back as I can scroll. I'll keep an eye out for it though!

You should keep an instance on the old API to be sure you have the data and/or continue to help me test it live. Thanks for testing!

I'm happy to keep helping you test. I stopped logging on the old API because I read somewhere that continuing to use it may result in me getting blocked. I use the Fleet API in EVCC and through my energy retailer for cheap charging of the car, so I didn't want to risk that getting blocked as well.

If you think it's safe to use the old owner API concurrently with the new Fleet API, then I'll spin up another instance of TM.

[guldil]

Hello,

I have same probleme today, i lost connection 3hours ago and now my log is full of "429" [warning] TeslaApi.Error / "Retry in 62875 seconds\n". I just stop the TeslaMate service then, have to wait 17hours ;-)

[brianmay]

I got this just today too. Hoping it might go away, but a bit skeptical. This is using the older home owner API.

With this error, I think you really need to be using the latest master which has support for the 429 messages. Otherwise there is the risk by continuously polling the API the time might not go down.

You should be able to check this, does the number of seconds keep going down?

[brianmay]

If this doesn't come good, we might have to investigate #3940 some more, to reduce the polling intervals. But I think this is going to be painful :-(

[jlestel]

You should be able to check this, does the number of seconds keep going down?

I can confirm that the Retry-After is decreasing, users are also seeing it now. Frankly, when I see the pressure that Teslamate can currently put on the API owner, I understand better why Tesla is introducing limits.

[guldil]

i moved to master version and it's back online. But i don't know i have to stay on "master" version...

[brianmay]

There doesn't seem to be any problem today. Which I guess is normal, people have reported 429 errors before, and then they just go away.

I am really doubtful that Tesla has even noticed TeslaMate. These servers would receive heaps of requests just from the app alone. I think what TeslaMate generates would be tiny in comparison. The number of TeslaMate installations compared with the number of sold cars I think is extremely small.

And the rate limit is going to increase the load, not decrease it, because current versions of software will retry when an error is detected. Yes, we should fix this for Teslamate (and have fixed this in master), but I imagine there is a lot of other software.

And rate limiting everything for hours is really extreme if it was just high traffic. I think they may be gearing up to start charging for premium access, which isn't rate limited. By making the free offering basically unusable.

It is also worth noting that the new 429 processing in master appeared to work fine.

[jlestel]

I agree with you.

I don't know if this is for the premium offer or to prepare for the discontinuation of the Owner API but it seems to announce changes.

[JakobLichterfeld]

Do not see the issue still persist nor TelsaMate can change Tesla API behaviour