RFC: Basic NOBIL implementation #363
Conversation
robho
force-pushed
the
nobil
branch
3 times, most recently
from
80cdd5e to
d0c19e3
Compare
|
Very nice, thanks a lot! I'll try to find some time in the weekend to try it out and provide some help with code cleanup.
Yeah, if there are properties that don't fit into EVMap's data structure we can think about adding new fields or expanding existing ones where it is useful. For payment methods, we have the
As previously mentioned, I guess all the filters would have to be implemented locally, as the Nobil API doesn't really have server-side filtering abilities, right? That should be possible (just like the min_connectors filter for GE and OCM), but in the longer term the better solution is probably to first download all chargers into the local DB (which is a capability I'm implementing for #290) and perform filtering there.
Chargeprice currently only support GoingElectric and OpenChargeMap unfortunately. And by now they have also created their own separate database of charging stations aggregated from different sources, so from what I heard they are not so interested in fixing issues with their adapters for GE and OCM, let alone adding new ones 🫤.
Likely yes, I can test that as well.
True, that is probably a general bug - for GoingElectric it's not so relevant as we are only allowed to cache the data for 24h anyway. But yeah, it probably also applies to OpenChargeMap. Let's track that in a separate issue. |
app/src/main/java/net/vonforst/evmap/api/nobil/NobilAdapters.kt
Outdated
Show resolved
Hide resolved
|
Regarding filtering..
Is your idea to replace the on-demand-loading with a full data fetch or are you thinking of having some combination of the two? The nobil data is ~130 Mbytes today (up ~6 Mbytes in ~2 months). This is uncompressed data, I haven't checked if the server can compresss data and what the compressed size is. I think I would prefer to use on-demand loading most of the time, but it would be valuable to have the possibility to cache all data when needed (no mobile connection or high mobile roaming prices, ...). Is the problem that you see with local filters that they need to fetch data for chargers that are uninteresting? Ie unnecessary network traffic and data processing? Are those "unnecessary" downloads cached even if they are not used? |
Gzip usually helps quite a lot with JSON data, and the Nobil server does seem to support it. According to the download size is pretty small at just ~5 MB. Of course it might occupy a bit more space in the local database (and from my experience with the implementation for OSM, parsing the JSON and inserting chargers into the DB also takes much more time than the download itself).
In general, EVMap will continue to support both. So we could offer both options for Nobil. On the other hand, with such a small download size, there's not much of a downside to storing it all locally - might also reduce the load on Nobil's servers in the long term.
Yeah. I think the data processing (JSON parsing into Kotlin objects, then filtering) is usually the main bottleneck when loading a large map region with thousands or even ten-thousands of chargers - server-side filtering and even server-side clustering makes it a lot faster for GoingElectric where that is supported. Also, depending on what filters we implement, there might be some filters where we can only determine the available options by iterating through the whole dataset once (e.g., the network filter mentioned in the comments) and then storing that as
not at the moment, because local filtering happens inside the API implementation |
|
I rebased to latest master. Some comments on this implementation and nobil..
|
|
Hi, sorry for the late reply!
|
nobil has data on biogas and hydrogen "chargers", but we don't want them in EVMap
There's a web page for Swedish chargers, but we need to send email for the other countries.
Remove duplicated url
.. and populate it in nobil data source
I think this is more in line with how other non-existent parameters are handled. Also added some code to merge evseIds when merging Chargepoints.
Nobil has no suitable sites to individual charging stations so url needs to be optional and then we use dataSourceUrl instead in "data source button".
URLEncoder.encode() encodes ' ' as '+' and that's not handled by all email apps. Uri.encode() uses "%20" for ' '.
In the charger details screen, set the share menu item's visibility in the same way as the favorite menu item's state is set.
All valid nobil images are [0-9]+.(jpg|png), all other file name variants should be discarded (such as "coming soon").
In nobil there's a connector type named "Type 2 + Schuko". If this means that both a type 2 connector and a schuko connector is available I think it makes more sense to parse this like a type 2 connector rather than a schuko connector since most people want type 2 connectors. A new EVMap connector type could be created, but it seems to add little value since "Type 2 + Schuko" is only used by a single charging station in nobil, ie the use would be extremely limited.
Copy-n-paste implementation that needs to be cleaned up. Nobil data for Tesla is outdated so locations may not have all the chargepoints that the Tesla API returns data for -> matching fails.
Doesn't quite work. When doing a full download there's a NetworkOnMainThreadException when moshi needs more bytes to parse.
| @GET("datadump.php") | ||
| suspend fun getAllChargingStations( | ||
| @Query("apikey") apikey: String, | ||
| @Query("countrycode") countrycode: String = "DAN", |
There was a problem hiding this comment.
This line should be removed. It's only there to bypass a NetworkOnMainThreadException I can't figure out.
|
I made an attempt to implement "full download" for nobil. It's based on and borrows lots of code from the OSM implementation. It almost works.. When downloading "many" chargers I get a NetworkOnMainThreadException after parsing ~2000 chargers. I haven't been able to figure out what the problem is. Is there a problem in my implementation or is there a problem in moshi/retrofit/okhttpclient? If someone has any ideas or could point me in some direction I'd appreciate it! From the backtrace I get it looks like moshi is happily parsing data, but then the input buffer is exhausted so more data needs to be fetched from the nobil server and this happens on the main thread: Compared to the OSM implementation (which works great), the only difference I can think of is that the nobil data consists of more and longer data structures which would need more buffer space to hold data for a complete chargepoint. |
This adds a NOBIL data source. Data is available in Denmark, Finland, Iceland, Norway and Sweden and the data is collected both from operators via OCPI and contributed by the public.
This is a draft implementation where some functionality is missing. I'm interested in hearing if this is of interest, and if so, what should I fix/work on going forward?
Known problems / missing features: