Data sources

geokrety.org

There are many available methods to obtain informations from geokrety.org. But none of them permit:

• full synchronization with all available data
• incremental synchronization with all available data
• partial syn are limited to 10 days
• not sure if export methods show updated data (description, log, logtype...)
• export2.php doesn't provide just created Geokrety

Those tables show which information is available per method.

The most effective method of synchronization in term of data completeness seems to be a combination of page crawling coupled with csv moves exports. Does this imply more load on geokrety.org server than using export.php?

Standard

field	page crawling	export2.php	export.php	export_oc.php	GPX	csv
gkid	X	X	X	X	X	X
name	X	X	X	X
waypoint	X	X	X	X	X	X
missing	X		X
distancetravelled	X	X	X	X
state	X	X	X	X
image avatar uri	X	X
last_pos_id	?	X
last_log_id	X	X	X			X

Details

field	parge parsing	export2.php	export.php	export_oc.php	GPX	csv
gkid	X	X	X	X	X
description	X		X
owner id	X	X	X
owner name	X		X
datecreated			X
places	X
position latitude		X	X	X	X	X
position longitude		X	X	X	X	X
type id	X	X	X
type name	X		X
image title	X
image uri	X
image avatar title	X
votes	X

Moves

field	parge parsing	export2.php	export.php	export_oc.php	GPX	csv
ALL LOGS	X
UPDATES LOGS
id	X		X			X
position latitude			X		X	X
position longitude			X		X	X
wpt	X		X		X	X
date moved	X		X		X	X
date logged			X
user id	X		X
user name	X		X
application using (api)	X
application name	X
message	X		X			X
logtype id	X		X			X
logtype name	X		X
distancetravelled unit	X
distancetravelled value	X					X
country	X				X	X

Moves comments

field	parge parsing	export2.php	export.php	export_oc.php	GPX	csv
comment	X
type	X

Synchronization strategy

Master

1. Get last update date from geokrety-details or 10 days
2. Poll @5min geokrety.org/export2.php?modifiedsince={lastupdate}
3. Save in pending-geokrety
4. Get @1min a chunk of pending gk
5. Crawl each page at geokrety.org?konkret={gkid} + gpx.xml
6. Merge data
7. Save as individual files /export/{gkid}.xml
8. Add date and ownername to the current gk from pending-geokrety
9. Merge pending-geokrety-details into geokrety-details
10. Save in geokrety-details
11. Merge pending-geokrety into geokrety
12. Save in geokrety

Slave

1. Get last update date from geokrety-details or ??? days
2. Poll @5min api.geokretymap.org/gk/details/${lastupdate}
3. Save in pending-geokrety-details
4. Convert to simple format and save in pending-geokrety
5. Merge pending-geokrety-details into geokrety-details
6. Save in geokrety-details
7. Merge pending-geokrety into geokrety
8. Save in geokrety