Skip to content

Commit 034ac5f

Browse files
jeffdefactojeffdefacto
authored
Disputed feature documentation (#2085)
This adds scripts and documentation needed to utilize our disputed border work.
1 parent afcf5c9 commit 034ac5f

8 files changed

+242280
-0
lines changed

‎docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ This repository contains documentation for Mapzen's Vector Tile service.
88
* [display-tiles.md](display-tiles.md) - Display vector tiles in a map in Tangram, MapboxGL, D3, and other frameworks.
99
* [layers.md](layers.md) - Layers in the vector tile service detailed incuding available `kind` values and other tags.
1010
* [data-sources.md](data-sources.md) - Where does Mapzen's data come from?
11+
* [disputed-features.md](disputed-features.md) - How we handle dynamic viewpoints for disputed features.
1112
* [http-status-codes.md](http-status-codes.md) - HTTP status codes.
1213
* [attribution.md](attribution.md) - Giving credit where credit is due, and the data licence requires it.
1314

‎docs/data-sources.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ With its coverage of roads, POIs, landuse, as well as rich metadata, OpenStreetM
2121

2222
All OpenStreetMap data is licensed under the [ODbL](http://opendatacommons.org/licenses/odbl/), a [share-alike](https://en.wikipedia.org/wiki/Share-alike) license which also requires attribution.
2323

24+
Tilezen uses [Daylight Distribution](https://daylightmap.org/) which combines OpenStreetMap data validated with quality and consistency checks with additional opensource datasets.
25+
26+
Dynamic viewpoint processing of OSM data is described in [disputed-features.md](disputed-features.md).
27+
2428
## osmdata.openstreetmap.de
2529

2630
`source:osmdata.openstreetmap.de`

‎docs/disputed-features.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
## Disputed borders
2+
3+
We use [Daylight Distribution](https://daylightmap.org/) over straight OpenStreetMap data for our data source. Daylight has quality control checks applied to it that limits the possibility of major vandalism getting onto the map. Additionally, Daylight provides optional 'sidecars' containing data such as ML based roads and buildings and other open source data sets. One potential drawback when considering Daylight is that the data will always be a month or so behind live OSM.
4+
5+
Daylight does not currently include the disputed borders that we need to render on the map. Instead, we download and patch them onto the pbf ourselves. This approach offers some advantages, as we can iterate on the borders as necessary without relying on Daylight to update, however, the mismatch between Daylight and live OSM can require some additional local edits to render properly.
6+
7+
We have a two-step process to patching in these disputes. First, we download and assemble a `.osm` file that contains all the dispute relations we are interested in. Then we convert and apply this file to the Daylight pbf using a script that assembles all the sidecar files into a final pbf.
8+
9+
10+
### Building a dispute patch
11+
1. When Daylight has a new release, it is best to start with a fresh disputed relation package. Using the overpass query in `disputed_relation_overpass_query.txt` run an overpass query to get the data.
12+
2. Click export and select download as raw osm data
13+
3. Open the file in JOSM and compare this file with the previous version. Features may have been altered or deleted since the last build and updates may be necessary. If there are fixes necessary, make and upload them.
14+
4. Some edits may need to be made locally on the patch without committing to OSM itself, such as when daylight borders are out of alignment with live OSM. These edits can be done in JOSM and the file then just saved without uploading. The reverter plugin is handy for realigning borders that have been moved.
15+
* Note: This will only work for modifying or deleting existing features. If features are created, they must be uploaded to OSM for osmium to work correctly.
16+
5. If you need to update features within the patch it is safest to select the relation or way then run update selection. This will limit the chances of unintentional changes being introduced to the .osm file once it’s been initially downloaded.
17+
6. The dispute patch may be used to bring in other features or updates as needed, such as missing labels. Just download the feature into the layer and save.
18+
7. Save the new patch file as something like `filtered_disputes.1.10.osm` using the desired Daylight version and upload to the [`scripts`](https://github.com/tilezen/vector-datasource/tree/master/scripts) folder for tracking.
19+
20+
### Patching Daylight
21+
1. Run osmium cat to convert the .osm file to a pbf. The script expects the name to be `filtered_disputes.pbf`. [filtered_disputes_example.osm](https://github.com/tilezen/vector-datasource/tree/master/scripts/filtered_disputes_example.osm) can be used to try out the process.
22+
* `osmium cat filtered_disputes.1.10.osm -o filtered_disputes.pbf --overwrite`
23+
2. Scp the file to the server
24+
* `scp -i tilezen.pem filtered_disputes.pbf ubuntu@1.2.3.4:/mnt/username/`
25+
3. ssh onto the server and navigate to /mnt/username/
26+
4. Run [`parallel_create_planet_file.sh`](https://github.com/tilezen/vector-datasource/tree/master/scripts/parallel_create_planet_file.sh) with a daylight version to build a fresh pbf. This will take some time.
27+
* example usage: `sh parallel_create_planet_file.sh 1.10`
28+
5. Once the `parallel_create_planet_file.sh` has been run on a Daylight version at least once, you can use the [`patch_disputes_into_pbf.sh`](https://github.com/tilezen/vector-datasource/tree/master/scripts/scripts/patch_disputes_into_pbf.sh) script to repatch the disputes if changes need to be made. This script just cuts out some of the early steps to save time.
29+
* example usage: `sh patch_disputes_into_pbf.sh 1.10`
30+
6. If you want to test out disputes in a build, run [`create_disputed_areas_pbf.sh`](https://github.com/tilezen/vector-datasource/tree/master/scripts/create_disputed_areas_pbf.sh) to create a smaller pbf with just the data within disputed areas.
31+
* example usage: `sh create_disputed_areas-pbf.sh 1.10`
32+
33+
34+
## Disputed capitals
35+
36+
Some cities are considered different administrative levels depending on a country's viewpoint. One country may consider a city to be a regional or country capital while a disputant does not. Natural Earth has this information in the `ne_10m_populated_places` table for affected countries in the `FCLASS_XX` columns. Using Wikidata ID tags, the NE `FCLASS_XX` data is joined to the OSM feature to create output kinds of `regional_capital:xx` or `country_capital:xx` with a true or false value to allow for different rendering options.
37+
38+
This should all happen automatically in Tilezen, however, Wikidata ID tag changes in OSM could cause breakages in the join. In this instance, a locally applied patch to Daylight like described in the disputed border section may be needed or an update to NE if the OSM change is valid.

‎scripts/create_disputed_areas_pbf.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# This creates a pbf with just the disputed areas as well as a kashmir specific pbf for testing.
2+
# Run with the daylight version number as the input.
3+
4+
osmium extract -s smart planet-v$1-plus-buildings-admin-filtered-disputes-renumbered.osm.pbf -p collected_disputes.geojson -o $1_disputed_areas.pbf --overwrite -S types=any
5+
md5sum $1_disputed_areas.pbf > $1_disputed_areas.pbf.md5
6+
aws s3 cp --acl public-read $1_disputed_areas.pbf.md5 s3://nextzen-tile-assets-us-east-1/$1_disputed_areas.pbf.md5
7+
aws s3 cp --acl public-read $1_disputed_areas.pbf s3://nextzen-tile-assets-us-east-1/$1_disputed_areas.pbf

‎scripts/disputed_relation_overpass_query.txt

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// Contains the relations needed for a filtered_dispute package.
2+
// Recommended to download fresh package with each new Daylight release,
3+
// then compare with previous package for edits or deletions that may need
4+
// correction or updating before using in a build.
5+
6+
[out:xml][timeout:25];
7+
(
8+
rel(9520114); // Extent of Moroccan Claim at Melilla
9+
rel(9520116); // Extent of Moroccan Claim at Peñon Velez de la Gomera (Badis)
10+
rel(9520181); // Extent of Kenyan claim at Ilemi Triange
11+
rel(9520182); // Extent of South Sudan claim at Abyei
12+
rel(9520183); // Extent of South Sudan claim at Ilemi Triange
13+
rel(9520185); // Extent of Sudanese claim at Abyei
14+
rel(9520224); // Extent of Western Sahara Claim with Morocco
15+
rel(9520245); // Extent of Russian claim with Ukraine at Crimea
16+
rel(9520246); // Extent of Ukranian claim with Russia at Crimea
17+
rel(9575754); // Extent of Chinese Claim - Bara Hoti area
18+
rel(9575755); // Extent of Chinese Claim at Demchok
19+
rel(9575756); // Extent of Chinese Claim at Samdu Valleys
20+
rel(9575757); // Extent of Chinese Claim at Tirpani Valleys
21+
rel(9575758); // Extent of Indian Claim western Kashmir
22+
rel(13559362); // Extent of Indian Claim at Kalapani Territory
23+
rel(13559363); // Extent of Nepalese Claims at Kalapani Territory
24+
rel(13559365); // Extent of Indian Claim at Tirpani Valleys
25+
rel(13559366); // Extent of Indian Claim at Samdu Valleys
26+
rel(13559368); // Extent of Indian Claim at Demchok
27+
rel(13559369); // Extent of Chinese Claim at Aksai Chin
28+
rel(13559370); // Extent of Indian Claim at Aksai Chin
29+
rel(13559371); // Extent of Chinese Claim at Shaksam Valley
30+
rel(13559372); // Extent of Indian Claim at Shaksgam Valley
31+
rel(13559373); // Extent of Chinese Claims to Arunachal Pradesh
32+
rel(13559374); // Extent of Indian Claim at Arunachal Pradesh
33+
rel(13559375); // Extent of Indian Claim at Bara Hotii Valleys
34+
rel(13559376); // Extent of Pakistani Claim at Jammu and Kashmir
35+
rel(13559516); // Line of Control between India and Pakistan
36+
rel(13559521); // Siachen Glacier Disputed Area
37+
rel(13559522); // Extent of Claim at Azad Kashmir
38+
rel(13562280); // Extent of French Claim at Lawa Headwaters
39+
rel(13562281); // Extent of Surinamese Claim at Lawa Headwaters
40+
rel(13562303); // Extent of Surinamese Claim at Courantyne Headwaters
41+
rel(13562304); // Extent of Guyanese Claim at Courantyne Headwaters
42+
rel(13562307); // Line of Control Between Morocco and Western Sahara
43+
rel(13562321); // Cyprus U.N. Buffer Zone
44+
rel(13562322); // Line of Control Between Cyprus and Northern Cyprus
45+
rel(13562333); // Extent of Guyana Claim to Esequibo
46+
rel(13562334); // Extent of Venezuela Claim to Esequibo
47+
rel(13562340); // Extent of Spanish Claim at Olivenza
48+
rel(13562341); // Former Olivenza region disputed between Portugal and Spain
49+
rel(13562391); // Extent of Bhutanese Claim at Doklam
50+
rel(13562392); // Extent of Chinese Claim at Doklam
51+
rel(13562614); // Extent of Serbian Claim at Šarengrad
52+
rel(13562615); // Extent of Croatian Claim at Šarengrad
53+
rel(13562617); // Extent of Serbian Claim at Vukovar
54+
rel(13562618); // Extent of Croatian Claim at Vukovar
55+
rel(13562869); // Extent of Nagorno-Karabakh Claim
56+
rel(13562870); // Nagorno-Karabakh Ceasefire Line 2020
57+
rel(13563019); // Extent of Sudan Claim at Bir Tawil
58+
rel(13563020); // Extent of Egypt Claim at Bir Tawil
59+
rel(13563021); // Extent of Sudan Claim at Halayib Triangle
60+
rel(13563022); // Extent of Egypt Claim at Halayib Triangle
61+
rel(13563042); // Extent of Uruguayan claim at Rincón de Artigas
62+
rel(13563043); // Extent of Brazilian claim at Rincón de Artigas
63+
rel(13563049); // Extent of Russian Claim at Kuril Islands
64+
rel(13563050); // Extent of Japanese Claim at Kuril Islands
65+
rel(13563052); // Extent of Uruguayan Claim at Brazilian Island
66+
rel(13563053); // Extent of Brazilian Claim at Brazilian Island
67+
rel(13563059); // Extent of Somaliland Claim
68+
rel(13563060); // Extent of Transnistria Claim
69+
rel(13563062); // UNDOF Zone
70+
rel(13563066); // Extent of Guantanamo Bay Naval Base
71+
rel(13563067); // Extent of Palestinian Claim at Gaza
72+
rel(13563071); // Line of Control Between Israel and West Bank
73+
rel(13563072); // Extent of Palestinian Claim at West Bank
74+
rel(13563074); // Extent of Spanish Claims at Gibraltor
75+
rel(13563075); // Extent of United Kingdom Claims at Gibraltar
76+
rel(13563077); // Extent of Morocco Claims at Ceuta
77+
rel(13563078); // Extent of Spanish Claims at Ceuta
78+
rel(13563114); // Extent of Spanish Claim at Melilla
79+
rel(13563115); // Extent of Spanish Claim at Peñon Velez de la Gomera (Badis)
80+
rel(13563116); // Military Demarcation Line North Korea - South Korea
81+
rel(13563117); // Korean Demilitarized Zone (North)
82+
rel(13563118); // Korean Demilitarized Zone (South)
83+
rel(13563121); // Extent of Claim at South Ossetia
84+
rel(13573737); // Extent of Chilean Claim at Southern Glaciers Field
85+
rel(13573738); // Extent of Argentinian Claim at Southern Glaciers Field
86+
rel(13573952); // Extent of Claim at Abkhazia
87+
rel(13573966); // Extent of Claim at Donbass
88+
rel(13573969); // Extent of Belize Claim
89+
rel(13573970); // Extent of Guatemalan Claim
90+
rel(13573985); // Extent of Kosovo Claim
91+
rel(13574165); // Extent of Israeli Claim at Golan Heights
92+
rel(13574166); // 1949 Israeli–Syrian DMZ
93+
rel(13574167); // Extent of Syrian Claims at Golan Heights
94+
rel(13790690); // Approximate Extent of Siachen Glacier Dispute
95+
rel(13812677); // Line of Control at Demchok
96+
rel(13812681); // Line of Control at Depsang Bulge
97+
rel(13812687); // Extent of Indian/Pakistani Claim - Chumar area
98+
rel(13812689); // Extent of Chinese Claim - Chumar area
99+
rel(13816439); // Line of Control at Doklam
100+
rel(13816659); // Line of Control at Pangong Tso
101+
rel(13879674); // Disputed province boundary at Jammu And Kashmir
102+
rel(13879681); // Disputed province boundary at Aksai Chin
103+
rel(13879689); // Extent of Disputed Province Boundary
104+
rel(13931905); // Extent of Moroccan Provinces within Western Saharan claim
105+
rel(13932537); // Disputed province boundary at Jammu And Kashmir
106+
rel(13932538); // Disputed province boundary at Jammu And Kashmir
107+
rel(13937522); // Line of Control between India and Pakistan
108+
);
109+
out meta;
110+
>;
111+
out meta qt;

0 commit comments

Comments
 (0)