Skip to content

Commit

Permalink
Merge pull request #3544 from magda-io/issue/3487
Browse files Browse the repository at this point in the history 
#3487: Update magda to use ASGS 2021 and latest LGAs boundaries
  • Loading branch information
@t83714
t83714 authored Jun 28, 2024
2 parents 3923276 + 834e7fd commit b05b773
Show file tree
Hide file tree
Showing 9 changed files with 455 additions and 435 deletions.
9 changes: 9 additions & 0 deletions CHANGES.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# CHANGELOG

## v4.1.0

- #3487: Update to use ASGS 2021 and latest LGAs (2023) boundaries
- Added `indexed` field to region index & Bump region index version to `27`
- Migration instruction can be found from [the migration doc](./docs/docs/migration/4.1.0.md)
- Adjusted & improved region config entry skip logic
- Improved the region loader performance
- Fixed: region loader failed to convert number type id property into a string

## v4.0.1

- Add helm chart configuration option allows users to disable gateway auto-gzip response feature
Expand Down
217 changes: 217 additions & 0 deletions docs/docs/migration/4.1.0.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
## v4.1.0

## Search Engine Region Index Upgrade

We upgrade region index mapping version to `27` and upgrade region data to use ASGS 2021 and latest LGAs (2023) boundaries. During the upgrade, indexer will auto-recreate the region index. Depends on your database size, this might take some time (5-30 mins depends on resources allocated).

> To build your own region files & MVT tile service, please refer to this repo: https://github.com/magda-io/magda-regions
To avoid serving incomplete region data during the re-indexing process, you can manually set the region index version that search API uses to the existing versions via [helm chart options](https://github.com/magda-io/magda/blob/944ae887842b98c51698d567435003be2e9dbefd/deploy/helm/internal-charts/search-api/values.yaml#L29).

Moreover, we also need to set the region mapping served from search API [Get Region Types
](https://dev.magda.io/api/v0/apidocs/index.html#api-Search-GetV0SearchRegionTypes) endpoint to ensure it matches the old region index version we set.

e.g. you can set the following in your helm deployment values file / config to make search api still queries the previous region index version (`26`) while indexer's creating the new indexes:

```
search-api:
# set region index version used to previous version: 26
regionsIndexVersion: 26
appConfig:
# Set to previous region mapping for regions in index version 26 or older
# You can find region mapping from the repo here: https://github.com/magda-io/magda/blob/main/magda-search-api/src/main/resources/regionMapping.json
regionMapping:
regionWmsMap:
STE:
layerName: FID_STE_2011_AUST
server: https://tiles.magda.io/FID_STE_2011_AUST/{z}/{x}/{y}.pbf
regionProp: STE_CODE11
aliases:
- ste_code
- ste_code_2011
- ste
digits: 1
description: States and Territories (STE)
regionIdsFile: data/regionids/region_map-FID_STE_2011_AUST_STE_CODE11.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.74050960300003
- 159.10921900799997
- -9.142175976999999
nameProp: STE_NAME11
SA4:
layerName: FID_SA4_2011_AUST
server: https://tiles.magda.io/FID_SA4_2011_AUST/{z}/{x}/{y}.pbf
regionProp: SA4_CODE11
aliases:
- sa4_code_2011
- sa4_code
- sa4
digits: 3
description: Statistical Area Level 4 (SA4)
regionIdsFile: data/regionids/region_map-FID_SA4_2011_AUST_SA4_CODE11.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.74050960300003
- 159.10921900799997
- -9.142175976999999
nameProp: SA4_NAME11
SA3:
layerName: FID_SA3_2011_AUST
server: https://tiles.magda.io/FID_SA3_2011_AUST/{z}/{x}/{y}.pbf
regionProp: SA3_CODE11
aliases:
- sa3_code_2011
- sa3_code
- sa3
digits: 5
description: Statistical Area Level 3 (SA3)
regionIdsFile: data/regionids/region_map-FID_SA3_2011_AUST_SA3_CODE11.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.74050960300003
- 159.10921900799997
- -9.142175976999999
nameProp: SA3_NAME11
SA2:
layerName: FID_SA2_2011_AUST
server: https://tiles.magda.io/FID_SA2_2011_AUST/{z}/{x}/{y}.pbf
regionProp: SA2_MAIN11
aliases:
- sa2_code_2011
- sa2_code
- sa2
digits: 9
description: Statistical Area Level 2 (SA2)
regionIdsFile: data/regionids/region_map-FID_SA2_2011_AUST_SA2_MAIN11.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.74050960300003
- 159.10921900799997
- -9.142175976999999
nameProp: SA2_NAME11
SA1:
layerName: FID_SA1_2011_AUST
server: https://tiles.magda.io/FID_SA1_2011_AUST/{z}/{x}/{y}.pbf
regionProp: SA1_MAIN11
aliases:
- sa1_code_2011
- sa1_maincode_2011
- sa1_code
- sa1
digits: 11
description: Statistical Area Level 1 (SA1)
regionIdsFile: data/regionids/region_map-FID_SA1_2011_AUST_SA1_MAIN11.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.74050960300003
- 159.10921900799997
- -9.142175976999999
nameProp: SA2_NAME11
LGA:
layerName: FID_LGA_2015_AUST
server: https://tiles.magda.io/FID_LGA_2015_AUST/{z}/{x}/{y}.pbf
regionProp: LGA_CODE15
aliases:
- lga_code_2015
- lga_code
- lga
- lga_code_2014
- lga_code_2012
- lga_code_2010
digits: 5
description: Local Government Area (LGA)
regionIdsFile: data/regionids/region_map-FID_LGA_2015_AUST_LGA_CODE15.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.740509602999985
- 159.10921900799997
- -9.142175976999999
nameProp: LGA_NAME15
POA:
layerName: FID_POA_2011_AUST
server: https://tiles.magda.io/FID_POA_2011_AUST/{z}/{x}/{y}.pbf
regionProp: POA_CODE
aliases:
- poa_2011
- postcode_2011
- poa
- poa_code
- poa_code_2011
- postcode
- postcode_2015
digits: 4
dataReplacements:
- - "^(?=\\d\\d\\d$)"
- '0'
description: Postal Area (POA)
regionIdsFile: data/regionids/region_map-FID_POA_2011_AUST_POA_CODE.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81694140799998
- -43.59821500299999
- 159.10921900799997
- -9.142175976999999
nameProp: POA_NAME
COM_ELB_ID_2016:
layerName: FID_COM20160509_ELB
server: https://tiles.magda.io/FID_COM20160509_ELB/{z}/{x}/{y}.pbf
regionProp: DIV_ID
aliases:
- divisionid
- com_elb_id_2016
- com_elb_id
- com_elb
digits: 3
description: Commonwealth Electoral District
regionIdsFile: data/regionids/region_map-FID_COM20160509_ELB_DIV_ID.json
serverType: MVT
serverSubdomains: []
serverMinZoom: 0
serverMaxNativeZoom: 12
serverMaxZoom: 28
bbox:
- 96.81676599999997
- -43.740509999999986
- 159.1092189999999
- -9.142175999999996
nameProp: SORTNAME
```

Once indexer fully completes the new region index creation / indexing, you can remove the above config from deployment config and deploy again to make search API uses the latest version index.

You can check the indexing progress via Indexer module logs.
2 changes: 1 addition & 1 deletion magda-int-test-ts/indexer-setup.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ registry.registerForWebhooks = false
# only enable / use one region file for speeding up the initialisation
regionSources = {
COUNTRY {
url = "https://github.com/magda-io/magda-regions/releases/download/v1.0.0/country.geojson"
url = "https://github.com/magda-io/magda-regions/releases/download/v2.0.0/country.ndjson"
idField = "id"
nameField = "name"
order = 9
Expand Down
Loading

0 comments on commit b05b773

Please sign in to comment.