Merge pull request #28 from ibi-group/elevation-improvements

Elevation improvements

docs/Configuration.md

@@ -84,7 +84,8 @@ config key | description | value type | value default | notes
 `matchBusRoutesToStreets` | Based on GTFS shape data, guess which OSM streets each bus runs on to improve stop linking | boolean | false |
 `fetchElevationUS` | Download US NED elevation data and apply it to the graph | boolean | false |
 `elevationBucket` | If specified, download NED elevation tiles from the given AWS S3 bucket | object | null | provide an object with `accessKey`, `secretKey`, and `bucketName` for AWS S3
-`elevationUnitMultiplier` | Specify a multiplier to convert elevation units from source to meters | double | 1.0 | see [Elevation unit conversion](#elevation-unit-conversion)
+`readCachedElevations` | If true, reads in pre-calculated elevation data. | boolean | true | see [Elevation Data Calculation Optimizations](#elevation-data-calculation-optimizations)
+`writeCachedElevations` | If true, writes the calculated elevation data. | boolean | false | see [Elevation Data Calculation Optimizations](#elevation-data-calculation-optimizations)
 `fares` | A specific fares service to use | object | null | see [fares configuration](#fares-configuration)
 `osmNaming` | A custom OSM namer to use | object | null | see [custom naming](#custom-naming)
 `osmWayPropertySet` | Custom OSM way properties | string | `default` | options: `default`, `norway`
@@ -197,6 +198,31 @@ You can configure it as follows in `build-config.json`:
 }
 ```
 
+### Geoid Difference
+
+With some elevation data, the elevation values are specified as relative to the a geoid (irregular estimate of mean sea level). See [issue #2301](https://github.com/opentripplanner/OpenTripPlanner/issues/2301) for detailed discussion of this. In these cases, it is necessary to also add this geoid value onto the elevation value to get the correct result. OTP can automatically calculate these values in one of two ways. 
+
+The first way is to use the geoid difference value that is calculated once at the center of the graph. This value is returned in each trip plan response in the [ElevationMetadata](http://otp-docs.ibi-transit.com/api/json_ElevationMetadata.html) field. Using a single value can be sufficient for smaller OTP deployments, but might result in incorrect values at the edges of larger OTP deployments. If your OTP instance uses this, it is recommended to set a default request value in the `router-config.json` file as follows:
+
+```JSON
+// router-config.json
+{
+    "routingDefaults": {
+        "geoidElevation ": true   
+    }
+}
+```
+
+The second way is to precompute these geoid difference values at a more granular level and include them when calculating elevations for each sampled point along each street edge. In order to speed up calculations, the geoid difference values are calculated and cached using only 2 significant digits of GPS coordinates. This is more than enough detail for most regions of the world and should result in less than one meter of difference in areas that have large changes in geoid difference values. To enable this, include the following in the `build-config.json` file: 
+
+```JSON
+// build-config.json
+{
+  "includeEllipsoidToGeoidDifference": true
+}
+```
+
+If the geoid difference values are precomputed, be careful to not set the routing resource value of `geoidElevation` to true in order to avoid having the graph-wide geoid added again to all elevation values in the relevant street edges in responses.
 
 ### Other raster elevation data
 
@@ -218,6 +244,28 @@ order as the above-mentioned SRTM data, which is also the default for the popula
 DEM files(USGS DEM) is not supported by OTP, but can be converted to GeoTIFF with tools like [GDAL](http://www.gdal.org/). 
 Use `gdal_merge.py -o merged.tiff *.dem` to merge a set of `dem` files into one `tif` file.
 
+### Elevation Data Calculation Optimizations
+
+Calculating elevations on all StreetEdges can take a dramatically long time. In a very large graph build for multiple Northeast US states, the time it took to download the elevation data and calculate all of the elevations took 5,509 seconds (roughly 1.5 hours).
+
+If you are using cloud computing for your OTP instances, it is recommended to create prebuilt images that contain the elevation data you need. This will save time because all of the data won't need to be downloaded.
+
+However, the bulk of the time will still be spent calculating elevations for all of the street edges. Therefore, a further optimization can be done to calculate and save the elevation data during a graph build and then save it for future use.
+
+In order to write out the precalculated elevation data, add this to your `build-config.json` file:
+
+```JSON
+// build-config.json
+{
+  "writeCachedElevations": true
+}
+```
+
+After building the graph, a file called `cached_elevations.obj` will be written to the cache directory. By default, this file is not written during graph builds. There is also a graph build parameter called `readCachedElevations` which is set to `true` by default.
+
+In graph builds, the elevation module will attempt to read the `cached_elevations.obj` file from the cache directory. The cache directory defaults to `/var/otp/cache`, but this can be overriden via the CLI argument `--cache <directory>`. For the same graph build for multiple Northeast US states, the time it took with using this predownloaded and precalculated data became 543.7 seconds (roughly 9 minutes).
+
+The cached data is a lookup table where the coordinate sequences of respective street edges are used as keys for calculated data. Therefore, it is expected that over time various edits to OpenStreetMap will cause this cached data to become stale and not include new OSM ways. Therefore, periodic update of this cached data is recommended.
 
 ## Fares configuration
 

src/main/java/org/opentripplanner/graph_builder/GraphBuilder.java

@@ -35,6 +35,7 @@
 
 import java.io.File;
 import java.io.IOException;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
@@ -179,6 +180,7 @@ public static GraphBuilder forDirectory(CommandLineParameters params, File dir)
             LOG.error("'{}' is not a readable directory.", dir);
             return null;
         }
+
         // Find and parse config files first to reveal syntax errors early without waiting for graph build.
         builderConfig = OTPMain.loadJson(new File(dir, BUILDER_CONFIG_FILENAME));
         GraphBuilderParameters builderParams = new GraphBuilderParameters(builderConfig);
@@ -272,30 +274,34 @@ public static GraphBuilder forDirectory(CommandLineParameters params, File dir)
         graphBuilder.addModule(streetLinkerModule);
         // Load elevation data and apply it to the streets.
         // We want to do run this module after loading the OSM street network but before finding transfers.
+        ElevationGridCoverageFactory gcf = null;
         if (builderParams.elevationBucket != null) {
             // Download the elevation tiles from an Amazon S3 bucket
             S3BucketConfig bucketConfig = builderParams.elevationBucket;
             File cacheDirectory = new File(params.cacheDirectory, "ned");
             DegreeGridNEDTileSource awsTileSource = new DegreeGridNEDTileSource();
-            awsTileSource = new DegreeGridNEDTileSource();
             awsTileSource.awsAccessKey = bucketConfig.accessKey;
             awsTileSource.awsSecretKey = bucketConfig.secretKey;
             awsTileSource.awsBucketName = bucketConfig.bucketName;
-            NEDGridCoverageFactoryImpl gcf = new NEDGridCoverageFactoryImpl(cacheDirectory);
-            gcf.tileSource = awsTileSource;
-            GraphBuilderModule elevationBuilder = new ElevationModule(gcf);
-            graphBuilder.addModule(elevationBuilder);
+            gcf = new NEDGridCoverageFactoryImpl(cacheDirectory, awsTileSource);
         } else if (builderParams.fetchElevationUS) {
             // Download the elevation tiles from the official web service
             File cacheDirectory = new File(params.cacheDirectory, "ned");
-            ElevationGridCoverageFactory gcf = new NEDGridCoverageFactoryImpl(cacheDirectory);
-            GraphBuilderModule elevationBuilder = new ElevationModule(gcf);
-            graphBuilder.addModule(elevationBuilder);
+            gcf = new NEDGridCoverageFactoryImpl(cacheDirectory);
         } else if (demFile != null) {
             // Load the elevation from a file in the graph inputs directory
-            ElevationGridCoverageFactory gcf = new GeotiffGridCoverageFactoryImpl(demFile);
-            GraphBuilderModule elevationBuilder = new ElevationModule(gcf);
-            graphBuilder.addModule(elevationBuilder);
+            gcf = new GeotiffGridCoverageFactoryImpl(demFile);
+        }
+        if (gcf != null) {
+            graphBuilder.addModule(
+                new ElevationModule(
+                    gcf,
+                    params.cacheDirectory,
+                    builderParams.readCachedElevations,
+                    builderParams.writeCachedElevations,
+                    builderParams.includeEllipsoidToGeoidDifference
+                )
+            );
         }
         if ( hasGTFS ) {
             // The stops can be linked to each other once they are already linked to the street network.

src/main/java/org/opentripplanner/graph_builder/module/ned/DegreeGridNEDTileSource.java

@@ -33,8 +33,6 @@
 public class DegreeGridNEDTileSource implements NEDTileSource {
     private static Logger log = LoggerFactory.getLogger(DegreeGridNEDTileSource.class);
 
-    private Graph graph;
-
     private File cacheDirectory;
 
     public String awsAccessKey;
@@ -43,18 +41,15 @@ public class DegreeGridNEDTileSource implements NEDTileSource {
 
     public String awsBucketName;
 
-    @Override
-    public void setGraph(Graph graph) {
-        this.graph = graph;
-    }
+    private List<File> nedTiles;
 
     @Override
-    public void setCacheDirectory(File cacheDirectory) {
-        this.cacheDirectory = cacheDirectory;
+    public List<File> getNEDTiles() {
+        return nedTiles;
     }
 
-    @Override
-    public List<File> getNEDTiles() {
+    @Override public void fetchData(Graph graph, File cacheDirectory) {
+        this.cacheDirectory = cacheDirectory;
 
         HashSet<P2<Integer>> tiles = new HashSet<P2<Integer>>();
 
@@ -67,9 +62,12 @@ public List<File> getNEDTiles() {
         for (P2<Integer> tile : tiles) {
             int x = tile.first - 1;
             int y = tile.second + 1;
-            paths.add(getPathToTile(x, y));
+            File tilePath = getPathToTile(x, y);
+            if (tilePath != null) {
+                paths.add(tilePath);
+            }
         }
-        return paths;
+        nedTiles = paths;
     }
 
     private String formatLatLon(int x, int y) {
@@ -97,14 +95,15 @@ private File getPathToTile(int x, int y) {
             path.getParentFile().mkdirs();
 
             if (awsAccessKey == null || awsSecretKey == null) {
-                throw new RuntimeException("Cannot download NED tiles from S3: awsAccessKey or awsSecretKey properties are not set");
+                throw new RuntimeException(
+                    "Cannot download NED tiles from S3: awsAccessKey or awsSecretKey properties are not set");
             }
             log.info("Downloading NED degree tile " + path);
             // download the file from S3.
             AWSCredentials awsCredentials = new AWSCredentials(awsAccessKey, awsSecretKey);
+            String key = formatLatLon(x, y) + ".tiff";
             try {
                 S3Service s3Service = new RestS3Service(awsCredentials);
-                String key = formatLatLon(x, y) + ".tiff";
                 S3Object object = s3Service.getObject(awsBucketName, key);
 
                 InputStream istream = object.getDataInputStream();
@@ -120,18 +119,10 @@ private File getPathToTile(int x, int y) {
                 }
                 ostream.close();
                 istream.close();
-            } catch (S3ServiceException e) {
-                path.deleteOnExit();
-                throw new RuntimeException(e);
-            } catch (ServiceException e) {
-                path.deleteOnExit();
-                throw new RuntimeException(e);
-            } catch (FileNotFoundException e) {
-                path.deleteOnExit();
-                throw new RuntimeException(e);
-            } catch (IOException e) {
+            } catch (ServiceException | IOException e) {
+                log.error("Error downloading tile {}! Error: {}.", key, e.getMessage());
                 path.deleteOnExit();
-                throw new RuntimeException(e);
+                return null;
             }
             return path;
         }