Add Berkley DB support

ochriste · Aug 9, 2013 · 683de3c · 683de3c
1 parent a3ef152
commit 683de3c
Show file tree

Hide file tree

Showing 10 changed files with 251 additions and 69 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -4,11 +4,15 @@ python:
 - 2.6
 - 2.7
 
+env:
+  global:
+  - BERKELEYDB_LIBDIR=/usr/lib/x86_64-linux-gnu BERKELEYDB_INCDIR=/usr/include
+
 before_install:
 - echo 'yes' | sudo add-apt-repository ppa:stephane-brunner/precise
 - echo 'yes' | sudo add-apt-repository ppa:mapnik/v2.2.0
 - sudo apt-get update
-- sudo apt-get install -y libmapnik-dev python-mapnik postgresql-9.1-postgis apache2 cgi-mapserver deploy
+- sudo apt-get install -y libmapnik-dev python-mapnik postgresql-9.1-postgis apache2 cgi-mapserver deploy libdb-dev
 - sudo -u postgres tilecloud_chain/tests/create_test_data.sh
 - sudo cp tilecloud_chain/tests/apache.conf /etc/apache2/sites-enabled/apache.conf
 - sudo apache2ctl graceful
@@ -35,11 +39,11 @@ script:
 - ./buildout/bin/python setup.py nosetests
 
 after_success:
-  # Report coverage results to coveralls.io
-  - pip install coveralls
-  - coveralls
+# Report coverage results to coveralls.io
+- pip install coveralls
+- coveralls
 
 notifications:
   email:
     recipients:
-      - stephane.brunner@gmail.com
+    - stephane.brunner@gmail.com
diff --git a/README.rst b/README.rst
@@ -7,7 +7,7 @@ Source: WMS, Mapnik.
 
 Optionally use an SQS queue, AWS host, SNS topic.
 
-Destination in WMTS layout, directly on local filesystem, on S3 or MBTiles on local filesystem.
+Destination in WMTS layout, on S3, on Berkley DB (``bsddb``), on MBTiles, or on local filesystem.
 
 Feature:
 
@@ -72,16 +72,16 @@ resolution without regenerate all the tiles, but it don't work with MapCache.
 Configure caches
 ----------------
 
-There tree available tiles cache: ``s3``, ``mbtile`` and ``filesystem``.
+The available tiles cache are: ``s3``, ``bsddb``, ``mbtile`` and ``filesystem``.
 
-The best solution to store the tiles is ``s3``, ``mbtiles`` has the advantage to have only one file per
-layer - style  dimensions.
+The best solution to store the tiles is ``s3``, ``mbtiles`` and ``bsddb`` has the advantage to have only one file per
+layer - style  dimensions. To serve the ``mbtile`` and the ``bsddb`` see `Distribute the tiles`_.
 
-``s3`` need a ``bucket`` an a ``folder`` (default to '').
+``s3`` need a ``bucket`` and a ``folder`` (default to '').
 
-``mbtiles`` and ``filesystem`` just need a ``folder``.
+``mbtiles``, ``bsddb`` and ``filesystem`` just need a ``folder``.
 
-On all the cache we can add some information to generate the url where the tiles are available.
+On all the cache we can add some information to generate the URL where the tiles are available.
 This is needed to generate the capabilities. We can specify:
 
 * ``http_url`` direct url to the tiles root.
@@ -90,6 +90,13 @@ This is needed to generate the capabilities. We can specify:
 
 In all case ``http_url`` or ``http_urls`` can include all attribute of this cache as ``%(attribute)s``.
 
+MBTiles vs Berkley DB (``bsddb``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Read performance: similar, eventually the MBTiles is 10% faster.
+* Write performance: The Berkley DB is largely faster, about 10 times.
+* List the tiles: the MBTiles is largely faster but we usually don't need it.
+
 
 Configure layers
 ----------------
@@ -219,37 +226,37 @@ The previously defined ``mime_type`` is also used in the WMS requests.
 To customise the request you also have the attributes ``params``, ``headers``
 and ``generate_salt``.
 In ``params`` you can specify additional parameter of the WMS request,
-in ``headers`` you can modify the request headers. See the chapter
-'Proxy/cache issue' for additional informations.
+in ``headers`` you can modify the request headers. See the
+`Proxy/cache issue`_ for additional informations.
 
 
 Mapnik layers
 ~~~~~~~~~~~~~
 
-We ned to specify the ``mapfile`` path.
+We need to specify the ``mapfile`` path.
 
-With mapnik we have the possibility to specify a ``data_buffer`` than we should set the unneeded ``meta_buffer`` to 0.
+With Mapnik we have the possibility to specify a ``data_buffer`` than we should set the unneeded ``meta_buffer`` to 0.
 
-And the ``output_format`` used for the mapnik renderer, can be ``png``, ``png256``, ``jpeg``, ``grid`` (grid_renderer).
+And the ``output_format`` used for the Mapnik renderer, can be ``png``, ``png256``, ``jpeg``, ``grid`` (grid_renderer).
 
 
 ~~~~~~~~~~~~~~~~~~
 Mapnik grid layers
 ~~~~~~~~~~~~~~~~~~
 
-With mapnik we can generate UTFGrid tiles (JSON format that describe the tiles present on a corresponding tile)
+With Mapnik we can generate UTFGrid tiles (JSON format that describe the tiles present on a corresponding tile)
 by using the ``output_format`` 'grid', see also: https://github.com/mapnik/mapnik/wiki/MapnikRenderers#grid_renderer.
 
 Specific configuration:
 
 We have a specific way to ``drop_empty_utfgrid`` by using the ``on`` value.
 
-We should specify the speudo pixel size [px] with the ``resolution``.
+We should specify the pseudo pixel size [px] with the ``resolution``.
 
 And the ``layers_fields`` that we want to get the attributes.
 Object withe the layer name as key and the values in an array as value.
 
-Il fact the Mapnik documentation say that's working only for one layer.
+In fact the Mapnik documentation say that's working only for one layer.
 
 And don't miss the change the ``extension`` to ``json``, and the ``mime_type`` to ``application/utfgrid``
 and the ``meta`` to ``off`` (not supported).
@@ -322,17 +329,49 @@ To generate the Apache configuration we use the command::
 
     ./buildout/bin/generate_controller --generate-apache-config
 
-The internal server can be used as a python view with:
+The server can be configure as it:
+
+.. code:: yaml
+
+    server:
+        layers: a_layer # Restrict to serve an certain number of layers [default to all]
+        cache: mbtiles # The used cache [default use generation/default_cache]
+        # the URL without location to MapCache, [default to http://localhost/]
+        mapcache_base: http://localhost/
+        mapcache_headers: # headers, can be used to access to an other Apache vhost [default to {}]
+            Host: localhost
+        geoms_redirect: true # use the geoms to redirect to MapCache [defaut to false]
+        # allowed extension in the static path (default value), not used for s3.
+        static_allow_extension: [jpeg, png, xml]
+
+The minimal config is to enable it:
+
+.. code:: yaml
+
+    server: {}
+
+You should also configure the ``http_url`` of the used `cache`, to something like
+``https://%(host)s/${instanceid}/tiles`` or like
+``https://%(host)s/${instanceid}/wsgi/tiles/wmts`` if you use the Pyramid view.
+
+Pyramid view
+~~~~~~~~~~~~
+
+To use the pyramid view use the following config:
 
 .. code:: python
 
     config.get_settings().update({
         'tilegeneration_configfile': '<the configuration file>',
     })
-    config.add_route('tiles', '/tiles/*path')
+    config.add_route('tiles', '/tiles/\*path')
     config.add_view('tilecloud_chain.server:PyramidView', route_name='tiles')
 
-or as a WSGI server with buildout, add in ``buildout.cfg``::
+
+Internal WSGI server
+~~~~~~~~~~~~~~~~~~~~
+
+To use the WSGI server with buildout, add in ``buildout.cfg``::
 
     [buildout]
         parts = ...
@@ -360,34 +399,6 @@ with the apache configuration::
         WSGIApplicationGroup %{GLOBAL}
     </Location>
 
-To use tue Python view you should used the following:
-
-
-And configure as it:
-
-.. code:: yaml
-
-    server:
-        layers: a_layer # Restrict to serve an sertain number of layers [default to all]
-        cache: mbtiles # The used cache [default use generation/default_cache]
-        # the url without location to MapCache, [default to http://localhost/]
-        mapcache_base: http://localhost/
-        mapcache_headers: # headers, can be used to acces to an other Apache vhost [default to {}]
-            Host: localhost
-        geoms_redirect: true # use the geoms to redirect to MapCache [defaut to false]
-        # allowed extension in the static path (default value), not used for s3.
-        static_allow_extension: [jpeg, png, xml]
-
-The minimal config is to enable it:
-
-.. code:: yaml
-
-    server: {}
-
-You should also configure the ``http_url`` of the used `cache`, to somthing like
-``https://%(host)s/${instanceid}/tiles`` to don't use the internal server, and
-``https://%(host)s/${instanceid}/wsgi/tiles/wmts`` to use it.
-
 
 Generate configuration in buildout
 ----------------------------------
@@ -514,7 +525,7 @@ If we set a file path in config file:
     generation:
         error_file: <path>
 
-The tiles that in error will be appen to the file, ant the tiles can be regenerated with
+The tiles that in error will be append to the file, ant the tiles can be regenerated with
 ``./buildout/bin/generate_tiles --layer <layer> --tiles <path>``.
 
 
@@ -543,7 +554,7 @@ To don't have cache we use the as default the headers:
     - Pragma: no-cache
 
 And if you steal have issue you can add a ``SALT`` random argument by setting
-the layer paramater ``generate_salt`` to ``true``.
+the layer parameter ``generate_salt`` to ``true``.
 
 
 Explain cost
@@ -650,7 +661,7 @@ Configure and explain AWS
 The generation can be deported on an external host.
 
 
-Other usefull options
+Other useful options
 ---------------------
 
 ``--verbose`` or ``-v``: used to display info message.

diff --git a/setup.py b/setup.py
@@ -20,7 +20,14 @@
     'requests',
 ]
 if sys.version_info < (2, 7):
-    install_requires.append('argparse')
+    install_requires.extend([
+        'argparse',
+        'bsddb3',
+    ])
+if sys.version_info >= (3, 0):
+    install_requires.extend([
+        'bsddb3',
+    ])
 
 setup_requires = [
     'nose==1.3.0',

diff --git a/tilecloud_chain/__init__.py b/tilecloud_chain/__init__.py
@@ -13,6 +13,11 @@
 from fractions import Fraction
 from datetime import datetime
 
+try:
+    import bsddb3 as bsddb
+except:  # pragma: no cover
+    import bsddb
+
 try:
     from PIL import Image
     Image  # suppress pyflakes warning
@@ -31,6 +36,7 @@
 from tilecloud.store.metatile import MetaTileSplitterTileStore
 from tilecloud.store.s3 import S3TileStore
 from tilecloud.store.mbtiles import MBTilesTileStore
+from tilecloud.store.bsddb import BSDDBTileStore
 from tilecloud.store.filesystem import FilesystemTileStore
 from tilecloud.layout.wmts import WMTSTileLayout
 from tilecloud.filter.error import LogErrors, MaximumConsecutiveErrors, DropErrors
@@ -327,15 +333,15 @@ def __init__(self, config_file, options=None, layer_name=None):
             error = self.validate(cache, name, 'name', attribute_type=str, default=cname) or error
             error = self.validate(
                 cache, name, 'type', attribute_type=str, required=True,
-                enumeration=['s3', 'filesystem', 'mbtiles']
+                enumeration=['s3', 'filesystem', 'mbtiles', 'bsddb']
             ) or error
             error = self.validate(
                 cache, 'cache[%s]' % cache['name'], 'wmtscapabilities_file', attribute_type=str,
                 default='/1.0.0/WMTSCapabilities.xml'
             ) or error
             if cache['wmtscapabilities_file'][0] != '/':
                 cache['wmtscapabilities_file'] = '/' + cache['wmtscapabilities_file']
-            if cache['type'] == 'filesystem' or cache['type'] == 'mbtiles':
+            if cache['type'] == 'filesystem' or cache['type'] == 'mbtiles' or cache['type'] == 'bsddb':
                 error = self.validate(cache, name, 'folder', attribute_type=str, required=True) or error
             elif cache['type'] == 's3':
                 error = self.validate(cache, name, 'bucket', attribute_type=str, required=True) or error
@@ -481,7 +487,7 @@ def _resolution_scale(self, resolutions):
             result *= fact ** nb
         return result
 
-    def get_store(self, cache, layer, dimensions=None):
+    def get_store(self, cache, layer, dimensions=None, read_only=False):
         # build layout
         grid = layer['grid_ref'] if 'grid_ref' in layer else None
         layout = WMTSTileLayout(
@@ -513,6 +519,21 @@ def get_store(self, cache, layer, dimensions=None):
                 content_type=layer['mime_type'],
                 tilecoord_in_topleft=True,
             )
+        elif cache['type'] == 'bsddb':
+            # on bsddb file
+            filename = layout.filename(TileCoord(0, 0, 0)).replace(
+                '/0/0/0', ''
+            ) + '.bsddb'
+            if not os.path.exists(os.path.dirname(filename)):
+                os.makedirs(os.path.dirname(filename))
+            cache_tilestore = BSDDBTileStore(
+                bsddb.btopen(
+                    filename,
+                    # and os.path.exists(filename) to avoid error on non existing file
+                    'r' if read_only and os.path.exists(filename) else 'c'
+                ),
+                content_type=layer['mime_type'],
+            )
         elif cache['type'] == 'filesystem':
             # on filesystem
             cache_tilestore = FilesystemTileStore(

diff --git a/tilecloud_chain/server.py b/tilecloud_chain/server.py
@@ -139,7 +139,7 @@ def seed_filter(tile):
                 store_defs = new_store_defs
             for store_def in store_defs:
                 self.stores['/'.join(store_def['ref'])] = \
-                    self.tilegeneration.get_store(self.cache, layer, store_def['dimensions'])
+                    self.tilegeneration.get_store(self.cache, layer, store_def['dimensions'], read_only=True)
 
     def __call__(self, environ, start_response):
         params = {}