v1.0.0-PRE2, "an embarassment of riches"

1.0.0-PRE2 brings in a lot of changes that got merged while we were testing out PRE1. so, hey. - Configurable names for the `maxwell` schema database (Kristian Kaufman) - Configurable key (primary key, id, database) into the kafka partition hash function (Kristian Kaufman) - Configurable Kafka partition hash function (java hashCode, murmur3) (Kristian Kaufman) - support GEOMETRY columns, output as well-known-text - add `--blacklist_tables` option to fully ignore excessive schema changes (Nicolas Maquet) - bootstrap rows now have 'bootstrap-insert' type
zendesk · Jan 30, 2016 · f1e2aee · f1e2aee
1 parent 5a812b4
commit f1e2aee
Show file tree

Hide file tree

Showing 8 changed files with 45 additions and 19 deletions.
diff --git a/docs/docs/bootstrapping.md b/docs/docs/bootstrapping.md
@@ -1,4 +1,4 @@
-*Maxwell's bootstrapping is available in the [1.0.0-PRE1 release](https://github.com/zendesk/maxwell/releases/tag/v1.0.0-PRE1)*
+*Maxwell's bootstrapping is available in the [1.0.0-PRE2 release](https://github.com/zendesk/maxwell/releases/tag/v1.0.0-PRE2)*
 
 ### Using the maxwell-bootstrap utility
 ***

diff --git a/docs/site/bootstrapping/index.html b/docs/site/bootstrapping/index.html
@@ -171,7 +171,7 @@
 </div></div>
             <div class="col-md-10" role="main">
 
-<p><em>Maxwell's bootstrapping is available in the <a href="https://github.com/zendesk/maxwell/releases/tag/v1.0.0-PRE1">1.0.0-PRE1 release</a></em></p>
+<p><em>Maxwell's bootstrapping is available in the <a href="https://github.com/zendesk/maxwell/releases/tag/v1.0.0-PRE2">1.0.0-PRE2 release</a></em></p>
 <h3 id="using-the-maxwell-bootstrap-utility">Using the maxwell-bootstrap utility</h3>
 <hr />
 <p>You can use the <code>maxwell-bootstrap</code> utility to bootstrap tables from the command-line.</p>
@@ -243,8 +243,8 @@ <h3 id="bootstrapping-data-format">Bootstrapping Data Format</h3>
 <pre><code>{&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;insert&quot;,&quot;ts&quot;:1450557598,&quot;xid&quot;:13,&quot;data&quot;:{&quot;txt&quot;:&quot;hello&quot;}}
 {&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;insert&quot;,&quot;ts&quot;:1450557598,&quot;xid&quot;:13,&quot;data&quot;:{&quot;txt&quot;:&quot;bootstrap!&quot;}}
 {&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;bootstrap-start&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{}}
-{&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;insert&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{&quot;txt&quot;:&quot;hello&quot;}}
-{&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;insert&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{&quot;txt&quot;:&quot;bootstrap!&quot;}}
+{&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;bootstrap-insert&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{&quot;txt&quot;:&quot;hello&quot;}}
+{&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;bootstrap-insert&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{&quot;txt&quot;:&quot;bootstrap!&quot;}}
 {&quot;database&quot;:&quot;fooDB&quot;,&quot;table&quot;:&quot;barTable&quot;,&quot;type&quot;:&quot;bootstrap-complete&quot;,&quot;ts&quot;:1450557744,&quot;data&quot;:{}}
 </code></pre>
 

diff --git a/docs/site/config/index.html b/docs/site/config/index.html
@@ -215,6 +215,11 @@ <h3 id="command-line-options">Command line options</h3>
 <td>3306</td>
 </tr>
 <tr>
+<td>--schema_database</td>
+<td>database name where maxwell stores schema and state</td>
+<td>maxwell</td>
+</tr>
+<tr>
 <td>--max_schemas</td>
 <td>how many old schemas maxwell should leave lying around in maxwell.schemas</td>
 <td>5</td>
@@ -240,6 +245,16 @@ <h3 id="command-line-options">Command line options</h3>
 <td></td>
 </tr>
 <tr>
+<td>--kafka_partition_hash</td>
+<td>which hash function to use: [default, murmur3]</td>
+<td>default</td>
+</tr>
+<tr>
+<td>--kafka_partition_by</td>
+<td>what fields to hash for partition key: [database, table, primary_key]</td>
+<td>database</td>
+</tr>
+<tr>
 <td>--kafka_topic</td>
 <td>kafka topic to write to.</td>
 <td>maxwell</td>
@@ -295,6 +310,11 @@ <h3 id="command-line-options">Command line options</h3>
 <td></td>
 </tr>
 <tr>
+<td>--blacklist_tables PATTERN</td>
+<td>ignore updates AND schema changes from tables named like PATTERN (see warnings below)</td>
+<td></td>
+</tr>
+<tr>
 <td>&nbsp;</td>
 <td></td>
 <td></td>
@@ -351,6 +371,10 @@ <h3 id="filters">Filters</h3>
 <li>reject tables in <code>exclude_tables</code></li>
 </ol>
 <p>So an example like <code>--include_dbs=/foo.*/ --exclude_tables=bar</code> will include <code>footy.zab</code> and exclude <code>footy.bar</code></p>
+<p>The option <code>blacklist_tables</code> controls whether Maxwell will send updates for a table to its producer AND whether
+it captures schema changes for that table. Note that once Maxwell has been running with a table marked as blacklisted,
+you <em>must</em> continue to run Maxwell with that table blacklisted or else Maxwell will halt. If you want to stop
+blacklisting a table, you will have to drop the maxwell schema first.</p>
 <script>
   jQuery(document).ready(function () {
     jQuery("table").addClass("table table-condensed table-bordered table-hover");

diff --git a/docs/site/kafka/index.html b/docs/site/kafka/index.html
@@ -190,10 +190,12 @@ <h3 id="kafka-key">Kafka key</h3>
 as a source of truth.</p>
 <h3 id="partitioning">Partitioning</h3>
 <hr />
-<p>A binlog event's partition is chosen by the following:</p>
-<pre><code>  DATABASE_NAME.hash() % TOPIC.NUMBER_OF_PARTITIONS
+<p>A binlog event's partition is determined by the selected hash function and hash string as follows</p>
+<pre><code>  HASH_FUNCTION(HASH_STRING) % TOPIC.NUMBER_OF_PARTITIONS
 </code></pre>
 
+<p>The HASH_FUNCTION is either java's <em>hashCode</em> or <em>murmurhash3</em>. The default HASH_FUNCTION is <em>hashCode</em>. Murmurhash3 may be set with the <code>kafka_partition_hash</code> option. The seed value for the murmurhash function is hardcoded to 25342 in the MaxwellKafkaPartitioner class.</p>
+<p>The HASH_STRING may be (<em>database</em>, <em>table</em>, <em>primary_key</em>).  The default HASH_STRING is the <em>database</em>. The partitioning field can be configured using the <code>kafka_partition_by</code> option.    </p>
 <p>Maxwell will discover the number of partitions in its kafka topic upon boot.  This means that you should pre-create your kafka topics,
 and with at least as many partitions as you have logical databases:</p>
 <pre><code>bin/kafka-topics.sh --zookeeper ZK_HOST:2181 --create \

diff --git a/docs/site/mkdocs/search_index.json b/docs/site/mkdocs/search_index.json
diff --git a/docs/site/quickstart/index.html b/docs/site/quickstart/index.html
@@ -193,7 +193,7 @@ <h3 id="row-based-replication">Row based replication</h3>
 in order to change to row-based replication, you must reconnect all active clients to the server.</p>
 <h3 id="mysql-permissions">Mysql permissions</h3>
 <hr />
-<p>Maxwell stores all the state it needs within the mysql server itself, in a database called <code>maxwell</code>.</p>
+<p>Maxwell stores all the state it needs within the mysql server itself, in the database called specified by the <em>schema_database</em> option. By default the database is named <code>maxwell</code>.</p>
 <pre><code>mysql&gt; GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';
 mysql&gt; GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';
 

diff --git a/docs/site/sitemap.xml b/docs/site/sitemap.xml
@@ -4,55 +4,55 @@
 
     <url>
      <loc>None/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/quickstart/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/config/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/kafka/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/dataformat/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/bootstrapping/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/compat/</loc>
-     <lastmod>2016-01-11</lastmod>
+     <lastmod>2016-01-29</lastmod>
      <changefreq>daily</changefreq>
     </url>
 

diff --git a/pom.xml b/pom.xml
@@ -4,7 +4,7 @@
 
   <groupId>com.zendesk</groupId>
   <artifactId>maxwell</artifactId>
-  <version>1.0.0-PRE1</version>
+  <version>1.0.0-PRE2</version>
   <packaging>jar</packaging>
 
   <name>maxwell</name>