Skip to content

Commit d8ec051

Browse files
committed
Refined readme.md
1 parent 343e844 commit d8ec051

File tree

1 file changed

+61
-22
lines changed

1 file changed

+61
-22
lines changed

readme.md

+61-22
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,22 @@
22

33
MySQL Binary Log connector.
44

5-
Initially project was started as a fork of [open-replicator](https://code.google.com/p/open-replicator), but ended up as a complete rewrite. Key differences/features:
5+
Initially project was started as a fork of [open-replicator](https://code.google.com/p/open-replicator),
6+
but ended up as a complete rewrite. Key differences/features:
67

7-
- automatic binlog filename/position resolution
8+
- automatic binlog filename/position | GTID resolution
89
- resumable disconnects
910
- plugable failover strategies
1011
- JMX exposure (optionally with statistics)
1112
- availability in Maven Central
1213
- no third-party dependencies
13-
- binlog_checksum support (for MySQL 5.6.2+ users)
14+
- binlog_checksum=CRC32 support (for MySQL 5.6.2+ users)
1415
- test suite over different versions of MySQL releases
1516

17+
> If you are looking for something similar in other languages - check out
18+
[siddontang/go-mysql](https://github.com/siddontang/go-mysql) (Go),
19+
[noplay/python-mysql-replication](https://github.com/noplay/python-mysql-replication) (Python).
20+
1621
## Usage
1722

1823
Get the latest JAR(s) from [here](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.github.shyiko%22%20AND%20a%3A%22mysql-binlog-connector-java%22). Alternatively you can include following Maven dependency (available through Maven Central):
@@ -38,19 +43,19 @@ The latest development version always available through Sonatype Snapshots repos
3843

3944
<repositories>
4045
<repository>
41-
<id>sonatype-snapshots</id>
42-
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
43-
<snapshots>
44-
<enabled>true</enabled>
45-
</snapshots>
46-
<releases>
47-
<enabled>false</enabled>
48-
</releases>
46+
<id>sonatype-snapshots</id>
47+
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
48+
<snapshots>
49+
<enabled>true</enabled>
50+
</snapshots>
51+
<releases>
52+
<enabled>false</enabled>
53+
</releases>
4954
</repository>
5055
</repositories>
5156
```
5257

53-
### Reading Binary Log file
58+
### Reading binary log file
5459

5560
```java
5661
File binlogFile = ...
@@ -88,6 +93,12 @@ kick off from a specific filename or position, use `client.setBinlogFilename(fil
8893

8994
### Controlling event deserialization
9095

96+
> You might need it for several reasons:
97+
you don't want to waste time deserializing events you won't need;
98+
there is no EventDataDeserializer defined for the event type you are interested in (or there is but it contains a bug);
99+
you want certain type of events to be deserialized in a different way (perhaps *RowsEventData should contain table
100+
name and not id?); etc.
101+
91102
```java
92103
EventDeserializer eventDeserializer = new EventDeserializer();
93104

@@ -109,7 +120,7 @@ BinaryLogClient client = ...
109120
client.setEventDeserializer(eventDeserializer);
110121
```
111122

112-
### Making client available through JMX
123+
### Exposing BinaryLogClient with JMX
113124

114125
```java
115126
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
@@ -128,17 +139,51 @@ mBeanServer.registerMBean(stats, statsObjectName);
128139
## Implementation notes
129140

130141
- data of numeric types (tinyint, etc) always returned signed(!) regardless of whether column definition includes "unsigned" keyword or not
131-
- data of \*text/\*blob types always returned as a byte array
142+
- data of var\*/\*text/\*blob types always returned as a byte array (for var\* this is true starting from mysql-binlog-connector-java@1.0.0).
132143

133144
## Frequently Asked Questions
134145

135-
Q: How do I get column names of a table?
136-
A: The easiest way is to use JDBC (as described [here](https://github.com/shyiko/mysql-binlog-connector-java/issues/24#issuecomment-43747417)). Binary log itself does not contain that piece of information.
146+
**Q**. How does a typical transaction look like?
147+
148+
**A**. GTID event (if gtid_mode=ON) -> QUERY event with "BEGIN" as sql -> ... -> XID event | QUERY event with "COMMIT" or "ROLLBACK" as sql.
149+
150+
**Q**. EventData for inserted/updated/deleted rows has no information about table (except for some weird id).
151+
How do I make sense out of it?
152+
153+
**A**. Each [WriteRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/WriteRowsEventData.java)/[UpdateRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/UpdateRowsEventData.java)/[DeleteRowsEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/DeleteRowsEventData.java) event is preceded by [TableMapEventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/TableMapEventData.java) which
154+
contains schema & table name. If for some reason you need to know column names (types, etc). - the easiest way is to
155+
156+
```sql
157+
select TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, ORDINAL_POSITION, COLUMN_DEFAULT, IS_NULLABLE,
158+
DATA_TYPE, CHARACTER_MAXIMUM_LENGTH, CHARACTER_OCTET_LENGTH, NUMERIC_PRECISION, NUMERIC_SCALE,
159+
CHARACTER_SET_NAME, COLLATION_NAME from INFORMATION_SCHEMA.COLUMNS;
160+
# see https://dev.mysql.com/doc/refman/5.6/en/columns-table.html for more information
161+
```
162+
163+
(yes, binary log DOES NOT include that piece of information).
164+
165+
You can find JDBC snippet [here](https://github.com/shyiko/mysql-binlog-connector-java/issues/24#issuecomment-43747417).
137166

138167
## Documentation
139168

169+
BASICS: There are two entry points - [BinaryLogClient](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java) (which you can use to read binary logs from a MySQL server) and
170+
[BinaryLogFileReader](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/BinaryLogFileReader.java) (for offline log processing). Both of them rely on [EventDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventDeserializer.java) to deserialize
171+
stream of events. Each [Event](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/Event.java) consists of [EventHeader](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventHeader.java) (containing among other things reference to [EventType](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventType.java)) and
172+
[EventData](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/EventData.java). The aforementioned EventDeserializer has one [EventHeaderDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventHeaderDeserializer.java) ([EventHeaderV4Deserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventHeaderV4Deserializer.java) by default)
173+
and [a collection of EventDataDeserializer|s](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/EventDeserializer.java#L82). If there is no EventDataDeserializer registered for
174+
some particular type of Event - default EventDataDeserializer kicks in ([NullEventDataDeserializer](https://github.com/shyiko/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/NullEventDataDeserializer.java)).
175+
140176
For the insight into the internals of MySQL look [here](https://dev.mysql.com/doc/internals/en/index.html). [MySQL Client/Server Protocol](http://dev.mysql.com/doc/internals/en/client-server-protocol.html) and [The Binary Log](http://dev.mysql.com/doc/internals/en/binary-log.html) sections are particularly useful as a reference documentation for the `com.**.mysql.binlog.network` and `com.**.mysql.binlog.event` packages.
141177

178+
## Real-world applications
179+
180+
Some of the OSS built on top of mysql-binlog-conector-java:
181+
[shyiko/rook](https://github.com/shyiko/rook) (generic Change Data Capture (CDC) toolkit),
182+
[mardambey/mypipe](https://github.com/mardambey/mypipe) (MySQL to Apache Kafka replicator),
183+
[ngocdaothanh/mydit](https://github.com/ngocdaothanh/mydit) (MySQL to MongoDB replicator),
184+
185+
It's also used [on a large scale](https://twitter.com/atwinmutt/status/626816601078300672) in MailChimp. You can read about it [here](http://devs.mailchimp.com/blog/powering-mailchimp-pro-reporting/).
186+
142187
## Development
143188

144189
```sh
@@ -147,12 +192,6 @@ cd mysql-binlog-connector-java
147192
mvn # shows how to build, test, etc. project
148193
```
149194

150-
## Used by
151-
152-
* [shyiko/rook](https://github.com/shyiko/rook) - Change Data Capture (CDC) toolkit for keeping system layers in sync with the database.
153-
* [ngocdaothanh/mydit](https://github.com/ngocdaothanh/mydit) - MySQL to MongoDB data replicator.
154-
* [mardambey/mypipe](https://github.com/mardambey/mypipe) - MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.
155-
156195
## Contributing
157196

158197
In lieu of a formal styleguide, please take care to maintain the existing coding style.

0 commit comments

Comments
 (0)