Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: html to markdown #48

Merged
merged 1 commit into from
Jun 22, 2017
Merged

docs: html to markdown #48

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 89 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,50 +1,89 @@
<h1>背景</h1>
<p>&nbsp; &nbsp;08年左右，阿里巴巴开始尝试MySQL的相关研究，并开发了基于MySQL分库分表技术的相关产品，Cobar/TDDL(目前为阿里云DRDS产品)，解决了单机Oracle无法满足的扩展性问题，当时也掀起一股去IOE项目的浪潮，愚公这项目因此而诞生，其要解决的目标就是帮助用户完成从Oracle数据迁移到MySQL上，完成去IOE的第一步.&nbsp;</p>
<h1>项目介绍</h1>
<p>名称: &nbsp; yugong</p>
<p>译意: &nbsp; 愚公移山</p>
<p>语言: &nbsp;&nbsp;纯java开发</p>
<p>定位: &nbsp; 数据库迁移 (目前主要支持oracle -&gt; mysql/DRDS)</p>
<h1>项目介绍</h1>
<p style="box-sizing: border-box; margin-bottom: 16px; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; line-height: 25.6px;">整个数据迁移过程，分为两部分：</p>
<ol>
<li>全量迁移</li>
<li>增量迁移</li>
</ol>
<p><img src="https://camo.githubusercontent.com/9a9cc09c5a7598239da20433857be61c54481b9c/687474703a2f2f646c322e69746579652e636f6d2f75706c6f61642f6174746163686d656e742f303131352f343531312f31306334666134632d626634342d333165352d623531312d6231393736643164373636392e706e67" alt="" /></p>
<p>过程描述：</p>
<ol>
<li>增量数据收集 (创建oracle表的增量物化视图)</li>
<li>进行全量复制</li>
<li>进行增量复制 (可并行进行数据校验)</li>
<li>原库停写，切到新库</li>
</ol>
<h1>架构</h1>
<p><img src="http://dl2.iteye.com/upload/attachment/0115/5473/8532d838-d4b2-371b-af9f-829d4127b1b8.png" alt="" width="584" height="206" /></p>
<p>说明:&nbsp;</p>
<ol>
<li><!--StartFragment-->
<div style="margin-top: 10.8pt; margin-bottom: 0pt; direction: ltr; unicode-bidi: embed; vertical-align: baseline;">一个Jvm Container对应多个instance，每个instance对应于一张表的迁移任务</div>
<!--EndFragment--></li>
<li><!--StartFragment-->&nbsp;instance分为三部分<br />a.&nbsp; <!--StartFragment-->extractor &nbsp;(从源数据库上提取数据，可分为全量/增量实现)<!--EndFragment--><br />b.&nbsp; <!--StartFragment-->translator &nbsp;(将源库上的数据按照目标库的需求进行自定义转化)<!--EndFragment--><br />c. &nbsp;applier<!--EndFragment-->&nbsp; (将数据更新到目标库，可分为全量/增量/对比的实现<!--EndFragment-->)</li>
</ol>
<h1>DevDesign</h1>
<p>See the page for dev design: <a href="https://github.com/alibaba/yugong/wiki/DevDesign">DevDesign</a></p>
<h1>QuickStart</h1>
<p>See the page for quick start: <a href="https://github.com/alibaba/yugong/wiki/QuickStart">QuickStart</a></p>
<h1>AdminGuide</h1>
<p>See the page for admin deploy guide: <a href="https://github.com/alibaba/yugong/wiki/AdminGuide">AdminGuide</a></p>
<h1>Performance</h1>
<p>See the page for yugong performance : <a href="https://github.com/alibaba/yugong/wiki/Performance">Performance</a></p>
<h1>相关资料</h1>
<ol>
<li>yugong简单介绍ppt : &nbsp;<a href="https://github.com/alibaba/yugong/blob/master/docs/yugong_Intro.ppt?raw=true">ppt</a></li>
<li><a href="https://www.aliyun.com/product/drds">分布式关系型数据库服务DRDS</a> (前身为阿里巴巴公司的Cobar/TDDL的演进版本, 基本原理为MySQL分库分表)</li>
</ol>
<h1>问题反馈</h1>
<ol>
<li><span style="line-height: 21px;">qq交流群： 537157866</span></li>
<li><span style="line-height: 21px;">邮件交流： jianghang115@gmail.com , zylicfc@gmail.com</span></li>
<li><span style="line-height: 21px;">新浪微博： agapple0002</span></li>
<li><span style="line-height: 21px;">报告issue：<a href="https://github.com/alibaba/yugong/issues">issues</a></span></li>
</ol>
## 背景

08年左右，阿里巴巴开始尝试MySQL的相关研究，并开发了基于MySQL分库分表技术的相关产品，Cobar/TDDL(目前为阿里云DRDS产品)，解决了单机Oracle无法满足的扩展性问题，当时也掀起一股去IOE项目的浪潮，愚公这项目因此而诞生，其要解决的目标就是帮助用户完成从Oracle数据迁移到MySQL上，完成去IOE的第一步. 

## 项目介绍


名称:   yugong

译意:   愚公移山

语言:   纯java开发

定位:   数据库迁移 (目前主要支持oracle / mysql / DRDS)

## 项目介绍


整个数据迁移过程，分为两部分：

1. 全量迁移
2. 增量迁移

![](https://camo.githubusercontent.com/9a9cc09c5a7598239da20433857be61c54481b9c/687474703a2f2f646c322e69746579652e636f6d2f75706c6f61642f6174746163686d656e742f303131352f343531312f31306334666134632d626634342d333165352d623531312d6231393736643164373636392e706e67)

过程描述：

1. 增量数据收集 (创建oracle表的增量物化视图)
2. 进行全量复制
3. 进行增量复制 (可并行进行数据校验)
4. 原库停写，切到新库

## 架构


![](http://dl2.iteye.com/upload/attachment/0115/5473/8532d838-d4b2-371b-af9f-829d4127b1b8.png){width="584"
height="206"}

说明: 

1. 一个Jvm Container对应多个instance，每个instance对应于一张表的迁移任务
2.  instance分为三部分
a.  extractor  (从源数据库上提取数据，可分为全量/增量实现)
b.  translator  (将源库上的数据按照目标库的需求进行自定义转化)
c.  applier  (将数据更新到目标库，可分为全量/增量/对比的实现)

## DevDesign


See the page for dev design:
[DevDesign](https://github.com/alibaba/yugong/wiki/DevDesign)

## QuickStart

See the page for quick start:
[QuickStart](https://github.com/alibaba/yugong/wiki/QuickStart)

## AdminGuide

See the page for admin deploy guide:
[AdminGuide](https://github.com/alibaba/yugong/wiki/AdminGuide)

## Performance

See the page for yugong performance :
[Performance](https://github.com/alibaba/yugong/wiki/Performance)

## 相关资料

1. yugong简单介绍ppt: [ppt](https://github.com/alibaba/yugong/blob/master/docs/yugong_Intro.ppt?raw=true)
2. [分布式关系型数据库服务DRDS](https://www.aliyun.com/product/drds)
(前身为阿里巴巴公司的Cobar/TDDL的演进版本, 基本原理为MySQL分库分表)

## 问题反馈

1. qq交流群： 537157866
2. 邮件交流： jianghang115@gmail.com, zylicfc@gmail.com
3. 新浪微博： agapple0002
4. 报告issue：[issues](https://github.com/alibaba/yugong/issues)


## Usage

```
mvn clean package
cp target/yugong-shaded.jar .
export JAVA_OPTIONS=-server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
java $JAVA_OPTIONS -jar yugong-shaded.jar -c yugong-product-dev-check.properties -y yugong-product.yaml
```