Skip to content

DTLE: Memory error,when the amount of data is too large,the dtle is 'panic' #462

Closed
@yaqigithub

Description

@yaqigithub

Description
When table has a large data,change some data ,the dtle is 'panic'.

Steps to reproduce the issue:
1.create a table in Src schema whit 100w data ,

create database dataTest;
use dataTest;
create table `tl`( `id` int(10) NOT NULL, `txt` VARCHAR(50) NOT NULL, `student` VARCHAR(50) NOT NULL, PRIMARY KEY (`id`));
delimiter //
CREATE PROCEDURE tl()
     begin
    declare v int default 1;
     while v <1000001
     do 
     insert into tl 
     values(v,'ttt','hg');
     set v=v+1;
     end while;
     end 
     //
delimiter ;
call  tl();

2.create a schema level job
job.json.txt

3.execute some DML
UPDATE tl set txt=REPLACE(txt,'yyy','huhu');

Describe the results you received:
The dtle is 'panic'

Describe the results you expected:
Gave some errorlog ang dtle is normal operation
Output of ./dtle version:

Dtle 2.19.05.0 (git: 2.19.05.0 6597c279707070cf5217082a229e4b592788ea2d)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):
1.

2019/06/18 11:14:18.840093 [INFO] [jobid:5e21c11c-d9d4-3973-373b-7818599454a0] mysql.applier: Initiated on 172.100.9.2:3306, version 5.7.23-log 
2019/06/18 11:14:18.840153 [ERR] [jobid:5e21c11c-d9d4-3973-373b-7818599454a0] mysql.applier: Can't connect nats server nats://. make sure a nats streaming server is 
running.nats: no servers available for connection 
2019/06/18 11:14:18.840209 [DEBUG] setState 4 
2019/06/18 11:14:18.840235 [DEBUG] setState.SaveState 
2019/06/18 11:14:18.840302 [INFO] [jobid:5e21c11c-d9d4-3973-373b-7818599454a0] mysql.applier: Shutting down 
2019/06/18 11:14:18.840391 [DEBUG] Worker.SaveState: lock: 0xc00039a310, 0xc000018d60 
2019/06/18 11:14:18.840402 [DEBUG] Worker.SaveState: after lock: 0xc00039a310 
2019/06/18 11:14:18.840413 [DEBUG] Worker.SaveState: after unlock: 0xc00039a310 
2019/06/18 11:14:18.840421 [DEBUG] updater 
2019/06/18 11:14:18.840443 [ERR] agent: Task "Dest" for alloc "3bbe9e9f-4ebd-1bef-08bb-45264c5637c1" failed: Wait returned exit code 2, and error nats: no servers av
ailable for connection 
2019/06/18 11:14:18.840456 [INFO] agent: Not restarting task: Dest for alloc: 3bbe9e9f-4ebd-1bef-08bb-45264c5637c1  
2019/06/18 11:14:18.840464 [DEBUG] setState restart 1 
2019/06/18 11:14:18.840474 [DEBUG] setState.SaveState 


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions