error E08 Synchronization loss fault

[gbreahna]

Hi guys,
I am trying to make an Inovance IS620N to work, using redtest model, but after 2 weeks I am ending into
Er.E08 (Synchronization loss fault)
Attached is the C code, ESI file, and a Wireshark dump.

red2-202104081738.pcapng.gz
red2.c.txt
03024278-IS620N-Ecat_v2.6.8.xml.txt
Please help me with this issue,
Regards,
Gabriel

[ArthurKetels]

From the screen shot I can see that the DCtime jitter is quite a lot. You are running 200us cycle time and a visible jitter of 120us. This will indeed not work. Improve your timing. The ec_sync function you copied is not the best for high performance. Replace it with a PI controller and tune the gains appropriately. You want to have below 30us average deviation with peaks below 50us. This is not easy to do on most hardware and a standard OS.

The first question here would be, do you need 200us cycle time? If you drop to 1000us the tolerances are more bearable.

[gbreahna]

I am using 1000 us cycle time. I double checked all values and maybe the confusion is because of the 20 ms output/printf cycle time.

1. if you divide the DCtime between two lines with the number of cycles (column after pdc), then avg=1000 us, max=1000.402, min=999.7005, stdev=0.1124
2. for jitter avg=1.11 us, max=63.8, min=-34.7, stdev=17.68
   My values seems to be perfect, but the error points to your values - please let me know where did you see 200 us for cycle time

[ArthurKetels]

Ehh, brain fart from my side. Lack of coffee I guess.
So I looked at the Wireshark data with some simple filters. See snippet below:

No.	Time	Source	Destination	Protocol	Length	Info
12475	0.001004083	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12477	0.001000024	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12479	0.000994394	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12481	0.001007778	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12483	0.000999728	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12485	0.000967984	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12487	0.001854841	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12489	0.000139607	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12491	0.001485909	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12493	0.000521613	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12495	0.001003782	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 
12497	0.001002268	Private_01:01:01	Broadcast	ECAT	88	2 Cmds, 'LRW': len 40, 'FRMW': len 8 

It shows the delta time between each LRW. According to your measurements the max. jitter is 63.8us. But from the sample above you can see packet no 12487 is 854.8us late. It does not happen often in your trace but there are more. Some servo slaves keep a simple counter for sync errors. When you reach the threshold it throws an error. It would have been better if they implemented it as a leaky integrator so it would forget old mishaps. But I guess you are unlucky with this slave.

But the data you provided combined with the raw data provides some clues. Most likely your system is hit by SMI.
https://www.kernel.org/doc/html/latest/trace/hwlat_detector.html
Another possibility is that the CPU for your rt-task is going into a deeper C-state. If it is SMI you are toast, when it is C-state then you could disable it by setting the proper system parameters.

[gbreahna]

Still working. Seems the servo has an old firmware. The factory will ship me the last version.

[gbreahna]

After
ec_dcsync0(1, TRUE, cycletime, 0);
usleep(10000000);
READ(1, 0x1c32, 0x01, buf16, "Synchronization type");
READ(1, 0x1c32, 0x02, buf32, "Cycle time");
READ(1, 0x1c33, 0x01, buf16, "Synchronization type");
READ(1, 0x1c33, 0x02, buf32, "Cycle time");
I am receiving
Slave: 1 - Read at 0x1c32:1 => wkc: 1; data: 0x01 (1) [Synchronization type]
Slave: 1 - Read at 0x1c32:2 => wkc: 1; data: 0x0000 (0) [Cycle time]
Slave: 1 - Read at 0x1c33:1 => wkc: 1; data: 0x22 (34) [Synchronization type]
Slave: 1 - Read at 0x1c33:2 => wkc: 1; data: 0x0000 (0) [Cycle time]
instead of 2 (0x1c32:1, 0x1c33:1) and 2000000 (0x1c32:2, 0x1c33:2) as TwinCAT has.
And is quite possible to have sync error because I have sync with a bad value.
Any idea?

[ArthurKetels]

Well, assumed your code still looks like something in your first post I am not surprised it does not work. You can not simply dump the slave in OP then start-up sync-0 and hope for the best. Please read #487 (comment)
Implement all steps carefully and then try again.

[gbreahna]

Thanks. Is too complicated. I will do it with TwinCAT.

[gbreahna]

I did it with TwinCAT, but I did it with SOEM too. The only change I did was to do it like below:
ec_dcsync0(slave, TRUE, CYCLE_TIME, 0);
ec_configdc();
ec_config_map(&IOmap);
Thank you @ArthurKetels