Profile velocity mode works fine, but Cyclic sync velocity mode doesn't move

[Linshiyuan]

Hi,

I met a strange problem when trying to control two servo drivers.

I modify the ebox.c example and let the two drivers works in Profile velocity mode ( OPmode 0x6060=3 ). When I change to cyclic sync velocity mode ( OPmode 0x6060=9 ), the driver reach state OP and switched on, but the motor didn't move. I use pc software of the driver to check the register and find the target velocity(0x60FF ) is changed over time, it seems the PBO works fine.

I guess there is something wrong with Distributed Clock settings?

But it can reach op and switched op, I am very confused now.

My code and running result:
move2driver.c.txt
output of move2driver.txt

./slaveinfo -map :
output of Slaveinfo.txt

slave's ESI file:
Stone -E XML_04.23(PC).xml.zip

Your help is very much appreciated!

[ArthurKetels]

You are almost there. In the ESI file the AssignActivate value is 0x0700. This means 0x00 has to be written to ESC register 0x0980 and 0x07 to ESC register 0x0981. This will configure both SYNC0 and SYNC1. You only activate SYNC0.

  ec_dcsync0(1, TRUE, 1000 * 1000, 500 * 1000); // SYNC0 on slave 1
  ec_dcsync0(2, TRUE, 1000 * 1000, 500 * 1000); // SYNC0 on slave 2 

Should be changed to:

  ec_dcsync01(1, TRUE, 1000 * 1000, 1000 * 1000, 0); // SYNC0/1 on slave 1
  ec_dcsync01(2, TRUE, 1000 * 1000, 1000 * 1000, 0); // SYNC0/1 on slave 2 

Cycleshift parameter does not need to be set.

A remark, your code is probably going to work, but it is better to enable DC and sync and stabilization at pre-OP.

[Linshiyuan]

Thanks for the reply!
I try to use ec_dcsync01 but the motor still doesn't move. Then I try to enable DC and sync after reach pre-OP, still not work...
Anything else seems ok, the driver can enter OP and Switched-On, the target velocity register(0x60FF ) is changed during time.
move2driver_2.c.txt

Here is the screenshot of driver's pc software:

[Linshiyuan]

I try to move the DC and ec_dcsync 's code around and found they working at a very strange position

 if (ec_config_init(FALSE) > 0)
        {
            printf("%d slaves found and configured.\n", ec_slavecount);

//////////////////////////////      both dcsync0 and dcsync01 works

            // ec_configdc();          //without it is still ok

            ec_dcsync0(1, TRUE, 1000 * 1000, 0 ); // SYNC0/1 on slave 1
            ec_dcsync0(2, TRUE, 1000 * 1000, 0 ); // SYNC0/1 on slave 2     
            // ec_dcsync01(1, TRUE, 1000 * 1000, 1000 * 1000, 0); // SYNC0/1 on slave 1
            // ec_dcsync01(2, TRUE, 1000 * 1000, 1000 * 1000, 0); // SYNC0/1 on slave 2

            ec_config_map(&IOmap);   // the order can't change, it must be after the ec_dcsync
            ec_config_init(FALSE);         //  can not reach OP without it, 

/////////////////////////////////

            // do some PDO mapping
            if ((ec_slavecount >= 1))
            {
                for (int i = 1; i <= ec_slavecount; i++)
                {
                  ****
                }
            }

             // do ec_config_map and ec_configdc again
            ec_config_map(&IOmap);
            ec_configdc();
            
            /* wait for all slaves to reach SAFE_OP state */
            ec_statecheck(0, EC_STATE_SAFE_OP, EC_TIMEOUTSTATE);

          // then all the code keep the same and finally the driver reach OP and works in CSV mode.

Here is the code:
move2driver4.c.txt

I don't know why they get worked here. What is the correct way to write the DC and ec_dcsync ?

Waiting for a your response, thank you.

[Linshiyuan]

It's not real working....I check the SYNC manager 2 parameter 0x1c32:01, it is 01 which is Synchronous: Synchronous (with SM2 event).

I use twincat to run a demo, the 0x1c32 is 02 and works fine.

Here is the wireshark data
soem
soem_1.zip

twincat
twincat_DC.zip

[Linshiyuan]

@ArthurKetels
I find some useful information in Logiclab PLC(it has a built in EtherCAT master). It can run smoothly in CSV mode and the 0x1c32 is 02.

The startup cmd is:

I think the problem is still at the DC and sync settings.

Should I change the ec_dcsync0 or ec_dcsync01 code to make it send the same cmd to let the DC working?

Any suggestion is appreciated.

[ArthurKetels]

Throwing mud to the wall to see if it sticks is not advisable to solve your problems. You need a better understanding.
I have written an in depth explanation about DC and SYNC here : #487 (comment)
Please follow all steps and check if you implemented them correctly. Most likely your problem will be solved.

[Linshiyuan]

Dear sir,

Huge thanks for your depth explanation, after reading that I finally get correct DC settings and let the driver working in csv mode.

I found ec_dcsync01 works in the following parameter. CylcTime1 should be 0.

static int slave_dc_config(uint16 slave)
{
  ec_dcsync01(1, TRUE, 1000 * 1000, 0, 0);
  return 0;
}

If CylcTime1=CylcTime0=1000, the 1c32:01 will be 3. There is no mode 3 in my driver so this won't works.
ec_dcsync01(1, TRUE, 1000 * 1000, 1000 * 1000, 0)

With the PREOP->SAFEOP hooks #142 , the ec_dcsync01 correctly set the 1c32:01 to 2 which is DC sync mode in my case.

...
ec_configdc();
...
for(cnt = 1; cnt <= ec_slavecount ; cnt++)
{
  if(ec_slave[cnt].hasdc > 0)
  {
    ec_slave[cnt].PO2SOconfig = slave_dc_config;
  }
}
ec_config_map(&IOmap);
...

If don't use the PREOP->SAFEOP hooks, dorun should be 1 after ec_configdc.

...
ec_configdc();
dorun = 1;
usleep(10000);
...
for(cnt = 1; cnt <= ec_slavecount ; cnt++)
{
  if(ec_slave[cnt].hasdc > 0)
  {
    ec_dcsync01(1, TRUE, 1000 * 1000, 0, 0);
  }
}
ec_config_map(&IOmap);
...

Then I try to test the real time performance. I use a FPGA based capture tool(which can take ns timestamp) and wireshark combine to catch 30 second data in OP. My CPU is i7-9700, NIC is intel82575(TwinCAT support hardware), debian 10 with preempt rt patch. The linux time is synced to DC time using the same code in red_test.c.

Most of time the jitter is within 50us, but there is still a lot of frames have 100~150us jitter.
I think it may be too big. Comparing to twincat use the same hardware, the jitter is within 20us most of the time.

How could I improve the real time performance?

Thanks.

[ArthurKetels]

Well, congratulations with your results. I am happy my write-up helped you in the right direction.

About Linux real-time performance. This is not so easy as you have noticed. Although this has nothing to do with SOEM or EtherCAT, here some pieces of advice.

Luckily you attached a graph with the measured timing latencies. Good data is always the basis for analysis and improvement. You can see 3 bands in your graph. Each separated almost a perfect 50us. Also larger deviations happen in bursts. This is most likely due to your microprocessor going into deeper C states to conserve energy. Waking up costs some time.

There is a tunable in the kernel to prevent this C state regulation. See : https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b
You can run this program along side your SOEM program or integrate it into your program. Set the max latency to 0 or 5. Zero might be a bit aggressive, but gives the best results.

Other things to try is to isolate a CPU to use only for SOEM and the NIC driver. Then you are sure no other programs are running on this CPU. This is actually what TwinCAT does. It runs a real-time OS next to windows on its own CPU(s).

A source of latency might be a GPU driver. The open linux drivers are mostly fine, but the proprietary versions are know for latency spikes.

Make sure you do not have devices on your system that block the system bus for a long time. Known devices are PS/2 keyboard and mouse, USB1 mouse, graphic cards. In general, clean-up the hardware in your system as much as possible. Disable all unnecessary drivers.

[Linshiyuan]

Thanks for your suggestions.

I try to prevent the C state regulation using the example code with max latency to 0 and run my program on a isolated CPU core. And unplug all the unused devices.

The jitter is dropped significantly. Even the biggest deviation is below 30us. Most of the time it is within 3us.

Then I run a 30 mins test to see if it is stable. There is warning in the terminal after the test.

I think everything may be ok now?

[Linshiyuan]

When I did another 30mins test, I got a few warnings in the terminal. They are not very often, 5~7 mins to got one. Then after about 10ms the driver go back to OP mode and keep moving.
WARNING : slave 1 is in SAFE_OP, change to OPERATIONAL.
WARNING : slave 2 is in SAFE_OP, change to OPERATIONAL.
May be there are still too much deviations for a stable ethercat communication?

[Linshiyuan]

After one week testing and debugging, I still have the random warnings . The driver fall into Safe OP during OP. Sometimes after a few mins to get the warnings, sometimes after a few hours(4~5). I have no idea now.
Any suggestion?
Thanks.

[ArthurKetels]

You will need to instrument your system to figure out what goes wrong and why. Google around to figure out the linux trace functions, and how to use them on the network stack.

Another approach is to split your system in VM's with a RT-hypervisor. One example:
https://projectacrn.github.io/latest/getting-started/roscube/roscube-gsg.html

Lastly you have to make sure to rule-out hardware stability issues. Is your cabling OK, your power supply, tested with other slaves, etc.?

[ArthurKetels]

A good video explaining general options to improve real-time behavior
https://linutronix.de/videos/A_Checklist_for_Writing_L-John_Ogness.mp4

[nakarlsson]

@Linshiyuan , can this issue be closed?