STM32 Cache coherence

[mulcmu]

@PeterChristen577 First off, thanks for contributing the STM32 port. Below problem was encountered running on a H7 cpu.

I got the demo running fine on the STM32F746-Disco board. I imported OpENer successfully into a new STM32F746-Disco TouchGFX template project. Needed one small change for freeRTOS CMSIS_V2 vs _V1. Both these were tested with enip/explorer and wireshark. Explicit and implicit messaging working perfect.

I imported into a STM32H745I-Disco TouchGFX template project. Got everything setup for this board but started getting some random data in the enip/explorer implicit messaging. Wireshark showed a few good packets were transmitted and then one or two malformed packets would be sent. Troubleshooting this lead to what appears to be a cache coherence issue.

In cipioconnection.c SendConnectData(...) I changed the the outgoing message to

static ENIPMessage outgoing_message __attribute__((section(".outgoing_message_hack")));

and stuck this with the non-cached lwip heap memory. This got the sample application running on this board without any malformed io packets (at least for single connection). The cpu frequency was higher on this board and the ETH is MII vs RMII which might be contributing factors. I didn't check the F746 to see if malformed packets were still occurring but just way less frequently.

Any ideas for a more elegant solution than the above hack?

[PeterChristen577]

Hi mulcmu

Same questions/ideas:

• Which heap size have you configured in FreeRTOSConfig.h?
• What is the MEM_SIZE in lwiopts.h
• Did you have a look to the document OpENer STM32 Port.pdf?
  There are some hint for setup of lwiopts.h

Best regards Peter

[PeterChristen577]

... another thought:
increase OPENER_STACK_SIZE in opener.c, is currently at 2000

[mulcmu]

The freeRTOS heap size is #define configTOTAL_HEAP_SIZE ((size_t)0x8000)

lwip heap is 96k. First 32k of D2 sram is used for Ethernet dma descriptors and rx buffers, rest of the 128k is used for lwip.
#define MEM_SIZE 0x18000

The graphics buffers are in the external 16mb SDRAM.

Here's snapshot of the stack sizes. OpENer was already increased to (4*1024) as well as increases to the default lwip tasks.

[PeterChristen577]

Everything looks good in the stack snapshot, let me do a similar analysis on the STM32F746 target.
One question: Which cycle time have you set up in enip/explorer? Is the rate of malformed packet related to the cycle time?

[mulcmu]

Cycle time was the default 200ms. At 200ms the malformed packets appeared randomly distributed, I didn't see any patterns.

This post on ST community forum discusses similar problem. Seems that on my H745, sometimes the processor cached data is written out to the memory before the DMA reads it for the valid packets. For the malformed packets the outgoing message buffer data is only in the MCU cache, so the DMA is copying the SRAM memory that holds random data.

OpENer/source/src/ports/generic_networkhandler.c

Lines 682 to 719 in 70d6947

	EipStatus SendUdpData(const struct sockaddr_in *const address,
	const ENIPMessage
	*const outgoing_message) {
	#if defined(OPENER_TRACE_ENABLED)
	static char ip_str[INET_ADDRSTRLEN];
	OPENER_TRACE_INFO(
	"UDP packet to be sent to: %s:%d\n",
	inet_ntop(AF_INET, &address->sin_addr, ip_str, sizeof ip_str),
	ntohs(address->sin_port) );
	#endif
	int sent_length = sendto( g_network_status.udp_io_messaging,
	(char *)outgoing_message->message_buffer,
	outgoing_message->used_message_length, 0,
	(struct sockaddr *) address, sizeof(*address) );
	if(sent_length < 0) {
	int error_code = GetSocketErrorNumber();
	char *error_message = GetErrorMessage(error_code);
	OPENER_TRACE_ERR(
	"networkhandler: error with sendto in SendUDPData: %d - %s\n",
	error_code,
	error_message);
	FreeErrorMessage(error_message);
	return kEipStatusError;
	}
	if(sent_length != outgoing_message->used_message_length) {
	OPENER_TRACE_WARN(
	"data length sent_length mismatch; probably not all data was sent in SendUdpData, sent %d of %d\n",
	sent_length,
	outgoing_message->used_message_length);
	return kEipStatusError;
	}
	return kEipStatusOk;
	}

I reverted my prior hack and then on line 693 of the SendUdpData() an explicit cache flush was added.

  SCB_CleanDCache_by_Addr(outgoing_message->message_buffer, sizeof(ENIPMessage));

This change also resolved the malformed packet issue for the implicit messaging.

[PeterChristen577]

Great, that you found out this so quickly!
If I understand correctly the situation this cache flush is needed only for H7 targets and not for F7, isn't it?

[mulcmu]

I was still working through things, I have 2 H7 (H757 & H745), an H563, and the F746. The clean cache() function doesn't seem to hurt the F746 but there were a few times ping, http, and list identity would work but then the rest of the ethernet/IP appeared with errors. Still trying to figure out what was changing. I started a branch with the changes. https://github.com/mulcmu/OpENer/tree/stm32-flush-cache-CMSISv2.

[elupus]

I think you are looking at this in the wrong location. I don't think OpENer is using DMA. If there is a problem with cache consistency here, it is in the ethernetdriver for your board. It should ensure memory is flushed from cpu cache if it is intending to copy it using some dma transfer.