ESP8266 Losing packets

[drmpf]

I am coding a UART to WiFi bridge for ESP8266 running as a server.
Using build 1.6.5-947-g39819f0
I am finding that the ESP client is losing packets between the ESP TCP/IP stack and the client.available() method.

ESP Code is

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
// normally DEBUG is commented out
#define DEBUG
bool NoDelayFlag = true;
WiFiServer server(80);
WiFiClient client;
uint32_t timeout = 0;

void setup ( void ) {
  WiFi.mode(WIFI_STA);
  Serial.begin (9600);
  delay(10);
  Serial.println();
  for (int i = 4; i > 0; i--) {
#ifdef DEBUG
    Serial.print(i);
    Serial.print(' ');
#endif
    delay(500);
  }
#ifdef DEBUG
  Serial.println();
  Serial.println(F("Starting Setup"));
#endif
  char ssid[] = "xxxxxx"; 
  char password[] = "xxxxxx";
  char staticIP[] = "192.168.0.100";
  uint16_t portNo = 4989;
  timeout = 15000;

  server = WiFiServer(portNo);
  // Initialise wifi module
#ifdef DEBUG
  Serial.println(F("Connecting to AP"));
  Serial.print("ssid '");
  Serial.print(ssid);
  Serial.println("'");
  Serial.print("password '");
  Serial.print(password);
  Serial.println("'");
#endif
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
#ifdef DEBUG
    Serial.print(".");
#endif
  }
#ifdef DEBUG
  Serial.println();
  Serial.println(F("Connected!"));
#endif

  if (*staticIP != '\0') {
    // config static IP
    IPAddress ip(192,168,0,100);
    IPAddress gateway(192,168,0,1); // set gatway to ... 1
#ifdef DEBUG
    Serial.print(F("Setting gateway to: "));
    Serial.println(gateway);
#endif
    IPAddress subnet(255, 255, 255, 0);
    WiFi.config(ip, gateway, subnet);
  } // else leave as DHCP

  // Start listening for connections
#ifdef DEBUG
  Serial.println(F("Start Server"));
#endif
  server.begin();
  server.setNoDelay(NoDelayFlag);
#ifdef DEBUG
  Serial.println(F("Server Started"));
  // Print the IP address
  Serial.print(WiFi.localIP());
  Serial.print(':');
  Serial.println(portNo);
  Serial.println(F("Listening for connections..."));
#endif

  client = server.available();
#ifdef DEBUG
  Serial.print("+++"); // end of setup start listening now
#endif
}


static const size_t bufferSize = 128;
static uint8_t sbuf[bufferSize];
unsigned long timeoutTimerStart = 0;
const unsigned long SEND_DELAY_TIME = 10; // 10mS delay befor sending buffer

static const size_t SEND_BUFFER_SIZE = 1300; 
static uint8_t sendBuffer[SEND_BUFFER_SIZE+10];
size_t sendBufferIdx = 0;
unsigned long sendTimerStart = 0;
bool alreadyConnected = false;
// the loop routine runs over and over again forever:
void loop() {
  timeout = 45000;   //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<,  remove this later
  if (!client) { // see if a client is available
    client = server.available(); // evaluates to false if no connection
  } else {
    // have client
    if (!client.connected()) {
      if (alreadyConnected) {
        // client closed so clean up
        closeConnection();
      }
    } else {
      // have connected client
      if (!alreadyConnected) {
        client.setNoDelay(NoDelayFlag);
        alreadyConnected = true;
        sendBufferIdx = 0;
        // start timer
        timeoutTimerStart = millis();
      }
    }
  }

  //check UART for data
  if (Serial.available()) {
    size_t len = Serial.available();

    if (len > 0) { // size_t is an unsigned type so >0 is actually redundent
      sendTimerStart = millis();
      size_t will_copy = (len < (SEND_BUFFER_SIZE - sendBufferIdx)) ? len : (SEND_BUFFER_SIZE - sendBufferIdx);
      Serial.readBytes(sendBuffer + sendBufferIdx, will_copy);
      sendBufferIdx += will_copy;
      if (alreadyConnected && client) {
        if (sendBufferIdx == SEND_BUFFER_SIZE) {
          // buffer full
          client.write((const uint8_t *)sendBuffer, sendBufferIdx);
          client.flush();
          sendBufferIdx = 0;
        }
        delay(0); // yield
      }
    }
  }

  if (alreadyConnected && client) {
    if ((sendBufferIdx > 0) && ((millis() - sendTimerStart) > SEND_DELAY_TIME)) {
      client.write((const uint8_t *)sendBuffer, sendBufferIdx);
      client.flush();
      sendBufferIdx = 0;
    }
    delay(0); // yield
  }

  if (client) {
    while (client.available()) {
       timeoutTimerStart = millis();  // reset timer
      int in = client.read();
     if ((in == '{') || (in=='}')) {
       sendBuffer[sendBufferIdx++] = '*';
     } else {
       sendBuffer[sendBufferIdx++] = in;
     }      
      Serial.write(in);
      delay(0); // yield
    }
  }
  // see if we should drop the connection
  if (alreadyConnected && (timeout > 0) && ((millis() - timeoutTimerStart) > timeout)) {
    closeConnection();
  }
}

void closeConnection() {
  alreadyConnected = false;
  WiFiClient::stopAll(); // stop all clients and release memory.
  client = server.available(); // evaluates to false if no connection
}

Testing with Wireshark from Windows, I find the following
where the client sends {.} to the ESP which responds with
. from the ESP code (above) and . from the Arduino code consuming the ESP serial out. E.g.

{.}*.*_._.
{.Uno Control Application.
`1000|d~Set/Read Digital Pins|p~Plot Digital Inputs|a~Set/Read Analog Inputs|b~Plot Analog Inputs|s~Set 

AREF Analog Ref Volts|h~Help}

A longer sequence of msgs is

{.}*.*_._.
{.Uno Control Application.
`1000|d~Set/Read Digital Pins|p~Plot Digital Inputs|a~Set/Read Analog Inputs|b~Plot Analog Inputs|s~Set 

AREF Analog Ref Volts|h~Help}38.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}*.*_._.
{.Uno Control Application.
`1000|d~Set/Read Digital Pins|p~Plot Digital Inputs|a~Set/Read Analog Inputs|b~Plot Analog Inputs|s~Set 

AREF Analog Ref Volts|h~Help}39.00,0,0,0,0,0,0,0,0,0,0,0
39.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}*.*_._.
{.Uno Control Application.
`1000|d~Set/Read Digital Pins|p~Plot Digital Inputs|a~Set/Read Analog Inputs|b~Plot Analog Inputs|s~Set 

AREF Analog Ref Volts|h~Help}40.09,0.00,0.00,0.00,0.00,0.00,0.00
41.00,0,0,0,0,0,0,0,0,0,0,0
41.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}42.00,0.00,0.00,0.00,0.00,0.00,0.00
43.00,0,0,0,0,0,0,0,0,0,0,0
43.00,0.00,0.00,0.00,0.00,0.00,0.00
44.00,0.00,0.00,0.00,0.00,0.00,0.00

however the ESP looses the last {.} above and does not respond and the client closes the connection due to lack of response.
Checking the WireShark files shows that the {.} packet was acked by the ESP, but the bytes did not make it to the client.available() call.
client sends {.}
24 3.833327000 192.168.0.2 192.168.0.100 TCP 57 59029→4989 [PSH, ACK] Seq=10 Ack=714 Win=16807 Len=3
Esp acks {.}
25 3.950966000 192.168.0.100 192.168.0.2 TCP 54 4989→59029 [ACK] Seq=714 Ack=13 Win=5828 Len=0

This problem only seems to happen when the ESP is also sending data back i.e. the
41.00,0,0,0,0,0,0,0,0,0,0,0
etc.
If I disable the sending of that extra data, things seem to work.
Data and responses are provided by an Arduino board connected to the ESP serial TXRX
EDIT: Actually also loose packet data even when not sending extra data back.

[igrr]

I suppose the reason is that client.flush(); discards all data in RX buffer.

On Thu, Oct 22, 2015, 01:38 drmpf notifications@github.com wrote:

I am coding a UART to WiFi bridge for ESP8266 running as a server.
Using build 1.6.5-947-g39819f0
I am finding that the ESP client is losing packets between the ESP TCP/IP
stack and the client.available() method.

ESP Code is

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
// normally DEBUG is commented out
#define DEBUG
bool NoDelayFlag = true;
WiFiServer server(80);
WiFiClient client;
uint32_t timeout = 0;

void setup ( void ) {
WiFi.mode(WIFI_STA);
Serial.begin (9600);
delay(10);
Serial.println();
for (int i = 4; i > 0; i--) {
#ifdef DEBUG
Serial.print(i);
Serial.print(' ');
#endif
delay(500);
}
#ifdef DEBUG
Serial.println();
Serial.println(F("Starting Setup"));
#endif
char ssid[] = "xxxxxx";
char password[] = "xxxxxx";
char staticIP[] = "192.168.0.100";
uint16_t portNo = 4989;
timeout = 15000;

server = WiFiServer(portNo);
// Initialise wifi module
#ifdef DEBUG
Serial.println(F("Connecting to AP"));
Serial.print("ssid '");
Serial.print(ssid);
Serial.println("'");
Serial.print("password '");
Serial.print(password);
Serial.println("'");
#endif
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
#ifdef DEBUG
Serial.print(".");
#endif
}
#ifdef DEBUG
Serial.println();
Serial.println(F("Connected!"));
#endif

if (*staticIP != '\0') {
// config static IP
IPAddress ip(192,168,0,100);
IPAddress gateway(192,168,0,1); // set gatway to ... 1
#ifdef DEBUG
Serial.print(F("Setting gateway to: "));
Serial.println(gateway);
#endif
IPAddress subnet(255, 255, 255, 0);
WiFi.config(ip, gateway, subnet);
} // else leave as DHCP

// Start listening for connections
#ifdef DEBUG
Serial.println(F("Start Server"));
#endif
server.begin();
server.setNoDelay(NoDelayFlag);
#ifdef DEBUG
Serial.println(F("Server Started"));
// Print the IP address
Serial.print(WiFi.localIP());
Serial.print(':');
Serial.println(portNo);
Serial.println(F("Listening for connections..."));
#endif

client = server.available();
#ifdef DEBUG
Serial.print("+++"); // end of setup start listening now
#endif
}

static const size_t bufferSize = 128;
static uint8_t sbuf[bufferSize];
unsigned long timeoutTimerStart = 0;
const unsigned long SEND_DELAY_TIME = 10; // 10mS delay befor sending buffer

static const size_t SEND_BUFFER_SIZE = 1300;
static uint8_t sendBuffer[SEND_BUFFER_SIZE+10];
size_t sendBufferIdx = 0;
unsigned long sendTimerStart = 0;
bool alreadyConnected = false;
// the loop routine runs over and over again forever:
void loop() {
timeout = 45000; //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<, remove this later
if (!client) { // see if a client is available
client = server.available(); // evaluates to false if no connection
} else {
// have client
if (!client.connected()) {
if (alreadyConnected) {
// client closed so clean up
closeConnection();
}
} else {
// have connected client
if (!alreadyConnected) {
client.setNoDelay(NoDelayFlag);
alreadyConnected = true;
sendBufferIdx = 0;
// start timer
timeoutTimerStart = millis();
}
}
}

//check UART for data
if (Serial.available()) {
size_t len = Serial.available();

if (len > 0) { // size_t is an unsigned type so >0 is actually redundent
  sendTimerStart = millis();
  size_t will_copy = (len < (SEND_BUFFER_SIZE - sendBufferIdx)) ? len : (SEND_BUFFER_SIZE - sendBufferIdx);
  Serial.readBytes(sendBuffer + sendBufferIdx, will_copy);
  sendBufferIdx += will_copy;
  if (alreadyConnected && client) {
    if (sendBufferIdx == SEND_BUFFER_SIZE) {
      // buffer full
      client.write((const uint8_t *)sendBuffer, sendBufferIdx);
      client.flush();
      sendBufferIdx = 0;
    }
    delay(0); // yield
  }
}

}

if (alreadyConnected && client) {
if ((sendBufferIdx > 0) && ((millis() - sendTimerStart) > SEND_DELAY_TIME)) {
client.write((const uint8_t *)sendBuffer, sendBufferIdx);
client.flush();
sendBufferIdx = 0;
}
delay(0); // yield
}

if (client) {
while (client.available()) {
timeoutTimerStart = millis(); // reset timer
int in = client.read();
if ((in == '{') || (in=='}')) {
sendBuffer[sendBufferIdx++] = '*';
} else {
sendBuffer[sendBufferIdx++] = in;
}
Serial.write(in);
delay(0); // yield
}
}
// see if we should drop the connection
if (alreadyConnected && (timeout > 0) && ((millis() - timeoutTimerStart) > timeout)) {
closeConnection();
}
}

void closeConnection() {
alreadyConnected = false;
WiFiClient::stopAll(); // stop all clients and release memory.
client = server.available(); // evaluates to false if no connection
}

Testing with Wireshark from Windows, I find the following
where the client sends {.} to the ESP which responds with
. from the ESP code (above) and . from the Arduino code consuming the
ESP serial out. E.g.

{.}.__..
{.Uno Control Application.
`1000|dSet/Read Digital Pins|pPlot Digital Inputs|aSet/Read Analog Inputs|bPlot Analog Inputs|s~Set

AREF Analog Ref Volts|h~Help}

A longer sequence of msgs is

{.}.__..
{.Uno Control Application.
`1000|dSet/Read Digital Pins|pPlot Digital Inputs|aSet/Read Analog Inputs|bPlot Analog Inputs|s~Set

AREF Analog Ref Volts|hHelp}38.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}.__..
{.Uno Control Application.
`1000|dSet/Read Digital Pins|pPlot Digital Inputs|aSet/Read Analog Inputs|bPlot Analog Inputs|sSet

AREF Analog Ref Volts|hHelp}39.00,0,0,0,0,0,0,0,0,0,0,0
39.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}.__..
{.Uno Control Application.
`1000|dSet/Read Digital Pins|pPlot Digital Inputs|aSet/Read Analog Inputs|bPlot Analog Inputs|sSet

AREF Analog Ref Volts|h~Help}40.09,0.00,0.00,0.00,0.00,0.00,0.00
41.00,0,0,0,0,0,0,0,0,0,0,0
41.00,0.00,0.00,0.00,0.00,0.00,0.00
{.}42.00,0.00,0.00,0.00,0.00,0.00,0.00
43.00,0,0,0,0,0,0,0,0,0,0,0
43.00,0.00,0.00,0.00,0.00,0.00,0.00
44.00,0.00,0.00,0.00,0.00,0.00,0.00

however the ESP looses the last {.} above and does not respond and the
client closes the connection due to lack of response.
Checking the WireShark files shows that the {.} packet was acked by the
ESP, but the bytes did not make it to the client.available() call.
client sends {.}
24 3.833327000 192.168.0.2 192.168.0.100 TCP 57 59029→4989 [PSH, ACK]
Seq=10 Ack=714 Win=16807 Len=3
Esp acks {.}
25 3.950966000 192.168.0.100 192.168.0.2 TCP 54 4989→59029 [ACK] Seq=714
Ack=13 Win=5828 Len=0

This problem only seems to happen when the ESP is also sending data back
i.e. the
41.00,0,0,0,0,0,0,0,0,0,0,0
etc.
If I disable the sending of that extra data, things seem to work.
Data and responses are provided by an Arduino board connected to the ESP
serial TXRX

—
Reply to this email directly or view it on GitHub
#917.

[me-no-dev]

https://github.com/esp8266/Arduino/blob/esp8266/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/examples/WiFiTelnetToSerial/WiFiTelnetToSerial.ino

[me-no-dev]

try this and see if it works better

[drmpf]

@igrr I changed my code to

  if (client) {
    while ((client.available()>0) &&  (Serial.availableForWrite()>0)) {
      timeoutTimerStart = millis();  // reset timer if we have incoming data
      Serial.write(client.read());
      delay(0); // yield
    }
  }

But still loose packet data about once in 5 times.

I had look at the HardwareSerial.cpp flush() method.

    while(_tx_buffer->getSize() || uart_get_tx_fifo_room(_uart) < UART_TX_FIFO_SIZE)
        yield();

Looks to me like it just waits until all the data has been taken?

@me-no-dev Thanks for the link. Looks good for Telnet, but not so good for general streaming data. ESP only seems to handle on packet at a time outgoing on the fly and then waits for the ACK to come back before sending the next one. NoTCPDelay does not make any difference. So my code attempts to send filled packets if you are sending streaming data.
Also coming back, I think the WiFiTelentToSerial code will block at

while(serverClients[i].available()) Serial.write(serverClients[i].read());

for a full incoming packet (1460 bytes) and so not handle the incoming Serial data.

[drmpf]

@me-no-dev Also the Telnet to Serial code has problems with half closed connections. If the client goes away (dies or wifi breaks), the server stays connected to its client connection and the no clients, even the original one, can connect until the connection times out. Not good if you have flaky WiFi or if your client gets killed by the user.

[igrr]

I thought your code called WiFiClient::flush, not HardwareSerial::flush. Unlike HardwareSerial::flush, WiFiClient::flush will discard all RX data.

Will see if i can write a sketch to reproduce your issue.

[drmpf]

@igrr OK, let me remove the client.flush() as well it may be causing problem.
Also I tested TelnetToSerial sketch and it runs well, so I will do some line be line changes and see I can locate the magic statement.

[igrr]

Just to make it more clear, client.write() blocks until either data is sent and ACKed, or timeout occurs (currently hard-coded to 5 seconds). So client.flush() doesn't "force" sending data after client.write(), because there is no output buffer which can be flushed. It does discard incoming data though.

[drmpf]

OK that would be the magic statement. The client.flush() would drop discard my incoming data :-(

Rather than having the sketch blocked for 5sec, is there some way to tell if last packet has been sent and Acked. Acking takes about 0.2sec with windows client.

That way I could buffer incoming serial until client available to take next packet.

[igrr]

Sketch is blocked until either packet is acked, or timeout expires. So if the packet is ACKed in 0.2 seconds, sketch will not get blocked for 5 sec.

While there is no "nonblocking write" API in WiFiClient, you may use LwIP APIs directly to process data in an event-driven manner.

[drmpf]

Thanks, was hoping to avoid the 0.2sec delay. At 115200 that is about 2K of incoming Serial data the sketch is not handling.
So was looking for something like
if (client.canWrite()) {....

[igrr]

This isn't available since ESP8266WiFi library API is a blocking one, like WiFi Shield library.
I would propose to close this issue (if it is resolved for you) and create a new feature request for an asynchronous WiFi library.

[me-no-dev]

200ms seems a bit high. I am able to make 8ms request/response times with the http server which does parsing and whatnot. Other than that I get your problem and I'm currently in the same boat, but i'm looking for a way without timeout

[drmpf]

@igrr Sure I will close this. thanks again and great work.
@me-no-dev 0.2sec is what you get from Windows, checked with wireshark, some Linux systems are 40mS. Because ESP only handles one packet at a time, it needs to keep the last packet for retransmission until it is acked. If you want to stack up packets you need to do it in your sketch.
But you still are faced with the ACK delay. No way around that.