Upload new firmware over Serial

[mikeme2009]

I use an esp32 that communicates with the Mega2560 on Serial (Esp32 - Serial2 and Mega - Serial3). I need to load a new firmware on the Mega2560 using serial communication. I use these 2 functions to download FW from http and save it in SPIFFS from esp32. How should I modify the optiboot so that I can load the new FW on the Mega? Thank you
void checkUpdateFirmwareMega2560() {
String payload;
Serial.println("Starting Mega2560 FW Update");

DynamicJsonDocument doc(288);
doc["mi"] = EspId;
doc["ip"] = EspIpAddress;
doc["Mega_firmware_id"] = firmwareId;

// Serialize JSON document
String json;
serializeJson(doc, json);
doc.clear();

HTTPClient http;
http.begin(mega2560FirmwareAddress);
http.addHeader("Content-Type", "application/json");

int httpCode = http.POST(json);

if (httpCode > 0) {
if (httpCode == HTTP_CODE_OK) {
payload = http.getString();
DynamicJsonDocument payloadDoc(payload.length() + 200);
deserializeJson(payloadDoc, payload);
bool updateFirmware = payloadDoc["updateFirmware"];
if (updateFirmware) {
String megaURL = payloadDoc["latestFirmware"]["firmware_link"];
Serial.println("Update available");

    // Update Arduino Mega Firmware
    if (!megaURL.isEmpty()) {
      Serial.println("Downloading firmware");
      HTTPClient firmwareHttp;
      firmwareHttp.begin(megaURL);
      int firmwareHttpCode = firmwareHttp.GET();
      if (firmwareHttpCode == HTTP_CODE_OK) {
        WiFiClient *stream = firmwareHttp.getStreamPtr();
        File firmwareFile = SPIFFS.open("/mega_firmware.bin", FILE_WRITE);
        if (!firmwareFile) {
          Serial.println("Failed to open file for writing");
          return;
        }

        while (firmwareHttp.connected() && stream->available()) {
          size_t size = stream->available();
          if (size) {
            uint8_t *buf = (uint8_t *)malloc(size);
            if (buf) {
              size_t bytesRead = stream->readBytes(buf, size);
              if (bytesRead > 0) {
                firmwareFile.write(buf, bytesRead);
                Serial.printf("Written %d bytes to file\n", bytesRead); // Debugging line
              }
              free(buf);
            } else {
              Serial.println("Failed to allocate memory");
              break;
            }
          }
        }

        firmwareFile.close();
        firmwareHttp.end();
        Serial.println("Firmware download complete");
        sendFirmwareToArduinoMega();
      } else {
        Serial.printf("Failed to download firmware, HTTP code: %d\n", firmwareHttpCode);
      }
    } else {
      Serial.println("Firmware URL is empty");
    }
  } else {
    Serial.println("No update needed");
  }
} else {
  Serial.printf("Failed to get update info, HTTP code: %d\n", httpCode);
}

} else {
Serial.printf("Failed to send POST request, HTTP code: %d\n", httpCode);
}
http.end();
startafterfirmware = true;
}

void sendFirmwareToArduinoMega() {
File firmwareFile = SPIFFS.open("/mega_firmware.bin", FILE_READ);
if (!firmwareFile) {
Serial.println("Failed to open firmware file");
return;
}

Serial2.println("START");
delay(1000); // Wait for the Mega to be ready

while (firmwareFile.available()) {
uint8_t buf[64];
size_t len = firmwareFile.read(buf, sizeof(buf));
Serial2.write(buf, len);
delay(1); // Small delay to prevent overflow
Serial.printf("Sent %d bytes to Arduino\n", len); // Debugging line
}
Serial2.println("END");
Serial.println("Sent END to Arduino"); // Debugging line

firmwareFile.close();
}

[WestfW]

Optiboot uses a protocol that is more complicated than simply sending the bytes over the serial port. The "host" (ESP32 in your case) checks the chip type and sends blocks of binary bytes, each tagged with the address where it is supposed to go, and waits for an acknowledgement that that block has been written, and well as commands to indicate that it is "done."
This is called the STK500v1 protocol; there are links to the specification in the Optiboot Wiki.
You could certainly implement this in Python, and I'm pretty sure that you could find an existing implementation that might be easily ported to circuitPython or MicroPython.