PropLoader thinks .eeprom file is corrupt or not valid Propeller Application

[PropGit]

The .eeprom image in the attached archive is generated by OpenSpin v1.0.78. It is a valid image (downloadable via Propellent and Propeller Tool) but PropLoader says "ERROR: File is corrupt or not a Propeller application."

ValidEEActsInvalid.zip

It does, however, work properly with a .binary image generated from the same source with the same OpenSpin compiler.

C:\Scribbler-3_Library-master>proploader -p COM33 ExampleS3.eeprom
Opening file 'ExampleS3.eeprom'
ERROR: File is corrupt or not a Propeller application

C:\Scribbler-3_Library-master>proploader -p COM33 ExampleS3.binary
Opening file 'ExampleS3.binary'
Downloading file to port COM33
14060 bytes remaining
13036 bytes remaining
12012 bytes remaining
10988 bytes remaining
9964 bytes remaining
8940 bytes remaining
7916 bytes remaining
6892 bytes remaining
5868 bytes remaining
4844 bytes remaining
3820 bytes remaining
2796 bytes remaining
1772 bytes remaining
748 bytes remaining
14060 bytes sent
Verifying RAM
Download successful!

[dbetz]

proploader doesn't know anything about .eeprom files currently.

[PropGit]

I did not realize that.

[dbetz]

propeller-load doesn't have this problem because it doesn't have the image validation code that we have in proploader. It just loads whatever bytes are in the file and doesn't worry about the exact length. It might complain about files bigger than 32K but that's about it.

[PropGit]

Understood. The same image verification code should apply to both .binary and .eeprom images; at least in the simple sense that we've used for Propeller Tool and Propellent, where we don't expect or allow non-zero data past the end of the executable code image.

There must be something in the image verification code in PropLoader that is doing a different logical check and deciding the image is bad based on that. Here's the exact Delphi code from Propeller Tool and Propellent.

procedure ValidateImageDataIntegrity(Buffer: PByteArray; ImageSize: Integer; Filename: String);
{Validate Propeller application image data integrity in Buffer.  This is done through a series of tests that verify that the file is not too small.
 Also installs initial call frame.

 PROPELLER APPLICATION FORMAT:
   The Propeller Application image consists of data blocks for initialization, program, variables, and data/stack space.  The first block, initialization, describes the application's
   startup paramemters, including the position of the other blocks within the image, as shown below.

     long 0 (bytes 0:3) - Clock Frequency
     byte 4             - Clock Mode
     byte 5             - Checksum (this value causes additive checksum of bytes 0 to ImageLimit-1 to equal 0)
     word 3             - Start of Code pointer (must always be $0010)
     word 4             - Start of Variables pointer
     word 5             - Start of Stack Space pointer
     word 6             - Current Program pointer (points to first public method of object)
     word 7             - Current Stack Space pointer (points to first run-time usable space of stack)

 WHAT GETS DOWNLOADED:
   To save time, the Propeller Tool does not download the entire Propeller Application Image.  Instead, it downloads only the parts of the image from long 0 through the end of code (up to the
   start of variables) and then the Propeller chip itself writes zeros (0) to the rest of the RAM/EEPROM, after the end of code (up to 32 Kbytes), and inserts the initial call frame in the
   proper location.  This effectively clears (initializes) all global variables to zero (0) and sets all available stack and free space to zero (0) as well.

 INITIAL CALL FRAME:
   The Initial Call Frame is stuffed into the Propeller Application's image at location DBase-8 (eight bytes (2 longs) before the start of stack space).  The Propeller Chip itself stores the
   Initial Call Frame into those locations at the end of the download process.  The initial call frame is exactly like standard run-time call frames in their format, but different in value.

         Call Frame Format:  PBase VBase DBase PCurr Return Extra... (each are words arranged in this order from Word0 to Word3; 4 words = two longs)
   Initial Call Frame Data:  $FFFF $FFF9 $FFFF $FFF9  n/a    n/a

   Note: PBase is Start of Object Program, VBase is Start of Variables, DBase is Start of Data/Stack, PCurr is current program location (PC).

   The Initial Call Frame is stuffed prior to DBase so that if one-too-many returns are executed by the Spin-based Propeller Application, the Initial Call Frame is popped off the stack next
   which instructs the Spin Interpreter to jump to location $FFF9 (PCurr) and execute the byte code there (which is two instructions to perform a "Who am I?" followed by a cog stop, "COGID ID"
   and "COGSTOP ID", to halt the cog).  NOTE: The two duplicate longs $FFFFFFF9 are used for simplicity and the first long just happens to set the right bits to indicate to an ABORT command
   that it has reached the point where it should stop popping the stack and actually execute code.

   Note that a Call Frame is followed by one or more longs of data.  Every call, whether it be to an inter-object method or an intra-object method, results in a call frame that consists
   of the following:

         Return Information     : 2 longs (first two longs shown in Call Frame Format, above, but with different data)
         Return Result          : 1 long
         Method Parameters      : x longs (in left-to-right order)
         Local Variables        : y longs (in left-to-right order)
         Intermediate Workspace : z longs

   The first four items in the call stack are easy to determine from the code.  The last, Intermediate Workspace, is much more difficult because it relates directly to how and when the
   interpreter pushes and pops items on the stack during expression evaluations.}

var
  Idx      : Integer;
  CheckSum : Byte;
begin
  {Raise exception if file truncated}
  if (ImageSize < 16) or (ImageSize < PWordArray(Buffer)[4]) then
    raise EFileCorrupt.Create(ifthen(Filename <> '', 'File '''+Filename+'''', 'Image') + ' is truncated or is not a Propeller Application' + ifthen(Filename <> '',' file', '') + '!'+#$D#$A#$D#$A+ifthen(Filename <> '', 'File', 'Image') + ' size is less than 16 bytes or is less than word 4 (VarBase) indicates.');
  if (PWordArray(Buffer)[3] <> $0010) then
    raise EFileCorrupt.Create('Initialization code invalid!  ' + ifthen(Filename <> '', 'File '''+Filename+'''', 'Image') + ' is corrupt or is not a Propeller Application' + ifthen(Filename <> '',' file', '') +'!'+#$D#$A#$D#$A+'Word 3 (CodeBase) must be $0010.');
  {Write initial call frame}
  copymemory(@Buffer[min($7FF8, max($0010, PWordArray(Buffer)[5] - 8))], @InitCallFrame[0], 8);
  {Raise exception if file's checksum incorrect}
  CheckSum := 0;
  for Idx := 0 to ImageLimit-1 do CheckSum := CheckSum + Buffer[Idx];
  if CheckSum <> 0 then
    raise EFileCorrupt.Create('Checksum Error!  ' + ifthen(Filename <> '', 'File '''+Filename+'''', 'Image') + ' is corrupt or is not a Propeller Application' + ifthen(Filename <> '',' file', '') +'!'+#$D#$A#$D#$A+'Byte 5 (Checksum) is incorrect.');
  {Check for extra data beyond code}
  Idx := PWordArray(Buffer)[4];
  while (Idx < ImageSize) and ((Buffer[Idx] = 0) or ((Idx >= PWordArray(Buffer)[5] - 8) and (Idx < PWordArray(Buffer)[5]) and (Buffer[Idx] = InitCallFrame[Idx-(PWordArray(Buffer)[5]-8)]))) do inc(Idx);
  if Idx < ImageSize then
    begin
    {$IFDEF ISLIB}
    if (CPrefs[GUIDisplay].IValue in [0, 2]) then  {Dialog display needed?  Show error.}
    {$ENDIF}
      begin
      MessageBeep(MB_ICONWARNING);
      messagedlg(ifthen(Filename <> '', 'File '''+Filename+'''', 'Image') + ' contains data after code space' + {$IFNDEF ISLIB}' that was not generated by the '+PropIDEName+{$ENDIF} '.  This data will not be displayed or downloaded.', mtWarning, [mbOK], 0);
      end;
    end;
end;

[dbetz]

I believe you told me that no data could follow the code. I don't think you said that only non-zero data could follow it.

[PropGit]

Okay, yes, I think you're right. That explains it! That sounds like an easy thing to fix.

[dbetz]

Should the code also verify that the entire image file is <= 32768 bytes? Your code doesn't do that.

[dbetz]

Also, what is "ImageLimit" in the above code? Also, I tried just translating your validation code to C and it doesn't work for me. It tries to check whether the value at vbase is zero but it's not in a test image I generated with openspin. The data at vbase is the start of the initial stack frame. I'll look at this later to see if I can figure out what's going wrong.

[PropGit]

Should the code also verify that the entire image file is <= 32768 bytes? Your code doesn't do that.

Yes, that is important and was done outside of the code I shared.

ImageLimit = 32768

The file loading code (not shown) only opens up to ImageLimit, 32768 bytes, and ignores the rest if there is more in the file, then it calls the ValidateImageDataIntegrity function and passes in the actual ImageSize.

I think you already have most of the verification code translated and working because I gave it to you before with explanations of various parts. Send me the code if you need help.

Right, vbase is the start of variables, which is after data and code space, and may look like the start of the initial stack frame if there's no variables defined in code.

[dbetz]

Do you add the initial stack frame to .binary images? It looks like your checksum code assumes they are present even though they are not actually in the .binary file.

[PropGit]

It's not actually added into the .binary, but upon load, the values are calculated into the checksum by the validate routine as if they were there in the .binary. [Correction] It is added into the binary in the validation routine, but just for checksum validation purposes... nothing past the end of code is actually transmitted to the Propeller's boot loader because the boot loader knows how to fill in the rest. The .eeprom image should already have the initial stack frame, but the Validate routine that I wrote will essentially treat it the same as the .binary for transmission purposes.

The reason for .eeprom images is really to support EEPROM writers in production boards that include a Propeller. The .eeprom image can be used by an EEPROM writer to pre-program thousands of EEPROMS with the exact image that would have been placed there by the Propeller during downloading. The loader should treat the .eeprom with that in mind, knowing that it should only contain 0's at VBase and after, except for the initial stack frame, otherwise, it declares it invalid for it's transmission needs.

[PropGit]

Thank you @dbetz for addressing this. I've verified PropLoader is able to verify .eeprom files (with the Windows platform executable) as of commit 7e5aa1f. I also reviewed the code changes and agree with them.

There's one last thing we need to do to close this issue- something I probably wasn't clear enough on.

After verification of the image passes all the tests we've implemented, regardless of whether the image originated as a .binary or a .eeprom, there's no need to transmit anything pointed to by vbase, or beyond, during the Propeller download process.

The .eeprom format was really intended for EEPROM writers so they can pre-program EEPROMs with the same exact final image that the Propeller's boot loader would end up generating; however, our loader can get and use the same information from just the leading parts of the .eeprom format, exactly as it can from the entire .binary format. The Propeller download process should not transmit anything at or after the location pointed to by vbase, since those later values are automatically generated by the Propeller's boot loader anyway, much faster than we can transmit them.

Here's a simple example with a Propeller application that blinks a PAB WX board's LEDs (pins 26 and 27) in a certain pattern.

Archive: Blinky.zip (NOTE: I updated this after initial posting as I may have zipped up an old version of the Spin source by accident)

The .binary generated from this Spin program looks like this:

The .eeprom generated from this same Spin program looks like this:

In each case, we can transmit only up to the value at vbase-1 and the transmission would be fast and looks like this from PropLoader:

C:\>proploader -p COM33 Blinky.binary
Opening file 'Blinky.binary'
Downloading file to port COM33
60 bytes remaining
60 bytes sent
Verifying RAM
Download successful!

But if we transmit all 32 KB of this .eeprom image, it takes much longer and looks like this:

C:\>proploader -p COM33 Blinky.eeprom
Opening file 'Blinky.eeprom'
Downloading file to port COM33
32768 bytes remaining
31744 bytes remaining
30720 bytes remaining
29696 bytes remaining
28672 bytes remaining
27648 bytes remaining
26624 bytes remaining
25600 bytes remaining
24576 bytes remaining
23552 bytes remaining
22528 bytes remaining
21504 bytes remaining
20480 bytes remaining
19456 bytes remaining
18432 bytes remaining
17408 bytes remaining
16384 bytes remaining
15360 bytes remaining
14336 bytes remaining
13312 bytes remaining
12288 bytes remaining
11264 bytes remaining
10240 bytes remaining
9216 bytes remaining
8192 bytes remaining
7168 bytes remaining
6144 bytes remaining
5120 bytes remaining
4096 bytes remaining
3072 bytes remaining
2048 bytes remaining
1024 bytes remaining
32768 bytes sent
Verifying RAM
Download successful!

Also, for comparison, the .binary and .eeprom form's of ExampleS3.spin that we've been using look like this:
.binary

.eeprom

[PropGit]

Just found something is broken. The checksum calculated in the downloading phase is broken, at least in the case of loading this specific .elf file (which I hadn't tried until now).

File: SampleBlocklyProject.zip

C:\>proploader -v -p COM44 SampleBlocklyProject.elf
Opening file 'SampleBlocklyProject.elf'
Delivering second-stage loader
Downloading file to port COM44
8248 bytes remaining
7224 bytes remaining
6200 bytes remaining
5176 bytes remaining
4152 bytes remaining
3128 bytes remaining
2104 bytes remaining
1080 bytes remaining
56 bytes remaining
8248 bytes sent
Verifying RAM
Checksum error: expected fff687ec, got fff68000
ERROR: Download failed: -1