Add predefined sections in platform.txt

[slabbi]

Please consider to change
compiler.c.elf.flags={compiler.warning_flags} -Os -Wl,--gc-sections
to
compiler.c.elf.flags={compiler.warning_flags} -Os -Wl,--gc-sections,--section-start=.FAR_MEM1=0x10000,--section-start=.FAR_MEM2=0x20000,--section-start=.FAR_MEM3=0x30000

This allows to use sections directly when coding on an e.g. ATmega2560 that has 256kb memory.
Example:

#define HIGHPROGMEM2 attribute((section(".FAR_MEM2")))
const char ttl_short[] HIGHPROGMEM2 = "...";

Thanks :)

[MCUdude]

To be honest, I don't have much experience with section. But I agree this is a good idea, for all AVRs that has more than 64kB of flash.

How would this be compared to this?

// https://forum.arduino.cc/index.php?topic=387506.msg2676442#msg2676442
// PROGMEM over 64KB
#define PROGMEM_LATE __attribute__ (( __section__(".fini1") ))
const PROGMEM_LATE uint16_t data1[] = {big array};

and how big .fini1 really? According to the avr-libc manual, it's user-definable, but I can't find any references to it in the official platform.txt file.

And would this affect a program that's larger than 64kBand uses one or more FARMEM sections?

a little futher down that Arduino post:

The compiler produces "relocatable" segments of code, and the linker figures out how to place these in real memory based on information in a "linker script" (contained in .../tools/avr/avr/lib/ldscripts/...)
All that "PROGMEM" really does is put the designated pieces of data in a section called .progmem.something.
The Linker script for an ATmega1280 is something like:

.text   :
  {
    *(.vectors)
    KEEP(*(.vectors))
    /* For data that needs to reside in the lower 64k of progmem.  */
     *(.progmem.gcc*)
     *(.progmem*)
    /* For code that needs to reside in the lower 128k progmem.  */
    *(.lowtext)
    *(.init0)  /* Start here after reset.  */
    KEEP (*(.init0))
    *(.init1)
    KEEP (*(.init1))
       :
    *(.init9)  /* Call main().  */
    KEEP (*(.init9))
    *(.text)
    . = ALIGN(2);
     *(.text.*)
    . = ALIGN(2);
    *(.fini9)  /* _exit() starts here.  */
    KEEP (*(.fini9))
      :
    *(.fini0)  /* Infinite loop after program termination.  */
    KEEP (*(.fini0))
     _etext = . ;
  }  > text
  .data ...

The PROGMEM stuff (.progmem) is intentionally loaded "early" in the memory so that it will be in the bottom 64k where it is easy to reach with pgm_get_byte() and similar. Which is just fine when there is a relatively small amount of PROGMEM data, but doesn't make sense at all if you KNOW that there is going to be more than 64k of PROGMEM. In that case, you want to put the data that is used by the generic libraries, that NEEDS it to be in the first 64k, in the early memory, and your large bulk data, that you know you'll need to use the _far() functions to access, after everything else. It would probably be nice if there were a section named ".farprogmem" or something that you could use, but there isn't, and the effort of adding it doesn't really seem justified as long as those .fini sections are there anyway...

[slabbi]

Well, in most cases you use the lower 64kb for your code and store data (e.g. two databases in my case, actually an array of a structure holding pointers) in the far memory. When you let the compiler decide where to store the arrays you have problems when the string #1 is placed in section 0x10000 and string #2 in section 0x20000. It is impossible to address both strings using the same structure. You will always run into these problems when you do following:

#define ADD_DATA(INDEX, DATA_CHECKSUM, DATA_ID, DATA_FUNC)                               \
  const char   data_##INDEX##_id[]   PROGMEM = CHIP_ID;                             \
  const char   data_##INDEX##_func[] PROGMEM = CHIP_FUNC;                           \
  const t_data data_##INDEX##_       PROGMEM = { .checksum = DATA_CHECKSUM,         \
                                                      .dataid   = data_##INDEX##_id,     \
                                                      .datafunc = data_##INDEX##_func,   \
                                                    };
ADD_DATA(    100, 0x12345678 , "xxx"   , "xxxxxxxxxxxxxx"              )
ADD_DATA(    101, 0x12345678 , "yyy"   , "yyyyyyyyyyyyyy"              )

The compiler will store the data in PROGMEM first (or .fini1 or whatever you decide to take) and when this section is not "full" it continues with the second database which might exceed the storage of this section.

Using above suggestion allows to tell the compiler to store e.g. database 1 in section 0x10000 and database 2 in section 0x20000. You are still resposible not to exceed the 64kb for one database but you will not have the problem that the compiler tries to store both databases in one section.

And very important:
The suggested change has no influence to programs that do not use the predefined sections. So there will be no side effects.

[slabbi]

This is a memory map of one of my programs. You can see how the two "databases" are stores in 0x10000 and 0x20000. This is very useful when dealing with large structures.

[MCUdude]

Wow, thanks for the visual representation! I just pulled out my custom ATmega2561 board to do a little testing.

So how do you access the data that's stored "way up there"? with pgm_read_byte_far?

[MCUdude]

I wrote a very simple test program to tun on an ATmega2561. It seems like it's working just fine! However, I think names like PROGMEM1..3 is "easier to understand. This will need a little note in the readme anyways. But again, I'll assume pgm_read_byte_far and pgm_get_far_address is what you use to fetch the address and the data? It seems like these two functions uses uint32 (or actually just 24 bit), so they don't care what section the data is located at. Heck you could even use pgm_read_byte_far to get the lower 0x0000-0xFFFF addresses, but it will be less efficient than pgm_read_byte.

#define PROGMEM1 __attribute__((section(".FAR_MEM1")))
#define PROGMEM2 __attribute__((section(".FAR_MEM2")))
#define PROGMEM3 __attribute__((section(".FAR_MEM3")))

const char far1[] PROGMEM1 = " Hello far1";
const char far2[] PROGMEM2 = "Hello far2";
const char far3[] PROGMEM3 = "Hello far3";

void setup() 
{
  uint8_t i = 0;
  uint8_t c = 0;
  Serial.begin(115200);
  do
  {
    c = pgm_read_byte_far(pgm_get_far_address(far1)+i);
    if (c)
      Serial.printf("char:%c\tnum:0x%02x\tmem_addr:0x%05lx\n",c, c, pgm_get_far_address(far1)+i);
    i++;
  } while (c);
  i = 0;
  Serial.printf("\n");
  do
  {
    c = pgm_read_byte_far(pgm_get_far_address(far2)+i);
    if (c)
      Serial.printf("char:%c\tnum:0x%02x\tmem_addr:0x%05lx\n",c, c, pgm_get_far_address(far2)+i);
    i++;
  } while (c);
  i = 0;
  Serial.printf("\n");
  do
  {
    c = pgm_read_byte_far(pgm_get_far_address(far3)+i);
    if (c)
      Serial.printf("char:%c\tnum:0x%02x\tmem_addr:0x%05lx\n",c, c, pgm_get_far_address(far3)+i);
    i++;
  } while (c);

}

void loop()
{

}

Output:

char: 	num:0x20	mem_addr:0x10000
char:H	num:0x48	mem_addr:0x10001
char:e	num:0x65	mem_addr:0x10002
char:l	num:0x6c	mem_addr:0x10003
char:l	num:0x6c	mem_addr:0x10004
char:o	num:0x6f	mem_addr:0x10005
char: 	num:0x20	mem_addr:0x10006
char:f	num:0x66	mem_addr:0x10007
char:a	num:0x61	mem_addr:0x10008
char:r	num:0x72	mem_addr:0x10009
char:1	num:0x31	mem_addr:0x1000a

char:H	num:0x48	mem_addr:0x20000
char:e	num:0x65	mem_addr:0x20001
char:l	num:0x6c	mem_addr:0x20002
char:l	num:0x6c	mem_addr:0x20003
char:o	num:0x6f	mem_addr:0x20004
char: 	num:0x20	mem_addr:0x20005
char:f	num:0x66	mem_addr:0x20006
char:a	num:0x61	mem_addr:0x20007
char:r	num:0x72	mem_addr:0x20008
char:2	num:0x32	mem_addr:0x20009

char:H	num:0x48	mem_addr:0x30000
char:e	num:0x65	mem_addr:0x30001
char:l	num:0x6c	mem_addr:0x30002
char:l	num:0x6c	mem_addr:0x30003
char:o	num:0x6f	mem_addr:0x30004
char: 	num:0x20	mem_addr:0x30005
char:f	num:0x66	mem_addr:0x30006
char:a	num:0x61	mem_addr:0x30007
char:r	num:0x72	mem_addr:0x30008
char:3	num:0x33	mem_addr:0x30009

[slabbi]

Here is an simplified example using the code above:

#define HIGHPROGMEM1  __attribute__((section(".FAR_MEM1")))

typedef struct {
    const uint32_t checksum;
    const char *dataid;
    const char *datafunc;
} t_data;

#define ADD_DATA(INDEX, DATA_CHECKSUM, DATA_ID, DATA_FUNC)                               \
  const char   data_##INDEX##_id[]   HIGHPROGMEM1  = CHIP_ID;                             \
  const char   data_##INDEX##_func[] HIGHPROGMEM1  = CHIP_FUNC;                           \
  const t_data data_##INDEX##_       HIGHPROGMEM1  = { .checksum = DATA_CHECKSUM,         \
                                                      .dataid   = data_##INDEX##_id,     \
                                                      .datafunc = data_##INDEX##_func,   \
                                                    };
ADD_DATA(    100, 0x12345678 , "xxx"   , "xxxxxxxxxxxxxx"              )
ADD_DATA(    101, 0x12345678 , "yyy"   , "yyyyyyyyyyyyyy"              )
// use as many ADD_DATA as you need

const t_data * const data[] HIGHPROGMEM1  = {
  &data_100_, &data_101_, // and so on
}
// needed to access the data

uint_farptr_t read_data_dataid(int i) {
        static char buffer[21];
        uint_farptr_t adr = pgm_get_far_address(data)+(int32_t)(i*2);
        uint_farptr_t base = adr & 0xFF0000ul;
        uint_farptr_t menuPtr = base + pgm_read_word_far ( adr );     // get address to structure
        memcpy_PF(buffer, menuPtr, sizeof(t_data));   // 21 bytes max
        uint_farptr_t text = base + ( ((uint_farptr_t)((t_data*)&buffer)->dataid) & 0xfffful );
        return text;
    }
// same for read_data_datafunc()

uint32_t read_data_checksum(int i) {
        static char buffer[21];
        uint_farptr_t adr = pgm_get_far_address(data)+(int32_t)(i*2);
        uint_farptr_t base = adr & 0xFF0000ul;
        uint_farptr_t menuPtr = base + pgm_read_word_far ( adr );     // get address to structure
        memcpy_PF(buffer, menuPtr, sizeof(t_data));
        uint32_t value = ((t_data*)&buffer)->checksum;
        return value;
    }

Hints:

• uint_farptr_t base = adr & 0xFF0000ul;
  is used to get the base address of the segment. There are other ways to do this (like inline assembler etc.) but this is quite convenient. When you know what the base address is, then you can also use "uint_farptr_t base = 0x10000" but my solution also works when you move the code to a different section without changing the code.
• buffer[] must be large enough to store the complete t_data structure. It must not be static but I have some functions that return the structure instead of a single value, so this is quite convenient.
• To visualize the code I used the free version of HEXit:
  http://www.hse-electronics.com/Deutsch/Produkte/dHexit/dhexit.html
  http://www.hse-electronics.com/Deutsch/dDownloads/ddownloads.html
  Unfortunately they do not have a affordable version for "normal" users :(

As you can see, using the compiler options working with high memory becomes quite simple. I have no idea why this is not part of the Arduino code.

And another nice fact I recognized recently:
When using LTO the compiler also seems to optimize const variables in flash within one section. So you can save lots of memory :)

[MCUdude]

Whoa, that some pretty powerful code right there. However, some users may find that macro a little hairy. But hey, powerful macros are what they are 🙂

As you can see, using the compiler options working with high memory becomes quite simple. I have no idea why this is not part of the Arduino code.

I was not aware that it was that simple to work with the "far" memory. @SpenceKonde did you know this trick? To Arduino's "defense", there is ALOT of stuff they have added and left out, that doesn't make sense. Take for instance a look at the max and min macros in Arduino.h. Since these are macros and not functions, you may get an incorrect number out if one parameter is a function such as rand().

Thanks for this contribution, you obviously know your stuff! Feel free to open more issues if you find other issues/improvements that this or some of my other Arduino cores may benefit from.

[slabbi]

Thanks ;)

I was not aware that it was that simple to work with the "far" memory.

Well, currently it is not simple because Arduino IDE does not allow to modify the compiler options. But since the "far" memory is very limited it is only necessary to define three (or four - when you also add 0x00000) sections to cover the whole memory ;)

[SpenceKonde]

I knew it was something like that, but I get agoraphobic programming devices with so much flash space anyway :-P
Like, I only have one set of devices that needs this and the core isn't even released yet (and only between two sections - I'm referring to Alpha Victor Romeo Delta Alpha ), but on that one what I want to do is have... maybe a menu option (?) to effect it's choice of linker script, and have the default option be for it to put .rodata at the highest possible addresses (and like, error out if you've got more than 32k of static data; ideally with an error message that tells users - sorry, we can only handle this with under 32K of const variables but see this section of core docs for ways around this... ooooh, what I should do is define sections like that for each of the 32KB sections of flash that can be mapped independently, so people can manually specify it if they want... Oh everything is coming together in my mind! Muahaha!