Proposal: Provide persistence in RAM and Flash across low-power and reset states

[kenbell]

I've been thinking about how to support super-low power states in TinyGo, and I think there are two interesting cases:

1. CPU core and peripherals powered off, but RAM still powered
2. CPU core, peripherals and RAM powered off

Various chips (I'm most aware of the STM32 line) support entering sleep states like these, being woken on specific GPIO pins, RTC alarms, and similar. In order to support these low-power states, it would be super-cool to have ways to persist data. A beneficial side-effect is that it would also enable some persistence across reset (and potentially flashing).

Example use-cases:

1. Battery-powered sensing device that wakes every minute (RTC alarm) to collect a sample, save to RAM, then returns to low-power state. Once per day it powers up wifi to deliver the samples collected.
2. Logging device that wants to ensure samples are not lost on a crash / watchdog event by holding samples in RAM until they can be delivered.
3. Configuration settings - allow users to configure a device and persist those settings to flash so they survive power-out and reset

I've put a prototype implementation for RAM persistence here: #1715

I've tried to put fairly details docs in the package runtime/persistence: https://github.com/tinygo-org/tinygo/blob/537ef22f7365bdc7d3e04b03dc943262f43730fe/src/runtime/persistence/persistence.go

[kenbell]

Example output of the example app

value:  85
value:  86
value:  87

*** ** * RESET * ** ***

Persistent Region Size:  100
value:  88
value:  89
value:  90
value:  91
value:  92
value:  93
value:  94

*** ** * RESET * ** ***

Persistent Region Size:  100
value:  95
value:  96
value:  97
value:  98
value:  99
value:  100

[aykevl]

There is actually a third category: the ESP32 chip has a special low-power RAM area. In deep sleep, almost the entire chip is powered off except for a small RTC area (that's actually much more than just an RTC and also includes a small low-power CPU core).

[aykevl]

This is an interesting idea, but there are a few subtleties around it. Also, I have a few questions.

Battery-powered sensing device that wakes every minute (RTC alarm) to collect a sample, save to RAM, then returns to low-power state. Once per day it powers up wifi to deliver the samples collected.

So I guess the difference with a regular sleep (time.Sleep) is that all peripherals are also powered down and are reset?
I think this would make programming much harder. If it is possible to somehow power down peripherals but retain their state (AFAIK this is usually possible) that would be much easier to work with.

Logging device that wants to ensure samples are not lost on a crash / watchdog event by holding samples in RAM until they can be delivered.

TinyGo currently supports neither, and I'm not sure what the semantics around both would be for a reliable recovery. I don't think we should focus on this use case until they are supported in TinyGo to a reasonable degree.

Configuration settings - allow users to configure a device and persist those settings to flash so they survive power-out and reset

Flash is a totally different beast and not at all comparable to RAM. Most importantly, while it is generally random-access for reads, you need to erase entire pages to be able to write it (and there are sometimes limitations regarding the number of writes per page per flash erase cycle).
I think flash should have a different API and should work well together with filesystems.

[kenbell]

The way I'm thinking of using RAM for logging is something like this...

import "github.com/random/persistentlog"

// Random value used as guard to detect if region initialized
const MAGIC_SEED = 0xde42ae42

func main() {
   // This will only reset RTC state if not already initialized (first boot or battery failed)
   if !machine.RTC.Init() {
      // Get time from NTP / control server, put into RTC
      machine.RTC.SetTime(xxxx)

      // Ask for RTC to wake processor with a reset every 60secs
      machine.RTC.SetPeriodicWake(60 * time.Seconds)
   }

  // Request 10KB RAM be preserved across reset
  rgn := persist.NewRAM(10 * 1024)
  
  // Use 8KB for log (persistentlog library responsible for guards / checksums / whatever to detect uninitialized RAM)
  persistentLog := persistentlog.Init(rgn.SubRegion(0, 8 * 1024), MAGIC_SEED)

  // Use 2KB for other things... calibration values?  randomly generated unique device id?
  randomStuff := rgn.SubRegion(8*1024, 2 * 1024)

  // Take measurement

  // Save in log
  persistentLog.Add(time.Now(),measurement)

  if persistentLog.Samples() > 100 {
    // Power-up wifi

    // Try to send logs.

    // Reset logs
    if was_able_to_send_logs {
      persistentLog.Clear()
    }
  }

  // Indicate were done until next RTC wake-up, at which time software reset will
  // cause main to start over.   os.Exit is a hint to the runtime to shutdown max
  // peripherals, etc as possible to minimize power draw.
  os.Exit(0)
}

So this would very much just be one building block - but enables code that is hopefully very chip agnostic.

I think if something like this is added to TinyGo, it should not be made use-case specific and should be as generic as possible. I'm sure there's a bunch of other use-cases for having persistent RAM regions.

Agree flash is very different, i need to do some research there - possibly have the concept of 'pages' in the interface, so maybe RAM has a page size of 4 (32-bit systems), Flash may have larger pages. At least some STM32 chips have multiple pages sizes even within a single chip - 16KB up to 128KB in one i'm looking at. Also open to RAM and Flash being so different that, yes, they should just be separate interfaces.

I wasn't aware of the ESP32 low-power RAM. Looking now, it looks like some STM32 chips have a small 'SRAM2' that can be left powered without powering the main SRAM. Looks like that's definitely something that should be incorporated on chips that support it.

[kenbell]

I've updated the branch to use //go:persist pragma. It's a much, much cleaner implementation.

At the moment, it doesn't do any type validation. Since the initializer is not actually run, it's definitely not safe to use pointers - thinking that it would make sense to limit it to built-in types and arrays of built-in types.

Updating the 'example' of how this might be used:

import "github.com/random/persistentlog"

// Random value used as guard to detect if region initialized
const MAGIC_SEED = 0xde42ae42

//go:persist
var log [8*1024]byte

//go:persist
var guard int

//go:persist
var calibration int

func main() {
   // This will only reset RTC state if not already initialized (first boot or battery failed)
   if !machine.RTC.Init() {
      // Get time from NTP / gps / control server, put into RTC
      machine.RTC.SetTime(xxxx)

      // Ask for RTC to wake processor with a reset every 60secs
      machine.RTC.SetPeriodicWake(60 * time.Seconds)
   }

  // persistentlog library responsible for guards / checksums / whatever to detect uninitialized RAM
  persistentlog = persistentlog.Init(logBuffer, MAGIC_SEED)

  // reset calibration if guard not initialized
  if guard != MAGIC_SEED {
   calibration = 0
   guard = MAGIC_SEED
  }

  // Take measurement

  // Save in log
  persistentLog.Add(time.Now(),measurement)

  if persistentLog.Samples() > 100 {
    // Power-up wifi

    // Try to send logs.

    // Reset logs
    if was_able_to_send_logs {
      persistentLog.Clear()
    }
  }

  // Indicate were done until next RTC wake-up, at which time software reset will
  // cause main to start over.   os.Exit is a hint to the runtime to shutdown max
  // peripherals, etc as possible to minimize power draw.
  os.Exit(0)
}

[kenbell]

There is actually a third category: the ESP32 chip has a special low-power RAM area. In deep sleep, almost the entire chip is powered off except for a small RTC area (that's actually much more than just an RTC and also includes a small low-power CPU core).

My latest evolution of the prototype creates a memory region PRAM on most chips this is aliased to RAM, on the STM32L5 it's configured to be SRAM2 (64KB), which can be left powered in low-power states. On the ESP32 it uses the RTC FAST RAM (8KB).

[deadprogram]

Now that we have machine.Flash support, can this issue be closed @kenbell ?