1.22.0

Added

• The Lifetime Estimation B value for monitored eMMCs is now reported as
  diskstats/<disk>/lifetime_b_pct
• A warning is now emitted when there is an issue parsing the output from
  memfault-device-info in memfault-core-handler. Previously, this would
  cause coredump collection to fail silently.
• Built-in CPU, memory, and thermal metrics can now be disabled individually
  with the new metrics.system_metric_collection.{cpu,memory,thermal}.enable
  boolean config.
• memfaultd's .cargo/config.toml now sets --compress-debug-sections to
  zlib and --force-unwind-tables to yes by default. These settings enable
  building efficient debug sections and binaries whose coredumps can be fully
  processed. We strongly recommend using them with any Rust programs you intend
  to monitor with Memfault.

Changed

• collectd-based metrics that overlap with built-in metrics will now be
  selectively filtered out if both are enabled based on whether the overlapping
  built-in metric group itself is enabled. This allows for using a mix of
  collectd-based metrics and built-in metrics more easily.

Fixed

• A bug in which the values for the MemfaultSdkMetric_os_name and
  MemfaultSdkMetric_os_version built-in Attributes were reversed.

1.21.1

This is a patch release that fixes an overflow issue with the recently added
diskstats/<disk>/bytes_written metric.

Fixed

• Fixed an issue with the diskstats/<disk>/bytes_written metric where it would
  overflow when the diskstats counter wrapped. This would cause very large
  readings to be sent. Whenever there is a potential overflow now the reading
  will be discarded.

1.21.0

This release adds several new metrics for tracking flash wear, as well as a new
syslog log source. It also includes a config flag to enable legacy statsd metric
names, and a data retention check to the upload process. For more information on
the new metrics, check out the
docs.

Added

• Added metrics for tracking flash wear for mmc block devices like eMMC and SD
  cards
  • diskstats/<disk_name>/bytes_written
  • diskstats/<disk_name>/lifetime_pct
  • diskstats/<disk_name>/manufacturer_id
  • diskstats/<disk_name>/name
• Added syslog as a logs source. This adds support for messages in both RFC5424
  and RFC3164 format sent over UDP. This is enabled with the syslog feature
  which defaults to enabled.
• Added a config flag to enable legacy statsd metric names. This will generate
  metrics that follow the format generated by the collectd statsd server. This
  is a helpful option for people looking to remove collectd as a dependency,
  but keep the legacy format for metric keys. The config value can be found in
  metrics->statsd_server->legacy_key_names.
• Added time-based data filtering: When upgrading multiple devices' visibility
  levels, the Memfault backend now tells memfaultd to only upload data from
  the time of the change forward. This prevents unexpected large data uploads
  from device history.
• Added interface/<interface>/total_bytes/{tx,rx} metrics. These allow us to
  track the total bytes sent over an interface over the course of an hour.

Changed

• Added intelligent traffic distribution: Device initial connections now include
  a random delay (up to 25% of the configured upload interval). This prevents
  traffic spikes when many devices connect simultaneously, ensuring smoother
  backend performance.
• ram and loop disks are no longer tracked with diskstats in auto mode.
  These are virtual disks, and provide little value when tracking metrics.

1.20.0

This release adds support for log filtering as well as a few bug fixes. Check
out the logging docs for more
information on log filtering!

Added

• Added support for log message filtering. This feature gives you the ability to
  exclude log lines that are spammy or contain potentially sensitive information
  from being uploaded to Memfault.
• Added built-in logs-to-metric rules that increment counter metrics when a
  systemd_restarts or oomkill event is detected. These are widely applicable
  to most Linux distros, and represent a good base for metric collection

Fixed

• Fixed a problem where large HRT files could take a long time to write. This
  was due to the file writer not being buffered.
• Fixed a race condition that caused the journald cursor to not be written.
  This would lead to the journald log collector capturing duplicate log lines
  if memfaultd is restarted.
• Fixed a problem where daily heartbeats were being saved when they were not
  enabled.

Removed

• Combined the regex feature with the logging feature to simplify feature
  management as log filtering and matching based on regexes becomes a key part
  of our logging offering.

1.19.0

This release adds memfaultctl upload as a command to allow
users to force memfaultd to upload data without forcing a
write and sync of in-progress data.

Added

• memfaultctl upload, which tells memfaultd to upload
  any data that has been written to the staging direcoty.

1.18.1

This release adds mickledore to the list of compatible
Yocto versions for meta-memfault.

Added

• mickledore to LAYERSERIES_COMPAT_memfault in meta-memfault's
  layer.conf

1.18.0

Added

• mar.mar_entry_max_count, which allows for setting the max number of MAR
  entries in your tmp directory. This will default to 1000.
• logs.max_buffered_lines, which sets the max number of lines that will be
  kept in memory before lines are dropped. Note that this is the same as
  fluent-bit.max_buffered_lines, but will apply to both the fluent-bit log
  source as well as the journald log source. If you have previously configured
  fluent-bit.max_buffered_lines that value will be used instead of the general
  config.
• Added the kind field to the MAR POST body. This is a field that is used by
  Memfault to differentiate the source of a MAR entry.
• Added logs.extra_attributes option to configure extra log attributes to keep
  in the log file (similar to
  fluent-bit.extra_attributes
  which still works but will be deprecated)

Changed

• The LogCollector has gone through a refactor to make the concurrency methods
  more consistent with the rest of the code base. This has also allowed us to
  make the following change.
• Moved recovery of logs in the logs directory into the LogCollector thread.
  This prevents a case where memfaultd could be slow to start when there is a
  large number of MAR entries on disk, and depending on systemd configuration,
  could be considered crashed and restarted before booting completely.

Fixed

• When passing a null to disable a feature there was a case where the config
  merging logic would fail. This would result in a case where a subset of
  configs were not possible. Fixed this logic to allow these valid use cases.

1.17.0

This release introduces the new memfaultctl write-metrics command
as well as some important bugfixes.

Added

• memfaultctl write-metrics, which allows users to write
  metrics to memfaultd from a script or shell by specifying
  them as arguments to the command in the form KEY=VALUE.
• 2 new built-in system metrics under the new diskstats
  metric category. These metrics follow the naming pattern
  diskstats/<device name>/reads_per_second and
  diskstats/<device name>/writes_per_second where
  <device name> is the name of a device listed in
  /proc/diskstats. The list of devices monitored
  can be configured with the metrics.system_metric_collection.diskstats
  configuration field.

Changed

• The log message emitted when the MAR cleaner
  encounters an invalid MAR entry in the MAR
  staging area has been lowered from WARN level
  to DEBUG level.
• A log message emitted when the configured
  high_resolution_telemetry.max_samples_per_minute
  rate limit is violated has been lowered from WARN
  level to DEBUG. This is to avoid repeatedly
  logging the same message at a high frequency
  when the system is sending more readings than
  permitted.

Fixed

• All deltas calculated based on a current and previous
  counter from procfs now take potential overflow into account.
• Fixed a sequencing issue with the cleaning of the MAR directory
  concerning an edge case where logs would fail to recover on
  devices that are near their disk space or inode quotas.
  This would result in stranded log files that would not
  get deleted or uploaded to Memfault until there is enough
  space to recover them.

memfaultd 1.16.1

This is a small release that adds coredump capture support for 32-bit ARM
targets using musl libc.

Fixed

• The coredump feature can now be compiled for the
  armv7-unknown-linux-musleabihf target.

memfaultd 1.16.0

This release includes two major new features:

• A new coredump capture mode to process the coredumps directly on device
  resulting in even smaller crash signature and guaranteeing that no user data
  is uploaded;
• High Resolution Telemetry for Linux, storing every single datapoint and
  enabling second-by-second data from Linux devices, for even deeper debugging
  capability.

Added

• Built-in wireless metric capture - wireless interfaces being monitored by the
  metrics.system_metric_collection.network_interfaces configuration will now
  have RSSI captured alongside the existing network metrics
• Added a boottime_duration_ms to Metric Reports that indicates the duration
  of the report independent of whether the CPU was running or not. This is
  intended to be used for debugging purposes for teams whose devices sleep often
  while in use.
• The new on-device stack unwinding option for capturing crash Traces, which can
  be enabled by setting coredump.capture_strategy.type to stacktrace in
  memfaultd.conf.
• High resolution telemetry, which is enabled by default and can be disabled or
  have its rate limiting configured with the metrics.high_resolution_telemetry
  field.
• Support for Yocto Scarthgap in meta-memfault. This support will be extended
  to meta-memfault-example in an upcoming release. Check out the scarthgap
  branch if you are using that version of Yocto!
• The built-in system memory metrics now include additional states such as
  Cached and Buffered.
• The metrics.statsd_server has had a new option, legacy_gauge_aggregation,
  added that averages Gauge metric readings rather than simply storing the
  most recent value. This is meant to allow for a smooth migration from
  collectd's StatsD server to memfaultd's. New integrations should
  use the h (Histogram) metric type for metrics whose readings should
  be aggregated via average.

Changed

• The default on-device rate limit for logs ingested by memfaultd has been
  increased from 500 lines per minute to 1000 lines per minute

Fixed

• Arbitrary ASCII strings between 1 and 128 in length can
  now be used in StatsD metric keys (besides the : delimiter),
  matching the behavior described in the docs.

Removed

Some built-in metrics were removed, as the naming conventions were not
consistent. All of these metrics were added in 1.15.0. The list can
be found below:

• cpu_usage_pct
• connectivity_recv_bytes
• connectivity_sent_bytes
• connectivity_<interface>_recv_bytes
• connectivity_<interface>_sent_bytes
• memory_used_pct
• memory_<process>_pct
• cpu_usage_<process>_pct
• storage_used_<disk>_pct