Add a diagnostic kstat for obtaining pool status #16026

don-brady · 2024-03-25T20:56:02Z

Motivation and Context

A hung pool process can be left holding the spa config lock or the spa namespace lock. If an admin wants to observe the status of a pool using the traditional zpool status, it could hang waiting for one of the locks held by the stuck process. It would be nice to observe pool status in this scenario without the risk of the inquiry hanging.

Description

Exploring Solutions

Ignore the locks. Here the admin knows that the pool state is not changing (they are not adding/removing disks, changing the config, etc). So we can traverse data structures without the normal protection required to guard against them changing underneath us. Two methods come to mind.

(a) Add a new kstat entry that reports the pool status in JSON and ignores any locking when gathering the pool stats.
(b) Add an '--unsafe' (or '--ignore-locking') flag to zpool status which informs the kernel to ignore locking when gathering the pool stats.

Infer that the lock is stuck (held for an extended period) and conclude that locking is not required to read the pool stats. This is somewhat a variant of 1, where the source code, instead of the admin user, is determining that it is safe to ignore locking since the pool configuration cannot be changing.
Refactor the spa code to have more fine grain locking and perhaps use reader/writer locks in lieu of mutex locks to alleviate the obvious points of lock contention when a pool gets stuck. Don't hold these global scope locks across disk I/O, etc.

This change is implementing option 1a -- adding a kstat at zfs/<pool>/stats.json which ignores any locking. This kstat can be used for investigations when pools are in a hung state while holding global locks required for a traditional 'zpool status' to proceed.

NOTE: This kstat is not safe to use in conditions where pools are in the process of configuration changes (i.e., adding/removing devices). Therefore, this kstat is not intended to be a general replacement or alternative to using 'zpool status'.

Sponsored-by: Wasabi Technology, Inc.
Sponsored-By: Klara Inc.

How Has This Been Tested?

Added new zpool_status_kstat_pos test to validate the JSON output
Manually tested the kstat with various pool configurations in various states of health

sample kstat output (degraded mirror):

$ sudo cat /proc/spl/kstat/zfs/tank/status.json | jq
{
  "status_json_version": 4,
  "scl_config_lock": true,
  "scan_error": 0,
  "scan_stats": {
    "func": "RESILVER",
    "state": "FINISHED",
    "start_time": 1710901463,
    "end_time": 1710901463,
    "to_examine": 165888,
    "examined": 165888,
    "processed": 316416,
    "errors": 0,
    "pass_exam": 0,
    "pass_start": 1711398055,
    "pass_scrub_pause": 0,
    "pass_scrub_spent_paused": 0,
    "pass_issued": 0,
    "issued": 113664
  },
  "state": "DEGRADED",
  "version": 5000,
  "name": "tank",
  "txg": 255,
  "pool_guid": 6645716700149381000,
  "errata": 0,
  "hostname": "discovery1",
  "com.delphix:has_per_vdev_zaps": true,
  "features_for_read": {
    "com.delphix:hole_birth": true,
    "com.delphix:embedded_data": true,
    "com.klarasystems:vdev_zaps_v2": true
  },
  "load_info": {
    "hostname": "discovery1",
    "enabled_feat": {
      "org.illumos:edonr": 0,
      "com.delphix:redaction_list_spill": 0,
      "org.zfsonlinux:large_dnode": 0,
      "com.delphix:bookmark_written": 0,
      "org.illumos:sha512": 0,
      "org.openzfs:raidz_expansion": 0,
      "org.freebsd:zstd_compress": 0,
      "com.joyent:multi_vdev_crash_dump": 0,
      "org.illumos:lz4_compress": 1,
      "org.illumos:skein": 0,
      "com.delphix:redaction_bookmarks": 0,
      "com.datto:encryption": 0,
      "com.delphix:extensible_dataset": 1,
      "com.datto:bookmark_v2": 0,
      "com.delphix:head_errlog": 1,
      "com.klarasystems:vdev_zaps_v2": 1,
      "com.delphix:hole_birth": 1,
      "com.delphix:redacted_datasets": 0,
      "org.open-zfs:large_blocks": 0,
      "com.delphix:embedded_data": 1,
      "org.openzfs:blake3": 0,
      "com.delphix:device_removal": 0,
      "org.openzfs:draid": 0,
      "com.delphix:empty_bpobj": 0,
      "com.delphix:obsolete_counts": 0,
      "org.zfsonlinux:allocation_classes": 0,
      "org.zfsonlinux:project_quota": 1,
      "org.openzfs:device_rebuild": 0,
      "com.delphix:livelist": 0,
      "com.fudosecurity:block_cloning": 0,
      "com.datto:resilver_defer": 0,
      "com.delphix:enabled_txg": 36,
      "com.delphix:spacemap_v2": 1,
      "com.delphix:zpool_checkpoint": 0,
      "org.zfsonlinux:userobj_accounting": 1,
      "org.openzfs:zilsaxattr": 0,
      "com.delphix:bookmarks": 0,
      "com.delphix:async_destroy": 0,
      "com.delphix:log_spacemap": 1,
      "com.delphix:spacemap_histogram": 18,
      "com.joyent:filesystem_limits": 0
    },
    "can_rdonly": true,
    "rewind_txg_ts": 1711139069,
    "seconds_of_rewind": -1711139069,
    "verify_meta_errors": 0,
    "verify_data_errors": 0
  },
  "spa_props": {
    "name": {
      "source": "ZPROP_SRC_NONE",
      "value": "tank"
    },
    "size": {
      "source": "ZPROP_SRC_NONE",
      "value": 503316480
    },
    "allocated": {
      "source": "ZPROP_SRC_NONE",
      "value": 167424
    },
    "free": {
      "source": "ZPROP_SRC_NONE",
      "value": 503149056
    },
    "checkpoint": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "fragmentation": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "expandsize": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "readonly": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "capacity": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "dedupratio": {
      "source": "ZPROP_SRC_NONE",
      "value": 100
    },
    "bcloneused": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "bclonesaved": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "bcloneratio": {
      "source": "ZPROP_SRC_NONE",
      "value": 100
    },
    "health": {
      "source": "ZPROP_SRC_NONE",
      "value": 6
    },
    "version": {
      "source": "ZPROP_SRC_DEFAULT",
      "value": 5000
    },
    "load_guid": {
      "source": "ZPROP_SRC_NONE",
      "value": 10978401526797912000
    },
    "freeing": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "leaked": {
      "source": "ZPROP_SRC_NONE",
      "value": 0
    },
    "guid": {
      "source": "ZPROP_SRC_NONE",
      "value": 6645716700149381000
    },
    "maxblocksize": {
      "source": "ZPROP_SRC_NONE",
      "value": 16777216
    },
    "maxdnodesize": {
      "source": "ZPROP_SRC_NONE",
      "value": 16384
    }
  },
  "initial_load_time": [
    1711398055,
    453276435
  ],
  "error_count": 0,
  "suspended": false,
  "feature_stats": {
    "com.delphix:async_destroy": 0,
    "com.delphix:empty_bpobj": 0,
    "org.illumos:lz4_compress": 1,
    "com.joyent:multi_vdev_crash_dump": 0,
    "com.delphix:spacemap_histogram": 21,
    "com.delphix:enabled_txg": 36,
    "com.delphix:hole_birth": 1,
    "com.delphix:extensible_dataset": 1,
    "com.delphix:embedded_data": 1,
    "com.delphix:bookmarks": 0,
    "com.joyent:filesystem_limits": 0,
    "org.open-zfs:large_blocks": 0,
    "org.zfsonlinux:large_dnode": 0,
    "org.illumos:sha512": 0,
    "org.illumos:skein": 0,
    "org.illumos:edonr": 0,
    "org.zfsonlinux:userobj_accounting": 1,
    "com.datto:encryption": 0,
    "org.zfsonlinux:project_quota": 1,
    "com.delphix:device_removal": 0,
    "com.delphix:obsolete_counts": 0,
    "com.delphix:zpool_checkpoint": 0,
    "com.delphix:spacemap_v2": 1,
    "org.zfsonlinux:allocation_classes": 0,
    "com.datto:resilver_defer": 0,
    "com.datto:bookmark_v2": 0,
    "com.delphix:redaction_bookmarks": 0,
    "com.delphix:redacted_datasets": 0,
    "com.delphix:bookmark_written": 0,
    "com.delphix:log_spacemap": 1,
    "com.delphix:livelist": 0,
    "org.openzfs:device_rebuild": 0,
    "org.freebsd:zstd_compress": 0,
    "org.openzfs:draid": 0,
    "org.openzfs:zilsaxattr": 0,
    "com.delphix:head_errlog": 1,
    "org.openzfs:blake3": 0,
    "com.fudosecurity:block_cloning": 0,
    "com.klarasystems:vdev_zaps_v2": 1,
    "com.delphix:redaction_list_spill": 0,
    "org.openzfs:raidz_expansion": 0
  },
  "vdev_tree": {
    "type": "root",
    "id": 0,
    "guid": 6645716700149381000,
    "vdev_children": 1,
    "children": [
      {
        "type": "mirror",
        "id": 0,
        "guid": 10500366840367100000,
        "asize": 519569408,
        "ashift": 9,
        "offline": false,
        "faulted": false,
        "degraded": false,
        "removed": false,
        "not_present": false,
        "is_log": false,
        "state": "DEGRADED",
        "vs_scan_removing": false,
        "vs_noalloc": false,
        "vs_resilver_deferred": false,
        "resilver_repair": "none",
        "initialize_state": {
          "vs_initialize_state": "VDEV_INITIALIZE_NONE",
          "vs_initialize_bytes_done": 0,
          "vs_initialize_bytes_est": 0,
          "vs_initialize_action_time": 0
        },
        "trim_state": {
          "vs_trim_state": "VDEV_UNTRIMMED",
          "vs_trim_action_time": 0,
          "vs_trim_bytes_done": 0,
          "vs_trim_bytes_est": 0
        },
        "read_errors": 0,
        "write_errors": 0,
        "checksum_errors": 0,
        "slow_ios": 0,
        "trim_errors": 0,
        "vdev_children": 2,
        "children": [
          {
            "type": "file",
            "id": 0,
            "guid": 135275325128040140,
            "asize": 519569408,
            "ashift": 9,
            "whole_disk": true,
            "offline": false,
            "faulted": false,
            "degraded": false,
            "removed": false,
            "not_present": false,
            "is_log": false,
            "path": "/var/tmp/zfs-vdev",
            "state": "HEALTHY",
            "vs_scan_removing": false,
            "vs_noalloc": false,
            "vs_resilver_deferred": false,
            "resilver_repair": "none",
            "initialize_state": {
              "vs_initialize_state": "VDEV_INITIALIZE_NONE",
              "vs_initialize_bytes_done": 0,
              "vs_initialize_bytes_est": 0,
              "vs_initialize_action_time": 0
            },
            "trim_state": {
              "vs_trim_state": "VDEV_UNTRIMMED",
              "vs_trim_action_time": 0,
              "vs_trim_bytes_done": 0,
              "vs_trim_bytes_est": 0
            },
            "read_errors": 0,
            "write_errors": 0,
            "checksum_errors": 0,
            "slow_ios": 0,
            "trim_errors": 0
          },
          {
            "type": "file",
            "id": 1,
            "guid": 15451542793245470000,
            "asize": 0,
            "ashift": 0,
            "whole_disk": true,
            "offline": false,
            "faulted": false,
            "degraded": false,
            "removed": false,
            "not_present": true,
            "is_log": false,
            "path": "/var/tmp/zfs-vdev2",
            "state": "UNAVAIL",
            "vs_scan_removing": false,
            "vs_noalloc": false,
            "vs_resilver_deferred": false,
            "resilver_repair": "none",
            "initialize_state": {
              "vs_initialize_state": "VDEV_INITIALIZE_NONE",
              "vs_initialize_bytes_done": 0,
              "vs_initialize_bytes_est": 0,
              "vs_initialize_action_time": 0
            },
            "trim_state": {
              "vs_trim_state": "VDEV_UNTRIMMED",
              "vs_trim_action_time": 0,
              "vs_trim_bytes_done": 0,
              "vs_trim_bytes_est": 0
            },
            "read_errors": 0,
            "write_errors": 0,
            "checksum_errors": 0,
            "slow_ios": 0,
            "trim_errors": 0
          }
        ]
      }
    ]
  }
}

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

This kstat output does not require taking the spa_namespace lock, as in the case for 'zpool status'. It can be used for investigations when pools are in a hung state while holding global locks required for a traditional 'zpool status' to proceed. This kstat is not safe to use in conditions where pools are in the process of configuration changes (i.e., adding/removing devices). Therefore, this kstat is not intended to be a general replacement or alternative to using 'zpool status'. Sponsored-by: Wasabi Technology, Inc. Sponsored-By: Klara Inc. Co-authored-by: Don Brady <don.brady@klarasystems.com> Signed-off-by: Don Brady <don.brady@klarasystems.com>

tonyhutter · 2024-03-27T01:04:48Z

Overall, I think JSON output is a good thing. It's something that's been on the ZFS wishlist since the dawn of time, and there have been some aborted attempts over the years to implement it.

Just some initial thoughts before I look at the code:

Even though this is a diagnostic kstat, everyone is going to want it for normal use since it's the only way to get JSON output. It will become a set in stone API. And if we're going to do a JSON API, we should just take the plunge and add it to the zpool commands proper. Having the JSON go though zpool status also allows us to later tack on the zpool status -c|-s fields to the JSON output, which would be nice.
If the desire is for a lockless option to get zpool status info (either in JSON or regular zpool status), then I like your 1b option to have a zpool status --ignore-locking. So you could do zpool status --ignore-locking or zpool status --ignore-locking --json depending on what you want. JSON in zpool status also allows you to get output from multiple pools in one command.
We may want to begin by just exposing the limited, historical, zpool status|get fields, and go from there. That would give us 90% of what people want, while initially limiting the scope to well-known, documented, fields. It also gets us out of the business of API-ifying esoteric internal variables like:

  "seconds_of_rewind": -1711139069,
...
  "initial_load_time": [
    1711398055,
    453276435
  ],
...
   "id": 0,

geoffamey · 2024-04-10T14:24:03Z

One major advantage that I see to this being a lockless kstat file is the ability for it to be parsed and used by metrics exporters like node_exporter, which currently uses the /proc/spl/kstat/zfs/<pool>/state file (among others).

Frequently, these tools are deployed in containers, where /proc/ is visible, and they are required to operate without calling third-party binaries or even having the zfs userspace utilities installed.

tonyhutter · 2024-04-10T23:13:31Z

Having the JSON go though zpool status also allows us to later tack on the zpool status -c|-s fields to the JSON output, which would be nice.
...
JSON in zpool status also allows you to get output from multiple pools in one command.

One major advantage that I see to this being a lockless kstat file is the ability for it to be parsed and used by metrics exporters like node_exporter, which currently uses the /proc/spl/kstat/zfs//state file

Just thinking out loud - we could potentially combine the two, and have zpool status --json grab the output from each of the pools in /proc/spl/kstat/zfs/<pool>/status.json, while tacking on any JSON from zpool status -c|-s output.

umsaleem · 2024-11-19T15:43:11Z

JFYI, if anybody is interested in carrying this work forward, it might be worth taking a look at #16484.

behlendorf added the Status: Code Review Needed Ready for review and testing label Mar 26, 2024

umsaleem mentioned this pull request Aug 28, 2024

Add a diagnostic kstat for obtaining pool status #16484

Closed

13 tasks

ihoro mentioned this pull request Feb 20, 2025

Add a diagnostic kstat for obtaining pool status #17076

Open

13 tasks

allanjude closed this Mar 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a diagnostic kstat for obtaining pool status #16026

Add a diagnostic kstat for obtaining pool status #16026

Uh oh!

don-brady commented Mar 25, 2024

Uh oh!

tonyhutter commented Mar 27, 2024

Uh oh!

geoffamey commented Apr 10, 2024

Uh oh!

tonyhutter commented Apr 10, 2024

Uh oh!

umsaleem commented Nov 19, 2024

Uh oh!

Uh oh!

Add a diagnostic kstat for obtaining pool status #16026

Add a diagnostic kstat for obtaining pool status #16026

Uh oh!

Conversation

don-brady commented Mar 25, 2024

Motivation and Context

Description

Exploring Solutions

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

tonyhutter commented Mar 27, 2024

Uh oh!

geoffamey commented Apr 10, 2024

Uh oh!

tonyhutter commented Apr 10, 2024

Uh oh!

umsaleem commented Nov 19, 2024

Uh oh!

Uh oh!