[Bug]: slurmctld crashes with double free or corruption when MetricsType=metrics/openmetrics enabled on 25.11 (Kubernetes + slurm-operator)

[narcotis]

Description

On Slurm 25.11, enabling the new metrics/openmetrics feature causes
slurmctld to periodically crash with glibc heap errors:

double free or corruption (fasttop)
double free or corruption (!prev)

Disabling only the metrics feature (removing MetricsType=metrics/openmetrics
from slurm.conf) fully resolves the problem. All other components and workflow
remain unchanged.

This appears to be a memory corruption issue triggered by the combination of:

• metrics/openmetrics,
• frequent node state updates (REQUEST_UPDATE_NODE) issued by a Kubernetes
  slurm-operator, and
• concurrent RPC requests (REQUEST_NODE_INFO, REQUEST_JOB_INFO,
  REQUEST_PARTITION_INFO) coming from local services.

Once the metrics plugin is disabled, slurmctld becomes stable under the same workload.

Steps to Reproduce

Environment:

• Slurmctld 25.11 (amd64, aarch64 both tested)
• Kubernetes deployment (controller + workers as Pods)
• slurm-operator (v1.0.0)
• Metrics plugin enabled:

slurm.conf

CgroupPlugin=autodetect
IgnoreSystemd=yes
EnableControllers=yes
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainDevices=yes
ConstrainSwapSpace=yes

AutoDetect=nvidia

MetricsType=metrics/openmetrics

Reproduction scenario:

1. Start slurmctld with metrics/openmetrics enabled.
2. Allow the slurm-operator to manage node states (Pod creation & deletion).
   This results in bursts of:

• REQUEST_UPDATE_NODE
• REQUEST_NODE_INFO
• REQUEST_PARTITION_INFO
• MESSAGE_NODE_REGISTRATION_STATUS

3. Trigger or wait for:

• reconfigures, or
• metric scrapes from monitoring systems.

4. Observe that slurmctld frequently aborts with:

double free or corruption (fasttop)
SIGABRT (core dumped)

Disabling only the metrics plugin makes all crashes disappear,
even under the same load, operator churn, and reconfigure events.

Expected Behavior

slurmctld should remain stable with metrics/openmetrics enabled,
including during:

• RECONFIGURE events,
• frequent node state updates from operators,
• concurrent metric scrapes,
• high RPC activity, and
• elevated debug logging (debug5).

Additional Context

Crash examples:

double free or corruption (!prev)
2025-12-09 05:16:11,712 WARN exited: slurmctld (SIGABRT, core dumped)

Crashes consistently align with:

• multiple simultaneous RPC connections from localhost:6817,
• rapid REQUEST_UPDATE_NODE (cordon/uncordon) calls,
• scheduler loop execution, and
• metric scrape or reconfigure timing.

Relevant configuration snippets:

MetricsType=metrics/openmetrics
SlurmctldDebug=debug5
CgroupPlugin=autodetect
TaskPlugin=task/cgroup,task/affinity
JobAcctGatherType=jobacct_gather/cgroup
GresTypes=gpu
AutoDetect=nvidia

For I'm working on AWS, I couldn't run more gdb bt to figure out the backtrace logs.

This Issue report is a summary created by ChatGPT, from the conversation debugging the issue with me.

[vivian-hafener]

Good morning @narcotis,

I will look into this.

Best,
Vivian Hafener

[vivian-hafener]

In the meantime, would it be possible for you to get a core dump for this, by using something like: https://github.com/IBM/core-dump-handler ?

[vivian-hafener]

Good morning @narcotis ,

I've spent some time trying to reproduce this this morning and have been unsuccessful. Could you provide more information about your environment? Of particular usefulness would be:

• Estimated job throughput of the cluster
• Cluster size
• Estimated frequency of metrics scraping
• Contents of controller.metrics block of helm/slurm/values.yaml that you used.

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener ,

Let me provide more informations about my environment:

controller.metrics block:

metrics:
    # -- Enable metrics.
    enabled: false
    # Prometheus service monitor configuration.
    serviceMonitor:
      # -- Enable serviceMonitor for metrics discovery.
      enabled: false
      # -- Annotations (metadata) added to the serviceMonitor.
      annotations: {}
      # -- Labels (metadata) added to the serviceMonitor.
      labels: {}
      # -- Interval at which Prometheus scrapes the metrics from the target (all endpoints).
      # If empty, the Prometheus default will be used instead.
      interval: 30s
      # -- ScrapeTimeout defines the timeout after which Prometheus considers the scrape to be failed (all endpoints).
      # If empty, the Prometheus default will be used instead.
      scrapeTimeout: 25s
      # -- Endpoint scrape configuration.
      # If empty, a default group of endpoints will be used.
      # Ref: https://slurm.schedmd.com/metrics.html#endpoints
      endpoints:
        - path: /metrics/jobs
        - path: /metrics/nodes
        - path: /metrics/partitions
        - path: /metrics/scheduler
        # - path: /metrics/jobs-users-accts

I've disabled the metrics for now, but It was enabled. I've tried both on/off for jobs-users-accts to check whether it's the problem but I couldn't get the answer.

In my cluster, the system can only scape one metric endpoint, so I used this metrics-merger sidecar:

# containers:
    #   - name: metrics-merger
    #     image: harbor.dev.hpcnt.com/devops/exporter-merger:v0.1.3
    #     args:
    #       - "--listen-port=6500"
    #       - "--url=http://localhost:6817/metrics/jobs"
    #       - "--url=http://localhost:6817/metrics/nodes"
    #       - "--url=http://localhost:6817/metrics/partitions"
    #       - "--url=http://localhost:6817/metrics/scheduler"
          # - "--url=http://localhost:6817/metrics/jobs-users-accts"
    # -- (map[string]string) Node label selector for pod assignment.

This is the same image from this repo : https://github.com/rebuy-de/exporter-merger

And I've checked the scape_interval, and it's 10s for the whole cluster.

Also, I've tried various persistent volumes to check whether the volume is the cause.
I tried EFS, FSx OpenZFS for the persistent volume, and neither worked.
But the interesting part was that the slurmctld didn't crash on the early period when I changed the volume, but it started soon to crash. slurmctld crashed irregularly, so I couldn't deep-dive more.

My Slinky Cluster has one Login Pod, and 10 GPU Pods (H100, p5.48xlarge).

Unfortunately my Slinky Cluster has just got released for beta usage in my team, so it's hard to try the metric value again now, but when the slurmctld started to crash, there was only me and my team debugging, so it was only about 1 ~ 2 users and 1 ~ 2 jobs running. (interactive pty bash jobs).

Actually I'm using slurm-exporter 0.4.1 now to provide metrics but I want to get the metrics about jobs-users-accts so I hope this Bug can be reproduced and fixed.

Let me know if more details is needed.

Best,
Narcotis

[vivian-hafener]

Good afternoon @narcotis ,

Thank you for the additional information. Could you provide more of the Slurmctld logs, ideally at debug2 or higher, from around the time of the crash? Having more context (ideally in the ballpark of 30+ lines) will help me gain a better understanding of what was occurring immediately prior to the crash. As of now I am still unable to reproduce this behavior.

Best,
Vivian Hafener

[narcotis]

@vivian-hafener Hi, this is the log when the slurmctld got crashed:

[2025-12-09T05:15:36] debug: sched: Running job scheduler for default depth.
[2025-12-09T05:15:36] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection
[2025-12-09T05:15:36] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401
[2025-12-09T05:15:36] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) was cordoned
[2025-12-09T05:15:36] update_node: node h100-1 state set to DRAINING 
[2025-12-09T05:15:36] debug2: _slurm_rpc_update_node complete for h100-1 usec=29
[2025-12-09T05:15:36] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:36] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:36] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401 
[2025-12-09T05:15:36] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) was cordoned pending termination 
[2025-12-09T05:15:36] update_node: node h100-1 state set to DRAINING 
[2025-12-09T05:15:36] debug2: _slurm_rpc_update_node complete for h100-1 usec=37 
[2025-12-09T05:15:36] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:36] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:36] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-09T05:15:36] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:36] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:36] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-09T05:15:36] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:38] debug: sackd_mgr_dump_state: saved state of 2 nodes 
[2025-12-09T05:15:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-09T05:15:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:38] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-09T05:15:38] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:38] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-09T05:15:38] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:39] debug: sched: Running job scheduler for default depth. 
[2025-12-09T05:15:40] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:40] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401 
[2025-12-09T05:15:40] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) was uncordoned 
[2025-12-09T05:15:40] update_node: node h100-1 state set to ALLOCATED 
[2025-12-09T05:15:40] debug2: _slurm_rpc_update_node complete for h100-1 usec=38 
[2025-12-09T05:15:40] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:40] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:40] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401 
[2025-12-09T05:15:40] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) has been uncordoned 
[2025-12-09T05:15:40] update_node: node h100-1 state set to ALLOCATED 
[2025-12-09T05:15:40] debug2: _slurm_rpc_update_node complete for h100-1 usec=42 
[2025-12-09T05:15:40] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:40] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:40] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-09T05:15:40] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:40] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:40] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-09T05:15:40] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:41] debug: Spawning registration agent for h100-1 1 hosts 
[2025-12-09T05:15:41] debug2: Spawning RPC agent for msg_type REQUEST_NODE_REGISTRATION_STATUS 
[2025-12-09T05:15:41] debug2: Tree head got back 0 looking for 1 
[2025-12-09T05:15:41] debug3: Tree sending to h100-1 
[2025-12-09T05:15:41] debug2: Tree head got back 1 
[2025-12-09T05:15:41] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:41] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from UID=0 
[2025-12-09T05:15:41] debug2: validate_jobs_on_node: registered JobId=8 SLUID=s1DNKC0KKNH700 StepId=8.extern on node h100-1 
[2025-12-09T05:15:41] debug3: Registered JobId=8 StepId=extern on node h100-1 
[2025-12-09T05:15:41] debug2: validate_jobs_on_node: registered JobId=9 SLUID=s1DNKC0KTAG300 StepId=9.extern on node h100-1 
[2025-12-09T05:15:41] debug3: Registered JobId=9 StepId=extern on node h100-1 
[2025-12-09T05:15:41] debug3: validate_node_specs: validating nodes h100-1 in state: ALLOCATED* 
[2025-12-09T05:15:41] Node h100-1 now responding 
[2025-12-09T05:15:41] debug2: _slurm_rpc_node_registration complete for h100-1 usec=78 
[2025-12-09T05:15:41] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:41] debug: sched: Running job scheduler for default depth. 
[2025-12-09T05:15:41] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:41] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-09T05:15:41] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:41] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:41] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-09T05:15:41] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:42] debug2: node_did_resp h100-1 
[2025-12-09T05:15:43] debug: sackd_mgr_dump_state: saved state of 2 nodes 
[2025-12-09T05:15:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:43] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401 
[2025-12-09T05:15:43] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) was cordoned 
[2025-12-09T05:15:43] update_node: node h100-1 state set to DRAINING 
[2025-12-09T05:15:43] debug2: _slurm_rpc_update_node complete for h100-1 usec=29 
[2025-12-09T05:15:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:43] debug2: Processing RPC: REQUEST_UPDATE_NODE from UID=401 
[2025-12-09T05:15:43] update_node: node h100-1 reason set to: slurm-operator: Pod (slinky/slurm-cluster-worker-h100-1) was cordoned pending termination 
[2025-12-09T05:15:43] update_node: node h100-1 state set to DRAINING 
[2025-12-09T05:15:43] debug2: _slurm_rpc_update_node complete for h100-1 usec=32 
[2025-12-09T05:15:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-09T05:15:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-09T05:15:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-09T05:15:44] debug: sched: Running job scheduler for default depth. 
[2025-12-09T05:15:44] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-09T05:15:44] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-09T05:15:44] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-09T05:15:44] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
double free or corruption (!prev) 
2025-12-09 05:16:11,712 WARN exited: slurmctld (terminated by SIGABRT (core dumped); not expected)
2025-12-09 05:16:11,712 INFO reaped unknown pid 26 (exit status 0)
2025-12-09 05:16:12,714 INFO spawned: 'slurmctld' with pid 121 + exec slurmctld --systemd
2025-12-09 05:16:13,719 INFO success: slurmctld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I have this too:

[2025-12-08T10:41:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:41:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:41:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:41:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:41:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:41:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:41:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:41:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:41:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:41:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:41:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:41:37] debug3: Called _msg_readable 
[2025-12-08T10:41:38] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:41:38] debug2: Performing purge of old job records 
[2025-12-08T10:41:38] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:41:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:41:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:41:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:41:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:41:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:41:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:41:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:41:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:41:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:41:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:41:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:41:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:41:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug: Spawning ping agent for h100-[0-9] 
[2025-12-08T10:42:00] debug2: Spawning RPC agent for msg_type REQUEST_PING 
[2025-12-08T10:42:00] debug3: Tree sending to h100-0 
[2025-12-08T10:42:00] debug3: Tree sending to h100-1 
[2025-12-08T10:42:00] debug3: Tree sending to h100-2 
[2025-12-08T10:42:00] debug3: Tree sending to h100-3 
[2025-12-08T10:42:00] debug3: Tree sending to h100-4 
[2025-12-08T10:42:00] debug3: Tree sending to h100-5 
[2025-12-08T10:42:00] debug2: Tree head got back 0 looking for 10 
[2025-12-08T10:42:00] debug3: Tree sending to h100-8 
[2025-12-08T10:42:00] debug3: Tree sending to h100-7 
[2025-12-08T10:42:00] debug3: Tree sending to h100-6 
[2025-12-08T10:42:00] debug3: Tree sending to h100-9 
[2025-12-08T10:42:00] debug2: Tree head got back 1 
[2025-12-08T10:42:00] debug2: Tree head got back 3 
[2025-12-08T10:42:00] debug2: Tree head got back 4 
[2025-12-08T10:42:00] debug2: Tree head got back 5 
[2025-12-08T10:42:00] debug2: Tree head got back 6 
[2025-12-08T10:42:00] debug2: Tree head got back 9 
[2025-12-08T10:42:00] debug2: Tree head got back 10 
[2025-12-08T10:42:00] debug2: Tree head got back 10 
[2025-12-08T10:42:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:00] debug: sched/backfill: _attempt_backfill: beginning 
[2025-12-08T10:42:00] debug: sched/backfill: _attempt_backfill: no jobs to backfill 
[2025-12-08T10:42:01] debug2: node_did_resp h100-1 
[2025-12-08T10:42:01] debug2: node_did_resp h100-0 
[2025-12-08T10:42:01] debug2: node_did_resp h100-7 
[2025-12-08T10:42:01] debug2: node_did_resp h100-5 
[2025-12-08T10:42:01] debug2: node_did_resp h100-2 
[2025-12-08T10:42:01] debug2: node_did_resp h100-6 
[2025-12-08T10:42:01] debug2: node_did_resp h100-9 
[2025-12-08T10:42:01] debug2: node_did_resp h100-3 
[2025-12-08T10:42:01] debug2: node_did_resp h100-4 
[2025-12-08T10:42:01] debug2: node_did_resp h100-8 
[2025-12-08T10:42:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:08] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:42:10] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:10] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:10] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:10] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:42:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:42:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:42:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:30] debug: sched/backfill: _attempt_backfill: beginning 
[2025-12-08T10:42:30] debug: sched/backfill: _attempt_backfill: no jobs to backfill 
[2025-12-08T10:42:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:42:37] debug3: Called _msg_readable 
[2025-12-08T10:42:38] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:42:38] debug2: Performing purge of old job records 
[2025-12-08T10:42:38] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:42:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:42:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:42:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:42:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:42:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:42:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:42:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:42:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:42:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:42:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:07] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:43:10] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:10] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:10] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:10] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:43:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:43:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:43:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:37] debug4: priority/multifactor: _write_last_decay_ran: done writing time 1765190617 
[2025-12-08T10:43:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:43:37] debug3: Called _msg_readable 
[2025-12-08T10:43:38] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:43:38] debug2: Performing purge of old job records 
[2025-12-08T10:43:38] debug2: Performing full system state save 
[2025-12-08T10:43:38] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:43:38] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/slinky_slurm-cluster/job_state` as buf_t 
[2025-12-08T10:43:38] debug3: Writing job id 18 to header record of job_state file 
[2025-12-08T10:43:38] debug: sackd_mgr_dump_state: saved state of 0 nodes 
[2025-12-08T10:43:38] debug2: accounting_storage/slurmdbd: _send_cluster_tres: Sending tres '1=1920,2=20479590,3=0,4=10,5=1920,6=0,7=0,8=0,1001=160,1002=0,1003=0' for cluster 
[2025-12-08T10:43:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:43:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:43:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:43:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:43:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:43:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:43:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:43:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:43:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:43:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:07] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:44:10] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:10] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:10] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:10] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:44:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:44:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:44:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:37] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:44:37] debug2: Performing purge of old job records 
[2025-12-08T10:44:37] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:44:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:44:37] debug3: Called _msg_readable 
[2025-12-08T10:44:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:44:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:44:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:44:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:44:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:44:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:44:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:44:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:44:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:44:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:07] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:45:10] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:10] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:10] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:10] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:45:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug: Spawning ping agent for h100-[0-9] 
[2025-12-08T10:45:20] debug2: Spawning RPC agent for msg_type REQUEST_PING 
[2025-12-08T10:45:20] debug3: Tree sending to h100-0 
[2025-12-08T10:45:20] debug3: Tree sending to h100-1 
[2025-12-08T10:45:20] debug3: Tree sending to h100-2 
[2025-12-08T10:45:20] debug3: Tree sending to h100-3 
[2025-12-08T10:45:20] debug3: Tree sending to h100-4 
[2025-12-08T10:45:20] debug2: Tree head got back 0 looking for 10 
[2025-12-08T10:45:20] debug3: Tree sending to h100-7 
[2025-12-08T10:45:20] debug3: Tree sending to h100-9 
[2025-12-08T10:45:20] debug3: Tree sending to h100-5 
[2025-12-08T10:45:20] debug3: Tree sending to h100-8 
[2025-12-08T10:45:20] debug3: Tree sending to h100-6 
[2025-12-08T10:45:20] debug2: Tree head got back 1 
[2025-12-08T10:45:20] debug2: Tree head got back 2 
[2025-12-08T10:45:20] debug2: Tree head got back 3 
[2025-12-08T10:45:20] debug2: Tree head got back 5 
[2025-12-08T10:45:20] debug2: Tree head got back 7 
[2025-12-08T10:45:20] debug2: Tree head got back 8 
[2025-12-08T10:45:20] debug2: Tree head got back 10 
[2025-12-08T10:45:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:20] debug: sched/backfill: _attempt_backfill: beginning 
[2025-12-08T10:45:20] debug: sched/backfill: _attempt_backfill: no jobs to backfill 
[2025-12-08T10:45:21] debug2: node_did_resp h100-5 
[2025-12-08T10:45:21] debug2: node_did_resp h100-9 
[2025-12-08T10:45:21] debug2: node_did_resp h100-6 
[2025-12-08T10:45:21] debug2: node_did_resp h100-2 
[2025-12-08T10:45:21] debug2: node_did_resp h100-8 
[2025-12-08T10:45:21] debug2: node_did_resp h100-7 
[2025-12-08T10:45:21] debug2: node_did_resp h100-0 
[2025-12-08T10:45:21] debug2: node_did_resp h100-1 
[2025-12-08T10:45:21] debug2: node_did_resp h100-3 
[2025-12-08T10:45:21] debug2: node_did_resp h100-4 
[2025-12-08T10:45:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:31] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:31] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:31] debug2: Processing RPC: REQUEST_COMPLETE_JOB_ALLOCATION from UID=1048 
[2025-12-08T10:45:31] debug3: Processing RPC details: REQUEST_COMPLETE_JOB_ALLOCATION for JobId=17 SLUID=s1DNKBYRK7MF00 rc=32512 
[2025-12-08T10:45:31] _job_complete: JobId=17 WEXITSTATUS 127 
[2025-12-08T10:45:31] debug2: Spawning RPC agent for msg_type SRUN_JOB_COMPLETE 
[2025-12-08T10:45:31] debug3: select/cons_tres: job_res_rm_job: JobId=17 action:normal 
[2025-12-08T10:45:31] debug3: select/cons_tres: job_res_rm_job: removed JobId=17 from part h100 row 0 
[2025-12-08T10:45:31] debug: slurm_cred_create: creating credential for JobId=17 SLUID=s1DNKBYRK7MF00 
[2025-12-08T10:45:31] debug2: slurm_cred_create: identity: uid=1048 gid=20000 pw_name=narco-y pw_gecos=narco-y pw_dir=/home/narco-y pw_shell=/bin/bash ngids=1 groups=hpcnt(20000) 
[2025-12-08T10:45:31] _job_complete: JobId=17 done 
[2025-12-08T10:45:31] debug2: _slurm_rpc_complete_job_allocation: JobId=17 
[2025-12-08T10:45:31] debug2: Spawning RPC agent for msg_type REQUEST_TERMINATE_JOB 
[2025-12-08T10:45:31] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:31] debug2: Tree head got back 0 looking for 1 
[2025-12-08T10:45:31] debug3: Tree sending to h100-0 
[2025-12-08T10:45:31] debug2: assoc_mgr_set_uid: uid=0 already known 
[2025-12-08T10:45:31] debug2: Processing RPC: REQUEST_DBD_RELAY from UID=0 
[2025-12-08T10:45:31] debug3: Processing RPC details: REQUEST_DBD_RELAY 
[2025-12-08T10:45:31] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:31] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:31] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/slinky_slurm-cluster/job_state` as buf_t 
[2025-12-08T10:45:31] debug3: Writing job id 18 to header record of job_state file 
[2025-12-08T10:45:31] debug: sackd_mgr_dump_state: saved state of 0 nodes 
[2025-12-08T10:45:31] debug2: Tree head got back 1 
[2025-12-08T10:45:31] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:31] debug2: assoc_mgr_set_uid: uid=0 already known 
[2025-12-08T10:45:31] debug2: Processing RPC: REQUEST_STEP_COMPLETE from UID=0 
[2025-12-08T10:45:31] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:31] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:31] debug2: Processing RPC: MESSAGE_EPILOG_COMPLETE from UID=0 
[2025-12-08T10:45:31] debug2: _slurm_rpc_epilog_complete: JobId=17 Node=h100-0 
[2025-12-08T10:45:31] debug: sched: Running job scheduler for default depth. 
[2025-12-08T10:45:31] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:32] debug2: node_did_resp h100-0 
[2025-12-08T10:45:34] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:34] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:34] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=1048 
[2025-12-08T10:45:34] debug3: _build_visible_parts_foreach: uid:1048 visible_parts:(null) 
[2025-12-08T10:45:34] debug3: part_not_on_list: partition: h100 on visible part list 
[2025-12-08T10:45:34] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:34] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:34] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:34] debug2: Processing RPC: REQUEST_NODE_INFO from UID=1048 
[2025-12-08T10:45:34] debug3: _build_visible_parts_foreach: uid:1048 visible_parts:(null) 
[2025-12-08T10:45:34] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:36] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/slinky_slurm-cluster/job_state` as buf_t 
[2025-12-08T10:45:36] debug3: Writing job id 18 to header record of job_state file 
[2025-12-08T10:45:36] debug: sackd_mgr_dump_state: saved state of 0 nodes 
[2025-12-08T10:45:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:45:37] debug3: Called _msg_readable 
[2025-12-08T10:45:38] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:45:38] debug2: Performing purge of old job records 
[2025-12-08T10:45:38] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:45:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:38] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:38] debug2: Processing RPC: REQUEST_JOB_INFO from UID=1048 
[2025-12-08T10:45:38] debug3: _build_visible_parts_foreach: uid:1048 visible_parts:(null) 
[2025-12-08T10:45:38] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:38] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:38] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:38] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=1048 
[2025-12-08T10:45:38] debug3: _build_visible_parts_foreach: uid:1048 visible_parts:(null) 
[2025-12-08T10:45:38] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:45:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:44] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:44] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:44] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:44] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:44] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:44] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_RESOURCE_ALLOCATION from UID=1048 
[2025-12-08T10:45:45] debug3: sched: Processing RPC: REQUEST_RESOURCE_ALLOCATION from uid=1048 
[2025-12-08T10:45:45] debug3: auth_g_get_host: using auth token: slurm-cluster-login-slinky-6996986d64-xjhqg 
[2025-12-08T10:45:45] debug3: JobDesc: user_id=1048 JobId=N/A partition=h100 name=bash 
[2025-12-08T10:45:45] debug3: cpus=1-4294967294 pn_min_cpus=-1 core_spec=-1 
[2025-12-08T10:45:45] debug3: Nodes=1-[4294967294] Sock/Node=65534 Core/Sock=65534 Thread/Core=65534 
[2025-12-08T10:45:45] debug3: pn_min_memory_job=18446744073709551615 pn_min_tmp_disk=-1 
[2025-12-08T10:45:45] debug3: immediate=0 reservation=(null) 
[2025-12-08T10:45:45] debug3: features=(null) batch_features=(null) cluster_features=(null) prefer=(null) 
[2025-12-08T10:45:45] debug3: req_nodes=(null) exc_nodes=(null) 
[2025-12-08T10:45:45] debug3: time_limit=-1--1 priority=-1 contiguous=0 shared=-1 
[2025-12-08T10:45:45] debug3: kill_on_node_fail=-1 script=(null) 
[2025-12-08T10:45:45] debug3: argv="bash" 
[2025-12-08T10:45:45] debug3: stdin=(null) stdout=(null) stderr=(null) 
[2025-12-08T10:45:45] debug3: work_dir=/home/narco-y alloc_node:sid=slurm-cluster-login-slinky-6996986d64-xjhqg:26350 
[2025-12-08T10:45:45] debug3: resp_host=10.121.50.34 alloc_resp_port=38681 other_port=42247 
[2025-12-08T10:45:45] debug3: dependency=(null) account=(null) qos=(null) comment=(null) 
[2025-12-08T10:45:45] debug3: mail_type=0 mail_user=(null) nice=0 num_tasks=-1 open_mode=0 overcommit=-1 acctg_freq=(null) 
[2025-12-08T10:45:45] debug3: network=(null) begin=Unknown cpus_per_task=-1 requeue=-1 licenses=(null) 
[2025-12-08T10:45:45] debug3: end_time= signal=0@0 wait_all_nodes=1 cpu_freq= 
[2025-12-08T10:45:45] debug3: ntasks_per_node=-1 ntasks_per_socket=-1 ntasks_per_core=-1 ntasks_per_tres=-1 
[2025-12-08T10:45:45] debug3: mem_bind=0:(null) plane_size:65534 
[2025-12-08T10:45:45] debug3: array_inx=(null) 
[2025-12-08T10:45:45] debug3: burst_buffer=(null) 
[2025-12-08T10:45:45] debug3: mcs_label=(null) 
[2025-12-08T10:45:45] debug3: deadline=Unknown 
[2025-12-08T10:45:45] debug3: bitflags=0x1a000000 delay_boot=4294967294 
[2025-12-08T10:45:45] debug3: User (null)(1048) doesn't have a default account 
[2025-12-08T10:45:45] debug3: User (null)(1048) doesn't have a default account 
[2025-12-08T10:45:45] debug3: found correct qos 
[2025-12-08T10:45:45] debug2: priority/multifactor: priority_p_set: initial priority for job 18 is 1 
[2025-12-08T10:45:45] debug2: found 10 usable nodes from config containing h100-[0-9] 
[2025-12-08T10:45:45] debug2: NodeSet for JobId=18 
[2025-12-08T10:45:45] debug2: NodeSet[0] Nodes:h100-[0-9] NodeWeight:1 Flags:0 FeatureBits:0 SchedWeight:511 
[2025-12-08T10:45:45] debug3: _pick_best_nodes: JobId=18 idle_nodes 10 share_nodes 1024 
[2025-12-08T10:45:45] debug2: select/cons_tres: select_p_job_test: evaluating JobId=18 
[2025-12-08T10:45:45] debug3: select/cons_tres: job_res_add_job: JobId=18 action:normal 
[2025-12-08T10:45:45] debug3: select/cons_tres: job_res_add_job: adding JobId=18 to part h100 row 0 
[2025-12-08T10:45:45] debug: slurm_cred_create: creating credential for JobId=18 SLUID=s1DNKBYS2FGD00 
[2025-12-08T10:45:45] debug2: slurm_cred_create: identity: uid=1048 gid=20000 pw_name=narco-y pw_gecos=narco-y pw_dir=/home/narco-y pw_shell=/bin/bash ngids=1 groups=hpcnt(20000) 
[2025-12-08T10:45:45] debug2: sched: JobId=18 allocated resources: NodeList=h100-0 
[2025-12-08T10:45:45] sched: _slurm_rpc_allocate_resources JobId=18 NodeList=h100-0 usec=3105 
[2025-12-08T10:45:45] debug2: Spawning RPC agent for msg_type REQUEST_LAUNCH_PROLOG 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug2: Tree head got back 0 looking for 1 
[2025-12-08T10:45:45] debug3: Tree sending to h100-0 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug3: create_mmap_buf: loaded file `/var/spool/slurmctld/slinky_slurm-cluster/job_state` as buf_t 
[2025-12-08T10:45:45] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:45] debug2: Tree head got back 1 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_JOB_READY from UID=1048 
[2025-12-08T10:45:45] debug2: _slurm_rpc_job_ready: JobId=18 result 7 in usec=3 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:20)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug3: Writing job id 19 to header record of job_state file 
[2025-12-08T10:45:45] debug: sackd_mgr_dump_state: saved state of 0 nodes 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_COMPLETE_PROLOG from UID=0 
[2025-12-08T10:45:45] debug3: Processing RPC details: REQUEST_COMPLETE_PROLOG from JobId=18 SLUID=s1DNKBYS2FGD00 
[2025-12-08T10:45:45] debug2: _slurm_rpc_complete_prolog: JobId=18 SLUID=s1DNKBYS2FGD00 usec=22 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_JOB_READY from UID=1048 
[2025-12-08T10:45:45] debug2: _slurm_rpc_job_ready: JobId=18 result 7 in usec=4 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug2: assoc_mgr_set_uid: user narco-y not in assoc_mgr_user_list 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_NODE_ALIAS_ADDRS from UID=1048 
[2025-12-08T10:45:45] debug3: Processing RPC details: REQUEST_NODE_ALIAS_ADDRS 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:45] debug2: assoc_mgr_set_uid: uid=0 already known 
[2025-12-08T10:45:45] debug2: Processing RPC: REQUEST_DBD_RELAY from UID=0 
[2025-12-08T10:45:45] debug3: Processing RPC details: REQUEST_DBD_RELAY 
[2025-12-08T10:45:45] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:46] debug2: node_did_resp h100-0 
[2025-12-08T10:45:46] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:46] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:46] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:46] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:46] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:46] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:47] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:47] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:47] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:47] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:47] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:47] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:49] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:49] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:49] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:49] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:49] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:49] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug: sched/backfill: _attempt_backfill: beginning 
[2025-12-08T10:45:50] debug: sched/backfill: _attempt_backfill: no jobs to backfill 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:45:50] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:50] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:45:50] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:45:50] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:45:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:45:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:05] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:07] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:46:10] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:10] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:10] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:10] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:10] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:13] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:46:13] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:46:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:13] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:13] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:46:13] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:15] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:20] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:20] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:20] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:20] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:20] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:25] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:30] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:30] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:30] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:30] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:30] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:35] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:37] debug2: Testing job time limits and checkpoints 
[2025-12-08T10:46:37] debug2: Performing purge of old job records 
[2025-12-08T10:46:37] debug: sched: Running job scheduler for full queue. 
[2025-12-08T10:46:37] debug4: eio: handling events for 1 objects 
[2025-12-08T10:46:37] debug3: Called _msg_readable 
[2025-12-08T10:46:40] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:40] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:40] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:40] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:40] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:43] debug2: Processing RPC: REQUEST_JOB_INFO from UID=401 
[2025-12-08T10:46:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:43] debug2: Processing RPC: REQUEST_NODE_INFO from UID=401 
[2025-12-08T10:46:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:43] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:43] debug2: Processing RPC: REQUEST_PARTITION_INFO from UID=401 
[2025-12-08T10:46:43] debug3: _on_primary_finish: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: RPC connection closed 
[2025-12-08T10:46:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:45] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:50] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:50] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:46:50] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection 
[2025-12-08T10:46:50] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection 
[2025-12-08T10:46:50] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:46:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:46:55] debug3: _on_primary_connection: [slurm-cluster-controller-0.slurm-cluster-controller.slinky.svc.cluster.local:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:47:00] debug3: _on_primary_connection: [localhost:6817(fd:19)] PRIMARY: New RPC connection 
[2025-12-08T10:47:00] debug3: _on_primary_connection: [localhost:6817(fd:4)] PRIMARY: New RPC connection 
[2025-12-08T10:47:00] debug3: _on_primary_connection: [localhost:6817(fd:20)] PRIMARY: New RPC connection 
[2025-12-08T10:47:00] debug3: _on_primary_connection: [localhost:6817(fd:21)] PRIMARY: New RPC connection double free or corruption (fasttop) 
[2025-12-08T10:47:00] debug3: _on_primary_connection: [localhost:6817(fd:22)] PRIMARY: New RPC connection
2025-12-08 10:47:29,102 WARN exited: slurmctld (terminated by SIGABRT (core dumped); not expected)
2025-12-08 10:47:29,102 INFO reaped unknown pid 26 (exit status 0)
2025-12-08 10:47:30,104 INFO spawned: 'slurmctld' with pid 111 + exec slurmctld --systemd
2025-12-08 10:47:31,109 INFO success: slurmctld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I've set slurmctld & slurmd log for debug5

SlurmctldDebug: debug5
SlurmdDebug: debug5

Please let me know, if you need more informations.

Best,
Narcotis

[vivian-hafener]

Hi @narcotis,

Thank you for providing that log. While it is helpful, I am unable to determine a root cause from it at this time. If you are able to provide us with a backtrace, that would be immensely helpful, as I am presently unable to reproduce this failure myself.

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener,

I was finally able to collect multiple coredumps for slurmctld.
To avoid disrupting my existing production Slurm cluster, I launched a separate debug-only cluster, consisting of a single CPU worker node, and reproduced the issue there.

Because the controller immediately enters CrashLoopBackOff and the environment is running on ARM64, it was difficult to capture these failures. I pulled the core files directly from the node’s /var/lib/systemd/coredump and analyzed them in an ARM64 Ubuntu environment using gdb.

I analyzed the coredumps inside a local ARM64 Ubuntu container (via Docker on macOS), using the exact same slurmctld binary extracted from the node.

In total, I examined 7 coredumps.
Among them:

• 6 out of 7 crashed with
  double free or corruption (fasttop)

• 1 out of 7 crashed with
  malloc(): unaligned fastbin chunk detected

Both error types indicate consistent heap corruption inside slurmctld, triggered either during free() or during malloc/calloc().

Below are the raw gdb bt full logs for three representative coredumps (no modifications).

If needed, I can also email the full .xz core files privately.

Core Dump No.1 - bt full

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `slurmctld --systemd'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=281473503850912, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
[Current thread is 1 (Thread 0xffffa835f1a0 (LWP 352))]
(gdb) bt full
#0  __pthread_kill_implementation (threadid=281473503850912, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = 352
        ret = 0
        pd = 0xffffa835f1a0
        old_mask = {__val = {18442257035476008960}}
        ret = <optimized out>
#1  0x0000ffffa8527670 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x0000ffffa84dcb3c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x0000ffffa84c7ea4 in __GI_abort () at ./stdlib/abort.c:100
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {18446744073709551615, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x0}
#4  0x0000ffffa851abf4 in __libc_message_impl (fmt=fmt@entry=0xffffa8603bc0 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
        ap = {__stack = 0xffffa835e250, __gr_top = 0xffffa835e250, __vr_top = 0xffffa835e210, __gr_offs = -48, __vr_offs = 0}
        fd = 2
        iov = {{iov_base = 0xffffa85ff0e0, iov_len = 35}, {iov_base = 0xffffa8603bc2, iov_len = 1}, {iov_base = 0xffff7c060380, iov_len = 0}, {iov_base = 0x0, iov_len = 281473503846864}, {iov_base = 0xffffa8536f88 <__GI___libc_free+216>, iov_len = 281472762136192}, {iov_base = 0xffffa835f8e0,
            iov_len = 281473506961136}, {iov_base = 0x20, iov_len = 11}}
        iovcnt = <optimized out>
        total = <optimized out>
        cp = <optimized out>
#5  0x0000ffffa853210c in malloc_printerr (str=str@entry=0xffffa85ff0e0 "double free or corruption (fasttop)") at ./malloc/malloc.c:5772
No locals.
#6  0x0000ffffa853445c in _int_free (av=<optimized out>, p=p@entry=0xffff8400aaf0, have_lock=have_lock@entry=0) at ./malloc/malloc.c:4607
        idx = 6
        old = 0xffff8400aaf0
        old2 = <optimized out>
        size = <optimized out>
        fb = <optimized out>
#7  0x0000ffffa8536f88 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3398
        ar_ptr = <optimized out>
        p = 0xffff8400aaf0
        err = <optimized out>
#8  0x0000ffffa87c3e8c in ?? ()
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Core Dump No.2 - bt full

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `slurmctld --systemd'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=281473533669792, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
[Current thread is 1 (Thread 0xffffa9fcf1a0 (LWP 9))]
(gdb) bt full
#0  __pthread_kill_implementation (threadid=281473533669792, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = 9
        ret = 0
        pd = 0xffffa9fcf1a0
        old_mask = {__val = {281473536386576}}
        ret = <optimized out>
#1  0x0000ffffaa197670 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x0000ffffaa14cb3c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x0000ffffaa137ea4 in __GI_abort () at ./stdlib/abort.c:100
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {18446744073709551615, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x0}
#4  0x0000ffffaa18abf4 in __libc_message_impl (fmt=fmt@entry=0xffffaa273bc0 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
        ap = {__stack = 0xffffa9fcdfc0, __gr_top = 0xffffa9fcdfc0, __vr_top = 0xffffa9fcdf80, __gr_offs = -48, __vr_offs = 0}
        fd = 2
        iov = {{iov_base = 0xffffaa26f2b8, iov_len = 42}, {iov_base = 0xffffaa273bc2, iov_len = 1}, {iov_base = 0xffffaa43cea0, iov_len = 0}, {iov_base = 0xffffaa43cf04, iov_len = 281473533665536}, {iov_base = 0xffffa9fce100, iov_len = 281473533665488}, {iov_base = 0xffffff80ffffffd0, iov_len = 281473533665536}, {
            iov_base = 0x59f1879398ac5100, iov_len = 281473533665168}}
        iovcnt = <optimized out>
        total = <optimized out>
        cp = <optimized out>
#5  0x0000ffffaa1a210c in malloc_printerr (str=str@entry=0xffffaa26f2b8 "malloc(): unaligned fastbin chunk detected") at ./malloc/malloc.c:5772
No locals.
#6  0x0000ffffaa1a4f88 in _int_malloc (av=av@entry=0xffff9c000030, bytes=bytes@entry=116) at ./malloc/malloc.c:3929
        fb = <optimized out>
        pp = <optimized out>
        nb = <optimized out>
        idx = <optimized out>
        bin = <optimized out>
        victim = <optimized out>
        size = <optimized out>
        victim_index = <optimized out>
        remainder = <optimized out>
        remainder_size = <optimized out>
        block = <optimized out>
        bit = <optimized out>
        map = <optimized out>
        fwd = <optimized out>
        bck = <optimized out>
        tcache_unsorted_count = <optimized out>
        tcache_nb = <optimized out>
        tc_idx = <optimized out>
        return_cached = <optimized out>
        __PRETTY_FUNCTION__ = "_int_malloc"
#7  0x0000ffffaa1a7aa4 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at ./malloc/malloc.c:3754
        av = <optimized out>
        oldtop = 0xffff9c066b90
        sz = 116
        oldtopsize = 263280
        mem = <optimized out>
        clearsize = <optimized out>
        nclears = <optimized out>
        d = <optimized out>
        bytes = 116
        __PRETTY_FUNCTION__ = "__libc_calloc"
        p = <optimized out>
        csz = <optimized out>
#8  0x0000ffffaa43a728 in ?? ()
No symbol table info available.
#9  0x0000ffffa9fce1a0 in ?? ()
No symbol table info available.
#10 0x0000ffff9c000030 in ?? ()
No symbol table info available.
#11 0x0000ffffa9fcf658 in ?? ()
No symbol table info available.
#12 0x0000ffffaa2bd1d8 in _nl_C_LC_ADDRESS () from /lib/aarch64-linux-gnu/libc.so.6
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Core Dump No.3 - bt full

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `slurmctld --systemd'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=281473760686496, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44

warning: 44	./nptl/pthread_kill.c: No such file or directory
[Current thread is 1 (Thread 0xffffb784f1a0 (LWP 14))]
(gdb) bt full
#0  __pthread_kill_implementation (threadid=281473760686496, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = 14
        ret = 0
        pd = 0xffffb784f1a0
        old_mask = {__val = {7237124276286089586}}
        ret = <optimized out>
#1  0x0000ffffba267670 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x0000ffffba21cb3c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x0000ffffba207ea4 in __GI_abort () at ./stdlib/abort.c:100
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {18446744073709551615, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x0}
#4  0x0000ffffba25abf4 in __libc_message_impl (fmt=fmt@entry=0xffffba343bc0 "%s\n") at ../sysdeps/posix/libc_fatal.c:134
        ap = {__stack = 0xffffb784e2c0, __gr_top = 0xffffb784e2c0, __vr_top = 0xffffb784e280, __gr_offs = -48, __vr_offs = 0}
        fd = 2
        iov = {{iov_base = 0xffffba33f0e0, iov_len = 35}, {iov_base = 0xffffba343bc2, iov_len = 1}, {iov_base = 0xffffa40664e0, iov_len = 0}, {iov_base = 0x0, iov_len = 281473760682560}, {iov_base = 0xffffba276f88 <__GI___libc_free+216>, iov_len = 281473433356688}, {iov_base = 0xffffb784f8e0,
            iov_len = 281473806067440}, {iov_base = 0x20, iov_len = 11}}
        iovcnt = <optimized out>
        total = <optimized out>
        cp = <optimized out>
#5  0x0000ffffba27210c in malloc_printerr (str=str@entry=0xffffba33f0e0 "double free or corruption (fasttop)") at ./malloc/malloc.c:5772
No locals.
#6  0x0000ffffba27445c in _int_free (av=<optimized out>, p=p@entry=0xffff980028f0, have_lock=have_lock@entry=0) at ./malloc/malloc.c:4607
        idx = 6
        old = 0xffff980028f0
        old2 = <optimized out>
        size = <optimized out>
        fb = <optimized out>
#7  0x0000ffffba276f88 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3398
        ar_ptr = <optimized out>
        p = 0xffff980028f0
        err = <optimized out>
#8  0x0000ffffba503e8c in ?? ()
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

---

Interpretation

Across 7 crashes:

• 6 show fasttop corruption during free()
• 1 shows fastbin corruption during calloc()
• Crash locations inside slurmctld repeatedly point to offsets such as
  …3e8c and …a728 (likely the same call sites across ASLR ranges)

This strongly suggests a reproducible heap corruption bug in slurmctld (Slurm 25.11.0).

Unfortunately, since the official container image is fully stripped, I cannot map these addresses to specific functions or source lines. With debug symbols, I could provide exact mappings.

---

Original coredump filenames (for reference)

[root@ip-10-121-38-78 /]# cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
[root@ip-10-121-38-78 /]# coredumpctl list | grep -Ei 'slurm|slurmd|slurmdbd'
Wed 2025-12-17 14:43:51 UTC 136137 401 401 SIGABRT present  /usr/sbin/slurmctld 400.6K
Wed 2025-12-17 14:57:01 UTC 140296 401 401 SIGABRT present  /usr/sbin/slurmctld 373.0K
Wed 2025-12-17 15:15:41 UTC 149523 401 401 SIGABRT present  /usr/sbin/slurmctld 387.8K
Wed 2025-12-17 15:57:41 UTC 183685 401 401 SIGABRT present  /usr/sbin/slurmctld 385.9K
Wed 2025-12-17 15:59:40 UTC 192520 401 401 SIGABRT present  /usr/sbin/slurmctld 350.8K
Wed 2025-12-17 16:12:11 UTC 194105 401 401 SIGABRT present  /usr/sbin/slurmctld 377.1K
Wed 2025-12-17 16:18:41 UTC 202919 401 401 SIGABRT present  /usr/sbin/slurmctld 365.1K
[root@ip-10-121-38-78 /]# ls -lh /var/lib/systemd/coredump | head
ls -lh /var/lib/systemd/coredump | grep slurm
total 2.6M
-rw-r----- 1 root root 401K Dec 17 14:43 core.slurmctld.401.a014944f95be420f888aca223e9031e5.136137.1765982609000000.xz
-rw-r----- 1 root root 374K Dec 17 14:57 core.slurmctld.401.a014944f95be420f888aca223e9031e5.140296.1765983399000000.xz
-rw-r----- 1 root root 388K Dec 17 15:15 core.slurmctld.401.a014944f95be420f888aca223e9031e5.149523.1765984519000000.xz
-rw-r----- 1 root root 386K Dec 17 15:57 core.slurmctld.401.a014944f95be420f888aca223e9031e5.183685.1765987039000000.xz
-rw-r----- 1 root root 351K Dec 17 15:59 core.slurmctld.401.a014944f95be420f888aca223e9031e5.192520.1765987159000000.xz
-rw-r----- 1 root root 378K Dec 17 16:12 core.slurmctld.401.a014944f95be420f888aca223e9031e5.194105.1765987909000000.xz
-rw-r----- 1 root root 366K Dec 17 16:18 core.slurmctld.401.a014944f95be420f888aca223e9031e5.202919.1765988299000000.xz
-rw-r----- 1 root root 401K Dec 17 14:43 core.slurmctld.401.a014944f95be420f888aca223e9031e5.136137.1765982609000000.xz
-rw-r----- 1 root root 374K Dec 17 14:57 core.slurmctld.401.a014944f95be420f888aca223e9031e5.140296.1765983399000000.xz
-rw-r----- 1 root root 388K Dec 17 15:15 core.slurmctld.401.a014944f95be420f888aca223e9031e5.149523.1765984519000000.xz
-rw-r----- 1 root root 386K Dec 17 15:57 core.slurmctld.401.a014944f95be420f888aca223e9031e5.183685.1765987039000000.xz
-rw-r----- 1 root root 351K Dec 17 15:59 core.slurmctld.401.a014944f95be420f888aca223e9031e5.192520.1765987159000000.xz
-rw-r----- 1 root root 378K Dec 17 16:12 core.slurmctld.401.a014944f95be420f888aca223e9031e5.194105.1765987909000000.xz
-rw-r----- 1 root root 366K Dec 17 16:18 core.slurmctld.401.a014944f95be420f888aca223e9031e5.202919.1765988299000000.xz

I analyzed seven of these; three are shown above, but I can share all of them if needed.

If you would like the full .xz core dump files, I can send them via email or through any secure method you prefer.

Thanks,
Narcotis

[vivian-hafener]

Hi @narcotis , thank you for this!

If you email me the full .xz core dumps at vivian@schedmd.com, I can get those over to the Slurm developers who wrote that plugin.

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener ,

Thanks for the quick response.
I’ve just sent the full .xz core dumps to your email (vivian@schedmd.com).
Please let me know if you need anything else on my side.

Best,
Narcotis