Got Essential task `overseer` failed error after upgrading Kusama and Polkadot validator to v1.5.0

[AlexZhenWang]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Hi, I am trying to upgrade our both Polkadot and Kusama validator to v1.5.0. But I got an Essential task overseer failed error after upgrading Kusama and Polkadot validator to v1.5.0.

After downgrading back to v1.4.0, the issue gone. This error happened on both Polkadot and Kusama validator.
The logs:

---

Update:
logs

2023-12-18 01:10:33 ----------------------------    
2023-12-18 01:10:33 This chain is not in any way    
2023-12-18 01:10:33       endorsed by the           
2023-12-18 01:10:33      KUSAMA FOUNDATION          
2023-12-18 01:10:33 ----------------------------    
2023-12-18 01:10:33 Parity Polkadot    
2023-12-18 01:10:33 ✌️  version 1.5.0-a3dc2f15f23    
2023-12-18 01:10:33 ❤️  by Parity Technologies <admin@parity.io>, 2017-2023    
2023-12-18 01:10:33 📋 Chain specification: Kusama    
2023-12-18 01:10:33 🏷  Node name: OnfinalityV#1    
2023-12-18 01:10:33 👤 Role: AUTHORITY    
2023-12-18 01:10:33 💾 Database: RocksDb at /chain-data/chains/ksmcc3/db/full    
2023-12-18 01:11:01 🏷  Local node identity is: 12D3KooWHcDiChr6KkEfgCQtSiLbTzcDRhyKHyqtMvn515QWrWWt    
2023-12-18 01:11:01 🔍 Discovered new external address for our node: /ip4/142.215.53.19/tcp/30000/p2p/12D3KooWHcDiChr6KkEfgCQtSiLbTzcDRhyKHyqtMvn515QWrWWt    
2023-12-18 01:11:08 🚀 Using prepare-worker binary at: "/usr/lib/polkadot/polkadot-prepare-worker"    
2023-12-18 01:11:08 🚀 Using execute-worker binary at: "/usr/lib/polkadot/polkadot-execute-worker"    
2023-12-18 01:11:09 💻 Operating system: linux    
2023-12-18 01:11:09 💻 CPU architecture: x86_64    
2023-12-18 01:11:09 💻 Target environment: gnu    
2023-12-18 01:11:09 💻 CPU: Intel Xeon Processor (Cascadelake)    
2023-12-18 01:11:09 💻 CPU cores: 16    
2023-12-18 01:11:09 💻 Memory: 32115MB    
2023-12-18 01:11:09 💻 Kernel: 5.4.0-89-generic    
2023-12-18 01:11:09 💻 Linux distribution: Ubuntu 22.04.3 LTS    
2023-12-18 01:11:09 💻 Virtual machine: yes    
2023-12-18 01:11:09 📦 Highest known block at #21034600    
2023-12-18 01:11:09 Running JSON-RPC server: addr=127.0.0.1:9900, allowed origins=["*"]    
2023-12-18 01:11:09 🏁 CPU score: 1.02 GiBs    
2023-12-18 01:11:09 🏁 Memory score: 4.26 GiBs    
2023-12-18 01:11:09 🏁 Disk score (seq. writes): 626.57 MiBs    
2023-12-18 01:11:09 🏁 Disk score (rand. writes): 327.96 MiBs    
2023-12-18 01:11:09 ⚠️  The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 4.26 GiBs), Seq Write(expected: 950.00 MiBs, found: 626.57 MiBs), Rnd Write(expected: 420.00 MiBs, found: 327.96 MiBs),  for role 'Authority' find out more at:
https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware    
2023-12-18 01:11:09 👶 Starting BABE Authorship worker    
2023-12-18 01:11:09 🚨 Your system cannot securely run a validator. 
Running validation of malicious PVF code has a higher risk of compromising this machine.
  - Cannot enable landlock, a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced
  - Cannot unshare user namespace and change root, which are Linux-specific kernel security features: not available: unshare user and mount namespaces: Operation not permitted (os error 1)
You can ignore this error with the `--insecure-validator-i-know-what-i-do` command line argument if you understand and accept the risks of running insecurely. With this flag, security features are enabled on a best-effort basis, but not mandatory. 
More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
2023-12-18 01:11:09 🥩 BEEFY gadget waiting for BEEFY pallet to become available...    
2023-12-18 01:11:09 subsystem exited with error subsystem="candidate-validation" err=FromOrigin { origin: "candidate-validation", source: Context("could not enable Secure Validator Mode; check logs") }
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="candidate-validation"
2023-12-18 01:11:09 subsystem finished unexpectedly subsystem=Ok(())
2023-12-18 01:11:09 Received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="pvf-checker"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="bitfield-signing"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="network-bridge-rx"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="candidate-backing"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="network-bridge-tx"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-recovery"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="runtime-api"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="collation-generation"
2023-12-18 01:11:09 received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="gossip-support"
2023-12-18 01:11:09 Conclude
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-store"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="provisioner"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="bitfield-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="dispute-distribution"
2023-12-18 01:11:09 received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="chain-selection"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="chain-api"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="statement-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="approval-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="collator-protocol"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="approval-voting"
2023-12-18 01:11:09 discovered: 12D3KooWPEovdRYLyAw8phHWXLmJncnRYM6euSvPiahthcYKdeLv /ip4/172.17.0.1/tcp/30333    
2023-12-18 01:11:09 discovered: 12D3KooWPEovdRYLyAw8phHWXLmJncnRYM6euSvPiahthcYKdeLv /ip4/142.215.53.19/tcp/30333    
2023-12-18 01:11:10 subsystem exited with error subsystem="prospective-parachains" err=FromOrigin { origin: "prospective-parachains", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
2023-12-18 01:11:10 Essential task `overseer` failed. Shutting down service.    
2023-12-18 01:11:10 subsystem exited with error subsystem="dispute-coordinator" err=FromOrigin { origin: "dispute-coordinator", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
Error: 
   0: Other: Essential task failed.

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                 ⋮ 1 frame hidden ⋮                               
   2: polkadot::main::h4cca9d3491727cb7
      at <unknown source file>:<unknown line>
   3: std::sys_common::backtrace::__rust_begin_short_backtrace::h94782a592969dae8
      at <unknown source file>:<unknown line>
   4: main<unknown>
      at <unknown source file>:<unknown line>
   5: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
   6: _start<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

parameters:
--chain=polkadot --base-path=/chain-data --rpc-cors=all --port=30333 --unsafe-rpc-external --node-key=<xxx> --rpc-methods=Unsafe --name=<name> --telemetry-url="wss://telemetry-backend.w3f.community/submit 1" --public-addr=/dns4/<xxx>/tcp/23739 --in-peers=100 --in-peers-light=0 --db-cache=512

[bkchr]

Please provide more logs and also the logs and not a screenshot.

[AlexZhenWang]

Please provide more logs and also the logs and not a screenshot.

Thanks for the response @bkchr.
Here are the logs

2023-12-18 01:10:33 ----------------------------    
2023-12-18 01:10:33 This chain is not in any way    
2023-12-18 01:10:33       endorsed by the           
2023-12-18 01:10:33      KUSAMA FOUNDATION          
2023-12-18 01:10:33 ----------------------------    
2023-12-18 01:10:33 Parity Polkadot    
2023-12-18 01:10:33 ✌️  version 1.5.0-a3dc2f15f23    
2023-12-18 01:10:33 ❤️  by Parity Technologies <admin@parity.io>, 2017-2023    
2023-12-18 01:10:33 📋 Chain specification: Kusama    
2023-12-18 01:10:33 🏷  Node name: OnfinalityV#1    
2023-12-18 01:10:33 👤 Role: AUTHORITY    
2023-12-18 01:10:33 💾 Database: RocksDb at /chain-data/chains/ksmcc3/db/full    
2023-12-18 01:11:01 🏷  Local node identity is: 12D3KooWHcDiChr6KkEfgCQtSiLbTzcDRhyKHyqtMvn515QWrWWt    
2023-12-18 01:11:01 🔍 Discovered new external address for our node: /ip4/142.215.53.19/tcp/30000/p2p/12D3KooWHcDiChr6KkEfgCQtSiLbTzcDRhyKHyqtMvn515QWrWWt    
2023-12-18 01:11:08 🚀 Using prepare-worker binary at: "/usr/lib/polkadot/polkadot-prepare-worker"    
2023-12-18 01:11:08 🚀 Using execute-worker binary at: "/usr/lib/polkadot/polkadot-execute-worker"    
2023-12-18 01:11:09 💻 Operating system: linux    
2023-12-18 01:11:09 💻 CPU architecture: x86_64    
2023-12-18 01:11:09 💻 Target environment: gnu    
2023-12-18 01:11:09 💻 CPU: Intel Xeon Processor (Cascadelake)    
2023-12-18 01:11:09 💻 CPU cores: 16    
2023-12-18 01:11:09 💻 Memory: 32115MB    
2023-12-18 01:11:09 💻 Kernel: 5.4.0-89-generic    
2023-12-18 01:11:09 💻 Linux distribution: Ubuntu 22.04.3 LTS    
2023-12-18 01:11:09 💻 Virtual machine: yes    
2023-12-18 01:11:09 📦 Highest known block at #21034600    
2023-12-18 01:11:09 Running JSON-RPC server: addr=127.0.0.1:9900, allowed origins=["*"]    
2023-12-18 01:11:09 🏁 CPU score: 1.02 GiBs    
2023-12-18 01:11:09 🏁 Memory score: 4.26 GiBs    
2023-12-18 01:11:09 🏁 Disk score (seq. writes): 626.57 MiBs    
2023-12-18 01:11:09 🏁 Disk score (rand. writes): 327.96 MiBs    
2023-12-18 01:11:09 ⚠️  The hardware does not meet the minimal requirements Failed checks: Copy(expected: 11.49 GiBs, found: 4.26 GiBs), Seq Write(expected: 950.00 MiBs, found: 626.57 MiBs), Rnd Write(expected: 420.00 MiBs, found: 327.96 MiBs),  for role 'Authority' find out more at:
https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware    
2023-12-18 01:11:09 👶 Starting BABE Authorship worker    
2023-12-18 01:11:09 🚨 Your system cannot securely run a validator. 
Running validation of malicious PVF code has a higher risk of compromising this machine.
  - Cannot enable landlock, a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced
  - Cannot unshare user namespace and change root, which are Linux-specific kernel security features: not available: unshare user and mount namespaces: Operation not permitted (os error 1)
You can ignore this error with the `--insecure-validator-i-know-what-i-do` command line argument if you understand and accept the risks of running insecurely. With this flag, security features are enabled on a best-effort basis, but not mandatory. 
More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
2023-12-18 01:11:09 🥩 BEEFY gadget waiting for BEEFY pallet to become available...    
2023-12-18 01:11:09 subsystem exited with error subsystem="candidate-validation" err=FromOrigin { origin: "candidate-validation", source: Context("could not enable Secure Validator Mode; check logs") }
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="candidate-validation"
2023-12-18 01:11:09 subsystem finished unexpectedly subsystem=Ok(())
2023-12-18 01:11:09 Received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="pvf-checker"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="bitfield-signing"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="network-bridge-rx"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="candidate-backing"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="network-bridge-tx"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-recovery"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="runtime-api"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="collation-generation"
2023-12-18 01:11:09 received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="gossip-support"
2023-12-18 01:11:09 Conclude
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-store"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="provisioner"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="bitfield-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="dispute-distribution"
2023-12-18 01:11:09 received `Conclude` signal, exiting
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="chain-selection"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="chain-api"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="statement-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="approval-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="availability-distribution"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="collator-protocol"
2023-12-18 01:11:09 Terminating due to subsystem exit subsystem="approval-voting"
2023-12-18 01:11:09 discovered: 12D3KooWPEovdRYLyAw8phHWXLmJncnRYM6euSvPiahthcYKdeLv /ip4/172.17.0.1/tcp/30333    
2023-12-18 01:11:09 discovered: 12D3KooWPEovdRYLyAw8phHWXLmJncnRYM6euSvPiahthcYKdeLv /ip4/142.215.53.19/tcp/30333    
2023-12-18 01:11:10 subsystem exited with error subsystem="prospective-parachains" err=FromOrigin { origin: "prospective-parachains", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
2023-12-18 01:11:10 Essential task `overseer` failed. Shutting down service.    
2023-12-18 01:11:10 subsystem exited with error subsystem="dispute-coordinator" err=FromOrigin { origin: "dispute-coordinator", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
Error: 
   0: Other: Essential task failed.

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                 ⋮ 1 frame hidden ⋮                               
   2: polkadot::main::h4cca9d3491727cb7
      at <unknown source file>:<unknown line>
   3: std::sys_common::backtrace::__rust_begin_short_backtrace::h94782a592969dae8
      at <unknown source file>:<unknown line>
   4: main<unknown>
      at <unknown source file>:<unknown line>
   5: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
   6: _start<unknown>
      at <unknown source file>:<unknown line>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

And here are the parameters that I used when running the validator
--chain=polkadot --base-path=/chain-data --rpc-cors=all --port=30333 --unsafe-rpc-external --node-key=<xxx> --rpc-methods=Unsafe --name=<name> --telemetry-url="wss://telemetry-backend.w3f.community/submit 1" --public-addr=/dns4/<xxx>/tcp/23739 --in-peers=100 --in-peers-light=0 --db-cache=512

[alexggh]

This is the root cause:

 - Cannot unshare user namespace and change root, which are Linux-specific kernel security features: not available: unshare user and mount namespaces: Operation not permitted (os error 1)
You can ignore this error with the `--insecure-validator-i-know-what-i-do` command line argument if you understand and accept the risks of running insecurely. With this flag, security features are enabled on a best-effort basis, but not mandatory. 
More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
2023-12-18 01:11:09 🥩 BEEFY gadget waiting for BEEFY pallet to become available...    
2023-12-18 01:11:09 subsystem exited with error subsystem="candidate-validation" err=FromOrigin { origin: "candidate-validation", source: Context("could not enable Secure Validator Mode; check logs") }

I think you are hiting the same problem as here: #2662

[mrcnski]

Yes, looks like some new security features couldn't be enabled. "Operation not permitted" is interesting. Can you share details of your setup, is there anything unusual about it? Is your database path on a mount or have any special restrictions?

By the way, upgrading to Linux 5.13+ would make this part of the error go away, and the other part (unshare) becomes optional:

  - Cannot enable landlock, a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced

If upgrading is not possible you can pass the CLI flag specified in the error.

[AlexZhenWang]

Yes, looks like some new security features couldn't be enabled. "Operation not permitted" is interesting. Can you share details of your setup, is there anything unusual about it? Is your database path on a mount or have any special restrictions?

By the way, upgrading to Linux 5.13+ would make this part of the error go away, and the other part (unshare) becomes optional:

  - Cannot enable landlock, a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced

If upgrading is not possible you can pass the CLI flag specified in the error.

Thanks for the reply! @mrcnski

I am running the node as a pod in a k8s cluster. And the database is in a PVC that is mounted on the pod.

The related settings for the pod:

...
spec:
  containers:
  - args:
    - --base-path=/chain-data
...
    volumeMounts:
    - mountPath: /chain-data
      name: chaindata-volume
...
  volumes:
  - name: chaindata-volume
    persistentVolumeClaim:
      claimName: pvc-0
 ...

[mrcnski]

I am completely unfamiliar with kubernetes, but I presume the node is running in a container. That is probably why certain operations are not allowed, and maybe it depends on the container settings. For example if there is a seccomp sandbox it could be blocking the syscall, but I think this can be turned off. What is your Linux kernel version?

[AlexZhenWang]

Thanks @mrcnski. Yeah, I think you are right. Here is my Linux kernel version

5.4.0-89-generic

[mrcnski]

Thanks @AlexZhenWang! If it's possible, you can upgrade to Linux 5.13+; you'll still get a warning due to running in a container, but it won't be a hard error. Otherwise you'll need to pass --insecure-validator-i-know-what-i-do. Appreciate the report, it's important for us to know what is supported by actual configurations in production usage.

[bkchr]

#2486 (comment) posting this here as I think it belongs here.

CC @s0me0ne-unkn0wn @matthewmarcus

[maksimryndin]

#2486 (comment) posting this here as I think it belongs here.

CC @s0me0ne-unkn0wn @matthewmarcus

@matthewmarcus Hi! Could you please provide an output/answer for the following commands/questions

1. uname -a
2. more detailed logs output from the console if possible (as a text like this Got Essential task overseer failed error after upgrading Kusama and Polkadot validator to v1.5.0 #2728 (comment))
3. in the terminal on the machine

      echo "      #include <fcntl.h>
      #include <linux/landlock.h>
      #include <sys/syscall.h>
      #include <unistd.h>

      int main (int argc, char *argv[]) {
        /* Get supported landlock ABI */
        int abi = syscall (SYS_landlock_create_ruleset, NULL, 0, LANDLOCK_CREATE_RULESET_VERSION);
        return abi < 0 ? 1 : 0;
      }
     " > landlock_test.c

clang landlock_test.c -o landlock_test

./landlock_test

echo $? <--- what is an output? is it 0?

Thank you!