"We want to know if htlc 1 is missing (immediate)"

[fabianfabian]

lightingd keeps crashing, this is in the log:

2018-02-04T23:36:23.636Z lightningd(4284): Adding block 507509: 0000000000000000003b0c2cc3034ea8d6c71dd77d820df8b24cf6e668ac8892
**2018-02-04T23:36:24.989Z lightningd(4284): Owning output 1 300000 (P2SH) txid 20d8e529408a281f43284e216b182e23aaabbe64c7167b9b384065401ed2281a**
2018-02-04T23:36:25.186Z lightningd(4284): Adding block 507510: 0000000000000000005fdf68e6a860c11cad255157504bde4ca090ac910b59b7
2018-02-04T23:36:25.490Z lightningd(4284): Adding block 507511: 00000000000000000028307274c3176ffe9cfe418ac47f27e515b2b19c51d74e
2018-02-04T23:36:25.828Z lightningd(4284): Adding block 507512: 0000000000000000001406f51c9785fb2b07f879f8279da383eba4044c021ba3
**2018-02-04T23:36:27.153Z lightningd(4284): Owning output 0 135980 (SEGWIT) txid cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81**
2018-02-04T23:36:27.295Z lightningd(4284): Adding block 507513: 00000000000000000044dac62e6b26827f7f717eda4dfdd32a740d9022b60af5
2018-02-04T23:36:28.811Z lightningd(4284): Adding block 507514: 0000000000000000003d8ccb167602660c5c2f6aca14344f7bc85ca21aa98e07
2018-02-04T23:36:30.307Z lightningd(4284): Adding block 507515: 0000000000000000004e25bb406c70ac566a73bd73af77b517e4cf7f53f2b365
2018-02-04T23:36:30.643Z lightningd(4284): Adding block 507516: 00000000000000000054cfd968c7b6d986f594b24a90c726c44db1c5f1e50c36
2018-02-04T23:36:30.941Z lightningd(4284): Adding block 507517: 0000000000000000005159fc290104ead48aebef3d49a53d9a05b4167cd8072f
2018-02-04T23:36:31.610Z lightningd(4284): Adding block 507518: 000000000000000000505633dbbc2050b349babf3fb22e79b54a4c635c96d24d
2018-02-04T23:36:32.701Z lightningd(4284): Adding block 507519: 000000000000000000178af595809bba6cdb8af5e599884dccfa14e93ccb783b
2018-02-04T23:36:33.365Z lightningd(4284): Adding block 507520: 000000000000000000293baaad7ae3594291e75e63affc8db928344e19848d21
2018-02-04T23:36:33.511Z lightningd(4284): Adding block 507521: 00000000000000000030a6806d06bb707e38f81f94ac2d4c5911c1c8fe71597a
**2018-02-04T23:36:33.712Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: Got UTXO spend for cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81:1: 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2
2018-02-04T23:36:33.712Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: Peer permanent failure in ONCHAIND_OUR_UNILATERAL: Funding transaction spent
2018-02-04T23:36:33.712Z lightningd(4284): 	 (tx 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2)
2018-02-04T23:36:33.712Z lightningd(4284): sendrawtransaction: 02000000000101815c859c45d4b4c9a22f5a892bc0f93e9b008ddd5a9519ffbd73973dd2583bcf01000000007d80ba800249d60000000000001600147dbd4634afc993b70fb5839db0047b84db9cd359c56b0100000000002200200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff219410400473044022048d58c831b5c1735ab2198bb5c91855966d5fcde07e491bcef3fb8190be5775c022075c89477b94a689bc4b12d1464462bfad3105011a58640bee05c24a1e9576ac601483045022100f4f75da84118a80bfe1ddfa813812a84e0dd25b8c8f414a8e4acfdf61f1ee13d02204cc4dc75c7e183e277d9837fbd1be9ff0dc3eff3ba5b531f01a325f0dde7b4c201475221027068cdeb935512faf54477427efe112e081df250509e4a356f164f5b072e673f2103058b89c005ddc30cf995da1e5eac9a360fd915db50d41501e2c0ff45c67e377252ae90382b20
2018-02-04T23:36:33.713Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from ONCHAIND_OUR_UNILATERAL to FUNDING_SPEND_SEEN
2018-02-04T23:36:33.714Z lightningd(4284): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): pid 4529, msgfd 17
2018-02-04T23:36:33.714Z lightningd(4284): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: We want to know if htlc 1 is missing (immediate)**

I'm not sure when it first crashed, all I did was fund a channel with blockstream store and bought a sticker (successfully), then tried to make donation to https://lnmainnet.gaben.win/?100= which failed I think, then the next day I check the logs and its in this infinite crash loop.

getinfo output

$ lightning-cli getinfo
{ "id" : "03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6", "port" : 9735, "address" : 
	[ 
		{ "type" : "ipv4", "address" : "X.X.X.X", "port" : 9735 } ], "version" : "v0.5.2-2016-11-21-1814-g5f2cd7c1", "blockheight" : 507439, "network" : "bitcoin" }

listpeers output

$ lightning-cli listpeers
{ "peers" : 
	[ 
		{ "id" : "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432", "connected" : false, "channels" : 
			[ 
				{ "state" : "ONCHAIND_OUR_UNILATERAL", "owner" : "lightning_onchaind", "short_channel_id" : "507512:2585:1", "funding_txid" : "cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81", "msatoshi_to_us" : 95142333, "msatoshi_total" : 150000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "to_self_delay" : 144, "max_accepted_htlcs" : 483 } ] } ] }

listpayments output

$ lightning-cli listpayments
{ "payments" : 
	[ 
		{ "id" : 1, "payment_hash" : "ba8e5352952ab59468d543334d9b2d765cf76be61c38d5644695fcbe48cb3126", "destination" : "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432", "msatoshi" : 54857667, "timestamp" : 1517713156, "created_at" : 1517713156, "status" : "complete", "payment_preimage" : "855dcf08dadd147f4b45745c40e59ea92a3324e8e31c7995f8930f099c0b50d5" }, 
		{ "id" : 2, "payment_hash" : "abb88bee3d98b0615e7856418cf3bf9be5ac4fda1cef678cc9f5097fe0a957a6", "destination" : "035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8", "msatoshi" : 500000, "timestamp" : 1517713524, "created_at" : 1517713524, "status" : "failed" } ] }

[fabianfabian]

In the console there is some more info that is not in the log file:

2018-02-05T00:46:38.450Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from ONCHAIND_OUR_UNILATERAL to FUNDING_SPEND_SEEN
2018-02-05T00:46:38.452Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): pid 8396, msgfd 16
2018-02-05T00:46:38.453Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: We want to know if htlc 1 is missing (immediate)
**2018-02-05T00:46:38.985Z lightningd(8149): sendrawtx exit 27, gave error code: -27
error message:
transaction already in block chain**

**2018-02-05T00:46:38.986Z lightning_gossipd(8157): TRACE: req: type WIRE_GOSSIP_DISABLE_CHANNEL len 12
2018-02-05T00:46:38.986Z lightning_gossipd(8157): TRACE: Unable to find channel 507512:2585:1/0
2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 1 of cf3b58d23d9773bdff19955add8d009b3ef9c02b895a2fa2c9b4d4459c855c81: FUNDING_TRANSACTION/FUNDING_OUTPUT
2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Remote per-commit point: 024d247a0d3fecaac73c175767e50d02bc01a77f51160abb9f09bda967e629ab10
2018-02-05T00:46:38.987Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Old remote per-commit point: 02d2c1e111a835f4d9732f240ced13b6366f5d5e0415bee80cb4790cf81c61bba0
2018-02-05T00:46:38.988Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: commitnum = 6, revocations_received = 6
2018-02-05T00:46:38.988Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): UPDATE WIRE_ONCHAIN_INIT_REPLY
2018-02-05T00:46:38.988Z lightningd(8149): peer 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432: State changed from FUNDING_SPEND_SEEN to ONCHAIND_OUR_UNILATERAL
2018-02-05T00:46:38.992Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Initial feerate 0 to 4167**
**2018-02-05T00:46:38.992Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Resolved FUNDING_TRANSACTION/FUNDING_OUTPUT by OUR_UNILATERAL (7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2)
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Deconstructing unilateral tx: 6 using keyset:  self_revocation_key: 02043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d self_delayed_payment_key: 029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d22 self_payment_key: 0346b6c9f949bcde1b70cdf6c95ecd8ce4546499f6bf5f3e9125d11549aff36ded other_payment_key: 03a32dc47746f8a4238454e5743ac45b53d240e2374a5e99b70b0f82a37a5e6d3a self_htlc_key: 0347ded51baed2e51c8f5034865a268c10b8da9f75f4a64e848bfcb55d5ae04773 other_htlc_key: 025ace4ca690b2f379396b33491367323f3e4931dbebfc68f29ad5d040fa345be6
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Script to-me: 144: 00200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff21941 (632102043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d67029000b27521029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d2268ac)
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Script to-them: 00147dbd4634afc993b70fb5839db0047b84db9cd359
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Output 0: 00147dbd4634afc993b70fb5839db0047b84db9cd359
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Output 1: 00200686d8b848eea5268a7886fbcab54518506f7c181c65ddfd54664c457ff21941
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 0 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/OUTPUT_TO_THEM
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Ignoring output 0 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/OUTPUT_TO_THEM
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Tracking output 1 of 7a149b2496d57e0597e9f90769574a2e92fb11c97909b1dd37444039631686b2: OUR_UNILATERAL/DELAYED_OUTPUT_TO_US
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): TRACE: Propose handling OUR_UNILATERAL/DELAYED_OUTPUT_TO_US by OUR_DELAYED_RETURN_TO_WALLET (02000000000101b286166339404437ddb10979c911fb922e4a576907f9e997057ed596249b147a01000000009000000001786a010000000000160014f467433419bf5f7ef177c5c52ffc63255eab399a03483045022100927bcbc742c3c15d9e4b624dc022389fb4ca2f0880f3bb1f13a96052bfec1f5c02201157f96c697c8d55a15c3f9b6be773e7679516609a3c007bc4da082875d4854f01004d632102043d09322c22b06803f15dfc92dd58692920d9b1ef1de38cb399349066f2892d67029000b27521029d95106bfd470cd571842dc63c1c0f05b58dbc7105eb48df175cdef53d8b9d2268ac00000000) in 144 blocks
2018-02-05T00:46:38.993Z lightningd(8149): lightning_onchaind(02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432): UPDATE WIRE_ONCHAIN_MISSING_HTLC_OUTPUT
2018-02-05T00:46:38.993Z lightningd(8149): Reporting route failure to gossipd: 0x4008 (WIRE_PERMANENT_CHANNEL_FAILURE) node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6 channel 507512:2585:1 update** 
**2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: req: type WIRE_GOSSIP_ROUTING_FAILURE len 47
2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: Received routing failure 0x4008 (WIRE_PERMANENT_CHANNEL_FAILURE), erring node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6, channel 507512:2585:1
2018-02-05T00:46:38.997Z lightning_gossipd(8157): TRACE: UNUSUAL routing_failure: Erring node 03de291f2e3e14cd3875b79bc8948f42e582ce7feda246314dfd282548dd1964a6 not in map**
2018-02-05T00:46:39.052Z lightningd(8149): Adding block 507522: 00000000000000000040ed0f2c2d686b75fb486c21f58e7cea4b91d9ab7b39e4
2018-02-05T00:46:39.405Z lightningd(8149): Adding block 507523: 00000000000000000044aae802ae8175223ce3fe9d07d9d9869d73dd98edadb5
2018-02-05T00:46:40.175Z lightningd(8149): Adding block 507524: 0000000000000000003a1f24cf071a242ec548f3b37e541cd1d1b177edaeadba
2018-02-05T00:46:40.723Z lightningd(8149): Adding block 507525: 000000000000000000464da0669b510d81e0465cf227462b3833c216b747a4e5

[rustyrussell]

There's no log of the crash? Nothing in here looks particularly bad, it's just following the chain.

You might find a crash.log file in your ~/.lightning directory?

[fabianfabian]

I think this may be a problem with my (custom) systemd service file, when starting lightningd as a service it restarts right after the "We want to know if htlc 1 is missing (immediate)**" message. But when I start lightningd manually it keeps running fine.

I couldn't find something like https://github.com/bitcoin/bitcoin/tree/master/contrib/init for c-lightning.

This is mine now:

root@bunker:~# cat /etc/systemd/system/lightningd.service 
[Unit]
Description=Lightning daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/lightningd --network bitcoin --log-level debug --log-file debug.log
RuntimeDirectory=lightningd
User=root
Type=forking
PrivateTmp=true

[Install]
WantedBy=multi-user.target

[plachta11b]

I create my own working by using Type=simple.

[Unit]
Description=Lightning Network implementation in C
After=network.target

[Service]
User=myUser
Group=myUser

Type=simple

ExecStart=/path/to/lightningd \
--bitcoin-datadir=/path/to/.bitcoin \
--network=bitcoin \
--alias=myCoolAlias \
--ipaddr X.X.X.X \
--log-file=/path/to/.lightning/log_file \
--log-level=debug

Restart=on-failure
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Look for "Type=" in https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#types-of-units for more informations

Is it safe to run lightningd with root user?

[fabianfabian]

@plachta11b I will try your configuration

Is it safe to run lightningd with root user?

Probably not, for me its just a testing machine.

[fabianfabian]

@plachta11b your config works!