sql/importer: TestImportWorkerFailure failed

[cockroach-teamcity]

sql/importer.TestImportWorkerFailure failed with artifacts on release-23.1 @ 4226a83871bbce776bc9389fca5cf084b4bb7632:

=== RUN   TestImportWorkerFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportWorkerFailure3379689604
    test_log_scope.go:79: use -show-logs to present logs inline
    import_stmt_test.go:5430: pq: updating job details after publishing schemas: job 862797201834016769: select-job: grpc: connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)" [code 14/Unavailable]
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/73dfce30b9a5630b1b4dabed3c94b32c/logTestImportWorkerFailure3379689604
--- FAIL: TestImportWorkerFailure (42.46s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-27679

[cucaroach]

The full failure looks like this:

I230506 12:13:30.736962 231027 jobs/registry.go:1582 ⋮ [T1,n1] 518  IMPORT job 862797201834016769: stepping through state reverting with error: updating job details after publishing schemas: job 862797201834016769: ‹select-job›: grpc: ‹connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)"› [code 14/Unavailable]
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +(1) attached stack trace
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  -- stack trace:
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).publishSchemas.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:1105
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3665
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1594
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1682
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:965
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:928
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:964
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:927
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:902
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1670
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1592
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3664
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).publishSchemas
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:1076
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).Resume
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:331
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1624
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1625
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:474
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:978
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +Wraps: (2) updating job details after publishing schemas
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +Wraps: (3) attached stack trace
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  -- stack trace:
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.Updater.update.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/update.go:83
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | [...repeated from below...]
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +Wraps: (4) job 862797201834016769
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +Wraps: (5) attached stack trace
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  -- stack trace:
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:971
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:418
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:469
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:583
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:635
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:621
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.Updater.update
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/update.go:102
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.Updater.Update
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/update.go:378
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.Updater.SetDetails
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:668
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).publishSchemas.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:1103
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3665
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1594
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1682
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn.func1
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:965
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:928
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:964
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:927
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:902
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1670
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1592
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3664
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).publishSchemas
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:1076
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*importResumer).Resume
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/sql/importer/import_job.go:331
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1624
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1625
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:474
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:978
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | runtime.goexit
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +  | 	src/runtime/asm_amd64.s:1594
I230506 12:13:30.736962 231027 jobs/registry.go:1582 ⋮ [T1,n1] 518 +Wraps: (6) ‹select-job›
I230506 12:13:30.736962 231027 jobs/registry.go:1582 ⋮ [T1,n1] 518 +Wraps: (7) grpc: ‹connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)"› [code 14/Unavailable]
I230506 12:13:30.736962 231027 jobs/registry.go:1582 â‹® [T1,n1] 518 +Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *status.Error

[cucaroach]

I wasn't able to repro this although the failure has happened twice. I wonder if someone with strong jobs mojo could take a look?

[cucaroach]

Using stress after 2885 runs I got:

test logs left over in: /tmp/_tmp/060246e25d27a0efc6dd14ec672a0f2b/logTestImportWorkerFailure3155831314
--- FAIL: TestImportWorkerFailure (17.54s)
    test_log_scope.go:161: test logs captured to: /tmp/_tmp/060246e25d27a0efc6dd14ec672a0f2b/logTestImportWorkerFailure3155831314
    test_log_scope.go:79: use -show-logs to present logs inline
    import_stmt_test.go:5435: query 'SELECT * FROM t ORDER BY i': expected:
        0
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        
        got:
        1
        2
        4
        5
        7
        8
        10
        11
        13
        14
        16
        17
        19
        
    panic.go:522: -- test log scope end --
FAIL
I230509 15:29:22.694464 1 (gostd) testmain.go:254  [-] 1  Test //pkg/sql/importer:importer_test exited with error code 1

[rytaft]

Assigning @cucaroach to skip this test (and mark this issue with skipped-test label), can put it on Bugs to Fix after if the fix is not obvious.

[dt]

on the first one that complained about transport closing, I suspect that is this SELECT query here: https://github.com/cockroachdb/cockroach/blob/439c515b2a0058648731da73993c409544404da1/pkg/jobs/update.go

My guess is that this SELECT is being distributed? and thus when a node (other than the gateway) is terminated, the execution of the distributed query fails, but this code doesn't retry it and assumes any error is a failure of the job and so the job fails?

I have no idea at all about that second one -- that one looks just... wrong... right?

[cucaroach]

on the first one that complained about transport closing, I suspect that is this SELECT query here: 439c515/pkg/jobs/update.go

My guess is that this SELECT is being distributed? and thus when a node (other than the gateway) is terminated, the execution of the distributed query fails, but this code doesn't retry it and assumes any error is a failure of the job and so the job fails?

Thanks for looking! I believe the internal executor will always run with DistSQLMode of 0 (ie off) but maybe the distsender GRPC failed? Its a little surprising a raw GRPC error made it all the way to the results iterator w/o getting wrapped/adorned with more info, if I could repro it I could try to fix that.