Index Task crashes in middle manager when ingesting JSON keys that contain a comma

[winsmith]

Affected Version

28.0.1

Description

I encountered an error during compaction and ingestion where tasks in middle manager would crash randomly when ingesting a JSON line where the key contains a comma (,). We call these "thorny" data because tasks that touch them crash.

Here's a row that crashes on ingest:

{"receivedAt": "2024-01-20","A,B": "C"}

Here's an ingestion spec that triggers the error:

{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "inline",
        "data": "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
      },
      "inputFormat": {
        "type": "json"
      }
    },
    "dataSchema": {
      "dataSource": "telemetry-signals-thorny-test",
      "timestampSpec": {
        "column": "receivedAt",
        "format": "iso"
      },
      "dimensionsSpec": {
        "dimensionExclusions": [
          "__time",
          "count",
          "receivedAt"
        ]
      }
    }
}

This results in task status failed:

{
  "id": "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "groupId": "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "type": "index_parallel",
  "createdTime": "2024-01-22T14:37:20.082Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": 5685,
  "location": {
    "host": "10.244.7.175",
    "port": 8102,
    "tlsPort": -1
  },
  "dataSource": "telemetry-signals-thorny-test",
  "errorMsg": "java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org..."
}

And here's the complete index log in the middle manager:

bash-5.1$ cat index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14\:37\:20.080Z.log 
2024-01-22T14:37:21,125 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final
2024-01-22T14:37:21,839 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar, druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar, snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:21,843 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-azure-extensions], jars: azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar, druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:21,844 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar, postgresql-42.6.0.jar
2024-01-22T14:37:21,846 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar, commons-math3-3.6.1.jar
2024-01-22T14:37:21,848 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-stats], jars: commons-math3-3.6.1.jar, druid-stats-28.0.1.jar
2024-01-22T14:37:22,116 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar, druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar, snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:22,119 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-azure-extensions], jars: azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar, druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:22,122 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar, postgresql-42.6.0.jar
2024-01-22T14:37:22,123 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar, commons-math3-3.6.1.jar
2024-01-22T14:37:22,132 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-stats], jars: commons-math3-3.6.1.jar, druid-stats-28.0.1.jar
2024-01-22T14:37:23,384 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=28.0.1}].
2024-01-22T14:37:23,714 WARN [main] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:23,747 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 3 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.curator.DruidConnectionStateListener, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2024-01-22T14:37:23,763 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors [7], memory [2,147,483,648], maxMemory [3,221,225,472], directMemory [17,179,869,184]. Properties follow.
2024-01-22T14:37:23,779 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS] 
2024-01-22T14:37:23,982 INFO [main] org.apache.druid.query.DruidProcessingConfig - Detected max direct memory size of [17,179,869,184] bytes
2024-01-22T14:37:24,000 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,050 INFO [main] org.apache.druid.guice.StorageNodeModule - Segment cache not configured on ServerType [indexer-executor]. It will not be assignable for segment placement
2024-01-22T14:37:24,108 INFO [main] org.eclipse.jetty.util.log - Logging initialized @4005ms to org.eclipse.jetty.util.log.Slf4jLog
2024-01-22T14:37:24,121 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8102]
2024-01-22T14:37:24,250 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23
2024-01-22T14:37:24,256 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[7,000,000]
2024-01-22T14:37:24,262 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[7,000,000]
2024-01-22T14:37:24,266 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[7,000,000]
2024-01-22T14:37:24,270 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[7,000,000]
2024-01-22T14:37:24,333 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [INIT]
2024-01-22T14:37:24,334 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [NORMAL]
2024-01-22T14:37:24,338 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.10-a32c7183d42325b03e44a06aade6a0f16955bf13, built on 05/29/2022 16:59 GMT
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid-druid-cluster-middlemanagers-1.druid-druid-cluster-middlemanagers.druid-main.svc.cluster.local
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.9
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/commons-io-2.11.0.jar:lib/jboss-logging-3.4.1.Final.jar:lib/aws-java-sdk-core-1.12.497.jar:lib/derbynet-10.14.2.0.jar:lib/error_prone_annotations-2.20.0.jar:lib/jetty-continuation-9.4.53.v20231009.jar:lib/antlr4-runtime-4.5.3.jar:lib/guice-assistedinject-4.1.0.jar:lib/druid-indexing-hadoop-28.0.1.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jetty-security-9.4.53.v20231009.jar:lib/jna-5.13.0.jar:lib/calcite-core-1.35.0.jar:lib/asm-commons-9.3.jar:lib/commons-compress-1.23.0.jar:lib/jakarta.activation-api-1.2.1.jar:lib/commons-codec-1.13.jar:lib/commons-dbcp2-2.0.1.jar:lib/jna-platform-5.13.0.jar:lib/druid-gcp-common-28.0.1.jar:lib/druid-indexing-service-28.0.1.jar:lib/google-oauth-client-1.34.1.jar:lib/resilience4j-core-1.3.1.jar:lib/config-magic-0.9.jar:lib/maven-repository-metadata-3.1.1.jar:lib/httpcore-4.4.11.jar:lib/proj4j-1.2.2.jar:lib/aws-java-sdk-s3-1.12.497.jar:lib/xz-1.9.jar:lib/httpclient5-5.1.3.jar:lib/jaxb-api-2.3.1.jar:lib/jetty-util-ajax-9.4.53.v20231009.jar:lib/commons-compiler-3.1.9.jar:lib/druid-processing-28.0.1.jar:lib/jmespath-java-1.12.497.jar:lib/curator-framework-5.5.0.jar:lib/rhino-1.7.14.jar:lib/lz4-java-1.8.0.jar:lib/jersey-guice-1.19.4.jar:lib/aopalliance-1.0.jar:lib/google-http-client-jackson2-1.42.3.jar:lib/netty-codec-4.1.100.Final.jar:lib/gson-2.10.jar:lib/web-console-28.0.1.jar:lib/aws-java-sdk-kms-1.12.497.jar:lib/jetty-server-9.4.53.v20231009.jar:lib/commons-pool2-2.2.jar:lib/jackson-annotations-2.12.7.jar:lib/plexus-utils-3.0.24.jar:lib/elasticache-java-cluster-client-1.2.0.jar:lib/avatica-metrics-1.23.0.jar:lib/jetty-http-9.4.53.v20231009.jar:lib/aether-util-0.9.0.M2.jar:lib/airline-io-2.8.4.jar:lib/fastutil-core-8.5.4.jar:lib/maven-settings-3.1.1.jar:lib/disruptor-3.3.6.jar:lib/metrics-core-4.2.19.jar:lib/avatica-server-1.23.0.jar:lib/maven-settings-builder-3.1.1.jar:lib/fastutil-8.5.4.jar:lib/netty-codec-http-4.1.100.Final.jar:lib/jcl-over-slf4j-1.7.36.jar:lib/calcite-linq4j-1.35.0.jar:lib/guava-31.1-jre.jar:lib/jetty-util-9.4.53.v20231009.jar:lib/maven-artifact-3.6.0.jar:lib/grpc-context-1.27.2.jar:lib/sigar-1.6.5.132.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/json-simple-1.1.1.jar:lib/hibernate-validator-6.2.5.Final.jar:lib/jetty-io-9.4.53.v20231009.jar:lib/ion-java-1.0.2.jar:lib/jackson-jq-0.0.10.jar:lib/jersey-core-1.19.4.jar:lib/aether-spi-0.9.0.M2.jar:lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/audience-annotations-0.5.0.jar:lib/httpclient-4.5.13.jar:lib/aggdesigner-algorithm-6.0.jar:lib/javax.activation-api-1.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/log4j-1.2-api-2.18.0.jar:lib/netty-transport-4.1.100.Final.jar:lib/log4j-api-2.18.0.jar:lib/jetty-client-9.4.53.v20231009.jar:lib/avatica-core-1.23.0.jar:lib/guice-4.1.0.jar:lib/jersey-server-1.19.4.jar:lib/druid-services-28.0.1.jar:lib/aws-java-sdk-ec2-1.12.497.jar:lib/oshi-core-6.4.4.jar:lib/netty-handler-4.1.100.Final.jar:lib/jackson-datatype-guava-2.12.7.jar:lib/joni-2.1.27.jar:lib/jackson-module-jaxb-annotations-2.12.7.jar:lib/javax.servlet-api-3.1.0.jar:lib/opencsv-4.6.jar:lib/derby-10.14.2.0.jar:lib/google-http-client-apache-v2-1.42.3.jar:lib/vavr-match-0.10.2.jar:lib/asm-9.3.jar:lib/zstd-jni-1.5.2-3.jar:lib/asm-tree-9.3.jar:lib/jackson-jaxrs-smile-provider-2.12.7.jar:lib/jackson-jaxrs-base-2.12.7.jar:lib/log4j-core-2.18.0.jar:lib/vavr-0.10.2.jar:lib/ipaddress-5.3.4.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/checker-qual-2.5.7.jar:lib/jetty-servlet-9.4.53.v20231009.jar:lib/jts-io-common-1.19.0.jar:lib/datasketches-java-4.2.0.jar:lib/druid-server-28.0.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/asm-analysis-9.3.jar:lib/jakarta.validation-api-2.0.2.jar:lib/datasketches-memory-2.2.0.jar:lib/joda-time-2.12.5.jar:lib/protobuf-java-3.24.0.jar:lib/accessors-smart-2.4.11.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/netty-resolver-dns-4.1.100.Final.jar:lib/jetty-rewrite-9.4.53.v20231009.jar:lib/httpcore5-5.1.3.jar:lib/janino-3.1.9.jar:lib/jackson-databind-2.12.7.jar:lib/google-api-client-2.2.0.jar:lib/curator-client-5.5.0.jar:lib/netty-common-4.1.100.Final.jar:lib/log4j-slf4j-impl-2.18.0.jar:lib/caffeine-2.8.0.jar:lib/stax-ex-1.8.jar:lib/airline-2.8.4.jar:lib/okhttp-1.0.2.jar:lib/jakarta.el-3.0.4.jar:lib/jts-core-1.19.0.jar:lib/compress-lzf-1.0.4.jar:lib/txw2-2.3.1.jar:lib/httpcore5-h2-5.1.3.jar:lib/FastInfoset-1.2.15.jar:lib/curator-recipes-5.5.0.jar:lib/netty-buffer-4.1.100.Final.jar:lib/jackson-dataformat-smile-2.12.7.jar:lib/apiguardian-api-1.1.2.jar:lib/jetty-servlets-9.4.53.v20231009.jar:lib/netty-codec-dns-4.1.100.Final.jar:lib/jackson-datatype-joda-2.12.7.jar:lib/maven-model-3.1.1.jar:lib/javax.activation-1.2.0.jar:lib/google-http-client-gson-1.42.3.jar:lib/slf4j-api-1.7.36.jar:lib/uzaygezen-core-0.2.jar:lib/jdbi-2.63.1.jar:lib/classmate-1.5.1.jar:lib/netty-codec-socks-4.1.100.Final.jar:lib/jetty-proxy-9.4.53.v20231009.jar:lib/async-http-client-2.5.3.jar:lib/aether-api-0.9.0.M2.jar:lib/jackson-jaxrs-json-provider-2.12.7.jar:lib/javax.el-api-3.0.0.jar:lib/tesla-aether-0.0.5.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/jsr311-api-1.1.1.jar:lib/guice-multibindings-4.1.0.jar:lib/netty-reactive-streams-2.0.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/zookeeper-3.5.10.jar:lib/netty-3.10.6.Final.jar:lib/json-path-2.3.0.jar:lib/netty-resolver-4.1.100.Final.jar:lib/jcodings-1.0.43.jar:lib/commons-collections4-4.2.jar:lib/jersey-servlet-1.19.4.jar:lib/commons-lang-2.6.jar:lib/RoaringBitmap-0.9.49.jar:lib/guice-servlet-4.1.0.jar:lib/opencensus-api-0.31.1.jar:lib/commons-lang3-3.12.0.jar:lib/commons-math3-3.6.1.jar:lib/druid-sql-28.0.1.jar:lib/shims-0.9.49.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/druid-aws-common-28.0.1.jar:lib/netty-transport-native-unix-common-4.1.100.Final.jar:lib/aether-impl-0.9.0.M2.jar:lib/curator-x-discovery-5.5.0.jar:lib/cron-scheduler-0.1.jar:lib/jackson-dataformat-cbor-2.12.7.jar:lib/zookeeper-jute-3.5.10.jar:lib/aws-java-sdk-sts-1.12.497.jar:lib/json-smart-2.4.11.jar:lib/maven-aether-provider-3.1.1.jar:lib/jackson-module-guice-2.12.7.jar:lib/jackson-core-2.12.7.jar:lib/commons-logging-1.1.1.jar:lib/netty-handler-proxy-4.1.100.Final.jar:lib/maven-model-builder-3.1.1.jar:lib/reactive-streams-1.0.2.jar:lib/icu4j-73.2.jar:lib/google-http-client-1.42.3.jar:lib/wagon-provider-api-2.4.jar:lib/opencensus-contrib-http-util-0.31.1.jar:lib/jsr305-2.0.1.jar:lib/failureaccess-1.0.1.jar:lib/log4j-jul-2.18.0.jar:lib/commons-collections-3.2.2.jar:lib/commons-beanutils-1.9.4.jar:lib/netty-transport-classes-epoll-4.1.100.Final.jar:lib/commons-text-1.10.0.jar:lib/plexus-interpolation-1.19.jar:lib/fastutil-extra-8.5.4.jar:lib/j2objc-annotations-1.3.jar:lib/javax.inject-1.jar:
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/druid/data
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.131.1-2.cm2
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1906MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=3072MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=2048MB
2024-01-22T14:37:24,349 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=tiny-cluster-zk-0.tiny-cluster-zk,tiny-cluster-zk-1.tiny-cluster-zk,tiny-cluster-zk-2.tiny-cluster-zk sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@731ab49b
2024-01-22T14:37:24,355 INFO [main] org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-01-22T14:37:24,359 INFO [main] org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2024-01-22T14:37:24,367 INFO [main] org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2024-01-22T14:37:24,374 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2024-01-22T14:37:24,380 INFO [main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181. Will not attempt to authenticate using SASL (unknown error)
2024-01-22T14:37:24,391 INFO [main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Socket error occurred: tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181: Connection refused
2024-01-22T14:37:24,415 WARN [main] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:24,455 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
  "type" : "index_parallel",
  "id" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "groupId" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "resource" : {
    "availabilityGroup" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
    "requiredCapacity" : 1
  },
  "spec" : {
    "dataSchema" : {
      "dataSource" : "telemetry-signals-thorny-test",
      "timestampSpec" : {
        "column" : "receivedAt",
        "format" : "iso",
        "missingValue" : null
      },
      "dimensionsSpec" : {
        "dimensions" : [ ],
        "dimensionExclusions" : [ "__time", "count", "receivedAt" ],
        "includeAllDimensions" : false,
        "useSchemaDiscovery" : false
      },
      "metricsSpec" : [ ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "DAY",
        "queryGranularity" : {
          "type" : "none"
        },
        "rollup" : true,
        "intervals" : [ ]
      },
      "transformSpec" : {
        "filter" : null,
        "transforms" : [ ]
      }
    },
    "ioConfig" : {
      "type" : "index_parallel",
      "inputSource" : {
        "type" : "inline",
        "data" : "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
      },
      "inputFormat" : {
        "type" : "json",
        "keepNullColumns" : false,
        "assumeNewlineDelimited" : false,
        "useJsonNodeReader" : false
      },
      "appendToExisting" : false,
      "dropExisting" : false
    },
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxRowsPerSegment" : null,
      "appendableIndexSpec" : {
        "type" : "onheap",
        "preserveExistingMetrics" : false
      },
      "maxRowsInMemory" : 1000000,
      "maxBytesInMemory" : 0,
      "skipBytesInMemoryOverheadCheck" : false,
      "maxTotalRows" : null,
      "numShards" : null,
      "splitHintSpec" : null,
      "partitionsSpec" : null,
      "indexSpec" : {
        "bitmap" : {
          "type" : "roaring"
        },
        "dimensionCompression" : "lz4",
        "stringDictionaryEncoding" : {
          "type" : "utf8"
        },
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "indexSpecForIntermediatePersists" : {
        "bitmap" : {
          "type" : "roaring"
        },
        "dimensionCompression" : "lz4",
        "stringDictionaryEncoding" : {
          "type" : "utf8"
        },
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "maxPendingPersists" : 0,
      "forceGuaranteedRollup" : false,
      "reportParseExceptions" : false,
      "pushTimeout" : 0,
      "segmentWriteOutMediumFactory" : null,
      "maxNumConcurrentSubTasks" : 1,
      "maxRetry" : 3,
      "taskStatusCheckPeriodMs" : 1000,
      "chatHandlerTimeout" : "PT10S",
      "chatHandlerNumRetries" : 5,
      "maxNumSegmentsToMerge" : 100,
      "totalNumMergeTasks" : 10,
      "logParseExceptions" : false,
      "maxParseExceptions" : 2147483647,
      "maxSavedParseExceptions" : 0,
      "maxColumnsToMerge" : -1,
      "awaitSegmentAvailabilityTimeoutMillis" : 0,
      "maxAllowedLockCount" : -1,
      "partitionDimensions" : [ ]
    }
  },
  "context" : {
    "forceTimeChunkLock" : true,
    "useLineageBasedSegmentAllocation" : true
  },
  "dataSource" : "telemetry-signals-thorny-test"
}
2024-01-22T14:37:24,457 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock].
2024-01-22T14:37:24,462 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock] in 4ms.
2024-01-22T14:37:24,464 INFO [main] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,467 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,474 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Running task: index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z
2024-01-22T14:37:24,476 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Intervals are missing in granularitySpec while this task is potentially overwriting existing segments. Forced to use timeChunk lock.
2024-01-22T14:37:24,477 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - firehoseFactory[InlineInputSource] is not splittable. Running sequentially.
2024-01-22T14:37:24,477 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [SERVER]
2024-01-22T14:37:24,479 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,479 INFO [main] org.eclipse.jetty.server.Server - jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.9+9-Debian-1deb12u1
2024-01-22T14:37:24,480 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Chat handler is already registered. Skipping chat handler registration.
2024-01-22T14:37:24,489 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Determining intervals and shardSpecs
2024-01-22T14:37:24,499 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181. Will not attempt to authenticate using SASL (unknown error)
2024-01-22T14:37:24,501 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.7.175:43330, server: tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181
2024-01-22T14:37:24,510 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181, sessionid = 0x101bf1fb80d0233, negotiated timeout = 30000
2024-01-22T14:37:24,517 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2024-01-22T14:37:24,530 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {}
2024-01-22T14:37:24,530 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {}
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type [CONNECTION_RECONNECTED] for node watcher of role [coordinator].
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type [CONNECTION_RECONNECTED] for node watcher of role [overlord].
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Found intervals and shardSpecs in 59ms
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [http://10.244.1.147:8088] of role [coordinator] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [http://10.244.1.150:8088] of role [overlord] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [coordinator] is now initialized with 1 nodes.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [overlord] is now initialized with 1 nodes.
2024-01-22T14:37:24,696 INFO [main] org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
2024-01-22T14:37:24,721 WARN [task-runner-0-priority-0] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:24,771 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
2024-01-22T14:37:24,771 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.StatusResource as a root resource class
2024-01-22T14:37:24,774 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
2024-01-22T14:37:24,799 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing [1] segments in background
2024-01-22T14:37:24,802 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing segments: [telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z]
2024-01-22T14:37:24,805 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted rows[1] and (estimated) bytes[154]
2024-01-22T14:37:24,837 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,848 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,925 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z] spill[0] to disk in [120] ms (1 rows).
2024-01-22T14:37:24,927 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z
2024-01-22T14:37:24,927 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [1], persisted rows[1], sinks: [1], total fireHydrants (across sinks): [1], persisted fireHydrants (across sinks): [1]
2024-01-22T14:37:24,928 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Preparing to push (stats): processed rows: [1], sinks: [1], fireHydrants (across sinks): [1]
2024-01-22T14:37:24,934 WARN [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to push merged index for segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z].
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-01-22T14:37:24,941 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1050) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:543) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1212) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-31.1-jre.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91) ~[guava-31.1-jre.jar:?]
        at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:151) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:134) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:131) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:940) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        ... 13 more
Caused by: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:998) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        ... 3 more
Caused by: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        ... 3 more
2024-01-22T14:37:24,948 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - handler[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z] not currently registered, ignoring.
2024-01-22T14:37:24,950 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "status" : "FAILED",
  "duration" : 474,
  "errorMsg" : "java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2024-01-22T14:37:25,157 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,181 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,189 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.QueryResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,190 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,193 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.security.ConfigResourceFilter to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,197 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,199 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,200 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:25,221 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@573f2e67{/,null,AVAILABLE}
2024-01-22T14:37:25,232 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,233 INFO [main] org.eclipse.jetty.server.Server - Started @5130ms
2024-01-22T14:37:25,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2024-01-22T14:37:25,252 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,255 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2024-01-22T14:37:25,261 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,261 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2024-01-22T14:37:25,262 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@573f2e67{/,null,STOPPED}
2024-01-22T14:37:25,291 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z].
2024-01-22T14:37:25,292 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2024-01-22T14:37:25,307 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2024-01-22T14:37:25,315 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2024-01-22T14:37:25,422 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x101bf1fb80d0233
2024-01-22T14:37:25,422 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x101bf1fb80d0233 closed
2024-01-22T14:37:25,434 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task

[ArvinZheng]

Hi @winsmith , @asdf2014 , this is because the column name A,B contains comma and it is stored into meta.smooth as is. I think we should document allowed or disallowed characters for column names in the following section.

druid/docs/ingestion/ingestion-spec.md

Line 247 in a6236c3

| name | The name of the dimension. This will be used as the field name to read from input records, as well as the column name stored in generated segments.<br /><br />Note that you can use a [`transformSpec`](#transformspec) if you want to rename columns during ingestion time. | none (required) |

[winsmith]

Documenting allowed or disallowed characters would be absolutely fantastic here <3