CollectD error when sending telemetry "SyntaxError: Unexpected token N"

[AlexFlat]

Hi,

I am trying to integrate CollectD (Ubuntu) with Application Insights.
When trying to previously use Log4J I was getting a similar error and downgraded to version 1.0.9 as per the article
https://stackoverflow.com/questions/46919565/unable-to-push-logs-to-application-insights-using-logback-appenders

Here is the errors I am getting for collectd after starting the service.

collectd.service - Statistics collection and monitoring daemon
   Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-12-10 20:10:31 EST; 2min 10s ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
           https://collectd.org
  Process: 42840 ExecStartPre=/usr/sbin/collectd -t (code=exited, status=0/SUCCESS)
 Main PID: 42841 (collectd)
    Tasks: 42 (limit: 19660)
   CGroup: /system.slice/collectd.service
          42841 /usr/sbin/collectd

Dec 10 21:12:03 osboxes collectd[42841]: AI: ERROR 10-12-2017 21:12, 36: Failed to send, Bad request  : {"itemsReceived":1,"itemsAccepted":0,"errors":[{"index":0,"statusCode":400,"message":"SyntaxError: Unexpected token N"}]}
Dec 10 21:12:03 osboxes collectd[42841]: AI: ERROR 10-12-2017 21:12, 26: Failed to send, Bad request  : {"itemsReceived":1,"itemsAccepted":0,"errors":[{"index":0,"statusCode":400,"message":"SyntaxError: Unexpected token N"}]}
Dec 10 21:12:03 osboxes collectd[42841]: AI: ERROR 10-12-2017 21:12, 31: Failed to send, Bad request  : {"itemsReceived":1,"itemsAccepted":0,"errors":[{"index":0,"statusCode":400,"message":"SyntaxError: Unexpected token N"}]}

However I am unable to locate the 1.0.9 version of CollectD.

Do you have a published link for this library?
Is this a known issue for collectd that will be fixed?

Thanks in advance,

Alex

[dhaval24]

Hi, @AlexFlat I think there was some issue with Log4J in 1.0.10 version that was probably because of some sanitization issue in the JSON packet created by SDK. The particular sanitization bug was fixed and merged into the master. However, there might be other potential issues with collectD that we might not be aware of. To isolate the root cause of this problem, I would suggest the following:

1. Install application insights sdk with collectD plugin, don't integrate Log4J. See if the similar errors come up.
2. If the above results in false there is a bug with Log4J adapter which should be fixed.

I would also suggest that you provide steps to reproduce this issue locally so we can debug and fix this issue. Thanks in advance!

[AlexFlat]

Thanks for the quick response.
Just to clarify, Log4J is not installed at all, I was previously testing Log4J in another environment.

I am attempting to use CollectD in conjunction with another plugin to monitor ElasticSearch with Application Insights.

As such, my setup was planning to;

• Collect metrics from ElasticSearch using a 3rd party Python script
• Integrate CollectD into App Insights
• View elastic cluster metrics in App Insights

Currently the collectd is sort of working as I see live data in the Azure Portal under "Live Stream".
However, I noticed that I was receiving these errors above in the systemctl logs.

Anyway, I have pulled back all the python stuff and am still receiving the errors in the log.

Here are the repro steps (pretty much as per the doco )

1. Install the Java SDK and CollectD as per the above article
2. Copy the applicationinsights-collectd-1.0.10.jar to /usr/share/collectd/java
3. Add ApplicationInsights.xml to /usr/share/collectd/java - see attached for my appinsights.xml
4. Edit the collecd.conf to include the JAVA plugin - see attached for my collectd.conf
5. Start collectd "systemctl start collectd"
6. Goto the Portal  - Live Stream displays CPU
7. Goto the log "journalctl -xn" - there are messages in the logs with
"SyntaxError: Unexpected token N"

I'm not sure what its attempting to send, but I thought it was my python plugin.
Even with this disabled the messages are appearing.
As such, I want to ensure collectd is operational before trying to get my 3rd party plugin to work.

Thanks

Alex

collectd.conf.txt
ApplicationInsights.xml.txt

[dhaval24]

It seems that this is some kind of sanitization bug where JSON is getting some unexpected character. I will have a look at this. In the mean while you might try to build the SDK locally which has the sanitization issues from the previous release fixed. You need to fork the repository, and run gradlew clean, assemble, build and test to build the SDK locally.

[mmaitre314]

I am hitting a similar issue trying to get perf metrics from an Azure Databricks cluster (no log4j). The memory and load metrics go through but not cpu or disk.

Config:

FQDNLookup true
Interval 10

# Filelog plugin
LoadPlugin logfile
<Plugin logfile>
	LogLevel "trace"
	File "/dbfs/databricks/init/output/TestAppInsights/collectd.log"
	Timestamp true
	PrintSeverity true
</Plugin>

# Perf counter plugins
LoadPlugin cpu
LoadPlugin disk
LoadPlugin load
LoadPlugin memory

# Java Plugin
LoadPlugin "java"
<Plugin "java">
  JVMArg "-verbose:jni"
  JVMArg "-Djava.class.path=/usr/share/collectd/java/applicationinsights-collectd-1.0.10.jar:/usr/share/collectd/java/collectd-api.jar"

  # Enabling Application Insights plugin
  LoadPlugin "com.microsoft.applicationinsights.collectd.ApplicationInsightsWriter"

  # Configuring Application Insights plugin
  <Plugin ApplicationInsightsWriter>
	InstrumentationKey "a0...99"
    SDKLogger true
  </Plugin>
</Plugin>

Error while running sudo collectd -f:

AI: ERROR 14-12-2017 01:04, 223: Failed to send, Throttling (Partial messages of the transmission were rejected)  : {"itemsReceived":41,"itemsAccepted":9,"errors":[{"index":0,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":2,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":4,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":5,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":7,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":8,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":9,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":10,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":11,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":12,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":13,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":14,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":15,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":16,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":17,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":18,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":19,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":20,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":21,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":22,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":23,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":24,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":25,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":26,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":27,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":28,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":29,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":30,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":31,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":32,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":33,"statusCode":400,"message":"SyntaxError: Unexpected token N"},{"index":40,"statusCode":400,"message":"SyntaxError: Unexpected token N"}]}

[mmaitre314]

Is it possible to have the CollectD plugin dump the JSON payload of the HTTP calls? The invalid token should be easy to find there.

[dhaval24]

@mmaitre314 I am not sure if we can dump that, I will double check, but what I know could be done for sure here is that: you can enable fidler on the application and see the post request packet itself. You should be able to see the invalid token there very easily. There could some serialization issue or some token being generated which is not properly escaped etc. Let me know if you could do that.

[mmaitre314]

Unfortunately, no Fiddler on Azure Databricks (that would have been too simple...). I was able to put a hook in InProcessTelemetryChannel though and it looks like the issue is handling of NaN.

• Hook:

            jsonWriter = new JsonTelemetryDataSerializer(writer);
            telemetry.serialize(jsonWriter);
            jsonWriter.close();
            String asJson = writer.toString();
            System.out.print("<1<" + asJson + ">1>");

• Example of JSON metric payload where max is NaN:

{
	"ver": 1,
	"name": "Microsoft.ApplicationInsights.a0e24c07ddae4bae8b0a8eb4fa9b2999.Metric",
	"time": "2017-12-15T08:17:12.332+0000",
	"sampleRate": 100.0,
	"iKey": "a0e24c07-ddae-4bae-8b0a-8eb4fa9b2999",
	"tags": {
		"ai.internal.sdkVersion": "java:2.0.0-beta-snapshot",
		"ai.device.id": "1213-024324-avow180_10_139_64_12",
		"ai.device.locale": "en-US",
		"ai.internal.nodename": "1213-024324-avow180_10_139_64_12",
		"MetricSource": "CollectD-Plugin",
		"ai.device.os": "Linux",
		"ai.device.roleInstance": "1213-024324-avow180_10_139_64_12",
		"ai.device.osVersion": "Linux"
	},
	"data": {
		"baseType": "MetricData",
		"baseData": {
			"ver": 2,
			"metrics": [{
				"name": "\/cpu\/0\/cpu\/user\/value",
				"kind": 1,
				"value": 11486.0,
				"min": 0.0,
				"max": NaN
			}],
			"properties": {
				"CollectD-Host": "1213-024324-avow180_10_139_64_12"
			}
		}
	}
}

[mmaitre314]

My guess is that the issue is in JsonTelemetryDataSerializer.write() within the WRAPPER_TYPES if statement the code writes Double as-is to a string and NaN is not a valid string.

    private <T> void write(T item) throws IOException {
        if (item instanceof JsonSerializable) {
            StringWriter stringWriter = new StringWriter();

            String jsonStringToAppend = createJsonFor((JsonSerializable)item);
            if (Strings.isNullOrEmpty(jsonStringToAppend)) {
                return;
            }

            out.write(jsonStringToAppend);
        } else {
            if (WRAPPER_TYPES.contains(item.getClass()))
            {
                out.write(String.valueOf(item));
            } else {
                String truncatedName = truncate(String.valueOf(item), 8192);
                String sanitizedItem = sanitizeStringForJSON(truncatedName, 8192);
                out.write(JSON_COMMA);
                out.write(sanitizedItem);
                out.write(JSON_COMMA);
            }
        }
    }

[dhaval24]

Hi @mmaitre314 thank you very much for digging into the root cause of the issue. I was suspecting some sort of sanitization issue but it was very essential to get the exact packet to find out the root cause. I really appreciate your digging around. Would it be possible for you to create a PR for this? It would be highly appreciable. Let me know!