Exception in mmap_dict.py with multiprocess in 0.5.0 #357
Description
What's the issue?
JSON unicode decode error using multiprocess with moderate throughput
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 166: invalid start byte
File "django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "newrelic/hooks/framework_django.py", line 544, in wrapper
return wrapped(*args, **kwargs)
File "django_prometheus/exports.py", line 115, in ExportToDjangoView
metrics_page = prometheus_client.generate_latest(registry)
File "prometheus_client/exposition.py", line 89, in generate_latest
for metric in registry.collect():
File "prometheus_client/registry.py", line 75, in collect
for metric in collector.collect():
File "prometheus_client/multiprocess.py", line 30, in collect
return self.merge(files, accumulate=True)
File "prometheus_client/multiprocess.py", line 44, in merge
for key, value in d.read_all_values():
File "prometheus_client/mmap_dict.py", line 100, in read_all_values
for k, v, _ in self._read_all_values():
File "prometheus_client/mmap_dict.py", line 95, in _read_all_values
yield encoded.decode('utf-8'), value, posWhat configuration?
prometheus-client==0.5.0django-prometheus==1.0.15- Apache 2.4 with
mod_wsgi - Kubernetes 1.11.3
Metrics are exported via HTTP.
What happens?
At minimal throughput (single requests per minute up to tens of requests per minute) there's no problem. Once throughput increases to around 200 requests/minute (quite moderate), we start seeing the above exceptions.
These are containerised workloads, so there are no metrics files that need to be cleared away when the main process starts.
I'm rebuilding against 0.4.2 to validate that the issue was introduced in 0.5.0.