UnicodeDecodeError in sql.py: leads to 500 internal error

[ChristopherRabotin]

It seems that silk does not support UTF-8 characters in SQL queries.

Internal Server Error: /
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 223, in get_response
    response = middleware_method(request, response)
  File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.54.0.41/newrelic/hooks/framework_django.py", line 331, in wrapper
    return wrapped(*args, **kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/silk/middleware.py", line 123, in process_response
    self._process_response(response)
  File "/app/.heroku/python/lib/python2.7/site-packages/silk/middleware.py", line 107, in _process_response
    silk_response.save()
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 827, in _save_table
    forced_update)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 877, in _do_update
    return filtered._update(values) > 0
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 580, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1062, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/app/.heroku/python/lib/python2.7/site-packages/silk/sql.py", line 38, in execute_sql
    sql_query = q % params
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)

[trik]

as a temporary fix you can change model_factory.py:261 from
silky_response.raw_body = content
to
silky_response.raw_body = content.encode('UTF-8', 'replace')

[avelis]

@trik I appreciate the helpful temporary solution. I can apply it for now if that works for everyone.

@ChristopherRabotin I will honestly have to look into official UTF-8 support.

If anyone has time for a pull request I can be happy to take a look at it and merge it in.

[trik]

@avelis I was already thinking to a pull request but i don't understand well enough those lines.. the UnicodeDecodeError is never raised, the code inside the try/except does nothing but an assignment. we could keep my patch, not ignoring errors and if UDE is raised cause the sql has some binary parameter, use a different encoding to log (maybe b64)

[ChristopherRabotin]

After a few days of testing when I opened this issue, we ended up not being able to use silk in production as it would lead to a substantial increase in the number of database queries making the main functionalities of the platform unavailable.