Problem with silk in child threads

[marino39]

Hi,

I'm working on synchronous API that is processing some data. In order to speed things up I have decided to spawn multiple threads in Django request and do processing there. The problem occurs when one of the threads tries to execute sql queries. Is there any way to run silk and spawn child threads?

The self.register_objects(TYP_QUERIES, *args) could just give a warning in that case and carry on.

File "/zzzz/service/yyyy/xxxx.py", line 883, in get_negative_info
    p_1 = Node.objects.get(id=node)
  File "/Library/Python/2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 381, in get
    num = len(clone)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 240, in __len__
    self._fetch_all()
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 52, in __iter__
    results = compiler.execute_sql()
  File "/Library/Python/2.7/site-packages/silk/sql.py", line 52, in execute_sql
    DataCollector().register_query(query_dict)
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 122, in register_query
    self.register_objects(TYP_QUERIES, *args)
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 109, in register_objects
    self.ensure_middleware_installed()
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 43, in ensure_middleware_installed
    raise_middleware_error()
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 24, in raise_middleware_error
    'Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can '
RuntimeError: Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can execute process_request and process_response. If an earlier middleware returns from either of these methods, Silk will not have the chance to inspect the request/response objects.

Thanks,
Marcin

[FraBle]

I have the same issue! Is there any way to temporarily store the request context information that Silk needs?

[tufelkinder]

Having the identical problem. Looking forward to finding some kind of solution...

[rubendura]

Just getting started with silk. Just run across this. The problem seems to be with a thread local on the DataCollector class, which is a Singleton. When a thread other than the first one tries to create a DataCollector it will retrieve the already initialised object, but as that object uses thread locals internally it is actually uninitialised (other than the first thread that created it).
I am just getting started with silk, so I'm not familiar enough with the code base to submit a PR yet. This is as far as I could get unfortunately.

[avelis]

@rubendura Thanks for the updated information on furthering the problem space for potential solution.

[FabioFleitas]

We are having the same issue

[vipul1994]

We are also having the same issue. Any update on this?

[lokeshatbigbasket]

Same issue, any update on this??? getting this when using django_extensions, django-debug-toolbar and silk, provided SilkMiddleware is the second one in Middleware List, where as GzipMiddleware is first. Still getting the issue

[kut]

Same issue. Any recommended way to dynamically disable the middleware (i.e. when we are using threads)?

[vinayan3]

I'm running into this exact problem. I'm running a SQL query in a thread after a request finishes for aggregation purposes. Is there a way I can disable silk for a thread?

[Atorich]

Same issue. Using with django-channels.

[Joeylhzn]

Same issue.I add DataCollector().configure() to my function and this error never show up. I don't know if there are any other problems.

[m-hu]

I have seen the similar issue, and it turned out to be a threading related issue.
In fact, the DataCollector is a singleton. Its instance contains a threading.local. When the current thread switches to another, due to singleton behavior, it will not be re-instantiated, but the local information that belongs to the previous thread will be lost, as they are local to its belonging thread.

My idea is to have a DataCollector class specific singleton implementation, in addition to the check of existence of instance itself, check also if this thread local info is lost. If this thread specific info is lost, it indicates a thread switch, then the data collector needs to be re-instantiated.

Otherwise, it is wiser to implement a threading aware singleton. That is to say, the singleton parent class will keep a mapping between the thread instance and its singleton instance, if thread context changes, the singleton instance needs to be changed as well.

[mdolsma]

It seems that commit f7fd730 has improved the situation...

I have spent time to reproduce this error in the simplest way, using the input from @rubendura and @pptime. I forked django-silk and added a view runworker to the example_app which would simply delete all Blind objects from the DB using a worker thread:

class Worker(threading.Thread):

    def __init__(self):
        super().__init__()

    def run(self):
        from .models import Blind
        print("worker started on {}".format(self.getName()))
        Blind.objects.all().delete()


def runworker(request):
    Worker().start()
    return HttpResponse("Worker started")

Using the dev server, I would expect that a request to http://127.0.0.1:8000/example_app/runworker triggers the middleware error. However, this actually worked fine. After checking the code and latest commits, i noticed the commit mentioned earlier. To verify, i commented these lines 14-15 from \silk\sql.py:

def _should_wrap(sql_query):
    # if not DataCollector().request:
    #     return False

    for ignore_str in SilkyConfig().SILKY_IGNORE_QUERIES:
        if ignore_str in sql_query:
            return False
    return True

and indeed... the MiddleWare error appeared again:

$> python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
March 15, 2019 - 00:25:09
Django version 2.1.7, using settings 'project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Init datacollector Thread-1
2019-03-15 00:25:29,289 DEBUG   process_request [process_request (middleware.py:101)]
2019-03-15 00:25:29,307 DEBUG   Created new request model with pk fd709875-9c96-408b-9e5d-ca210ba4ec37 [construct_request_model (model_factory.py:225)]
2019-03-15 00:25:29,317 DEBUG   Process response [_process_response (middleware.py:112)]
2019-03-15 00:25:29,317 DEBUG   Creating response model for request model with pk fd709875-9c96-408b-9e5d-ca210ba4ec37 [construct_response_model (model_factory.py:280)]
2019-03-15 00:25:29,339 DEBUG   Process response done. [_process_response (middleware.py:133)]
2019-03-15 00:25:35,602 DEBUG   process_request [process_request (middleware.py:101)]
2019-03-15 00:25:35,621 DEBUG   Created new request model with pk 6ec7afec-562e-46b6-88a1-57ae112d5d0e [construct_request_model (model_factory.py:225)]
2019-03-15 00:25:37,055 DEBUG   Process response [_process_response (middleware.py:112)]
2019-03-15 00:25:37,056 DEBUG   Creating response model for request model with pk 6ec7afec-562e-46b6-88a1-57ae112d5d0e [construct_response_model (model_factory.py:280)]
2019-03-15 00:25:37,082 DEBUG   Process response done. [_process_response (middleware.py:133)]
2019-03-15 00:25:40,598 DEBUG   process_request [process_request (middleware.py:101)]
2019-03-15 00:25:40,614 DEBUG   Created new request model with pk b0bd570a-d8ff-4545-9d69-867eed40511b [construct_request_model (model_factory.py:225)]
worker started on Thread-5
2019-03-15 00:25:40,617 DEBUG   Process response [_process_response (middleware.py:112)]
2019-03-15 00:25:40,618 DEBUG   Creating response model for request model with pk b0bd570a-d8ff-4545-9d69-867eed40511b [construct_response_model (model_factory.py:280)]
2019-03-15 00:25:40,631 DEBUG   Process response done. [_process_response (middleware.py:133)]
Exception in thread Thread-5:
Traceback (most recent call last):
  File "~\AppData\Local\Programs\Python\Python36\Lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "xxx\django-silk\project\example_app\views.py", line 28, in run
    Blind.objects.all().delete()
  File "~\.virtualenvs\django-silk-DFO8FG2O\lib\site-packages\django\db\models\query.py", line 663, in delete
    deleted, _rows_count = collector.delete()
  File "~\.virtualenvs\django-silk-DFO8FG2O\lib\site-packages\django\db\models\deletion.py", line 282, in delete
    count = qs._raw_delete(using=self.using)
  File "~\.virtualenvs\django-silk-DFO8FG2O\lib\site-packages\django\db\models\query.py", line 677, in _raw_delete
    return sql.DeleteQuery(self.model).delete_qs(self, using)
  File "~\.virtualenvs\django-silk-DFO8FG2O\lib\site-packages\django\db\models\sql\subqueries.py", line 75, in delete_qs
    cursor = self.get_compiler(using).execute_sql(CURSOR)
  File "xxx\django-silk\project\silk\sql.py", line 55, in execute_sql
    DataCollector().register_query(query_dict)
  File "xxx\django-silk\project\silk\collector.py", line 125, in register_query
    self.register_objects(TYP_QUERIES, *args)
  File "xxx\django-silk\project\silk\collector.py", line 109, in register_objects
    self.ensure_middleware_installed()
  File "xxx\django-silk\project\silk\collector.py", line 45, in ensure_middleware_installed
    raise_middleware_error()
  File "xxx\django-silk\project\silk\collector.py", line 25, in raise_middleware_error
    'Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can '
RuntimeError: Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can execute process_request and process_response. If 
an earlier middleware returns from either of these methods, Silk will not have the chance to inspect the request/response objects.

Hope this helps ...
Matthijs