Skip to content

Import of data fails #208

Open
Open
@sawan

Description

Checklist

  • I have verified that I am using a GIS-enabled database, such as PostGIS or Spatialite.
  • I have verified that that issue exists against the master branch of django-cities.
  • I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
  • I have reduced the issue to the simplest possible case.
  • I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

Steps to reproduce

python manage.py cities --import=all

Expected behavior

Import of all data.

Actual behavior

Fails at the index build step.

python manage.py cities --import=all
Importing countries: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [00:00<00:00, 505.84it/s]
Importing country neighbours: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [00:00<00:00, 389.58it/s]
Building country index: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [00:00<00:00, 26404.51it/s]
Importing regions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 3972/3972 [00:07<00:00, 535.45it/s]
Building region index: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 3972/3972 [00:00<00:00, 18794.72it/s]
Importing subregions: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 45731/45731 [01:35<00:00, 480.91it/s]
Building region index:   0%|                                                                                                     | 1/47993 [00:00<3:25:01,  3.90it/s]Traceback (most recent call last):
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/cities/management/commands/cities.py", line 160, in handle
    func()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/cities/management/commands/cities.py", line 470, in import_city
    self.build_region_index()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/cities/management/commands/cities.py", line 396, in build_region_index
    desc="Building region index"):
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/tqdm/_tqdm.py", line 979, in __iter__
    for obj in iterable:
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 268, in __iter__
    self._fetch_all()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 1188, in _fetch_all
    self._prefetch_related_objects()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 723, in _prefetch_related_objects
    prefetch_related_objects(self._result_cache, *self._prefetch_related_lookups)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 1569, in prefetch_related_objects
    obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 1699, in prefetch_one_level
    all_related_objects = list(rel_qs)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 268, in __iter__
    self._fetch_all()
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1065, in execute_sql
    cursor.execute(sql, params)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/sawanvithlani/src/virtual-envs/xxx/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 296, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: too many SQL variables

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions