Traceback when importing a very long source string

[friedelwolff]

When importing a very long source string (the one causing this was over 18 000 characters), I get the following traceback from the import script (caused by PostgreSQL):

Traceback (most recent call last):
File "bin/amagama-manage", line 45, in
manager.run()
File "env/lib/python2.7/site-packages/flask_script.py", line 731, in run
result = self.handle(sys.argv[0], command, sys.argv[2:])
File "env/lib/python2.7/site-packages/flask_script.py", line 704, in handle
return command.handle(app, _positional_args, *_command_namespace.dict)
File "env/lib/python2.7/site-packages/flask_script.py", line 193, in handle
return self.run(_args, *_kwargs)
File "amagama/amagama/commands.py", line 123, in run
self.real_run(slang, tlang, filename)
File "amagama/amagama/commands.py", line 138, in real_run
self.handlefile(filename)
File "amagama/amagama/commands.py", line 161, in handlefile
current_app.tmdb.add_store(store, source_lang, target_lang, commit=True)
File "amagama/amagama/tmdb.py", line 302, in add_store
return self.add_list(units, source_lang, target_lang, commit)
File "amagama/amagama/tmdb.py", line 321, in add_list
self.get_all_sids(units, source_lang)
File "amagama/amagama/tmdb.py", line 283, in get_all_sids
cursor.executemany(insert_query, params)
psycopg2.OperationalError: index row requires 8960 bytes, maximum size is 8191

I guess the issue is the unique index. Maybe we could simply decide not to support such long strings, but a translator hitting that (and this was from a real life file) would definitely appreciate a TM match for it. (The string was the whole GPL and was just copied from source for the file I had, though.)

We will probably benefit from partly going back to using hashes, but only for indexes (not in a column as we did before):
http://gjsql.wordpress.com/2009/04/19/how-to-speed-up-index-on-bytea-text-etc/
I'm not sure if that is the right solution, though.