Long error message get truncated

[tkilias]

Environment

• Python: 64-bit, Python 2.7 and 3.6
• pyodbc: 4.0.24
• OS: Ubuntu 18 LTS using unixodbc
• DB: Exasol 6.2
• driver: Exasol 6.2

Issue

Observed behavior:

• Exasol can return very long error messages in the context of User Defined Functions (UDF), because the user code might throw an exception with a very long stack trace. These long error messages get truncated by pyodbc which makes the debugging of the these UDFs more difficult.

Expected behavior:

• No truncation of the error message if possible.

Suspected cause:

• I already checked the driver and it returns the complete error message.
  I also attached the odbc log of the driver in case it helps: exaodbc.log.
• Afterwards, I searched in the code of pyodbc where the truncation might happen and I think the following line is responsible:

  pyodbc/src/errors.cpp

  Line 250 in a3693bf

  ODBCCHAR szMsg[1024];

Proposed solution:

• Do you think it is possible to increase the buffer size or actually allocate it dynamically depending on size of the error message?

I might try to build a test version of pyodbc with a large buffer to test this theory. If you need more information don't hesitate to contact me.

[v-chojas]

Increasing it in pyODBC won't help because there's another fixed-size buffer in the driver manager - I believe it is either 1k or 512 too. The real issue is with the driver returning excessively long error messages, if it is more than SQL_MAX_MESSAGE_LENGTH (that corresponds to the fixed-size buffer in the DM) it should be splitting them into multiple records.

[tkilias]

Ok, that I didn't know. Thanks for the information. A question to understand ODBC a bit better, I think, I saw that multiple records get concatenated with ";" in between which would indicate they are different error messages. Are records usually meant to be different error message?

[v-chojas]

pyODBC will concatenate multiple error records together, since the driver can post more than one for each API call (and as mentioned, should do so if the total error message exceeds the limit.)

[tkilias]

Thanks. Hopefully, a last question.
Does the concatenation also work with unixodbc, because I saw the following comment in the code?

pyodbc/src/errors.cpp

Line 269 in a3693bf

// than once. I hate to do this, but I'm going to only call it once for non-Windows platforms for now...

[v-chojas]

Looks like it doesn't, to work around a different ODBC driver's bug. Maybe @mkleehammer can consider making these workarounds optional/configurable.

[keitherskine]

Just for info, the comment in the code was put there 11 years ago, so perhaps unixodbc and/or the Postgres drivers have been fixed by now to prevent those crashes from happening. Maybe the restriction doesn't apply anymore.

[reedjosh]

I see a ton of issues about error messages here in PyODBC. #489 #640 .

I think as mentioned in #489, driver awareness/dialecting would be useful for a number of scenarios. In this case, only call once if on windows with a postgres driver of a version that is broken--instead of crippling PyODBC for all OS/Driver Combos.

Many drivers are broken in unique ways. Some drivers are closed source and others are like moving a glacier to get fixed.

So, being able to react based upon OS, ODBC driver manager, and ODBC DB driver would allow for workarounds of these sorts.

PyODBC is great, but it could be much better if these sort of issues could be worked around--hopefully driving further adoption of ODBC in general.

[keitherskine]

This may be addressed and/or fixed with PR #881 .

[gordthompson]

#881 has been merged. If this particular problem persists then please re-open.