Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long error message get truncated #747

Closed
tkilias opened this issue Apr 20, 2020 · 9 comments
Closed

Long error message get truncated #747

tkilias opened this issue Apr 20, 2020 · 9 comments

Comments

@tkilias
Copy link

tkilias commented Apr 20, 2020

Environment

  • Python: 64-bit, Python 2.7 and 3.6
  • pyodbc: 4.0.24
  • OS: Ubuntu 18 LTS using unixodbc
  • DB: Exasol 6.2
  • driver: Exasol 6.2

Issue

Observed behavior:

  • Exasol can return very long error messages in the context of User Defined Functions (UDF), because the user code might throw an exception with a very long stack trace. These long error messages get truncated by pyodbc which makes the debugging of the these UDFs more difficult.

Expected behavior:

  • No truncation of the error message if possible.

Suspected cause:

  • I already checked the driver and it returns the complete error message.
    I also attached the odbc log of the driver in case it helps: exaodbc.log.
  • Afterwards, I searched in the code of pyodbc where the truncation might happen and I think the following line is responsible:
    ODBCCHAR szMsg[1024];

Proposed solution:

  • Do you think it is possible to increase the buffer size or actually allocate it dynamically depending on size of the error message?

I might try to build a test version of pyodbc with a large buffer to test this theory. If you need more information don't hesitate to contact me.

@v-chojas
Copy link
Contributor

Increasing it in pyODBC won't help because there's another fixed-size buffer in the driver manager - I believe it is either 1k or 512 too. The real issue is with the driver returning excessively long error messages, if it is more than SQL_MAX_MESSAGE_LENGTH (that corresponds to the fixed-size buffer in the DM) it should be splitting them into multiple records.

@tkilias
Copy link
Author

tkilias commented Apr 20, 2020

Ok, that I didn't know. Thanks for the information. A question to understand ODBC a bit better, I think, I saw that multiple records get concatenated with ";" in between which would indicate they are different error messages. Are records usually meant to be different error message?

@v-chojas
Copy link
Contributor

pyODBC will concatenate multiple error records together, since the driver can post more than one for each API call (and as mentioned, should do so if the total error message exceeds the limit.)

@tkilias
Copy link
Author

tkilias commented Apr 21, 2020

Thanks. Hopefully, a last question.
Does the concatenation also work with unixodbc, because I saw the following comment in the code?

// than once. I hate to do this, but I'm going to only call it once for non-Windows platforms for now...

@v-chojas
Copy link
Contributor

Looks like it doesn't, to work around a different ODBC driver's bug. Maybe @mkleehammer can consider making these workarounds optional/configurable.

@keitherskine
Copy link
Collaborator

Just for info, the comment in the code was put there 11 years ago, so perhaps unixodbc and/or the Postgres drivers have been fixed by now to prevent those crashes from happening. Maybe the restriction doesn't apply anymore.

@reedjosh
Copy link

reedjosh commented May 7, 2020

I see a ton of issues about error messages here in PyODBC. #489 #640 .

I think as mentioned in #489, driver awareness/dialecting would be useful for a number of scenarios. In this case, only call once if on windows with a postgres driver of a version that is broken--instead of crippling PyODBC for all OS/Driver Combos.

Many drivers are broken in unique ways. Some drivers are closed source and others are like moving a glacier to get fixed.

So, being able to react based upon OS, ODBC driver manager, and ODBC DB driver would allow for workarounds of these sorts.

PyODBC is great, but it could be much better if these sort of issues could be worked around--hopefully driving further adoption of ODBC in general.

@keitherskine
Copy link
Collaborator

This may be addressed and/or fixed with PR #881 .

@gordthompson
Copy link
Collaborator

#881 has been merged. If this particular problem persists then please re-open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants