Skip to content

Conversation

anton-abushkevich
Copy link
Contributor

@anton-abushkevich anton-abushkevich commented Dec 13, 2022

Backend for OHDSI/Atlas#2800

@chrisknoll
Copy link
Collaborator

I tried to compare with the attached concept set CSV and got this error:

Caused by: org.postgresql.util.PSQLException: ERROR: stack depth limit exceeded
  Hint: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)

Looks like this error isn't a java stack limit, but rather something in PGAdmin.

concept_compare_1.csv

@chrisknoll
Copy link
Collaborator

Just as an implementation note: I think this is a case where it may make more sense to do this in java-space: by sorting the list and scanning through each result, you can determine which elements are absent/present from each side by scanning through the lists (something not easily done on sql side), but a sql solution could be done using a combinatino of joins (inner join will find 'both', left join will find in C1, and right join could find those in C2.

@@ -0,0 +1 @@
select concept_id from @vocabulary_database_schema.CONCEPT where (concept_code, vocabulary_id) in (@conceptColumns)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From discussion on Atlas WG, this query could be problematic and the root cause of the issue reported in #2176 (comment). This may require refactoring to use a temp table that is created using a UNION ALL with all of the concepts. Tagging @chrisknoll as he may be able to provide a code pointer to other places in the code base where this is done.

working with tamp table was added

private String getQuery(final ConceptSetExpression csExpression, final ExpressionType type,
final Source source, final JdbcTemplate jdbcTemplate) {
final ConceptSetExpressionQueryBuilder builder = new ConceptSetExpressionQueryBuilder();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move the instantiation of the builder into 'else' block.

working with tamp table was added
@anton-abushkevich
Copy link
Contributor Author

@chrisknoll, The fix is ready - please review.

@chrisknoll chrisknoll merged commit 00cfbfa into master Feb 6, 2023
@delete-merged-branch delete-merged-branch bot deleted the add_csv_for_comparison_conceptsets branch February 6, 2023 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants