Query Optimizations #482
Merged
Query Optimizations #482
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Added new environment variables for document processing services in `.envs/.test/.django`. - Removed unused `.claude/settings.local.json` file. - Updated `.gitignore` to include additional files. - Enhanced `README.md` with more detailed project description. - Updated various GitHub Actions workflows to use the latest `actions/checkout` version. - Introduced new GraphQL resolvers and optimized file handling in the backend. - Added performance optimizations and new test cases for document handling.
- Updated `docker-compose.yml` to improve service dependencies with health checks for `postgres` and `redis`. - Removed `celeryworker` from dependencies to prevent circular dependency issues. - Enhanced GraphQL resolvers for optimized document annotations, adding filter checks and improving permission handling. - Deleted unused progressive types and related GraphQL fields to streamline the schema.
- Introduced `created_by_analysis` and `created_by_extract` fields in the Annotation model to enforce privacy for annotations created by analyses and extracts. - Implemented validation to ensure an annotation cannot be created by both an analysis and an extract. - Enhanced the AnnotationQueryOptimizer to filter annotations based on user permissions, ensuring private annotations are only visible to users with the appropriate access. - Updated GraphQL resolvers to respect the new permission model, requiring both object and corpus permissions for visibility. - Added comprehensive tests to validate the new privacy model and permission checks for analyses and extracts. - Updated documentation to reflect changes in the permissioning system and the new annotation privacy features. - Created a debug script for testing permission checks in a controlled environment. - Optimized existing queries to improve performance and reduce unnecessary database hits.
- Added detailed permission checks for `RemoveAnnotation`, `RejectAnnotation`, `ApproveAnnotation`, and `AddRelationship` mutations to ensure users have the appropriate access rights based on the privacy model. - Introduced informative messages in mutation responses to clarify permission issues and operation outcomes. - Implemented comprehensive tests for annotation permission handling, ensuring that privacy rules are respected across different user roles. - Updated the `user_has_permission_for_obj` function to include special handling for annotations with privacy fields, enhancing permission validation logic.
- Replaced the deprecated `resolve_oc_model_queryset` function with a more secure and maintainable `visible_to_user` method across various models. - Updated GraphQL resolvers to utilize the new permission logic, ensuring users can only access objects they are permitted to see. - Enhanced the `BaseVisibilityManager` to provide consistent permission filtering for all models. - Removed legacy resolver code and improved test coverage for visibility checks, ensuring robust permission handling. - Updated documentation to reflect the changes in permissioning logic and the deprecation of the old resolver function.
- Removed IMAGE_PREFIX environment variable and replaced it with a step to convert the repository owner to lowercase. - Updated image tagging to use the lowercase repository owner, ensuring consistent naming conventions for built images. - Enhanced workflow clarity by streamlining the environment variable usage.
…ermission handling - Added a custom resolver for the annotations field in the CorpusType class to compute permissions accurately using AnnotationQueryOptimizer. - Introduced a new migration to modify constraints on the Annotation model, ensuring annotations can only be created by one source at a time. - Updated UserFeedbackManager to delegate visibility checks to the queryset's method - Enhanced UserFeedbackQuerySet to refine visibility logic for user feedback based on creator and public status. - Added comprehensive tests for permission filtering across corpuses, documents, annotations, and labels, ensuring robust access control.
…reuse - Updated PooledS3Boto3Storage to improve performance by reusing boto3 S3 clients within threads and configuring connection pool size for concurrent operations. - Added retry logic for resilience during S3 operations. - Removed the redundant OptimizedS3Boto3Storage class and its related functionality to streamline the codebase.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…k mutations permission checks - Introduced a new COMMENT permission type, allowing users to comment on annotations and relationships based on document and corpus permissions. - Updated RejectAnnotation and ApproveAnnotation mutations to enforce COMMENT permission checks, ensuring only authorized users can provide feedback. - Enhanced permission handling in the AnnotationQueryOptimizer to support the new COMMENT permission logic, including a special mode for open commenting. - Added comprehensive tests to validate the new COMMENT permission system and its integration with existing feedback functionalities. - Updated documentation to reflect changes in permissioning logic and the introduction of the COMMENT permission.
- Added model-specific optimizations in the PermissionedTreeQuerySet for the "corpus" model, utilizing select_related and prefetch_related to enhance database query efficiency. - Introduced new tests to validate the optimizations applied in the Corpus and Document models, ensuring that the expected queryset behaviors are maintained. - Enhanced existing test coverage for visibility checks, confirming that optimizations do not affect permission logic.
- Introduced a new COMMENT permission for the Analysis model, allowing users to comment on analyses. - Created a migration to update the model options and include the new permission in the permissions list. - Ensured consistency with existing permission structures in the application.
…g migrations - Introduced COMMENT permissions for multiple models including GremlinEngine, Analyzer, AnnotationLabel, LabelSet, Note, Conversation, ChatMessage, CorpusQuery, CorpusAction, DocumentAnalysisRow, DocumentRelationship, Column, Datacell, Extract, Fieldset, UserFeedback, Assignment, UserExport, and UserImport. - Created migrations to update model options and include the new COMMENT permissions in the permissions list for each model. - Ensured consistency with existing permission structures throughout the application.
Closed
- Introduced a new test method to validate various annotation permission types including CREATE, CRUD, ALL, and unsupported permissions. - Ensured that permissions are correctly enforced for users with and without access, enhancing coverage of the permissioning logic.
…ive tests - Updated comments in BaseVisibilityManager to clarify the purpose of the top-level permission logic. - Introduced a new test suite for BaseVisibilityManager, ensuring full coverage of the visible_to_user method across various user scenarios and permissions. - Enhanced test cases to validate edge cases, including handling of anonymous users, superusers, and authenticated users with specific permissions.
…iltering - Introduced a new test suite for Query Optimizer methods, covering AnnotationQueryOptimizer, RelationshipQueryOptimizer, and ExtractQueryOptimizer. - Validated permission-based access control for various user roles, including owners, collaborators, strangers, and superusers. - Ensured thorough testing of extract visibility, relationship retrieval, and annotation summaries based on user permissions and document access. - Enhanced test coverage for edge cases, including handling of non-existent extracts and structural relationships.
Signed-off-by: JSIV <5049984+JSv4@users.noreply.github.com>
- Added time freezing to ensure consistent rate limit testing across corpuses, documents, and labelsets queries. - Improved assertions to include the number of requests made before hitting the rate limit, providing clearer feedback on test outcomes. - Refactored permission setting in query optimizer tests for better readability and consistency.
- Added failure messages to ensure clarity when expected rate limits are not hit. - Implemented time freezing in tests to maintain consistent request timing. - Enhanced assertions to validate that both heavy and light queries hit their respective rate limits within the expected window.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Substantial Changes: