Add tree cache to ASTAnalyzer and optimize vector projection #196
Conversation
Introduces a tree cache to ASTAnalyzer to avoid re-parsing unchanged files, improving performance for repeated analysis. Enhances vector projection in scripts/ingest/vectors.py by using numpy when available for significant speedup, with a fallback to pure Python. Updates Qdrant upsert logic to support async/sync modes via parameter or environment variable, and adds a flush_upserts utility to ensure data persistence after async upserts.
🤖 Augment PR SummarySummary: Performance-focused update to reuse parsed ASTs, speed up mini-vector projection, and optionally run Qdrant upserts asynchronously. Changes:
Technical Notes: Default behavior remains synchronous unless async mode is enabled; projection matrices are cached and NumPy uses vectorized matmul + L2 normalization. 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
scripts/ast_analyzer.py
Outdated
| path = Path(file_path) if file_path else None | ||
|
|
||
| # Try to get cached tree (only for real files, not in-memory content) | ||
| if self._tree_cache and path and path.exists(): |
There was a problem hiding this comment.
_parse_with_cache can return a cached tree even when analyze_file(..., content=...) is called with explicit content for an on-disk file_path, which risks analyzing a stale tree if the provided content differs from the file on disk.
🤖 Was this useful? React with 👍 or 👎
| return | ||
| try: | ||
| # Force a sync operation to ensure all pending writes are flushed | ||
| client.get_collection(collection) |
There was a problem hiding this comment.
flush_upserts() calls get_collection(), but Qdrant’s wait=False semantics are “confirmed received” rather than “applied”, so this may not actually guarantee prior async upserts are committed before reads.
🤖 Was this useful? React with 👍 or 👎
| [scale * (1.0 if rnd.random() < 0.5 else -1.0) for _ in range(out_dim)] | ||
| for _ in range(in_dim) | ||
| ] | ||
| M = np.array(M_list, dtype=np.float32) |
There was a problem hiding this comment.
The NumPy path uses float32 for both the projection matrix and input vector, so results will differ from the pure-Python (float64) path despite the “reproducibility” comment; if determinism across environments matters, this could be surprising.
🤖 Was this useful? React with 👍 or 👎
Refines ASTAnalyzer to avoid returning stale trees when content is provided in-memory, by tracking content provenance and skipping cache in such cases. Enhances flush_upserts in Qdrant ingest to clarify consistency semantics and add a minimal scroll for better write visibility. Adds comprehensive tests for chunk deduplication, ingest infrastructure, and tree cache integration to ensure correctness and robustness.
Introduces a tree cache to ASTAnalyzer to avoid re-parsing unchanged files, improving performance for repeated analysis. Enhances vector projection in scripts/ingest/vectors.py by using numpy when available for significant speedup, with a fallback to pure Python. Updates Qdrant upsert logic to support async/sync modes via parameter or environment variable, and adds a flush_upserts utility to ensure data persistence after async upserts.