Skip to content

refactor(delphi): Remove legacy poller/system architecture and dead code #2343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jucor wants to merge 8 commits into
base: edge
Choose a base branch
from
Open

refactor(delphi): Remove legacy poller/system architecture and dead code #2343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cebd9d2
refactor: remove legacy poller/system architecture
jucor Jan 5, 2026
cfc1518
refactor: remove dead code and archive outdated docs
jucor Jan 5, 2026
4c82531
docs: update RUNNING_THE_SYSTEM.md and archive more outdated docs
jucor Jan 5, 2026
0cc52a6
docs: add dead code cleanup report with vulture analysis
jucor Jan 6, 2026
c56beb2
fix: restore incorrectly deleted general.py and remove unused imports
jucor Jan 7, 2026
645894d
docs: unify dead code cleanup reports
jucor Jan 7, 2026
2d625df
docs: add raw Vulture static analysis output
jucor Jan 7, 2026
4daa841
refactor: remove high-confidence unused imports (17 total)
jucor Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
434 changes: 434 additions & 0 deletions delphi/docs/DEAD_CODE_CLEANUP_REPORT.md
View file
Open in desktop

Large diffs are not rendered by default.

88 changes: 59 additions & 29 deletions delphi/docs/RUNNING_THE_SYSTEM.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,38 +98,61 @@ Test results for real data are saved to the `python_output` directory within eac

## Using the System

### Basic Usage
### Running the Full Pipeline

Here's a basic example of how to use the system in Python:
The recommended way to run Delphi is via the main orchestrator script:

```python
from polismath import SystemManager
from polismath.conversation import Conversation
```bash
# Run the full pipeline for a conversation
python run_delphi.py --zid=<CONVERSATION_ID>

# With options
python run_delphi.py --zid=12345 --include_moderation=true
```

# Start the system manager
system = SystemManager.start()
### Using the CLI

# Create a conversation manager
conv_manager = system.conversation_manager
For job queue-based execution:

```bash
# Interactive mode
./delphi

# Submit a job directly
./delphi submit --zid=12345

# Check job status
./delphi list
./delphi status 12345
```

# Create a new conversation
conv_id = "my-conversation"
conv = conv_manager.create_conversation(conv_id)
### Basic Python Usage

# Process votes
votes = [
{"pid": "participant1", "tid": "comment1", "vote": 1}, # Agree
{"pid": "participant1", "tid": "comment2", "vote": -1}, # Disagree
{"pid": "participant2", "tid": "comment1", "vote": 1}, # Agree
{"pid": "participant2", "tid": "comment3", "vote": 1}, # Agree
]
Here's how to use the core components directly in Python:

```python
from polismath.conversation.conversation import Conversation

# Create a conversation
conv = Conversation("my-conversation")

# Process votes (vote convention: +1=agree, -1=disagree, 0=pass)
votes = {
"votes": [
{"pid": 0, "tid": 0, "vote": 1}, # Agree
{"pid": 0, "tid": 1, "vote": -1}, # Disagree
{"pid": 1, "tid": 0, "vote": 1}, # Agree
{"pid": 1, "tid": 2, "vote": 1}, # Agree
]
}

# Update the conversation with votes
updated_conv = conv_manager.process_votes(conv_id, {"votes": votes})
conv.update_votes(votes)

# Access results
group_clusters = updated_conv.group_clusters
repness = updated_conv.repness
pca_results = conv.pca
group_clusters = conv.group_clusters
repness = conv.repness
```

### Loading Real Data
Expand Down Expand Up @@ -199,19 +222,26 @@ cd delphi/eda_notebooks

## Command-line Interface

The package includes a basic command-line interface:
The package provides several CLI entry points:

```bash
# Run with default settings
polismath
# Run the full Delphi pipeline
run-delphi --zid=12345

# Show help
polismath --help
# Run just the math pipeline (PCA/K-means/representativeness)
run-math-pipeline --zid=12345

# Run with custom settings
polismath --config config.yaml --port 8000 --log-level DEBUG
# Run just the UMAP/narrative pipeline
run-umap-pipeline --zid=12345

# Use the interactive job CLI
delphi
delphi submit --zid=12345
delphi list
```

See `pyproject.toml` for the full list of CLI entry points.

## Running the Simplified Test Scripts

The repository includes simplified versions of the core algorithms that can be run independently:
Expand Down
147 changes: 147 additions & 0 deletions delphi/docs/vulture_analysis_output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
polismath/benchmarks/bench_repness.py:29: unused import 'add_comparative_stats' (90% confidence)
polismath/benchmarks/bench_repness.py:29: unused import 'finalize_cmt_stats' (90% confidence)
polismath/benchmarks/bench_repness.py:29: unused import 'select_rep_comments' (90% confidence)
polismath/benchmarks/benchmark_utils.py:62: unused function 'run_benchmark' (60% confidence)
polismath/components/config.py:12: unused import 'Set' (90% confidence)
polismath/components/config.py:59: unused function 'to_bool' (60% confidence)
polismath/components/config.py:133: unused function 'get_env_value' (60% confidence)
polismath/components/config.py:403: unused method 'save_to_file' (60% confidence)
polismath/components/config.py:423: unused method 'load_from_file' (60% confidence)
polismath/components/config.py:444: unused class 'ConfigManager' (60% confidence)
polismath/components/config.py:452: unused method 'get_config' (60% confidence)
polismath/conversation/conversation.py:10: unused import 'Set' (90% confidence)
polismath/conversation/conversation.py:75: unused attribute 'subgroup_clusters' (60% confidence)
polismath/conversation/conversation.py:541: unused attribute 'subgroup_clusters' (60% confidence)
polismath/conversation/conversation.py:571: unused attribute 'subgroup_clusters' (60% confidence)
polismath/conversation/conversation.py:949: unused method '_compute_votes_base' (60% confidence)
polismath/conversation/conversation.py:1081: unused method '_compute_user_vote_counts' (60% confidence)
polismath/conversation/conversation.py:1560: unused method '_convert_to_clojure_format' (60% confidence)
polismath/conversation/conversation.py:1717: unused method '_reset_conversion_cache' (60% confidence)
polismath/conversation/manager.py:10: unused import 'Set' (90% confidence)
polismath/conversation/manager.py:26: unused class 'ConversationManager' (60% confidence)
polismath/conversation/manager.py:151: unused method 'process_votes' (60% confidence)
polismath/conversation/manager.py:256: unused method 'export_conversation' (60% confidence)
polismath/conversation/manager.py:287: unused method 'import_conversation' (60% confidence)
polismath/conversation/manager.py:324: unused method 'delete_conversation' (60% confidence)
polismath/database/dynamodb.py:518: unused variable 'last_log_time' (60% confidence)
polismath/database/dynamodb.py:559: unused variable 'last_log_time' (60% confidence)
polismath/database/dynamodb.py:582: unused method 'write_projections_separately' (60% confidence)
polismath/database/dynamodb.py:770: unused method 'read_latest_math' (60% confidence)
polismath/database/postgres.py:13: unused import 'Set' (90% confidence)
polismath/database/postgres.py:383: unused method 'get_zinvite_from_zid' (60% confidence)
polismath/database/postgres.py:419: unused method 'poll_votes' (60% confidence)
polismath/database/postgres.py:471: unused method 'poll_moderation' (60% confidence)
polismath/database/postgres.py:550: unused method 'load_math_main' (60% confidence)
polismath/database/postgres.py:582: unused method 'write_math_main' (60% confidence)
polismath/database/postgres.py:629: unused method 'write_participant_stats' (60% confidence)
polismath/database/postgres.py:655: unused method 'write_correlation_matrix' (60% confidence)
polismath/database/postgres.py:684: unused method 'increment_math_tick' (60% confidence)
polismath/database/postgres.py:718: unused method 'poll_tasks' (60% confidence)
polismath/database/postgres.py:761: unused method 'mark_task_complete' (60% confidence)
polismath/database/postgres.py:784: unused method 'create_task' (60% confidence)
polismath/database/postgres.py:812: unused class 'PostgresManager' (60% confidence)
polismath/database/postgres.py:821: unused method 'get_client' (60% confidence)
polismath/pca_kmeans_rep/clusters.py:462: unused function 'silhouette' (60% confidence)
polismath/pca_kmeans_rep/corr.py:257: unused function 'save_correlation_to_json' (60% confidence)
polismath/pca_kmeans_rep/corr.py:327: unused function 'participant_correlation_matrix' (60% confidence)
polismath/pca_kmeans_rep/pca.py:28: unused function 'vector_length' (60% confidence)
polismath/pca_kmeans_rep/pca.py:147: unused variable 'last_vector' (60% confidence)
polismath/pca_kmeans_rep/pca.py:197: unused variable 'last_vector' (60% confidence)
polismath/pca_kmeans_rep/repness.py:51: unused function 'z_score_sig_95' (60% confidence)
polismath/pca_kmeans_rep/repness.py:157: unused function 'add_comparative_stats' (60% confidence)
polismath/pca_kmeans_rep/repness.py:213: unused function 'finalize_cmt_stats' (60% confidence)
polismath/pca_kmeans_rep/repness.py:315: unused function 'select_rep_comments' (60% confidence)
polismath/pca_kmeans_rep/repness.py:395: unused function 'calculate_kl_divergence' (60% confidence)
polismath/pca_kmeans_rep/repness.py:413: unused function 'select_consensus_comments' (60% confidence)
polismath/pca_kmeans_rep/stats.py:82: unused function 'z_sig_90' (60% confidence)
polismath/pca_kmeans_rep/stats.py:95: unused function 'z_sig_95' (60% confidence)
polismath/pca_kmeans_rep/stats.py:108: unused function 'shannon_entropy' (60% confidence)
polismath/pca_kmeans_rep/stats.py:123: unused function 'gini_coefficient' (60% confidence)
polismath/pca_kmeans_rep/stats.py:159: unused function 'weighted_stddev' (60% confidence)
polismath/pca_kmeans_rep/stats.py:186: unused function 'ci_95' (60% confidence)
polismath/pca_kmeans_rep/stats.py:215: unused function 'bayesian_ci_95' (60% confidence)
polismath/pca_kmeans_rep/stats.py:240: unused function 'bootstrap_ci_95' (60% confidence)
polismath/pca_kmeans_rep/stats.py:275: unused function 'binomial_test' (60% confidence)
polismath/pca_kmeans_rep/stats.py:323: unused function 'fisher_exact_test' (60% confidence)
polismath/regression/comparer.py:700: unused method 'generate_report' (60% confidence)
polismath/regression/datasets.py:186: unused function 'find_dataset_file' (60% confidence)
polismath/run_math_pipeline.py:73: unused function 'fetch_votes' (60% confidence)
polismath/utils/general.py:8: unused import 'itertools' (90% confidence)
polismath/utils/general.py:10: unused import 'Set' (90% confidence)
polismath/utils/general.py:17: unused variable 'PASS' (60% confidence)
polismath/utils/general.py:44: unused function 'delphi_vote_to_postgres' (60% confidence)
polismath/utils/general.py:60: unused function 'xor' (60% confidence)
polismath/utils/general.py:74: unused function 'round_to' (60% confidence)
polismath/utils/general.py:88: unused function 'zip_collections' (60% confidence)
polismath/utils/general.py:102: unused function 'with_indices' (60% confidence)
polismath/utils/general.py:115: unused function 'filter_by_index' (60% confidence)
polismath/utils/general.py:153: unused function 'mapv_rest' (60% confidence)
polismath/utils/general.py:167: unused function 'typed_indexof' (60% confidence)
polismath/utils/general.py:184: unused function 'hash_map_subset' (60% confidence)
polismath/utils/general.py:198: unused function 'distinct' (60% confidence)
polismath/utils/general.py:237: unused function 'weighted_means' (60% confidence)
scripts/compare_implementations.py:41: unused function 'compare_numerical_values' (60% confidence)
scripts/compare_implementations.py:528: unused variable 'use_manual_pipeline' (100% confidence)
scripts/delphi_cli.py:23: unused import 'Text' (90% confidence)
scripts/delphi_cli.py:24: unused import 'rprint' (90% confidence)
scripts/job_poller.py:395: unused variable 'frame' (100% confidence)
scripts/job_poller.py:395: unused variable 'sig' (100% confidence)
setup_minio.py:50: unused variable 'bucket_exists' (60% confidence)
setup_minio.py:54: unused variable 'bucket_exists' (60% confidence)
umap_narrative/500_generate_embedding_umap_cluster.py:283: unused function 'generate_basic_cluster_labels' (60% confidence)
umap_narrative/701_static_datamapplot_for_layer.py:423: unused variable 'local_dir' (60% confidence)
umap_narrative/701_static_datamapplot_for_layer.py:486: unused variable 'static_html' (60% confidence)
umap_narrative/801_narrative_report_batch.py:40: unused import 'csv' (90% confidence)
umap_narrative/801_narrative_report_batch.py:41: unused import 'io' (90% confidence)
umap_narrative/801_narrative_report_batch.py:124: unused method 'get_report' (60% confidence)
umap_narrative/801_narrative_report_batch.py:1105: unused method 'process_request' (60% confidence)
umap_narrative/803_check_batch_status.py:30: unused variable 'TERMINAL_BATCH_STATES' (60% confidence)
umap_narrative/803_check_batch_status.py:192: unused method 'check_and_process_jobs' (60% confidence)
umap_narrative/llm_factory_constructor/model_provider.py:64: unused attribute 'api_base' (60% confidence)
umap_narrative/llm_factory_constructor/model_provider.py:296: unused method 'get_batch_responses' (60% confidence)
umap_narrative/polismath_commentgraph/core/clustering.py:6: unused import 'hdbscan' (90% confidence)
umap_narrative/polismath_commentgraph/core/clustering.py:14: unused import 'delayed' (90% confidence)
umap_narrative/polismath_commentgraph/core/clustering.py:14: unused import 'Parallel' (90% confidence)
umap_narrative/polismath_commentgraph/core/clustering.py:131: unused method 'evoc_cluster' (60% confidence)
umap_narrative/polismath_commentgraph/core/clustering.py:223: unused method 'analyze_cluster' (60% confidence)
umap_narrative/polismath_commentgraph/core/embedding.py:169: unused method 'calculate_similarity' (60% confidence)
umap_narrative/polismath_commentgraph/core/embedding.py:229: unused method 'find_nearest_neighbors' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:34: unused variable 'y' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:40: unused variable 'dimensions' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:57: unused variable 'umap_parameters' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:58: unused variable 'evoc_parameters' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:77: unused variable 'is_outlier' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:79: unused variable 'layer0_cluster_id' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:80: unused variable 'layer1_cluster_id' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:81: unused variable 'layer2_cluster_id' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:82: unused variable 'layer3_cluster_id' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:83: unused variable 'layer4_cluster_id' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:84: unused variable 'distance_to_centroid' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:85: unused variable 'cluster_confidence' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:97: unused variable 'centroid_coordinates' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:116: unused variable 'shared_cluster_layers' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:131: unused function 'create_cluster_key' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:147: unused function 'create_topic_key' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:163: unused variable 'created_at' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:165: unused function 'create_topic_key' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:177: unused variable 'body' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:180: unused variable 'agree_vote_count' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:181: unused variable 'disagree_vote_count' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:182: unused variable 'pass_vote_count' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:186: unused class 'CommentMetadata' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:188: unused variable 'is_seed' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:189: unused variable 'is_moderated' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:190: unused variable 'moderation_status' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:215: unused variable 'confidence_scores' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:228: unused variable 'similar_comments' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:229: unused variable 'predicted_clusters' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:240: unused class 'CommentExtremity' (60% confidence)
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py:246: unused variable 'calculation_timestamp' (60% confidence)
umap_narrative/polismath_commentgraph/utils/converter.py:544: unused method 'batch_convert_enhanced_topic_names' (60% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:42: unused import 'QueuePool' (90% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:518: unused method 'list_conversations' (60% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:580: unused method 'get_comment_embedding' (60% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:1047: unused method 'get_cluster_characteristics_by_layer' (60% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:1123: unused method 'batch_create_enhanced_topic_names' (60% confidence)
umap_narrative/polismath_commentgraph/utils/storage.py:1414: unused method 'get_visualization_data' (60% confidence)
umap_narrative/run_pipeline.py:753: unused function 'create_static_datamapplot' (60% confidence)
1 change: 0 additions & 1 deletion delphi/polismath/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,4 @@

__version__ = '0.1.0'

from polismath.system import System, SystemManager
from polismath.components.config import Config, ConfigManager
Loading
Loading