Skip to content

Commit 9b58c13

Browse files
authored
Feature/testing pipelines
- Fix pipeline - Updated tests
1 parent 3bc29c5 commit 9b58c13

33 files changed

+18546
-1835
lines changed
Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
# Pipeline Design Philosophy
2+
3+
## The Problem with Inline Tests in CI/CD
4+
5+
### ❌ What Was Wrong
6+
7+
The original pipeline had several anti-patterns:
8+
9+
#### 1. **Inline Test Code in YAML**
10+
11+
```yaml
12+
# BAD: Embedding test logic in pipeline
13+
run: |
14+
cat > test-claude-detection.js << 'EOF'
15+
const { exec } = require('child_process');
16+
// ... complex test logic here
17+
EOF
18+
node test-claude-detection.js
19+
```
20+
21+
**Problems**:
22+
23+
- Test logic is not version controlled properly
24+
- No IDE support for the embedded code
25+
- Hard to debug and maintain
26+
- Cannot be run locally for development
27+
- No proper error handling or logging
28+
- Duplicates test logic across pipeline steps
29+
30+
#### 2. **Redundant Testing**
31+
32+
```yaml
33+
# BAD: Testing the same thing multiple times
34+
- name: "Test A"
35+
- name: "Test B that does the same as A"
36+
- name: "Test C that also does the same"
37+
```
38+
39+
**Problems**:
40+
41+
- Wastes CI/CD time and resources
42+
- Creates confusion about what's actually being tested
43+
- Makes failures harder to diagnose
44+
45+
#### 3. **Poor Separation of Concerns**
46+
47+
```yaml
48+
# BAD: Mixing infrastructure and test logic
49+
run: |
50+
# Setup stuff
51+
export DISPLAY=:99
52+
# Test stuff embedded here
53+
# More setup
54+
# More test stuff
55+
```
56+
57+
**Problems**:
58+
59+
- Infrastructure concerns mixed with test logic
60+
- Hard to understand what each step does
61+
- Difficult to reuse or modify
62+
63+
### ✅ The Correct Approach
64+
65+
#### 1. **Tests in Codebase, Pipeline Runs Tests**
66+
67+
```yaml
68+
# GOOD: Pipeline just orchestrates, tests are in codebase
69+
- name: Run Without Claude CLI tests
70+
run: npm run test:ci:without-claude-cli
71+
```
72+
73+
**Benefits**:
74+
75+
- All test logic is in the codebase
76+
- Can be run locally for debugging
77+
- Proper version control and IDE support
78+
- Clear separation of concerns
79+
- Reusable across different CI systems
80+
81+
#### 2. **Dedicated Test Scripts**
82+
83+
```javascript
84+
// GOOD: Proper test file with full functionality
85+
// scripts/test-claude-detection.js
86+
class ClaudeDetectionTester {
87+
async runAllTests() {
88+
// Comprehensive, well-structured test logic
89+
}
90+
}
91+
```
92+
93+
**Benefits**:
94+
95+
- Full programming language features
96+
- Proper error handling and logging
97+
- Can be unit tested itself
98+
- Clear documentation and comments
99+
100+
#### 3. **Clear Pipeline Responsibilities**
101+
102+
**Pipeline Responsibilities**:
103+
104+
- Environment setup (Docker, dependencies)
105+
- Artifact management (build, upload, download)
106+
- Test orchestration (run test commands)
107+
- Result reporting (success/failure, summaries)
108+
109+
**Test Code Responsibilities**:
110+
111+
- Actual testing logic and assertions
112+
- Error handling and reporting
113+
- Test data management
114+
- Mock setup and teardown
115+
116+
## Our Two-Stage Testing Strategy
117+
118+
### Without Claude CLI: Detection Tests
119+
120+
```bash
121+
# What it runs
122+
npm run test:ci:without-claude-cli
123+
124+
# What that includes
125+
npm run test:unit # Unit tests
126+
npm run test:main-window # VS Code extension test
127+
npm run test:claude-detection # CLI detection logic
128+
```
129+
130+
**Purpose**: Verify the extension handles missing Claude CLI gracefully
131+
132+
### With Claude CLI: Integration Tests
133+
134+
```bash
135+
# What it runs
136+
npm run test:ci:with-claude-cli
137+
138+
# What that includes
139+
npm run test:ci:without-claude-cli # All Without Claude CLI tests
140+
npm run test:e2e # End-to-end workflows
141+
npm run test:integration # Integration tests
142+
```
143+
144+
**Purpose**: Verify full functionality when Claude CLI is available
145+
146+
## Why This Design is Better
147+
148+
### 🏗️ **Maintainability**
149+
150+
- Test logic is in proper source files
151+
- Can be modified with IDE support
152+
- Version controlled like other code
153+
- Can be refactored and improved
154+
155+
### 🧪 **Testability**
156+
157+
- Tests can be run locally during development
158+
- Easy to debug when they fail
159+
- Can add more tests without touching pipeline
160+
- Test the tests themselves
161+
162+
### 🔄 **Reusability**
163+
164+
- Same tests work on different CI systems
165+
- Developers can run the same tests locally
166+
- Docker containers can use the same test commands
167+
- Easy to create new test combinations
168+
169+
### 📊 **Clarity**
170+
171+
- Pipeline shows high-level flow
172+
- Test details are in appropriate files
173+
- Clear separation between infrastructure and logic
174+
- Easy to understand what each phase does
175+
176+
### **Performance**
177+
178+
- No redundant testing
179+
- Tests can be optimized independently
180+
- Better caching and parallelization
181+
- Faster feedback loops
182+
183+
## Test Organization
184+
185+
```
186+
├── .github/workflows/ # CI/CD orchestration only
187+
│ ├── test-pipeline.yml # Main 2-stage pipeline
188+
│ └── docker-e2e.yml # Docker-based testing
189+
├── scripts/ # Utility test scripts
190+
│ └── test-claude-detection.js
191+
├── tests/ # Test suites
192+
│ ├── e2e/ # End-to-end tests
193+
│ └── integration/ # Integration tests
194+
├── src/test/ # VS Code extension tests
195+
│ └── suite/
196+
└── package.json # Test command definitions
197+
```
198+
199+
## Commands and Their Purpose
200+
201+
### Local Development
202+
203+
```bash
204+
npm run test:claude-detection # Test CLI detection logic
205+
npm run test:main-window # Test VS Code integration
206+
npm run test:unit # Test individual functions
207+
```
208+
209+
### CI Simulation
210+
211+
```bash
212+
npm run test:ci:without-claude-cli # Simulate Without Claude CLI
213+
npm run test:ci:with-claude-cli # Simulate With Claude CLI
214+
```
215+
216+
### Individual Categories
217+
218+
```bash
219+
npm run test:e2e # End-to-end workflows
220+
npm run test:integration # Service integration
221+
npm run test:all:coverage # Full coverage report
222+
```
223+
224+
This design ensures that:
225+
226+
1. **Pipeline focuses on orchestration**, not test implementation
227+
2. **Tests are proper code** with full language features
228+
3. **Local development** mirrors CI/CD exactly
229+
4. **Debugging is easy** when tests fail
230+
5. **Maintenance is simple** with standard code practices

0 commit comments

Comments
 (0)