Rewrite and modernize test suite for current architecture

[aarora79]

Summary

The existing test suite in the tests/ folder needs to be completely rewritten to align with the current architecture, new authentication system, and recent feature additions. Most tests are outdated, use old authentication mocks, and don't cover new functionality.

Current State Assessment

Existing Test Structure

tests/
├── conftest.py                              # Shared fixtures
├── integration/
│   └── test_server_routes.py               # Integration tests
├── unit/
│   ├── auth/
│   │   ├── test_auth_dependencies.py       # Auth dependency tests
│   │   └── test_auth_routes.py             # Auth route tests
│   ├── servers/
│   │   └── test_server_service.py          # Server service tests
│   ├── services/
│   │   └── test_access_control_service.py  # Access control tests
│   ├── search/
│   │   └── test_faiss_service.py           # FAISS search tests
│   ├── health/
│   │   ├── test_health_service.py          # Health monitoring tests
│   │   └── test_health_routes.py           # Health route tests
│   └── core/
│       ├── test_config.py                  # Config tests
│       ├── test_main.py                    # Main app tests
│       └── test_nginx_service.py           # Nginx service tests
└── fixtures/
    └── factories.py                         # Test data factories

Problems with Current Tests

1. Outdated Authentication Mocking

• Tests use old session-based auth mocks (mock_authenticated_user)
• Don't account for new Keycloak/OAuth integration
• Missing M2M authentication testing
• No testing for enhanced_auth dependency with user context
• No testing for JWT token validation
• Missing group-based authorization tests

2. Missing Coverage for New Features

Recent features with NO test coverage:

• CLI tools (cli/mcp_client.py, cli/service_mgmt.sh)
• Scopes management (registry/utils/scopes_manager.py)
• New internal API endpoints (/internal/register, /internal/remove, etc.)
• MCP Gateway server tools (servers/mcpgw/server.py)
• Token generation endpoints (/tokens/generate)
• Well-known URL endpoint (/.well-known/mcp-servers)
• Scopes group management (add_server_to_groups, remove_server_from_groups)

3. Incomplete Service Testing

• Health service tests don't cover WebSocket connections
• Server service tests don't cover permission filtering
• No tests for get_filtered_servers() and get_all_servers_with_permissions()
• Missing tests for user_can_access_server_path()
• No tests for FAISS service initialization and indexing

4. No Integration Testing for Real Flows

Missing end-to-end tests for:

• Complete server registration flow (register → scopes update → FAISS index → health check)
• Server deletion flow (remove → cleanup scopes → cleanup FAISS)
• User authentication flow (Keycloak OAuth → session creation → permission loading)
• Agent M2M authentication flow (client credentials → JWT → API access)
• Health monitoring cycle (periodic checks → WebSocket updates → status persistence)

5. Test Infrastructure Issues

• conftest.py mocks are outdated
• Missing pytest fixtures for Keycloak/OAuth mocking
• No fixtures for M2M authentication
• Missing fixtures for user context with groups/scopes
• No test data for tool lists and server metadata

Required Changes

1. Update Authentication Test Infrastructure

Create new fixtures in conftest.py:

@pytest.fixture
def mock_keycloak_user_context():
    """Mock user context from Keycloak authentication."""
    return {
        "username": "testuser",
        "is_admin": False,
        "groups": ["mcp-servers-unrestricted"],
        "scopes": [
            "mcp-servers-unrestricted/read",
            "mcp-servers-unrestricted/execute"
        ],
        "accessible_servers": ["currenttime", "mcpgw"],
        "accessible_services": ["all"],
        "ui_permissions": {
            "toggle_service": ["all"],
            "modify_service": ["all"],
            "register_service": ["all"],
            "health_check_service": ["all"]
        }
    }

@pytest.fixture
def mock_admin_user_context():
    """Mock admin user context."""
    return {
        "username": "admin",
        "is_admin": True,
        "groups": ["mcp-servers-unrestricted", "admins"],
        "scopes": ["mcp-servers-unrestricted/read", "mcp-servers-unrestricted/execute"],
        "accessible_servers": ["all"],
        "accessible_services": ["all"],
        "ui_permissions": {
            "toggle_service": ["all"],
            "modify_service": ["all"],
            "register_service": ["all"],
            "health_check_service": ["all"]
        }
    }

@pytest.fixture
def mock_m2m_token():
    """Mock M2M JWT token for agent authentication."""
    import jwt
    from datetime import datetime, timedelta
    
    payload = {
        "sub": "agent-test-m2m",
        "scope": "mcp-servers-unrestricted/read mcp-servers-unrestricted/execute",
        "exp": datetime.utcnow() + timedelta(hours=1),
        "iat": datetime.utcnow(),
        "client_id": "agent-test-m2m"
    }
    return jwt.encode(payload, "test-secret", algorithm="HS256")

@pytest.fixture
def mock_enhanced_auth(monkeypatch, mock_keycloak_user_context):
    """Mock enhanced_auth dependency."""
    def mock_auth(session=None, authorization=None):
        return mock_keycloak_user_context
    
    monkeypatch.setattr("registry.auth.dependencies.enhanced_auth", mock_auth)
    return mock_auth

2. Add Tests for New Features

Scopes Manager Tests (tests/unit/utils/test_scopes_manager.py)

- test_add_server_to_scopes_unrestricted_only()
- test_add_server_to_groups_custom()
- test_remove_server_from_scopes()
- test_remove_server_from_groups()
- test_update_server_scopes()
- test_trigger_auth_server_reload()
- test_read_scopes_file()
- test_write_scopes_file()

Internal API Tests (tests/integration/test_internal_api.py)

- test_internal_register_with_auth()
- test_internal_register_without_auth()
- test_internal_remove_with_auth()
- test_internal_toggle_with_auth()
- test_internal_healthcheck_with_auth()
- test_internal_add_to_groups()
- test_internal_remove_from_groups()
- test_internal_list_services()

MCP Gateway Server Tests (tests/unit/servers/test_mcpgw_server.py)

- test_list_services_tool()
- test_healthcheck_services_tool()
- test_register_service_tool()
- test_remove_service_tool()
- test_toggle_service_tool()
- test_add_server_to_scopes_groups_tool()
- test_remove_server_from_scopes_groups_tool()
- test_intelligent_tool_finder()

CLI Tests (tests/unit/cli/test_mcp_client.py)

- test_ping_command()
- test_list_command()
- test_call_command()
- test_m2m_authentication()
- test_error_handling()
- test_json_output_format()

3. Update Existing Tests

Server Routes Tests

Update tests/integration/test_server_routes.py:

• Replace mock_authenticated_user with mock_enhanced_auth
• Add permission-based access tests
• Add tests for filtered server listings
• Add tests for UI permission checks

Server Service Tests

Update tests/unit/servers/test_server_service.py:

• Add tests for get_filtered_servers()
• Add tests for get_all_servers_with_permissions()
• Add tests for user_can_access_server_path()
• Add tests for remove_server()
• Update existing tests to use realistic server data

Auth Tests

Update tests/unit/auth/test_auth_dependencies.py:

• Add tests for Keycloak OAuth flow
• Add tests for M2M JWT validation
• Add tests for enhanced_auth dependency
• Add tests for permission extraction from groups/scopes
• Add tests for UI permission checking

4. Add Integration Tests

Create tests/integration/test_server_lifecycle.py:

- test_complete_server_registration_flow()
  * Register server
  * Verify in server list
  * Verify in scopes.yml
  * Verify in FAISS index
  * Verify health check runs

- test_complete_server_deletion_flow()
  * Delete server
  * Verify removed from server list
  * Verify removed from scopes.yml
  * Verify removed from FAISS index

- test_server_toggle_with_health_check()
  * Toggle server on
  * Verify immediate health check
  * Verify FAISS metadata updated
  * Toggle server off
  * Verify status changed

Create tests/integration/test_authentication_flow.py:

- test_keycloak_oauth_login_flow()
- test_m2m_token_authentication()
- test_permission_based_access_control()
- test_group_based_authorization()

5. Add E2E Tests

Create tests/e2e/test_complete_workflows.py:

- test_new_server_registration_to_first_call()
  * Admin registers new server
  * Server appears in registry UI
  * Health check completes successfully
  * User can discover server via search
  * User can call server tools

- test_user_permission_restrictions()
  * Restricted user logs in
  * Can only see permitted servers
  * Cannot toggle/modify restricted servers
  * Can toggle/modify permitted servers

Testing Standards

Required Test Coverage

• Minimum: 80% code coverage overall
• Critical paths: 95% coverage (auth, server registration, health checks)
• New features: 90% coverage minimum

Test Organization

• Use pytest markers: @pytest.mark.unit, @pytest.mark.integration, @pytest.mark.e2e
• Follow AAA pattern: Arrange, Act, Assert
• One assertion per test when possible
• Clear, descriptive test names

Test Data

• Use factories for generating test data (update factories.py)
• Create realistic server metadata
• Use actual tool schemas from real servers
• Mock external services (Keycloak, FAISS, MCP servers)

Implementation Plan

Phase 1: Infrastructure (High Priority)

• Update conftest.py with new authentication fixtures
• Add Keycloak/OAuth mocking utilities
• Update factories.py with new data generators
• Set up pytest configuration for markers

Phase 2: Critical Path Tests (High Priority)

• Rewrite authentication tests
• Rewrite server service tests
• Add scopes manager tests
• Add internal API tests

Phase 3: New Feature Tests (Medium Priority)

• CLI tools tests
• MCP Gateway server tests
• Token generation tests
• Well-known URL tests

Phase 4: Integration Tests (Medium Priority)

• Server lifecycle integration tests
• Authentication flow tests
• Health monitoring tests

Phase 5: E2E Tests (Lower Priority)

• Complete workflow tests
• Permission restriction tests
• Multi-user scenario tests

Acceptance Criteria

• All tests pass with updated architecture
• 80%+ code coverage achieved
• All new features have test coverage
• CI/CD integration working (GitHub Actions)
• Test documentation updated
• No deprecated mocks or fixtures
• pytest runs cleanly without warnings

Resources

Testing Tools

All required testing dependencies are in pyproject.toml:

[project.optional-dependencies]
dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.23.0",
    "pytest-cov>=4.1.0",
    "pytest-mock>=3.12.0",
    "pytest-xdist>=3.5.0",
    "coverage[toml]>=7.4.0",
    "httpx>=0.27.0",
    "pytest-html>=4.1.1",
    "pytest-json-report>=1.5.0",
    "factory-boy>=3.3.0",
    "faker>=24.0.0",
    "freezegun>=1.4.0",
]

Installation

# Install dev dependencies
uv sync --extra dev

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=registry --cov-report=html

# Run specific test categories
uv run pytest -m unit
uv run pytest -m integration
uv run pytest -m e2e

Documentation

• pytest documentation
• pytest-asyncio
• Factory Boy
• Coverage.py

Related Issues

• This work should be done after CLI Tool for MCP Server Registration and Health Validation (#120) #143 (CLI tools) is merged
• Relates to authentication improvements and new feature additions
• Supports CI/CD workflow improvements

Notes

• Tests should be runnable both locally and in CI/CD
• Mock external dependencies (Keycloak, FAISS, MCP servers)
• Use environment variables for test configuration
• Keep test data fixtures in version control
• Document any test-specific environment setup