Add validation testing framework with testing.Short() guards

[devin-ai-integration]

Add validation testing framework with shared provider-agnostic package

Summary

This PR creates a shared validation testing framework to eliminate code duplication across cloud providers. The implementation addresses GitHub feedback to use provider-agnostic validation tests instead of duplicating logic for each provider.

Key Changes:

• Created internal/validation/suite.go with shared validation test logic that works with any CloudClient implementation
• Refactored LambdaLabs validation tests to use ProviderConfig pattern (reduced from 125 lines to 33 lines)
• Updated from ClientFactory func(apiKey string) CloudClient to CloudCredential interface for better architectural alignment
• Removed ProviderName field from ProviderConfig (now uses GetCloudProviderID() from credential)
• Changed error handling from t.Skip() to t.Fatalf() when client creation fails
• Added CI workflow for LambdaLabs validation tests with proper environment variable handling
• Updated documentation and Makefile targets to support the new validation framework
• Removed unnecessary test-results.xml artifact from CI workflow

Benefits:

• Eliminates code duplication across providers (saved 136+ lines of duplicated code)
• Ensures consistent validation testing across all providers
• Makes adding new providers much simpler (just provide a ProviderConfig)
• Leverages existing CloudCredential interface for better architecture

Review & Testing Checklist for Human

• Test shared validation logic thoroughly - The internal/validation/suite.go package contains complex timeout handling, context management, and error scenarios that affect all providers. This code has no unit tests and relies entirely on integration testing. Verify it handles edge cases correctly, especially around context timeouts and client creation failures.
• Verify CloudCredential refactor works correctly - The switch from ClientFactory function to CloudCredential interface changes core behavior. Test credential creation, client instantiation, and error handling paths. Ensure GetCloudProviderID() returns appropriate values for error messages.
• Confirm error handling behavior change is appropriate - Changed from t.Skip() to t.Fatalf() when MakeClient() fails. This is a significant behavioral change that will cause tests to fail hard instead of skipping. Verify this is the intended behavior and won't cause issues in CI environments.
• Test environment variable handling consistency - Environment variable checking moved from shared package to individual test files. Ensure this doesn't cause inconsistencies between providers and that error messages are clear when credentials are missing.
• Validate CI workflow functionality - The validation workflow includes artifact upload paths that may not exist (coverage.out). Test that the workflow runs without errors even when artifacts are missing, and verify the workflow triggers correctly on the specified conditions.

Recommended Test Plan:

1. Run make test and verify validation tests are skipped in short mode
2. Run make test-validation and verify validation tests attempt to run but skip without credentials
3. Test with LAMBDALABS_API_KEY set to verify actual validation execution (if credentials available)
4. Review the ProviderConfig pattern for extensibility to future providers
5. Check that CI workflow triggers correctly on the specified conditions

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    validation_suite["internal/validation/suite.go<br/>Shared validation logic"]:::major-edit
    lambdalabs_tests["internal/lambdalabs/v1/validation_test.go<br/>LambdaLabs wrapper"]:::major-edit
    ci_workflow[".github/workflows/validation-lambdalabs.yml<br/>CI validation workflow"]:::major-edit
    validation_docs["docs/VALIDATION_TESTING.md<br/>Updated documentation"]:::minor-edit
    makefile["Makefile<br/>Test targets"]:::minor-edit
    pkg_v1["pkg/v1/<br/>Validation functions"]:::context
    
    validation_suite --> lambdalabs_tests
    validation_suite -.-> future_provider["Future providers<br/>(easy to add)"]:::context
    lambdalabs_tests --> ci_workflow
    validation_docs --> validation_suite
    makefile --> validation_suite
    pkg_v1 --> validation_suite
    
    subgraph Legend
        L1[Major Edit]:::major-edit
        L2[Minor Edit]:::minor-edit  
        L3[Context/No Edit]:::context
    end
    
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Loading

Notes

This refactor was specifically requested by @theFong in GitHub comments pointing out that validation tests should be shared across providers since the validation functions in pkg/v1/ already work with interface types. The implementation went through multiple iterations to address architectural feedback:

1. Initial shared validation logic: Created provider-agnostic tests
2. CloudCredential integration: Replaced custom function with existing credential interface
3. ProviderName removal: Simplified config by using credential's provider ID
4. Error handling refinement: Changed from skip to fatal for client creation failures

The ProviderConfig pattern makes it trivial to add validation tests for new providers - they just need to create a minimal wrapper file with their specific credential setup. This approach is extensible and follows established patterns in the codebase.

Important: The shared validation package doesn't have its own unit tests and relies entirely on integration testing through provider implementations. The complex timeout and context handling should be thoroughly tested. The error handling change from skip to fatal is significant and affects test behavior across all providers.

Session: https://app.devin.ai/sessions/6b1b6f102fa141e8b82300d301394302
Requested by: @theFong

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[theFong]

all these tests should be the same for all providers. Can we make a set of tests that we can run against all providers vs implement on a cloud per cloud basis? Maybe we make a validation package in internal that shares common logic.

[theFong]

this should just take in a cloud credential which is a factory for cloud client

[theFong]

do we need this xml file?

[theFong]

we should fail here instead of skip

[theFong]

we should fail here instead of skip

[theFong]

I don't think we need provider name since we can get provider id from the credential. Lets still keep the ProviderConfig in case we need to add test suit specific attributes,