Feature Request: Add Cross-Version Compatibility Check Task for Platform Subprojects

[Jaffe2718]

Background

Architectury plugin is widely used for multi-platform (Fabric/Forge/NeoForge/Quilt) mod development, but developers currently lack a unified way to check cross-version compatibility within the Gradle ecosystem. Existing solutions (online tools/manual testing) are fragmented, not integrated with the subproject-based compilation structure of Architectury, and cannot automatically verify class loading/runtime stability.

Proposed Solution

Add a checkCompatibility Gradle task at the platform subproject level ([fabric/neoforge/quilt]/build.gradle) with multi-profile support and configurable test items. This task should isolate the classpath for each profile, verify class existence/compilation, and optionally run client/server to test runtime class loading (with auto-created test worlds and graceful exit).

Core Design (You can adjust it according to the actual situation)

1. Example Configuration Syntax (Subproject build.gradle)

// Example: fabric/build.gradle
checkCompatibility {                // group = test
    failOnError = false             // Fail all tests if any fails
    
    // Multi-profile support (isolate different MC versions/platforms)
    profile("mc1.21-fabric") {
        failOnError = false             // Fail this profile's tests if any fails
        
        // Dependencies (reuse Gradle native syntax)
        dependencies {
            minecraft "com.mojang:minecraft:1.21"
            fabricApi "net.fabricmc.fabric-api:fabric-api:0.97.0+1.21"
            mappings "net.fabricmc:yarn:1.21+build.1:mergedv2"
            // Mod dependencies (reuse Gradle native syntax)
        }
        
        // Java configuration
        java {
            sourceCompatibility = JavaVersion.VERSION_21
            targetCompatibility = JavaVersion.VERSION_21
        }
        
        // Configurable test items (enable/disable per test)
        tests {
            // Test 1: Isolated compilation (check class existence)
            compileIsolated {
                reportFormat = ["html", "json"]
            }
            
            // Test 2: Runtime class loading (force load all mod classes)
            runtimeClassLoading {
                reportFormat = ["html", "json"]
                
                // Client configuration (auto-create test world + exit once world is loaded)
                client {
                    jvmArgs = [
                        "-Dfabric.api.logger.level=WARN",
                    ]
                    programArgs = []
                    timeout = 120 // Seconds
                }
                
                // Server configuration (auto-create world + exit once world is loaded)
                server {
                    programArgs = [
                        "--world", "TestServerWorld_${profileName}",
                        "--nogui",
                        "--port", "25565"
                    ]
                    timeout = 60 // Seconds
                }

            }
        }
        
        // Report configuration
        reports {
            dir = file("${project.buildDir}/reports/compatibility/${profileName}")
            mergeReports = true
        }
    }
    
    // Additional profile example
    profile("mc1.21.6-fabric") {
        // ... Similar configuration
    }
}

2. Key Test Capabilities (Configurable Enable/Disable)

Test Item	Purpose
compileIsolated	Isolate the profile's classpath (independent of the project's original classpath), attempt to compile the mod, and generate a report to check for missing classes/methods.
runtimeClassLoading	Use the subproject's classpath compiled JAR (from remapJar), then use profile to setup MC testing instance, force load all classes via ClassLoader (no lazy loading), optionally run client/server with auto-created test worlds, and verify runtime class accessibility.

3. Auto-Create Test World for client/server

• Setup Minecraft environment with the profile's dependencies, and because client/server will not auto exit, we need to inject System.exit(0) after world load.
• Client: Use MC's native --quickPlaySingleplayer <WorldName> parameter to auto-create a test world and enter it directly; inject System.exit(0) after world load to exit gracefully.
• Server: Use --world <WorldName> to auto-create a server world, start the server, and exit gracefully after startup by injecting System.exit(0) after world load.

4. Workflow Diagram (Mermaid)

flowchart TD
    A["gradle subproject:checkCompatibility"] --> B[Profile-0]
    A --> C[Profile-1]
    A --> D[Profile-2]
    A --> E["......"]
    
    B --> F["compileIsolated"]
    F --> G["Try using the isolated classpath to compile the mod"]
    G --> H["Report"]
    
    B --> I["runtimeClassLoading"]
    I --> J["compile the mod with the subproject's classpath"]
    J --> K["Execute the mod (client/server) with isolated classpath defined in the profile"]
    K --> H["Report"]

Loading

Expected Benefits

1. Subproject-First Design: Aligns with Architectury's platform-specific subproject structure (fabric/neoforge), avoiding root-project configuration clutter.
2. Isolated Testing: Each profile has independent dependencies/classpath, preventing conflicts with the project's original build environment.
3. Configurable Tests: Developers can enable/disable specific test items based on needs (e.g., skip runtime tests for fast compilation checks).
4. Automated Runtime Verification: Auto-create test worlds eliminate manual world setup, and forced class loading catches lazy-loading compatibility issues.
5. Unified Reporting: Generate structured HTML/JSON reports for both compilation and runtime issues, easy to debug.

Use Case

A developer working on an Architectury mod for Fabric/NeoForge can:

1. Add checkCompatibility profiles for MC 1.21 and 1.21.11 in fabric/neoforge subprojects.
2. Run ./gradlew fabric:checkCompatibility neoforge:checkCompatibility to verify cross-version compatibility.
3. Review reports to fix missing classes/methods for specific versions/platforms.