Ultron 2.0

Ultron 2.0 is a prototype autograder system designed to automatically grade Android application (APK) submissions. It orchestrates the entire process, from setting up an environment to running tests and generating a final report.

This is the successor to Ultron Beginner, updated for compatibility with the new Direviu contract.

Modules

The project is divided into two main modules:

• app: A Kotlin-based command-line application that acts as the main orchestrator. It handles setting up the environment, running tests, and processing the results.
• tester: An Android project containing the instrumented tests that are executed against the submitted APK.

Prerequisites

To run the Ultron Autograder, you will need the following installed on your system:

• Java Development Kit (JDK) 11 or higher
• Android SDK with adb and aapt2 available in your system's PATH
• Docker

How to Run

You can run the autograder from the command line using Gradle.

1. Navigate to the app directory:

   cd app

2. Run the application using the run Gradle task, providing the path to the submission directory.

   ./gradlew run --args="-s /path/to/submission/directory"

Example:

1. Approved Sample

    ./gradlew run --args="-s FIXTURES/submission-approved -v"

   Output:

   [{"submission_id":2,"rating":3,"checklist_keys":["DISPLAY_INFO_IN_LIST_MIN_10ITEMS","DISPLAY_DETAIL_PAGE_AFTER_CLICKING_ITEM","DISPLAY_RELEVANT_INFO_IN_DETAIL","DISPLAY_ABOUT_PAGE"],"message":"Congratulations! You have passed the submission.","is_draft":true,"is_passed":true}]

2. Rejected Sample

    ./gradlew run --args="-s FIXTURES/submission-rejected -v"

   Output:

   [{"submission_id":4,"rating":0,"checklist_keys":["DISPLAY_INFO_IN_LIST_MIN_10ITEMS","DISPLAY_DETAIL_PAGE_AFTER_CLICKING_ITEM","DISPLAY_RELEVANT_INFO_IN_DETAIL"],"message":"Sorry, you have not passed the submission. Please check the following criteria: \nCriteria DISPLAY_ABOUT_PAGE not met","is_draft":true,"is_passed":false}]

Command-Line Arguments

• -s, --source: (Required) The absolute path to the directory containing the student's submission (.apk file and auto-review-config.json).
• -v, --verbose: (Optional) Enables verbose logging for all subprocess commands.

How It Works

The autograder follows these steps to grade a submission:

1. Initialization: Reads the submission information from submission.json.
2. Emulator Setup: Starts an Android emulator inside a Docker container using a pre-configured image. It waits until the emulator is fully booted and ready.
3. Testing:
   • Installs the submitted APK on the emulator.
   • Executes the instrumented tests defined in the tester module against the APK.
   • Pulls the test result XML file and any generated screenshots from the emulator.
4. Grading: Parses the XML test results to determine which criteria passed or failed.
5. Reporting: Generates a report.json file in the submission directory, containing the final grade, a list of completed checklist items, and feedback.
6. Cleanup: Stops the Docker container and the ADB server to release all resources.