Build Intelligent Visual Automations with Java

Why Brobot • Quick Start • Features • Use Cases • Documentation • Contributing

---

🤖 Why Brobot?

Brobot is a powerful Java framework that combines SikuliX and OpenCV to create intelligent visual automations. Unlike traditional GUI automation tools, Brobot uses a state-based approach that makes your automations resilient, maintainable, and truly intelligent.

🎯 Perfect for:

• 🎮 Game Automation - Build bots that can play games autonomously
• 🧪 Visual Testing - Create robust image-based test suites
• 🔄 Process Automation - Automate repetitive visual tasks
• 🔬 Research - Develop visual APIs for AI/ML experiments

💡 What makes Brobot different?

// Traditional automation: brittle and sequential
click(100, 200);
wait(2);
type("username");
click(100, 250);
wait(2);
type("password");

// Brobot: intelligent and state-aware
stateNavigator.openState("login");
// Brobot automatically finds the best path, handles errors, and recovers from failures

🚀 Quick Start

Installation

🎉 Version 1.1.0 is now available on Maven Central! This is the final major release of Brobot with comprehensive documentation, enhanced CI/CD support, and full cross-platform compatibility.

Maven

<dependency>
    <groupId>io.github.jspinak</groupId>
    <artifactId>brobot</artifactId>
    <version>1.1.0</version>
</dependency>

Gradle

implementation 'io.github.jspinak:brobot:1.1.0'

Your First Automation

@Component
public class SimpleAutomation {
    
    @Autowired
    private Action action;
    
    public void actOnButton() {
        // Define what to look for
        StateImage button = new StateImage.Builder()
                .setName("button.png")
                .build();
        
        // Configure the search
        PatternFindOptions findOptions = new PatternFindOptions.Builder()
                .setSimilarity(0.7)
                .build();
        
        // Find 
        ActionResult findResult = action.perform(findOptions, button);

        if (findResult.isSuccess()) {
        // Get the first match
        Match firstMatch = findResult.getFirstMatch();
    
        // Get all matches
        List<Match> allMatches = findResult.getMatchList();
    
        // Work with each match
        for (Match match : allMatches) {
            // Highlight each found instance
            action.perform(ActionType.HIGHLIGHT, match.getRegion());
        
            // Click each one
            action.perform(ActionType.CLICK, match.getRegion());
    }
}

📖 View Full Quickstart Guide →

✨ Features

🎯 State-Based Architecture

Model your application as states and transitions, not just sequences of clicks. Brobot automatically handles navigation, error recovery, and dynamic UI changes.

🔍 Advanced Image Recognition

• Multi-pattern matching with similarity thresholds
• Color-based detection for dynamic content
• Motion tracking for moving objects
• OCR support for text recognition

🛡️ Built-in Resilience

• Automatic error recovery - Brobot recalculates paths when things go wrong
• Dynamic state detection - Knows where it is at all times
• Smart waiting - No more arbitrary sleep() calls

🧪 Testable Automations

Unlike traditional GUI automation, Brobot automations are fully testable:

// Test your automation logic without running the GUI
BrobotSettings.mock = true;
// Your tests run with simulated responses

🔧 Modern Java API

• Type-safe builders for all configurations
• Spring Boot integration out of the box
• Comprehensive logging and debugging tools
• JavaFX Desktop Runner for visual automation development

📚 Documentation

Getting Started Quick Start Guide Installation Core Concepts Your First State

Guides Finding Objects State Management Color Detection Motion Tracking

API Reference Migration Guide (v1.1.0) Action Hierarchy JavaDoc (v1.0.7) JavaDoc (all versions) Example: Login Automation

🎮 Use Cases

Game Automation

Build sophisticated game bots that understand game states and react intelligently:

// Define game states
@State(initial = true)
@Getter
public class HomeBaseState {
    private final StateImage baseFlag;
    private final StateImage attackButton;
    
    public HomeBaseState() {
        baseFlag = new StateImage.Builder()
            .setName("home-flag")
            .addPatterns("game/home-flag")
            .build();
            
        attackButton = new StateImage.Builder()
            .setName("attack-button")
            .addPatterns("game/attack-button")
            .build();
    }
}

@State
@Getter
public class EnemyBaseState {
    private final StateImage enemyFlag;
    private final StateImage enemyUnits;
    
    public EnemyBaseState() {
        enemyFlag = new StateImage.Builder()
            .setName("enemy-flag")
            .addPatterns("game/enemy-flag")
            .build();
            
        enemyUnits = new StateImage.Builder()
            .setName("enemy-units")
            .addPatterns("game/enemy-units")
            .build();
    }
}

// Define transition from Home Base to Enemy Base
@Transition(from = HomeBaseState.class, to = EnemyBaseState.class)
@RequiredArgsConstructor
public class AttackEnemyBaseTransition {
    private final HomeBaseState homeBase;
    private final Action action;
    
    public boolean execute() {
        // Click attack button to navigate to enemy base
        ActionResult result = action.click(homeBase.getAttackButton());
        return result.isSuccess();
    }
}

// To go to the enemy base
stateNavigator.openState("EnemyBase");

Visual Testing

Create maintainable visual regression tests:

@Test
void loginPageShouldDisplayCorrectly() {
    // Define expected UI elements
    StateImage logo = new StateImage.Builder()
        .setName("company-logo")
        .addPatterns("login/logo")
        .setSimilarity(0.95)  // High similarity for visual tests
        .build();
    
    StateImage loginForm = new StateImage.Builder()
        .setName("login-form")
        .addPatterns("login/form")
        .build();
    
    // Verify all elements are present
    ActionResult logoResult = action.find(logo);
    assertTrue(logoResult.isSuccess(), "Logo should be visible");
    
    ActionResult formResult = action.find(loginForm);
    assertTrue(formResult.isSuccess(), "Login form should be visible");
}

Process Automation

Automate complex workflows across multiple applications:

// Define UI elements
StateImage exportButton = new StateImage.Builder()
    .addPatterns("app/export-button")
    .build();

StateImage fileNameField = new StateImage.Builder()
    .addPatterns("dialog/filename-field")
    .build();

// Complete workflow in one conditional chain
ConditionalActionChain
    .find(new PatternFindOptions.Builder().build())  // Find export button
    .ifFoundClick()     // Click it
    .then(new PatternFindOptions.Builder().build())  // Find filename field
    .ifFoundType("export_" + System.currentTimeMillis() + ".csv")  // Type filename
    .perform(action, new ObjectCollection.Builder()
        .withImages(exportButton, fileNameField)
        .build());

🏗️ Architecture

brobot/
├── library/          # Core Brobot framework
├── library-test/     # Integration tests with GUI
├── runner/           # JavaFX Desktop Runner (in development)
└── docs/             # Documentation website

Test Organization

• Unit tests (library/) - Fast, isolated component tests
• Integration tests (library-test/) - Full GUI automation tests with Spring context

🤝 Contributing

We love contributions! Whether it's bug reports, feature requests, documentation improvements, or code contributions, all are welcome.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

See CONTRIBUTING.md for detailed guidelines.

🛠️ Development Setup

# Clone the repository
git clone https://github.com/jspinak/brobot.git
cd brobot

# Build the project
./gradlew build

# Run tests
./gradlew test

# Run integration tests
./gradlew :library-test:test

📄 License

Brobot is MIT licensed. The documentation is Creative Commons licensed.

🙏 Acknowledgments

Brobot is built on the shoulders of giants:

• SikuliX - Computer vision engine
• OpenCV - Image processing
• Spring Boot - Application framework

🌟 Star History

---

_{Built with ❤️ by jspinak and contributors}