Flaky Test Framework

Overview

This project investigates methods for reducing flaky test failures in Continuous Integration and Continuous Deployment (CI/CD) pipelines using Selenium WebDriver automation.

The framework compares a baseline Selenium implementation against an enhanced implementation incorporating synchronization mechanisms and retry strategies. Experimental execution was performed using GitHub Actions and Maven TestNG automation on the Demoblaze e-commerce application.

This repository supports the research conducted for a BSc Information Technology dissertation project.

---

Research Title

Reducing Flaky Tests in CI/CD Pipelines using Selenium Automation

---

Project Duration

February 2026 – July 2026 (6 Months)

---

Research Aim

To evaluate whether automated mitigation techniques can reduce flaky test behaviour and improve test stability within CI/CD environments.

---

Research Objectives

1. Investigate the causes of flaky tests in Selenium automation.
2. Develop a baseline automation framework without mitigation mechanisms.
3. Implement mitigation strategies including:
   • Explicit waits
   • Retry mechanisms
   • Screenshot capture
   • CI/CD execution
4. Compare baseline and enhanced framework performance.
5. Analyse improvements in test stability and reliability.

---

Technology Stack

Technology	Purpose
Java 17	Programming Language
Selenium WebDriver	Browser Automation
TestNG	Test Framework
Maven	Dependency Management
GitHub Actions	CI/CD Pipeline
ChromeDriver	Browser Driver
WebDriverManager	Driver Management
CSV Reports	Result Storage

---

Application Under Test

Website:

https://www.demoblaze.com/

Tested functionalities include:

• Login
• Product Selection
• Add To Cart
• Category Navigation
• Cart Navigation
• Home Navigation

---

Framework Architecture

src/test/java
│
├── base
│   └── BaseTest.java
│
├── pages
│   ├── LoginPage.java
│   ├── LoginPageBaseline.java
│   └── HomePage.java
│
├── tests
│   ├── BaselineTests.java
│   ├── EnhancedTests.java
│   └── EcommerceTests.java
│
└── utils
    ├── WaitUtils.java
    ├── RetryAnalyzer.java
    ├── ScreenshotUtils.java
    └── ExcelUtils.java

---

Flaky Test Mitigation Techniques

Baseline Framework

The baseline implementation executes Selenium tests without any stabilization mechanism.

Characteristics:

• No explicit waits
• No retry logic
• Higher susceptibility to timing issues

Enhanced Framework

The enhanced implementation includes:

Explicit Waits

WaitUtils.waitForElement(driver, By.id("login2"));

Retry Mechanism

retryAnalyzer = RetryAnalyzer.class

Screenshot Capture

ScreenshotUtils.capture(driver, "EnhancedFail");

CI/CD Integration

Automated execution through GitHub Actions.

---

Running Tests Locally

Clone Repository

git clone https://github.com/SapteshPawar/Flakytest-framework.git

Navigate to Project

cd Flakytest-framework

Run All Tests

mvn clean test

Run Baseline Tests

mvn -Dtest=BaselineTests test

Run Enhanced Tests

mvn -Dtest=EnhancedTests test

Run in Headless Mode

mvn clean test -Dheadless=true

---

GitHub Actions CI/CD

The project includes automated test execution using GitHub Actions.

Pipeline workflow:

GitHub Push
      ↓
Maven Build
      ↓
Selenium Test Execution
      ↓
CSV Result Generation
      ↓
Artifact Upload

Workflow file:

.github/workflows/selenium-tests.yml

---

Experimental Design

Two implementations were evaluated:

Framework	Description
Baseline	No mitigation mechanisms
Enhanced	Explicit waits and retry logic

Experimental metrics collected:

• Total Test Executions
• Passed Tests
• Failed Tests
• Pass Rate
• Flakiness Rate
• Execution Time

---

Results Summary

The experimental evaluation demonstrated:

• Reduction in flaky test failures
• Improved pass rates
• Enhanced framework stability
• More reliable CI/CD execution

Result files are available in:

docs/results/

---

Repository Structure

.github/
docs/
src/
pom.xml
README.md
RESEARCH.md

---

Releases

Version	Description
v0.1	Project Proposal
v0.2	Framework Setup
v0.3	Baseline Framework
v0.4	Enhanced Framework
v0.5	CI/CD Integration
v1.0	Final Dissertation Submission

---

Research Evidence

The repository includes:

• Framework architecture diagrams
• GitHub Actions execution evidence
• Experimental result datasets
• Test execution screenshots
• CI/CD workflow screenshots
• Comparative analysis charts

Located in:

docs/research evidence/

---

Limitations

• Single application under test (Demoblaze)
• Chrome browser only
• Limited project duration (6 months)
• Simulated flaky scenarios rather than enterprise-scale environments

---

Future Work

Potential future enhancements include:

• Cross-browser testing
• Parallel execution
• Selenium Grid integration
• AI-assisted flaky test detection
• Cloud-based execution environments

---

Author

Saptesh Pawar

MSc Information Systems Dissertation Project

2026

---

Citation

If referencing this work academically:

Pawar, S. (2026) Enhancing Web Application Testing Reliability: A Selenium-Java Automation Framework with CI/CD Integration.
MSc Information Systems Dissertation Project.

---

License

This project is licensed under the MIT License.