Reducing CI Execution Time Using Impact-Based Test Selection Across Repositories

CI optimization using Git diff and JGit to selectively run impacted Karate tests, reducing regression execution while preserving safe fallback coverage.

Jul. 21, 26 · Analysis

Likes (0)

Comment

Save

270 Views

Modern CI/CD pipelines often execute complete regression suites for every code change, regardless of the actual impact of the modification. While this approach guarantees broad validation coverage, it also introduces unnecessary test execution, slower feedback loops, and increased infrastructure cost.

This challenge becomes more visible in microservice-based systems where repositories, services, and automation suites are distributed across multiple projects. A small change in one module can unintentionally trigger an entire regression pipeline containing tests unrelated to the updated code.

To explore a lightweight solution for this problem, I built a personal engineering project that performs impact-based test selection using Git diff analysis, Spring Boot, JGit, GitHub Actions, and Karate.

The goal of the project was simple: instead of executing the full regression suite for every change, dynamically determine which tests are actually impacted and execute only those tests.

The project demonstrates how selective test execution can help reduce unnecessary CI workload while maintaining baseline validation coverage through fallback smoke testing.

The Problem With Traditional Regression Execution

In many CI/CD pipelines, regression execution is static.

Every push event or pull request triggers:

Full API regression suites
Complete integration testing
Broad validation across unrelated modules

Although this guarantees high coverage, it creates several practical problems.

Slow Feedback Cycles

Developers may wait several minutes or even longer to receive pipeline feedback for small localized changes.

For example, updating a single payments API controller might still trigger:

transactions tests
transfer tests
authentication regression tests
unrelated smoke validations

As projects scale, this delay affects engineering productivity and release speed.

Unnecessary Infrastructure Usage

Executing the same large regression suites repeatedly consumes unnecessary CI resources.

This becomes more expensive when:

Pipelines run in parallel
Multiple pull requests are active
Cloud runners are billed by execution time

Reduced Pipeline Efficiency

In many cases, only a small subset of tests is truly relevant to the code change. Running the full suite results in redundant execution and inefficient utilization of CI infrastructure.

The objective of this project was to explore whether lightweight impact analysis could reduce unnecessary test execution without introducing complicated tooling or dependency management systems.

Project Overview

The solution uses Git diff analysis to identify changed files and map those changes to relevant Karate test tags.

The architecture consists of two repositories:

      Plain Text
     
 

     Developer Repository
(fintech-impact-services)

        ↓ Push Event

GitHub Repository Dispatch

        ↓

Automation Repository
(karate-change-impact-test)

        ├── Start Spring Boot Application
        ├── Perform Git Diff Analysis
        ├── Generate Impacted Test Tags
        ├── Execute Targeted Karate Tests
        └── Generate Execution Metrics
    

The development repository contains the Spring Boot microservice and impact analysis API. The automation repository contains Karate test suites and GitHub Actions workflows responsible for selective test execution.

This separation allowed the automation layer to remain reusable and independently managed.

Cross-Repository Workflow

The workflow begins when code is pushed to the development repository. A GitHub Repository Dispatch event triggers the automation repository pipeline.

Example dispatch payload:

      Shell
     
 

     curl -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <TOKEN>" \
  https://api.github.com/repos/{owner}/karate-change-impact-test/dispatches \
  -d '{
    "event_type": "dev_push"
  }'
    

The automation repository then performs the following steps:

Checkout repositories
Start the Spring Boot impact-analysis service
Wait for API readiness
Call the impact-analysis endpoint
Retrieve impacted Karate tags
Execute only selected tests
Publish execution metrics

This design helped simulate a lightweight cross-repository CI orchestration model using only GitHub-native capabilities.

Implementing Git Diff Analysis Using JGit

The core functionality of the project relies on identifying changed files between commits.

Instead of using shell-based Git commands inside CI scripts, the implementation uses JGit, a Java library that provides Git functionality directly within Java applications.

The service compares the current branch with a target branch reference and extracts modified file paths.

Example implementation:

      Java
     
 

     public Set<String> getImpactedTags(String targetBranch) throws Exception {

    FileRepositoryBuilder builder = new FileRepositoryBuilder();

    Repository repository = builder.readEnvironment()
            .findGitDir(new File("."))
            .build();

    try (Git git = new Git(repository)) {

        AbstractTreeIterator oldTreeParser =
                prepareTreeParser(repository, targetBranch);

        List<DiffEntry> diffs = git.diff()
                .setOldTree(oldTreeParser)
                .call();

        return calculateTags(diffs);
    }
}
    

Once the diff entries are collected, the application extracts changed paths and sends them to the mapping engine.

Using JGit provided several advantages:

Better portability across environments
Easier integration with Spring Boot
Reduced dependency on shell scripting
Cleaner CI pipeline implementation

One challenge encountered during implementation was ensuring full Git history availability inside GitHub Actions runners. Shallow clones occasionally caused incorrect branch comparisons, so complete fetch depth was required for accurate analysis.

Rule-Based Impact Mapping

After identifying changed files, the framework maps those files to relevant Karate execution tags.

Example mapping rules:

Changed Path	Karate Tag
`/payments/`	`@payments`
`/transactions/`	`@transactions`
`/transfer/`	`@transfers`
`/auth/`	`@regression`
`pom.xml`	`@regression`

Example implementation:

     Java
    
    if (path.contains("/payments/")) {
    tags.add("@payments");
}

if (path.contains("/transactions/")) {
    tags.add("@transactions");
}

The project intentionally uses deterministic rule-based mapping instead of advanced dependency graph analysis or machine learning models.

The primary reasons were:

Simplicity
Predictability
Easier debugging
Faster implementation
Lower maintenance overhead

Although more sophisticated impact-analysis systems exist, lightweight rule-based mapping was sufficient to demonstrate measurable CI optimization in this project.

Dynamic Karate Test Execution

Once impacted tags are generated, the automation repository dynamically executes only the relevant Karate tests.

Example command:

     Shell
    
    mvn test -Dkarate.options="--tags @payments,@transactions"

The Karate framework worked particularly well for this implementation because feature files were already organized by business capability.

Example structure:

     Gherkin
    
 

    features/
 ├── payments.feature
 ├── transactions.feature
 ├── transfers.feature
 └── smoke.feature
   

This made selective execution straightforward without requiring custom runners or additional orchestration frameworks.

Safe Fallback Mechanism

One important concern with selective execution is the possibility of hidden dependencies. A code change may indirectly impact areas not captured by simple path-based rules.

To reduce this risk, the framework includes a fallback strategy. If no impacted tags are identified, the pipeline automatically executes baseline smoke tests.

Example:

     Shell
    
    @smoke

This ensured that critical application validation still occurred even when no direct impact mapping was detected.

The fallback mechanism helped balance optimization with CI reliability.

GitHub Actions Integration

GitHub Actions was used to orchestrate the complete workflow.

Example workflow steps:

     YAML
    
 

    - name: Start Spring Boot Service
  run: mvn spring-boot:run &

- name: Wait for API Readiness
  run: |
    curl --retry 10 \
         --retry-delay 5 \
         http://localhost:8080/actuator/health
   

One practical issue encountered during implementation was synchronization between application startup and API invocation.

Without readiness checks, the workflow occasionally attempted to call the API before the Spring Boot application was fully initialized.

Adding retry-based health checks improved workflow stability significantly.

Execution Metrics

To evaluate the effectiveness of the approach, the framework generates execution metrics after each run.

Example output:

     JSON
    
 

    {
  "impacted_tags": "@payments,@transactions",
  "scenarios_executed": 6,
  "scenarios_skipped": 12,
  "test_reduction_rate": "66%",
  "timing_metrics": {
    "total_workflow_seconds": 52,
    "isolated_test_seconds": 18,
    "api_overhead_seconds": 6
  }
}
   

In sample execution scenarios from this project, the framework reduced executed tests by approximately 60–70% depending on the scope of code changes.

Localized feature updates benefited the most, while shared dependency changes still triggered broader regression execution.

Although these results came from a personal engineering project rather than a production enterprise system, the experiment demonstrated how lightweight impact-aware execution can improve CI efficiency.

Limitations and Future Improvements

The project also exposed several limitations.

Manual Mapping Maintenance

Rule-based mappings require periodic updates as repositories evolve.

Hidden Dependency Risks

Indirect service dependencies may not always be detected through simple path matching.

Git Comparison Accuracy

Accurate impact analysis depends heavily on proper branch comparison and repository history availability.

Future improvements could include:

dependency graph analysis
code coverage–based impact detection
historical test-failure analysis
ML-assisted impact prediction
multi-module dependency propagation

These enhancements could improve precision while preserving the lightweight nature of the framework.

The complete implementation for this project, including the Spring Boot impact-analysis service and GitHub Actions workflow, is available on GitHub.

Source Code:

Dev repository: https://github.com/raakeshdev20/fintech-impact-services
Automation repository:https://github.com/raakeshdev20/karate-change-impact-test

Conclusion

This project explored how Git diff analysis and selective test execution can help optimize CI/CD pipelines without requiring complex external tooling.

By combining the following, the framework demonstrated a practical approach to reducing unnecessary regression execution for localized changes.

JGit-based change detection
deterministic impact mapping
dynamic Karate execution
GitHub Actions orchestration

While the implementation is intentionally lightweight, the experiment highlights how impact-aware testing strategies can improve feedback cycles, reduce redundant execution, and make CI pipelines more efficient as systems continue to scale.

Repository (version control) Spring Boot Testing

Opinions expressed by DZone contributors are their own.

Related

Trending