Build Pipeline Bottlenecks: Diagnosing Slow CI at Enterprise Scale

A developer at a large financial institution submits a pull request. The CI pipeline starts. Forty-seven minutes later, if the queue was short, they get a result. If the queue was long, they have already switched contexts, started something else, and must now re-establish the mental state they were in when they submitted the PR. That context switch is not a productivity inconvenience. Multiply it by 800 engineers, each waiting for 47-minute feedback loops, and the aggregate cost in developer hours is measurable on a finance spreadsheet, not just a DevOps dashboard. Slow CI at enterprise scale is a competitive disadvantage quantified in engineering capacity that could otherwise be building features.

Teams using real-time dashboards and immediate alerts reduced their mean-time-to-resolution by up to 50%, with a 30% improvement in response times when alerts were routed effectively. Those improvements are achievable, but they require diagnosing the actual bottleneck before applying fixes. Most CI optimization advice applies one-size-fits-all solutions to problems that have specific root causes. Caching dependencies helps when dependency installation is the bottleneck. It does nothing when the bottleneck is a 20,000-test suite with 3 percent flakiness, a security scan that serializes after all tests, or a build graph that recompiles 4,000 modules when one shared library changes. Enterprise CI bottlenecks require enterprise diagnosis before enterprise fixes.

Why Enterprise CI Is Different

The advice that works for a 10-engineer startup, cache your node_modules, parallelize your tests, use GitHub Actions, is not wrong for an enterprise. It is insufficient. Enterprise CI operates at a scale that introduces failure modes that simply do not exist in smaller environments:

Scale of the codebase. A monorepo with 50 million lines of code across 400 services cannot be treated like a 50,000-line single-service repository. Incremental build correctness, ensuring that only the modules affected by a change are rebuilt, requires an accurate dependency graph at the repository level. Without it, the only safe option is to build everything, and building everything in a 50-million-line codebase takes a long time regardless of how much you cache.

Scale of the team. In a tight talent market, engineers avoid companies with frustrating workflows. A sluggish CI pipeline becomes a morale issue. When 500 engineers are all pushing code during business hours, shared runner pools queue. A pipeline that runs in 15 minutes when the queue is empty runs in 45 minutes when 200 jobs are waiting. Queue time is a capacity planning problem, not a code optimization problem, and the two require different solutions.

Scale of the compliance requirement. With regulations like SOC 2, ISO 27001, and GDPR, security checks are embedded into pipelines. SAST, DAST, container scanning, these add time unless optimized properly. An enterprise that has added SAST scanning, SCA dependency scanning, secret detection, and IaC policy checking to its pipeline in response to regulatory requirements may have added 15-20 minutes of compliance gate time that runs serially after tests. These checks were not there three years ago; they are now the longest stage.

Scale of the test surface. A 20,000-test suite with 2 percent test flakiness has 400 tests that fail non-deterministically on any given run. Each flaky failure triggers an investigation, a re-run, or worse, a normalization of failure where engineers learn to ignore red pipelines. At enterprise scale, flaky tests are not an inconvenience; they are a systemic reliability problem that erodes trust in the CI system itself.

Measure First: The Four Pipeline Metrics That Matter

You cannot fix what you have not measured. Before any optimization, establish baseline measurements for the four metrics that characterize pipeline health at enterprise scale:

P50 and P95 build time. The median build time (P50) tells you the typical developer experience. The 95th percentile (P95) tells you how bad the worst-case experience is. A pipeline with P50 of 12 minutes and P95 of 45 minutes has a long-tail problem, some combination of flakiness, queue contention, and resource limits is creating severe outliers. A pipeline with P50 of 30 minutes and P95 of 35 minutes has a uniformly slow build that needs structural optimization.

darbe

# GitHub Actions: extract P50 and P95 build times via API
gh api repos/{owner}/{repo}/actions/runs \
  --field per_page=100 \
  --jq '[.workflow_runs[] | select(.status=="completed") |
         .run_duration_ms / 1000] |
        sort |
        {
          p50: .[(length * 0.5 | floor)],
          p95: .[(length * 0.95 | floor)],
          count: length
        }'

Queue time vs execution time ratio. If a pipeline runs for 15 minutes but sits in the queue for 20 minutes before it starts, the execution optimization problem is less important than the runner capacity problem. Measure both separately.

darbe

# GitLab CI: extract queue duration and execution duration
# Via GitLab API for recent pipelines
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$GITLAB_URL/api/v4/projects/$PROJECT_ID/pipelines?per_page=50" |
  jq '[.[] | {
    id: .id,
    duration: .duration,
    queued_duration: .queued_duration,
    queue_ratio: (.queued_duration / .duration * 100)
  }] | sort_by(.queue_ratio) | reverse'

Flaky test rate. The percentage of pipeline failures caused by tests that pass on re-run without any code change. A flaky test rate above 1 percent creates enough noise that developers begin ignoring pipeline failures.

Cost per pipeline run. According to Flexera’s 2025 State of the Cloud Report, organizations overspend an estimated 28% on cloud resources. CI/CD workloads, especially container builds and ephemeral environments, are a major contributor. Tracking cost per run makes optimization measurable in financial terms that resonate with engineering leadership.

The Seven Enterprise CI Bottleneck Types

Understanding the root cause of slowness determines which fix applies. Not all slow pipelines are slow for the same reason:

Bottleneck 1: Build graph over-approximation

The build system rebuilds modules it did not need to rebuild because it does not have an accurate dependency graph. A change to a utility library triggers a rebuild of every service in the monorepo because the build system cannot determine that only two services actually import that library. This is the root cause of the “everything rebuilds on every change” phenomenon in large monorepos.

Bottleneck 2: Cache invalidation failures

Caching is configured but not effective. Cache keys are too broad (invalidated by unrelated changes), too narrow (misses are too frequent), or based on mutable inputs (timestamps, branch names) rather than content hashes. The effect: the cache exists, the pipeline appears to use it, but cache hits are rare in practice.

Bottleneck 3: Test suite bloat and flakiness

The test suite has grown to include slow integration tests that run on every PR, flaky tests that cause re-runs without signal value, and redundant tests that cover the same code paths. At 20,000 tests, even a 3ms average test execution time produces a 60-second test stage, before parallelization is accounted for.

Bottleneck 4: Serial security scanning

SAST, SCA, secret detection, and container scanning run sequentially after tests complete. Each scan tool runs independently, scans the full codebase, and takes 5-8 minutes. Four scans in series add 20-32 minutes. Running them in parallel, as a separate job group that starts when tests start, not after, eliminates this overhead.

Bottleneck 5: Runner pool contention

The runner pool is sized for average load, not peak load. On days when the team is active, sprint end, release days, Friday afternoon, pipeline queue times spike because every available runner is occupied. The fix is autoscaling runners, not larger runners.

Bottleneck 6: Artifact and image bloat

Docker images have grown to 4GB because no one removed unnecessary packages. Build artifacts are copied in full even when only a small subset is needed downstream. Transferring 4GB between pipeline stages takes longer than the stages themselves.

Bottleneck 7: Dependency installation without effective caching

npm install, pip install, mvn dependency:resolve, package installation that runs on every pipeline execution because the cache key does not correctly reflect the dependency lockfile content. This is the most commonly addressed bottleneck and still the most commonly misconfigured fix.

The Dependency Graph Problem: Why Knowing What to Build Matters Most

Of the seven bottleneck types, Bottleneck 1, build graph over-approximation, has the largest potential impact at enterprise scale and receives the least attention in standard CI optimization guidance. The reason is that solving it requires structural knowledge of the codebase, not just pipeline configuration changes.

Modern build systems for JavaScript and TypeScript monorepos, Nx, Turborepo, Bazel, implement “affected target” analysis: given a set of changed files, determine the minimal set of packages, services, and tests that could be affected by those changes, and build/test only those targets. The analysis is fast because these tools maintain an explicit dependency graph of the repository’s packages.

For enterprise Java, C#, or Python monorepos, equivalent tools (Bazel, Pants, Gradle with build scans) exist but require significant investment to configure correctly. For polyglot enterprise environments that span multiple languages, the dependency graph must cover cross-language dependencies, a Java service that calls a Python utility must be in the same dependency graph as the Python utility it calls, or a change to the Python utility will not be detected as affecting the Java service.

tatlım

# GitHub Actions: affected-only build using Nx (monorepo example)
name: CI

on: [pull_request]

jobs:
  affected:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.affected.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Required: Nx needs full history to compute affected

      - name: Compute affected projects
        id: affected
        run: |
          # Nx identifies only affected packages since base branch
          AFFECTED=$(npx nx show projects --affected --base=origin/main \
                     --json 2>/dev/null | jq -c '{include: [.[] | {project: .}]}')
          echo "matrix=$AFFECTED" >> $GITHUB_OUTPUT

  build-affected:
    needs: affected
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJson(needs.affected.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v4
      - name: Build ${{ matrix.project }}
        run: npx nx build ${{ matrix.project }}
      - name: Test ${{ matrix.project }}
        run: npx nx test ${{ matrix.project }}

The matrix dynamically contains only the affected projects, if a change touches auth-service hem de auth-service is depended upon by user-service, both are in the matrix. If it touches analytics-service sadece, auth-service hem de user-service are not built at all.

A Systematic Diagnosis Protocol

Before any optimization decision, follow this sequence:

Step 1: Collect baseline measurements. Run the metric extraction commands above against the last 100 pipeline executions. Establish P50, P95, queue ratio, flaky test rate, and cost per run before touching anything.

Step 2: Visualize the pipeline critical path. Identify the longest sequential chain of stages. Most pipeline optimizations make the wrong stage faster, parallelizing something that was not on the critical path produces no improvement in total pipeline time.

Step 3: Categorize the bottleneck. Against the seven bottleneck types: is the slowness in queue time (Bottleneck 5) or execution time (Bottlenecks 1-4, 6-7)? If queue time is more than 30 percent of total pipeline duration, runner capacity is the primary problem and code-level optimizations are secondary.

Step 4: Target the longest critical-path stage. Measure individual stage durations. The stage that takes longest on the critical path is the stage worth fixing first. A 15-minute test stage with 5-minute security scans running in parallel is a testing problem. The same pipeline with scans running serially after tests is a pipeline structure problem.

Step 5: Apply the specific fix for the identified bottleneck. Match the fix to the root cause rather than applying every optimization from a checklist.

Step 6: Measure the improvement. Re-run the baseline measurement after applying the fix. A fix that reduces P50 by 5 minutes but increases P95 by 10 minutes has made the typical experience better and the worst-case experience worse, which may not be the correct trade-off.

Specific Fixes for the Most Impactful Bottlenecks

Caching that actually works (Bottleneck 7):

tatlım

# GitHub Actions: content-hash caching for Node.js
- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-
# hashFiles produces a cache key based on lockfile content, not branch name
# The key changes only when the lockfile changes, not on every commit

Parallel security scanning (Bottleneck 4):

tatlım

# Run security scans in parallel with tests, not after
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  security:
    runs-on: ubuntu-latest   # Starts at the same time as 'test'
    steps:
      - uses: semgrep/semgrep-action@v1
      - uses: snyk/actions/node@master

  deploy:
    needs: [test, security]   # Both must pass before deploy
    # Total time: max(test_duration, security_duration)
    # Not: test_duration + security_duration

Flaky test quarantine (Bottleneck 3):

The correct approach to flaky tests is not to tolerate them and not to retry silently. Tag them explicitly, route them to a quarantine suite that runs separately from the main gate, and track their flakiness rate over time:

tatlım

# Jest: quarantine flaky tests with a custom tag
# Run quarantined tests separately, not as PR gate
- name: Run stable tests (PR gate)
  run: jest --testPathIgnorePatterns="quarantine"

- name: Run quarantined tests (non-blocking, for tracking)
  run: jest --testPathPattern="quarantine" || true
  # || true: never fails the pipeline; results tracked separately

The Legacy Pipeline: COBOL Compile Scope

The enterprise CI bottleneck problem extends into mainframe environments where the “build” step is COBOL compilation on z/OS. COBOL compile times for large programs are significant, and the default approach in legacy environments, recompile everything when anything changes, produces the same over-build problem that Nx and Bazel address for modern monorepos.

In a COBOL environment, the dependency that drives compile scope is the copybook: a shared member included by any number of programs. A change to a frequently included copybook, one shared by 300 programs, triggers recompilation of all 300 programs if the build system does not have a fine-grained dependency map. If the dependency map is accurate, if the build system knows which programs include the changed copybook, it can reduce the compile scope to exactly those programs that are actually affected.

This is the equivalent of “affected target” analysis applied to COBOL: not build everything, build only what the changed copybook reaches. For a copybook change that actually affects 12 programs rather than 300, the compile scope reduction is a factor of 25. At enterprise COBOL compile times, that reduction translates directly into pipeline speed.

Ne kadar SMART TS XL Supports Accurate Dependency-Driven CI

SMART TS XL'S uygulama bağımlılık eşlemesi produces the cross-language dependency graph that accurate affected-target analysis requires. For COBOL environments, this means: every program that includes each copybook, every program that calls each subprogram, and every JCL job that invokes each program. When a copybook changes, the dependency map immediately produces the exact list of programs that must be recompiled, not the full portfolio, but the specifically affected subset.

For enterprise environments that span COBOL, Java, Python, and JCL, the dependency map covers cross-language dependencies: a Java service that calls a COBOL program via a CICS interface has a dependency on that COBOL program in the map. If the COBOL program changes, the Java service’s integration tests should be in the affected scope. Without this cross-language dependency graph, the CI system either misses the Java integration test (false safe) or rebuilds everything (false expensive).

MKS etki analizi capability makes this affected-target analysis queryable in real time: given a specific changed component, what is the complete set of programs, services, and tests that must be validated? This query is the input to the CI scheduling decision, which jobs run on this pipeline because this change requires them, and which jobs can be skipped because the dependency graph shows they are not affected.

MKS statik kod analizi capability identifies the code patterns that slow builds at the code level: deeply nested conditional structures that produce long compilation times, duplicate code that inflates build scope, and dead code that occupies compilation capacity without contributing to the running system. Removing dead code from the compilation scope reduces build time proportionally, code that does not exist cannot slow the build.

MKS kurumsal arama capability supports CI diagnosis in multi-language portfolios: find every program that depends on a specific component (to scope an affected-target rebuild), every copybook that is included by more than a threshold number of programs (to identify the highest-impact compilation dependencies), every module with no inbound references (dead code that can be excluded from build scope). These search queries are the diagnostic queries that precede dependency-driven CI optimization.

Conclusion: Diagnosis Before Optimization, Every Time

The pattern that characterizes every failed CI optimization effort is the same: applying a generic fix to an undiagnosed specific problem. Parallelizing tests when the bottleneck is queue time. Scaling runners when the bottleneck is build scope. Adding caching when the bottleneck is flaky tests causing re-runs that invalidate cached results.

Slow CI at enterprise scale is diagnosable. The measurement framework produces the data. The seven bottleneck types provide the diagnostic taxonomy. The dependency graph analysis identifies the root cause of the build scope problems that account for the largest optimization opportunities. The specific fixes follow from the specific diagnosis.

The pipeline that runs in 12 minutes instead of 47 is not the pipeline that applied every optimization from a checklist. It is the pipeline that measured first, identified the longest stage on the critical path, fixed that stage specifically, and repeated until there was nothing left worth fixing. That discipline, measure, diagnose, target, fix, validate, is faster and more reliable than any individual optimization technique.