↓ Skip to main content
Oh My Ops | DevOps Guides, stacks & tools for engineers

CI/CD for Small Teams: The Pipeline and the Six Ways It Breaks

·2008 words·10 mins
Author
Maksim P.
DevOps Engineer / SRE
Table of Contents

TL;DR
#

  • Use the CI that ships with your git host. The integration is worth more than any feature you would gain elsewhere.
  • Build the artifact once and promote it. Rebuilding for production means shipping something you never tested.
  • Two independently green pull requests can break main the moment they merge. Required checks do not prevent this.
  • Set concurrency and timeout-minutes on day one. Without them, two merges race each other and one hung job can eat a third of your monthly minutes.
  • A rollback that reverts the code but not the schema is not a rollback.

Who this is for
#

Teams of 3-10 engineers shipping a web application or API, on GitHub or GitLab, without anyone whose job is the pipeline. You want deploys that are boring and a process that survives the person who set it up going on holiday.

Pick the CI that comes with your git host
#

GitHub → Actions, GitLab → GitLab CI, Bitbucket → Pipelines. Not because they are the best tools, but because the alternative costs you a system to authenticate, patch and debug, and buys you features you will not use for years.

The one thing worth checking before committing: the free minutes. On GitHub, private repositories get 2,000 minutes a month on Free and 3,000 on Team, billed per job and rounded up to the minute. Extra Linux minutes are $0.006 each. A ten-minute pipeline gives you roughly 200 runs a month before you pay — comfortable for a team of five, tight for a team of ten who push often.

Two things blow that budget faster than expected. Matrix jobs multiply: three Node versions on every push is three times the spend, and most small teams do not need to support three Node versions. And macOS runners bill at roughly ten times the Linux rate ($0.062 vs $0.006 a minute), so an iOS build turns the allowance into a rounding error.

The pipeline
#

Two workflows, and the split matters: what runs on a pull request, and what runs after it merges.

name: CI

on:
  pull_request:
  push:
    branches: [main]

# Two merges landing minutes apart would otherwise deploy in parallel and
# race each other. cancel-in-progress stays false because a deploy that is
# half-finished is worse than a deploy that is late.
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: false

jobs:
  test:
    runs-on: ubuntu-latest
    # Without this the ceiling is GitHub's own: 6 hours. One hung job
    # then costs 360 of your 2,000 monthly minutes.
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: '22'
          cache: 'npm'
      - run: npm ci
      - run: npm run lint
      - run: npm test

  build:
    needs: test
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      # Declaring any permission zeroes out the rest. Without contents:
      # read, checkout fails with a 403 before anything else runs.
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v7
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        run: |
          IMAGE=ghcr.io/${GITHUB_REPOSITORY,,}:${GITHUB_SHA}
          docker build -t "$IMAGE" .
          docker push "$IMAGE"

  staging:
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment: staging
    steps:
      - run: ./deploy.sh staging "${GITHUB_SHA}"
      - run: ./smoke-test.sh https://staging.example.com

  production:
    needs: staging
    runs-on: ubuntu-latest
    timeout-minutes: 15
    # Required reviewers live on the environment, not in the workflow file.
    # A pipeline gate someone can edit in a PR is not a gate.
    environment: production
    steps:
      - run: ./deploy.sh production "${GITHUB_SHA}"
      - run: ./smoke-test.sh https://example.com

Note what the deploy jobs do not do: they never build. The image tagged with the commit sha was built once, tested on staging, and the same bytes go to production. This is the difference between promoting an artifact and rebuilding one, and it is not academic — a rebuild picks up a new base image layer, a floating dependency, a changed lockfile resolution. You then deploy something that has never run anywhere.

The six ways this breaks
#

Everything above is the easy part. These are the failures that survive a green pipeline.

1. Two green pull requests that break main together
#

Required status checks run each pull request against its own branch. Nothing checks the combination. Alice removes a function she believes is unused; Bob adds a caller for it. Both branches pass. Both merge. main is broken, and neither pull request was wrong.

The cheap mitigation is GitHub’s “Require branches to be up to date before merging”, which forces a rebase and re-run before merge. It costs you a re-run per merge and it stops working the moment two people merge within the same few minutes. The real fix at any volume is a merge queue, which tests the actual merge result in sequence.

For a team of five this is a once-a-quarter annoyance and the branch protection setting is enough. Know that it exists so you recognise it when it happens, rather than concluding your tests are flaky.

2. A path filter that blocks pull requests forever
#

This one is a trap with a delightful shape:

on:
  pull_request:
    paths:
      - 'src/**'

Combine that with a required status check of the same name. Someone opens a pull request touching only README.md. The workflow does not run, the required check never reports, and the pull request sits at “Expected — Waiting for status to be reported” forever. It cannot be merged and nothing is wrong with it.

The fix is a companion workflow with the same job name and no path filter, which does nothing and exits successfully:

name: CI
on:
  pull_request:
    paths-ignore:
      - 'src/**'
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - run: echo "No source changes"

Ugly, and the documented approach. The alternative — no path filters at all — is often the better trade for a small repository.

3. Two deploys at once
#

Merge two pull requests a minute apart and, with no concurrency block, both deploy jobs run at the same time. Whichever finishes last wins, which is not necessarily the newer commit. If your deploy syncs files (aws s3 sync --delete, rsync --delete), the two runs interleave and you can serve a genuinely inconsistent state for a minute.

The concurrency block at the top of the workflow above fixes it. The choice worth thinking about is cancel-in-progress: true for pull request checks, where cancelling a superseded run saves minutes and costs nothing; false for deploys, where cancelling mid-way leaves the target half-updated.

4. Secrets and pull requests from forks
#

Workflows triggered by pull_request from a fork get no access to secrets — by design, because otherwise anyone could open a pull request that prints them. Teams discover this when their deploy-preview step starts failing on a contributor’s first pull request.

The tempting fix is pull_request_target, which does have secrets. Do not reach for it without understanding what it does. It runs in the context of the base repository with write permissions and secrets available; if you then check out the pull request’s head and run its code — its build script, its dependencies, its test suite — you have handed an unreviewed contributor your credentials. This is the single most common serious misconfiguration in public repositories.

If you have no outside contributors, this problem does not exist for you. If you do, keep untrusted code and secrets in separate workflows.

5. A cache that serves yesterday’s answer
#

cache: 'npm' in setup-node keys on the lockfile and is safe. Hand-rolled caches usually are not:

# Wrong: the key never changes, so the cache is never refreshed
- uses: actions/cache@v4
  with:
    path: ~/.cache/build
    key: build-cache

A key without a content hash means the first successful run pins the cache forever. Builds get faster and then quietly stop reflecting your dependencies. Key on the hash of whatever the cache derives from — hashFiles('**/package-lock.json') — and let a changed lockfile produce a changed key.

6. A rollback that cannot roll back
#

This is the one that turns a bad deploy into an incident.

Reverting a deploy puts the previous image back. It does not put the previous database schema back. If the release you are reverting also renamed a column, the old code now runs against a schema it does not understand, and your rollback has produced a second outage on top of the first.

The pipeline cannot solve this; the schema discipline has to. Every migration must leave the previous version of the application able to run — which means expand-contract, and three deploys rather than two. Until that is habit, treat any release containing a migration as one you cannot roll back by redeploying the previous tag, and say so out loud when you ship it.

What to actually check
#

A short audit you can run in ten minutes:

  • timeout-minutes on every job — the default ceiling is six hours
  • concurrency set, with cancel-in-progress: false on anything that deploys
  • Production deploys go through an environment with required reviewers, not an if: in the workflow file
  • The artifact deployed to production is the one tested on staging, byte for byte
  • Required checks cannot be skipped by a path filter
  • No pull_request_target checking out untrusted code
  • Cache keys contain a hash of what they cache
  • Cloud credentials come from OIDC, not from long-lived keys in repository secrets
  • Rollback has been performed once, on purpose, and someone timed it
  • Someone other than the author can explain what the pipeline does

The last two are the ones that fail in practice, and neither is a YAML problem.

Questions people ask
#

What CI should a small team use?
#

Whatever your git host provides — GitHub Actions, GitLab CI, Bitbucket Pipelines. The integration with pull requests, required checks and environments is worth more than any feature a dedicated CI system offers a team of this size, and there is no server to patch. Revisit the decision when you have a concrete requirement the built-in tool cannot meet, not before.

How long should a CI pipeline take?
#

Under ten minutes for the pull request pipeline, because that is roughly how long a developer will wait before context-switching. Beyond that, people stop watching their builds and start merging on the assumption it is fine — which converts your test suite from a gate into a formality. If the suite genuinely takes longer, split it: fast tests on the pull request, slow ones after merge.

Should staging and production use the same pipeline?
#

The same deploy logic, yes — same scripts, same manifests, differing only in configuration. But more important than sharing the logic is sharing the artifact: production should receive the exact image that was tested on staging, identified by commit sha. Rebuilding for production means deploying something no environment has run.

Why did my pull request pass CI and still break main?
#

Most likely because both your branch and someone else’s passed independently, and the conflict only exists in the combination. Required checks validate each pull request against its own base, not against the merge result. Enable “require branches to be up to date before merging” as a first step, and use a merge queue when merges get frequent enough for that to be annoying.

Do I need staging at all?
#

If deploying broken code costs you real money or real trust, yes. If you are pre-launch with ten users, a staging environment is a second system to maintain and pay for while the feedback it gives you is available from production at almost no cost. The honest test is not “is staging good practice” but “what would we do differently if staging caught something” — if the answer is nothing, you have built a copy of production for decoration.

Related reads #

Reply by Email