CI/CD pipelines have changed a lot over the years. What worked in 2020 doesn't necessarily work in 2024. Tools have improved. Practices have evolved. Teams have learned hard lessons.
I've been building and managing CI/CD pipelines for years. Here's what the best teams are doing differently now, and what's actually worth adopting.
1. Pipeline as Code (This One is Non-Negotiable Now)
If your pipelines aren't version-controlled, you're doing it wrong. Period.
All major CI/CD platforms support pipeline as code now:
- GitHub Actions: Workflows are YAML files in your repo
- GitLab CI/CD: .gitlab-ci.yml file
- Jenkins: Jenkinsfile (Groovy or Declarative)
- Azure DevOps: YAML pipelines
Benefits of pipeline as code:
- Version control: Track changes, see history, roll back
- Code review: Pipelines get reviewed like any other code
- Reusability: Share pipeline templates across projects
- Reproducibility: Anyone can see and run the same pipeline
2. Multi-Stage Pipelines (But Keep Them Simple)
I see teams create pipelines with 20 stages. That's overkill. Here's what actually works:
- Build: Compile code, run unit tests
- Test: Integration tests, security scans
- Deploy to Dev: Automatic deployment
- Deploy to Staging: Manual approval, then deploy
- Deploy to Prod: Manual approval, then deploy
That's it. Five stages. You don't need more unless you have a really good reason.
3. Parallel Execution (Speed Matters)
If your tests take 30 minutes, your pipeline takes 30 minutes. That's too slow.
Run tests in parallel:
- Split unit tests across multiple runners
- Run integration tests in parallel
- Run security scans alongside tests (don't wait for them)
- Build Docker images while running tests
A pipeline that takes 30 minutes sequentially might take 10 minutes in parallel. That's a huge productivity boost.
4. Fast Feedback (Fail Fast)
The earlier you catch issues, the cheaper they are to fix. Structure your pipeline to fail fast:
- Run quick checks first (linting, formatting, unit tests)
- Then run slower checks (integration tests, security scans)
- Only deploy if everything passes
If unit tests fail, don't run integration tests. If linting fails, don't run anything else. Fail fast, fix fast.
5. Secrets Management (Don't Hardcode Secrets)
I've seen teams put API keys, passwords, and tokens directly in pipeline files. Please don't do this.
Use secret management:
- GitHub Actions: Secrets in repository settings
- GitLab CI/CD: CI/CD variables (masked)
- Jenkins: Credentials plugin or HashiCorp Vault
- External: AWS Secrets Manager, HashiCorp Vault
Never commit secrets to Git. Ever.
6. Artifact Management (Build Once, Deploy Many)
Build your Docker images or application artifacts once, then reuse them across environments:
- Build artifact in CI
- Tag it with version (e.g., v1.2.3) and commit SHA
- Push to artifact registry (Docker Hub, AWS ECR, GitHub Container Registry)
- Deploy the same artifact to dev, staging, and prod
This ensures dev and prod run the exact same code. No "it works on my machine" issues.
7. Automated Rollback (Because Things Break)
Things will break in production. That's reality. Have automated rollback strategies:
- Health checks after deployment: If health checks fail, automatically roll back
- Canary deployments: Deploy to a small percentage first, monitor, then expand
- Blue-green deployments: Keep old version running, switch traffic if new version works
- Feature flags: Turn off new features if issues are detected
Automated rollback means you can fix issues in minutes, not hours.
8. Pipeline Notifications (Keep Teams Informed)
When pipelines fail, teams need to know. Set up notifications for:
- Pipeline failures (Slack, email, PagerDuty)
- Successful deployments (optional, but nice to know)
- Long-running pipelines (if a pipeline takes more than X minutes, notify)
But don't over-notify. Too many notifications means teams ignore them.
9. Pipeline Caching (Speed Up Builds)
Caching dependencies can cut build times in half:
- Cache npm/node_modules
- Cache Docker layers
- Cache Maven/Gradle dependencies
- Cache Python packages
Most CI/CD platforms support caching. Use it. Your builds will be faster, and you'll use fewer CI minutes.
10. Environment Parity (But Not Too Much)
Dev, staging, and prod should be similar, but not identical:
- Same infrastructure: Use the same cloud provider, same services
- Same code: Deploy the same artifacts
- Similar configuration: Same environment variables, same structure
- Different scale: Prod is bigger, that's fine
The goal is: if it works in staging, it should work in prod. But you don't need prod-scale staging environments.
What's Changed Since 2020
Here's what I'm seeing teams do differently now:
- More automation: Less manual approval, more automated gates
- Better security: Security scans are built into pipelines, not afterthoughts
- Faster feedback: Parallel execution and caching make pipelines much faster
- GitOps adoption: More teams using GitOps workflows (ArgoCD, Flux)
- Infrastructure as Code: Infrastructure changes go through CI/CD too
What Doesn't Work Anymore
Things I see teams doing that don't work well:
- Manual deployments: Clicking buttons in a UI - too slow, too error-prone
- Complex pipelines: 20+ stages that nobody understands
- No testing: Deploying without running tests first
- Hardcoded secrets: Still seeing this, and it's still a bad idea
- Slow pipelines: If your pipeline takes an hour, it's too slow
Your Next Steps
Want to improve your CI/CD pipelines? Here's where to start:
- Move pipelines to code (if they're not already)
- Add parallel execution to speed things up
- Set up automated rollback strategies
- Enable caching for dependencies
- Add security scans to your pipeline
- Set up notifications for failures
CI/CD pipelines are about speed and reliability. The faster you can deploy, and the more confident you can be that deployments work, the better.
What CI/CD practices have you adopted this year? What's working (or not working) for your team? I'd love to hear what you're trying.