Building GitOps Pipelines With Helm on OpenShift: Lessons From the Trenches

After spending the last two years knee-deep in Kubernetes deployments and watching too many "quick fixes" turn into production incidents, I've become a true believer in GitOps. Not because it's the latest buzzword, but because it actually works when you need to sleep at night.

Last month, our team finally finished migrating our entire microservices platform to a GitOps workflow using Helm and OpenShift. It wasn't pretty, and we definitely learned some things the hard way. But now that the dust has settled, I wanted to share what we've discovered about making this stack work in the real world.

What GitOps Actually Means (Beyond the Marketing)

Look, GitOps isn't rocket science. It's basically treating Git like your single source of truth for everything that runs in your cluster. Instead of frantically running kubectl commands when things break (we've all been there), you make changes in Git and let automation handle the rest.

The core idea is simple:

• Write your infrastructure as code (YAML, Helm charts, whatever).
• Store it in Git, where you can track every change.
• Let tools like ArgoCD automatically sync your cluster to match what's in Git.

The beauty is that rolling back a broken deployment becomes as easy as reverting a commit. No more "did anyone remember what we changed last Tuesday?"

Why We Chose Helm + OpenShift + ArgoCD

Honestly, we didn't start with this stack. We tried a few different approaches first:

• Git – Obviously needed this for version control. Nothing controversial here.
• Helm – After writing thousands of lines of repetitive YAML, Helm templates became a lifesaver. Sure, the template syntax can be a bit weird, but it beats copy-pasting deployment files.
• OpenShift – Our company was already standardized on Red Hat, and OpenShift gives us enterprise features like proper RBAC and security scanning out of the box. The web console is actually pretty decent, too.
• ArgoCD – We initially tried Flux, but ArgoCD's UI won our operations team over. Being able to visualize what's happening during deployments is surprisingly valuable when things go sideways.

This combination solved our biggest headaches: inconsistent environments, no rollback strategy, and way too much manual intervention during deployments.

How We Structured Everything

Here's what our Git repo looks like after several iterations:

platform-gitops/
├── applications/
│   ├── user-service/
│   │   ├── Chart.yaml
│   │   ├── templates/
│   │   ├── values-development.yaml
│   │   ├── values-staging.yaml
│   │   └── values-production.yaml
│   ├── payment-service/
│   │   └── (same structure)
├── infrastructure/
│   ├── monitoring/
│   ├── ingress-controllers/
│   └── cert-management/
└── argocd-apps/
    ├── dev-apps.yaml
    └── prod-apps.yaml

A few things we learned about organization:

• Keep your values files separate for each environment. Trying to use a single values file with overrides gets messy fast.
• Put shared infrastructure stuff in its own directory. You don't want monitoring configurations mixed with application code.
• Name things clearly. Future you will thank present you.

Setting Up ArgoCD on OpenShift

Getting ArgoCD running was pretty straightforward, though we hit a few gotchas:

oc new-project argocd-system
kubectl apply -n argocd-system -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Create a route so we can access the UI
oc expose svc argocd-server --port=https -n argocd-system

Pro tip: Change the default admin password immediately. We learned this one the embarrassing way.

Getting ArgoCD to talk to our private Git repos took some figuring out. SSH keys work fine, but we ended up using a service account token for simplicity.

Application Definitions That Actually Work

Here's what one of our ArgoCD application definitions looks like:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: user-service-prod
  namespace: argocd-system
spec:
  project: production
  source:
    repoURL: https://github.com/ourcompany/platform-gitops
    path: applications/user-service
    targetRevision: release
    helm:
      valueFiles:
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: user-service-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

A couple of things worth noting:

• We use different Git branches for different environments. main for dev, release for production.
• prune: true automatically removes resources that are no longer in Git. This caught us off guard initially when we renamed some services.
• selfHeal: true fixes configuration drift, which happens more often than you'd think.

Integrating With Our CI Pipeline

GitOps doesn't replace CI/CD — it changes how the CD part works. Our Jenkins pipeline now looks like this:

• Developer pushes code
• Jenkins runs tests and builds Docker image
• Jenkins tags the image and pushes to our registry
• Jenkins updates the Helm values file with the new image tag
• ArgoCD notices the Git change and deploys automatically

The key insight was realizing that CI should update Git, not directly deploy to Kubernetes. It felt weird at first, but now it's natural.

Here's a snippet of how we update the image tag from Jenkins:

# Update the values file with new image tag

sed -i "s/tag: .*/tag: ${BUILD_NUMBER}/" applications/user-service/values-production.yaml
git add .
git commit -m "Deploy user-service build ${BUILD_NUMBER} to production"
git push origin release

Things That Bit Us (And How We Fixed Them)

• Secret management: Don't put secrets in Git. Seriously. We use Sealed Secrets now, which encrypts secrets that only the cluster can decrypt. External Secrets Operator is another good option.
• Resource limits: Make sure your Helm charts include proper resource requests and limits. We had some services that would consume entire nodes because we forgot to set limits in the templates.
• RBAC complexity: OpenShift's RBAC, combined with ArgoCD's project system, can get complicated. Document your permission model early and stick to it.
• Sync waves: When you have dependencies between services, use ArgoCD sync waves to control deployment order. We had databases trying to start before their persistent volumes were ready.
• Chart dependencies: Helm's dependency management works, but it's not obvious. Use Helm dependency update religiously, and version your chart dependencies properly.

What We Got Out of This

After six months of running everything through GitOps:

• Deployment time went from 30+ minutes of manual steps to under 5 minutes, fully automated
• We haven't had a single "oops, wrong environment" incident
• Rolling back deployments went from panic-inducing to boring (in a good way)
• Our audit compliance got way easier because everything is tracked in Git
• New developers can understand our deployment process by reading the repo

The biggest win wasn't technical — it was cultural. Our developers stopped being afraid of deployments. When deployment is just a Git commit, people are more willing to ship small, frequent changes.

Some Hard-Earned Advice

• Start small: Don't try to migrate everything at once. Pick one simple service and get the workflow solid before expanding.
• Invest in your Helm charts: Spend time making them configurable and well-documented. You'll be living with these templates for years.
• Monitor ArgoCD itself: ArgoCD going down means deployments stop. Set up proper monitoring and alerts.
• Train your team: The tooling is only as good as the people using it. Make sure everyone understands the new workflow.
• Plan for secrets: Figure out your secret management strategy early. Retrofitting it later is painful.
• Use preview environments: One unexpected benefit was being able to spin up temporary environments for feature branches. Great for testing before merge.

Looking Back

Building this GitOps pipeline wasn't just about adopting new tools — it fundamentally changed how we think about deployments. Git isn't just where we store code anymore; it's our deployment control plane.

The learning curve was real, and we definitely made mistakes along the way. But now that everything is running smoothly, I can't imagine going back to the old way of doing things. When someone asks "what version is running in production?", the answer is always "whatever's in the release branch."

If you're thinking about making this jump, my advice is simple: start with one application, get it working well, then expand from there. The investment in automation pays dividends every single day.

What's your experience been with GitOps? Hit me up in the comments if you've run into similar challenges or found better solutions.