Drift Detection
Detect and resolve configuration drift by comparing Git branch state and ArgoCD sync metadata.
What is Drift
In GitOpsHQ, drift has two distinct meanings depending on the level:
- Tenant / project drift is detected by comparing the Git
mainbranch against thedeployedbranch for each tenant-environment. When these branches diverge, changes have been committed but not yet deployed. This is Git-branch divergence, not a comparison of live cluster state versus Git. - Cluster drift is derived by merging persisted drift findings with synthetic rows generated from ArgoCD application and resource health/sync metadata. This is closer to an ArgoCD sync-status aggregation than a field-level resource comparison.
Drift can surface because of:
| Cause | Example |
|---|---|
| Undeployed commits | Values were changed on main but the delivery has not been promoted to the deployed branch |
| Failed syncs | ArgoCD fails to apply the latest manifests due to validation errors or resource conflicts |
| Partial rollbacks | A rollback reverts some but not all resources to a previous state |
| Never deployed | The deployed branch does not exist for a tenant-environment (neverDeployed status) |
GitOpsHQ surfaces drift at three levels: project, cluster, and tenant-environment.
Drift Detection Levels
1. Project Drift Summary
UI Location: Project detail page → header badge
The project-level drift summary provides an aggregate view across all tenants and environments in a project. It answers the question: "Is anything in this project out of sync?" Drift here means the main branch has commits that are not yet on the deployed branch for one or more tenant-environments.
| Badge | Meaning |
|---|---|
| No Drift (green) | main and deployed branches are identical for all tenant-environments |
| Drift Detected (yellow, with count) | N tenant-environments have diverged branches |
| Never Deployed (gray) | The deployed branch does not yet exist for this tenant-environment |
Clicking the badge opens a filtered view showing only the affected tenant-environment combinations.
2. Cluster Drift Findings
UI Location: Cluster detail page → Drift tab
Cluster-level drift merges persisted findings with synthetic rows generated from ArgoCD application and resource health/sync metadata. This gives an aggregated view of sync status across all projects and tenants deployed to the cluster.
The drift findings table includes:
| Column | Description |
|---|---|
| Application | ArgoCD application name |
| Project | Which project owns this resource |
| Tenant | Which tenant this resource belongs to |
| Environment | Which environment this resource is deployed in |
| Sync Status | ArgoCD sync status (Synced, OutOfSync, Unknown) |
| Health | ArgoCD health status (Healthy, Degraded, Progressing, Missing) |
| Detected At | When the finding was first observed |
3. Tenant Drift Status
UI Location: /projects/:slug/tenants/:tenantId → environment view
Per-tenant-environment drift shows whether the main branch has diverged from the deployed branch for this specific tenant-environment. This is the most granular view and the one most operators interact with daily.
Each tenant-environment row displays:
- Drift status: Whether
mainhas commits not yet ondeployed - Never deployed: Shown when the
deployedbranch does not exist - Last deploy time: When the last promotion to
deployedoccurred - Pending changes: Whether there are committed-but-not-yet-deployed changes on
main
How Drift is Detected
GitOpsHQ detects drift through two complementary mechanisms: Git branch comparison for tenant/project drift, and ArgoCD sync metadata aggregation for cluster drift.
Detection Sources
For tenant and project drift, GitOpsHQ compares the main branch against the deployed branch for each tenant-environment:
- Changes are committed to
mainby the delivery generator or by value edits - When a deployment is promoted, commits move to the
deployedbranch - If
mainis ahead ofdeployed, the tenant-environment is in drift - If the
deployedbranch does not exist, the status isneverDeployed
This approach detects undeployed configuration changes without requiring live cluster state.
For cluster-level drift, GitOpsHQ merges persisted findings with synthetic rows from ArgoCD:
- Synced: ArgoCD confirms the live state matches the desired state
- OutOfSync: ArgoCD detects differences between the Git revision and live state
- Progressing: A sync operation is in progress
- Degraded: Resources exist but are unhealthy (CrashLoopBackOff, pending pods)
ArgoCD application and resource health metadata flows through the agent to the hub.
Drift Indicators in the UI
Drift indicators appear in multiple locations to ensure visibility regardless of where you are navigating.
Dashboard
The environment matrix on the dashboard shows drift as colored indicators per project-environment cell. Yellow indicates branch divergence; red indicates failed syncs or degraded health.
Project Detail
The project header shows an aggregate drift badge. The environments tab shows per-environment drift counts based on main vs deployed branch comparison.
Tenant Environment View
Each tenant-environment row shows whether pending changes exist (branch divergence) or whether the environment has never been deployed.
Cluster Detail
The cluster's drift tab shows findings merged from persisted data and ArgoCD sync/health metadata across all projects and tenants deployed to that cluster.
Resolving Drift
When drift is detected, GitOpsHQ provides multiple resolution paths depending on the cause and intent.
Review the Drift
For tenant/project drift, check which commits are on main but not on deployed. The project detail page shows the divergence.
For cluster drift, review the ArgoCD sync status and health information on the cluster drift tab.
Choose a Resolution
When to use: The changes on main are intentional and should go live.
Trigger a deployment to promote the main branch content to the deployed branch. This clears the tenant/project drift.
When to use: Cluster drift shows ArgoCD is out of sync.
Trigger a Sync command to force ArgoCD to reconcile the cluster with the desired state.
Action: Cluster Command → Sync
Target: Specific ArgoCD Application
Effect: Cluster state is reconciled to match GitWhen to use: The drift is part of a broader failed deployment.
Trigger a Rollback command to revert the ArgoCD application to a previous known-good revision. This restores both the desired and actual state to a previous point.
Verify Resolution
After deploying, the main and deployed branches converge and tenant/project drift clears. For cluster drift, a successful sync updates the ArgoCD status on the next agent report cycle.
Drift in Release Workflow
When creating or deploying a release, the UI surfaces existing drift that might be affected.
Deploying a release on top of existing branch divergence is allowed but may produce unpredictable results if the pending changes conflict with the release. Review drift status before major releases when possible.
Cluster Commands for Drift Resolution
Commands are dispatched from the hub to the cluster agent for execution. The following commands are relevant to drift resolution:
| Command | Description | Approval Required |
|---|---|---|
| Sync | Force ArgoCD to reconcile a specific application | Configurable per environment |
| Sync All | Reconcile all applications in a project-environment | Configurable per environment |
| Rollback | Revert an ArgoCD application to a specific revision | Always (if approval policies enabled) |
| Hard Refresh | Clear ArgoCD cache and re-read manifests from Git | No |
| Restart | Rolling restart of a deployment (useful for config reload) | Configurable per environment |
Command Execution Flow
Drift Prevention Best Practices
| Practice | Recommendation |
|---|---|
| Avoid manual kubectl changes | All changes should flow through GitOpsHQ → Git → ArgoCD. Document this as a team policy. |
| Use ArgoCD self-heal | Enable ArgoCD's selfHeal option to automatically revert manual changes. |
| Scope HPA carefully | If using HPA, exclude replica count from ArgoCD diff tracking to avoid false drift signals. |
| Monitor external controllers | Identify and document all controllers that modify resources (VPA, cert-manager, external-dns). Configure drift detection to ignore their managed fields. |
| Review drift regularly | Check the dashboard weekly for persistent drift. Chronic drift often indicates a configuration gap. |
| Use environment freezes | Freeze production environments during maintenance windows to prevent accidental drift from automated processes. |
Related Pages
- Core Architecture — Delivery philosophy and agent architecture
- Dashboard — Dashboard drift indicators
- Tenant Management — Tenant-level drift tracking
- Environment Compare — Compare configurations to identify potential drift sources