Observability

Health Endpoints

GitOpsHQ exposes standard health and metrics endpoints for integration with load balancers, orchestrators, and monitoring systems.

Liveness vs Readiness

• /healthz (liveness) — Returns 200 as long as the GitOpsHQ process is running and responsive. A failure here means the process should be restarted. This endpoint does not check external dependencies.
• /readyz (readiness) — Returns 200 when the service is fully initialized and can handle requests. During startup, this returns 503 until database connections, cache, and agent communication channels are established.

# Kubernetes probe configuration example
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 15
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Dashboard Signals

The GitOpsHQ dashboard organizes health signals into four categories. Each provides a different lens on your operational state.

Key Metrics to Monitor

GitOpsHQ exposes the following metrics via the /metrics endpoint in Prometheus format.

Delivery Metrics

Governance Metrics

Runtime Metrics

Automation Metrics

Daily Monitoring Checklist

Run through this checklist at the start of each operational shift.

Integration with Observability Platforms

Prometheus + Grafana

GitOpsHQ's /metrics endpoint is compatible with standard Prometheus scraping.

# prometheus.yml scrape config
scrape_configs:
  - job_name: 'gitopshq'
    metrics_path: '/metrics'
    scheme: 'https'
    bearer_token_file: '/etc/prometheus/gitopshq-token'
    static_configs:
      - targets: ['api.gitopshq.io:443']
    scrape_interval: 30s

Recommended Grafana dashboards:

Datadog

Export GitOpsHQ metrics to Datadog via Prometheus-to-Datadog bridge or the OpenMetrics integration.

# Datadog agent config
instances:
  - prometheus_url: https://api.gitopshq.io/metrics
    namespace: gitopshq
    metrics:
      - gitopshq_releases_total
      - gitopshq_agent_heartbeat_age_seconds
      - gitopshq_drift_detected_total
    headers:
      Authorization: Bearer <GITOPSHQ_TOKEN>

PagerDuty / Opsgenie

Configure webhook notifications in GitOpsHQ to send alerts to your incident management platform:

• Critical: Agent disconnected > 5 min, production deployment failed, break-glass session opened
• Warning: Production drift detected, approval queue > 1 hour, rollback executed
• Info: Release deployed successfully, promotion completed

Distributed Tracing

GitOpsHQ supports OpenTelemetry Protocol (OTLP) for distributed tracing. Every API request generates a trace with spans for:

• Request authentication and authorization
• Policy evaluation (OPA + approval checks)
• Delivery generation (template rendering, Git commit)
• Agent communication (sync trigger, health check)

Configure the OTLP exporter endpoint in GitOpsHQ settings to send traces to your Jaeger, Zipkin, or cloud tracing backend. Traces include the X-Correlation-ID header value for cross-system correlation.

Alerting Recommendations

Critical Alerts (Page Immediately)

Warning Alerts (Investigate Within 1 Hour)

Informational Alerts (Review Daily)

Signal Views Deep Dive

Delivery Health

Delivery health tells you whether your deployment pipeline is functioning correctly.

• Release progression: Track releases as they move through Draft → Approved → Deployed → Healthy. Bottlenecks appear as accumulation at any stage.
• Promotion lead time: The time from the first environment deployment to the final production deployment. Long lead times indicate governance overhead or testing gaps.
• Rollback volume: A sudden increase in rollbacks signals a quality problem in the development or testing process. Investigate the root cause rather than just rolling back.

Runtime Health

Runtime health tells you whether your live clusters match the desired state.

• Connected/disconnected trend: A gradual increase in disconnected clusters may indicate network policy changes, certificate expiration, or agent version drift.
• Drift correlation: Drift that appears immediately after a deployment may indicate the deployment was partially applied. Drift that appears gradually may indicate manual cluster edits or operator interference.
• Sync timing: If syncs are consistently slow, check cluster resource availability and agent pod health.

Governance Health

Governance health tells you whether your policies are effective and not creating unnecessary friction.

• Approval aging: If approvals consistently take too long, consider adjusting quorum requirements or adding more approved reviewers.
• Break-glass frequency: More than one break-glass session per month suggests a gap in normal operational capabilities. Review policies and permissions.
• OPA evaluation trend: A decreasing denial rate over time indicates teams are learning and adapting to the policies. A persistent high denial rate may mean the policy is too restrictive or unclear.