Secrets Management Best Practices for DevOps Pipelines

Secrets sprawl refers to credentials distributed across multiple systems with no central inventory, rotation, or audit trail. It accumulates over years because the path of least resistance in development is to hardcode or copy-paste credentials.

Common locations where secrets hide:

• Source code repositories (the obvious one — but also git history)
• CI/CD pipeline environment variables (Jenkins, GitHub Actions, GitLab CI)
• Container images baked into layers
• Configuration management tooling (Ansible vaults with weak keys, Chef encrypted data bags)
• Developer laptops in .env files, shell history, SSH config
• Slack, Jira, Confluence (secrets pasted into tickets or messages)
• Build artifacts and Docker image layers
• Log files from applications that print connection strings

The git history problem: Removing a secret from the current HEAD of a repository does not remove it from git history. An attacker with repo access can retrieve it from any prior commit. Purging secrets from git history requires git filter-repo (the maintained successor to BFG Repo Cleaner) and force-pushing — a destructive operation that rewrites history for every branch. Prevention is far less painful than remediation.

The rotation problem: Secrets that never rotate have indefinite exposure windows. If a credential was leaked two years ago and you never knew, that credential is still active and potentially being used by a threat actor. Rotation without a secrets management platform is a manual, error-prone process that most teams avoid — which is why static long-lived credentials dominate enterprise environments.

A complete secrets management architecture has four components: centralized storage, access control, dynamic secrets, and audit logging.

Centralized storage: All secrets live in one system of record with encryption at rest. No secrets in environment variables checked into repos, no plaintext config files, no shared password documents.

Fine-grained access control: Applications and pipelines are granted access to only the secrets they need, authenticated by identity (cloud IAM role, Kubernetes service account, CI/CD identity token) rather than a shared static credential. If a CI/CD pipeline only needs to deploy to a staging environment, it should not have production database credentials.

Dynamic secrets: Instead of storing a long-lived database password, the secrets manager generates a temporary credential valid for the duration of a job or session, then revokes it automatically. The credential that was used is never used again. This eliminates the exposure window entirely for supported backends.

Comprehensive audit logging: Every secret read, write, rotation, and access denial is logged with identity, timestamp, and source IP. This is the forensic record that lets you answer: who accessed production database credentials at 3 AM last Tuesday?

Integration pattern for applications: The application authenticates to the secrets manager using its platform identity (IAM role, workload identity), retrieves secrets at startup or on demand, and never caches them to disk. The secrets manager handles rotation transparently for supported backends.

HashiCorp Vault (HCP Vault or self-hosted): The de facto standard for multi-cloud and hybrid environments. Supports dynamic secrets for databases, AWS, Azure, GCP, SSH, PKI, and Kubernetes. Fine-grained policy engine based on HCL. Multiple authentication backends: AWS IAM, Azure AD, GCP service accounts, Kubernetes service accounts, LDAP, AppRole, JWT/OIDC. Audit log shipped to Splunk, Elasticsearch, or syslog. Self-hosted requires operational expertise (HA setup, storage backend, seal management). HCP Vault (managed) removes operational overhead at a cost.

Best fit: Organizations with multi-cloud deployments, Kubernetes-heavy environments, or complex dynamic secrets requirements.

AWS Secrets Manager: Native integration with the AWS ecosystem. Automatic rotation for supported AWS services (RDS, Redshift, DocumentDB) via Lambda rotation functions. IAM-based access control using resource-based policies. No separate infrastructure to manage. Replication across regions for disaster recovery. Cross-account access via IAM roles. At $0.40 per secret per month, cost scales with secret count.

Best fit: AWS-centric organizations that want managed secrets without operational overhead.

Azure Key Vault: Equivalent to AWS Secrets Manager in the Azure ecosystem. Stores secrets, keys, and certificates in one service. Managed HSM option for FIPS 140-2 Level 3 hardware-backed keys. Integration with Azure Policy for compliance enforcement. Soft-delete and purge protection prevent accidental or malicious deletion.

Best fit: Azure-centric organizations, especially those with compliance requirements for hardware-backed key storage.

CyberArk Conjur (open source) / CyberArk Secrets Manager: Strong PAM heritage — better fit for organizations that need to manage secrets alongside privileged accounts. Conjur open source provides a Vault-like secrets API. Full CyberArk integration adds PAM capabilities (session recording, just-in-time access). Common in regulated industries.

Best fit: Enterprises with existing CyberArk PAM deployments or strict regulatory requirements for privileged credential management.

Google Cloud Secret Manager: Equivalent to AWS Secrets Manager for GCP workloads. IAM-integrated, regional replication, automatic version management, and event notifications via Pub/Sub when secrets are updated.

Dynamic secrets are temporary credentials generated on demand and automatically revoked after use or expiration. They are the single most impactful secrets management control because they eliminate the concept of a long-lived credential that can be leaked and reused.

How Vault dynamic secrets work for databases:

1. Vault connects to your database using a long-lived root credential (stored in Vault, never exposed to applications)
2. Application authenticates to Vault using its platform identity
3. Vault generates a new database user with a time-limited TTL (e.g., 1 hour)
4. Application connects to the database using the ephemeral credential
5. TTL expires; Vault revokes the credential
6. The credential that was used will never exist again

Even if an attacker intercepts the credential in transit, it is useless after the TTL expires. There is no rotation needed because there is nothing to rotate.

Supported backends for Vault dynamic secrets:

• Databases: PostgreSQL, MySQL, Oracle, MSSQL, MongoDB, Cassandra, Elasticsearch
• Cloud IAM: AWS (STS-based IAM credentials), Azure (service principal passwords), GCP (service account keys)
• PKI: Short-lived X.509 certificates signed by Vault's CA
• SSH: Signed SSH certificates for just-in-time server access
• Kubernetes: Short-lived service account tokens

AWS Secrets Manager rotation for RDS: For teams not using Vault, AWS Secrets Manager with automatic rotation achieves similar results for RDS-backed credentials. A Lambda function rotates the password on a schedule (every 30 or 90 days), transparently for applications using the SDK. Not truly dynamic (the credential persists between rotations) but eliminates the set-it-and-forget-it problem.

The CI/CD pipeline is the highest-risk location for secrets because it has legitimate access to production systems and its configuration is often version-controlled and widely readable.

GitHub Actions — native secrets: GitHub's built-in secrets are encrypted environment variables injected at workflow runtime. They do not appear in logs (redacted). Organization and repository-level secrets with environment-level restrictions. For a complete secrets management solution, use the HashiCorp Vault GitHub Action or AWS credentials action with OIDC:

# GitHub Actions + Vault via OIDC (no long-lived credentials)
jobs:
  deploy:
    permissions:
      id-token: write  # Required for OIDC
    steps:
      - uses: hashicorp/vault-action@v3
        with:
          url: https://vault.example.com
          method: jwt
          role: github-deploy
          secrets: |
            secret/data/prod/db password | DB_PASSWORD

The workflow authenticates to Vault using a JWT token signed by GitHub — no stored credentials anywhere.

AWS with GitHub Actions OIDC:

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/github-deploy
    aws-region: us-east-1
    # No access keys — role assumed via OIDC

GitLab CI: GitLab supports native CI/CD variables (masked and protected) and native Vault integration via JWT authentication. GitLab Premium adds external secret management with Vault and Azure Key Vault.

Jenkins: Use the HashiCorp Vault Plugin or AWS Secrets Manager Credentials Provider. Avoid Jenkins' built-in credentials store for production secrets — it lacks the audit logging and rotation capabilities of a dedicated secrets manager.

What never belongs in CI/CD:

• Long-lived AWS IAM access keys (use OIDC-based role assumption)
• Database passwords (use dynamic secrets or short-lived credentials)
• Private keys for code signing or deployment (use short-lived certificates)
• API tokens with no expiration (use short-lived tokens or OAuth client credentials flow)

Kubernetes Secrets are base64-encoded, not encrypted. By default, they are stored in plaintext in etcd. Anyone with read access to the namespace — or to the etcd backup — can decode every secret. This is a widespread misconception: many teams believe Kubernetes Secrets are secure because they are not plaintext in YAML. They are not.

Minimum baseline for Kubernetes secrets:

1. Enable etcd encryption at rest (EncryptionConfiguration with AES-CBC or AES-GCM)
2. Enable RBAC — restrict get and list on Secrets to only the service accounts and humans that need them
3. Never mount secrets as environment variables (visible in kubectl describe pod and process environment); use volume mounts with projected volumes instead
4. Audit every Secret read in the Kubernetes audit log

External Secrets Operator (recommended): ESO synchronizes secrets from external providers (Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) into Kubernetes Secrets automatically. Applications continue to consume secrets as native Kubernetes Secrets, but the source of truth is your secrets manager:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: prod-db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: secret/prod/database
        property: password

Vault Agent Injector / Vault Secrets Operator: Vault's Kubernetes integration injects secrets directly into pods via a sidecar (Agent Injector) or manages synchronization via an operator. Supports dynamic secrets with automatic renewal before TTL expiration.

Prevention is the goal, but detection is the safety net.

Pre-commit scanning: Install detect-secrets or trufflehog as a pre-commit hook to block commits containing credential patterns before they reach the repository:

pip install detect-secrets
detect-secrets scan > .secrets.baseline
# Add to .pre-commit-config.yaml
- repo: https://github.com/Yelp/detect-secrets
  rev: v1.5.0
  hooks:
    - id: detect-secrets
      args: ['--baseline', '.secrets.baseline']

Repository scanning:

• GitHub Secret Scanning: Automatic scanning for 200+ token patterns from major providers. Push protection blocks commits containing detected secrets. Free for public repos; available for private repos on GitHub Advanced Security.
• GitGuardian: More comprehensive pattern coverage than GitHub's native scanning. Historical scanning across all branches and git history. Integration with Slack, Jira, and PagerDuty for alerting.
• TruffleHog: Open-source, scans git history with entropy analysis plus regex patterns. Run in CI/CD to catch secrets before merging.

Runtime monitoring: CSPM tools (Wiz, Orca) scan cloud environments for exposed credentials in S3 buckets, environment variables in Lambda functions, EC2 instance metadata exposure, and IAM keys that have not been used or rotated. Set CloudTrail alerts for IAM key usage from unexpected regions or at unusual times — a reliable signal of compromised static credentials.