Blocking Secret Leaks in CI/CD: Shannon Entropy, High-Precision Patterns, and In-Diff Checks

The 3-Second Threat: How Leaked Secrets Are Exploited

Automated threat actors scan GitHub's public event stream (`https://api.github.com/events`) in real time. Research demonstrates that when an AWS access key or OpenAI API secret is committed to a public repository: - Automated botnets detect the key within **2 to 4 seconds**. - Unauthorized API calls (such as provisioning high-memory GPU instances for crypto-mining) occur within **60 to 90 seconds**.

Removing the key in a subsequent commit does **not** solve the problem. The credential remains permanently embedded in the Git directed acyclic graph (DAG) and reachable via historical commit trees.

The Dual-Engine Approach: Regex Patterns + Shannon Entropy

Effective secret scanning cannot rely on static pattern matching alone. A platform must balance two opposing challenges: - **False Negatives**: Missing custom API tokens or database passwords that don't match standard prefixes. - **False Positives**: Flagging benign UUIDs, Git commit hashes, and base64-encoded SVG strings as passwords.

To solve this, DevAudit pairs strict high-precision signature expressions with mathematical **Shannon Entropy analysis**.

Understanding Shannon Entropy

Named after Claude Shannon, entropy measures the degree of randomness or information density in a sequence of characters:

\[ H(X) = - \sum_{i=1}^{n} P(x_i) \log_2 P(x_i) \]

Where \(P(x_i)\) is the probability of character \(x_i\) appearing in the string.

• Predictable source code identifiers (`userFirstName`, `handleHttpRequest`) have low entropy (typically \(H < 3.2\)).
• High-security cryptographic keys and random passwords have high entropy (typically \(H > 4.5\) for Base64 and \(H > 3.8\) for Hexadecimal).

// Shannon Entropy computation engine
export function calculateShannonEntropy(str: string): number {
  if (!str) return 0;
  
  const frequencies: Record<string, number> = {};
  for (const char of str) {
    frequencies[char] = (frequencies[char] || 0) + 1;
  }

  let entropy = 0;
  const len = str.length;

  for (const char in frequencies) {
    const probability = frequencies[char] / len;
    entropy -= probability * Math.log2(probability);
  }

  return entropy;
}

High-Precision Secret Signatures

Certain platforms provide prefix guarantees that eliminate false positives:

When a token matches a known provider prefix, DevAudit immediately flags it with 100% confidence. For unformatted assignments (e.g., `DATABASE_PASSWORD = "..."`), the assignment is validated against the Shannon entropy threshold.

In-Diff PR Checking: Stop Secrets Before Merge

The most effective place to stop a secret leak is **before the commit merges into the default branch**.

With the DevAudit PR Guard, checks run ephemerally across pull request diffs:

# Example: Blocking secrets in CI/CD pipeline
name: DevAudit PR Security Guard
on: [pull_request]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run Secret & Dependency Guard
        run: |
          curl -X POST https://auditweb.onlinecloud.in/api/scan-repo \
            -H "Content-Type: application/json" \
            -d '{"repo": "${{ github.repository }}", "pr": ${{ github.event.pull_request.number }}}'

What to Do If You Leak a Secret

1. **Revoke Immediately**: Invalidate the key in the provider dashboard (AWS IAM, Stripe, OpenAI). Do not wait to purge Git history first.
2. **Audit Access Logs**: Inspect CloudTrail, Stripe audit logs, or server auth logs to determine if unauthorized operations occurred.
3. **Purge Git History**: Use `git-filter-repo` or BFG Repo-Cleaner to rewrite commits:

   git filter-repo --replace-text expressions.txt
   git push origin --force --all