# How I Built an In-Browser Secret Scanner: Client-Side Entropy & Heuristic Pattern Matching

> Detecting API keys, private certs, and bearer tokens in real-time before code touches Git or the wire.

> Platform: Engineering Deep Dive | Published: Published | Read time: 6 min read
> Canonical URL: https://www.arman-portfolio.online/blog/secret-scanner-deep-dive/

## Summary
A technical deep-dive into building zero-dependency, high-throughput secret detection directly in the browser using Shannon entropy heuristics, compiled regex tokenizers, and Web Worker thread isolation.

## Key Takeaways
- Static regular expressions miss 60%+ of non-standard API keys; Shannon Entropy H(X) solves this by quantifying randomness.
- Running multi-pattern tokenizers on large text buffers in the UI thread causes frame drops; Web Worker isolation preserves 60fps responsiveness.
- False positives (UUIDs, SHA256 hashes, base64 asset digests) are systematically eliminated using length boundary filters and character distribution checks.
- Client-side secret scanning complements CI/CD pre-commit hooks by intercepting credentials before they ever touch git history.

## 1. The Problem — Accidental Secrets in the Modern Web

During local development and rapid prototyping, API tokens, JWT private keys, and cloud credentials routinely slip into code snippets, test fixtures, and mock payloads. Once committed to git history, a secret is considered permanently compromised — requiring revocation, rotation, and incident triage. I wanted to build a zero-dependency, instantaneous tool that scans pasted code directly inside the browser before it leaves the developer's machine.



## 2. Shannon Entropy Algorithm (H(X) = -Σ P(x) log2 P(x))

Standard secret scanners rely entirely on known pattern prefixes (e.g., 'AKIA...' for AWS or 'ghp_...' for GitHub). But high-risk credentials like custom HMAC secrets, JWT private signing keys, and database passwords have no standard prefix. To catch these, the scanner computes the Shannon entropy of candidate strings:

```javascript
// Shannon Entropy calculation in JS
function calculateEntropy(str) {
  const len = str.length;
  if (len === 0) return 0;
  const frequencies = {};
  for (let ch of str) frequencies[ch] = (frequencies[ch] || 0) + 1;
  return Object.values(frequencies).reduce((entropy, count) => {
    const p = count / len;
    return entropy - p * Math.log2(p);
  }, 0);
}
```


## 3. Heuristic Filtering & False Positive Suppression

Raw entropy calculations flag everything that looks random — including Git SHA-1 hashes, UUID v4 strings, base64 CSS fonts, and random hex colors. To prevent alarm fatigue, the engine applies layered contextual heuristics:


- Known Hash Whitelisting: 32-char hex strings matching MD5 or 40-char SHA-1 with lowercase-only distributions are categorized as non-secret hashes.
- Character Set Diversity: Strings require a mixture of character classes (alphanumeric + symbol) or entropy >= 4.5 bits/char to trigger generic secret alerts.
- Delimiter & Keyword Context: Strings preceded by assignment operators and keywords (e.g., 'apiKey = ', 'SECRET=', 'PRIVATE_KEY') receive boosted confidence scores.

## 4. Web Worker Thread Isolation

Scanning 5,000+ lines of minified JavaScript or JSON dump against 20+ compiled regex patterns and entropy tokenizers takes 30–80ms. Executing this on the main JavaScript thread causes noticeable typing lag and UI freezes. By delegating the raw text buffer to a dedicated Web Worker via structured cloning, the UI remains perfectly responsive at 60 FPS while the worker streams back structured findings.



## 5. Defense-in-Depth & Production Integration

In-browser scanning is one layer of a complete DevSecOps strategy. In production pipelines, teams should enforce pre-commit hooks (Gitleaks, TruffleHog), automated SAST in CI/CD, and dynamic secret injection via HashiCorp Vault or AWS Secrets Manager so plaintext secrets never reside in codebases.



---
Author: Arman Ahemad Khan (https://www.arman-portfolio.online/)