From 685270b0ad2674998fc2b8d05a1c97c65af6f39d Mon Sep 17 00:00:00 2001 From: Miccah Date: Thu, 23 Apr 2026 10:52:59 -0700 Subject: [PATCH] Document SecretParts contract in detector-authoring docs (#4912) * Rename AnalysisInfo field to SecretParts on detectors.Result Mechanical rename of the detectors.Result.AnalysisInfo field to SecretParts to prepare for its replacement for Raw. * Document SecretParts contract in detector-authoring docs Adds a "Populating SecretParts" section to both the external and internal detector-authoring guides. Covers what SecretParts is, the rule that every Result must populate it, and the shape for single-part vs multi-part credentials with code examples. Establishes "key" as the convention for single-part detectors (matches ~48 existing detectors vs ~13 using "token"). --- hack/docs/Adding_Detectors_Internal.md | 50 ++++++++++++++++++++++++-- hack/docs/Adding_Detectors_external.md | 49 +++++++++++++++++++++++-- 2 files changed, 93 insertions(+), 6 deletions(-) diff --git a/hack/docs/Adding_Detectors_Internal.md b/hack/docs/Adding_Detectors_Internal.md index f24148e48..c1951feb4 100644 --- a/hack/docs/Adding_Detectors_Internal.md +++ b/hack/docs/Adding_Detectors_Internal.md @@ -16,7 +16,9 @@ The purpose of Secret Detectors is to discover secrets with exceptionally high s - [Development Guidelines](#development-guidelines) - [Development Dependencies](#development-dependencies) - [Creating a new Secret Scanner](#creating-a-new-secret-detector) + - [Populating SecretParts](#populating-secretparts) - [Addendum](#addendum) + - [Verification indeterminacy](#verification-indeterminacy) - [Managing Test Secrets](#managing-test-secrets) - [Setting up Google Cloud SDK](#setting-up-google-cloud-sdk) @@ -80,15 +82,57 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or 2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/). 3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not. * Make sure you understand [verification indeterminacy](#verification-indeterminacy). - 4. Update the tests with these test cases at minimum: + 4. Populate `SecretParts` on every `Result` your detector emits. See [Populating SecretParts](#populating-secretparts). + 5. Update the tests with these test cases at minimum: 1. Found and verified (using a credential loaded from GCP Secrets) 2. Found and unverified (determinately, i.e. the secret is invalid) 3. Found and unverified (indeterminately due to timeout) 4. Found and unverified (indeterminately due to an unexpected API response) 5. Not found 6. Any false positive cases that you come across - 5. Add your new detector to DefaultDetectors in `/pkg/engine/defaults/defaults.go` - 6. Create a merge request for review. CI tests must be passing. + 6. Add your new detector to DefaultDetectors in `/pkg/engine/defaults/defaults.go` + 7. Create a merge request for review. CI tests must be passing. + +## Populating SecretParts + +`SecretParts` is the structured source of truth for the credential components a detector found. It is a `map[string]string` on [`detectors.Result`](/pkg/detectors/detectors.go) that stores each part of the credential under a descriptive key. Downstream consumers (analyzers and enterprise TruffleHog) rely on it. + +**Every `Result` your detector emits must populate `SecretParts`.** Populate it whether or not the secret is verified. + +### Single-part credentials + +Most detectors find a single opaque token. Use one entry keyed by `"key"` — this is the established convention in the codebase: + +```go +s1 := detectors.Result{ + DetectorType: detector_typepb.DetectorType_Example, + Raw: []byte(match), + SecretParts: map[string]string{"key": match}, +} +``` + +### Multi-part credentials + +When the credential has more than one component (e.g. AWS access-key + secret-access-key, OAuth client-id + client-secret, or a token bound to an endpoint/host), use one entry per part with descriptive keys: + +```go +s1 := detectors.Result{ + DetectorType: detector_typepb.DetectorType_Example, + Raw: []byte(accessKeyID), + RawV2: []byte(accessKeyID + secretAccessKey), + SecretParts: map[string]string{ + "access_key_id": accessKeyID, + "secret_access_key": secretAccessKey, + }, +} +``` + +### Key-naming guidance + +- Single-part: use `"key"`. This matches the bulk of existing detectors in `pkg/detectors/`. +- Multi-part: pick descriptive, lowercase, `snake_case` keys that name each part (e.g. `client_id`, `client_secret`, `access_key_id`, `secret_access_key`, `username`, `password`, `domain`, `host`, `endpoint`). If the detector also has a corresponding analyzer in `pkg/analyzer/analyzers/`, the keys must match what the analyzer expects, since analyzers read directly from this map. +- Only secret parts that uniquely identify the credential belong in `SecretParts`. Unrelated metadata belongs in `ExtraData`. +- This field is the source of truth for uniquely identifying a credential. ## Addendum diff --git a/hack/docs/Adding_Detectors_external.md b/hack/docs/Adding_Detectors_external.md index dc4d0d001..cf338db9f 100644 --- a/hack/docs/Adding_Detectors_external.md +++ b/hack/docs/Adding_Detectors_external.md @@ -17,7 +17,9 @@ The purpose of Secret Detectors is to discover secrets with exceptionally high s + [Development Dependencies](#development-dependencies) + [Creating a new Secret Detector](#creating-a-new-secret-detector) + [Testing the Detector](#testing-the-detector) + * [Populating SecretParts](#populating-secretparts) * [Addendum](#addendum) + + [Verification indeterminacy](#verification-indeterminacy) + [Adding Protos in Windows](#adding-protos-in-windows) ## Getting Started @@ -82,9 +84,10 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or 1. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/). 2. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not. * Make sure you understand [verification indeterminacy](#verification-indeterminacy). - 3. Create a [test for the detector](#testing-the-detector). - 4. Add your new detector to DefaultDetectors in `/pkg/engine/defaults/defaults.go`. - 5. Create a pull request for review. + 3. Populate `SecretParts` on every `Result` your detector emits. See [Populating SecretParts](#populating-secretparts). + 4. Create a [test for the detector](#testing-the-detector). + 5. Add your new detector to DefaultDetectors in `/pkg/engine/defaults/defaults.go`. + 6. Create a pull request for review. ### Testing the Detector To ensure the quality of your PR, make sure your tests are passing with verified credentials. @@ -128,6 +131,46 @@ To ensure the quality of your PR, make sure your tests are passing with verified If the tests are passing, feel free to open a PR! +## Populating SecretParts + +`SecretParts` is the structured source of truth for the credential components a detector found. It is a `map[string]string` on [`detectors.Result`](/pkg/detectors/detectors.go) that stores each part of the credential under a descriptive key. Downstream consumers (analyzers, Secret Storage, and — in future work — the `Raw`/`RawV2` mapping layer and dedup hashing) rely on it. + +**Every `Result` your detector emits must populate `SecretParts`.** Populate it whether or not the secret is verified. + +### Single-part credentials + +Most detectors find a single opaque token. Use one entry keyed by `"key"` — this is the established convention in the codebase: + +```go +s1 := detectors.Result{ + DetectorType: detector_typepb.DetectorType_Example, + Raw: []byte(match), + SecretParts: map[string]string{"key": match}, +} +``` + +### Multi-part credentials + +When the credential has more than one component (e.g. AWS access-key + secret-access-key, OAuth client-id + client-secret, or a token bound to an endpoint/host), use one entry per part with descriptive keys: + +```go +s1 := detectors.Result{ + DetectorType: detector_typepb.DetectorType_Example, + Raw: []byte(accessKeyID), + RawV2: []byte(accessKeyID + secretAccessKey), + SecretParts: map[string]string{ + "access_key_id": accessKeyID, + "secret_access_key": secretAccessKey, + }, +} +``` + +### Key-naming guidance + +- Single-part: use `"key"`. This matches the bulk of existing detectors in `pkg/detectors/`. +- Multi-part: pick descriptive, lowercase, `snake_case` keys that name each part (e.g. `client_id`, `client_secret`, `access_key_id`, `secret_access_key`, `username`, `password`, `domain`, `host`, `endpoint`). If the detector also has a corresponding analyzer in `pkg/analyzer/analyzers/`, the keys must match what the analyzer expects, since analyzers read directly from this map. +- Only secret parts that uniquely identify the credential belong in `SecretParts`. Unrelated metadata belongs in `ExtraData`. +- This field is the source of truth for uniquely identifying a credential. ## Addendum