# FLOSS Migration: Implementation Spec for an Autonomous Agent

**Source:** `FLOSS Migration Document 8_10_26.md` (the single reference)
**Status:** Ready to implement
**Target repo:** `mandiant/flare-floss`, branches `quantumstrand` (primary) and `master`

> All file paths below are relative to the flare-floss repo root. The target repo is **not** present in this working directory; the implementing agent must run inside a checkout of the repo.


## 0. How to use this document

This spec is written for an agent that will implement the FLOSS migration. Each task is numbered, self-contained, and has:

- **Goal**: what to change.
- **Files**: the files most likely affected (agent must confirm by reading the repo).
- **Acceptance criteria**: how to verify the change is done.

Work through the phases in order. Phases have hard dependencies (Phase 1 must precede Phase 2–5). Phase 6 items are independent and may be done last. Commit after each phase with a clear message.

**Golden rule:** the behavior and decisions in the source document are normative. Where this spec makes an explicit "Decision:", that decision resolves an open question in the source document and is binding for this implementation.


## 1. Goals (from source §3.1)

1. Improve usability for human analysts and AI agents.
2. Keep the flag count small.
3. Keep each flag independent from the others.

Personas:
- **Human analyst:** interactive-style tree navigation, hierarchical section/structure grouping, in-tree regex search that preserves headings, automatic noise reduction, single option to toggle metadata columns.
- **AI agent:** token-efficient summary schemas, deterministic string-count limits, line-delimited streaming output for ingestion pipelines, structured error objects on STDERR when machine output is active.

## 2. Scope and non-goals

### 2.1 In scope
- Static-string layout reconstruction and tag-aware output (tree/grouped/flat views).
- CLI redesign: removals, renames, new filter flags, output flags.
- Machine-readable output: `--json`, `--json-summary`, TOON rendering, structured JSON errors.
- Result caching layer.
- Deferred cleanups that are safe to do now (source §2.6).

### 2.2 Non-goals (do NOT do now)
- Layout context or tags for **recovered** (non-static) strings. Stack, tight, decoded, and language strings keep emulation/parser extraction (source §1.1, §3.6). Tagging/layout for them is a TODO roadmap only (§3.6.2).
- Shell completion scripts (`zsh`, `bash`, `fish`): post-GSoC, tracked in issue #1350.
- Reintroducing `Runtime.layout`/`Runtime.tags` timing fields: decision deferred until after QS merge and before auto DB build (source §2.1).
- Consolidating render modules / removing classic view: happens in the PR that removes the classic renderer (source §2.3).


## 3. Target flag reference (authoritative)

### 3.1 Positional arguments
| Flag | Type | Notes |
|---|---|---|
| `sample` | path | Target binary to analyze. Required unless `--load` data is supplied. |

### 3.2 Core options
| Flag | Values | Default | Notes |
|---|---|---|---|
| `-n`, `--minimum-length` | int | `4` | Minimum string length. |
| *(no arguments)* | |: | Print simple help; **exit code 1**. |
| `--help` | |: | Print advanced options; **exit code 0**. |
| `--version` | |: | Print version; exit. |

### 3.3 Analysis and extraction options
| Flag | Values | Default | Notes |
|---|---|---|---|
| `-t`, `--string-type` | `static, stack, tight, decoded, language, all` | `all` | Select string types. Multi-value = OR. |
| `--no-string-type` | same | | Exclude string types. Multi-value = OR. |

> **Decision (source §3.7.3 review):** keep both the `-t` short flag and the `--string-type` long flag. The short flag is a convenience; the long form is primary in docs/help.

### 3.4 Layout and structure options
| Flag | Values | Default | Notes |
|---|---|---|---|
| `--section` | section names, e.g. `.rdata`, `.text` | | Restrict static strings to these sections. OR across values. |
| `--no-section` | section names | | Exclude strings in these sections. |
| `--structure` | structure names, e.g. `import_table`, `pe_header` | | Restrict static strings to these structures. |
| `--no-structure` | structure names | | Exclude strings in these structures. |

### 3.5 Tagging, searching, noise filtering
| Flag | Values | Default | Notes |
|---|---|---|---|
| `-Q`, `--query` | regex | | Filter strings by regex, preserving tree/grouped headings. Repeated flag = AND of all expressions. |
| `--tag` | tag names, e.g. `winapi`, `crypto`, `crt`, `oss` | | Emit only strings matching these tags. |
| `--no-tag` | tag names | | Suppress strings matching these tags. Only way to disable tag evaluation. |
| `--interesting` | |: | Shortcut equivalent to `--no-tag #common #duplicate #code #reloc #code-junk`. |
| `--max-strings` | int | | Cap emitted strings per section to top-N by relevance. |

> **Decision (source §3.3.2 note):** the phrase "Exclusions strictly use `--no <type>`" means the `--no-*` flag family (`--no-string-type`, `--no-section`, `--no-structure`, `--no-tag`). There is no generic `--no` flag.

### 3.6 Output and rendering options
| Flag | Values | Default | Notes |
|---|---|---|---|
| `--view` | `tree, grouped, flat` | `tree` if layout present, else `flat` | Terminal display format. |
| `--columns` | `tags, offset, structure, encoding` | `tags, offset` | Comma-separated column list for flat view (and per §8 decision, tree view). Omit `tags` to hide tag column. |
| `-j`, `--json` | |: | Full structured ResultDocument. |
| `--json-summary` | |: | Token-efficient JSON summary for LLM prompts. Independent of `--json`. |
| `--toon` | |: | Token-optimized compact table (see §6.4). |
| `-v`, `--verbose` | |: | Verbose: cache hits, analysis metadata, offsets. Keep simple. |
| `--color` | `auto, always, never` | `auto` | ANSI color. |

### 3.7 Advanced / decoder control
| Flag | Values | Default | Notes |
|---|---|---|---|
| `-f`, `--format` | `auto, pe, sc32, sc64` | `auto` | Sample file format. |
| `--language` | `auto, go, rust, none` | `auto` | Language-specific extractor. |
| `-l`, `--load` | path | | **DEPRECATED; remove.** |
| `--analyze-functions` | addresses | | Restrict emulation to these function addresses. |
| `--signatures` | path | | Custom FLIRT `.sig`/`.pat` file or directory. |
| `-L`, `--large-file` | |: | Allow files > 100 MB. |
| `--disable-progress` | |: | Disable progress bars. |

### 3.8 Logging and cache options
| Flag | Values | Default | Notes |
|---|---|---|---|
| `-d`, `--debug` | |: | Debug output on STDERR. |
| `-q`, `--quiet` | |: | Suppress STDOUT status except fatal errors. |
| `--clean` | |: | Renamed from `--noise-reduction` (see §7.1). |
| `--cache-dir` | path | platform cache dir | Overrides `FLOSS_CACHE_DIR`. |
| `--no-cache` | |: | Disable cache reads/writes (CI, one-offs). |
| `--prune-cache` | |: | Evict stale/expired cache entries and exit. |

### 3.9 Removed flags (must error as unknown)
| Removed | Replacement |
|---|---|
| `--exclude-aliases` | `--no-tag` (suppression) |
| `--noise-reduction` | `--clean` (hidden alias may exist for one release, then removed) |
| `--show-encoding` / `--show-offset` / `--show-tags` | `--columns` |
| `--no-layout` | *(gone; layout is automatic)* |
| `-l`, `--load` | *(deprecated; do not implement)* |


## 4. Semantics and edge cases

### 4.1 Multi-value matching
Multiple values to any single flag use **OR**. Across different flags, semantics compose:
- `--string-type static --string-type stack` → static OR stack.
- `--string-type static --no-string-type stack` → static AND NOT stack (restriction applied, then exclusion).
- If a value appears in both allow and deny lists for the same dimension, **deny wins**.
- `all` expands to the full set of types. `--string-type all --no-string-type stack` → all except stack.
- `--query` combined with `--tag` or `--interesting` → AND across flags (string must satisfy all active filters).
- `--section`/`--structure` filters only affect static strings (recovered strings have no layout). They do not error when recovered types are selected; they simply do not apply to them.

### 4.2 Tag evaluation
- `--no-tag` is the only way to disable tag evaluation.
- Hiding the tag column is done by omitting `tags` from `--columns`: this does **not** disable evaluation, only display.
- `--interesting` excludes the noisy tag set: `#common, #duplicate, #code, #reloc, #code-junk`.
- No `--tag none` sentinel exists (source review comment). Use `--no-tag <tags>`.

### 4.3 Validation and errors
- Unknown values for `--string-type`, `--section`, `--structure`, `--tag`, `--view`, `--columns`, `--color`, `--format`, `--language` → fatal usage error listing valid values.
- Invalid regex for `--query` → fatal error.
- `--max-strings < 1` → fatal error.
- Conflicting output modes (`--json` + `--toon`, `--view tree` + `--toon`) → error, or silently let TOON/JSON take precedence with a `--verbose` note. **Decision:** JSON/TOON take precedence over terminal view/columns; emit a warning on STDERR in verbose mode. This keeps each flag independent (goal 3.1.3).
- Sample > 100 MB without `-L` → error unless `--no-cache`+`--disable-progress` in a configurable strictness mode; default behavior: refuse with a clear message and point to `-L`. (Align with existing capa behavior; confirm against repo.)

### 4.4 Exit codes
| Code | Meaning |
|---|---|
| `0` | Success. |
| `1` | Simple help (no args) and fatal runtime/analysis errors. |
| `2` | Usage errors from argparse defaults (confirm existing behavior; keep consistent). |


## 5. String display order and headings (source §1.2)

Order of output sections:
1. Static strings (layout-aware tree/grouped/flat rendering).
2. Language strings.
3. Stack strings.
4. Tight strings.
5. Decoded strings.

Rules:
- Static strings get rich, layout-aware rendering.
- Recovered kinds (2–5) render separately, **after** static strings, still in the rich tabular format, each under a heading naming the kind, matching the layout-section style. Example:

```
----- decoded strings -----
MR: see what's there currently and reuse or improve that
```

- This order is the current behavior in `floss/render/default.py`: preserve and extend it, do not invent a new one.


## 6. Output formats

### 6.1 Full JSON (`-j`, `--json`)
Emit the complete structured ResultDocument. This is the canonical schema the cache stores (§8) and the base for all other renderers.

### 6.2 Structured JSON summary (`--json-summary`)
- Independent of `--json` (source open question 3, Decision).
- Token-efficient JSON designed for LLM context windows.
- Contents: sample metadata, per-section string counts, tag histograms, and strings matching high-value tags.
- Derived from the same ResultDocument as `--json`.

### 6.3 Summary (human/agent, `--summary`: see §7.4)
The non-JSON summary option requested in source §2.4: include file path, language, runtime, and version metadata that is currently hidden when layout succeeds. Rendered from the ResultDocument.

### 6.4 TOON rendering (`--toon`)
Token-optimized output with the least possible characters: no borders/box-drawing, minimal whitespace, only relevant columns, compact table format. Includes interactive affordance hints such as:

```
showing x/y strings by default, use "…options…" to see more
```

- TOON implies flat layout; tree headers are omitted.
- Line-delimited rows suitable for streaming ingestion pipelines.

### 6.5 Structured JSON errors
When any JSON output mode is active (`--json`, `--json-summary`, and any future machine mode):
- Fatal CLI or parsing errors emit a structured JSON object on **STDERR**, not unstructured text, e.g.:
  `{"error": "...", "code": <int>}`
- `code` uses the values from §4.4 where applicable; analysis failures get a distinct code (e.g. `3`).
- When no JSON mode is active, existing plain-text errors are unchanged.


## 7. Phase-by-phase implementation plan

### Phase 1: CLI flag surface

**Task 1.1: Remove deprecated flags**
- Remove `--exclude-aliases`, `--no-layout`, `--show-encoding`, `--show-offset`, `--show-tags`, and `-l`/`--load`.
- Unknown flags must produce the standard usage error.
- **Files:** CLI argument definition module (e.g. `floss/arguments.py` or the argparse block in `floss/main.py`; confirm in repo).
- **Accept:** `floss --exclude-aliases sample.exe` errors; `--load` is absent from `--help`.

**Task 1.2: Rename `--noise-reduction` → `--clean`**
- Rename the flag; keep a hidden alias `--noise-reduction` for one release if the repo convention allows, otherwise remove entirely. **Decision:** remove entirely; `--clean` is the only name.
- **Accept:** `--clean` is accepted; `--noise-reduction` errors.

**Task 1.3: Add new flags**
Add the full flag surface from §3.3–3.8: `--no-string-type`, `--section`, `--no-section`, `--structure`, `--no-structure`, `-Q`/`--query`, `--tag`, `--no-tag`, `--interesting`, `--max-strings`, `--columns`, `--view`, `--color`, `--json-summary`, `--toon`, `--cache-dir`, `--no-cache`, `--prune-cache`.
- **Accept:** `floss --help` lists every flag in §3 with documented defaults; `--version` and no-arg help exit codes per §3.2.

### Phase 2: Filter and query engine

**Task 2.1: Filter application order**
Implement a single pipeline that, per string, applies in order: string-type allow/deny → section allow/deny → structure allow/deny → tag allow/deny → `--interesting` → `--query` → `--max-strings` cap.
- **Accept:** unit tests cover every flag in isolation and in combination, per §4.1.

**Task 2.2: In-tree query**
`--query` filters strings while preserving parent section and structure headings in tree and grouped modes.
- **Accept:** a tree view filtered by regex still shows section/structure headers for sections that have matching strings.

**Task 2.3: Relevance ordering for `--max-strings`**
Define deterministic relevance: strings tagged with non-noisy tags first, then by ascending offset (stable tie-break by string value). Document the ordering in the code.
- **Accept:** `--max-strings 10` returns the same 10 strings for the same input across runs.

### Phase 3: Rendering

**Task 3.1: Section ordering and headings (§5)**
**Task 3.2: `--view` modes**
tree (default when layout present), grouped, flat (default when no layout).
**Task 3.3: `--columns` in flat and tree views**
Per §3.6 and the §8 decision, apply the column list in flat view; in tree view show the selected columns under headings where applicable. Omitting `tags` hides the tag column.
**Task 3.4: `--color`**
Auto-detect TTY; `always`/`never` force. Default `auto`.
**Task 3.5: TOON rendering (§6.4)**
- **Accept:** each mode renders deterministically; snapshot tests exist.

### Phase 4: Machine-readable output

**Task 4.1: Full JSON (§6.1)**
- **Accept:** schema matches the cached ResultDocument exactly.

**Task 4.2: Structured JSON errors (§6.5)**
- **Accept:** with `--json` active, an invalid regex or missing file yields `{"error": ..., "code": ...}` on STDERR only; STDOUT carries no error text.

**Task 4.3: JSON summary (§6.2) and human summary (§6.3)**
Include the metadata currently hidden when layout succeeds (path, language, runtime, version).
- **Accept:** `--summary` shows file metadata that the layout view omits.

### Phase 5: Result caching (source §3.8)

**Task 5.1: Cache module** (`floss/cache.py`, new)
- **Cache directory:** platform cache dirs (`$XDG_CACHE_HOME/floss` or `~/.cache/floss` on Linux; `~/Library/Caches/floss` on macOS; `%LOCALAPPDATA%\floss\Cache` on Windows), overridable by `FLOSS_CACHE_DIR` or `--cache-dir`.
- **Cache key:** SHA-256 of sample bytes **mixed with** the FLOSS version and the layout/tag engine version (any code change invalidates stale entries). Use a stable string concatenation (e.g. `f"{version}|{layout_version}|{tags_version}|{sha256}"`).
- **Layout:** fan-out `{cache-dir}/{key[0:2]}/{key[2:]}.json`.
- **Stored payload:** the full ResultDocument JSON (same schema as `--json`).
- **Write:** atomic write (temp file + rename) guarded by a lock file to prevent concurrent-write races.
- **Hit path:** compute key on startup; on a valid hit, load cached ResultDocument, **skip** extraction/layout/tagging, render from it. Surface hits via `--verbose`.
- **Validation:** drop and re-analyze on parse failure or checksum mismatch. Cache is a speedup, never a source of correctness.
- **Lifecycle:** LRU/TTL eviction, `--prune-cache` command, warn when total cache size exceeds a configured cap.
- **Opt-out:** `--no-cache` disables reads and writes.
- **Accept:** run the same sample twice; second run is served from cache (visible with `-v`) and skips analysis; corrupt the cache file → full re-analysis; cache honored the memory regression note in §2.5.

### Phase 6: Deferred items and cleanups (source §2)

**Task 6.1: Fix stale data paths (source §2.6)**
- Fix old `floss/qs/db/data` paths in:
  - `scripts/tags/query_string.py:27`
  - `scripts/tags/build_oss_db.py:1102`
- **Accept:** both files reference the moved tags database location; their tests pass.

**Task 6.2: Large-sample size warning (source §2.5)**
- Keep the full-buffer `sample.read_bytes()` read for now (documented regression).
- Add a size warning when the sample is large enough to matter (align threshold with `-L`, 100 MB).
- **Accept:** a 150 MB sample warns on STDERR unless `-L`.

**Task 6.3: Meta visibility with layout (source §2.4)**
- Keep current behavior (meta table hidden when layout renders).
- File an issue referencing this decision; metadata is surfaced via `--summary` (§Task 4.3).
- **Accept:** issue exists; `--summary` output includes path/language/runtime/version.

**Task 6.4: Renderer consolidation note (source §2.3)**
- Keep `render_strings` in `render/layout_text.py`; default renderer stays the coordinator. No code change; add a code comment referencing the future consolidation PR.

**Task 6.5: `Runtime.layout` / `Runtime.tags` fields (source §2.1)**
- Already removed; do not re-add. Add a code comment/TODO noting the decision window (after QS merge, before auto DB build).


## 8. Decisions on open questions (from source)

| # | Question | Decision |
|---|---|---|
| 1 | Does `--query` accept multiple expressions? | Yes: repeat the flag. Multiple expressions are **ANDed**. |
| 2 | Does `--columns` apply to the tree view? | Yes: show selected columns under headings where applicable. |
| 3 | When does `--json-summary` ship relative to `--json`? | Independent. Summary is a human/agent-focused token-efficient rendering derived from the same ResultDocument, **not** a JSON-specific structure. Other consumers build their own structured summaries. |

Additional binding decisions made in this spec:
- `-t` short flag retained alongside `--string-type` (§3.3).
- No `--no` generic flag; `--no-*` family only (§3.5 decision).
- TOON/JSON take precedence over terminal view/columns (§4.3).
- `--noise-reduction` removed, not aliased (§Task 1.2).
- `--max-strings` relevance = non-noisy tags first, then ascending offset (§Task 2.3).


## 9. Testing strategy

- **Framework:** follow the repo's existing test conventions (`pytest` + fixtures in `tests/`). Do not introduce new test tooling.
- **Golden/snapshot tests:** every render mode (tree, grouped, flat, TOON, JSON, JSON summary) on a fixed sample.
- **CLI tests:** subprocess tests for exit codes, help text, and structured JSON errors.
- **Filter matrix:** combinatorial test of the §4.1 semantics (allow/deny pairs, `all` expansion, deny-wins, cross-flag AND).
- **Cache tests:** hit/miss, corruption → re-analysis, version-bump → invalidation, `--no-cache`, concurrent-write race (two processes).
- **Regression:** preserve existing FLOSS analyses on `master` samples that do not use new flags (no behavioral drift for `-n 4 sample` with default view).

## 10. Definition of done

- All Phase 1–6 tasks implemented and their acceptance criteria passing.
- `pytest` green; lint/typecheck as configured in the repo run clean.
- No uncommitted secrets; changes are on the `quantumstrand` branch unless a fix is master-specific.
- Open questions are resolved per §8 and documented in the code where relevant.
- No implementation of non-goals in §2.2 (no tags/layout for recovered strings, no shell completions, no render consolidation).

## 11. Out of scope (future work, do not implement)

- Tags/layout for stack, tight, decoded, go, rust strings (source §3.6.2): VA→LayoutMap mapping in `floss/enrich.py`, tag evaluation via `floss.tags.engine.evaluate_tags()`, unified tree integration.
- Shell completion scripts (issue #1350).
- Render module consolidation and classic-view removal.
- `Runtime.layout`/`Runtime.tags` metrics.
- Post-GSoC scripting/instrumentation for metrics.