FLOSS Migration: Implementation Spec for an Autonomous Agent

Source: FLOSS Migration Document 8_10_26.md (the single reference) Status: Ready to implement Target repo: mandiant/flare-floss, branches quantumstrand (primary) and master

All file paths below are relative to the flare-floss repo root. The target repo is not present in this working directory; the implementing agent must run inside a checkout of the repo.

0. How to use this document

This spec is written for an agent that will implement the FLOSS migration. Each task is numbered, self-contained, and has:

• Goal: what to change.
• Files: the files most likely affected (agent must confirm by reading the repo).
• Acceptance criteria: how to verify the change is done.

Work through the phases in order. Phases have hard dependencies (Phase 1 must precede Phase 2–5). Phase 6 items are independent and may be done last. Commit after each phase with a clear message.

Golden rule: the behavior and decisions in the source document are normative. Where this spec makes an explicit "Decision:", that decision resolves an open question in the source document and is binding for this implementation.

1. Goals (from source §3.1)

1. Improve usability for human analysts and AI agents.
2. Keep the flag count small.
3. Keep each flag independent from the others.

Personas:

• Human analyst: interactive-style tree navigation, hierarchical section/structure grouping, in-tree regex search that preserves headings, automatic noise reduction, single option to toggle metadata columns.
• AI agent: token-efficient summary schemas, deterministic string-count limits, line-delimited streaming output for ingestion pipelines, structured error objects on STDERR when machine output is active.

2. Scope and non-goals

2.1 In scope

• Static-string layout reconstruction and tag-aware output (tree/grouped/flat views).
• CLI redesign: removals, renames, new filter flags, output flags.
• Machine-readable output: --json, --json-summary, TOON rendering, structured JSON errors.
• Result caching layer.
• Deferred cleanups that are safe to do now (source §2.6).

2.2 Non-goals (do NOT do now)

• Layout context or tags for recovered (non-static) strings. Stack, tight, decoded, and language strings keep emulation/parser extraction (source §1.1, §3.6). Tagging/layout for them is a TODO roadmap only (§3.6.2).
• Shell completion scripts (zsh, bash, fish): post-GSoC, tracked in issue #1350.
• Reintroducing Runtime.layout/Runtime.tags timing fields: decision deferred until after QS merge and before auto DB build (source §2.1).
• Consolidating render modules / removing classic view: happens in the PR that removes the classic renderer (source §2.3).

3. Target flag reference (authoritative)

3.1 Positional arguments

3.2 Core options

3.3 Analysis and extraction options

Decision (source §3.7.3 review): keep both the -t short flag and the --string-type long flag. The short flag is a convenience; the long form is primary in docs/help.

3.4 Layout and structure options

3.5 Tagging, searching, noise filtering

Decision (source §3.3.2 note): the phrase "Exclusions strictly use --no <type>" means the --no-* flag family (--no-string-type, --no-section, --no-structure, --no-tag). There is no generic --no flag.

3.6 Output and rendering options

3.7 Advanced / decoder control

3.8 Logging and cache options

3.9 Removed flags (must error as unknown)

4. Semantics and edge cases

4.1 Multi-value matching

Multiple values to any single flag use OR. Across different flags, semantics compose:

• --string-type static --string-type stack → static OR stack.
• --string-type static --no-string-type stack → static AND NOT stack (restriction applied, then exclusion).
• If a value appears in both allow and deny lists for the same dimension, deny wins.
• all expands to the full set of types. --string-type all --no-string-type stack → all except stack.
• --query combined with --tag or --interesting → AND across flags (string must satisfy all active filters).
• --section/--structure filters only affect static strings (recovered strings have no layout). They do not error when recovered types are selected; they simply do not apply to them.

4.2 Tag evaluation

• --no-tag is the only way to disable tag evaluation.
• Hiding the tag column is done by omitting tags from --columns: this does not disable evaluation, only display.
• --interesting excludes the noisy tag set: #common, #duplicate, #code, #reloc, #code-junk.
• No --tag none sentinel exists (source review comment). Use --no-tag <tags>.

4.3 Validation and errors

• Unknown values for --string-type, --section, --structure, --tag, --view, --columns, --color, --format, --language → fatal usage error listing valid values.
• Invalid regex for --query → fatal error.
• --max-strings < 1 → fatal error.
• Conflicting output modes (--json + --toon, --view tree + --toon) → error, or silently let TOON/JSON take precedence with a --verbose note. Decision: JSON/TOON take precedence over terminal view/columns; emit a warning on STDERR in verbose mode. This keeps each flag independent (goal 3.1.3).
• Sample > 100 MB without -L → error unless --no-cache+--disable-progress in a configurable strictness mode; default behavior: refuse with a clear message and point to -L. (Align with existing capa behavior; confirm against repo.)

4.4 Exit codes

5. String display order and headings (source §1.2)

Order of output sections:

1. Static strings (layout-aware tree/grouped/flat rendering).
2. Language strings.
3. Stack strings.
4. Tight strings.
5. Decoded strings.

Rules:

• Static strings get rich, layout-aware rendering.
• Recovered kinds (2–5) render separately, after static strings, still in the rich tabular format, each under a heading naming the kind, matching the layout-section style. Example:

----- decoded strings -----
MR: see what's there currently and reuse or improve that

• This order is the current behavior in floss/render/default.py: preserve and extend it, do not invent a new one.

6. Output formats

6.1 Full JSON (-j, --json)

Emit the complete structured ResultDocument. This is the canonical schema the cache stores (§8) and the base for all other renderers.

6.2 Structured JSON summary (--json-summary)

• Independent of --json (source open question 3, Decision).
• Token-efficient JSON designed for LLM context windows.
• Contents: sample metadata, per-section string counts, tag histograms, and strings matching high-value tags.
• Derived from the same ResultDocument as --json.

6.3 Summary (human/agent, --summary: see §7.4)

The non-JSON summary option requested in source §2.4: include file path, language, runtime, and version metadata that is currently hidden when layout succeeds. Rendered from the ResultDocument.

6.4 TOON rendering (--toon)

Token-optimized output with the least possible characters: no borders/box-drawing, minimal whitespace, only relevant columns, compact table format. Includes interactive affordance hints such as:

showing x/y strings by default, use "…options…" to see more

• TOON implies flat layout; tree headers are omitted.
• Line-delimited rows suitable for streaming ingestion pipelines.

6.5 Structured JSON errors

When any JSON output mode is active (--json, --json-summary, and any future machine mode):

• Fatal CLI or parsing errors emit a structured JSON object on STDERR, not unstructured text, e.g.: {"error": "...", "code": <int>}
• code uses the values from §4.4 where applicable; analysis failures get a distinct code (e.g. 3).
• When no JSON mode is active, existing plain-text errors are unchanged.

7. Phase-by-phase implementation plan

Phase 1: CLI flag surface

Task 1.1: Remove deprecated flags

• Remove --exclude-aliases, --no-layout, --show-encoding, --show-offset, --show-tags, and -l/--load.
• Unknown flags must produce the standard usage error.
• Files: CLI argument definition module (e.g. floss/arguments.py or the argparse block in floss/main.py; confirm in repo).
• Accept: floss --exclude-aliases sample.exe errors; --load is absent from --help.

Task 1.2: Rename --noise-reduction → --clean

• Rename the flag; keep a hidden alias --noise-reduction for one release if the repo convention allows, otherwise remove entirely. Decision: remove entirely; --clean is the only name.
• Accept: --clean is accepted; --noise-reduction errors.

Task 1.3: Add new flags Add the full flag surface from §3.3–3.8: --no-string-type, --section, --no-section, --structure, --no-structure, -Q/--query, --tag, --no-tag, --interesting, --max-strings, --columns, --view, --color, --json-summary, --toon, --cache-dir, --no-cache, --prune-cache.

• Accept: floss --help lists every flag in §3 with documented defaults; --version and no-arg help exit codes per §3.2.

Phase 2: Filter and query engine

Task 2.1: Filter application order Implement a single pipeline that, per string, applies in order: string-type allow/deny → section allow/deny → structure allow/deny → tag allow/deny → --interesting → --query → --max-strings cap.

• Accept: unit tests cover every flag in isolation and in combination, per §4.1.

Task 2.2: In-tree query --query filters strings while preserving parent section and structure headings in tree and grouped modes.

• Accept: a tree view filtered by regex still shows section/structure headers for sections that have matching strings.

Task 2.3: Relevance ordering for --max-strings Define deterministic relevance: strings tagged with non-noisy tags first, then by ascending offset (stable tie-break by string value). Document the ordering in the code.

• Accept: --max-strings 10 returns the same 10 strings for the same input across runs.

Phase 3: Rendering

Task 3.1: Section ordering and headings (§5) Task 3.2: --view modes tree (default when layout present), grouped, flat (default when no layout). Task 3.3: --columns in flat and tree views Per §3.6 and the §8 decision, apply the column list in flat view; in tree view show the selected columns under headings where applicable. Omitting tags hides the tag column. Task 3.4: --color Auto-detect TTY; always/never force. Default auto. Task 3.5: TOON rendering (§6.4)

• Accept: each mode renders deterministically; snapshot tests exist.

Phase 4: Machine-readable output

Task 4.1: Full JSON (§6.1)

• Accept: schema matches the cached ResultDocument exactly.

Task 4.2: Structured JSON errors (§6.5)

• Accept: with --json active, an invalid regex or missing file yields {"error": ..., "code": ...} on STDERR only; STDOUT carries no error text.

Task 4.3: JSON summary (§6.2) and human summary (§6.3) Include the metadata currently hidden when layout succeeds (path, language, runtime, version).

• Accept: --summary shows file metadata that the layout view omits.

Phase 5: Result caching (source §3.8)

Task 5.1: Cache module (floss/cache.py, new)

• Cache directory: platform cache dirs ($XDG_CACHE_HOME/floss or ~/.cache/floss on Linux; ~/Library/Caches/floss on macOS; %LOCALAPPDATA%\floss\Cache on Windows), overridable by FLOSS_CACHE_DIR or --cache-dir.
• Cache key: SHA-256 of sample bytes mixed with the FLOSS version and the layout/tag engine version (any code change invalidates stale entries). Use a stable string concatenation (e.g. f"{version}|{layout_version}|{tags_version}|{sha256}").
• Layout: fan-out {cache-dir}/{key[0:2]}/{key[2:]}.json.
• Stored payload: the full ResultDocument JSON (same schema as --json).
• Write: atomic write (temp file + rename) guarded by a lock file to prevent concurrent-write races.
• Hit path: compute key on startup; on a valid hit, load cached ResultDocument, skip extraction/layout/tagging, render from it. Surface hits via --verbose.
• Validation: drop and re-analyze on parse failure or checksum mismatch. Cache is a speedup, never a source of correctness.
• Lifecycle: LRU/TTL eviction, --prune-cache command, warn when total cache size exceeds a configured cap.
• Opt-out: --no-cache disables reads and writes.
• Accept: run the same sample twice; second run is served from cache (visible with -v) and skips analysis; corrupt the cache file → full re-analysis; cache honored the memory regression note in §2.5.

Phase 6: Deferred items and cleanups (source §2)

Task 6.1: Fix stale data paths (source §2.6)

• Fix old floss/qs/db/data paths in:
  • scripts/tags/query_string.py:27
  • scripts/tags/build_oss_db.py:1102
• Accept: both files reference the moved tags database location; their tests pass.

Task 6.2: Large-sample size warning (source §2.5)

• Keep the full-buffer sample.read_bytes() read for now (documented regression).
• Add a size warning when the sample is large enough to matter (align threshold with -L, 100 MB).
• Accept: a 150 MB sample warns on STDERR unless -L.

Task 6.3: Meta visibility with layout (source §2.4)

• Keep current behavior (meta table hidden when layout renders).
• File an issue referencing this decision; metadata is surfaced via --summary (§Task 4.3).
• Accept: issue exists; --summary output includes path/language/runtime/version.

Task 6.4: Renderer consolidation note (source §2.3)

• Keep render_strings in render/layout_text.py; default renderer stays the coordinator. No code change; add a code comment referencing the future consolidation PR.

Task 6.5: Runtime.layout / Runtime.tags fields (source §2.1)

• Already removed; do not re-add. Add a code comment/TODO noting the decision window (after QS merge, before auto DB build).

8. Decisions on open questions (from source)

Additional binding decisions made in this spec:

• -t short flag retained alongside --string-type (§3.3).
• No --no generic flag; --no-* family only (§3.5 decision).
• TOON/JSON take precedence over terminal view/columns (§4.3).
• --noise-reduction removed, not aliased (§Task 1.2).
• --max-strings relevance = non-noisy tags first, then ascending offset (§Task 2.3).

9. Testing strategy

• Framework: follow the repo's existing test conventions (pytest + fixtures in tests/). Do not introduce new test tooling.
• Golden/snapshot tests: every render mode (tree, grouped, flat, TOON, JSON, JSON summary) on a fixed sample.
• CLI tests: subprocess tests for exit codes, help text, and structured JSON errors.
• Filter matrix: combinatorial test of the §4.1 semantics (allow/deny pairs, all expansion, deny-wins, cross-flag AND).
• Cache tests: hit/miss, corruption → re-analysis, version-bump → invalidation, --no-cache, concurrent-write race (two processes).
• Regression: preserve existing FLOSS analyses on master samples that do not use new flags (no behavioral drift for -n 4 sample with default view).

10. Definition of done

• All Phase 1–6 tasks implemented and their acceptance criteria passing.
• pytest green; lint/typecheck as configured in the repo run clean.
• No uncommitted secrets; changes are on the quantumstrand branch unless a fix is master-specific.
• Open questions are resolved per §8 and documented in the code where relevant.
• No implementation of non-goals in §2.2 (no tags/layout for recovered strings, no shell completions, no render consolidation).

11. Out of scope (future work, do not implement)

• Tags/layout for stack, tight, decoded, go, rust strings (source §3.6.2): VA→LayoutMap mapping in floss/enrich.py, tag evaluation via floss.tags.engine.evaluate_tags(), unified tree integration.
• Shell completion scripts (issue #1350).
• Render module consolidation and classic-view removal.
• Runtime.layout/Runtime.tags metrics.
• Post-GSoC scripting/instrumentation for metrics.