Login Register
vee1e

vee1e / f5d28b8b9f2e414907191dcd0eb39508

Last active 4 days ago

Like 0
migration-doc.md Raw

FLOSS Migration Document

Date: 2026-08-06

Branches: quantumstrand, master

Related PR: https://github.com/mandiant/flare-floss/pull/1347

Related issues:

Purpose

This document is the single reference for the FLOSS migration.

It contains three parts:

  1. The follow-up decisions for the string kinds.
  2. The deferred items from PR #1347.
  3. The CLI specification and design for the major release.

The review comments appear inline in the CLI specification.

Part 1: String display decisions

1.1 Scope for now

Recommendation: focus on static strings only for this phase.

The layout and tag features work on static strings.

Do not add layout context to recovered strings yet.

1.2 Position of recovered strings

Show stack, tight, and decoded strings at the end of the output.

This is the current behavior in floss/render/default.py.

Keep this order:

  1. Static strings (layout-aware when available).
  2. Language strings.
  3. Stack strings.
  4. Tight strings.
  5. Decoded strings.

Do not merge recovered strings into the layout tree.

1.3 Separate rendering

Use separate rendering and tables for each string kind.

Static strings use the layout-aware section tree.

Recovered strings use classic tables.

Do not mix the two styles.

1.4 Plain output

Open question: keep a --plain output mode?

Current behavior: default text output uses Rich tables.

The web view is out of scope.

Decide whether a plain, table-free mode is needed for scripts.

1.5 Decoded string placement

Where should decoded strings appear?

Possible feature: use offsets to place them.

This is not 100% guaranteed. It may change or not ship.

Example structure:

Function 0x401000
    Short        <offset>
    mal.com      <decoded string>

Group decoded strings by their decoding function.

Show the offset next to each decoded string.

1.6 Filter option

Support a filter by tag.

Example: --filter mal shows only strings with the tag mal.

Base other options on what a human does in the interactive web view.

Part 2: Deferred items from PR #1347

2.1 Runtime timing fields (layout, tags)

Discussion: Runtime had the fields layout and tags. A reviewer asked if these fields are worth tracking.

Decision: Removed both in commit 75c973a. Metrics and performance tracking is an open question for a future iteration.

Status: Complete.

Action: Address this later, after the FLOSS QS merge is complete but before the auto DB build starts.

2.2 Load flag ignores layout flags

Discussion: floss --load file.json --no-layout still renders the layout view. The load function does not clear results.layout or apply enable_layout.

Decision: Noted as a TODO and issue #1348. We are removing --no-layout entirely. Whether to remove it completely is a matter of debate.

Status: Open. TODOs at floss/results.py:350 and floss/render/default.py:361.

Action: Make load() honor enable_layout and enable_tags. Clear results.layout when layout is disabled.

2.3 Bulk analysis script only does static

Discussion: scripts/analysis/bulk_analyze.py excludes stack, tight, and decoded strings. The script is inherited from QuantumStrand, which was static-only.

Decision: Mark as a TODO for when recovered strings are first-class in layout output. Add a doc note.

Status: Open. TODO at scripts/analysis/bulk_analyze.py:24.

Action: Run full deobfuscation when recovered strings are first-class in layout output. Update the script doc.

2.4 Render function location

Discussion: The render_strings function lives in floss/render/layout_text.py. A reviewer asked if it should move to default.py.

Decision: Keep it in layout_text.py for now. It owns layout rendering. The default renderer stays the coordinator. Consolidate when Rich rendering covers every string kind.

Status: Complete.

Action: Consolidate render modules in the PR that removes the classic view. Remove obsolete classic helpers then.

2.5 Render meta not shown with layout

Discussion: When layout succeeds, the classic meta table is never shown. This includes file path, language, runtime, and version. A reviewer asked to confirm this is intended.

Decision: This matches the PR intent. Keep as-is.

Status: Complete.

Action: Mandatory for this PR: show meta above the layout tree. Or show meta under --verbose, without --no-layout.

2.6 Full-buffer read for static extraction

Discussion: The pipeline reads the whole file into memory with sample.read_bytes(). Classic extraction previously used mmap. This is a memory regression for very large files.

Decision: Fixed in commit 6890d76, then reverted in bedeab4. The layout path needs the full buffer. Added a TODO.

Status: Open. TODO at floss/pipeline.py:278.

Action: Add an mmap-only classic path if layout stops being default. Consider a size warning before full materialization.

2.7 Remaining migration items

Two files still point to the old floss/qs/db/data path:

  • scripts/tags/query_string.py:27
  • scripts/tags/build_oss_db.py:1102

Fix these paths.

2.8 Issues to address

Part 3: CLI specification and design (layout and tag aware)

3.1 Overview

This specification defines the redesigned command-line interface for FLOSS.

It uses binary layout reconstruction and string classification tags.

The goals are:

  • Improve usability for human analysts and AI agents.
  • Keep the flag count small.
  • Keep each flag independent from the others.

3.2 Goals and user personas

3.2.1 Human analyst persona

  • Interactive-style terminal navigation inspired by qs-viewer.
  • Hierarchical section and structure grouping.
  • In-tree regex searching without losing section and structure headers.
  • Automatic noise reduction for compiler boilerplate and common runtime libraries.
  • Single option to toggle string metadata columns.

3.2.2 AI agent persona

  • Token-efficient summary schemas for prompt context windows.
  • Deterministic string count limits to prevent context window overflow.
  • Line-delimited streaming JSON for automated ingestion pipelines.
  • Structured JSON error objects on STDERR when JSON output is active.

3.3 Removals, simplifications, and anti-bloat rules

3.3.1 Removed redundant aliases

  • Removed --exclude-aliases. Exclusions strictly use --no <type> for less bloat.
  • Removed --noise-reduction alias. Noise suppression strictly uses --clean.
  • Removed boolean column toggles (--show-encoding, --show-offset, --show-tags). A single --columns option replaces them.

3.3.2 Symmetrical filtering flags

  • --string-type <type...> and --no-string-type <type...>: Select or exclude string types (static, stack, tight, decoded, language, all).
  • --section <name...> and --no-section <name...>: Filter or exclude by binary section name (.rdata, .text).
  • --structure <name...> and --no-structure <name...>: Filter or exclude by binary structure (import_table, pe_header).
  • --tag <tag...> and --no-tag <tag...>: Filter or exclude by semantic classification tag (winapi, crypto, crt, oss).

3.3.3 Matching logic and shortcuts

  • Multi-value matching: Passing multiple values to a flag uses OR logic. Strings matching any of the specified values are included or excluded.
  • --interesting: Automatically excludes noisy tags (#common, #duplicate, #code, #reloc, #code-junk).
  • --no-tag <tag...>: Suppress strings matching specified tags. This is the only way to disable tag evaluation.

Final decision on --no-tag-engine:

Rejected. The review comments show the trade-off:

  • (Willi Ballenthin) Is this necessary?
  • (Moritz Raabe) Maybe? We could do it with --tag none or similar.
  • (Lakshit Verma) Yes. --tag none would avoid bloating the argument list and is easy to explain.

Final decision: keep --no tag only.

3.4 Output rendering and terminal views

3.4.1 Terminal display modes (--view)

  • tree: Hierarchical layout tree (Section -> Structure -> Strings) matching qs-viewer navigation. Collapses noisy CRT and OSS strings into branch count summaries. Default when binary layout analysis succeeds.
  • grouped: Strings grouped by section or function header.
  • flat: Classical line-by-line listing.

Review comment on --view:

  • (Moritz Raabe) TBD.

Note

The default view mode is not final. Confirm before the major release.

3.4.2 In-tree search and filtering

  • --query, -Q: Filter strings by regular expression while preserving parent tree and structure headings in tree and grouped modes.
  • --max-strings: Limit emitted strings per section to the top-N highest-relevance strings. Prioritize tagged API and crypto strings over untagged entries.

3.4.3 Column display option

In flat and grouped views, a comma-separated list controls column visibility:

  • --columns <col...>: Available columns are tags, offset, structure, encoding. Default: tags, offset.

3.5 Output formats

  • -j, --json: Full structured ResultDocument.

Review comments on -j, --json:

  • (Willi Ballenthin) Consider the TOON format for token efficiency: https://github.com/toon-format/toon. But maybe translating to a different format is the responsibility of the caller.
  • (Moritz Raabe) Good reference. We are going for something like this with the text output, so consider leveraging it directly.
  • --summary: Summary containing sample metadata, section counts, tag histograms, and strings matching high-value tags (#capa, winapi, crypto). Ideally base everything on JSON output for programmatic access, then render from there for humans and agents.

Note

TBD: should --summary also work with -j?

  • Structured JSON errors: When any JSON output mode is active, fatal CLI or parsing errors emit a structured JSON object to STDERR (for example {"error": "...", "code": ...}) rather than unstructured text.

Review comments on JSON output:

  • (Willi Ballenthin) Do agents really do a better job with JSON? Or is this more to support programmatic use of the output?
  • (Moritz Raabe) TBD, but yes. Refining this here.

Note

The JSON output direction is still under review.

3.6 Non-static string processing (TODO roadmap)

3.6.1 Current scope

  • Layout reconstruction and string tagging apply to static strings.
  • Stack, tight, decoded, and language strings continue to be extracted via emulation and language-specific parsers.

3.6.2 Future extensions (TODO)

  • TODO (Non-static section mapping): Map emulated function Virtual Address (VA) to the binary LayoutMap in floss/enrich.py so decoded and stack strings inherit their parent section (.text) and structure.
  • TODO (Non-static string tagging): Run extracted string buffers from stack, tight, decoded, go, and rust through floss.tags.engine.evaluate_tags().
  • TODO (Unified tree integration): Attach non-static strings under their corresponding parent section in the hierarchical layout tree in ResultDocument.

3.7 Complete command-line interface reference

3.7.1 Positional arguments

  • sample: Path to the target binary sample to analyze.

3.7.2 Core options

  • -n, --minimum-length: Minimum string length threshold (default: 4).
  • (no arguments): simple help.
  • --help: Display advanced options and exit.
  • --version: Display program version and exit.

Note

Add zsh, bash, and fish completion scripts for the FLOSS arguments. Tracked in https://github.com/mandiant/flare-floss/issues/1350. A tab press then shows completions without running --help.

3.7.3 Analysis and extraction options

  • -t <type...>, --string-type <type...>: Select string types to extract (static, stack, tight, decoded, language, all; default: all).

Review comments on -t, --string-type:

  • (Willi Ballenthin) These short options are hard to remember. Unless a user types them often (like with ls), the longer flag is better.
  • (Moritz Raabe) Good point.
  • --no-string-type <type...>: Exclude specific string types.

3.7.4 Layout and structure options

  • --section <name...>: Restrict static strings to specified binary sections.
  • --no-section <name...>: Exclude strings in specified binary sections.
  • --structure <name...>: Restrict static strings to specified binary structures.
  • --no-structure <name...>: Exclude strings in specified structures.
  • --no-layout: Disable binary layout reconstruction engine. Being removed entirely; whether to remove it completely is a matter of debate.

Review comment on --no-layout:

  • (Moritz Raabe) Do this via --view instead.

Note

The final removal of --no-layout is not confirmed.

3.7.5 Tagging, searching, and noise filtering options

  • --query: Search and filter strings by regex while preserving hierarchy.
  • --tag <tag...>: Emit only strings matching specified tags.
  • --no-tag <tag...>: Suppress strings matching specified tags.
  • --interesting: Shortcut to emit only strings with non-noisy semantic tags.
  • --max-strings: Cap emitted strings per section to top-N highest-relevance items.

3.7.6 Output and rendering options

  • --view: Terminal display format (tree, grouped, flat; default: tree when layout is present, otherwise flat).
  • --columns <col...>: Comma-separated list of columns to display in flat view (tags, offset, structure, encoding; default: tags, offset).
  • -j, --json: Emit full structured JSON document.
  • --json-summary: Emit token-efficient JSON summary document for LLM prompts.
  • -v, --verbose: Enable verbose output.

Review comments on -v, --verbose:

  • (Willi Ballenthin) What is this?
  • (Willi Ballenthin) The format is already specified with --view, and --debug handles logging, right?
  • (Moritz Raabe) Artifact of current FLOSS with the -v option (like in capa) showing offsets. Let us simplify this.
  • --color: Enable ANSI color formatting (auto, always, never; default: auto).

3.7.7 Advanced and decoder control options

  • -f, --format: Target sample file format (auto, pe, sc32, sc64; default: auto).
  • --language: Language-specific string extractor control (auto, go, rust, none; default: auto).
  • -l, --load: Load analysis from an existing FLOSS results JSON document.

Review comments on -l, --load:

  • (Willi Ballenthin) Why not just detect from the input file type whether it is a JSON document? A separate flag is not needed.
  • (Moritz Raabe) Agreed.
  • --analyze-functions <addr...>: Restrict emulation analysis to specified function addresses.
  • --signatures: Path to custom FLIRT .sig/.pat signature file or directory.

Review comments on --signatures:

  • (Willi Ballenthin) Is this an implementation detail of the backend? Might it change? Is a user likely to change it? Recommend getting rid of it, or making it more hidden.
  • (Moritz Raabe) Yes. Let us try to get rid of it or simplify and clarify it.
  • (Lakshit Verma) FLIRT signatures help reverse Rust and Go binaries. Keep this option.
  • -L, --large-file: Allow processing files larger than 100 MB.

Review comment on -L, --large-file:

  • (Moritz Raabe) Is this needed? If people want to do it, let them. I vote to remove it and process any file.
  • --disable-progress: Disable all CLI progress bars.

3.7.8 Logging and debugging options

  • -d, --debug: Enable debug output on STDERR.
  • -q, --quiet: Disable all status messages on STDOUT except fatal errors.

Part 4: CLI decisions from the follow-up

The CLI specification above refines the earlier design options.

Apply these decisions:

  • Use --string-type and --no-string-type for string kinds. This replaces the current --no and --only.
  • Use --no-layout only as a temporary flag. The final design uses --view flat instead. Removing --no-layout entirely is under debate.
  • Use --tag and --no-tag for filter by tag. This replaces the proposed --filter.
  • Use --view for display modes. Do not add a separate --plain mode.
  • Remove --large-file. Process any file.
  • Keep --signatures. FLIRT signatures help reverse Rust and Go binaries.
  • Detect JSON input from the file type. Remove the -l, --load flag.
  • Use --json-summary for token-efficient output. Consider the TOON format.

Open questions

  1. Should --query accept multiple expressions?
  2. Does --columns apply to the tree view?
  3. When does --json-summary ship relative to --json?