vee1e

vee1e / 828e8ff65b06fc44691ec5bf28ec5d4f

Last active 4 days ago

Like 0

Revision e50de023e0bfc52554c495de5cd347dbdf7a4bb3

quantumstrand-migration.md Raw

QUANTUMSTRAND FLOSS Master Migration Guide

  • Status: draft for GSoC / integration planning
  • Branches compared: quantumstrand @ f367032 vs master @ 98e2259
  • Merge-base: 9a7f351
  • Divergence: ~313 commits on quantumstrand not in master; ~44 commits on master not in quantumstrand
  • Local worktrees:

This document is a full inventory of what exists on quantumstrand today, how it differs from the GSoC 2026 proposal assumptions, and a concrete plan to fold QS features, workflows, CI, and the viewer interface into master without breaking FLOSS consumers.

1. Executive summary

QUANTUMSTRAND (QS) is already a mature parallel product living under floss/qs/, with its own CLI (qs), JSON schema, PyInstaller build, GitHub Actions, tag databases (Git LFS), OSS rebuild pipeline, and a React single-file viewer (qs-viewer/).

Critical reality vs. the original GSoC proposal:

Proposal assumption (older) Current quantumstrand state
Start with PE-only layout + thin adapter PE + ELF + Mach-O (thin & fat) layout already implemented
Build OSS pipeline from scratch scripts/build_oss_db.py + bi-weekly build-oss-db.yml already land PRs
Migrate DBs to Git LFS first Done (.gitattributes + LFS for DBs and FLIRT .sig files)
Slice 1 creates qs_integration.py + --quantum Not started -- QS is still a separate binary/entry point
Merge into FLOSS ResultDocument Not started -- QS has its own Pydantic ResultDocument (meta + recursive layout)
floss-viewer/ port Exists as qs-viewer/ consuming QS-native JSON only

Bottom line: most "feature build" work on the QS experiment branch is done. The remaining hard problem is integration architecture: how to expose QS capabilities through FLOSS's CLI/JSON/renderers while preserving backward compatibility, then retire the parallel codename.

Recommended posture: treat quantumstrand as the feature source of truth, rebase/merge recent master into it (or land incremental PRs onto master), and use a slice-based integration that starts with optional enrichments rather than rewriting FLOSS's static-string path overnight.

2. Worktree setup (local)

Already created:

# from repo root (quantumstrand checkout)
git worktree add ../flare-floss-master master

# quantumstrand is the primary worktree; a second worktree for the same branch
# is not needed and git will refuse it if the branch is already checked out.
git worktree list
# /Users/verma/Projects/Mandiant/flare-floss         ... [quantumstrand]
# /Users/verma/Projects/Mandiant/flare-floss-master  ... [master]

Use the master worktree for migration implementation branches:

cd ../flare-floss-master
git checkout -b feat/qs-integration-slice-1

Note: LFS objects on the quantumstrand worktree may appear as pointer stubs until git lfs pull is run. Master currently has full FLIRT .sig binaries without LFS; quantumstrand tracks them via LFS.

3. Architecture comparison

3.1 Master FLOSS (today)

flare-floss/
|-- floss/
|   |-- main.py              # sole CLI entry: floss
|   |-- results.py           # dataclasses ResultDocument
|   |-- strings.py           # static ASCII/UTF-16 extraction
|   |-- stackstrings.py / tightstrings.py / string_decoder.py
|   |-- language/            # Go / Rust language-specific strings
|   |-- render/{default,json,sanitize}.py
|   \-- sigs/                # FLIRT signatures (plain git blobs)
|-- .github/workflows/
|   |-- build.yml, tests.yml, publish.yml, black-format.yml
|   \-- verify-pins.yml      # pinact -- master-only hygiene
|-- scripts/                 # IDA/Ghidra/Binja/r2/x64dbg import helpers
\-- doc/

FLOSS ResultDocument shape (simplified):

{
  "metadata": {
        "file_path": "...",
        "version": "...",
        "imagebase": 0,
        "min_length": 4,
        "runtime": {},
        "language": ""
    },
  "analysis": { "enable_static_strings": true, "enable_stack_strings": true, ... },
  "strings": {
    "static_strings":   [{ "string": "...", "offset": 0, "encoding": "ASCII" }],
    "stack_strings":    [...],
    "tight_strings":    [...],
    "decoded_strings":  [...],
    "language_strings": [...],
    "language_strings_missed": [...]
  }
}

StaticString has only: string, offset, encoding. No tags, no section, no structure.

Analysis pipeline: vivisect-backed deobfuscation + language ID + static strings from whole-file extraction. No recursive PE/ELF/Mach-O layout tree; no string tagging DBs.

3.2 Quantumstrand (today)

flare-floss/   (on quantumstrand)
|-- floss/
|   |-- main.py              # UNCHANGED vs master (still pure FLOSS)
|   |-- results.py           # UNCHANGED
|   \-- qs/                  # entire experiment lives here
|       |-- main.py          # ~2585 lines: layout + tag + render + CLI
|       |-- bulk.py          # directory batch runner
|       |-- db/
|       |   |-- {gp,oss,expert,winapi}.py
|       |   \-- data/        # Git LFS: .jsonl.gz, .bin, capa.jsonl
|       \-- scripts/         # VT fetch, GP generation helpers
|-- qs-viewer/               # Vite + React single-file GUI
|-- scripts/build_oss_db.py  # vcpkg + jh automation
|-- .github/
|   |-- pyinstaller/qs.spec
|   \-- workflows/
|       |-- build-qs.yml
|       |-- tests-qs.yml
|       |-- build-oss-db.yml
|       \-- web-release.yml  # GitHub Pages for qs-viewer
|-- .gitattributes           # LFS rules
\-- pyproject.toml           # extra [qs] deps + `qs` console script

QS ResultDocument shape (Pydantic):

{
  "meta": {
    "version": "0.3.0",
    "timestamp": "...",
    "sample": { "md5": "...", "sha1": "...", "sha256": "...", "path": "..." },
    "min_str_len": 4
  },
  "layout": {
    "name": "pe",
    "offset": 0,
    "length": 3584,
    "strings": [
      {
        "string": "WriteLine",
        "offset": 1036,
        "size": 18,
        "encoding": "ascii",
        "tags": ["#common", "#msvc"],
        "structure": ""
      }
    ],
    "children": [
      { "name": "header", "offset": 0, "length": 512, "strings": [], "children": [] },
      { "name": ".text",  "offset": 512, "length": 1024, "strings": [...], "children": [] }
    ]
  }
}

Important: QS does not implement stack/tight/decoded string recovery. It is a contextual static-strings tool. FLOSS remains the deobfuscation engine.

3.3 Entry points and packaging

Concern Master Quantumstrand
Console scripts floss = floss.main:main + qs = floss.qs.main:main
Optional deps dev, build + qs extra: dnfile, colorama, machofile, msgspec, python-lancelot, pyelftools
PyInstaller floss.spec + qs.spec (bundles all DB data paths)
Binary name floss quantumstrand / quantumstrand.exe
Version floss.version -> 3.1.1 QS reports QS_VERSION = "0.3.0" independently

viv-utils pin is slightly higher on QS (>=0.8.0 vs >=0.7.9); otherwise core FLOSS deps are aligned, though master has newer dev/tooling pins (black, mypy, pre-commit, etc.).

4. Feature inventory (what must move)

4.1 Binary layout analysis (floss/qs/main.py)

Recursive Layout tree with non-overlapping ordered children and gap strings.

Format Function Capabilities
PE compute_pe_layout Sections, header/overlay segments, Authenticode, resources (readable names via PE_RESOURCE_TYPES), gap nodes; structures: section headers, import/export tables, rich header; #code via lancelot BinExport2 ranges; #reloc via reloc table; single-byte XOR MZ decoding
ELF compute_elf_layout Robust section/segment fallback, SHT_NOBITS skip, exec ranges, narrow metadata regions (ELF/program/section headers, string/symbol tables); ELFError / out-of-range hardening
Mach-O compute_macho_layout Thin + fat; segments/sections; code signature, certs, entitlements plist
Fallback SegmentLayout(name="binary") Whole-file string extraction

GSoC proposal note on section parsing: memory-dumped PEs (SectionAlignment + slack) vs disk PEs (FileAlignment) remain a testing concern. PE path already truncates out-of-range sections and logs warnings, but corpus coverage for AD1/static vs memory dumps should still be expanded during integration (Slice 3 testing deliverable in the proposal is still valid).

4.2 String tagging system

Taggers loaded by load_databases():

Source Module / data Tags produced Default render rule
WinAPI db/winapi + apis.txt.gz, dlls.txt.gz #winapi mute (default)
Expert db/expert + capa.jsonl e.g. #capa highlight
OSS libs db/oss + 17x *.jsonl.gz + crt/msvc_v143.jsonl.gz #zlib, #openssl, #msvc, ... mute
Global prevalence db/gp + gp.jsonl.gz, cwindb, hash DBs #common mute
Junk code strings gp/junk-code.jsonl.gz #code-junk mute
Layout-derived PE/ELF code & reloc ranges #code, #reloc hide
XOR decode PE path #decoded default
Duplicates same string text twice #duplicate mute

Post-processing:

  • remove_false_positive_lib_strings -- drops library tags if too few hits (default threshold 5).
  • hide_strings_by_rules -- removes #code / #reloc (etc.) from visible render.
  • Tag display consolidates adjacent identical tag groups; suppresses #common when more specific tags exist.

4.3 Tag databases (Git LFS)

.gitattributes:

floss/qs/db/data/**/*.bin  filter=lfs ...
floss/qs/db/data/**/*.gz   filter=lfs ...
floss/qs/db/data/**/*.jsonl filter=lfs ...
floss/sigs/*.sig           filter=lfs ...

30 LFS-tracked objects on quantumstrand. Master has no .gitattributes and stores FLIRT sigs as normal blobs (~15 MB total).

Checked-in OSS DBs (17 + CRT): brotli, bzip2, cryptopp, curl, detours, jemalloc, jsoncpp, kcp, liblzma, libpcap, libsodium, mbedtls, openssl, sqlite3, tomcrypt, wolfssl, zlib, + crt/msvc_v143.

Build manifest (libraries.json) already lists 62 vcpkg packages for the automated pipeline (expansion beyond the original "top 25" goal). Checked-in DBs and the build matrix are intentionally not 1:1 yet; rebuild CI merges into existing .jsonl.gz files.

4.4 OSS database construction pipeline

libraries.json --> vcpkg (x64-windows-static, msvc143, release)
                       |
                       v
                 jh (lancelot-bin) extract strings from .lib
                       |
                       v
              convert -> JSONL -> gzip
                       |
                       v
         merge with existing DB (new wins on collision)
                       |
                       v
         PR to quantumstrand (build-oss-db.yml)

Key implementation: scripts/build_oss_db.py (~878 lines)

  • Modular Vcpkg, JHExtractor, Converter classes
  • Cross-library strings are kept (not globally deduped); per-library still collapses duplicates by default
  • Metrics written to build_metrics.json
  • Windows-focused by design (PE-first rationale documented in floss/qs/db/data/oss/readme.md)

CI: .github/workflows/build-oss-db.yml

  • Schedule: bi-weekly (0 0 1,15 * *) + workflow_dispatch
  • Runs on windows-latest, builds jh from williballenthin/lancelot
  • Opens PR with base quantumstrand (must change to master after merge)
  • Requires contents: write + pull-requests: write

4.5 CLI & bulk analysis

qs CLI (floss/qs/main.py:main):

Flag Behavior
path sample path
-n / --minimum-length min string length (default 4)
-j / --json emit QS ResultDocument JSON
-l / --load re-render from saved QS JSON
-d / -q debug / quiet (reuses floss.main.set_log_config)

No mute/highlight/hide CLI overrides yet (hardcoded tag_rules in main). GSoC proposal's "tag filters as CLI arguments" is still open work.

floss/qs/bulk.py: walks a directory, shells python -m floss.qs.main --json, optional rendered .txt.

4.6 Terminal rendering

Rich-based columns: tags | string | structure | offset, with box-drawing layout delimiters, muted/highlight styles, tag-group consolidation. Depends on colorama for Windows console.

4.7 Web viewer (qs-viewer/)

Item Detail
Stack Vite + React + TypeScript, single-file build
Input QS JSON only (meta + recursive layout)
Features drag-drop upload, search, min length, tag multi-select, structure multi-select, show/hide untagged / no-structure, column toggles, copy filtered strings
Noisy tags default #common, #duplicate, #code, #reloc, #code-junk treated as muted styling; #capa highlighted
Deploy web-release.yml -> GitHub Pages on qs-viewer/** changes to quantumstrand
Sample data src/pma0303_qs.json baked in for preview

Not yet: browsing inside tag databases when clicking a tag (GSoC / issues #782, #784 -- still open product work).

4.8 Tests (quantumstrand-only)

File Focus
tests/test_qs.py end-to-end PMA sample, JSON round-trip
tests/test_qs_unit.py Range / Slice
tests/test_qs_offset_ranges.py reloc/code range structure
tests/test_qs_code_ranges.py merge overlapping ranges
tests/test_qs_elf.py ELF layout sanity
tests/test_qs_macho.py Mach-O thin layout + structures
tests/test_qs_pma0101.py PMA fixture expectations
tests/test_qs_oss_db.py OSS DB load/query
tests/test_qs_build_oss_db.py build script unit + orchestration (~625 lines)
tests/test_gp_db.py GP + hash DBs
tests/test_winapi_db.py winapi DB

tests-qs.yml runs pytest -k qs on Ubuntu with LFS + submodules.

Test data submodule: quantumstrand pins tests/data @ 6eba5b1 (includes PMA samples QS needs); master pins older 53e9101. Integration must bump the submodule on master carefully.

4.9 CI / release matrix

Workflow Master Quantumstrand Migration action
tests.yml Yes Yes keep; extend matrix or jobs for QS
build.yml Yes FLOSS binary Yes (minor branch filter) keep FLOSS build; optionally embed QS later
publish.yml Yes Yes unchanged
black-format.yml Yes Yes unchanged
verify-pins.yml Yes No bring to integrated branch (master hygiene)
build-qs.yml No Yes QS PyInstaller either merge into build.yml or keep as floss-qs artifact job on master
tests-qs.yml No Yes fold into tests.yml or keep named job on master
build-oss-db.yml No Yes retarget base branch -> master
web-release.yml No Yes Pages retarget to master; path may become floss-viewer/
mypy.ini extras -- intervaltree, lancelot, capa, virustotal3, colorama merge

4.10 Ancillary / scripts

Path Role Migrate?
floss/qs/scripts/extract_strings.py string extract helper yes (or fold into scripts/)
floss/qs/scripts/fetch_vt_hashes.py VT corpus for GP / library selection yes (dev tooling)
floss/qs/scripts/generate_gp_db.py GP DB generation yes
floss/qs/scripts/query_string.py DB query debug CLI yes
floss/qs/db/data/expert/import_from_capa.py capa -> expert rules yes
floss/qs/db/data/oss/jh_to_qs.py CSV->JSONL (legacy; build script supersedes bulk use) keep for manual rebuilds
floss/logging_.py timestamps in log format on QS only decide: adopt on master or keep FLOSS format

5. What does not need to be reinvented

These GSoC "new features" are largely complete on quantumstrand and should be ported as-is, then refined:

  1. ELF section parsing -- done with robustness work (PR history includes #1312, #1319).
  2. Git LFS for DBs -- done (#1309).
  3. Automatic tag DB construction pipeline -- done (script + bi-weekly CI).
  4. Mach-O static layout -- done beyond original "StaticString-only" sketch.
  5. qs-viewer baseline -- done; DB peek / tag drill-down still missing.

Remaining product gaps from the proposal + community threads:

  1. FLOSS integration (--quantum / always-on enrichments, merged JSON).
  2. CLI tag filters (mute/hide/highlight overrides).
  3. Viewer: open/browse tag databases (#782, #784).
  4. Diverse section-parsing corpus (disk vs memory dump, binwalk padding, stripped malware).
  5. Library manifest expansion -- pipeline supports 62 libs; only 17 DBs checked in until CI rebuilds land.
  6. Deprecate QUANTUMSTRAND codename after stable merge.

6. Integration design (target state)

6.1 Design principles

  1. Backward compatible JSON: legacy FLOSS roots analysis, metadata, strings remain valid without QS fields.
  2. --quantum is a feature flag: default off for this migration phase; not flipping to default-on yet.
  3. Do not break deobfuscation: stack/tight/decoded recovery stays FLOSS; when --quantum is on, text tags apply to static + stack/tight/decoded/language strings; section/structure mapping is primarily for static offsets.
  4. No GUI migration for now: qs-viewer/ and Pages deploy stay out of scope.
  5. QS runtime deps are core: python-lancelot, pyelftools, machofile, msgspec, etc. install with FLOSS always (not a long-lived optional extra).
  6. Single binary: only floss; no parallel qs / quantumstrand artifact.

6.2 Target merged JSON (proposal-aligned, updated)

When QS mode is enabled:

{
  "analysis": { "enable_static_strings": true, "...": "..." },
  "metadata": {
    "file_path": "dotnet-hello.exe",
    "version": "3.x.y",
    "qs_enabled": true,
    "qs_version": "0.3.0"
  },
  "strings": {
    "static_strings": [
      {
        "string": "WriteLine",
        "offset": 1036,
        "encoding": "ASCII",
        "qs_tags": ["#common", "#msvc"],
        "qs_structure": "",
        "qs_section": ".text"
      }
    ],
    "stack_strings": [],
    "decoded_strings": []
  },
  "qs_layout": {
    "name": "pe",
    "offset": 0,
    "length": 3584,
    "children": [
      { "name": "header", "offset": 0, "length": 512, "children": [] },
      { "name": ".text", "offset": 512, "length": 1024, "children": [] }
    ]
  }
}

Semantics:

  • qs_layout holds structural hierarchy only (no per-node string lists required for FLOSS JSON -- avoids doubling string storage).
  • Per-string qs_* fields are optional; absent when QS disabled.
  • Existing FLOSS tooling that ignores unknown fields keeps working.
  • Add a checked-in JSON Schema + tests/test_results_schema_compat.py as proposed.

6.3 Mapping QS layout -> FLOSS static strings

FLOSS StaticString(offset, string, encoding)
        |
        |- match offset in layout node -> qs_section = nearest section/segment name
        |- match offset in Structure map -> qs_structure = structure name
        \- tag string text via load_databases() -> qs_tags

Edge cases:

Case Strategy
String only in FLOSS (language/Go extractor path) tag by text; section may be unknown
String only in QS layout gaps optional: union into static_strings or leave QS-only in layout
Encoding enum mismatch (ASCII vs ascii) normalize in adapter
Multiple sections claim offset prefer deepest layout leaf
Memory dump / invalid PE layout may fall back to binary; tags still apply

6.4 Target directory structure (post-migration)

Aligned with the proposal, adjusted to current code:

flare-floss/
|-- floss/
|   |-- main.py                 # --quantum / tag filter CLI (feature flag, default off)
|   |-- results.py              # optional qs_* fields on string types + metadata
|   |-- qs/                     # layout, tags, dbs (library code; no separate shipped CLI)
|   |   |-- main.py             # may remain as module internals; do not ship `qs` console script
|   |   |-- layout.py           # NEW: split compute_*_layout out of main.py (optional refactor)
|   |   |-- tags.py             # NEW: load_databases + tag rules (optional refactor)
|   |   |-- render.py           # NEW: rich renderer (optional refactor)
|   |   |-- db/                 # unchanged data + loaders
|   |   \-- ...
|   |-- qs_integration.py       # NEW: FLOSS<->QS adapter (proposal Slice 1)
|   \-- render/
|       |-- default.py          # optional QS-aware columns when enabled
|       \-- json.py             # serializes optional qs fields
|-- scripts/build_oss_db.py
|-- tests/
|   |-- test_qs_*.py            # existing
|   |-- test_qs_integration_cli.py
|   |-- test_qs_layout_mapping.py
|   |-- test_qs_tagging.py
|   |-- test_qs_visibility.py
|   \-- test_results_schema_compat.py
|-- doc/
|   |-- quantumstrand-migration.md   # this file
|   \-- quantumstrand.md             # user-facing architecture + usage (to write)
\-- .github/workflows/
    |-- tests.yml               # include qs jobs
    |-- build.yml               # single floss binary (QS DBs embedded)
    |-- build-oss-db.yml        # base: master
    \-- verify-pins.yml

Not migrated (locked): qs-viewer/, web-release.yml, standalone qs entry point / build-qs.yml / qs.spec.

Refactor recommendation: floss/qs/main.py is a 2.5k-line monolith. Prefer splitting during migration without changing behavior, so FLOSS integration imports layout/tags cleanly. Not a blocker for Slice 1 if adapter imports from floss.qs.main temporarily.

7. Slice-based migration plan (updated)

The GSoC four-slice model remains the right process; contents are rebased onto current code.

Slice 0 -- Foundation (prerequisite, 0.5-1 week)

Goal: make master able to host QS artifacts safely.

  1. Merge or rebase latest master into integration branch (deps, verify-pins, AGENTS.md, mypy/black bumps).
  2. Land .gitattributes + Git LFS for:
    • floss/qs/db/data/**
    • optionally migrate floss/sigs/*.sig to LFS (coordinate: history rewrite not required if only new commits use LFS; existing master blobs remain until rewritten -- prefer forward-only LFS for new paths first, then sigs carefully).
  3. Bump tests/data submodule to QS pin if PMA fixtures are required.
  4. Document git lfs install && git lfs pull in doc/installation.md.
  5. CI: ensure all workflows that need DBs set lfs: true.

Exit criteria: clean install on Linux CI with LFS; FLOSS tests still green; no accidental 15MB re-add of DB binaries as non-LFS.

Slice 1 -- Adapter baseline (1-2 weeks)

Goal: FLOSS can optionally compute layout and attach section/structure to static strings (PE first).

Deliverables:

  • floss/qs_integration.py
    • analyze_layout(sample_bytes) -> ResultLayout-like tree
    • map_strings_to_layout(static_strings, layout) -> enriched strings
    • to_qs_document / from_floss_document helpers for tests
  • CLI: --quantum (store_true, default False) on floss.main — permanent feature flag for this migration phase
  • results.StaticString: optional qs_tags, qs_section, qs_structure (default empty / omit in JSON if unset -- decide one serialization policy and stick to it)
  • metadata.qs_enabled: bool
  • qs_layout optional top-level field on ResultDocument (or nested under metadata -- prefer top-level as in proposal)
  • Tests: test_qs_integration_cli.py, test_qs_layout_mapping.py (PE fixtures from PMA)
  • JSON schema draft + load old FLOSS JSON without error

Non-goals this slice: tagging DBs, viewer rename, OSS CI retarget.

Risk: lancelot panics -- already mitigated with broad BaseException catch in QS PE path; keep that in the adapter.

Slice 2 -- Tagging + CI + DBs (2-3 weeks)

Goal: full tag pipeline available under FLOSS.

  • Load DBs via existing floss.qs.db.* loaders
  • Apply taggers to static and stack / tight / decoded / language strings
  • CLI visibility controls, e.g.:
    • --qs-hide-tag #code
    • --qs-mute-tag #common
    • --qs-highlight-tag #capa
    • or a compact --tag-rule '#code=hide,#capa=highlight'
  • Default rules match current QS behavior when --quantum is on
  • Copy/adapt workflows:
    • tests-qs -> master
    • build-oss-db.yml base -> master
    • PyInstaller data paths if FLOSS binary should embed DBs when built with QS
  • Package data: ensure wheels/sdists include floss/qs/db/data (MANIFEST / package-data)
  • Tests: test_qs_tagging.py, test_qs_visibility.py, DB load tests

Dependency decision (locked): always install QS deps + ship DBs with FLOSS.

Slice 3 -- Multi-format layout + corpus (2 weeks)

Already implemented on QS branch -- focus is wiring + testing under FLOSS, not greenfield parsers.

  • Ensure adapter calls compute_layout (PE/ELF/Mach-O dispatch) not PE-only

  • ELF/Mach-O mapping tests under FLOSS CLI

  • Build corpus matrix:

    Source type Why
    Clean PE from disk / VT FileAlignment, normal sections
    Memory / crash dump PE SectionAlignment, slack, heap garbage
    binwalk-carved / padded imperfect offsets
    Stripped ELF section vs segment fallback
    Fat Mach-O multi-arch children
    XOR'd PE #decoded path
    Go/Rust FLOSS language samples interaction with language strings
  • Document sample-source consideration in doc/quantumstrand.md

Slice 4 -- Polish & single-binary cleanup (ongoing)

  • Out of scope: qs-viewer/, web-release.yml, GUI / DB-browser work — do not migrate
  • Do not ship a parallel qs / quantumstrand binary — only floss with --quantum
  • Drop or never land on master: build-qs.yml, .github/pyinstaller/qs.spec, project.scripts entry qs = ...
  • Apply text taggers to static and stack / tight / decoded (and language) strings when --quantum
  • User docs + changelog for --quantum
  • Community feedback loop (proposal Figure 2)

Optional Slice 5 -- Performance & DB productization

  • Audit jh_to_qs / build_oss_db for CI time (proposal concern)
  • Land expanded OSS DBs via automated PRs (62-lib matrix)
  • Consider multi-triplet matrix only if string coverage gaps are measured
  • Publish release artifacts when DB growth is significant

8. File-level migration checklist

Must copy / land onto master

.gitattributes                     # LFS: qs db data + floss/sigs/*.sig
floss/qs/                          # package (layout, tags, dbs) — not a separate CLI long-term
scripts/build_oss_db.py
.github/workflows/tests-qs.yml     # or fold into tests.yml
.github/workflows/build-oss-db.yml
.github/mypy/mypy.ini              # merge QS ignore modules
pyproject.toml                     # QS deps as core runtime deps (not optional extra)
tests/test_qs*.py
tests/test_gp_db.py
tests/test_winapi_db.py
tests/data                         # submodule bump

Do not copy to master (locked out of scope):

qs-viewer/
.github/workflows/web-release.yml
.github/workflows/build-qs.yml      # no parallel quantumstrand binary
.github/pyinstaller/qs.spec
# pyproject [project.scripts] qs = ...  — do not add

Must implement new (not pure copy)

floss/qs_integration.py
floss/main.py                      # flags + call adapter
floss/results.py                   # optional fields
floss/render/default.py            # optional QS rendering
floss/render/json.py               # already dataclass-friendly; ensure qs_* serialize
doc/quantumstrand.md               # user docs
schemas/floss-results.json         # optional but proposed
tests/test_qs_integration_*.py
tests/test_results_schema_compat.py

Must retarget after merge

Item From To
build-oss-db.yml PR base quantumstrand master
build-qs.yml quantumstrand drop, no parallel binary
tests-qs.yml triggers quantumstrand or fold into tests.yml
web-release.yml quantumstrand do not migrate now
Release assets quantumstrand-* only floss-*; embed QS DBs in floss PyInstaller datas when --quantum support lands
README / qs readme experiment language integrated --quantum feature language

Master-only assets to preserve

.github/workflows/verify-pins.yml
AGENTS.md
newer dependabot pins in pyproject / requirements

Explicit non-goals for first merge

  • Replacing FLOSS static string extractor with QS extractor wholesale
  • Making stack/decoded strings participate in layout trees (tags yes; section/structure usually N/A)
  • Migrating qs-viewer / any GUI / GitHub Pages web-release
  • Shipping a parallel qs / quantumstrand binary
  • Multi-OS vcpkg matrix for OSS DBs (Windows static remains justified)
  • Rewriting full git history for LFS (forward-only LFS tracking of sigs + DBs is enough)

9. Compatibility & risk matrix

Risk Impact Mitigation
JSON field additions break strict parsers Medium optional fields; schema version; golden tests for legacy JSON
LFS not pulled -> empty DBs / broken tags High CI lfs: true; clear install docs; runtime warn if DB missing
lancelot / native deps fail on some platforms Medium catch panics; degrade to section-only PE via pefile
PyInstaller size growth from DBs Medium measure; document; optional slim builds
Dual CLI confusion (floss vs qs) N/A locked: no qs binary on master
Test submodule mismatch Medium single pin; QS tests skip if fixture absent
Tag false positives from stale OSS DBs Medium keep build-oss-db cron; FP lib threshold
Section mapping wrong on memory dumps Medium corpus tests; document limitations
Logging format change (logging_.py timestamps) Low intentional — adopt QS timestamps globally; note in changelog
requirements.txt drift master vs QS Low regenerate from one lock strategy after merge

10. Testing strategy

Unit

  • Layout pure functions (OffsetRanges, range merge, Range/Slice)
  • DB loaders (oss, gp, winapi, expert)
  • build_oss_db converters/merge without network

Integration

  • floss --quantum sample.exe -j produces valid merged JSON
  • floss sample.exe -j bit-identical structure to pre-QS (no qs fields)
  • Load legacy JSON with floss -l still works
  • Tag rules hide/mute/highlight

Golden / fixture

  • PMA 01-01, 03-03 (existing QS tests)
  • ELF + Mach-O fixtures in testfiles submodule
  • One intentionally truncated / padded PE

CI gates before merge to master

pytest                          # full suite
pytest -k qs                    # QS suite
mypy (project config)
black / isort / pre-commit
verify-pins on workflow edits
smoke: pyinstaller floss (with QS DB datas embedded)

Manual acceptance

  • Terminal UX for floss --quantum matches expected tags/sections on a known sample
  • OSS workflow dry-run (workflow_dispatch) opens a PR against master

11. Suggested PR stack (Graphite / stacked PRs)

Keep PRs reviewable (~reviewer-hour sized):

  1. chore: LFS + .gitattributes + install docs (no feature code)
  2. feat: vendor floss/qs/ package + deps + unit tests (no FLOSS CLI wiring)
  3. feat: qs_integration + --quantum PE section mapping
  4. feat: tagging + tag-rule CLI
  5. ci: fold QS tests + OSS DB workflow retarget
  6. feat: multi-format layout coverage tests under FLOSS
  7. docs: usage + schema for --quantum (no viewer PR)
  8. chore: single floss release binary embeds QS DBs; no quantumstrand artifact

Avoid one mega-PR of 15k insertions even though the branch already has them; landing in stacks preserves bisectability.

12. Mapping GSoC proposal deliverables -> status

Deliverable Status on quantumstrand Migration work left
Merge-ready QS codebase Yes (experiment branch) Integrate into master FLOSS UX
Section mapping PE Done Adapter + FLOSS fields
ELF section parsing Done Wire through adapter + tests on master
Mach-O static Done Wire + tests
Tag databases + loaders Done Package data on master builds
OSS auto pipeline Done Retarget to master; expand checked-in DBs
Git LFS Done Introduce on master
Tag filters CLI Partial (hardcoded rules only) Implement args
Merged FLOSS+QS JSON Not started Core integration
JSON Schema + compat tests Not started Add
GUI viewer Done on QS branch Out of scope — do not migrate
Visualize / browse DBs Not started Out of scope for this migration
qs_integration.py Not started Slice 1
--quantum flag Not started Slice 1 (opt-in feature flag)
Single floss binary only N/A Locked — no parallel qs artifact
Tag stack/decoded/tight Static-only in QS CLI Yes under FLOSS --quantum

13. Decisions (locked)

# Question Decision
1 --quantum default Feature flag only for now — opt-in via --quantum; not the default for static strings. Revisit later if desired.
2 QS deps (lancelot / pyelftools / machofile / …) Always install — promote into core runtime dependencies (not a [qs] extra long-term).
3 Standalone qs / quantumstrand binary No — single floss binary; QS only via --quantum. Drop build-qs.yml / qs.spec / qs console script (do not land on master).
4 Viewer / GUI Do not migrate. Ignore qs-viewer/, web-release.yml, and all GUI work for this migration.
5 FLIRT .sig + QS DBs on LFS All LFS — both floss/sigs/*.sig and floss/qs/db/data/** tracked with Git LFS on master.
6 JSON field naming on FLOSS objects qs_* prefix (qs_tags, qs_section, qs_structure, qs_layout, qs_enabled) for safety / clarity.
7 Tag decoded / stack / tight strings Yes — apply text-based taggers to those string types too. Layout section/structure usually N/A for non-static strings.
8 Logging timestamps Adopt QS format globally — all FLOSS logs get %Y-%m-%d %H:%M:%S timestamps (same as quantumstrand floss/logging_.py).

13.1 Logging format (decision #8 locked)

Master (pre-migration) Target (post-migration)
Format string {color}%(levelname)s{RESET}: %(name)s: %(message)s
Date format (none)
Example line INFO: floss.main: extracting strings

Action on master: take the quantumstrand floss/logging_.py change as-is when integrating. Global for all FLOSS logs (not gated on --quantum).

14. Operational runbook (post-merge)

Developer setup

git clone --recurse-submodules https://github.com/mandiant/flare-floss.git
cd flare-floss
git lfs install
git lfs pull
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,qs]"   # or whatever extra name is chosen
pytest -k qs

Rebuilding OSS DBs locally (Windows)

# build jh from lancelot, then:
python scripts/build_oss_db.py `
  --config floss\qs\db\data\oss\libraries.json `
  --jh-path path\to\jh.exe `
  --output-dir floss\qs\db\data\oss `
  --continue-on-error

Viewer

Out of scope for this migration (do not set up or deploy qs-viewer).

15. Success metrics

Migration is "done" when:

  1. A user on master can run floss --quantum sample.exe and get tagged, section-annotated static strings.
  2. floss sample.exe -j without the flag remains backward compatible.
  3. CI on master runs QS unit/integration tests with LFS.
  4. Bi-weekly OSS DB workflow opens PRs against master.
  5. Only one shipped binary (floss); --quantum is the sole entry to QS features.
  6. Stack / tight / decoded / language strings receive text tags when --quantum is on.
  7. No regression in FLOSS deobfuscation tests (test_main, language suites, etc.).
  8. GUI / qs-viewer intentionally not part of success criteria.

16. Appendix A -- Tag reference

Tag Meaning Typical visibility
#common globally prevalent string mute
#duplicate repeated string content mute
#code overlaps identified instructions hide
#reloc overlaps relocation data hide
#code-junk junk-code string DB hit mute
#winapi Windows API/DLL name mute
#capa expert/capa-derived interest highlight
#zlib, #openssl, #msvc, ... OSS / CRT library provenance mute
#decoded from XOR-decoded PE image default

17. Appendix B -- PE structures annotated by QS

  • section header
  • import table
  • export table
  • rich header
  • layout regions: header, section names, overlay, Authenticode digital signature, gap, rsrc: ...

ELF: elf header, program header, section header, string table, symbol table

Mach-O: macho header, load command, segment header, section header, code signature, certificates, plist: entitlements

18. Appendix C -- Quick command cheat sheet for explorers

# Diff surface area
git diff --stat master...quantumstrand

# Only non-data code
git diff --name-status master...quantumstrand -- \
  ':(exclude)floss/qs/db/data' ':(exclude)qs-viewer/package-lock.json'

# Run QS locally (quantumstrand worktree)
pip install -e ".[qs]"
qs /path/to/sample -j | head
python -m floss.qs.main /path/to/sample

# Run FLOSS on master worktree
cd ../flare-floss-master
pip install -e ".[dev]"
floss /path/to/sample -j

19. Appendix D -- Divergence notes for integrators

  • floss/main.py and floss/results.py are currently identical between tips of the two branches. Integration edits land cleanly without resolving QS-vs-FLOSS conflicts in those files.
  • Master is ahead on dependency hygiene (verify-pins, AGENTS.md, newer pins). Always merge master -> integration branch first.
  • Quantumstrand FLIRT .sig files in a non-LFS-pulled tree look like 132-byte pointer files; master trees have full multi-MB binaries. Don't commit pointer stubs onto master by mistake.
  • libraries.json (62 libs) is the future set; DEFAULT_FILENAMES in oss.py (17 + CRT) is the runtime set. After CI rebuilds, update DEFAULT_FILENAMES (or auto-discover *.jsonl.gz) so new DBs are actually loaded.

20. Document history

Date Author Notes
2026-07-09 migration planning Initial detailed inventory from quantumstrand @ f367032 vs master @ 98e2259; incorporates GSoC 2026 proposal slices with current-code corrections
2026-07-09 decisions locked Feature-flag --quantum; core deps always; single floss binary; no GUI migration; all LFS including sigs; qs_* JSON fields; tag non-static strings; adopt QS log timestamps globally (§13.1)

When this plan is executed, keep this file updated per slice (checkboxes, open decisions, and any schema changes). Prefer linking PRs next to each Slice checklist item.