QUANTUMSTRAND FLOSS Master Migration Guide

• Status: draft for GSoC / integration planning
• Branches compared: quantumstrand @ f367032 vs master @ 98e2259
• Merge-base: 9a7f351
• Divergence: ~313 commits on quantumstrand not in master; ~44 commits on master not in quantumstrand

This document is a full inventory of what exists on quantumstrand today, how it differs from the GSoC 2026 proposal assumptions, and a concrete plan to fold QS features, workflows, CI, and the viewer interface into master without breaking FLOSS consumers.

1. Executive summary

QUANTUMSTRAND (QS) is already a mature parallel product living under floss/qs/, with its own CLI (qs), JSON schema, PyInstaller build, GitHub Actions, tag databases (Git LFS), OSS rebuild pipeline, and a React single-file viewer (qs-viewer/).

Critical reality vs. the original GSoC proposal:

Bottom line: most "feature build" work on the QS experiment branch is done. The remaining hard problem is integration architecture: how to expose QS capabilities through FLOSS's CLI/JSON/renderers while preserving backward compatibility, then retire the parallel codename.

Recommended posture: treat quantumstrand as the feature source of truth, rebase/merge recent master into it (or land incremental PRs onto master), and use a slice-based integration that starts with optional enrichments rather than rewriting FLOSS's static-string path overnight.

2. Worktree setup (local)

Already created:

# from repo root (quantumstrand checkout)
git worktree add ../flare-floss-master master

# quantumstrand is the primary worktree; a second worktree for the same branch
# is not needed and git will refuse it if the branch is already checked out.
git worktree list
# /Users/verma/Projects/Mandiant/flare-floss         ... [quantumstrand]
# /Users/verma/Projects/Mandiant/flare-floss-master  ... [master]

Use the master worktree for migration implementation branches:

cd ../flare-floss-master
git checkout -b feat/qs-integration-slice-1

Note: LFS objects on the quantumstrand worktree may appear as pointer stubs until git lfs pull is run. Master currently has full FLIRT .sig binaries without LFS; quantumstrand tracks them via LFS.

3. Architecture comparison

3.1 Master FLOSS (today)

flare-floss/
|-- floss/
|   |-- main.py              # sole CLI entry: floss
|   |-- results.py           # dataclasses ResultDocument
|   |-- strings.py           # static ASCII/UTF-16 extraction
|   |-- stackstrings.py / tightstrings.py / string_decoder.py
|   |-- language/            # Go / Rust language-specific strings
|   |-- render/{default,json,sanitize}.py
|   \-- sigs/                # FLIRT signatures (plain git blobs)
|-- .github/workflows/
|   |-- build.yml, tests.yml, publish.yml, black-format.yml
|   \-- verify-pins.yml      # pinact -- master-only hygiene
|-- scripts/                 # IDA/Ghidra/Binja/r2/x64dbg import helpers
\-- doc/

FLOSS ResultDocument shape (simplified):

{
  "metadata": {
        "file_path": "...",
        "version": "...",
        "imagebase": 0,
        "min_length": 4,
        "runtime": {},
        "language": ""
    },
  "analysis": { "enable_static_strings": true, "enable_stack_strings": true, ... },
  "strings": {
    "static_strings":   [{ "string": "...", "offset": 0, "encoding": "ASCII" }],
    "stack_strings":    [...],
    "tight_strings":    [...],
    "decoded_strings":  [...],
    "language_strings": [...],
    "language_strings_missed": [...]
  }
}

StaticString has only: string, offset, encoding. No tags, no section, no structure.

Analysis pipeline: vivisect-backed deobfuscation + language ID + static strings from whole-file extraction. No recursive PE/ELF/Mach-O layout tree; no string tagging DBs.

3.2 Quantumstrand (today)

flare-floss/   (on quantumstrand)
|-- floss/
|   |-- main.py              # UNCHANGED vs master (still pure FLOSS)
|   |-- results.py           # UNCHANGED
|   \-- qs/                  # entire experiment lives here
|       |-- main.py          # ~2585 lines: layout + tag + render + CLI
|       |-- bulk.py          # directory batch runner
|       |-- db/
|       |   |-- {gp,oss,expert,winapi}.py
|       |   \-- data/        # Git LFS: .jsonl.gz, .bin, capa.jsonl
|       \-- scripts/         # VT fetch, GP generation helpers
|-- qs-viewer/               # Vite + React single-file GUI
|-- scripts/build_oss_db.py  # vcpkg + jh automation
|-- .github/
|   |-- pyinstaller/qs.spec
|   \-- workflows/
|       |-- build-qs.yml
|       |-- tests-qs.yml
|       |-- build-oss-db.yml
|       \-- web-release.yml  # GitHub Pages for qs-viewer
|-- .gitattributes           # LFS rules
\-- pyproject.toml           # extra [qs] deps + `qs` console script

QS ResultDocument shape (Pydantic):

{
  "meta": {
    "version": "0.3.0",
    "timestamp": "...",
    "sample": { "md5": "...", "sha1": "...", "sha256": "...", "path": "..." },
    "min_str_len": 4
  },
  "layout": {
    "name": "pe",
    "offset": 0,
    "length": 3584,
    "strings": [
      {
        "string": "WriteLine",
        "offset": 1036,
        "size": 18,
        "encoding": "ascii",
        "tags": ["#common", "#msvc"],
        "structure": ""
      }
    ],
    "children": [
      { "name": "header", "offset": 0, "length": 512, "strings": [], "children": [] },
      { "name": ".text",  "offset": 512, "length": 1024, "strings": [...], "children": [] }
    ]
  }
}

Important: QS does not implement stack/tight/decoded string recovery. It is a contextual static-strings tool. FLOSS remains the deobfuscation engine.

3.3 Entry points and packaging

viv-utils pin is slightly higher on QS (>=0.8.0 vs >=0.7.9); otherwise core FLOSS deps are aligned, though master has newer dev/tooling pins (black, mypy, pre-commit, etc.).

4. Feature inventory (what must move)

4.1 Binary layout analysis (floss/qs/main.py)

Recursive Layout tree with non-overlapping ordered children and gap strings.

GSoC proposal note on section parsing: memory-dumped PEs (SectionAlignment + slack) vs disk PEs (FileAlignment) remain a testing concern. PE path already truncates out-of-range sections and logs warnings, but corpus coverage for AD1/static vs memory dumps should still be expanded during integration (Slice 3 testing deliverable in the proposal is still valid).

4.2 String tagging system

Taggers loaded by load_databases():

Post-processing:

• remove_false_positive_lib_strings -- drops library tags if too few hits (default threshold 5).
• hide_strings_by_rules -- removes #code / #reloc (etc.) from visible render.
• Tag display consolidates adjacent identical tag groups; suppresses #common when more specific tags exist.

4.3 Tag databases (Git LFS)

.gitattributes:

floss/qs/db/data/**/*.bin  filter=lfs ...
floss/qs/db/data/**/*.gz   filter=lfs ...
floss/qs/db/data/**/*.jsonl filter=lfs ...
floss/sigs/*.sig           filter=lfs ...

30 LFS-tracked objects on quantumstrand. Master has no .gitattributes and stores FLIRT sigs as normal blobs (~15 MB total).

Checked-in OSS DBs (17 + CRT): brotli, bzip2, cryptopp, curl, detours, jemalloc, jsoncpp, kcp, liblzma, libpcap, libsodium, mbedtls, openssl, sqlite3, tomcrypt, wolfssl, zlib, + crt/msvc_v143.

Build manifest (libraries.json) already lists 62 vcpkg packages for the automated pipeline (expansion beyond the original "top 25" goal). Checked-in DBs and the build matrix are intentionally not 1:1 yet; rebuild CI merges into existing .jsonl.gz files.

4.4 OSS database construction pipeline

libraries.json --> vcpkg (x64-windows-static, msvc143, release)
                       |
                       v
                 jh (lancelot-bin) extract strings from .lib
                       |
                       v
              convert -> JSONL -> gzip
                       |
                       v
         merge with existing DB (new wins on collision)
                       |
                       v
         PR to quantumstrand (build-oss-db.yml)

Key implementation: scripts/build_oss_db.py (~878 lines)

• Modular Vcpkg, JHExtractor, Converter classes
• Cross-library strings are kept (not globally deduped); per-library still collapses duplicates by default
• Metrics written to build_metrics.json
• Windows-focused by design (PE-first rationale documented in floss/qs/db/data/oss/readme.md)

CI: .github/workflows/build-oss-db.yml

• Schedule: bi-weekly (0 0 1,15 * *) + workflow_dispatch
• Runs on windows-latest, builds jh from williballenthin/lancelot
• Opens PR with base quantumstrand (must change to master after merge)
• Requires contents: write + pull-requests: write

4.5 CLI & bulk analysis

qs CLI (floss/qs/main.py:main):

No mute/highlight/hide CLI overrides yet (hardcoded tag_rules in main). GSoC proposal's "tag filters as CLI arguments" is still open work.

floss/qs/bulk.py: walks a directory, shells python -m floss.qs.main --json, optional rendered .txt.

4.6 Terminal rendering

Rich-based columns: tags | string | structure | offset, with box-drawing layout delimiters, muted/highlight styles, tag-group consolidation. Depends on colorama for Windows console.

4.7 Web viewer (qs-viewer/)

Not yet: browsing inside tag databases when clicking a tag (GSoC / issues #782, #784 -- still open product work).

4.8 Tests (quantumstrand-only)

tests-qs.yml runs pytest -k qs on Ubuntu with LFS + submodules.

Test data submodule: quantumstrand pins tests/data @ 6eba5b1 (includes PMA samples QS needs); master pins older 53e9101. Integration must bump the submodule on master carefully.

4.9 CI / release matrix

4.10 Ancillary / scripts

5. What does not need to be reinvented

These GSoC "new features" are largely complete on quantumstrand and should be ported as-is, then refined:

1. ELF section parsing -- done with robustness work (PR history includes #1312, #1319).
2. Git LFS for DBs -- done (#1309).
3. Automatic tag DB construction pipeline -- done (script + bi-weekly CI).
4. Mach-O static layout -- done beyond original "StaticString-only" sketch.
5. qs-viewer baseline -- done; DB peek / tag drill-down still missing.

Remaining product gaps from the proposal + community threads:

1. FLOSS integration (--quantum / always-on enrichments, merged JSON).
2. CLI tag filters (mute/hide/highlight overrides).
3. Viewer: open/browse tag databases (#782, #784).
4. Diverse section-parsing corpus (disk vs memory dump, binwalk padding, stripped malware).
5. Library manifest expansion -- pipeline supports 62 libs; only 17 DBs checked in until CI rebuilds land.
6. Deprecate QUANTUMSTRAND codename after stable merge.

6. Integration design (target state)

6.1 Design principles

1. Backward compatible JSON: legacy FLOSS roots analysis, metadata, strings remain valid without QS fields.
2. --quantum is a feature flag: default off for this migration phase; not flipping to default-on yet.
3. Do not break deobfuscation: stack/tight/decoded recovery stays FLOSS; when --quantum is on, text tags apply to static + stack/tight/decoded/language strings; section/structure mapping is primarily for static offsets.
4. No GUI migration for now: qs-viewer/ and Pages deploy stay out of scope.
5. QS runtime deps are core: python-lancelot, pyelftools, machofile, msgspec, etc. install with FLOSS always (not a long-lived optional extra).
6. Single binary: only floss; no parallel qs / quantumstrand artifact.

6.2 Target merged JSON (proposal-aligned, updated)

When QS mode is enabled:

{
  "analysis": { "enable_static_strings": true, "...": "..." },
  "metadata": {
    "file_path": "dotnet-hello.exe",
    "version": "3.x.y",
    "qs_enabled": true,
    "qs_version": "0.3.0"
  },
  "strings": {
    "static_strings": [
      {
        "string": "WriteLine",
        "offset": 1036,
        "encoding": "ASCII",
        "qs_tags": ["#common", "#msvc"],
        "qs_structure": "",
        "qs_section": ".text"
      }
    ],
    "stack_strings": [],
    "decoded_strings": []
  },
  "qs_layout": {
    "name": "pe",
    "offset": 0,
    "length": 3584,
    "children": [
      { "name": "header", "offset": 0, "length": 512, "children": [] },
      { "name": ".text", "offset": 512, "length": 1024, "children": [] }
    ]
  }
}

Semantics:

• qs_layout holds structural hierarchy only (no per-node string lists required for FLOSS JSON -- avoids doubling string storage).
• Per-string qs_* fields are optional; absent when QS disabled.
• Existing FLOSS tooling that ignores unknown fields keeps working.
• Add a checked-in JSON Schema + tests/test_results_schema_compat.py as proposed.

6.3 Mapping QS layout -> FLOSS static strings

FLOSS StaticString(offset, string, encoding)
        |
        |- match offset in layout node -> qs_section = nearest section/segment name
        |- match offset in Structure map -> qs_structure = structure name
        \- tag string text via load_databases() -> qs_tags

Edge cases:

6.4 Target directory structure (post-migration)

Aligned with the proposal, adjusted to current code:

flare-floss/
|-- floss/
|   |-- main.py                 # --quantum / tag filter CLI (feature flag, default off)
|   |-- results.py              # optional qs_* fields on string types + metadata
|   |-- qs/                     # layout, tags, dbs (library code; no separate shipped CLI)
|   |   |-- main.py             # may remain as module internals; do not ship `qs` console script
|   |   |-- layout.py           # NEW: split compute_*_layout out of main.py (optional refactor)
|   |   |-- tags.py             # NEW: load_databases + tag rules (optional refactor)
|   |   |-- render.py           # NEW: rich renderer (optional refactor)
|   |   |-- db/                 # unchanged data + loaders
|   |   \-- ...
|   |-- qs_integration.py       # NEW: FLOSS<->QS adapter (proposal Slice 1)
|   \-- render/
|       |-- default.py          # optional QS-aware columns when enabled
|       \-- json.py             # serializes optional qs fields
|-- scripts/build_oss_db.py
|-- tests/
|   |-- test_qs_*.py            # existing
|   |-- test_qs_integration_cli.py
|   |-- test_qs_layout_mapping.py
|   |-- test_qs_tagging.py
|   |-- test_qs_visibility.py
|   \-- test_results_schema_compat.py
|-- doc/
|   |-- quantumstrand-migration.md   # this file
|   \-- quantumstrand.md             # user-facing architecture + usage (to write)
\-- .github/workflows/
    |-- tests.yml               # include qs jobs
    |-- build.yml               # single floss binary (QS DBs embedded)
    |-- build-oss-db.yml        # base: master
    \-- verify-pins.yml

Not migrated (locked): qs-viewer/, web-release.yml, standalone qs entry point / build-qs.yml / qs.spec.

Refactor recommendation: floss/qs/main.py is a 2.5k-line monolith. Prefer splitting during migration without changing behavior, so FLOSS integration imports layout/tags cleanly. Not a blocker for Slice 1 if adapter imports from floss.qs.main temporarily.

7. Slice-based migration plan (updated)

The GSoC four-slice model remains the right process; contents are rebased onto current code.

Slice 0 -- Foundation (prerequisite, 0.5-1 week)

Goal: make master able to host QS artifacts safely.

1. Merge or rebase latest master into integration branch (deps, verify-pins, AGENTS.md, mypy/black bumps).
2. Land .gitattributes + Git LFS for:
   • floss/qs/db/data/**
   • optionally migrate floss/sigs/*.sig to LFS (coordinate: history rewrite not required if only new commits use LFS; existing master blobs remain until rewritten -- prefer forward-only LFS for new paths first, then sigs carefully).
3. Bump tests/data submodule to QS pin if PMA fixtures are required.
4. Document git lfs install && git lfs pull in doc/installation.md.
5. CI: ensure all workflows that need DBs set lfs: true.

Exit criteria: clean install on Linux CI with LFS; FLOSS tests still green; no accidental 15MB re-add of DB binaries as non-LFS.

Slice 1 -- Adapter baseline (1-2 weeks)

Goal: FLOSS can optionally compute layout and attach section/structure to static strings (PE first).

Deliverables:

• floss/qs_integration.py
  • analyze_layout(sample_bytes) -> ResultLayout-like tree
  • map_strings_to_layout(static_strings, layout) -> enriched strings
  • to_qs_document / from_floss_document helpers for tests
• CLI: --quantum (store_true, default False) on floss.main — permanent feature flag for this migration phase
• results.StaticString: optional qs_tags, qs_section, qs_structure (default empty / omit in JSON if unset -- decide one serialization policy and stick to it)
• metadata.qs_enabled: bool
• qs_layout optional top-level field on ResultDocument (or nested under metadata -- prefer top-level as in proposal)
• Tests: test_qs_integration_cli.py, test_qs_layout_mapping.py (PE fixtures from PMA)
• JSON schema draft + load old FLOSS JSON without error

Non-goals this slice: tagging DBs, viewer rename, OSS CI retarget.

Risk: lancelot panics -- already mitigated with broad BaseException catch in QS PE path; keep that in the adapter.

Slice 2 -- Tagging + CI + DBs (2-3 weeks)

Goal: full tag pipeline available under FLOSS.

• Load DBs via existing floss.qs.db.* loaders
• Apply taggers to static and stack / tight / decoded / language strings
• CLI visibility controls, e.g.:
  • --qs-hide-tag #code
  • --qs-mute-tag #common
  • --qs-highlight-tag #capa
  • or a compact --tag-rule '#code=hide,#capa=highlight'
• Default rules match current QS behavior when --quantum is on
• Copy/adapt workflows:
  • tests-qs -> master
  • build-oss-db.yml base -> master
  • PyInstaller data paths if FLOSS binary should embed DBs when built with QS
• Package data: ensure wheels/sdists include floss/qs/db/data (MANIFEST / package-data)
• Tests: test_qs_tagging.py, test_qs_visibility.py, DB load tests

Dependency decision (locked): always install QS deps + ship DBs with FLOSS.

Slice 3 -- Multi-format layout + corpus (2 weeks)

Already implemented on QS branch -- focus is wiring + testing under FLOSS, not greenfield parsers.

• Ensure adapter calls compute_layout (PE/ELF/Mach-O dispatch) not PE-only

• ELF/Mach-O mapping tests under FLOSS CLI

• Build corpus matrix:

  Source type	Why
  Clean PE from disk / VT	FileAlignment, normal sections
  Memory / crash dump PE	SectionAlignment, slack, heap garbage
  binwalk-carved / padded	imperfect offsets
  Stripped ELF	section vs segment fallback
  Fat Mach-O	multi-arch children
  XOR'd PE	#decoded path
  Go/Rust FLOSS language samples	interaction with language strings

• Document sample-source consideration in doc/quantumstrand.md

Slice 4 -- Polish & single-binary cleanup (ongoing)

• Out of scope: qs-viewer/, web-release.yml, GUI / DB-browser work — do not migrate
• Do not ship a parallel qs / quantumstrand binary — only floss with --quantum
• Drop or never land on master: build-qs.yml, .github/pyinstaller/qs.spec, project.scripts entry qs = ...
• Apply text taggers to static and stack / tight / decoded (and language) strings when --quantum
• User docs + changelog for --quantum
• Community feedback loop (proposal Figure 2)

Optional Slice 5 -- Performance & DB productization

• Audit jh_to_qs / build_oss_db for CI time (proposal concern)
• Land expanded OSS DBs via automated PRs (62-lib matrix)
• Consider multi-triplet matrix only if string coverage gaps are measured
• Publish release artifacts when DB growth is significant

8. File-level migration checklist

Must copy / land onto master

.gitattributes                     # LFS: qs db data + floss/sigs/*.sig
floss/qs/                          # package (layout, tags, dbs) — not a separate CLI long-term
scripts/build_oss_db.py
.github/workflows/tests-qs.yml     # or fold into tests.yml
.github/workflows/build-oss-db.yml
.github/mypy/mypy.ini              # merge QS ignore modules
pyproject.toml                     # QS deps as core runtime deps (not optional extra)
tests/test_qs*.py
tests/test_gp_db.py
tests/test_winapi_db.py
tests/data                         # submodule bump

Do not copy to master (locked out of scope):

qs-viewer/
.github/workflows/web-release.yml
.github/workflows/build-qs.yml      # no parallel quantumstrand binary
.github/pyinstaller/qs.spec
# pyproject [project.scripts] qs = ...  — do not add

Must implement new (not pure copy)

floss/qs_integration.py
floss/main.py                      # flags + call adapter
floss/results.py                   # optional fields
floss/render/default.py            # optional QS rendering
floss/render/json.py               # already dataclass-friendly; ensure qs_* serialize
doc/quantumstrand.md               # user docs
schemas/floss-results.json         # optional but proposed
tests/test_qs_integration_*.py
tests/test_results_schema_compat.py

Must retarget after merge

Master-only assets to preserve

.github/workflows/verify-pins.yml
AGENTS.md
newer dependabot pins in pyproject / requirements

Explicit non-goals for first merge

• Replacing FLOSS static string extractor with QS extractor wholesale
• Making stack/decoded strings participate in layout trees (tags yes; section/structure usually N/A)
• Migrating qs-viewer / any GUI / GitHub Pages web-release
• Shipping a parallel qs / quantumstrand binary
• Multi-OS vcpkg matrix for OSS DBs (Windows static remains justified)
• Rewriting full git history for LFS (forward-only LFS tracking of sigs + DBs is enough)

9. Compatibility & risk matrix

10. Testing strategy

Unit

• Layout pure functions (OffsetRanges, range merge, Range/Slice)
• DB loaders (oss, gp, winapi, expert)
• build_oss_db converters/merge without network

Integration

• floss --quantum sample.exe -j produces valid merged JSON
• floss sample.exe -j bit-identical structure to pre-QS (no qs fields)
• Load legacy JSON with floss -l still works
• Tag rules hide/mute/highlight

Golden / fixture

• PMA 01-01, 03-03 (existing QS tests)
• ELF + Mach-O fixtures in testfiles submodule
• One intentionally truncated / padded PE

CI gates before merge to master

pytest                          # full suite
pytest -k qs                    # QS suite
mypy (project config)
black / isort / pre-commit
verify-pins on workflow edits
smoke: pyinstaller floss (with QS DB datas embedded)

Manual acceptance

• Terminal UX for floss --quantum matches expected tags/sections on a known sample
• OSS workflow dry-run (workflow_dispatch) opens a PR against master

11. Suggested PR stack (Graphite / stacked PRs)

Keep PRs reviewable (~reviewer-hour sized):

1. chore: LFS + .gitattributes + install docs (no feature code)
2. feat: vendor floss/qs/ package + deps + unit tests (no FLOSS CLI wiring)
3. feat: qs_integration + --quantum PE section mapping
4. feat: tagging + tag-rule CLI
5. ci: fold QS tests + OSS DB workflow retarget
6. feat: multi-format layout coverage tests under FLOSS
7. docs: usage + schema for --quantum (no viewer PR)
8. chore: single floss release binary embeds QS DBs; no quantumstrand artifact

Avoid one mega-PR of 15k insertions even though the branch already has them; landing in stacks preserves bisectability.

12. Mapping GSoC proposal deliverables -> status

13. Decisions (locked)

13.1 Logging format (decision #8 locked)

Action on master: take the quantumstrand floss/logging_.py change as-is when integrating. Global for all FLOSS logs (not gated on --quantum).

14. Operational runbook (post-merge)

Developer setup

git clone --recurse-submodules https://github.com/mandiant/flare-floss.git
cd flare-floss
git lfs install
git lfs pull
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,qs]"   # or whatever extra name is chosen
pytest -k qs

Rebuilding OSS DBs locally (Windows)

# build jh from lancelot, then:
python scripts/build_oss_db.py `
  --config floss\qs\db\data\oss\libraries.json `
  --jh-path path\to\jh.exe `
  --output-dir floss\qs\db\data\oss `
  --continue-on-error

Viewer

Out of scope for this migration (do not set up or deploy qs-viewer).

15. Success metrics

Migration is "done" when:

1. A user on master can run floss --quantum sample.exe and get tagged, section-annotated static strings.
2. floss sample.exe -j without the flag remains backward compatible.
3. CI on master runs QS unit/integration tests with LFS.
4. Bi-weekly OSS DB workflow opens PRs against master.
5. Only one shipped binary (floss); --quantum is the sole entry to QS features.
6. Stack / tight / decoded / language strings receive text tags when --quantum is on.
7. No regression in FLOSS deobfuscation tests (test_main, language suites, etc.).
8. GUI / qs-viewer intentionally not part of success criteria.

16. Appendix A -- Tag reference

17. Appendix B -- PE structures annotated by QS

• section header
• import table
• export table
• rich header
• layout regions: header, section names, overlay, Authenticode digital signature, gap, rsrc: ...

ELF: elf header, program header, section header, string table, symbol table

Mach-O: macho header, load command, segment header, section header, code signature, certificates, plist: entitlements

18. Appendix C -- Quick command cheat sheet for explorers

# Diff surface area
git diff --stat master...quantumstrand

# Only non-data code
git diff --name-status master...quantumstrand -- \
  ':(exclude)floss/qs/db/data' ':(exclude)qs-viewer/package-lock.json'

# Run QS locally (quantumstrand worktree)
pip install -e ".[qs]"
qs /path/to/sample -j | head
python -m floss.qs.main /path/to/sample

# Run FLOSS on master worktree
cd ../flare-floss-master
pip install -e ".[dev]"
floss /path/to/sample -j

19. Appendix D -- Divergence notes for integrators

• floss/main.py and floss/results.py are currently identical between tips of the two branches. Integration edits land cleanly without resolving QS-vs-FLOSS conflicts in those files.
• Master is ahead on dependency hygiene (verify-pins, AGENTS.md, newer pins). Always merge master -> integration branch first.
• Quantumstrand FLIRT .sig files in a non-LFS-pulled tree look like 132-byte pointer files; master trees have full multi-MB binaries. Don't commit pointer stubs onto master by mistake.
• libraries.json (62 libs) is the future set; DEFAULT_FILENAMES in oss.py (17 + CRT) is the runtime set. After CI rebuilds, update DEFAULT_FILENAMES (or auto-discover *.jsonl.gz) so new DBs are actually loaded.

20. Document history

When this plan is executed, keep this file updated per slice (checkboxes, open decisions, and any schema changes). Prefer linking PRs next to each Slice checklist item.

21. First-class modular architecture (post master-merge)

Context (2026-07-10): master has been merged into quantumstrand (dep/CI updates). Direction is now a single land onto master, not a long parallel-branch life. Constraints from locked decisions:

• --quantum feature flag (default off)
• No separate qs binary / no GUI
• No long-lived floss/qs/ package — features are first-class under floss/
• Core deps always installed; DBs + FLIRT sigs on LFS
• JSON enrichments use qs_* field names on FLOSS objects
• Tag static and stack / tight / decoded / language strings
• Global log timestamps (QS logging_.py)

21.1 Problems with the current shape

Goal: one product, thin CLI, libraries with clear boundaries, same import style as floss.language.* / floss.render.*.

21.2 Design principles

1. main.py is only a conductor — parse args, call pipeline, print. No PE parsing, no tag DB I/O, no Rich layout drawing.
2. One concept → one module (or small subpackage) — layout, tags, enrich, render stay separable and unit-testable.
3. Match existing FLOSS style — subpackages for multi-file domains (language/, render/); flat modules for single concerns (stackstrings.py).
4. No qs package name — use domain names: layout, tags, enrich. Keep qs_* only on JSON/API fields and the --quantum flag (experiment branding without folder branding).
5. Internal models may stay rich (Pydantic/Layout tree); public FLOSS results stay dataclasses with optional qs_* fields.
6. Move-only refactors first, then wire — preserve behavior with import shims until tests are green, then delete shims in the same PR series.
7. Scripts are not libraries — VT/GP/bulk builders live under scripts/, not floss/.

21.3 Target package layout

floss/
├── main.py                 # thin CLI entry (target ~150–250 lines)
├── cli.py                  # ArgumentParser helpers, make_parser(), StringType enum
├── pipeline.py             # run_analysis(sample, args) -> ResultDocument  (today’s main() body)
├── results.py              # + optional qs_* on string types + metadata + qs_layout
├── strings.py              # existing static extractors
├── utils.py / identify.py / stackstrings.py / tightstrings.py / string_decoder.py
├── logging_.py             # timestamps (QS form)
├── version.py
│
├── ranges.py               # Range, Slice, OffsetRanges  (from qs/main foundation)
│
├── layout/                 # binary structure tree (from compute_*_layout)
│   ├── __init__.py         # public: compute_layout(buf) -> Layout
│   ├── base.py             # Layout ABC, SectionLayout, SegmentLayout, Structure, helpers
│   ├── pe.py               # compute_pe_layout, collect_pe_structures, reloc/code
│   ├── elf.py              # compute_elf_layout, ELF robustness helpers
│   ├── macho.py            # fat/thin Mach-O
│   └── extract.py          # extract_layout_strings, gap walking, collect_strings
│
├── tags/                   # string classification (from qs/db + taggers)
│   ├── __init__.py         # load_taggers(), apply_tags(), default TagRules
│   ├── rules.py            # TagRules type, hide/mute/highlight helpers, FP lib filter
│   ├── expert.py           # was qs/db/expert.py
│   ├── gp.py
│   ├── oss.py
│   ├── winapi.py
│   └── data/               # LFS: was qs/db/data/{crt,expert,gp,oss,winapi}
│       ├── crt/
│       ├── expert/
│       ├── gp/
│       ├── oss/
│       └── winapi/
│
├── enrich.py               # --quantum glue: map FLOSS strings ↔ layout + tags
│                           #   enrich_result_document(doc, sample_bytes, rules) -> doc
│
├── render/
│   ├── default.py          # existing FLOSS tables; optional qs columns when enabled
│   ├── json.py             # serializes optional qs_* (dataclass encoder already ok)
│   ├── sanitize.py
│   └── layout_text.py      # optional: Rich section-tree renderer (from qs render_*)
│                           #   only used when --quantum and not --json, if we want QS TUI
│
├── language/               # unchanged
├── features/               # unchanged
├── sigs/                   # LFS FLIRT
└── (no qs/)

scripts/
├── build_oss_db.py         # already here; paths → floss/tags/data/oss
├── bulk_quantum.py         # was floss/qs/bulk.py (optional)
├── tags/                   # was floss/qs/scripts/*
│   ├── extract_strings.py
│   ├── fetch_vt_hashes.py
│   ├── generate_gp_db.py
│   └── query_string.py
└── ...

tests/
├── test_layout_*.py        # was test_qs_elf/macho/code_ranges/offset_ranges/unit
├── test_tags_*.py          # was test_gp_db, test_winapi_db, test_qs_oss_db
├── test_enrich_*.py        # integration: --quantum CLI, mapping, visibility
└── test_results_schema_compat.py

What deliberately does not exist after the land:

• floss/qs/
• qs console script
• build-qs.yml / qs.spec / qs-viewer/ (GUI already out of scope)

21.4 How qs/main.py splits (responsibility map)

floss/qs/db/*.py → rename package to floss/tags/*.py and fix pathlib.Path(__file__).parent / "data".

21.5 How floss/main.py splits

pipeline.py sketch:

run(sample, opts) -> ResultDocument:
    doc = empty ResultDocument(analysis=...)
    static = get_static_strings(...)
    language identify / extract ...
    if deobfuscation enabled:
        vw = load_vw(...); stack/tight/decoded ...
    if opts.quantum:
        doc = enrich.apply(doc, sample_bytes, tag_rules=opts.tag_rules)
    doc.metadata.runtime = ...
    return doc

main.py then: parse → run → render.json|default → print.

21.6 Enrichment API (first-class --quantum)

floss/enrich.py is the only place that knows about both pipeline outputs and layout/tags:

apply(doc: ResultDocument, data: bytes, *, min_length: int, tag_rules: TagRules) -> ResultDocument:
    layout = floss.layout.compute_layout(data)
    # optional: still extract layout-local strings for structure marking /
    #           or only use layout tree for section maps
    section_index = build_offset_index(layout)      # offset -> section name
    structure_index = build_structure_index(layout) # offset -> structure name
    taggers = floss.tags.load_taggers()

    for each string list in doc.strings:
        # static, stack, tight, decoded, language_*:
        for s in list:
            s.qs_tags = tag_string(taggers, s.string)
            if hasattr offset and static-like:
                s.qs_section = section_index.lookup(s.offset)
                s.qs_structure = structure_index.lookup(s.offset)

    doc.metadata.qs_enabled = True
    doc.qs_layout = to_layout_dto(layout)  # structure-only tree, no nested strings
    apply_visibility_rules(doc, tag_rules)  # hide/mute for render path
    return doc

Section/structure: meaningful for static (and language static) offsets.
Tags: all string kinds (locked decision).

21.7 Results model extensions

# floss/results.py (illustrative)

@dataclass(frozen=True)
class StaticString:
    string: str
    offset: int
    encoding: StringEncoding
    qs_tags: tuple[str, ...] = ()
    qs_section: str = ""
    qs_structure: str = ""

# StackString / TightString / DecodedString: add qs_tags (and empty section/structure defaults)

@dataclass
class Metadata:
    ...
    qs_enabled: bool = False

@dataclass
class ResultDocument:
    metadata: Metadata
    analysis: Analysis = ...
    strings: Strings = ...
    qs_layout: Optional[dict] = None  # or a small LayoutDTO dataclass

JSON stays backward compatible when --quantum is off (empty defaults omitted if encoder skips empties — pick one policy and test it).

21.8 CLI surface (single binary)

floss [options] sample
  --quantum                 enable layout + tagging enrichments
  --qs-hide-tag TAG         (repeatable)   # optional slice 2
  --qs-mute-tag TAG
  --qs-highlight-tag TAG

No qs entry point in pyproject.toml.
PyInstaller floss.spec gains datas= for floss/tags/data/** (from current qs.spec).

21.9 Dependency / packaging

• Move current [project.optional-dependencies] qs packages into core dependencies (lancelot, pyelftools, machofile, msgspec, colorama, dnfile as needed).
• packages / package-data must include floss/tags/data.
• .gitattributes LFS paths update: floss/tags/data/** and floss/sigs/*.sig.
• build-oss-db.yml output dir → floss/tags/data/oss.

21.10 Implementation order inside one giant PR

Prefer one PR into master with a linear commit series (reviewable, bisectable). Suggested commits:

Rule: after each commit, pytest -k 'layout or tags or qs or enrich or main' (names evolve) stays green.

Anti-pattern: one giant “move everything + rewrite main” commit — unreviewable and hard to fix.

21.11 Temporary compatibility (optional, same PR only)

If useful during commits 2–7:

# floss/qs/main.py  (shim only, deleted before merge)
from floss.layout import compute_layout, ...
from floss.ranges import Range, Slice, OffsetRanges
# re-export for old tests

Do not merge the shim to master; final tree has zero floss/qs.

21.12 Testing map after rename

21.13 Size budget (sanity targets)

If pe.py stays >500 after move, extract layout/pe_structures.py and layout/pe_code.py (lancelot ranges).

21.14 Mental model

                    ┌─────────────┐
                    │  floss CLI  │  main.py + cli.py
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │  pipeline   │  static / language / deobfuscation
                    └──────┬──────┘
                           │  if --quantum
                    ┌──────▼──────┐
                    │   enrich    │
                    └──┬───────┬──┘
                       │       │
              ┌────────▼──┐  ┌─▼────────┐
              │  layout   │  │   tags   │
              │ pe/elf/.. │  │ data+DB  │
              └───────────┘  └──────────┘
                       │
                    ┌──▼──────────┐
                    │ results +   │
                    │ render/json │
                    └─────────────┘

21.15 Out of scope (unchanged)

• GUI / qs-viewer
• Making quantum default-on
• Parallel quantumstrand release binary
• Rewriting stack/decoded algorithms to be layout-aware (tags only)