# QUANTUMSTRAND FLOSS Master Migration Guide - **Status:** draft for GSoC / integration planning - **Branches compared:** `quantumstrand` @ `f367032` vs `master` @ `98e2259` - **Merge-base:** `9a7f351` - **Divergence:** ~313 commits on `quantumstrand` not in `master`; ~44 commits on `master` not in `quantumstrand` This document is a full inventory of what exists on `quantumstrand` today, how it differs from the GSoC 2026 proposal assumptions, and a concrete plan to fold QS features, workflows, CI, and the viewer interface into `master` without breaking FLOSS consumers. ## 1. Executive summary QUANTUMSTRAND (QS) is **already a mature parallel product** living under `floss/qs/`, with its own CLI (`qs`), JSON schema, PyInstaller build, GitHub Actions, tag databases (Git LFS), OSS rebuild pipeline, and a React single-file viewer (`qs-viewer/`). **Critical reality vs. the original GSoC proposal:** | Proposal assumption (older) | Current `quantumstrand` state | |----------------------------|-------------------------------| | Start with PE-only layout + thin adapter | PE + **ELF** + **Mach-O** (thin & fat) layout already implemented | | Build OSS pipeline from scratch | `scripts/build_oss_db.py` + bi-weekly `build-oss-db.yml` already land PRs | | Migrate DBs to Git LFS first | **Done** (`.gitattributes` + LFS for DBs and FLIRT `.sig` files) | | Slice 1 creates `qs_integration.py` + `--quantum` | **Not started** -- QS is still a separate binary/entry point | | Merge into FLOSS `ResultDocument` | **Not started** -- QS has its own Pydantic `ResultDocument` (`meta` + recursive `layout`) | | `floss-viewer/` port | Exists as `qs-viewer/` consuming **QS-native** JSON only | **Bottom line:** most "feature build" work on the QS experiment branch is done. The remaining hard problem is **integration architecture**: how to expose QS capabilities through FLOSS's CLI/JSON/renderers while preserving backward compatibility, then retire the parallel codename. Recommended posture: **treat `quantumstrand` as the feature source of truth**, rebase/merge recent `master` into it (or land incremental PRs onto `master`), and use a **slice-based integration** that starts with optional enrichments rather than rewriting FLOSS's static-string path overnight. ## 2. Worktree setup (local) Already created: ```bash # from repo root (quantumstrand checkout) git worktree add ../flare-floss-master master # quantumstrand is the primary worktree; a second worktree for the same branch # is not needed and git will refuse it if the branch is already checked out. git worktree list # /Users/verma/Projects/Mandiant/flare-floss ... [quantumstrand] # /Users/verma/Projects/Mandiant/flare-floss-master ... [master] ``` Use the master worktree for migration implementation branches: ```bash cd ../flare-floss-master git checkout -b feat/qs-integration-slice-1 ``` **Note:** LFS objects on the quantumstrand worktree may appear as pointer stubs until `git lfs pull` is run. Master currently has full FLIRT `.sig` binaries **without** LFS; quantumstrand tracks them via LFS. ## 3. Architecture comparison ### 3.1 Master FLOSS (today) ``` flare-floss/ |-- floss/ | |-- main.py # sole CLI entry: floss | |-- results.py # dataclasses ResultDocument | |-- strings.py # static ASCII/UTF-16 extraction | |-- stackstrings.py / tightstrings.py / string_decoder.py | |-- language/ # Go / Rust language-specific strings | |-- render/{default,json,sanitize}.py | \-- sigs/ # FLIRT signatures (plain git blobs) |-- .github/workflows/ | |-- build.yml, tests.yml, publish.yml, black-format.yml | \-- verify-pins.yml # pinact -- master-only hygiene |-- scripts/ # IDA/Ghidra/Binja/r2/x64dbg import helpers \-- doc/ ``` **FLOSS `ResultDocument` shape (simplified):** ```json { "metadata": { "file_path": "...", "version": "...", "imagebase": 0, "min_length": 4, "runtime": {}, "language": "" }, "analysis": { "enable_static_strings": true, "enable_stack_strings": true, ... }, "strings": { "static_strings": [{ "string": "...", "offset": 0, "encoding": "ASCII" }], "stack_strings": [...], "tight_strings": [...], "decoded_strings": [...], "language_strings": [...], "language_strings_missed": [...] } } ``` `StaticString` has only: `string`, `offset`, `encoding`. No tags, no section, no structure. **Analysis pipeline:** vivisect-backed deobfuscation + language ID + static strings from whole-file extraction. No recursive PE/ELF/Mach-O layout tree; no string tagging DBs. ### 3.2 Quantumstrand (today) ``` flare-floss/ (on quantumstrand) |-- floss/ | |-- main.py # UNCHANGED vs master (still pure FLOSS) | |-- results.py # UNCHANGED | \-- qs/ # entire experiment lives here | |-- main.py # ~2585 lines: layout + tag + render + CLI | |-- bulk.py # directory batch runner | |-- db/ | | |-- {gp,oss,expert,winapi}.py | | \-- data/ # Git LFS: .jsonl.gz, .bin, capa.jsonl | \-- scripts/ # VT fetch, GP generation helpers |-- qs-viewer/ # Vite + React single-file GUI |-- scripts/build_oss_db.py # vcpkg + jh automation |-- .github/ | |-- pyinstaller/qs.spec | \-- workflows/ | |-- build-qs.yml | |-- tests-qs.yml | |-- build-oss-db.yml | \-- web-release.yml # GitHub Pages for qs-viewer |-- .gitattributes # LFS rules \-- pyproject.toml # extra [qs] deps + `qs` console script ``` **QS `ResultDocument` shape (Pydantic):** ```json { "meta": { "version": "0.3.0", "timestamp": "...", "sample": { "md5": "...", "sha1": "...", "sha256": "...", "path": "..." }, "min_str_len": 4 }, "layout": { "name": "pe", "offset": 0, "length": 3584, "strings": [ { "string": "WriteLine", "offset": 1036, "size": 18, "encoding": "ascii", "tags": ["#common", "#msvc"], "structure": "" } ], "children": [ { "name": "header", "offset": 0, "length": 512, "strings": [], "children": [] }, { "name": ".text", "offset": 512, "length": 1024, "strings": [...], "children": [] } ] } } ``` **Important:** QS does **not** implement stack/tight/decoded string recovery. It is a **contextual static-strings** tool. FLOSS remains the deobfuscation engine. ### 3.3 Entry points and packaging | Concern | Master | Quantumstrand | |---------|--------|---------------| | Console scripts | `floss = floss.main:main` | + `qs = floss.qs.main:main` | | Optional deps | `dev`, `build` | + `qs` extra: `dnfile`, `colorama`, `machofile`, `msgspec`, `python-lancelot`, `pyelftools` | | PyInstaller | `floss.spec` | + `qs.spec` (bundles all DB data paths) | | Binary name | `floss` | `quantumstrand` / `quantumstrand.exe` | | Version | `floss.version` -> `3.1.1` | QS reports `QS_VERSION = "0.3.0"` independently | `viv-utils` pin is slightly higher on QS (`>=0.8.0` vs `>=0.7.9`); otherwise core FLOSS deps are aligned, though master has newer dev/tooling pins (black, mypy, pre-commit, etc.). ## 4. Feature inventory (what must move) ### 4.1 Binary layout analysis (`floss/qs/main.py`) Recursive `Layout` tree with non-overlapping ordered children and gap strings. | Format | Function | Capabilities | |--------|----------|--------------| | **PE** | `compute_pe_layout` | Sections, header/overlay segments, Authenticode, resources (readable names via `PE_RESOURCE_TYPES`), gap nodes; **structures**: section headers, import/export tables, rich header; **#code** via lancelot BinExport2 ranges; **#reloc** via reloc table; single-byte **XOR MZ** decoding | | **ELF** | `compute_elf_layout` | Robust section/segment fallback, SHT_NOBITS skip, exec ranges, narrow metadata regions (ELF/program/section headers, string/symbol tables); ELFError / out-of-range hardening | | **Mach-O** | `compute_macho_layout` | Thin + fat; segments/sections; code signature, certs, entitlements plist | | **Fallback** | `SegmentLayout(name="binary")` | Whole-file string extraction | **GSoC proposal note on section parsing:** memory-dumped PEs (SectionAlignment + slack) vs disk PEs (FileAlignment) remain a testing concern. PE path already truncates out-of-range sections and logs warnings, but corpus coverage for AD1/static vs memory dumps should still be expanded during integration (Slice 3 testing deliverable in the proposal is still valid). ### 4.2 String tagging system Taggers loaded by `load_databases()`: | Source | Module / data | Tags produced | Default render rule | |--------|---------------|---------------|---------------------| | WinAPI | `db/winapi` + `apis.txt.gz`, `dlls.txt.gz` | `#winapi` | mute (default) | | Expert | `db/expert` + `capa.jsonl` | e.g. `#capa` | **highlight** | | OSS libs | `db/oss` + 17x `*.jsonl.gz` + `crt/msvc_v143.jsonl.gz` | `#zlib`, `#openssl`, `#msvc`, ... | mute | | Global prevalence | `db/gp` + `gp.jsonl.gz`, cwindb, hash DBs | `#common` | **mute** | | Junk code strings | `gp/junk-code.jsonl.gz` | `#code-junk` | mute | | Layout-derived | PE/ELF code & reloc ranges | `#code`, `#reloc` | **hide** | | XOR decode | PE path | `#decoded` | default | | Duplicates | same string text twice | `#duplicate` | **mute** | Post-processing: - `remove_false_positive_lib_strings` -- drops library tags if too few hits (default threshold 5). - `hide_strings_by_rules` -- removes `#code` / `#reloc` (etc.) from visible render. - Tag display consolidates adjacent identical tag groups; suppresses `#common` when more specific tags exist. ### 4.3 Tag databases (Git LFS) `.gitattributes`: ``` floss/qs/db/data/**/*.bin filter=lfs ... floss/qs/db/data/**/*.gz filter=lfs ... floss/qs/db/data/**/*.jsonl filter=lfs ... floss/sigs/*.sig filter=lfs ... ``` **30 LFS-tracked objects** on quantumstrand. Master has **no** `.gitattributes` and stores FLIRT sigs as normal blobs (~15 MB total). **Checked-in OSS DBs (17 + CRT):** brotli, bzip2, cryptopp, curl, detours, jemalloc, jsoncpp, kcp, liblzma, libpcap, libsodium, mbedtls, openssl, sqlite3, tomcrypt, wolfssl, zlib, + `crt/msvc_v143`. **Build manifest** (`libraries.json`) already lists **62** vcpkg packages for the automated pipeline (expansion beyond the original "top 25" goal). Checked-in DBs and the build matrix are intentionally not 1:1 yet; rebuild CI merges into existing `.jsonl.gz` files. ### 4.4 OSS database construction pipeline ``` libraries.json --> vcpkg (x64-windows-static, msvc143, release) | v jh (lancelot-bin) extract strings from .lib | v convert -> JSONL -> gzip | v merge with existing DB (new wins on collision) | v PR to quantumstrand (build-oss-db.yml) ``` Key implementation: `scripts/build_oss_db.py` (~878 lines) - Modular `Vcpkg`, `JHExtractor`, `Converter` classes - Cross-library strings are **kept** (not globally deduped); per-library still collapses duplicates by default - Metrics written to `build_metrics.json` - Windows-focused by design (PE-first rationale documented in `floss/qs/db/data/oss/readme.md`) **CI:** `.github/workflows/build-oss-db.yml` - Schedule: bi-weekly (`0 0 1,15 * *`) + `workflow_dispatch` - Runs on `windows-latest`, builds jh from `williballenthin/lancelot` - Opens PR with base **`quantumstrand`** (must change to `master` after merge) - Requires `contents: write` + `pull-requests: write` ### 4.5 CLI & bulk analysis **`qs` CLI** (`floss/qs/main.py:main`): | Flag | Behavior | |------|----------| | `path` | sample path | | `-n / --minimum-length` | min string length (default 4) | | `-j / --json` | emit QS ResultDocument JSON | | `-l / --load` | re-render from saved QS JSON | | `-d / -q` | debug / quiet (reuses `floss.main.set_log_config`) | No mute/highlight/hide CLI overrides yet (hardcoded `tag_rules` in `main`). GSoC proposal's "tag filters as CLI arguments" is still open work. **`floss/qs/bulk.py`:** walks a directory, shells `python -m floss.qs.main --json`, optional rendered `.txt`. ### 4.6 Terminal rendering Rich-based columns: tags | string | structure | offset, with box-drawing layout delimiters, muted/highlight styles, tag-group consolidation. Depends on `colorama` for Windows console. ### 4.7 Web viewer (`qs-viewer/`) | Item | Detail | |------|--------| | Stack | Vite + React + TypeScript, single-file build | | Input | QS JSON only (`meta` + recursive `layout`) | | Features | drag-drop upload, search, min length, tag multi-select, structure multi-select, show/hide untagged / no-structure, column toggles, copy filtered strings | | Noisy tags default | `#common`, `#duplicate`, `#code`, `#reloc`, `#code-junk` treated as muted styling; `#capa` highlighted | | Deploy | `web-release.yml` -> GitHub Pages on `qs-viewer/**` changes to `quantumstrand` | | Sample data | `src/pma0303_qs.json` baked in for preview | **Not yet:** browsing inside tag databases when clicking a tag (GSoC / issues #782, #784 -- still open product work). ### 4.8 Tests (quantumstrand-only) | File | Focus | |------|--------| | `tests/test_qs.py` | end-to-end PMA sample, JSON round-trip | | `tests/test_qs_unit.py` | `Range` / `Slice` | | `tests/test_qs_offset_ranges.py` | reloc/code range structure | | `tests/test_qs_code_ranges.py` | merge overlapping ranges | | `tests/test_qs_elf.py` | ELF layout sanity | | `tests/test_qs_macho.py` | Mach-O thin layout + structures | | `tests/test_qs_pma0101.py` | PMA fixture expectations | | `tests/test_qs_oss_db.py` | OSS DB load/query | | `tests/test_qs_build_oss_db.py` | build script unit + orchestration (~625 lines) | | `tests/test_gp_db.py` | GP + hash DBs | | `tests/test_winapi_db.py` | winapi DB | `tests-qs.yml` runs `pytest -k qs` on Ubuntu with LFS + submodules. **Test data submodule:** quantumstrand pins `tests/data` @ `6eba5b1` (includes PMA samples QS needs); master pins older `53e9101`. Integration must bump the submodule on master carefully. ### 4.9 CI / release matrix | Workflow | Master | Quantumstrand | Migration action | |----------|--------|---------------|------------------| | `tests.yml` | Yes | Yes | keep; extend matrix or jobs for QS | | `build.yml` | Yes FLOSS binary | Yes (minor branch filter) | keep FLOSS build; optionally embed QS later | | `publish.yml` | Yes | Yes | unchanged | | `black-format.yml` | Yes | Yes | unchanged | | `verify-pins.yml` | Yes | No | **bring to integrated branch** (master hygiene) | | `build-qs.yml` | No | Yes QS PyInstaller | either merge into `build.yml` or keep as floss-qs artifact job on `master` | | `tests-qs.yml` | No | Yes | fold into `tests.yml` or keep named job on `master` | | `build-oss-db.yml` | No | Yes | retarget base branch -> `master` | | `web-release.yml` | No | Yes Pages | retarget to `master`; path may become `floss-viewer/` | | `mypy.ini` extras | -- | intervaltree, lancelot, capa, virustotal3, colorama | merge | ### 4.10 Ancillary / scripts | Path | Role | Migrate? | |------|------|----------| | `floss/qs/scripts/extract_strings.py` | string extract helper | yes (or fold into scripts/) | | `floss/qs/scripts/fetch_vt_hashes.py` | VT corpus for GP / library selection | yes (dev tooling) | | `floss/qs/scripts/generate_gp_db.py` | GP DB generation | yes | | `floss/qs/scripts/query_string.py` | DB query debug CLI | yes | | `floss/qs/db/data/expert/import_from_capa.py` | capa -> expert rules | yes | | `floss/qs/db/data/oss/jh_to_qs.py` | CSV->JSONL (legacy; build script supersedes bulk use) | keep for manual rebuilds | | `floss/logging_.py` | timestamps in log format on QS only | **decide:** adopt on master or keep FLOSS format | ## 5. What does *not* need to be reinvented These GSoC "new features" are largely complete on `quantumstrand` and should be **ported as-is**, then refined: 1. **ELF section parsing** -- done with robustness work (PR history includes #1312, #1319). 2. **Git LFS for DBs** -- done (#1309). 3. **Automatic tag DB construction pipeline** -- done (script + bi-weekly CI). 4. **Mach-O static layout** -- done beyond original "StaticString-only" sketch. 5. **qs-viewer baseline** -- done; DB peek / tag drill-down still missing. Remaining product gaps from the proposal + community threads: 1. **FLOSS integration** (`--quantum` / always-on enrichments, merged JSON). 2. **CLI tag filters** (mute/hide/highlight overrides). 3. **Viewer: open/browse tag databases** (#782, #784). 4. **Diverse section-parsing corpus** (disk vs memory dump, binwalk padding, stripped malware). 5. **Library manifest expansion** -- pipeline supports 62 libs; only 17 DBs checked in until CI rebuilds land. 6. **Deprecate QUANTUMSTRAND codename** after stable merge. ## 6. Integration design (target state) ### 6.1 Design principles 1. **Backward compatible JSON:** legacy FLOSS roots `analysis`, `metadata`, `strings` remain valid without QS fields. 2. **`--quantum` is a feature flag:** default off for this migration phase; not flipping to default-on yet. 3. **Do not break deobfuscation:** stack/tight/decoded recovery stays FLOSS; when `--quantum` is on, **text tags** apply to static + stack/tight/decoded/language strings; section/structure mapping is primarily for static offsets. 4. **No GUI migration for now:** `qs-viewer/` and Pages deploy stay out of scope. 5. **QS runtime deps are core:** `python-lancelot`, `pyelftools`, `machofile`, `msgspec`, etc. install with FLOSS always (not a long-lived optional extra). 6. **Single binary:** only `floss`; no parallel `qs` / `quantumstrand` artifact. ### 6.2 Target merged JSON (proposal-aligned, updated) When QS mode is enabled: ```json { "analysis": { "enable_static_strings": true, "...": "..." }, "metadata": { "file_path": "dotnet-hello.exe", "version": "3.x.y", "qs_enabled": true, "qs_version": "0.3.0" }, "strings": { "static_strings": [ { "string": "WriteLine", "offset": 1036, "encoding": "ASCII", "qs_tags": ["#common", "#msvc"], "qs_structure": "", "qs_section": ".text" } ], "stack_strings": [], "decoded_strings": [] }, "qs_layout": { "name": "pe", "offset": 0, "length": 3584, "children": [ { "name": "header", "offset": 0, "length": 512, "children": [] }, { "name": ".text", "offset": 512, "length": 1024, "children": [] } ] } } ``` **Semantics:** - `qs_layout` holds structural hierarchy only (no per-node string lists required for FLOSS JSON -- avoids doubling string storage). - Per-string `qs_*` fields are optional; absent when QS disabled. - Existing FLOSS tooling that ignores unknown fields keeps working. - Add a checked-in **JSON Schema** + `tests/test_results_schema_compat.py` as proposed. ### 6.3 Mapping QS layout -> FLOSS static strings ``` FLOSS StaticString(offset, string, encoding) | |- match offset in layout node -> qs_section = nearest section/segment name |- match offset in Structure map -> qs_structure = structure name \- tag string text via load_databases() -> qs_tags ``` Edge cases: | Case | Strategy | |------|----------| | String only in FLOSS (language/Go extractor path) | tag by text; section may be unknown | | String only in QS layout gaps | optional: union into static_strings or leave QS-only in layout | | Encoding enum mismatch (`ASCII` vs `ascii`) | normalize in adapter | | Multiple sections claim offset | prefer deepest layout leaf | | Memory dump / invalid PE | layout may fall back to `binary`; tags still apply | ### 6.4 Target directory structure (post-migration) Aligned with the proposal, adjusted to current code: ``` flare-floss/ |-- floss/ | |-- main.py # --quantum / tag filter CLI (feature flag, default off) | |-- results.py # optional qs_* fields on string types + metadata | |-- qs/ # layout, tags, dbs (library code; no separate shipped CLI) | | |-- main.py # may remain as module internals; do not ship `qs` console script | | |-- layout.py # NEW: split compute_*_layout out of main.py (optional refactor) | | |-- tags.py # NEW: load_databases + tag rules (optional refactor) | | |-- render.py # NEW: rich renderer (optional refactor) | | |-- db/ # unchanged data + loaders | | \-- ... | |-- qs_integration.py # NEW: FLOSS<->QS adapter (proposal Slice 1) | \-- render/ | |-- default.py # optional QS-aware columns when enabled | \-- json.py # serializes optional qs fields |-- scripts/build_oss_db.py |-- tests/ | |-- test_qs_*.py # existing | |-- test_qs_integration_cli.py | |-- test_qs_layout_mapping.py | |-- test_qs_tagging.py | |-- test_qs_visibility.py | \-- test_results_schema_compat.py |-- doc/ | |-- quantumstrand-migration.md # this file | \-- quantumstrand.md # user-facing architecture + usage (to write) \-- .github/workflows/ |-- tests.yml # include qs jobs |-- build.yml # single floss binary (QS DBs embedded) |-- build-oss-db.yml # base: master \-- verify-pins.yml ``` **Not migrated (locked):** `qs-viewer/`, `web-release.yml`, standalone `qs` entry point / `build-qs.yml` / `qs.spec`. **Refactor recommendation:** `floss/qs/main.py` is a 2.5k-line monolith. Prefer splitting during migration *without* changing behavior, so FLOSS integration imports `layout`/`tags` cleanly. Not a blocker for Slice 1 if adapter imports from `floss.qs.main` temporarily. ## 7. Slice-based migration plan (updated) The GSoC four-slice model remains the right process; contents are rebased onto **current** code. ### Slice 0 -- Foundation (prerequisite, 0.5-1 week) **Goal:** make `master` able to host QS artifacts safely. 1. Merge or rebase latest `master` into integration branch (deps, `verify-pins`, `AGENTS.md`, mypy/black bumps). 2. Land `.gitattributes` + Git LFS for: - `floss/qs/db/data/**` - optionally migrate `floss/sigs/*.sig` to LFS (coordinate: history rewrite **not** required if only new commits use LFS; existing master blobs remain until rewritten -- prefer *forward-only* LFS for new paths first, then sigs carefully). 3. Bump `tests/data` submodule to QS pin if PMA fixtures are required. 4. Document `git lfs install && git lfs pull` in `doc/installation.md`. 5. CI: ensure all workflows that need DBs set `lfs: true`. **Exit criteria:** clean install on Linux CI with LFS; FLOSS tests still green; no accidental 15MB re-add of DB binaries as non-LFS. ### Slice 1 -- Adapter baseline (1-2 weeks) **Goal:** FLOSS can optionally compute layout and attach section/structure to static strings (PE first). Deliverables: - [ ] `floss/qs_integration.py` - `analyze_layout(sample_bytes) -> ResultLayout-like tree` - `map_strings_to_layout(static_strings, layout) -> enriched strings` - `to_qs_document` / `from_floss_document` helpers for tests - [ ] CLI: `--quantum` (store_true, default False) on `floss.main` — permanent feature flag for this migration phase - [ ] `results.StaticString`: optional `qs_tags`, `qs_section`, `qs_structure` (default empty / omit in JSON if unset -- decide one serialization policy and stick to it) - [ ] `metadata.qs_enabled: bool` - [ ] `qs_layout` optional top-level field on `ResultDocument` (or nested under metadata -- prefer top-level as in proposal) - [ ] Tests: `test_qs_integration_cli.py`, `test_qs_layout_mapping.py` (PE fixtures from PMA) - [ ] JSON schema draft + load old FLOSS JSON without error **Non-goals this slice:** tagging DBs, viewer rename, OSS CI retarget. **Risk:** lancelot panics -- already mitigated with broad `BaseException` catch in QS PE path; keep that in the adapter. ### Slice 2 -- Tagging + CI + DBs (2-3 weeks) **Goal:** full tag pipeline available under FLOSS. - [ ] Load DBs via existing `floss.qs.db.*` loaders - [ ] Apply taggers to static **and** stack / tight / decoded / language strings - [ ] CLI visibility controls, e.g.: - `--qs-hide-tag #code` - `--qs-mute-tag #common` - `--qs-highlight-tag #capa` - or a compact `--tag-rule '#code=hide,#capa=highlight'` - [ ] Default rules match current QS behavior when `--quantum` is on - [ ] Copy/adapt workflows: - `tests-qs` -> master - `build-oss-db.yml` base -> `master` - PyInstaller data paths if FLOSS binary should embed DBs when built with QS - [ ] Package data: ensure wheels/sdists include `floss/qs/db/data` (MANIFEST / package-data) - [ ] Tests: `test_qs_tagging.py`, `test_qs_visibility.py`, DB load tests **Dependency decision (locked):** always install QS deps + ship DBs with FLOSS. ### Slice 3 -- Multi-format layout + corpus (2 weeks) Already implemented on QS branch -- focus is **wiring + testing under FLOSS**, not greenfield parsers. - [ ] Ensure adapter calls `compute_layout` (PE/ELF/Mach-O dispatch) not PE-only - [ ] ELF/Mach-O mapping tests under FLOSS CLI - [ ] Build **corpus matrix**: | Source type | Why | |-------------|-----| | Clean PE from disk / VT | FileAlignment, normal sections | | Memory / crash dump PE | SectionAlignment, slack, heap garbage | | binwalk-carved / padded | imperfect offsets | | Stripped ELF | section vs segment fallback | | Fat Mach-O | multi-arch children | | XOR'd PE | `#decoded` path | | Go/Rust FLOSS language samples | interaction with language strings | - [ ] Document sample-source consideration in `doc/quantumstrand.md` ### Slice 4 -- Polish & single-binary cleanup (ongoing) - [ ] **Out of scope:** `qs-viewer/`, `web-release.yml`, GUI / DB-browser work — do not migrate - [ ] Do **not** ship a parallel `qs` / `quantumstrand` binary — only `floss` with `--quantum` - [ ] Drop or never land on master: `build-qs.yml`, `.github/pyinstaller/qs.spec`, `project.scripts` entry `qs = ...` - [ ] Apply text taggers to static **and** stack / tight / decoded (and language) strings when `--quantum` - [ ] User docs + changelog for `--quantum` - [ ] Community feedback loop (proposal Figure 2) ### Optional Slice 5 -- Performance & DB productization - [ ] Audit `jh_to_qs` / `build_oss_db` for CI time (proposal concern) - [ ] Land expanded OSS DBs via automated PRs (62-lib matrix) - [ ] Consider multi-triplet matrix only if string coverage gaps are measured - [ ] Publish release artifacts when DB growth is significant ## 8. File-level migration checklist ### Must copy / land onto master ``` .gitattributes # LFS: qs db data + floss/sigs/*.sig floss/qs/ # package (layout, tags, dbs) — not a separate CLI long-term scripts/build_oss_db.py .github/workflows/tests-qs.yml # or fold into tests.yml .github/workflows/build-oss-db.yml .github/mypy/mypy.ini # merge QS ignore modules pyproject.toml # QS deps as core runtime deps (not optional extra) tests/test_qs*.py tests/test_gp_db.py tests/test_winapi_db.py tests/data # submodule bump ``` **Do not copy to master (locked out of scope):** ``` qs-viewer/ .github/workflows/web-release.yml .github/workflows/build-qs.yml # no parallel quantumstrand binary .github/pyinstaller/qs.spec # pyproject [project.scripts] qs = ... — do not add ``` ### Must implement new (not pure copy) ``` floss/qs_integration.py floss/main.py # flags + call adapter floss/results.py # optional fields floss/render/default.py # optional QS rendering floss/render/json.py # already dataclass-friendly; ensure qs_* serialize doc/quantumstrand.md # user docs schemas/floss-results.json # optional but proposed tests/test_qs_integration_*.py tests/test_results_schema_compat.py ``` ### Must retarget after merge | Item | From | To | |------|------|----| | `build-oss-db.yml` PR base | `quantumstrand` | `master` | | `build-qs.yml` | `quantumstrand` | **drop**, no parallel binary | | `tests-qs.yml` triggers | `quantumstrand` | or fold into `tests.yml` | | `web-release.yml` | `quantumstrand` | **do not migrate now** | | Release assets | `quantumstrand-*` | **only** `floss-*`; embed QS DBs in floss PyInstaller datas when `--quantum` support lands | | README / qs readme | experiment language | integrated `--quantum` feature language | ### Master-only assets to preserve ``` .github/workflows/verify-pins.yml AGENTS.md newer dependabot pins in pyproject / requirements ``` ### Explicit non-goals for first merge - Replacing FLOSS static string extractor with QS extractor wholesale - Making stack/decoded strings participate in layout trees (tags yes; section/structure usually N/A) - Migrating `qs-viewer` / any GUI / GitHub Pages web-release - Shipping a parallel `qs` / `quantumstrand` binary - Multi-OS vcpkg matrix for OSS DBs (Windows static remains justified) - Rewriting full git history for LFS (forward-only LFS tracking of sigs + DBs is enough) ## 9. Compatibility & risk matrix | Risk | Impact | Mitigation | |------|--------|------------| | JSON field additions break strict parsers | Medium | optional fields; schema version; golden tests for legacy JSON | | LFS not pulled -> empty DBs / broken tags | High | CI `lfs: true`; clear install docs; runtime warn if DB missing | | lancelot / native deps fail on some platforms | Medium | catch panics; degrade to section-only PE via pefile | | PyInstaller size growth from DBs | Medium | measure; document; optional slim builds | | Dual CLI confusion (`floss` vs `qs`) | N/A | locked: no `qs` binary on master | | Test submodule mismatch | Medium | single pin; QS tests skip if fixture absent | | Tag false positives from stale OSS DBs | Medium | keep `build-oss-db` cron; FP lib threshold | | Section mapping wrong on memory dumps | Medium | corpus tests; document limitations | | Logging format change (`logging_.py` timestamps) | Low | **intentional** — adopt QS timestamps globally; note in changelog | | `requirements.txt` drift master vs QS | Low | regenerate from one lock strategy after merge | ## 10. Testing strategy ### Unit - Layout pure functions (`OffsetRanges`, range merge, Range/Slice) - DB loaders (oss, gp, winapi, expert) - `build_oss_db` converters/merge without network ### Integration - `floss --quantum sample.exe -j` produces valid merged JSON - `floss sample.exe -j` bit-identical structure to pre-QS (no qs fields) - Load legacy JSON with `floss -l` still works - Tag rules hide/mute/highlight ### Golden / fixture - PMA 01-01, 03-03 (existing QS tests) - ELF + Mach-O fixtures in testfiles submodule - One intentionally truncated / padded PE ### CI gates before merge to master ```text pytest # full suite pytest -k qs # QS suite mypy (project config) black / isort / pre-commit verify-pins on workflow edits smoke: pyinstaller floss (with QS DB datas embedded) ``` ### Manual acceptance - Terminal UX for `floss --quantum` matches expected tags/sections on a known sample - OSS workflow dry-run (`workflow_dispatch`) opens a PR against master ## 11. Suggested PR stack (Graphite / stacked PRs) Keep PRs reviewable (~reviewer-hour sized): 1. **chore:** LFS + `.gitattributes` + install docs (no feature code) 2. **feat:** vendor `floss/qs/` package + deps + unit tests (no FLOSS CLI wiring) 3. **feat:** `qs_integration` + `--quantum` PE section mapping 4. **feat:** tagging + tag-rule CLI 5. **ci:** fold QS tests + OSS DB workflow retarget 6. **feat:** multi-format layout coverage tests under FLOSS 7. **docs:** usage + schema for `--quantum` (no viewer PR) 8. **chore:** single floss release binary embeds QS DBs; no `quantumstrand` artifact Avoid one mega-PR of 15k insertions even though the branch already has them; landing in stacks preserves bisectability. ## 12. Mapping GSoC proposal deliverables -> status | Deliverable | Status on quantumstrand | Migration work left | |-------------|-------------------------|---------------------| | Merge-ready QS codebase | **Yes** (experiment branch) | Integrate into master FLOSS UX | | Section mapping PE | **Done** | Adapter + FLOSS fields | | ELF section parsing | **Done** | Wire through adapter + tests on master | | Mach-O static | **Done** | Wire + tests | | Tag databases + loaders | **Done** | Package data on master builds | | OSS auto pipeline | **Done** | Retarget to master; expand checked-in DBs | | Git LFS | **Done** | Introduce on master | | Tag filters CLI | **Partial** (hardcoded rules only) | Implement args | | Merged FLOSS+QS JSON | **Not started** | Core integration | | JSON Schema + compat tests | **Not started** | Add | | GUI viewer | Done on QS branch | **Out of scope** — do not migrate | | Visualize / browse DBs | Not started | **Out of scope** for this migration | | `qs_integration.py` | **Not started** | Slice 1 | | `--quantum` flag | **Not started** | Slice 1 (opt-in feature flag) | | Single floss binary only | N/A | **Locked** — no parallel `qs` artifact | | Tag stack/decoded/tight | Static-only in QS CLI | **Yes** under FLOSS `--quantum` | ## 13. Decisions (locked) | # | Question | Decision | |---|----------|----------| | 1 | `--quantum` default | **Feature flag only for now** — opt-in via `--quantum`; not the default for static strings. Revisit later if desired. | | 2 | QS deps (lancelot / pyelftools / machofile / …) | **Always install** — promote into core runtime dependencies (not a `[qs]` extra long-term). | | 3 | Standalone `qs` / `quantumstrand` binary | **No** — single `floss` binary; QS only via `--quantum`. Drop `build-qs.yml` / `qs.spec` / `qs` console script (do not land on master). | | 4 | Viewer / GUI | **Do not migrate.** Ignore `qs-viewer/`, `web-release.yml`, and all GUI work for this migration. | | 5 | FLIRT `.sig` + QS DBs on LFS | **All LFS** — both `floss/sigs/*.sig` and `floss/qs/db/data/**` tracked with Git LFS on master. | | 6 | JSON field naming on FLOSS objects | **`qs_*` prefix** (`qs_tags`, `qs_section`, `qs_structure`, `qs_layout`, `qs_enabled`) for safety / clarity. | | 7 | Tag decoded / stack / tight strings | **Yes** — apply text-based taggers to those string types too. Layout section/structure usually N/A for non-static strings. | | 8 | Logging timestamps | **Adopt QS format globally** — all FLOSS logs get `%Y-%m-%d %H:%M:%S` timestamps (same as quantumstrand `floss/logging_.py`). | ### 13.1 Logging format (decision #8 locked) | Master (pre-migration) | Target (post-migration) | |------------------------|-------------------------| | Format string | `{color}%(levelname)s{RESET}: %(name)s: %(message)s` | `%(asctime)s {color}%(levelname)s{RESET}: %(name)s: %(message)s` | | Date format | (none) | `%Y-%m-%d %H:%M:%S` | | Example line | `INFO: floss.main: extracting strings` | `2026-07-09 13:51:00 INFO: floss.main: extracting strings` | **Action on master:** take the quantumstrand `floss/logging_.py` change as-is when integrating. Global for all FLOSS logs (not gated on `--quantum`). ## 14. Operational runbook (post-merge) ### Developer setup ```bash git clone --recurse-submodules https://github.com/mandiant/flare-floss.git cd flare-floss git lfs install git lfs pull python -m venv .venv && source .venv/bin/activate pip install -e ".[dev,qs]" # or whatever extra name is chosen pytest -k qs ``` ### Rebuilding OSS DBs locally (Windows) ```powershell # build jh from lancelot, then: python scripts/build_oss_db.py ` --config floss\qs\db\data\oss\libraries.json ` --jh-path path\to\jh.exe ` --output-dir floss\qs\db\data\oss ` --continue-on-error ``` ### Viewer Out of scope for this migration (do not set up or deploy `qs-viewer`). ## 15. Success metrics Migration is "done" when: 1. A user on `master` can run `floss --quantum sample.exe` and get tagged, section-annotated static strings. 2. `floss sample.exe -j` without the flag remains backward compatible. 3. CI on `master` runs QS unit/integration tests with LFS. 4. Bi-weekly OSS DB workflow opens PRs against `master`. 5. Only one shipped binary (`floss`); `--quantum` is the sole entry to QS features. 6. Stack / tight / decoded / language strings receive text tags when `--quantum` is on. 7. No regression in FLOSS deobfuscation tests (`test_main`, language suites, etc.). 8. GUI / qs-viewer intentionally not part of success criteria. ## 16. Appendix A -- Tag reference | Tag | Meaning | Typical visibility | |-----|---------|-------------------| | `#common` | globally prevalent string | mute | | `#duplicate` | repeated string content | mute | | `#code` | overlaps identified instructions | hide | | `#reloc` | overlaps relocation data | hide | | `#code-junk` | junk-code string DB hit | mute | | `#winapi` | Windows API/DLL name | mute | | `#capa` | expert/capa-derived interest | highlight | | `#zlib`, `#openssl`, `#msvc`, ... | OSS / CRT library provenance | mute | | `#decoded` | from XOR-decoded PE image | default | ## 17. Appendix B -- PE structures annotated by QS - section header - import table - export table - rich header - layout regions: `header`, section names, `overlay`, `Authenticode digital signature`, `gap`, `rsrc: ...` ELF: `elf header`, `program header`, `section header`, `string table`, `symbol table` Mach-O: `macho header`, `load command`, `segment header`, `section header`, `code signature`, `certificates`, `plist: entitlements` ## 18. Appendix C -- Quick command cheat sheet for explorers ```bash # Diff surface area git diff --stat master...quantumstrand # Only non-data code git diff --name-status master...quantumstrand -- \ ':(exclude)floss/qs/db/data' ':(exclude)qs-viewer/package-lock.json' # Run QS locally (quantumstrand worktree) pip install -e ".[qs]" qs /path/to/sample -j | head python -m floss.qs.main /path/to/sample # Run FLOSS on master worktree cd ../flare-floss-master pip install -e ".[dev]" floss /path/to/sample -j ``` ## 19. Appendix D -- Divergence notes for integrators - **`floss/main.py` and `floss/results.py` are currently identical** between tips of the two branches. Integration edits land cleanly without resolving QS-vs-FLOSS conflicts in those files. - Master is **ahead** on dependency hygiene (`verify-pins`, AGENTS.md, newer pins). Always **merge master -> integration branch first**. - Quantumstrand FLIRT `.sig` files in a non-LFS-pulled tree look like 132-byte pointer files; master trees have full multi-MB binaries. Don't commit pointer stubs onto master by mistake. - `libraries.json` (62 libs) is the future set; `DEFAULT_FILENAMES` in `oss.py` (17 + CRT) is the runtime set. After CI rebuilds, update `DEFAULT_FILENAMES` (or auto-discover `*.jsonl.gz`) so new DBs are actually loaded. ## 20. Document history | Date | Author | Notes | |------|--------|-------| | 2026-07-09 | migration planning | Initial detailed inventory from `quantumstrand` @ f367032 vs `master` @ 98e2259; incorporates GSoC 2026 proposal slices with current-code corrections | | 2026-07-09 | decisions locked | Feature-flag `--quantum`; core deps always; single floss binary; no GUI migration; all LFS including sigs; `qs_*` JSON fields; tag non-static strings; adopt QS log timestamps globally (§13.1) | | 2026-07-10 | architecture | master merged into quantumstrand; §21 first-class modular layout (no `floss/qs/`), split mains, single giant PR commit series | *When this plan is executed, keep this file updated per slice (checkboxes, open decisions, and any schema changes). Prefer linking PRs next to each Slice checklist item.* --- ## 21. First-class modular architecture (post master-merge) **Context (2026-07-10):** `master` has been merged into `quantumstrand` (dep/CI updates). Direction is now a **single land onto master**, not a long parallel-branch life. Constraints from locked decisions: - `--quantum` feature flag (default off) - No separate `qs` binary / no GUI - No long-lived `floss/qs/` package — features are first-class under `floss/` - Core deps always installed; DBs + FLIRT sigs on LFS - JSON enrichments use `qs_*` field names on FLOSS objects - Tag static **and** stack / tight / decoded / language strings - Global log timestamps (QS `logging_.py`) ### 21.1 Problems with the current shape | File / tree | Lines | Issue | |-------------|------:|-------| | `floss/qs/main.py` | ~2585 | Monolith: types + PE/ELF/Mach-O parsers + tagging + Rich UI + CLI + ResultDocument | | `floss/main.py` | ~843 | CLI + workspace load + full analysis orchestration in one `main()` (~334 lines) + `make_parser` (~206) | | `floss/qs/` | package | Signals “experiment / second product”; import path `floss.qs.*` everywhere | | Dual models | — | QS Pydantic `ResultDocument` vs FLOSS dataclass `ResultDocument` | Goal: **one product**, thin CLI, libraries with clear boundaries, same import style as `floss.language.*` / `floss.render.*`. ### 21.2 Design principles 1. **`main.py` is only a conductor** — parse args, call pipeline, print. No PE parsing, no tag DB I/O, no Rich layout drawing. 2. **One concept → one module (or small subpackage)** — layout, tags, enrich, render stay separable and unit-testable. 3. **Match existing FLOSS style** — subpackages for multi-file domains (`language/`, `render/`); flat modules for single concerns (`stackstrings.py`). 4. **No `qs` package name** — use domain names: `layout`, `tags`, `enrich`. Keep `qs_*` **only** on JSON/API fields and the `--quantum` flag (experiment branding without folder branding). 5. **Internal models may stay rich** (Pydantic/Layout tree); **public FLOSS results stay dataclasses** with optional `qs_*` fields. 6. **Move-only refactors first, then wire** — preserve behavior with import shims until tests are green, then delete shims in the same PR series. 7. **Scripts are not libraries** — VT/GP/bulk builders live under `scripts/`, not `floss/`. ### 21.3 Target package layout ``` floss/ ├── main.py # thin CLI entry (target ~150–250 lines) ├── cli.py # ArgumentParser helpers, make_parser(), StringType enum ├── pipeline.py # run_analysis(sample, args) -> ResultDocument (today’s main() body) ├── results.py # + optional qs_* on string types + metadata + qs_layout ├── strings.py # existing static extractors ├── utils.py / identify.py / stackstrings.py / tightstrings.py / string_decoder.py ├── logging_.py # timestamps (QS form) ├── version.py │ ├── ranges.py # Range, Slice, OffsetRanges (from qs/main foundation) │ ├── layout/ # binary structure tree (from compute_*_layout) │ ├── __init__.py # public: compute_layout(buf) -> Layout │ ├── base.py # Layout ABC, SectionLayout, SegmentLayout, Structure, helpers │ ├── pe.py # compute_pe_layout, collect_pe_structures, reloc/code │ ├── elf.py # compute_elf_layout, ELF robustness helpers │ ├── macho.py # fat/thin Mach-O │ └── extract.py # extract_layout_strings, gap walking, collect_strings │ ├── tags/ # string classification (from qs/db + taggers) │ ├── __init__.py # load_taggers(), apply_tags(), default TagRules │ ├── rules.py # TagRules type, hide/mute/highlight helpers, FP lib filter │ ├── expert.py # was qs/db/expert.py │ ├── gp.py │ ├── oss.py │ ├── winapi.py │ └── data/ # LFS: was qs/db/data/{crt,expert,gp,oss,winapi} │ ├── crt/ │ ├── expert/ │ ├── gp/ │ ├── oss/ │ └── winapi/ │ ├── enrich.py # --quantum glue: map FLOSS strings ↔ layout + tags │ # enrich_result_document(doc, sample_bytes, rules) -> doc │ ├── render/ │ ├── default.py # existing FLOSS tables; optional qs columns when enabled │ ├── json.py # serializes optional qs_* (dataclass encoder already ok) │ ├── sanitize.py │ └── layout_text.py # optional: Rich section-tree renderer (from qs render_*) │ # only used when --quantum and not --json, if we want QS TUI │ ├── language/ # unchanged ├── features/ # unchanged ├── sigs/ # LFS FLIRT └── (no qs/) scripts/ ├── build_oss_db.py # already here; paths → floss/tags/data/oss ├── bulk_quantum.py # was floss/qs/bulk.py (optional) ├── tags/ # was floss/qs/scripts/* │ ├── extract_strings.py │ ├── fetch_vt_hashes.py │ ├── generate_gp_db.py │ └── query_string.py └── ... tests/ ├── test_layout_*.py # was test_qs_elf/macho/code_ranges/offset_ranges/unit ├── test_tags_*.py # was test_gp_db, test_winapi_db, test_qs_oss_db ├── test_enrich_*.py # integration: --quantum CLI, mapping, visibility └── test_results_schema_compat.py ``` **What deliberately does *not* exist after the land:** - `floss/qs/` - `qs` console script - `build-qs.yml` / `qs.spec` / `qs-viewer/` (GUI already out of scope) ### 21.4 How `qs/main.py` splits (responsibility map) | Current region in `floss/qs/main.py` | Approx. | Destination | |--------------------------------------|--------:|-------------| | `Range`, `Slice`, `OffsetRanges` | foundation | `floss/ranges.py` | | `ExtractedString` / layout string helpers that only serve layout tree | — | prefer **not** duplicating FLOSS `StaticString`; layout extract can emit offsets+text then convert | | `Layout` ABC + children/tag_strings/mark_structures | ~150 | `floss/layout/base.py` | | `PELayout` / `compute_pe_layout` / structures / lancelot code | ~400+ | `floss/layout/pe.py` | | ELF helpers + `compute_elf_layout` | ~250 | `floss/layout/elf.py` | | Mach-O parsers + `compute_macho_layout` | ~400 | `floss/layout/macho.py` | | `compute_layout` dispatch + XOR MZ | ~60 | `floss/layout/__init__.py` | | `extract_layout_strings` / gaps / `collect_strings` | ~100 | `floss/layout/extract.py` | | Query fns + `load_databases` | ~80 | `floss/tags/__init__.py` | | `remove_false_positive_lib_strings`, hide rules, styles | ~100 | `floss/tags/rules.py` | | Rich `render_*` | ~400 | `floss/render/layout_text.py` (optional UX) | | QS-only `ResultDocument` Pydantic models | ~80 | **drop as public API**; keep only if needed as internal DTO during extract, or build `qs_layout` dict directly | | QS `main()` CLI | ~120 | **delete**; behavior via `floss --quantum` | `floss/qs/db/*.py` → rename package to `floss/tags/*.py` and fix `pathlib.Path(__file__).parent / "data"`. ### 21.5 How `floss/main.py` splits | Current piece | Destination | |---------------|-------------| | `StringType`, `ArgumentParser`, `ArgumentValueError`, `make_parser` | `floss/cli.py` | | `set_log_config` | stay in `logging_.py` or `cli.py` | | `load_vw`, `get_file_type`, `get_signatures`, `select_functions`, workspace errors | `floss/workspace.py` (optional; keeps vivisect loading out of CLI) | | Analysis orchestration (`main()` body: static → language → viv → stack/tight/decoded → render) | `floss/pipeline.py` → `run(sample, options) -> ResultDocument` | | `if __name__` / argv handling / exit codes | `floss/main.py` only | **`pipeline.py` sketch:** ```text run(sample, opts) -> ResultDocument: doc = empty ResultDocument(analysis=...) static = get_static_strings(...) language identify / extract ... if deobfuscation enabled: vw = load_vw(...); stack/tight/decoded ... if opts.quantum: doc = enrich.apply(doc, sample_bytes, tag_rules=opts.tag_rules) doc.metadata.runtime = ... return doc ``` `main.py` then: parse → `run` → `render.json|default` → print. ### 21.6 Enrichment API (first-class `--quantum`) `floss/enrich.py` is the only place that knows about both pipeline outputs and layout/tags: ```text apply(doc: ResultDocument, data: bytes, *, min_length: int, tag_rules: TagRules) -> ResultDocument: layout = floss.layout.compute_layout(data) # optional: still extract layout-local strings for structure marking / # or only use layout tree for section maps section_index = build_offset_index(layout) # offset -> section name structure_index = build_structure_index(layout) # offset -> structure name taggers = floss.tags.load_taggers() for each string list in doc.strings: # static, stack, tight, decoded, language_*: for s in list: s.qs_tags = tag_string(taggers, s.string) if hasattr offset and static-like: s.qs_section = section_index.lookup(s.offset) s.qs_structure = structure_index.lookup(s.offset) doc.metadata.qs_enabled = True doc.qs_layout = to_layout_dto(layout) # structure-only tree, no nested strings apply_visibility_rules(doc, tag_rules) # hide/mute for render path return doc ``` **Section/structure:** meaningful for static (and language static) offsets. **Tags:** all string kinds (locked decision). ### 21.7 Results model extensions ```python # floss/results.py (illustrative) @dataclass(frozen=True) class StaticString: string: str offset: int encoding: StringEncoding qs_tags: tuple[str, ...] = () qs_section: str = "" qs_structure: str = "" # StackString / TightString / DecodedString: add qs_tags (and empty section/structure defaults) @dataclass class Metadata: ... qs_enabled: bool = False @dataclass class ResultDocument: metadata: Metadata analysis: Analysis = ... strings: Strings = ... qs_layout: Optional[dict] = None # or a small LayoutDTO dataclass ``` JSON stays backward compatible when `--quantum` is off (empty defaults omitted if encoder skips empties — pick one policy and test it). ### 21.8 CLI surface (single binary) ```text floss [options] sample --quantum enable layout + tagging enrichments --qs-hide-tag TAG (repeatable) # optional slice 2 --qs-mute-tag TAG --qs-highlight-tag TAG ``` No `qs` entry point in `pyproject.toml`. PyInstaller `floss.spec` gains `datas=` for `floss/tags/data/**` (from current `qs.spec`). ### 21.9 Dependency / packaging - Move current `[project.optional-dependencies] qs` packages into **core** `dependencies` (lancelot, pyelftools, machofile, msgspec, colorama, dnfile as needed). - `packages` / package-data must include `floss/tags/data`. - `.gitattributes` LFS paths update: `floss/tags/data/**` and `floss/sigs/*.sig`. - `build-oss-db.yml` output dir → `floss/tags/data/oss`. ### 21.10 Implementation order inside one giant PR Prefer **one PR into master** with a **linear commit series** (reviewable, bisectable). Suggested commits: | Commit | Intent | Risk | |--------|--------|------| | 1 | LFS `.gitattributes` for sigs + future `tags/data`; install docs | ops | | 2 | Add `floss/ranges.py`, `floss/layout/*` by **move** from `qs/main.py` (behavior-identical `compute_layout`) | low if tests still import via thin shim | | 3 | Add `floss/tags/*` + move `db/data` → `tags/data`; fix loaders | low | | 4 | Add `floss/enrich.py` + extend `results.py` with `qs_*` | medium | | 5 | Split `floss/cli.py` + `pipeline.py`; thin `main.py`; add `--quantum` | medium | | 6 | Render: optional tag columns / layout_text when quantum | medium | | 7 | Update tests imports `floss.qs.*` → `floss.layout/tags`; add enrich CLI tests | medium | | 8 | Delete `floss/qs/`, `qs` script, build-qs, qs-viewer (if still present), fix CI paths | high-touch but mechanical | | 9 | OSS workflow + scripts path updates; mypy; changelog / `doc/quantumstrand.md` user docs | low | **Rule:** after each commit, `pytest -k 'layout or tags or qs or enrich or main'` (names evolve) stays green. **Anti-pattern:** one giant “move everything + rewrite main” commit — unreviewable and hard to fix. ### 21.11 Temporary compatibility (optional, same PR only) If useful during commits 2–7: ```python # floss/qs/main.py (shim only, deleted before merge) from floss.layout import compute_layout, ... from floss.ranges import Range, Slice, OffsetRanges # re-export for old tests ``` Do **not** merge the shim to master; final tree has zero `floss/qs`. ### 21.12 Testing map after rename | Old test | New focus / name | |----------|------------------| | `test_qs_unit` | `test_ranges` | | `test_qs_offset_ranges`, `test_qs_code_ranges` | `test_layout_ranges` | | `test_qs_elf`, `test_qs_macho`, `test_qs` / PMA | `test_layout_*` | | `test_gp_db`, `test_winapi_db`, `test_qs_oss_db` | `test_tags_*` | | `test_qs_build_oss_db` | keep; update paths | | new | `test_enrich_cli`, `test_results_schema_compat`, `test_tag_all_string_types` | ### 21.13 Size budget (sanity targets) | Module | Soft max lines | Notes | |--------|---------------:|-------| | `main.py` | ~250 | conductor only | | `cli.py` | ~250 | parser text is long; ok | | `pipeline.py` | ~300 | orchestration | | `layout/pe.py` | ~400 | largest parser; split further only if needed | | `layout/macho.py` | ~400 | | | `layout/elf.py` | ~250 | | | `layout/base.py` | ~200 | | | `enrich.py` | ~200 | | | `tags/*.py` | ~150 each | already small | If `pe.py` stays >500 after move, extract `layout/pe_structures.py` and `layout/pe_code.py` (lancelot ranges). ### 21.14 Mental model ```text ┌─────────────┐ │ floss CLI │ main.py + cli.py └──────┬──────┘ │ ┌──────▼──────┐ │ pipeline │ static / language / deobfuscation └──────┬──────┘ │ if --quantum ┌──────▼──────┐ │ enrich │ └──┬───────┬──┘ │ │ ┌────────▼──┐ ┌─▼────────┐ │ layout │ │ tags │ │ pe/elf/.. │ │ data+DB │ └───────────┘ └──────────┘ │ ┌──▼──────────┐ │ results + │ │ render/json │ └─────────────┘ ``` ### 21.15 Out of scope (unchanged) - GUI / qs-viewer - Making quantum default-on - Parallel `quantumstrand` release binary - Rewriting stack/decoded algorithms to be layout-aware (tags only)