QUANTUMSTRAND FLOSS Master Migration Guide
- Status: draft for GSoC / integration planning
- Branches compared:
quantumstrand@f367032vsmaster@98e2259 - Merge-base:
9a7f351 - Divergence: ~313 commits on
quantumstrandnot inmaster; ~44 commits onmasternot inquantumstrand - Local worktrees:
This document is a full inventory of what exists on quantumstrand today, how it differs from the GSoC 2026 proposal assumptions, and a concrete plan to fold QS features, workflows, CI, and the viewer interface into master without breaking FLOSS consumers.
1. Executive summary
QUANTUMSTRAND (QS) is already a mature parallel product living under floss/qs/, with its own CLI (qs), JSON schema, PyInstaller build, GitHub Actions, tag databases (Git LFS), OSS rebuild pipeline, and a React single-file viewer (qs-viewer/).
Critical reality vs. the original GSoC proposal:
| Proposal assumption (older) | Current quantumstrand state |
|---|---|
| Start with PE-only layout + thin adapter | PE + ELF + Mach-O (thin & fat) layout already implemented |
| Build OSS pipeline from scratch | scripts/build_oss_db.py + bi-weekly build-oss-db.yml already land PRs |
| Migrate DBs to Git LFS first | Done (.gitattributes + LFS for DBs and FLIRT .sig files) |
Slice 1 creates qs_integration.py + --quantum |
Not started -- QS is still a separate binary/entry point |
Merge into FLOSS ResultDocument |
Not started -- QS has its own Pydantic ResultDocument (meta + recursive layout) |
floss-viewer/ port |
Exists as qs-viewer/ consuming QS-native JSON only |
Bottom line: most "feature build" work on the QS experiment branch is done. The remaining hard problem is integration architecture: how to expose QS capabilities through FLOSS's CLI/JSON/renderers while preserving backward compatibility, then retire the parallel codename.
Recommended posture: treat quantumstrand as the feature source of truth, rebase/merge recent master into it (or land incremental PRs onto master), and use a slice-based integration that starts with optional enrichments rather than rewriting FLOSS's static-string path overnight.
2. Worktree setup (local)
Already created:
# from repo root (quantumstrand checkout)
git worktree add ../flare-floss-master master
# quantumstrand is the primary worktree; a second worktree for the same branch
# is not needed and git will refuse it if the branch is already checked out.
git worktree list
# /Users/verma/Projects/Mandiant/flare-floss ... [quantumstrand]
# /Users/verma/Projects/Mandiant/flare-floss-master ... [master]
Use the master worktree for migration implementation branches:
cd ../flare-floss-master
git checkout -b feat/qs-integration-slice-1
Note: LFS objects on the quantumstrand worktree may appear as pointer stubs until git lfs pull is run. Master currently has full FLIRT .sig binaries without LFS; quantumstrand tracks them via LFS.
3. Architecture comparison
3.1 Master FLOSS (today)
flare-floss/
|-- floss/
| |-- main.py # sole CLI entry: floss
| |-- results.py # dataclasses ResultDocument
| |-- strings.py # static ASCII/UTF-16 extraction
| |-- stackstrings.py / tightstrings.py / string_decoder.py
| |-- language/ # Go / Rust language-specific strings
| |-- render/{default,json,sanitize}.py
| \-- sigs/ # FLIRT signatures (plain git blobs)
|-- .github/workflows/
| |-- build.yml, tests.yml, publish.yml, black-format.yml
| \-- verify-pins.yml # pinact -- master-only hygiene
|-- scripts/ # IDA/Ghidra/Binja/r2/x64dbg import helpers
\-- doc/
FLOSS ResultDocument shape (simplified):
{
"metadata": {
"file_path": "...",
"version": "...",
"imagebase": 0,
"min_length": 4,
"runtime": {},
"language": ""
},
"analysis": { "enable_static_strings": true, "enable_stack_strings": true, ... },
"strings": {
"static_strings": [{ "string": "...", "offset": 0, "encoding": "ASCII" }],
"stack_strings": [...],
"tight_strings": [...],
"decoded_strings": [...],
"language_strings": [...],
"language_strings_missed": [...]
}
}
StaticString has only: string, offset, encoding. No tags, no section, no structure.
Analysis pipeline: vivisect-backed deobfuscation + language ID + static strings from whole-file extraction. No recursive PE/ELF/Mach-O layout tree; no string tagging DBs.
3.2 Quantumstrand (today)
flare-floss/ (on quantumstrand)
|-- floss/
| |-- main.py # UNCHANGED vs master (still pure FLOSS)
| |-- results.py # UNCHANGED
| \-- qs/ # entire experiment lives here
| |-- main.py # ~2585 lines: layout + tag + render + CLI
| |-- bulk.py # directory batch runner
| |-- db/
| | |-- {gp,oss,expert,winapi}.py
| | \-- data/ # Git LFS: .jsonl.gz, .bin, capa.jsonl
| \-- scripts/ # VT fetch, GP generation helpers
|-- qs-viewer/ # Vite + React single-file GUI
|-- scripts/build_oss_db.py # vcpkg + jh automation
|-- .github/
| |-- pyinstaller/qs.spec
| \-- workflows/
| |-- build-qs.yml
| |-- tests-qs.yml
| |-- build-oss-db.yml
| \-- web-release.yml # GitHub Pages for qs-viewer
|-- .gitattributes # LFS rules
\-- pyproject.toml # extra [qs] deps + `qs` console script
QS ResultDocument shape (Pydantic):
{
"meta": {
"version": "0.3.0",
"timestamp": "...",
"sample": { "md5": "...", "sha1": "...", "sha256": "...", "path": "..." },
"min_str_len": 4
},
"layout": {
"name": "pe",
"offset": 0,
"length": 3584,
"strings": [
{
"string": "WriteLine",
"offset": 1036,
"size": 18,
"encoding": "ascii",
"tags": ["#common", "#msvc"],
"structure": ""
}
],
"children": [
{ "name": "header", "offset": 0, "length": 512, "strings": [], "children": [] },
{ "name": ".text", "offset": 512, "length": 1024, "strings": [...], "children": [] }
]
}
}
Important: QS does not implement stack/tight/decoded string recovery. It is a contextual static-strings tool. FLOSS remains the deobfuscation engine.
3.3 Entry points and packaging
| Concern | Master | Quantumstrand |
|---|---|---|
| Console scripts | floss = floss.main:main |
+ qs = floss.qs.main:main |
| Optional deps | dev, build |
+ qs extra: dnfile, colorama, machofile, msgspec, python-lancelot, pyelftools |
| PyInstaller | floss.spec |
+ qs.spec (bundles all DB data paths) |
| Binary name | floss |
quantumstrand / quantumstrand.exe |
| Version | floss.version -> 3.1.1 |
QS reports QS_VERSION = "0.3.0" independently |
viv-utils pin is slightly higher on QS (>=0.8.0 vs >=0.7.9); otherwise core FLOSS deps are aligned, though master has newer dev/tooling pins (black, mypy, pre-commit, etc.).
4. Feature inventory (what must move)
4.1 Binary layout analysis (floss/qs/main.py)
Recursive Layout tree with non-overlapping ordered children and gap strings.
| Format | Function | Capabilities |
|---|---|---|
| PE | compute_pe_layout |
Sections, header/overlay segments, Authenticode, resources (readable names via PE_RESOURCE_TYPES), gap nodes; structures: section headers, import/export tables, rich header; #code via lancelot BinExport2 ranges; #reloc via reloc table; single-byte XOR MZ decoding |
| ELF | compute_elf_layout |
Robust section/segment fallback, SHT_NOBITS skip, exec ranges, narrow metadata regions (ELF/program/section headers, string/symbol tables); ELFError / out-of-range hardening |
| Mach-O | compute_macho_layout |
Thin + fat; segments/sections; code signature, certs, entitlements plist |
| Fallback | SegmentLayout(name="binary") |
Whole-file string extraction |
GSoC proposal note on section parsing: memory-dumped PEs (SectionAlignment + slack) vs disk PEs (FileAlignment) remain a testing concern. PE path already truncates out-of-range sections and logs warnings, but corpus coverage for AD1/static vs memory dumps should still be expanded during integration (Slice 3 testing deliverable in the proposal is still valid).
4.2 String tagging system
Taggers loaded by load_databases():
| Source | Module / data | Tags produced | Default render rule |
|---|---|---|---|
| WinAPI | db/winapi + apis.txt.gz, dlls.txt.gz |
#winapi |
mute (default) |
| Expert | db/expert + capa.jsonl |
e.g. #capa |
highlight |
| OSS libs | db/oss + 17x *.jsonl.gz + crt/msvc_v143.jsonl.gz |
#zlib, #openssl, #msvc, ... |
mute |
| Global prevalence | db/gp + gp.jsonl.gz, cwindb, hash DBs |
#common |
mute |
| Junk code strings | gp/junk-code.jsonl.gz |
#code-junk |
mute |
| Layout-derived | PE/ELF code & reloc ranges | #code, #reloc |
hide |
| XOR decode | PE path | #decoded |
default |
| Duplicates | same string text twice | #duplicate |
mute |
Post-processing:
remove_false_positive_lib_strings-- drops library tags if too few hits (default threshold 5).hide_strings_by_rules-- removes#code/#reloc(etc.) from visible render.- Tag display consolidates adjacent identical tag groups; suppresses
#commonwhen more specific tags exist.
4.3 Tag databases (Git LFS)
.gitattributes:
floss/qs/db/data/**/*.bin filter=lfs ...
floss/qs/db/data/**/*.gz filter=lfs ...
floss/qs/db/data/**/*.jsonl filter=lfs ...
floss/sigs/*.sig filter=lfs ...
30 LFS-tracked objects on quantumstrand. Master has no .gitattributes and stores FLIRT sigs as normal blobs (~15 MB total).
Checked-in OSS DBs (17 + CRT):
brotli, bzip2, cryptopp, curl, detours, jemalloc, jsoncpp, kcp, liblzma, libpcap, libsodium, mbedtls, openssl, sqlite3, tomcrypt, wolfssl, zlib, + crt/msvc_v143.
Build manifest (libraries.json) already lists 62 vcpkg packages for the automated pipeline (expansion beyond the original "top 25" goal). Checked-in DBs and the build matrix are intentionally not 1:1 yet; rebuild CI merges into existing .jsonl.gz files.
4.4 OSS database construction pipeline
libraries.json --> vcpkg (x64-windows-static, msvc143, release)
|
v
jh (lancelot-bin) extract strings from .lib
|
v
convert -> JSONL -> gzip
|
v
merge with existing DB (new wins on collision)
|
v
PR to quantumstrand (build-oss-db.yml)
Key implementation: scripts/build_oss_db.py (~878 lines)
- Modular
Vcpkg,JHExtractor,Converterclasses - Cross-library strings are kept (not globally deduped); per-library still collapses duplicates by default
- Metrics written to
build_metrics.json - Windows-focused by design (PE-first rationale documented in
floss/qs/db/data/oss/readme.md)
CI: .github/workflows/build-oss-db.yml
- Schedule: bi-weekly (
0 0 1,15 * *) +workflow_dispatch - Runs on
windows-latest, builds jh fromwilliballenthin/lancelot - Opens PR with base
quantumstrand(must change tomasterafter merge) - Requires
contents: write+pull-requests: write
4.5 CLI & bulk analysis
qs CLI (floss/qs/main.py:main):
| Flag | Behavior |
|---|---|
path |
sample path |
-n / --minimum-length |
min string length (default 4) |
-j / --json |
emit QS ResultDocument JSON |
-l / --load |
re-render from saved QS JSON |
-d / -q |
debug / quiet (reuses floss.main.set_log_config) |
No mute/highlight/hide CLI overrides yet (hardcoded tag_rules in main). GSoC proposal's "tag filters as CLI arguments" is still open work.
floss/qs/bulk.py: walks a directory, shells python -m floss.qs.main --json, optional rendered .txt.
4.6 Terminal rendering
Rich-based columns: tags | string | structure | offset, with box-drawing layout delimiters, muted/highlight styles, tag-group consolidation. Depends on colorama for Windows console.
4.7 Web viewer (qs-viewer/)
| Item | Detail |
|---|---|
| Stack | Vite + React + TypeScript, single-file build |
| Input | QS JSON only (meta + recursive layout) |
| Features | drag-drop upload, search, min length, tag multi-select, structure multi-select, show/hide untagged / no-structure, column toggles, copy filtered strings |
| Noisy tags default | #common, #duplicate, #code, #reloc, #code-junk treated as muted styling; #capa highlighted |
| Deploy | web-release.yml -> GitHub Pages on qs-viewer/** changes to quantumstrand |
| Sample data | src/pma0303_qs.json baked in for preview |
Not yet: browsing inside tag databases when clicking a tag (GSoC / issues #782, #784 -- still open product work).
4.8 Tests (quantumstrand-only)
| File | Focus |
|---|---|
tests/test_qs.py |
end-to-end PMA sample, JSON round-trip |
tests/test_qs_unit.py |
Range / Slice |
tests/test_qs_offset_ranges.py |
reloc/code range structure |
tests/test_qs_code_ranges.py |
merge overlapping ranges |
tests/test_qs_elf.py |
ELF layout sanity |
tests/test_qs_macho.py |
Mach-O thin layout + structures |
tests/test_qs_pma0101.py |
PMA fixture expectations |
tests/test_qs_oss_db.py |
OSS DB load/query |
tests/test_qs_build_oss_db.py |
build script unit + orchestration (~625 lines) |
tests/test_gp_db.py |
GP + hash DBs |
tests/test_winapi_db.py |
winapi DB |
tests-qs.yml runs pytest -k qs on Ubuntu with LFS + submodules.
Test data submodule: quantumstrand pins tests/data @ 6eba5b1 (includes PMA samples QS needs); master pins older 53e9101. Integration must bump the submodule on master carefully.
4.9 CI / release matrix
| Workflow | Master | Quantumstrand | Migration action |
|---|---|---|---|
tests.yml |
Yes | Yes | keep; extend matrix or jobs for QS |
build.yml |
Yes FLOSS binary | Yes (minor branch filter) | keep FLOSS build; optionally embed QS later |
publish.yml |
Yes | Yes | unchanged |
black-format.yml |
Yes | Yes | unchanged |
verify-pins.yml |
Yes | No | bring to integrated branch (master hygiene) |
build-qs.yml |
No | Yes QS PyInstaller | either merge into build.yml or keep as floss-qs artifact job on master |
tests-qs.yml |
No | Yes | fold into tests.yml or keep named job on master |
build-oss-db.yml |
No | Yes | retarget base branch -> master |
web-release.yml |
No | Yes Pages | retarget to master; path may become floss-viewer/ |
mypy.ini extras |
-- | intervaltree, lancelot, capa, virustotal3, colorama | merge |
4.10 Ancillary / scripts
| Path | Role | Migrate? |
|---|---|---|
floss/qs/scripts/extract_strings.py |
string extract helper | yes (or fold into scripts/) |
floss/qs/scripts/fetch_vt_hashes.py |
VT corpus for GP / library selection | yes (dev tooling) |
floss/qs/scripts/generate_gp_db.py |
GP DB generation | yes |
floss/qs/scripts/query_string.py |
DB query debug CLI | yes |
floss/qs/db/data/expert/import_from_capa.py |
capa -> expert rules | yes |
floss/qs/db/data/oss/jh_to_qs.py |
CSV->JSONL (legacy; build script supersedes bulk use) | keep for manual rebuilds |
floss/logging_.py |
timestamps in log format on QS only | decide: adopt on master or keep FLOSS format |
5. What does not need to be reinvented
These GSoC "new features" are largely complete on quantumstrand and should be ported as-is, then refined:
- ELF section parsing -- done with robustness work (PR history includes #1312, #1319).
- Git LFS for DBs -- done (#1309).
- Automatic tag DB construction pipeline -- done (script + bi-weekly CI).
- Mach-O static layout -- done beyond original "StaticString-only" sketch.
- qs-viewer baseline -- done; DB peek / tag drill-down still missing.
Remaining product gaps from the proposal + community threads:
- FLOSS integration (
--quantum/ always-on enrichments, merged JSON). - CLI tag filters (mute/hide/highlight overrides).
- Viewer: open/browse tag databases (#782, #784).
- Diverse section-parsing corpus (disk vs memory dump, binwalk padding, stripped malware).
- Library manifest expansion -- pipeline supports 62 libs; only 17 DBs checked in until CI rebuilds land.
- Deprecate QUANTUMSTRAND codename after stable merge.
6. Integration design (target state)
6.1 Design principles
- Backward compatible JSON: legacy FLOSS roots
analysis,metadata,stringsremain valid without QS fields. --quantumis a feature flag: default off for this migration phase; not flipping to default-on yet.- Do not break deobfuscation: stack/tight/decoded recovery stays FLOSS; when
--quantumis on, text tags apply to static + stack/tight/decoded/language strings; section/structure mapping is primarily for static offsets. - No GUI migration for now:
qs-viewer/and Pages deploy stay out of scope. - QS runtime deps are core:
python-lancelot,pyelftools,machofile,msgspec, etc. install with FLOSS always (not a long-lived optional extra). - Single binary: only
floss; no parallelqs/quantumstrandartifact.
6.2 Target merged JSON (proposal-aligned, updated)
When QS mode is enabled:
{
"analysis": { "enable_static_strings": true, "...": "..." },
"metadata": {
"file_path": "dotnet-hello.exe",
"version": "3.x.y",
"qs_enabled": true,
"qs_version": "0.3.0"
},
"strings": {
"static_strings": [
{
"string": "WriteLine",
"offset": 1036,
"encoding": "ASCII",
"qs_tags": ["#common", "#msvc"],
"qs_structure": "",
"qs_section": ".text"
}
],
"stack_strings": [],
"decoded_strings": []
},
"qs_layout": {
"name": "pe",
"offset": 0,
"length": 3584,
"children": [
{ "name": "header", "offset": 0, "length": 512, "children": [] },
{ "name": ".text", "offset": 512, "length": 1024, "children": [] }
]
}
}
Semantics:
qs_layoutholds structural hierarchy only (no per-node string lists required for FLOSS JSON -- avoids doubling string storage).- Per-string
qs_*fields are optional; absent when QS disabled. - Existing FLOSS tooling that ignores unknown fields keeps working.
- Add a checked-in JSON Schema +
tests/test_results_schema_compat.pyas proposed.
6.3 Mapping QS layout -> FLOSS static strings
FLOSS StaticString(offset, string, encoding)
|
|- match offset in layout node -> qs_section = nearest section/segment name
|- match offset in Structure map -> qs_structure = structure name
\- tag string text via load_databases() -> qs_tags
Edge cases:
| Case | Strategy |
|---|---|
| String only in FLOSS (language/Go extractor path) | tag by text; section may be unknown |
| String only in QS layout gaps | optional: union into static_strings or leave QS-only in layout |
Encoding enum mismatch (ASCII vs ascii) |
normalize in adapter |
| Multiple sections claim offset | prefer deepest layout leaf |
| Memory dump / invalid PE | layout may fall back to binary; tags still apply |
6.4 Target directory structure (post-migration)
Aligned with the proposal, adjusted to current code:
flare-floss/
|-- floss/
| |-- main.py # --quantum / tag filter CLI (feature flag, default off)
| |-- results.py # optional qs_* fields on string types + metadata
| |-- qs/ # layout, tags, dbs (library code; no separate shipped CLI)
| | |-- main.py # may remain as module internals; do not ship `qs` console script
| | |-- layout.py # NEW: split compute_*_layout out of main.py (optional refactor)
| | |-- tags.py # NEW: load_databases + tag rules (optional refactor)
| | |-- render.py # NEW: rich renderer (optional refactor)
| | |-- db/ # unchanged data + loaders
| | \-- ...
| |-- qs_integration.py # NEW: FLOSS<->QS adapter (proposal Slice 1)
| \-- render/
| |-- default.py # optional QS-aware columns when enabled
| \-- json.py # serializes optional qs fields
|-- scripts/build_oss_db.py
|-- tests/
| |-- test_qs_*.py # existing
| |-- test_qs_integration_cli.py
| |-- test_qs_layout_mapping.py
| |-- test_qs_tagging.py
| |-- test_qs_visibility.py
| \-- test_results_schema_compat.py
|-- doc/
| |-- quantumstrand-migration.md # this file
| \-- quantumstrand.md # user-facing architecture + usage (to write)
\-- .github/workflows/
|-- tests.yml # include qs jobs
|-- build.yml # single floss binary (QS DBs embedded)
|-- build-oss-db.yml # base: master
\-- verify-pins.yml
Not migrated (locked): qs-viewer/, web-release.yml, standalone qs entry point / build-qs.yml / qs.spec.
Refactor recommendation: floss/qs/main.py is a 2.5k-line monolith. Prefer splitting during migration without changing behavior, so FLOSS integration imports layout/tags cleanly. Not a blocker for Slice 1 if adapter imports from floss.qs.main temporarily.
7. Slice-based migration plan (updated)
The GSoC four-slice model remains the right process; contents are rebased onto current code.
Slice 0 -- Foundation (prerequisite, 0.5-1 week)
Goal: make master able to host QS artifacts safely.
- Merge or rebase latest
masterinto integration branch (deps,verify-pins,AGENTS.md, mypy/black bumps). - Land
.gitattributes+ Git LFS for:floss/qs/db/data/**- optionally migrate
floss/sigs/*.sigto LFS (coordinate: history rewrite not required if only new commits use LFS; existing master blobs remain until rewritten -- prefer forward-only LFS for new paths first, then sigs carefully).
- Bump
tests/datasubmodule to QS pin if PMA fixtures are required. - Document
git lfs install && git lfs pullindoc/installation.md. - CI: ensure all workflows that need DBs set
lfs: true.
Exit criteria: clean install on Linux CI with LFS; FLOSS tests still green; no accidental 15MB re-add of DB binaries as non-LFS.
Slice 1 -- Adapter baseline (1-2 weeks)
Goal: FLOSS can optionally compute layout and attach section/structure to static strings (PE first).
Deliverables:
-
floss/qs_integration.pyanalyze_layout(sample_bytes) -> ResultLayout-like treemap_strings_to_layout(static_strings, layout) -> enriched stringsto_qs_document/from_floss_documenthelpers for tests
- CLI:
--quantum(store_true, default False) onfloss.main— permanent feature flag for this migration phase -
results.StaticString: optionalqs_tags,qs_section,qs_structure(default empty / omit in JSON if unset -- decide one serialization policy and stick to it) -
metadata.qs_enabled: bool -
qs_layoutoptional top-level field onResultDocument(or nested under metadata -- prefer top-level as in proposal) - Tests:
test_qs_integration_cli.py,test_qs_layout_mapping.py(PE fixtures from PMA) - JSON schema draft + load old FLOSS JSON without error
Non-goals this slice: tagging DBs, viewer rename, OSS CI retarget.
Risk: lancelot panics -- already mitigated with broad BaseException catch in QS PE path; keep that in the adapter.
Slice 2 -- Tagging + CI + DBs (2-3 weeks)
Goal: full tag pipeline available under FLOSS.
- Load DBs via existing
floss.qs.db.*loaders - Apply taggers to static and stack / tight / decoded / language strings
- CLI visibility controls, e.g.:
--qs-hide-tag #code--qs-mute-tag #common--qs-highlight-tag #capa- or a compact
--tag-rule '#code=hide,#capa=highlight'
- Default rules match current QS behavior when
--quantumis on - Copy/adapt workflows:
tests-qs-> masterbuild-oss-db.ymlbase ->master- PyInstaller data paths if FLOSS binary should embed DBs when built with QS
- Package data: ensure wheels/sdists include
floss/qs/db/data(MANIFEST / package-data) - Tests:
test_qs_tagging.py,test_qs_visibility.py, DB load tests
Dependency decision (locked): always install QS deps + ship DBs with FLOSS.
Slice 3 -- Multi-format layout + corpus (2 weeks)
Already implemented on QS branch -- focus is wiring + testing under FLOSS, not greenfield parsers.
-
Ensure adapter calls
compute_layout(PE/ELF/Mach-O dispatch) not PE-only -
ELF/Mach-O mapping tests under FLOSS CLI
-
Build corpus matrix:
Source type Why Clean PE from disk / VT FileAlignment, normal sections Memory / crash dump PE SectionAlignment, slack, heap garbage binwalk-carved / padded imperfect offsets Stripped ELF section vs segment fallback Fat Mach-O multi-arch children XOR'd PE #decodedpathGo/Rust FLOSS language samples interaction with language strings -
Document sample-source consideration in
doc/quantumstrand.md
Slice 4 -- Polish & single-binary cleanup (ongoing)
- Out of scope:
qs-viewer/,web-release.yml, GUI / DB-browser work — do not migrate - Do not ship a parallel
qs/quantumstrandbinary — onlyflosswith--quantum - Drop or never land on master:
build-qs.yml,.github/pyinstaller/qs.spec,project.scriptsentryqs = ... - Apply text taggers to static and stack / tight / decoded (and language) strings when
--quantum - User docs + changelog for
--quantum - Community feedback loop (proposal Figure 2)
Optional Slice 5 -- Performance & DB productization
- Audit
jh_to_qs/build_oss_dbfor CI time (proposal concern) - Land expanded OSS DBs via automated PRs (62-lib matrix)
- Consider multi-triplet matrix only if string coverage gaps are measured
- Publish release artifacts when DB growth is significant
8. File-level migration checklist
Must copy / land onto master
.gitattributes # LFS: qs db data + floss/sigs/*.sig
floss/qs/ # package (layout, tags, dbs) — not a separate CLI long-term
scripts/build_oss_db.py
.github/workflows/tests-qs.yml # or fold into tests.yml
.github/workflows/build-oss-db.yml
.github/mypy/mypy.ini # merge QS ignore modules
pyproject.toml # QS deps as core runtime deps (not optional extra)
tests/test_qs*.py
tests/test_gp_db.py
tests/test_winapi_db.py
tests/data # submodule bump
Do not copy to master (locked out of scope):
qs-viewer/
.github/workflows/web-release.yml
.github/workflows/build-qs.yml # no parallel quantumstrand binary
.github/pyinstaller/qs.spec
# pyproject [project.scripts] qs = ... — do not add
Must implement new (not pure copy)
floss/qs_integration.py
floss/main.py # flags + call adapter
floss/results.py # optional fields
floss/render/default.py # optional QS rendering
floss/render/json.py # already dataclass-friendly; ensure qs_* serialize
doc/quantumstrand.md # user docs
schemas/floss-results.json # optional but proposed
tests/test_qs_integration_*.py
tests/test_results_schema_compat.py
Must retarget after merge
| Item | From | To |
|---|---|---|
build-oss-db.yml PR base |
quantumstrand |
master |
build-qs.yml |
quantumstrand |
drop, no parallel binary |
tests-qs.yml triggers |
quantumstrand |
or fold into tests.yml |
web-release.yml |
quantumstrand |
do not migrate now |
| Release assets | quantumstrand-* |
only floss-*; embed QS DBs in floss PyInstaller datas when --quantum support lands |
| README / qs readme | experiment language | integrated --quantum feature language |
Master-only assets to preserve
.github/workflows/verify-pins.yml
AGENTS.md
newer dependabot pins in pyproject / requirements
Explicit non-goals for first merge
- Replacing FLOSS static string extractor with QS extractor wholesale
- Making stack/decoded strings participate in layout trees (tags yes; section/structure usually N/A)
- Migrating
qs-viewer/ any GUI / GitHub Pages web-release - Shipping a parallel
qs/quantumstrandbinary - Multi-OS vcpkg matrix for OSS DBs (Windows static remains justified)
- Rewriting full git history for LFS (forward-only LFS tracking of sigs + DBs is enough)
9. Compatibility & risk matrix
| Risk | Impact | Mitigation |
|---|---|---|
| JSON field additions break strict parsers | Medium | optional fields; schema version; golden tests for legacy JSON |
| LFS not pulled -> empty DBs / broken tags | High | CI lfs: true; clear install docs; runtime warn if DB missing |
| lancelot / native deps fail on some platforms | Medium | catch panics; degrade to section-only PE via pefile |
| PyInstaller size growth from DBs | Medium | measure; document; optional slim builds |
Dual CLI confusion (floss vs qs) |
N/A | locked: no qs binary on master |
| Test submodule mismatch | Medium | single pin; QS tests skip if fixture absent |
| Tag false positives from stale OSS DBs | Medium | keep build-oss-db cron; FP lib threshold |
| Section mapping wrong on memory dumps | Medium | corpus tests; document limitations |
Logging format change (logging_.py timestamps) |
Low | intentional — adopt QS timestamps globally; note in changelog |
requirements.txt drift master vs QS |
Low | regenerate from one lock strategy after merge |
10. Testing strategy
Unit
- Layout pure functions (
OffsetRanges, range merge, Range/Slice) - DB loaders (oss, gp, winapi, expert)
build_oss_dbconverters/merge without network
Integration
floss --quantum sample.exe -jproduces valid merged JSONfloss sample.exe -jbit-identical structure to pre-QS (no qs fields)- Load legacy JSON with
floss -lstill works - Tag rules hide/mute/highlight
Golden / fixture
- PMA 01-01, 03-03 (existing QS tests)
- ELF + Mach-O fixtures in testfiles submodule
- One intentionally truncated / padded PE
CI gates before merge to master
pytest # full suite
pytest -k qs # QS suite
mypy (project config)
black / isort / pre-commit
verify-pins on workflow edits
smoke: pyinstaller floss (with QS DB datas embedded)
Manual acceptance
- Terminal UX for
floss --quantummatches expected tags/sections on a known sample - OSS workflow dry-run (
workflow_dispatch) opens a PR against master
11. Suggested PR stack (Graphite / stacked PRs)
Keep PRs reviewable (~reviewer-hour sized):
- chore: LFS +
.gitattributes+ install docs (no feature code) - feat: vendor
floss/qs/package + deps + unit tests (no FLOSS CLI wiring) - feat:
qs_integration+--quantumPE section mapping - feat: tagging + tag-rule CLI
- ci: fold QS tests + OSS DB workflow retarget
- feat: multi-format layout coverage tests under FLOSS
- docs: usage + schema for
--quantum(no viewer PR) - chore: single floss release binary embeds QS DBs; no
quantumstrandartifact
Avoid one mega-PR of 15k insertions even though the branch already has them; landing in stacks preserves bisectability.
12. Mapping GSoC proposal deliverables -> status
| Deliverable | Status on quantumstrand | Migration work left |
|---|---|---|
| Merge-ready QS codebase | Yes (experiment branch) | Integrate into master FLOSS UX |
| Section mapping PE | Done | Adapter + FLOSS fields |
| ELF section parsing | Done | Wire through adapter + tests on master |
| Mach-O static | Done | Wire + tests |
| Tag databases + loaders | Done | Package data on master builds |
| OSS auto pipeline | Done | Retarget to master; expand checked-in DBs |
| Git LFS | Done | Introduce on master |
| Tag filters CLI | Partial (hardcoded rules only) | Implement args |
| Merged FLOSS+QS JSON | Not started | Core integration |
| JSON Schema + compat tests | Not started | Add |
| GUI viewer | Done on QS branch | Out of scope — do not migrate |
| Visualize / browse DBs | Not started | Out of scope for this migration |
qs_integration.py |
Not started | Slice 1 |
--quantum flag |
Not started | Slice 1 (opt-in feature flag) |
| Single floss binary only | N/A | Locked — no parallel qs artifact |
| Tag stack/decoded/tight | Static-only in QS CLI | Yes under FLOSS --quantum |
13. Decisions (locked)
| # | Question | Decision |
|---|---|---|
| 1 | --quantum default |
Feature flag only for now — opt-in via --quantum; not the default for static strings. Revisit later if desired. |
| 2 | QS deps (lancelot / pyelftools / machofile / …) | Always install — promote into core runtime dependencies (not a [qs] extra long-term). |
| 3 | Standalone qs / quantumstrand binary |
No — single floss binary; QS only via --quantum. Drop build-qs.yml / qs.spec / qs console script (do not land on master). |
| 4 | Viewer / GUI | Do not migrate. Ignore qs-viewer/, web-release.yml, and all GUI work for this migration. |
| 5 | FLIRT .sig + QS DBs on LFS |
All LFS — both floss/sigs/*.sig and floss/qs/db/data/** tracked with Git LFS on master. |
| 6 | JSON field naming on FLOSS objects | qs_* prefix (qs_tags, qs_section, qs_structure, qs_layout, qs_enabled) for safety / clarity. |
| 7 | Tag decoded / stack / tight strings | Yes — apply text-based taggers to those string types too. Layout section/structure usually N/A for non-static strings. |
| 8 | Logging timestamps | Adopt QS format globally — all FLOSS logs get %Y-%m-%d %H:%M:%S timestamps (same as quantumstrand floss/logging_.py). |
13.1 Logging format (decision #8 locked)
| Master (pre-migration) | Target (post-migration) |
|---|---|
| Format string | {color}%(levelname)s{RESET}: %(name)s: %(message)s |
| Date format | (none) |
| Example line | INFO: floss.main: extracting strings |
Action on master: take the quantumstrand floss/logging_.py change as-is when integrating. Global for all FLOSS logs (not gated on --quantum).
14. Operational runbook (post-merge)
Developer setup
git clone --recurse-submodules https://github.com/mandiant/flare-floss.git
cd flare-floss
git lfs install
git lfs pull
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,qs]" # or whatever extra name is chosen
pytest -k qs
Rebuilding OSS DBs locally (Windows)
# build jh from lancelot, then:
python scripts/build_oss_db.py `
--config floss\qs\db\data\oss\libraries.json `
--jh-path path\to\jh.exe `
--output-dir floss\qs\db\data\oss `
--continue-on-error
Viewer
Out of scope for this migration (do not set up or deploy qs-viewer).
15. Success metrics
Migration is "done" when:
- A user on
mastercan runfloss --quantum sample.exeand get tagged, section-annotated static strings. floss sample.exe -jwithout the flag remains backward compatible.- CI on
masterruns QS unit/integration tests with LFS. - Bi-weekly OSS DB workflow opens PRs against
master. - Only one shipped binary (
floss);--quantumis the sole entry to QS features. - Stack / tight / decoded / language strings receive text tags when
--quantumis on. - No regression in FLOSS deobfuscation tests (
test_main, language suites, etc.). - GUI / qs-viewer intentionally not part of success criteria.
16. Appendix A -- Tag reference
| Tag | Meaning | Typical visibility |
|---|---|---|
#common |
globally prevalent string | mute |
#duplicate |
repeated string content | mute |
#code |
overlaps identified instructions | hide |
#reloc |
overlaps relocation data | hide |
#code-junk |
junk-code string DB hit | mute |
#winapi |
Windows API/DLL name | mute |
#capa |
expert/capa-derived interest | highlight |
#zlib, #openssl, #msvc, ... |
OSS / CRT library provenance | mute |
#decoded |
from XOR-decoded PE image | default |
17. Appendix B -- PE structures annotated by QS
- section header
- import table
- export table
- rich header
- layout regions:
header, section names,overlay,Authenticode digital signature,gap,rsrc: ...
ELF: elf header, program header, section header, string table, symbol table
Mach-O: macho header, load command, segment header, section header, code signature, certificates, plist: entitlements
18. Appendix C -- Quick command cheat sheet for explorers
# Diff surface area
git diff --stat master...quantumstrand
# Only non-data code
git diff --name-status master...quantumstrand -- \
':(exclude)floss/qs/db/data' ':(exclude)qs-viewer/package-lock.json'
# Run QS locally (quantumstrand worktree)
pip install -e ".[qs]"
qs /path/to/sample -j | head
python -m floss.qs.main /path/to/sample
# Run FLOSS on master worktree
cd ../flare-floss-master
pip install -e ".[dev]"
floss /path/to/sample -j
19. Appendix D -- Divergence notes for integrators
floss/main.pyandfloss/results.pyare currently identical between tips of the two branches. Integration edits land cleanly without resolving QS-vs-FLOSS conflicts in those files.- Master is ahead on dependency hygiene (
verify-pins, AGENTS.md, newer pins). Always merge master -> integration branch first. - Quantumstrand FLIRT
.sigfiles in a non-LFS-pulled tree look like 132-byte pointer files; master trees have full multi-MB binaries. Don't commit pointer stubs onto master by mistake. libraries.json(62 libs) is the future set;DEFAULT_FILENAMESinoss.py(17 + CRT) is the runtime set. After CI rebuilds, updateDEFAULT_FILENAMES(or auto-discover*.jsonl.gz) so new DBs are actually loaded.
20. Document history
| Date | Author | Notes |
|---|---|---|
| 2026-07-09 | migration planning | Initial detailed inventory from quantumstrand @ f367032 vs master @ 98e2259; incorporates GSoC 2026 proposal slices with current-code corrections |
| 2026-07-09 | decisions locked | Feature-flag --quantum; core deps always; single floss binary; no GUI migration; all LFS including sigs; qs_* JSON fields; tag non-static strings; adopt QS log timestamps globally (§13.1) |
When this plan is executed, keep this file updated per slice (checkboxes, open decisions, and any schema changes). Prefer linking PRs next to each Slice checklist item.