QUANTUMSTRAND → FLOSS Integration Guide

Status: living plan (exploration POC in progress)
Base branch: quantumstrand (now up to date with master)
Exploration branch: refactor/first-class-modular (fork for modular layout; 2–3 iterations expected)

0. Locked product constraints (2026-07 mentor sync)

These supersede earlier GSoC “backward compatible / opt-in flag” assumptions.

What this means technically

• FLOSS ResultDocument may break old consumers; design the right schema once.
• Do not invest in dual-format adapters or long-lived qs_* optional fields “for safety” unless they still read well as first-class names (e.g. tags, section, structure, layout).
• Prefer one pipeline: deobfuscation (FLOSS) + layout/tag context (QS) → one document → one renderer.

1. Executive summary

QUANTUMSTRAND is a mature experiment on quantumstrand: layout-aware static strings, tag databases, OSS rebuild CI, LFS, tests. It still lives under floss/qs/ as a second product (qs entry point + separate JSON).

Remaining work is productization, not greenfield features:

1. Make layout + tagging first-class FLOSS modules (no floss/qs/ ghetto).
2. Split monoliths (qs/main.py ~2.5k lines, floss/main.py ~850 lines).
3. Unify CLI/output for a major release.
4. Keep OSS DB pipeline healthy and reviewable (see §4).

Not remaining work (already on QS branch): PE/ELF/Mach-O layout, tag loaders, LFS, OSS build script + CI, most unit tests.

2. Branch / process model

master ──┐
         ├── already merged ──► quantumstrand (integration base)
                                    │
                                    ├── refactor/first-class-modular  (POC #1)
                                    ├── … possible POC #2–3 …
                                    └── stacked small PRs → master (major release)

3. Current inventory (quantumstrand)

3.1 Still true

3.2 OSS library builds (fork status)

Successful library build work tracked on the fork:

• https://github.com/vee1e/flare-floss/pull/13
• https://github.com/vee1e/flare-floss/pull/14

Expect updated DBs + workflow fixes to land on QS; migration paths must follow whatever tree those PRs use.

3.3 Build-time profile (vcpkg matrix, ranked)

Longest library builds observed (profiled run). Use this to prioritize CI time, timeouts, and optional continue-on-error / shard strategy — not as a reason to block migration.

Takeaways:

• libmysql alone ~33 minutes dominates wall time; consider excluding from default weekly matrix, building on a slower cadence, or isolating in its own job.
• Top ~5 libraries are most of the cost; document matrix exclusions explicitly in libraries.json / workflow comments.
• CI caching of vcpkg may help but native builds will remain expensive.

4. OSS DB PR reviewability (new requirement)

Problem: updating .jsonl.gz LFS blobs makes GitHub diffs opaque (“binary files differ”).

Requirement: when OSS DBs change, emit a text summary of string-level changes for humans.

Proposed behavior (build_oss_db.py + workflow)

1. After rebuild/merge per library, compute a textual diff vs previous DB:
   • added / removed / changed string keys (and maybe library_version).
2. Write artifact e.g. floss/tags/data/oss/build_diff.txt (or upload CI artifact).
3. PR body (create-pull-request action) includes a truncated summary:
   • ~100 lines max of the text diff (head + “… N more lines” footer).
4. Full diff remains as workflow artifact for download.

Example PR body fragment:

## OSS DB update summary (truncated)

### openssl.jsonl.gz
+ 12 strings
- 3 strings
~ 1 version stamp change on existing keys

```diff
+ "OpenSSL 3.x..."
- "OpenSSL 3.0.7..."
...

(… truncated; see build_diff.txt artifact …)

This is independent of the modular refactor but should land with the OSS pipeline PRs.

---

## 5. Target architecture (first-class, no `qs/` package)

### 5.1 Package layout

```text
floss/
  main.py              # thin CLI entry only
  cli.py               # argparse / StringType
  pipeline.py          # full analysis orchestration
  results.py           # major-release schema (layout + tags + deobfuscated strings)
  ranges.py            # Range, Slice, OffsetRanges
  enrich.py            # apply layout mapping + taggers to result strings
  logging_.py          # timestamps
  strings.py / utils.py / stackstrings.py / tightstrings.py / string_decoder.py
  identify.py / language/ / features/ / sigs/

  layout/              # binary structure (from qs/main compute_*)
    __init__.py        # compute_layout()
    base.py            # Layout ABC, section/segment nodes, Structure
    pe.py / elf.py / macho.py
    extract.py         # strings in layout gaps

  tags/                # classification DBs (from qs/db)
    __init__.py        # load_taggers(), apply
    rules.py           # mute/hide/highlight, FP lib filter
    expert.py / gp.py / oss.py / winapi.py
    data/              # LFS (crt, expert, gp, oss, winapi)

  render/
    default.py         # primary human output (context-aware)
    json.py
    layout_text.py     # Rich section tree helpers if still needed

scripts/
  build_oss_db.py
  tags/                # VT/GP helpers moved out of package

Deleted for release direction: floss/qs/, qs console script, build-qs.yml, qs.spec, qs-viewer/ (unmigrated).

5.2 Pipeline (major release mental model)

sample
  → static / language strings
  → layout.compute_layout (PE/ELF/Mach-O)
  → tags on all string kinds
  → section/structure on offset-bearing strings
  → vivisect deobfuscation (stack/tight/decoded) when enabled
  → tags on recovered strings
  → single ResultDocument → render

No dual “FLOSS mode vs QS mode” long term.

5.3 Schema naming (preference under major bump)

Because compatibility is not required, prefer plain names over permanent qs_ prefixes:

POC may still use temporary names; final major release should look like FLOSS, not like an experiment bolted on.

5.4 Split map (qs/main.py → modules)

5.5 Split map (floss/main.py)

6. Exploration plan (2–3 iterations)

Iteration 1 — Modular POC (this branch)

• Physically remove floss/qs/ ghetto: layout/, tags/, ranges.py, render split.
• Keep behavior: existing QS tests green under new imports.
• Optionally thin-shim nothing on final tip (no floss.qs imports).
• Do not require perfect unified schema yet.

Iteration 2 — Unified pipeline / schema

• One ResultDocument including deobfuscated strings + tags + layout.
• Default output is context-rich (no --quantum).
• Drop parallel entry points and unused release workflows.

Iteration 3 — Polish

• OSS text diffs in PR bodies.
• CI matrix tuning using build-time profile (libmysql etc.).
• Docs + release notes for major bump.
• Delete dead code / experiment naming.

Each iteration should yield reviewable PR(s), not only a private branch.

7. File / CI checklist (release direction)

Move / add

• floss/ranges.py, floss/layout/**, floss/tags/**, floss/enrich.py, floss/cli.py, floss/pipeline.py
• LFS attributes for floss/tags/data/** and floss/sigs/*.sig
• Core deps in pyproject.toml (no optional [qs] long-term)
• build-oss-db.yml paths + text diff in PR body
• Tests renamed/imported from floss.layout / floss.tags

Do not land

• qs-viewer/, web-release.yml (GUI)
• build-qs.yml, qs.spec, qs script (parallel product)
• Long-term dual JSON compatibility layers

Preserve from master (already on QS)

• verify-pins.yml, AGENTS.md, modern dep pins

8. Testing gates

pytest                       # full
pytest -k "layout or tags or qs or enrich or main"   # during rename churn
# smoke
floss sample.exe
floss sample.exe -j

Corpus still needed later: disk PE vs memory dump, ELF stripped, fat Mach-O (section parsing edge cases from GSoC notes remain valid).

9. Open questions (narrowed)

10. Document history

Update this file when POC outcomes change the preferred schema or module map.

11. POC iteration 1 status (refactor/first-class-modular)

Date: 2026-07-11

Done

• Removed floss/qs/ package entirely.
• First-class modules:
  • floss/ranges.py
  • floss/layout/ (pe, elf, macho, base, extract, checks, util)
  • floss/tags/ (+ LFS data under tags/data/)
  • floss/document.py (layout-aware result document)
  • floss/analyze_static.py (CLI for layout+tags static analysis)
  • floss/render/layout_text.py
  • floss/cli.py (parser extracted from main.py)
  • floss/enrich.py (stub for iteration 2 unify)
• Core deps promoted in pyproject.toml (no [qs] extra / no qs script).
• Dropped build-qs.yml + qs.spec.
• OSS workflow paths → floss/tags/data/oss.
• Tests updated to new imports.

Tests

250 passed, 2 skipped, 6 xfailed

(full pytest tests/ on this branch)

Not done yet (iteration 2+)

• Unify deobfuscation pipeline with layout/tags (single default output, no dual docs).
• Extract remaining main.py body into pipeline.py.
• Textual OSS DB diffs in PR bodies.
• Plain field names (tags/section/layout) for major release schema.
• PyInstaller floss.spec datas for tags/data (currently only FLOSS sigs).
• Landing as small stacked PRs onto master.