# QUANTUMSTRAND → FLOSS Integration Guide

**Status:** living plan (exploration POC in progress)  
**Base branch:** `quantumstrand` (now up to date with `master`)  
**Exploration branch:** `refactor/first-class-modular` (fork for modular layout; 2–3 iterations expected)

---

## 0. Locked product constraints (2026-07 mentor sync)

These supersede earlier GSoC “backward compatible / opt-in flag” assumptions.

| Topic | Decision |
|-------|----------|
| Compatibility | **No legacy JSON/API compatibility required** — this is a **major FLOSS release** |
| Feature flag | **`--quantum` is not the end state.** Prefer **QS-style output as the default** where applicable; flag only OK briefly during exploration |
| Naming | **Prefer domain names** (`layout`, `tags`, `enrich`) over long-lived `qs/` package branding. CLI experiment names can die with the major release |
| Binary | **One `floss` binary only** — no parallel `quantumstrand` / `qs` release artifact |
| GUI | **Out of scope** — do not migrate `qs-viewer` / Pages |
| Dependencies | **Always install** layout/tag deps (lancelot, pyelftools, machofile, msgspec, …) as core |
| LFS | **All** tag DBs + FLIRT `.sig` files on Git LFS |
| Tags on string kinds | Apply text tags to **static + stack + tight + decoded + language** strings |
| Logging | **Global timestamps** (`%Y-%m-%d %H:%M:%S` prefix) as on QS `logging_.py` |
| PR shape | **Smaller reviewable PRs** along the way (not one unreviewable mega-dump). Exploration may be 1 branch; land as stacked/small PRs |
| Iterations | **~2–3 exploration rounds** expected (“POC → keep what we like / rewrite if needed”) — not endless |
| Master sync | `master` **already merged into** `quantumstrand` (deps/CI); work forks from QS |

### What this means technically

- FLOSS `ResultDocument` **may break** old consumers; design the *right* schema once.
- Do **not** invest in dual-format adapters or long-lived `qs_*` optional fields “for safety” unless they still read well as first-class names (e.g. `tags`, `section`, `structure`, `layout`).
- Prefer **one pipeline**: deobfuscation (FLOSS) + layout/tag context (QS) → one document → one renderer.

---

## 1. Executive summary

QUANTUMSTRAND is a mature experiment on `quantumstrand`: layout-aware static strings, tag databases, OSS rebuild CI, LFS, tests. It still lives under `floss/qs/` as a **second product** (`qs` entry point + separate JSON).

**Remaining work is productization, not greenfield features:**

1. Make layout + tagging **first-class FLOSS modules** (no `floss/qs/` ghetto).
2. Split monoliths (`qs/main.py` ~2.5k lines, `floss/main.py` ~850 lines).
3. Unify CLI/output for a major release.
4. Keep OSS DB pipeline healthy and **reviewable** (see §4).

**Not remaining work (already on QS branch):** PE/ELF/Mach-O layout, tag loaders, LFS, OSS build script + CI, most unit tests.

---

## 2. Branch / process model

```text
master ──┐
         ├── already merged ──► quantumstrand (integration base)
                                    │
                                    ├── refactor/first-class-modular  (POC #1)
                                    ├── … possible POC #2–3 …
                                    └── stacked small PRs → master (major release)
```

| Practice | Guidance |
|----------|----------|
| Base | Fork exploration off **`quantumstrand`**, not stale master |
| POC | Allowed to be messy; **throw away or cherry-pick** what works |
| Landing | Prefer **small PRs** (layout move, tags move, CLI wire, CI paths…) for review |
| Rounds | Budget **2–3** exploration iterations, not open-ended rewrite loops |
| Mega-PR | Avoid one PR that is both “all features” and “all renames” without history; stack if needed |

---

## 3. Current inventory (quantumstrand)

### 3.1 Still true

| Area | Location | Notes |
|------|----------|-------|
| Layout + tags + Rich UI + QS CLI | `floss/qs/main.py` (~2585 lines) | Monolith to split |
| Tag DBs | `floss/qs/db/` + `data/` (LFS) | Move to `floss/tags/` |
| OSS builder | `scripts/build_oss_db.py` | Keep; fix paths after move |
| OSS CI | `.github/workflows/build-oss-db.yml` | Bi-weekly + dispatch |
| FLOSS CLI / deobfuscation | `floss/main.py` (~843) | Split parser vs pipeline |
| Viewer | `qs-viewer/` | **Do not migrate** |
| Parallel build | `build-qs.yml`, `qs.spec` | **Drop** for release |

### 3.2 OSS library builds (fork status)

Successful library build work tracked on the fork:

- https://github.com/vee1e/flare-floss/pull/13  
- https://github.com/vee1e/flare-floss/pull/14  

Expect updated DBs + workflow fixes to land on QS; migration paths must follow whatever tree those PRs use.

### 3.3 Build-time profile (vcpkg matrix, ranked)

Longest library builds observed (profiled run). Use this to prioritize CI time, timeouts, and optional `continue-on-error` / shard strategy — **not** as a reason to block migration.

| Rank | Library | Duration | Seconds |
|-----:|---------|----------|--------:|
| 1 | libmysql | 00:32:55 | 1975 |
| 2 | libarchive | 00:10:24 | 624 |
| 3 | libxml2 | 00:05:09 | 309 |
| 4 | poco | 00:04:06 | 246 |
| 5 | boost-graph | 00:03:40 | 220 |
| 6 | libsndfile | 00:03:31 | 211 |
| 7 | capnproto | 00:02:44 | 164 |
| 8 | libev | 00:02:27 | 147 |
| 9 | avro-c | 00:02:20 | 140 |
| 10 | boost-log | 00:02:18 | 138 |
| 11 | libgit2 | 00:02:18 | 138 |
| 12 | sail | 00:02:09 | 129 |
| 13 | libevent | 00:01:48 | 108 |
| 14 | czmq | 00:01:29 | 89 |
| 15 | sdl2 | 00:01:28 | 88 |
| 16 | boost-test | 00:01:19 | 79 |
| 17 | libsrt | 00:01:16 | 76 |
| 18 | libhv | 00:01:14 | 74 |
| 19 | boost-filesystem | 00:01:04 | 64 |
| 20 | boost-chrono | 00:01:04 | 64 |

**Takeaways:**

- **libmysql alone ~33 minutes** dominates wall time; consider excluding from default weekly matrix, building on a slower cadence, or isolating in its own job.
- Top ~5 libraries are most of the cost; document matrix exclusions explicitly in `libraries.json` / workflow comments.
- CI caching of vcpkg may help but native builds will remain expensive.

---

## 4. OSS DB PR reviewability (new requirement)

**Problem:** updating `.jsonl.gz` LFS blobs makes GitHub diffs **opaque** (“binary files differ”).

**Requirement:** when OSS DBs change, emit a **text summary of string-level changes** for humans.

### Proposed behavior (`build_oss_db.py` + workflow)

1. After rebuild/merge per library, compute a textual diff vs previous DB:
   - added / removed / changed string keys (and maybe library_version).
2. Write artifact e.g. `floss/tags/data/oss/build_diff.txt` (or upload CI artifact).
3. **PR body** (create-pull-request action) includes a **truncated** summary:
   - ~**100 lines** max of the text diff (head + “… N more lines” footer).
4. Full diff remains as workflow artifact for download.

Example PR body fragment:

```markdown
## OSS DB update summary (truncated)

### openssl.jsonl.gz
+ 12 strings
- 3 strings
~ 1 version stamp change on existing keys

```diff
+ "OpenSSL 3.x..."
- "OpenSSL 3.0.7..."
...
```
(… truncated; see build_diff.txt artifact …)
```

This is independent of the modular refactor but should land with the OSS pipeline PRs.

---

## 5. Target architecture (first-class, no `qs/` package)

### 5.1 Package layout

```text
floss/
  main.py              # thin CLI entry only
  cli.py               # argparse / StringType
  pipeline.py          # full analysis orchestration
  results.py           # major-release schema (layout + tags + deobfuscated strings)
  ranges.py            # Range, Slice, OffsetRanges
  enrich.py            # apply layout mapping + taggers to result strings
  logging_.py          # timestamps
  strings.py / utils.py / stackstrings.py / tightstrings.py / string_decoder.py
  identify.py / language/ / features/ / sigs/

  layout/              # binary structure (from qs/main compute_*)
    __init__.py        # compute_layout()
    base.py            # Layout ABC, section/segment nodes, Structure
    pe.py / elf.py / macho.py
    extract.py         # strings in layout gaps

  tags/                # classification DBs (from qs/db)
    __init__.py        # load_taggers(), apply
    rules.py           # mute/hide/highlight, FP lib filter
    expert.py / gp.py / oss.py / winapi.py
    data/              # LFS (crt, expert, gp, oss, winapi)

  render/
    default.py         # primary human output (context-aware)
    json.py
    layout_text.py     # Rich section tree helpers if still needed

scripts/
  build_oss_db.py
  tags/                # VT/GP helpers moved out of package
```

**Deleted for release direction:** `floss/qs/`, `qs` console script, `build-qs.yml`, `qs.spec`, `qs-viewer/` (unmigrated).

### 5.2 Pipeline (major release mental model)

```text
sample
  → static / language strings
  → layout.compute_layout (PE/ELF/Mach-O)
  → tags on all string kinds
  → section/structure on offset-bearing strings
  → vivisect deobfuscation (stack/tight/decoded) when enabled
  → tags on recovered strings
  → single ResultDocument → render
```

No dual “FLOSS mode vs QS mode” long term.

### 5.3 Schema naming (preference under major bump)

Because compatibility is not required, prefer **plain names** over permanent `qs_` prefixes:

| Concept | Prefer | Avoid long-term |
|---------|--------|-----------------|
| tags on a string | `tags` | `qs_tags` |
| section name | `section` | `qs_section` |
| structure name | `structure` | `qs_structure` |
| layout tree | `layout` | `qs_layout` |
| feature enabled meta | omit or `version` only | `qs_enabled` |

POC may still use temporary names; final major release should look like FLOSS, not like an experiment bolted on.

### 5.4 Split map (`qs/main.py` → modules)

| Region | Destination |
|--------|-------------|
| `Range`, `Slice`, `OffsetRanges` | `floss/ranges.py` |
| ExtractedString + ascii/unicode extractors used by layout | `floss/layout/extract.py` (or share `floss/strings.py` later) |
| `Layout` hierarchy + Structure | `floss/layout/base.py` |
| PE / ELF / Mach-O compute | `floss/layout/{pe,elf,macho}.py` |
| `compute_layout` | `floss/layout/__init__.py` |
| DB query + `load_databases` | `floss/tags/__init__.py` |
| hide/mute/FP rules | `floss/tags/rules.py` |
| Rich render | `floss/render/layout_text.py` |
| Pydantic QS ResultDocument | fold into `floss/results.py` or `floss/document.py` during unify |
| QS `main()` | delete → `floss.main` / `pipeline` |

### 5.5 Split map (`floss/main.py`)

| Region | Destination |
|--------|-------------|
| `make_parser`, enums, ArgumentParser | `floss/cli.py` |
| viv load / sigs / file type | `floss/workspace.py` (optional) |
| analysis body | `floss/pipeline.py` |
| entry | `floss/main.py` |

---

## 6. Exploration plan (2–3 iterations)

### Iteration 1 — Modular POC (this branch)

- Physically remove `floss/qs/` ghetto: `layout/`, `tags/`, `ranges.py`, render split.
- Keep behavior: existing QS tests green under new imports.
- Optionally thin-shim nothing on final tip (no `floss.qs` imports).
- **Do not** require perfect unified schema yet.

### Iteration 2 — Unified pipeline / schema

- One `ResultDocument` including deobfuscated strings + tags + layout.
- Default output is context-rich (no `--quantum`).
- Drop parallel entry points and unused release workflows.

### Iteration 3 — Polish

- OSS text diffs in PR bodies.
- CI matrix tuning using build-time profile (libmysql etc.).
- Docs + release notes for major bump.
- Delete dead code / experiment naming.

Each iteration should yield **reviewable PR(s)**, not only a private branch.

---

## 7. File / CI checklist (release direction)

### Move / add

- `floss/ranges.py`, `floss/layout/**`, `floss/tags/**`, `floss/enrich.py`, `floss/cli.py`, `floss/pipeline.py`
- LFS attributes for `floss/tags/data/**` and `floss/sigs/*.sig`
- Core deps in `pyproject.toml` (no optional `[qs]` long-term)
- `build-oss-db.yml` paths + text diff in PR body
- Tests renamed/imported from `floss.layout` / `floss.tags`

### Do not land

- `qs-viewer/`, `web-release.yml` (GUI)
- `build-qs.yml`, `qs.spec`, `qs` script (parallel product)
- Long-term dual JSON compatibility layers

### Preserve from master (already on QS)

- `verify-pins.yml`, AGENTS.md, modern dep pins

---

## 8. Testing gates

```text
pytest                       # full
pytest -k "layout or tags or qs or enrich or main"   # during rename churn
# smoke
floss sample.exe
floss sample.exe -j
```

Corpus still needed later: disk PE vs memory dump, ELF stripped, fat Mach-O (section parsing edge cases from GSoC notes remain valid).

---

## 9. Open questions (narrowed)

| Question | Status |
|----------|--------|
| Backward compatible JSON? | **No** (major release) |
| Permanent `--quantum`? | **No** — default QS-style output |
| Keep `floss/qs/` name? | **No** long-term |
| GUI? | **No** for now |
| How many rewrite rounds? | **~2–3** |
| One mega-PR? | Prefer **small PRs**; exploration branch OK |
| DB PR visibility? | **Yes** — truncated text diff in PR body |
| Final field names `tags` vs `qs_tags`? | Prefer plain `tags` under major bump; confirm in Iteration 2 |

---

## 10. Document history

| Date | Notes |
|------|-------|
| 2026-07-09 | Initial inventory vs master; GSoC slices; early open decisions |
| 2026-07-09 | Locked: feature flag, core deps, single binary, no GUI, LFS, tag all string kinds, log timestamps |
| 2026-07-10 | §21 modular first-class layout proposal |
| 2026-07-11 | Mentor sync: no back-compat; QS output default; small PRs; 2–3 iterations; master∈QS; OSS PR #13/#14; library build profile; opaque DB diff → text summary; exploration branch process |

---

*Update this file when POC outcomes change the preferred schema or module map.*

---

## 11. POC iteration 1 status (`refactor/first-class-modular`)

**Date:** 2026-07-11

### Done

- Removed `floss/qs/` package entirely.
- First-class modules:
  - `floss/ranges.py`
  - `floss/layout/` (pe, elf, macho, base, extract, checks, util)
  - `floss/tags/` (+ LFS data under `tags/data/`)
  - `floss/document.py` (layout-aware result document)
  - `floss/analyze_static.py` (CLI for layout+tags static analysis)
  - `floss/render/layout_text.py`
  - `floss/cli.py` (parser extracted from `main.py`)
  - `floss/enrich.py` (stub for iteration 2 unify)
- Core deps promoted in `pyproject.toml` (no `[qs]` extra / no `qs` script).
- Dropped `build-qs.yml` + `qs.spec`.
- OSS workflow paths → `floss/tags/data/oss`.
- Tests updated to new imports.

### Tests

```
250 passed, 2 skipped, 6 xfailed
```

(full `pytest tests/` on this branch)

### Not done yet (iteration 2+)

- Unify deobfuscation pipeline with layout/tags (single default output, no dual docs).
- Extract remaining `main.py` body into `pipeline.py`.
- Textual OSS DB diffs in PR bodies.
- Plain field names (`tags`/`section`/`layout`) for major release schema.
- PyInstaller `floss.spec` datas for `tags/data` (currently only FLOSS sigs).
- Landing as small stacked PRs onto master.