vee1e

vee1e / d561fcadc2627cc3dcb115d18847a236

Last active 4 days ago

Like 0

Revision 65718922d95d52b4e4ee2371deef3c8096db2792

gsoc-2026-final-submission.md Raw

Google Summer of Code 2026 @ FLARE: Integrating QuantumStrand into FLOSS

Organization: FLARE (Mandiant, a Google Cloud company)

Contributor: Lakshit Verma (GitHub: @vee1e)

Mentor: Moritz Raabe

Repository: mandiant/flare-floss

Project overview

FLOSS is the FLARE team's command line tool for extracting obfuscated and plaintext strings from malware. An analyst runs it against a binary and gets the readable text the malware uses: URLs, registry paths, API names, and more. That output is useful but bare: it is a flat list of strings with no context.

QuantumStrand (QS) is an experimental branch of FLOSS that adds context. It parses the binary's structure, splits it into sections and regions, and tags each string with extra meaning: which library it likely comes from, whether it is common noise, which API family it belongs to. It also renders results in a hierarchical tree instead of a flat list.

The goal of this project was to take the best of QS and make it the default behavior of FLOSS, so every user gets layout-aware, tag-enriched output without extra flags. The project also covered the supporting infrastructure: automated string database builds, a redesigned command line, result caching, and a web viewer.

Project status

The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master, which is open as PR #1372 at the time of writing.

A thousand-sample parity test proved the big refactor did not change behavior: 905 of 905 layout outputs matched upstream QS exactly, and 918 of 918 classic-mode outputs matched master exactly.

Deliverables

1. Format parsing for ELF, Mach-O, and PE structures

QS previously understood PE files well but had gaps elsewhere. These PRs added structural parsing so layout rendering works across formats.

  • PR #1312: native ELF parsing. Builds a section tree, tags relocation sections so they are not misread as strings, detects executable sections, and handles XOR-decoded ELFs. A test corpus of ELF binaries was added to the testfiles submodule.
  • PR #1221: Mach-O parsing, including universal (fat) binaries, segments, code signatures, and entitlements.
  • PR #1230: PE export table parsing, extracting DLL names, exported symbols, and forwarder names.
  • PR #1231 and PR #1233: PE Rich header detection, plus a crash fix for empty headers.
  • PR #1229: human-readable PE resource names instead of raw numeric IDs.

2. Automated open-source string database builds

FLOSS tags strings by matching them against databases of known library strings. Those databases used to be built by hand. This work automated the whole pipeline so they stay fresh.

  • PR #1327: a build script and bi-weekly CI workflow that compiles open-source libraries with vcpkg, extracts their strings, converts and deduplicates them into the FLOSS database format, and opens an automated PR with build metrics when anything changes.
  • PR #1334: a readable text diff of database changes, embedded in the auto-generated PR descriptions.
  • PR #1332: workflow dependency and versioning fixes.
  • PR #1309: moved the string databases to Git LFS, saving roughly 24 MB per shallow clone.
  • The library list was settled at around 60 open-source libraries, chosen by auditing false-positive rates against a global prevalence database built from 150,000 real-world samples.

The database builder lives in scripts/build_oss_db.py and the automated update workflow in .github/workflows/build-oss-db.yml.

3. The QS to FLOSS integration

This is the core of the project: turning QS's separate package into the default FLOSS pipeline.

  • PR #1337: split the monolithic QS module into first-class floss/layout and floss/tags packages and unified the command line. Roughly 4,000 lines moved and rewritten with no behavior change.
  • PR #1347: made the unified pipeline the default. Every string type flows through the same stages into one ResultDocument, and every output mode renders from that document.

The pipeline is coordinated in floss/pipeline.py. The layout tree, tag engine, and renderers live in floss/layout, floss/tags, and floss/render. The results schema lives in floss/results.py.

4. The redesigned command line

QS had its own set of flags that did not match FLOSS. The spec called for one coherent interface, delivered in three parts.

  • PR #1355: Part 1. Symmetrical include and exclude flags for string types (--string-type / --no-string-type), renamed --functions to --analyze-functions, removed the manual load flag in favor of automatic results detection, and added runtime timing fields to the results.
  • PR #1360: Part 2. Render-time filters by section, structure, and tag (--section, --structure, --tag, with matching --no-* forms), --query for regex filtering that preserves the tree, --max-strings to cap output, an --interesting shortcut that drops noisy tags, a new --summary output, and sorted stable JSON with structured JSON errors on stderr.
  • PR #1367: Part 3. Result caching (see below).

5. Result caching

  • PR #1367: automatic analysis caching. The cache is keyed by a SHA-256 of the sample bytes plus the FLOSS version, stored in the platform cache directory, and written atomically behind a lock. Filters apply at render time, so cached results respect any new flags. Moritz benchmarked a sample going from about 30 seconds to about 3.5 seconds on cache hit, roughly an 8 to 10x speedup for repeat runs.

The module is floss/cache.py, with tests in tests/test_cache.py.

6. The web viewer

  • PR #1358: reworked the QS web viewer to match the unified ResultDocument schema, added light and dark themes, fuzzy search, and virtualized rendering for very large string corpora. Deployed live for iteration during development.

The viewer lives in the viewer/ directory and is served from a Vercel deployment during development; a standalone release path is planned.

7. Reproducibility and CI hygiene

  • PR #1303: a pinact workflow that verifies GitHub Actions are pinned to commit hashes, reducing supply chain risk.
  • PR #1308: an AGENTS.md so AI coding tools have the context they need before generating code in this repo.
  • PR #1304: fixed a stale pinned SHA in the build workflow.
  • PR #1370: deprecated all qs / QuantumStrand naming, renamed the viewer, retargeted CI to master, and cleaned up stale branches, preparing the branch for the final merge.

Pull request index

Deliverable PRs
Format parsing #1312, #1221, #1230, #1231, #1233, #1229
String databases #1327, #1334, #1332, #1309
QS to FLOSS integration #1337, #1347
Command line #1355, #1360, #1367
Web viewer #1358
CI and hygiene #1303, #1308, #1304, #1370
Merge to master #1372 (open)

Additional fixes merged along the way: PR #1217 (zlib false negatives), PR #1225 (consolidated adjacent tag display), PR #1242 (CallsTo scoring), PR #1243 (import taint), PR #1244 (clean SIGINT handling), PR #1259 (Rust stack string crash).

I also contributed to the upstream lancelot project, which the string extractor depends on: issue #235 and PR #237 fixed a crash on weak external C++ symbols during database builds.

Reviews I contributed

Part of the work was reviewing others. The notable PRs I was asked to review:

  • PR #1319: ELF parsing robustness, including fallback for corrupt binaries. I checked the new fallback logic and requested a real corrupted test file over a mock.
  • PR #1326: global prevalence database update. I asked whether the large size gap between the Go and Rust sub-databases was intentional before approving.
  • PR #1366: summary view revamp. I caught that the new filtering let untagged generic strings crowd out tagged interesting ones, and flagged now-dead code, before it merged.
  • PR #1273, PR #1294, PR #1328, PR #1240: robustness, performance, and cleanup reviews.

What is left to do

  • Merge QuantumStrand into master. PR #1372 is open and green, awaiting final review. This is the last gate.
  • Web viewer distribution. Decide how the viewer ships: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically.
  • Documentation. The usage guide does not yet document the new flags, the summary output, or the cache environment variables.
  • Cache maintenance. A v1 tradeoff: there is no eviction or quota for the cache directory yet, and very large result documents load fully into memory.

None of these block the core functionality. The new pipeline, CLI, and cache are all merged and working.

Challenges

The largest single risk was the big refactor in PR #1337. Moving 4,000 lines around risks quietly changing behavior. We addressed it with evidence instead of argument: a parity harness ran old and new code over a thousand random samples, and every output matched. That became the review bar for the rest of the project.

The automated database pipeline surfaced a real bug in the upstream lancelot string extractor. Two common C++ libraries (cryptopp and jsoncpp) triggered a crash on weak external symbols. I filed a reproduction and shipped the fix upstream, which is exactly the kind of problem a maintainer wants contributors to solve at the root.

The planning document was the other hard part. A discussion doc and an implementation spec are different things, and it took a while to realize we needed the latter. Rewriting it as a spec that could be handed to coding agents unblocked the final months of the project. It also became the design reference for the new CLI and the caching module.

Acknowledgements

Thanks to my mentor Moritz Raabe for the straight talk and the support, and for reviewing far more than his share of large diffs. Thanks to Willi Ballenthin for the sharp reviews and for merging the upstream lancelot fix. Thanks to the rest of the FLARE team for their library recommendations and guidance. This project would not have happened without them.