gsoc-logo

# Extend FLOSS to use the rendering techniques pioneered by QUANTUMSTRAND
**Organization:** FLARE / Mandiant **Contributor:** Lakshit Verma ([@vee1e](https://github.com/vee1e)) **Mentor:** Moritz Raabe ([@mr-tz](https://github.com/mr-tz)) **Repository:** [mandiant/flare-floss](https://github.com/mandiant/flare-floss) ## Project overview FLOSS is the FLARE team's command line tool for extracting obfuscated and plaintext strings from malware. An analyst runs it against a binary and gets the readable text the malware uses: URLs, registry paths, API names, and more. That output is useful but bare: it is a flat list of strings with no context. QuantumStrand (QS) is an experimental branch of FLOSS that adds context. It parses the binary's structure, splits it into sections and regions, and tags each string with extra meaning: which library it likely comes from, whether it is common noise, which API family it belongs to. It also renders results in a hierarchical tree instead of a flat list. The goal of this project was to take the best of QS and make it the default behavior of FLOSS, so every user gets layout-aware, tag-enriched output without extra flags. The project also covered the supporting infrastructure: automated string database builds, a redesigned command line, result caching, and a web viewer. This submission is split into two parts. The pre-GSoC work below landed before the coding period started in June, mostly while I was getting familiar with the QS branch. The coding-period deliverables in the second half are the core of the GSoC project. ## Project status **The project is fully complete.** QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. A source-level review of the merge [surfaced ten issues](https://github.com/mandiant/flare-floss/issues/1375), and I [fixed the code issues](https://github.com/mandiant/flare-floss/pull/1376). **QuantumStrand was then [merged into master](https://github.com/mandiant/flare-floss/pull/1372), [closing the project issue](https://github.com/mandiant/flare-floss/issues/943).** Three product decisions from the review are still tracked as open issues: the [viewer tag filter](https://github.com/mandiant/flare-floss/issues/1377), the [version bump and flag aliases](https://github.com/mandiant/flare-floss/issues/1378), and the [default deobfuscation behavior](https://github.com/mandiant/flare-floss/issues/1379). ## Pre-GSoC work These PRs landed between March and May, before the coding period. They fixed QS bugs, added format parsing, and covered community bonding tasks like CI hygiene. ### Format and structure parsing - [Mach-O parsing](https://github.com/mandiant/flare-floss/pull/1221): universal (fat) binaries, segments, code signatures, and entitlements. - [PE export table parsing](https://github.com/mandiant/flare-floss/pull/1230): DLL names, exported symbols, and forwarder names. - [PE Rich header detection](https://github.com/mandiant/flare-floss/pull/1231) and [its crash fix](https://github.com/mandiant/flare-floss/pull/1233): Rich header detection plus a fix for empty headers. - [Readable PE resource names](https://github.com/mandiant/flare-floss/pull/1229): human-readable names instead of raw numeric IDs. ### Correctness fixes - [zlib false negatives](https://github.com/mandiant/flare-floss/pull/1217): fixed missing zlib error strings. - [adjacent tag display](https://github.com/mandiant/flare-floss/pull/1225): consolidated the display of adjacent tags in the output. - [decoder scoring](https://github.com/mandiant/flare-floss/pull/1242): honored the `skip_libs` setting in decoder scoring. - [import taint detection](https://github.com/mandiant/flare-floss/pull/1243): tightened import taint detection. - [clean Ctrl+C handling](https://github.com/mandiant/flare-floss/pull/1244): removed the traceback on Ctrl+C. - [Rust stack string crash](https://github.com/mandiant/flare-floss/pull/1259): fixed a crash rendering stack strings from Rust binaries. ### Exploratory and community bonding - [Junk string heuristics prototype](https://github.com/mandiant/flare-floss/pull/1224) (closed): prototyped heuristics for junk strings in Rust binaries; we agreed a regex band-aid was not the right fix. - [pinact workflow](https://github.com/mandiant/flare-floss/pull/1303): verifies GitHub Actions are pinned to commit hashes, reducing supply chain risk. - [stale SHA fix](https://github.com/mandiant/flare-floss/pull/1304): fixed a stale pinned SHA in the build workflow. - [AGENTS.md first draft](https://github.com/mandiant/flare-floss/pull/1301) (closed): a first, over-detailed draft that we later rewrote into the slim version below. ## Coding-period deliverables ### 1. Native ELF parsing - [Native ELF parsing](https://github.com/mandiant/flare-floss/pull/1312): builds a section tree, tags relocation sections so they are not misread as strings, detects executable sections, and handles XOR-decoded ELFs. A test corpus of ELF binaries was added to the testfiles submodule. This closed the format gap left by the pre-GSoC Mach-O and PE work, so layout rendering now works across all three major formats. ### 2. Automated open-source string database builds FLOSS tags strings by matching them against databases of known library strings. Those databases used to be built by hand. This work automated the whole pipeline so they stay fresh. - [Automated database builds](https://github.com/mandiant/flare-floss/pull/1327): a build script and bi-weekly CI workflow that compiles open-source libraries with vcpkg, extracts their strings, converts and deduplicates them into the FLOSS database format, and opens an automated PR with build metrics when anything changes. - [Database-change diffs](https://github.com/mandiant/flare-floss/pull/1334): a readable text diff of database changes, embedded in the auto-generated PR descriptions. - [Workflow dependency fixes](https://github.com/mandiant/flare-floss/pull/1332): dependency and versioning fixes. - [Git LFS migration](https://github.com/mandiant/flare-floss/pull/1309): moved the string databases to Git LFS, saving roughly 24 MB per shallow clone. - The library list was settled at around 60 open-source libraries, chosen by auditing false-positive rates against a global prevalence database built from 150,000 real-world samples. Building the databases surfaced several bugs in the upstream lancelot project, which the pipeline depends on. I filed reproductions and shipped fixes for all of them: - **Weak external symbols crash.** Two common C++ libraries (cryptopp and jsoncpp) triggered a crash on weak external symbols. I [filed the reproduction](https://github.com/williballenthin/lancelot/issues/235) and [shipped the fix](https://github.com/williballenthin/lancelot/pull/237), which Willi merged. - **Fragile CSV output.** The `jh` string extractor emitted hand-rolled CSV that broke on commas, quotes, newlines, and C++ mangled names. I [filed the issue](https://github.com/williballenthin/lancelot/issues/238) and [replaced the CSV with JSONL output](https://github.com/williballenthin/lancelot/pull/239), which Willi merged. - **Deprecated zydis build config.** A stale zydis dependency broke newer CMake versions, which the flare-floss review of my database build surfaced. I [filed the issue](https://github.com/williballenthin/lancelot/issues/240) and Willi [fixed it](https://github.com/williballenthin/lancelot/pull/241). The database builder lives in `scripts/build_oss_db.py` and the automated update workflow in `.github/workflows/build-oss-db.yml`. ### 3. The QS to FLOSS integration This is the core of the project: turning QS's separate package into the default FLOSS pipeline. - [Module split](https://github.com/mandiant/flare-floss/pull/1337): split the monolithic QS module into first-class `floss/layout` and `floss/tags` packages and unified the command line. Roughly 4,000 lines moved and rewritten with no behavior change. - [Unified pipeline](https://github.com/mandiant/flare-floss/pull/1347): made the unified pipeline the default. Every string type flows through the same stages into one `ResultDocument`, and every output mode renders from that document. A thousand-sample parity test proved the refactor did not change behavior: 905 of 905 layout outputs matched upstream QS exactly, and 918 of 918 classic-mode outputs matched master exactly. The pipeline is coordinated in `floss/pipeline.py`. The layout tree, tag engine, and renderers live in `floss/layout`, `floss/tags`, and `floss/render`. The results schema lives in `floss/results.py`. ### 4. The redesigned command line QS had its own set of flags that did not match FLOSS. The spec called for one coherent interface, delivered in three parts. - [Part 1: string type flags](https://github.com/mandiant/flare-floss/pull/1355): symmetrical include and exclude flags for string types (`--string-type` / `--no-string-type`), renamed `--functions` to `--analyze-functions`, removed the manual load flag in favor of automatic results detection, and added runtime timing fields to the results. - [Part 2: filters and summary](https://github.com/mandiant/flare-floss/pull/1360): render-time filters by section, structure, and tag (`--section`, `--structure`, `--tag`, with matching `--no-*` forms), `--query` for regex filtering that preserves the tree, `--max-strings` to cap output, an `--interesting` shortcut that drops noisy tags, a new `--summary` output, and sorted stable JSON with structured JSON errors on stderr. - [Part 3: caching](https://github.com/mandiant/flare-floss/pull/1367): result caching (see below). ### 5. Result caching - [Result caching](https://github.com/mandiant/flare-floss/pull/1367): automatic analysis caching. The cache is keyed by a SHA-256 of the sample bytes plus the FLOSS version, stored in the platform cache directory, and written atomically behind a lock. Filters apply at render time, so cached results respect any new flags. Moritz benchmarked a sample going from about 30 seconds to about 3.5 seconds on cache hit, roughly an 8 to 10x speedup for repeat runs. The module is `floss/cache.py`, with tests in `tests/test_cache.py`. ### 6. The web viewer - [The web viewer](https://github.com/mandiant/flare-floss/pull/1358): reworked the QS web viewer to match the unified `ResultDocument` schema, added light and dark themes, fuzzy search, and virtualized rendering for very large string corpora. Deployed live for iteration during development. The viewer lives in the `viewer/` directory and is now deployed at https://mandiant.github.io/flare-floss/ on every change. Serving it from the standalone binary is tracked as an [open issue](https://github.com/mandiant/flare-floss/issues/1371). ### 7. Reproducibility and CI hygiene (coding period) - [AGENTS.md for AI contributors](https://github.com/mandiant/flare-floss/pull/1308): the slimmed-down AGENTS.md so AI coding tools have the context they need before generating code in this repo. - [Merge preparation](https://github.com/mandiant/flare-floss/pull/1370): deprecated all `qs` / QuantumStrand naming, renamed the viewer, retargeted CI to master, and cleaned up stale branches, preparing the branch for the final merge. ## Timeline of feature work The coding period ran from June through August. This is how the work above landed, in order. **June** - Early June: [AGENTS.md for AI contributors](https://github.com/mandiant/flare-floss/pull/1308) and the [move of the string databases to Git LFS](https://github.com/mandiant/flare-floss/pull/1309). - Mid-June: [native ELF parsing](https://github.com/mandiant/flare-floss/pull/1312) completed the format coverage started before the coding period. **July** - Early July: the [automated string database pipeline](https://github.com/mandiant/flare-floss/pull/1327) landed, followed by [dependency fixes](https://github.com/mandiant/flare-floss/pull/1332) and [database-change diffs](https://github.com/mandiant/flare-floss/pull/1334) in the auto-generated PRs. - Late July: the first big integration step, [splitting QS into first-class modules](https://github.com/mandiant/flare-floss/pull/1337). **August** - Early August: the [unified pipeline](https://github.com/mandiant/flare-floss/pull/1347) became the default output path. - Mid-August: the [new command line in two parts](https://github.com/mandiant/flare-floss/pull/1355) and the [render-time filters and summary output](https://github.com/mandiant/flare-floss/pull/1360). The [web viewer](https://github.com/mandiant/flare-floss/pull/1358) was reworked to match the new results schema. - Late August: [result caching](https://github.com/mandiant/flare-floss/pull/1367) and the [cleanup that deprecated the old `qs` naming](https://github.com/mandiant/flare-floss/pull/1370). The [final merge of QuantumStrand into master](https://github.com/mandiant/flare-floss/pull/1372) then completed the project. ## Reviews I contributed Part of the work was reviewing others. The notable PRs I was asked to review: - [ELF parsing robustness](https://github.com/mandiant/flare-floss/pull/1319): including fallback for corrupt binaries. I checked the new fallback logic and requested a real corrupted test file over a mock. - [Global prevalence database update](https://github.com/mandiant/flare-floss/pull/1326): I asked whether the large size gap between the Go and Rust sub-databases was intentional before approving. - [Summary view revamp](https://github.com/mandiant/flare-floss/pull/1366): I caught that the new filtering let untagged generic strings crowd out tagged interesting ones, and flagged now-dead code, before it merged. - Robustness, performance, and cleanup reviews: [one](https://github.com/mandiant/flare-floss/pull/1273), [two](https://github.com/mandiant/flare-floss/pull/1294), [three](https://github.com/mandiant/flare-floss/pull/1328), and [four](https://github.com/mandiant/flare-floss/pull/1240). ## Future work What to do next, roughly in order: - **Rebuild and verify the OSS string databases.** QuantumStrand is in master now, so the databases get regenerated and validated against the merged code. A [CI fix for the build workflow](https://github.com/mandiant/flare-floss/pull/1384) is in progress. - **Finish the UI release.** The web viewer is already deployed at https://mandiant.github.io/flare-floss/ (deployed on every change via the web-release workflow). What remains is serving it from the standalone binary, tracked as an [open issue](https://github.com/mandiant/flare-floss/issues/1371). - **Resolve the tracked product decisions.** [Viewer tag filtering](https://github.com/mandiant/flare-floss/issues/1377), [version bump and flag aliases](https://github.com/mandiant/flare-floss/issues/1378), and [default deobfuscation behavior](https://github.com/mandiant/flare-floss/issues/1379). - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. - **Move string extraction in-house.** The pipeline still leans on lancelot; replacing it with native FLOSS extraction would remove an unmaintained dependency. - **Widen the viewer's reach.** The interactive viewer is CLI-oriented; adding agent-friendly entry points like an llms.txt page would make the tool usable through AI browsing tools. - **Expand database coverage.** More languages and runtime sub-databases (the Go and Rust ones landed) would tighten noise filtering further. None of these block the core functionality. The new pipeline, CLI, and cache are all merged and working. ## Challenges The largest single risk was [the big refactor](https://github.com/mandiant/flare-floss/pull/1337). Moving 4,000 lines around risks quietly changing behavior. We addressed it with evidence instead of argument: a parity harness ran old and new code over a thousand random samples, and every output matched. That became the review bar for the rest of the project. The planning document was the other hard part. A discussion doc and an implementation spec are different things, and it took a while to realize we needed the latter. Rewriting it as a spec that could be handed to coding agents unblocked the final months of the project. It also became the design reference for the new CLI and the caching module. ## Acknowledgements Thanks to my mentor Moritz Raabe for the straight talk and the support, and for reviewing far more than his share of large diffs. Thanks to Willi Ballenthin for the sharp secondary reviews. Finally, thanks to the rest of the FLARE team for their work in creating FLOSS. This project would not have happened without them.