| @@ -28,7 +28,7 @@ This submission is split into two parts. The pre-GSoC work below landed before t | |||
| 28 | 28 | ||
| 29 | 29 | ## Project status | |
| 30 | 30 | ||
| 31 | - | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master. A source-level review of that merge surfaced ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)), and I addressed the fixable ones in [#1376](https://github.com/mandiant/flare-floss/pull/1376). The remaining product decisions are tracked as issues I opened: the viewer tag filter ([#1377](https://github.com/mandiant/flare-floss/issues/1377)), the version bump and flag aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)), and the default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)). Both the merge ([#1372](https://github.com/mandiant/flare-floss/pull/1372)) and the fixes ([#1376](https://github.com/mandiant/flare-floss/pull/1376)) are open at the time of writing. | |
| 31 | + | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master. The remaining product decisions are tracked as issues I opened: the [viewer tag filter in #1377](https://github.com/mandiant/flare-floss/issues/1377), the [version bump and flag aliases in #1378](https://github.com/mandiant/flare-floss/issues/1378), and the [default deobfuscation behavior in #1379](https://github.com/mandiant/flare-floss/issues/1379). Both the merge ([#1372](https://github.com/mandiant/flare-floss/pull/1372)) and the fixes ([#1376](https://github.com/mandiant/flare-floss/pull/1376)) are open at the time of writing. | |
| 32 | 32 | ||
| 33 | 33 | ## Pre-GSoC work | |
| 34 | 34 | ||
| @@ -155,7 +155,6 @@ What to do next, roughly in order: | |||
| 155 | 155 | - Version bump and `--only`/`--no` aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)) | |
| 156 | 156 | - Default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)) | |
| 157 | 157 | ||
| 158 | - | Once those are decided, the branch rebases and merges. | |
| 159 | 158 | - **Ship the web viewer.** Decide how it distributes: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 160 | 159 | - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. | |
| 161 | 160 | - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. | |
vee1e / d561fcadc2627cc3dcb115d18847a236
Last active 4 days ago
Lakshit Verma revised this gist 2 weeks ago · e17c455
1 file changed, 1 insertion, 2 deletions
| @@ -150,7 +150,12 @@ Part of the work was reviewing others. The notable PRs I was asked to review: | |||
| 150 | 150 | ||
| 151 | 151 | What to do next, roughly in order: | |
| 152 | 152 | ||
| 153 | - | - **Finish the merge of QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open. A review of it filed ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)); six are fixed in [#1376](https://github.com/mandiant/flare-floss/pull/1376), and the rest need product decisions I tracked as issues: viewer tag filtering ([#1377](https://github.com/mandiant/flare-floss/issues/1377)), the version bump and `--only`/`--no` aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)), and the default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)). Once decided, the branch rebases and merges. | |
| 153 | + | - **Finish the merge of QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open. A review of it filed ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)); six are fixed in [#1376](https://github.com/mandiant/flare-floss/pull/1376). The rest need product decisions I tracked as issues: | |
| 154 | + | - Viewer tag filtering ([#1377](https://github.com/mandiant/flare-floss/issues/1377)) | |
| 155 | + | - Version bump and `--only`/`--no` aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)) | |
| 156 | + | - Default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)) | |
| 157 | + | ||
| 158 | + | Once those are decided, the branch rebases and merges. | |
| 154 | 159 | - **Ship the web viewer.** Decide how it distributes: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 155 | 160 | - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. | |
| 156 | 161 | - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. | |
Lakshit Verma revised this gist 2 weeks ago · c1dd9c0
No changes
| @@ -28,7 +28,7 @@ This submission is split into two parts. The pre-GSoC work below landed before t | |||
| 28 | 28 | ||
| 29 | 29 | ## Project status | |
| 30 | 30 | ||
| 31 | - | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master. A source-level review of that merge surfaced ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)), and I addressed the fixable ones in [#1376](https://github.com/mandiant/flare-floss/pull/1376). Both the merge ([#1372](https://github.com/mandiant/flare-floss/pull/1372)) and the fixes ([#1376](https://github.com/mandiant/flare-floss/pull/1376)) are open at the time of writing. | |
| 31 | + | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master. A source-level review of that merge surfaced ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)), and I addressed the fixable ones in [#1376](https://github.com/mandiant/flare-floss/pull/1376). The remaining product decisions are tracked as issues I opened: the viewer tag filter ([#1377](https://github.com/mandiant/flare-floss/issues/1377)), the version bump and flag aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)), and the default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)). Both the merge ([#1372](https://github.com/mandiant/flare-floss/pull/1372)) and the fixes ([#1376](https://github.com/mandiant/flare-floss/pull/1376)) are open at the time of writing. | |
| 32 | 32 | ||
| 33 | 33 | ## Pre-GSoC work | |
| 34 | 34 | ||
| @@ -150,7 +150,7 @@ Part of the work was reviewing others. The notable PRs I was asked to review: | |||
| 150 | 150 | ||
| 151 | 151 | What to do next, roughly in order: | |
| 152 | 152 | ||
| 153 | - | - **Finish the merge of QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open. A review of it filed ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)); six are fixed in [#1376](https://github.com/mandiant/flare-floss/pull/1376), and the rest need product decisions from the maintainers before the branch rebases and merges. | |
| 153 | + | - **Finish the merge of QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open. A review of it filed ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)); six are fixed in [#1376](https://github.com/mandiant/flare-floss/pull/1376), and the rest need product decisions I tracked as issues: viewer tag filtering ([#1377](https://github.com/mandiant/flare-floss/issues/1377)), the version bump and `--only`/`--no` aliases ([#1378](https://github.com/mandiant/flare-floss/issues/1378)), and the default deobfuscation behavior ([#1379](https://github.com/mandiant/flare-floss/issues/1379)). Once decided, the branch rebases and merges. | |
| 154 | 154 | - **Ship the web viewer.** Decide how it distributes: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 155 | 155 | - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. | |
| 156 | 156 | - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. | |
| @@ -28,7 +28,7 @@ This submission is split into two parts. The pre-GSoC work below landed before t | |||
| 28 | 28 | ||
| 29 | 29 | ## Project status | |
| 30 | 30 | ||
| 31 | - | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master, which is open as [#1372](https://github.com/mandiant/flare-floss/pull/1372) at the time of writing. | |
| 31 | + | The work is essentially complete. QS was refactored into first-class FLOSS modules, the unified pipeline is the default output path, the new command line is merged, caching ships with the tool, and the web viewer is merged. The final step is the merge of the QuantumStrand branch into master. A source-level review of that merge surfaced ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)), and I addressed the fixable ones in [#1376](https://github.com/mandiant/flare-floss/pull/1376). Both the merge ([#1372](https://github.com/mandiant/flare-floss/pull/1372)) and the fixes ([#1376](https://github.com/mandiant/flare-floss/pull/1376)) are open at the time of writing. | |
| 32 | 32 | ||
| 33 | 33 | ## Pre-GSoC work | |
| 34 | 34 | ||
| @@ -150,7 +150,7 @@ Part of the work was reviewing others. The notable PRs I was asked to review: | |||
| 150 | 150 | ||
| 151 | 151 | What to do next, roughly in order: | |
| 152 | 152 | ||
| 153 | - | - **Merge QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open and green, awaiting final review. This is the last gate. | |
| 153 | + | - **Finish the merge of QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open. A review of it filed ten issues ([#1375](https://github.com/mandiant/flare-floss/issues/1375)); six are fixed in [#1376](https://github.com/mandiant/flare-floss/pull/1376), and the rest need product decisions from the maintainers before the branch rebases and merges. | |
| 154 | 154 | - **Ship the web viewer.** Decide how it distributes: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 155 | 155 | - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. | |
| 156 | 156 | - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. | |
| @@ -146,12 +146,17 @@ Part of the work was reviewing others. The notable PRs I was asked to review: | |||
| 146 | 146 | - [#1366](https://github.com/mandiant/flare-floss/pull/1366): summary view revamp. I caught that the new filtering let untagged generic strings crowd out tagged interesting ones, and flagged now-dead code, before it merged. | |
| 147 | 147 | - [#1273](https://github.com/mandiant/flare-floss/pull/1273), [#1294](https://github.com/mandiant/flare-floss/pull/1294), [#1328](https://github.com/mandiant/flare-floss/pull/1328), [#1240](https://github.com/mandiant/flare-floss/pull/1240): robustness, performance, and cleanup reviews. | |
| 148 | 148 | ||
| 149 | - | ## What is left to do | |
| 149 | + | ## Future work | |
| 150 | + | ||
| 151 | + | What to do next, roughly in order: | |
| 150 | 152 | ||
| 151 | 153 | - **Merge QuantumStrand into master.** [#1372](https://github.com/mandiant/flare-floss/pull/1372) is open and green, awaiting final review. This is the last gate. | |
| 152 | - | - **Web viewer distribution.** Decide how the viewer ships: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 153 | - | - **Documentation.** The usage guide does not yet document the new flags, the summary output, or the cache environment variables. | |
| 154 | - | - **Cache maintenance.** A v1 tradeoff: there is no eviction or quota for the cache directory yet, and very large result documents load fully into memory. | |
| 154 | + | - **Ship the web viewer.** Decide how it distributes: inside the released binary, served from the tool, or as a standalone site. Add a deploy workflow so it updates automatically. | |
| 155 | + | - **Document the new interface.** The usage guide does not yet cover the new flags, the summary output, or the cache environment variables. | |
| 156 | + | - **Grow the cache.** A v1 tradeoff: no eviction or quota on the cache directory yet, and very large result documents load fully into memory. | |
| 157 | + | - **Move string extraction in-house.** The pipeline still leans on lancelot; replacing it with native FLOSS extraction would remove an unmaintained dependency. | |
| 158 | + | - **Widen the viewer's reach.** The interactive viewer is CLI-oriented; adding agent-friendly entry points like an llms.txt page would make the tool usable through AI browsing tools. | |
| 159 | + | - **Expand database coverage.** More languages and runtime sub-databases (the Go and Rust ones landed) would tighten noise filtering further. | |
| 155 | 160 | ||
| 156 | 161 | None of these block the core functionality. The new pipeline, CLI, and cache are all merged and working. | |
| 157 | 162 | ||
Lakshit Verma revised this gist 2 weeks ago · 7f741ba
1 file changed, 2 deletions
| @@ -159,8 +159,6 @@ None of these block the core functionality. The new pipeline, CLI, and cache are | |||
| 159 | 159 | ||
| 160 | 160 | The largest single risk was the big refactor in [#1337](https://github.com/mandiant/flare-floss/pull/1337). Moving 4,000 lines around risks quietly changing behavior. We addressed it with evidence instead of argument: a parity harness ran old and new code over a thousand random samples, and every output matched. That became the review bar for the rest of the project. | |
| 161 | 161 | ||
| 162 | - | The database pipeline surfaced real bugs in upstream lancelot, and fixing them became a theme of the project: a crash on weak C++ symbols ([#235](https://github.com/williballenthin/lancelot/issues/235), fixed in [PR #237](https://github.com/williballenthin/lancelot/pull/237)), fragile CSV output replaced with JSONL ([#238](https://github.com/williballenthin/lancelot/issues/238), fixed in [PR #239](https://github.com/williballenthin/lancelot/pull/239)), and a deprecated zydis build config ([#240](https://github.com/williballenthin/lancelot/issues/240), fixed by Willi in [PR #241](https://github.com/williballenthin/lancelot/pull/241)). | |
| 163 | - | ||
| 164 | 162 | The planning document was the other hard part. A discussion doc and an implementation spec are different things, and it took a while to realize we needed the latter. Rewriting it as a spec that could be handed to coding agents unblocked the final months of the project. It also became the design reference for the new CLI and the caching module. | |
| 165 | 163 | ||
| 166 | 164 | ## Acknowledgements | |
Lakshit Verma revised this gist 2 weeks ago · 74ba310
1 file changed, 1 insertion, 1 deletion
| @@ -165,4 +165,4 @@ The planning document was the other hard part. A discussion doc and an implement | |||
| 165 | 165 | ||
| 166 | 166 | ## Acknowledgements | |
| 167 | 167 | ||
| 168 | - | Thanks to my mentor Moritz Raabe for the straight talk and the support, and for reviewing far more than his share of large diffs. Thanks to Willi Ballenthin for the sharp reviews and for merging the upstream lancelot fix. Thanks to the rest of the FLARE team for their library recommendations and guidance. This project would not have happened without them. | |
| 168 | + | Thanks to my mentor Moritz Raabe for the straight talk and the support, and for reviewing far more than his share of large diffs. Thanks to Willi Ballenthin for the sharp secondary reviews. Finally, thanks to the rest of the FLARE team for their work in creating FLOSS. This project would not have happened without them. | |
Lakshit Verma revised this gist 2 weeks ago · 146eeb5
3 files changed, 1 insertion, 2 deletions
Binary file changes are not shown
| @@ -1,6 +1,5 @@ | |||
| 1 | 1 | <p align="center"> | |
| 2 | - | <img src="https://gist.githubusercontent.com/vee1e/d561fcadc2627cc3dcb115d18847a236/raw/gsoc-logo.png" width="120" style="vertical-align: middle;"> | |
| 3 | - | <img src="https://gist.githubusercontent.com/vee1e/d561fcadc2627cc3dcb115d18847a236/raw/flare-logo.png" width="220" style="vertical-align: middle;"> | |
| 2 | + | <img height="220" alt="gsoc-logo" src="https://gist.github.com/user-attachments/assets/5fcbd2b1-625f-4904-9f81-0b0c828bdd23" style="vertical-align: middle;"> | |
| 4 | 3 | </p> | |
| 5 | 4 | ||
| 6 | 5 | <div align="center"> | |
Binary file changes are not shown
| @@ -160,7 +160,7 @@ None of these block the core functionality. The new pipeline, CLI, and cache are | |||
| 160 | 160 | ||
| 161 | 161 | The largest single risk was the big refactor in [#1337](https://github.com/mandiant/flare-floss/pull/1337). Moving 4,000 lines around risks quietly changing behavior. We addressed it with evidence instead of argument: a parity harness ran old and new code over a thousand random samples, and every output matched. That became the review bar for the rest of the project. | |
| 162 | 162 | ||
| 163 | - | The automated database pipeline surfaced real bugs in the upstream lancelot string extractor, and fixing them became a theme of the project. A crash on weak external C++ symbols ([issue #235](https://github.com/williballenthin/lancelot/issues/235), fixed in [PR #237](https://github.com/williballenthin/lancelot/pull/237)), fragile CSV output that broke on special characters ([issue #238](https://github.com/williballenthin/lancelot/issues/238), fixed with JSONL output in [PR #239](https://github.com/williballenthin/lancelot/pull/239)), and a deprecated zydis build config ([issue #240](https://github.com/williballenthin/lancelot/issues/240), fixed by Willi in [PR #241](https://github.com/williballenthin/lancelot/pull/241)). This is exactly the kind of problem a maintainer wants contributors to solve at the root. | |
| 163 | + | The database pipeline surfaced real bugs in upstream lancelot, and fixing them became a theme of the project: a crash on weak C++ symbols ([#235](https://github.com/williballenthin/lancelot/issues/235), fixed in [PR #237](https://github.com/williballenthin/lancelot/pull/237)), fragile CSV output replaced with JSONL ([#238](https://github.com/williballenthin/lancelot/issues/238), fixed in [PR #239](https://github.com/williballenthin/lancelot/pull/239)), and a deprecated zydis build config ([#240](https://github.com/williballenthin/lancelot/issues/240), fixed by Willi in [PR #241](https://github.com/williballenthin/lancelot/pull/241)). | |
| 164 | 164 | ||
| 165 | 165 | The planning document was the other hard part. A discussion doc and an implementation spec are different things, and it took a while to realize we needed the latter. Rewriting it as a spec that could be handed to coding agents unblocked the final months of the project. It also became the design reference for the new CLI and the caching module. | |
| 166 | 166 | ||