Method and evidence
Adaptive File Compression System Using Multi-Level Frequency Analysis & Hybrid Huffman
ByteSize is the working interface for the thesis system. It keeps the project’s Hybrid Huffman pipeline intact and verifies every completed compression round trip with SHA-256.
Pipeline
Three decisions, one lossless stream
- 01
Multi-tier frequency scan
Counts individual bytes and repeated multi-byte structures at several candidate lengths.
- 02
Bit Cost Decision Engine
Compares the encoded cost of structural blocks against literal-byte coding and keeps only useful candidates.
- 03
Hybrid Huffman encoding
Builds a canonical Huffman code over the selected structural blocks and remaining literal bytes, then records enough metadata for exact reconstruction.
Measured evidence
Balanced preset, native backend
Every row below records byte equality and SHA-256 equality for the whole round trip. Sizes are reproducible; timings are machine-dependent and no speed claim is made from a single trial.
| Corpus | Files | Original bytes | AFC bytes | Saved | Source |
|---|---|---|---|---|---|
| Silesia current engine | 12 | 211,938,580 | 87,133,292 | 58.89% | benchmarks/silesia_repaired.csv |
| Silesia AFC 1.3 audit | 12 | 211,938,580 | 91,315,157 | 56.91% | benchmarks/afc_1_3_silesia_native_summary.csv |
| GovDocs1 thread 000 · current engine | 980 | 627,788,678 | 395,305,152 | 37.03% | benchmarks/thread000_external_corpus.csv |
| Repository corpus current engine | 31 | 13,747,990 | 4,743,979 | 65.49% | benchmarks/v9_repo_corpus.csv |
| Canterbury current engine | 11 | 2,810,784 | 789,536 | 71.91% | benchmarks/canterbury_full.csv |
| Canterbury AFC 1.3 audit | 11 | 2,810,784 | 826,154 | 70.61% | benchmarks/afc_1_3_canterbury_native_summary.csv |
The two Silesia rows are the same twelve files, measured twice.
The originals total 211,938,580 bytes in both, so the comparison is exact: the current engine writes
4,181,865 fewer bytes than the AFC 1.3 audit at the Balanced preset (−4.58%), and
8,212,404 fewer at Maximum (60.79% saved) — smaller on every one of the twelve files, not merely on average.
Silesia is regenerated with benchmarks/external_corpus.py, which takes the corpus as a path
because it is separately licensed and not redistributed here; the repository-corpus row is regenerated with
benchmarks/v9_repo_corpus.py and needs no download. The two Canterbury rows are likewise the
same eleven canonical files measured twice: the current engine writes 36,618 fewer bytes
than the AFC 1.3 audit at the Balanced preset (70.61% to 71.91%), smaller on nine of the eleven files and
identical on the other two, larger on none. Canterbury is regenerated the same way Silesia is and is always
measured whole — a partial re-run over only its text files would flatter the result by omitting the
binary members that pull the ratio down.
PDF and DOCX need class-qualified interpretation, and GovDocs1 shows why. Across 980 real government documents the result splits sharply by what is already compressed inside the file. Structured and textual formats reduce well — XLS 68.75% over 62 files, CSV 66.00%, TXT 65.57% over 154 files, HTML 64.40% over 181 files, PostScript 62.87% — while formats whose payloads are already entropy-coded barely move: PDF 12.50% over 200 files, PPT 9.21%, SWF 2.99%, GIF 2.88%, JPEG 1.50%, gzip 0.93%. Of the 980 files, 60 (6.1%) did not shrink at all and were stored verbatim rather than expanded. The corpus is thread 000 exactly as distributed, less one 131,010,734-byte member that exceeds the 100 MB per-file ceiling ByteSize enforces. These broader corpus measurements were conducted at the compression-engine level to characterize behavior across file classes and helped define the deployed system scope; they do not mean that every measured format is accepted by the current web interface. New web compression accepts only PDF, TXT, CSV, TSV, JSON, SQL, XML, HTML, HTM, LOG, PY, JS, C, CPP, H, and HPP. Standalone images, Microsoft Word OOXML files, and other unsupported extensions are rejected. Existing AFC containers that restore DOCX remain decodable for backward compatibility. These results therefore make no general compression-percentage claim for PDF or DOCX: a PDF of uncompressed text streams behaves like the text it contains, and a PDF of JPEGs behaves like the JPEGs. Every file in the broader benchmark — including the ones that did not shrink — was reconstructed byte for byte and SHA-256 verified. The measurements are retained as evidence supporting the deployed scope decisions, not as a claim that every benchmarked format is currently accepted for new compression.