starred/turboquant_plus
1
0 Fork 0
mirror of https://github.com/TheTom/turboquant_plus.git synced 2026-10-22 02:53:13 -05:00
Code Issues Packages Projects Releases Wiki Activity
342 Commits 14 Branches 3 Tags
main
Commit Graph
11 Commits
Author SHA1 Message Date
TheTomClaude Opus 4.6
 		86bcbbe59a Add norm correction to Python prototype quantization path 
PolarQuant.dequantize() now re-normalizes reconstructed rotated
coordinates to unit norm before inverse rotation, matching the
production llama.cpp grp_norm / recon_norm correction.

On Qwen2.5-3B, this drops 4-bit PPL from 19.39 to 8.47 (near-lossless)
and 3-bit from 1369 to 70.6. No regression on TinyLlama or Mistral.

Includes benchmark scripts for TQ vs IsoQuant/RotorQuant comparison
and norm correction A/B testing.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 10:57:38 -05:00
TheTomClaude Opus 4.6
 		deb2d8409d experiment: temporal decay prototype — 3→2 requant VIABLE 
Synthetic (d=128, 1000 vectors):
  turbo3 cos=0.983, decay cos=0.940 (>0.80 threshold)
  Decay within 5% of direct 2-bit — double-quant adds minimal error
  Inner product error 1.76x worse (acceptable for old tokens)

Real Qwen3-1.7B KV tensors:
  3-bit cos=0.985, decay cos=0.949 (>0.75 threshold)
  Layer 0 (early): 0.971 — most decay-tolerant
  Layer 2 (mid): 0.927 — worst case, still viable

Memory savings: 30-34% at 32K-256K context
  256K: 1120 MB → 742 MB (378 MB saved)

Approach: dequant turbo3 → find nearest 2-bit centroid → recompute
norm correction → repack. Codex confirmed zero-signs approach is
broken (biased remap), proper requant is required.

Next: Metal implementation in llama_kv_cache maintenance path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
 2026-03-26 17:29:58 -05:00
TheTomClaude Opus 4.6
 		7e469a0dc9 feat: benchmark results — turbo3/turbo4 vs q8_0/q4_0 on M5 Max #22 
Results on Qwen 3.5 35B-A3B MoE and Qwopus v2 27B:
- turbo3: 4.9× compression, 2.4 tok/s MoE / 1.3 tok/s Dense
- q8_0 baseline: 85.0 tok/s MoE / 17.6 tok/s Dense
- Speed regression: 13-35× (unoptimized Metal shader, tracked in #23)
- Compression ratio target met, speed optimization needed

Codex review corrections applied:
- Fixed methodology description (was inconsistent with actual test params)
- Changed "~35×" to "13-35×" (varies by model)
- Marked quality claims as qualitative/unquantified
- Noted compression ratios are theoretical not empirically measured
- Added missing analysis caveats (single run, no stddev, no warm-up)

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 01:06:59 -05:00
TheTomClaude Opus 4.6
 		6634deebd0 feat: README overhaul, Apache 2.0 license, benchmark script #22 
README updates (per Codex review):
- Fixed overstated "zero accuracy loss" claim — now shows actual metrics
- Added proper llama.cpp fork build instructions with prerequisites
- Added "Plus" branding explanation (post-v1 improvements planned)
- Added prerequisites section (Python, cmake, Xcode)
- Fixed inconsistent build paths (build-turbo vs build)
- Removed unsubstantiated Prince Canuma comparison claim

Added:
- Apache 2.0 LICENSE file (Copyright 2026 Tom Turney)
- benchmarks/benchmark_llama.sh script

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-25 00:09:59 -05:00
TheTomClaude Opus 4.6
 		338e978b09 feat: validate rotation Gaussianization on real KV tensors #18 #20 
Critical experiment: real Qwen3-1.7B K cache has kurtosis=900.
After norm extraction + random rotation: kurtosis=2.9 (Gaussian=3.0).
Std matches 1/sqrt(d) to 6 decimal places. Ratio = 1.000 exactly.

Paper's core mathematical claim is validated on real data.
Gaussian codebook is correct — no per-layer calibration needed (#20 closed).

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:36:34 -05:00
TheTomClaude Opus 4.6
 		4e8443f24e fix: real model validation — DynamicCache API, Phase A results #18 
Fixed HuggingFace DynamicCache extraction (transformers 5.x API).
First real tensor validation on Qwen3-1.7B complete.

Key findings:
- Real KV tensors have kurtosis=918 (K) vs Gaussian=3.0
- Norm extraction critical: norms range 6.8-603.8
- 3.5-bit outlier: cosine 0.945, ratio 8.0× on real tensors
- 4-bit uniform: cosine 0.975 on real tensors
- Attention cosines 0.64-0.82 (lower than synthetic due to heavy tails)

Results posted to issue #18.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:34:03 -05:00
TheTomClaude Opus 4.6
 		c1a10e4a64 feat: input validation, README, real model validation script #12 #18 
- Add input validation to rotation.py: d>=1, pow2 checks on hadamard
  and FWHT, with ValueError messages (#12)
- Add README.md with quick start, results, architecture, roadmap
- Add benchmarks/validate_real_model.py for Phase A validation (#18)
- Close #9, #16 (superseded by #18)

5 new tests for validation guards. 137 total tests passing.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:29:24 -05:00
TheTomClaude Opus 4.6
 		1fa6d79b10 feat: outlier comparison benchmark — matches Prince Canuma's ratios 
2.5-bit: 4.9× compression (Prince: 4.9×) ✅
3.5-bit: 3.8× compression (Prince: 3.8×) ✅

Quality at 3.5-bit: MSE 0.0009, cosine 0.947
Quality at 2.5-bit: MSE 0.0029, cosine 0.856

Pending: real KV tensor validation from llama.cpp NIAH test.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:21:53 -05:00
TheTomClaude Opus 4.6
 		f1c2397680 feat: integration test at real Qwen 3.5 KV cache dimensions #16 
Simulates KV cache compression at Qwen 3.5 27B and 35B-A3B dimensions:
28 layers × 8 KV heads × 128 head_dim × up to 8192 tokens.

Results:
- 3-bit K+V: 2.6× compression, K cosine 0.91, attention cosine 0.98
- 4-bit K+V: 1.9× compression, K cosine 0.96, attention cosine 0.99
- Prince Canuma gets 4.9× at 2.5-bit (uses outlier channel strategy)

Next: implement outlier channel strategy for non-integer bit rates
(2.5-bit, 3.5-bit) to match Prince's compression ratios.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:15:14 -05:00
TheTomClaude Opus 4.6
 		361ea4cab0 feat: add demo benchmark script for quick validation 
Shows single-vector compression, full KV cache compression, and inner
product preservation at 2/3/4-bit quantization levels.

Results:
- 3-bit: 4.9× compression, cosine sim 0.83, IP error 0.02
- 4-bit: 3.8× compression, cosine sim 0.86, IP error 0.01
- KV cache: 4 layers × 8 heads × 512 tokens compressed in 60ms

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 21:05:53 -05:00
TheTomClaude Opus 4.6
 		8fa1d293f9 infra: project scaffold — CI, pyproject, CLAUDE.md, PLAN.md #1 
Set up bare project infrastructure:
- .gitignore, pyproject.toml, setup.py
- GitHub Actions CI with Python 3.10/3.11/3.12 matrix
- CI enforces >95% coverage when tests exist, skips gracefully when empty
- CLAUDE.md with full development workflow rules
- PLAN.md with phased implementation plan
- Empty turboquant/, tests/, benchmarks/ packages

No algorithm code — scaffold only.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-24 20:43:16 -05:00