smolmaxxing patches llama.cpp to run 8B and 16B mixture-of-experts models on old iPads and TV boxes with about 1 GB of usable memory. It caches frequently used experts in RAM, reads others from flash and uses chip-specific kernels. The developer reports 5.1 tokens per second for a 16B model on the TV box and 2.6–4.5 on a 2013 iPad Air. Stock llama.cpp cannot load the supplied models, the iPad test uses a jailbreak, and these are the project’s own measurements on those devices.
Back to Tech
smolmaxxing runs large MoE models on roughly 1 GB of usable memory
Flash-streamed weights require custom patches, and reported device-specific results are not universal benchmarks.