Local ModelsMedium importance
The smolmaxxing project runs a 16-billion-parameter model on a TV box and a 2013 iPad with 1 GB of RAM, and a 35-billion one on a phone
smolmaxxing, an open-source project (MIT license) that reached the Hacker News front page, looks for the biggest model each old or cheap device can run and speeds it up with custom kernels, careful memory use and weights read straight from storage. According to numbers measured on the devices, a Xiaomi TV Box S, with about 1 GB of free RAM, runs Qwen3.5-2B at 7.7 tokens per second and Ling-mini-2.0, with 16 billion, at 5.1; a 2013 iPad Air runs the 1-bit Bonsai-1.7B at 24 to 26 tokens per second; and a OnePlus Nord CE 2, from 2022, with about 3 GB free, runs Qwen3.6-35B-A3B at 4.9 to 5.7 tokens per second. The trick is to keep only part of the model in memory and read the rest as needed.
Published in edition no. 27 of Noticiário do Joãozinho ·
Translated by Joãozinho AI from the original Brazilian Portuguese edition. Facts, numbers and sources are the same as in the original; quotes, video titles and source names stay in their original language. Read the original in Portuguese
Original source
https://github.com/hyperionaidev/smolmaxxing