Thursday, 10 July 2025

Mac Mini for LLMs Content

A little research into using Macs for running LLMs locally.

• Number of GPU cores matter for the Prompt Processing stage
• Memory Bandwidth matters for the Token Generation stage

For short conversations, ~20% of the processing would be for prompt processing and 80% for the token generation response.

Going by this summary, despite being 5 years old, the M1 Max and Ultra are still competitive for token generation (400GB/s and 800GB/s), matching the M2 Max & Ultra, M3 Max & Ultra while barely being surpassed by the M4 Max & Ultra (410GB/s & 820GB/s).

While memory bandwidth determines how fast you can run models, RAM size determines which models you can run at all.

Modern LLMs require substantial memory - a 7B parameter model needs roughly 14GB RAM for full precision, or 7GB with 8-bit quantization. The M1/M2 base models with 8GB RAM can only handle smaller quantized models, while the 32GB+ configurations of Max and Ultra variants can comfortably run 13B models and even some 30B models with aggressive quantization.

For local LLM work, that older M1 Max or Ultra might actually be the better buy than the latest M4 - especially on the used market where you can get serious memory bandwidth and capacity for much less.

However, NVIDIA chips still offer far faster memory bandwidth, but also require much more power.

• Mac M1 Max - 400GB/s
• Mac M4 Max - 410GB/s
• NVIDIA 4080 - 717GB/s
• NVIDIA 3080 - 760GB/s
• Mac M1 Ultra - 800GB/s
• Mac M4 Ultra - 810GB/s
• NVIDIA 3090 - 936GB/s
• NVIDIA 5080 - 960GB/s
• NVIDIA 4090 - 1,018GB/s
• NVIDIA 5090 - 1,792GB/s