Running an LLM on a simple Raspberry Pi

undefined 21, 2024

Source: blaess

I attempted to run a Large Language Model (LLM) on a Raspberry Pi 5 with 8GB RAM to see if it could work on a small system with limited or no internet. I chose the Phi-3 model via Ollama, a tool for running LLMs locally. With 8GB RAM, models over 7 billion parameters aren't practical, so I tested the 3.8 billion parameter version. This setup showed it's possible to use LLMs on a Raspberry Pi, albeit with limited capacity for larger models.

Share on