Running Local LLMs with RamaLama and Docker on a Mac: A Hands-On Guide

RamaLama runs large language models as OCI containers, so a single command (ramalama run smollm:135m) pulls a model and starts talking to it, with no Python environment to babysit. I spent an afternoon putting it through its paces on an Apple Silicon Mac (Apple M4 Pro, 48 GB RAM, macOS 26.6) with Docker 29.4 provided by OrbStack. This guide is what I actually saw: the install, the first model, an OpenAI-compatible server, and the one macOS-specific catch that isn't obvious from the docs. Every command and number below is from that run, on RamaLama 0.24.0.

What is RamaLama?

RamaLama is an open-source CLI from the container-tooling community that treats models like container images. Instead of assembling an inference stack yourself, it pulls a hardened OCI image containing llama.cpp (or vLLM/MLX) plus your chosen model and runs it with Podman or Docker. If you've used Ollama the ergonomics feel familiar (run, serve, list, pull), but the runtime and model live inside containers you can inspect and sign, and weights come straight from Hugging Face, Ollama, or any OCI registry.

Installing RamaLama on macOS

With Homebrew it's one command:

brew install ramalama

That pulled RamaLama 0.24.0 and, notably, its own copy of llama.cpp, ggml, and libomp as dependencies. Hold onto that detail; it matters for GPU acceleration later. Confirm the install:

ramalama version
# ramalama version 0.24.0

You also need a container engine running. I used Docker through OrbStack; Podman works too and is RamaLama's default on Linux.

Running your first model

The headline command:

ramalama run smollm:135m "In one sentence, what is a Linux container?"

Passing a prompt as an argument gives you one-shot output instead of dropping into a chat REPL. On first run this pulled the RamaLama container image, downloaded the model, and answered. smollm:135m resolves to hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF, a 138 MB, 8-bit quantized GGUF from Hugging Face.

First-run wall-clock was 2 minutes 56 seconds, but almost all of that was downloads (the ~1 GB image plus the model); the 135M model itself is near-instant on CPU. It is also not smart: asked about containers it invented "2048-bit containers" and a docker-compose up -v command that doesn't exist. That's expected at 135M parameters. Use a model this small to validate your setup, not to do real work; a 1B model like llama3.2:1b (a 770 MB Q4_K_M download) answers the same question correctly. (For which models are actually worth running today, see the open-weight coding leaderboard shake-up.)

Check what you've downloaded:

ramalama list
# SHORTNAME    NAME                                                    SIZE
# llama3.2:1b  hf://bartowski/Llama-3.2-1B-Instruct-GGUF               770.28 MB
# smollm:135m  hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF  138.1 MB

Models live under ~/.local/share/ramalama, separate from your container images.

What RamaLama actually runs

Before running anything for real, --dryrun prints the exact command without executing it:

ramalama --dryrun run smollm:135m "hi"

On my Mac that expands to a hardened docker run:

docker run ... --security-opt=label=disable --cap-drop=all \
  --security-opt=no-new-privileges --pull always -d -p 8080:8080 \
  --init quay.io/ramalama/ramalama:0.24 \
  llama-server --host :: --port 8080 --model /path/to/model --threads 7 ...

Note what it does by default: drops all Linux capabilities, disables privilege escalation, and starts llama-server, the same server that backs the OpenAI-compatible API below. The base image (quay.io/ramalama/ramalama:0.24) is about 1 GB, downloaded once and reused.

The macOS gotcha: containers run on the CPU

Here is the part that trips people up. On Apple Silicon, a model running inside a Linux container cannot reach the Mac's GPU, because Docker's Linux VM has no path to Metal. ramalama info reports the container engine's accelerator as none:

{ "Accelerator": "none", "Config": { "runtimes": { "llama_cpp": {}, "mlx": {} } } }

So the default containerized run is CPU-only. Fine for a 135M toy, painful for anything larger. The fix is --nocontainer, which runs the host's llama.cpp (the copy Homebrew installed) directly:

ramalama --nocontainer serve -p 8081 llama3.2:1b

I benchmarked the difference on the same model and prompt. Served natively, Llama-3.2-1B (Q4_K_M) loads straight onto the Apple GPU. Its startup log shows:

load_tensors: offloaded 17/17 layers to GPU
ggml_metal_init: found device: Apple M4 Pro

and it generated at ~206 tokens/sec. The same model served in the default container has no GPU to offload to and ran at ~102 tokens/sec on the CPU, about half the speed on this M4 Pro. RamaLama also exposes an mlx runtime if you'd rather use Apple's own inference framework than llama.cpp.

The trade-off is genuine: containers give you isolation and reproducibility; native gives you the GPU. On a Mac doing real work, --nocontainer is usually what you want. On Linux with an NVIDIA GPU, the container path keeps both.

Serving an OpenAI-compatible API

This is where RamaLama earns its place. serve starts the same llama-server as a local endpoint:

ramalama serve -d --name tdm-lab -p 8080 smollm:135m

It speaks the OpenAI API, so anything that talks to OpenAI can point at it:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"smollm","messages":[{"role":"user","content":"Say hello."}]}'

The response is standard OpenAI JSON: choices[].message.content, a usage block, and timings. Swap the base URL in your existing OpenAI client and your app runs locally with no code changes. Stop it when done:

ramalama stop tdm-lab

If you'd rather wire a local endpoint into your editor, the same idea powers our guide on connecting Copilot Chat to a local API.

So, is RamaLama worth it?

If you already live in containers, yes. Its strengths are the security defaults (cap-drop, no-new-privileges, signed OCI images), pulling from Hugging Face, Ollama, and OCI registries interchangeably, and the zero-friction OpenAI server. If you just want the fastest local chat on a Mac with a GUI, LM Studio is gentler. The two aren't mutually exclusive: I keep LM Studio for exploring and RamaLama for scripting reproducible, servable model runs.

Plan for two things before you graduate from the toy model. Pick a real quantized model that fits your RAM (a 7–8B Q4 model wants roughly 6–8 GB free), and on a Mac decide up front whether you're optimizing for isolation (container, CPU) or speed (native, Apple GPU).

Key takeaways

• Install: brew install ramalama (bundles llama.cpp); needs Docker or Podman running.
• Run: ramalama run <model> "prompt" for one-shot output; models come from Hugging Face, Ollama, or OCI registries.
• Inspect first: ramalama --dryrun run <model> prints the exact hardened docker run.
• macOS catch: containerized runs are CPU-only; --nocontainer offloads to the Apple GPU (Metal). On this M4 Pro, Llama-3.2-1B ran ~206 tok/s native versus ~102 tok/s in-container, about 2x faster.
• Serve: ramalama serve exposes a drop-in OpenAI-compatible API on port 8080.
• Tested with RamaLama 0.24.0 on macOS 26.6 (Apple M4 Pro, 48 GB), Docker 29.4 via OrbStack; models smollm:135m and llama3.2:1b (Q4_K_M).