Running a local LLM with llama.cpp: build, download, serve, benchmark

A minimal llama.cpp workflow for running GGUF models on your own machine — from the cmake build to an OpenAI-compatible server and llama-bench.

· 3 min read · #llama-cpp #llm #inference #gguf #local-llm · @deep-learning · 한국어

Here is the minimal workflow I use to run GGUF models on my own hardware without a cloud API. The order is build → download the model → serve → benchmark.

Build

The basic build is two cmake lines.

# build whole
cmake -B build
cmake --build build --config Release

To pull models straight from Hugging Face you need curl enabled. Before passing -DLLAMA_CURL=ON, make sure curl and its development headers are installed on the system.

# for hugging-face, add curl option
# make sure to install curl
sudo apt install curl libcurl4-openssl-dev
cmake -B build -DLLAMA_CURL=ON

If you only need the server binary rather than the whole tree, name the target.

# build subset
cmake -B build build --config Release -t llama-server

Downloading models

GGUF files usually run to several gigabytes, so there are two ways to fetch them.

When cloning the whole repository with git, you need git-lfs for the large files to come along properly.

# for large file download from git
sudo apt install git-lfs
git clone {hf repository}

If you want to pick specific files or directories, huggingface-cli is more convenient.

# or, you can use huggingface-cli
pip install -U "huggingface_hub[cli]"
huggingface-cli download {hf repository name} --local-dir .

Watch memory and core usage during download and loading with htop.

Starting the server

The Python binding llama-cpp-python ships an OpenAI-compatible server directly.

# llama-cpp-python
pip install llama-cpp-python[server]
python3 -m llama_cpp.server --model {model_gguf_path} --host 0.0.0.0

Because the server follows the OpenAI spec, the client can just use the openai SDK. Point base_url at the local server, and put anything in api_key since it is never checked. (Match the port in base_url to whatever port the server actually came up on.)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080",
    api_key="none",
)

stream = client.chat.completions.create(
    model="random",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is your name?"},
    ],
    stream=True,
    temperature=0.9,
    max_tokens=1000,
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

The C++ llama-server built above exposes the same OpenAI-compatible endpoints. Use that one if you would rather run without the Python dependency.

Benchmarking

To measure a model’s throughput (tokens/s and friends), use llama-bench.

build/bin/llama-bench -m {gguf_path}

The built binaries land under build/bin/, so you can point at that path directly.