How an AI Request Travels Through the System

Step 2

Request reaches the OpenAI-compatible API

The API layer is the only entry point; applications never address nodes directly.

Step 3

Controller authenticates and validates

Credentials, model reference and request shape are checked before any work is created.

Step 4

Scheduler evaluates available compute nodes

Node selection uses the current cluster inventory and health state.

Step 5

Controller creates and dispatches the execution workload

A job record is created and an execution plan is dispatched to the selected nodes.

Step 6

Node Agents receive the required work

Each participating agent accepts its assignment and prepares the local runtime.

Step 7

Distributed runtime coordinates inference across nodes

The runtime — not the controller — performs the distributed execution.

Step 8

llama.cpp / model runtime executes inference

GGUF model weights are executed on available GPU/CPU compute.

Step 9

Generated tokens are streamed back

Tokens travel upward through the agents and controller to the API layer.

Step 10

Application receives the AI response

The client consumes a familiar streamed completion response.