Loading page
Run inference on CPU, CUDA, ROCm, or Apple MPS.
Workers pick an inference device with the ML_DEVICE environment
variable. The default, ML_DEVICE=auto, tries devices in this order:
CUDA → ROCm → MPS → CPU, falling back to the next option if a device
isn't available.
You can also pin a worker to a specific device explicitly:
ML_DEVICE=cpu, mps, cuda, cuda:N (a specific GPU index), or rocm.
# NVIDIA
docker compose --profile gpu up --build \
--scale worker-model-local=0 \
redis api scheduler frontend worker-cpu worker-gpuThe NVIDIA variant (worker-gpu) requests an NVIDIA GPU reservation and
needs driver 555+. The AMD variant (worker-amd) needs /dev/kfd and
/dev/dri exposed and a ROCm-supported GPU on Linux. Both profiles scale
worker-model-local to 0 so the GPU-capable worker takes over the
gpu-fast/gpu-large queues instead.
ATTENTION_FORCE_CPU defaults to true. It forces the attention operation
onto CPU regardless of ML_DEVICE, working around a GPU Triton segfault.
Leave it at its default unless you've specifically confirmed attention is
stable on GPU in your environment.