Apply this guide to your situation
This answer covers “[AI-004] The inference runtime does not detect the GPU”. Read it with the summary and confirm that the symptom and service match what you are experiencing. Available options can vary by plan and account setup.
- Before changing anything, note what you expected, what happened and roughly when. Check whether the issue affects one page or the whole service.
- Save the full error message and note recent changes. Hide passwords, access codes, personal data and keys before sharing screenshots or logs.
- If a step requires administrator access, restarting services or editing server files, stop if you do not administer that server. Send support the service name, error and checks you have already made.
L3 technical diagnosis
User symptoms
The model runs on CPU, takes much longer or the container cannot see CUDA devices.
Observed code or message
`CUDA driver version is insufficient` / `No CUDA devices found`Likely root cause
A missing/incompatible host driver, GPU runtime not assigned to the container, mismatched CUDA libraries or a model without GPU backend can cause silent fallback.
Likely-cause and resolution matrix
Follow a branch only when evidence supports it. A symptom may have more than one cause; do not change layers by guesswork.
01Professional English translation pending.
Why it can happen: Professional English translation pending.
Professional English translation pending.Step-by-step resolution:
Professional English translation pending.
02Professional English translation pending.
Why it can happen: Professional English translation pending.
Professional English translation pending.Step-by-step resolution:
Professional English translation pending.
03Professional English translation pending.
Why it can happen: Professional English translation pending.
Professional English translation pending.Step-by-step resolution:
Professional English translation pending.
04Professional English translation pending.
Why it can happen: Professional English translation pending.
Professional English translation pending.Step-by-step resolution:
Professional English translation pending.
05Professional English translation pending.
Why it can happen: Professional English translation pending.
Professional English translation pending.Step-by-step resolution:
Professional English translation pending.
Prerequisites
- Authorised access to the involved runtime, provider or automation.
- A test environment and synthetic data; never use customer prompts for diagnostics.
- A technical owner who can approve network, secret, GPU or resource changes.
Important warnings
- Do not expose local inference endpoints or webhooks without appropriate authentication, TLS and limits. Never put API keys, private prompts or personal data in commands, screenshots or logs.
- Commands are diagnostic references. Production changes, Docker/firewall rules and credential rotation require service-owner authorisation.
Step-by-step walkthrough
1. Confirm the symptom and scope
nvidia-smi
docker info --format '{{json .Runtimes}}'
docker exec ai-runtime nvidia-smi`CUDA driver version is insufficient` / `No CUDA devices found`
2. Correct the responsible component
3. Verify and close exposure
The check confirms the expected result without exposing services or data.
Troubleshooting & Solutions
`CUDA driver version is insufficient` / `No CUDA devices found`
Root cause: A missing/incompatible host driver, GPU runtime not assigned to the container, mismatched CUDA libraries or a model without GPU backend can cause silent fallback.
Resolution: Ask the administrator to validate GPU model, driver and toolkit compatibility with the published runtime. Grant GPU access in Compose only to the authorised service; schedule a controlled runtime restart after changes and retain diagnostic output.
Rollback and recovery plan
Restore the last approved configuration, stop the affected workflow and revoke any exposed credential. If real data was processed through an unauthorised path, retain minimal redacted records and follow the internal incident-response process.
Related runbooks in this category:
Was this technical guide helpful?
Facing a production outage?
Hosting Anayil system administrators can directly intervene on your server infrastructure.