LibreChatVultr Cloud Inference
Configure Vultr Cloud Inference as a custom endpoint in LibreChat.
Archived documentation
You are reading the frozen documentation for v0.8.7, which is no longer maintained. Go to the latest docs.
Vultr Cloud Inference serves open models through an OpenAI-compatible API.
Get an API key
Create a key from the Vultr Cloud Inference console. Add it to your .env file:
VULTRINFERENCE_TOKEN=your-api-keyConfiguration
Add the endpoint under endpoints.custom in your librechat.yaml:
- name: 'Vultr Cloud Inference'
apiKey: '${VULTRINFERENCE_TOKEN}'
baseURL: 'https://api.vultrinference.com/v1/chat/completions'
models:
default: [
"llama2-7b-chat-Q5_K_M.gguf",
"llama2-13b-chat-Q5_K_M.gguf",
"mistral-7b-Q5_K_M.gguf",
"zephyr-7b-beta-Q5_K_M.gguf",
]
fetch: true
titleConvo: true
titleModel: "llama2-7b-chat-Q5_K_M.gguf"
modelDisplayLabel: "Vultr Cloud Inference"Notes
- The example lists four models optimized for chat, last updated June 28, 2024.
- Only
llama2-7b-chat-Q5_K_M.ggufcurrently supports title generation.