LibreChatVultr Cloud Inference
Configure Vultr Cloud Inference as a custom endpoint in LibreChat.
Vultr Cloud Inference serves open models through an OpenAI-compatible API.
Get an API key
Create a key from the Vultr Cloud Inference console. Add it to your .env file:
VULTRINFERENCE_TOKEN=your-api-keyConfiguration
Add the endpoint under endpoints.custom in your librechat.yaml:
- name: 'Vultr Cloud Inference'
apiKey: '${VULTRINFERENCE_TOKEN}'
baseURL: 'https://api.vultrinference.com/v1/chat/completions'
models:
default: [
"llama2-7b-chat-Q5_K_M.gguf",
"llama2-13b-chat-Q5_K_M.gguf",
"mistral-7b-Q5_K_M.gguf",
"zephyr-7b-beta-Q5_K_M.gguf",
]
fetch: true
titleConvo: true
titleModel: "llama2-7b-chat-Q5_K_M.gguf"
modelDisplayLabel: "Vultr Cloud Inference"Notes
- The example lists four models optimized for chat, last updated June 28, 2024.
- Only
llama2-7b-chat-Q5_K_M.ggufcurrently supports title generation.
How is this guide?