LibreChatVultr Cloud Inference
Configure Vultr Cloud Inference as a custom endpoint in LibreChat.
Vultr Cloud Inference serves open models through an OpenAI-compatible API.
Get an API key
Create a key from the Vultr Cloud Inference console. Add it to your .env file:
VULTRINFERENCE_TOKEN=your-api-keyConfiguration
Add the endpoint to the custom list under endpoints in your librechat.yaml. If your file already has an endpoints.custom list, paste only the - name: entry into it, keeping the same indentation:
endpoints:
custom:
- name: 'Vultr Cloud Inference'
apiKey: '${VULTRINFERENCE_TOKEN}'
baseURL: 'https://api.vultrinference.com/v1'
models:
default: [
"llama2-7b-chat-Q5_K_M.gguf",
"llama2-13b-chat-Q5_K_M.gguf",
"mistral-7b-Q5_K_M.gguf",
"zephyr-7b-beta-Q5_K_M.gguf",
]
fetch: true
titleConvo: true
titleModel: "llama2-7b-chat-Q5_K_M.gguf"
modelDisplayLabel: "Vultr Cloud Inference"Notes
- The example lists four models optimized for chat, last updated June 28, 2024.
- Only
llama2-7b-chat-Q5_K_M.ggufcurrently supports title generation.
How is this guide?