LibreChatHuggingface
Configure Huggingface as a custom endpoint in LibreChat.
Huggingface exposes hosted models through an OpenAI-compatible inference API, which you can add to LibreChat as a custom endpoint.
Get an API key
Create a token at huggingface.co/settings/tokens. Add it to your .env file:
HUGGINGFACE_TOKEN=your-api-keyConfiguration
Add the endpoint under endpoints.custom in your librechat.yaml:
- name: 'HuggingFace'
apiKey: '${HUGGINGFACE_TOKEN}'
baseURL: 'https://api-inference.huggingface.co/v1'
models:
default: [
"codellama/CodeLlama-34b-Instruct-hf",
"google/gemma-1.1-2b-it",
"google/gemma-1.1-7b-it",
"HuggingFaceH4/starchat2-15b-v0.1",
"HuggingFaceH4/zephyr-7b-beta",
"meta-llama/Meta-Llama-3-8B-Instruct",
"microsoft/Phi-3-mini-4k-instruct",
"mistralai/Mistral-7B-Instruct-v0.1",
"mistralai/Mistral-7B-Instruct-v0.2",
"mistralai/Mixtral-8x7B-Instruct-v0.1",
"NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO",
]
fetch: true
titleConvo: true
titleModel: "NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO"
dropParams: ["top_p"]
modelDisplayLabel: "HuggingFace"The model list above was last updated on May 09, 2024.
Notes
- The listed models are free but rate limited, and answers can be very short on the free tier. Some models work better than others.
- Fetching the model list is not supported, so set the
defaultarray yourself. dropParams: ["top_p"]is required. Without it, requests fail because Huggingface rejects thetop_pparameter. SeedropParams.
How is this guide?