Anyscale

Anyscale Endpoints serves open models such as Llama through an OpenAI-compatible API that you can add to LibreChat as a custom endpoint.

Get an API key

Create a key from your Anyscale credentials. Add it to your .env file:

ANYSCALE_API_KEY=your-api-key

Configuration

Add the endpoint to the custom list under endpoints in your librechat.yaml. If your file already has an endpoints.custom list, paste only the - name: entry into it, keeping the same indentation:

endpoints:
  custom:
    - name: "Anyscale"
      apiKey: "${ANYSCALE_API_KEY}"
      baseURL: "https://api.endpoints.anyscale.com/v1"
      models:
        default: [
          "meta-llama/Llama-2-7b-chat-hf",
          ]
        fetch: true
      titleConvo: true
      titleModel: "meta-llama/Llama-2-7b-chat-hf"
      modelDisplayLabel: "Anyscale"

Notes

• With fetch: true, LibreChat loads the available model list from Anyscale, so the default array is only the initial selection.