Groq

Groq runs open models such as Llama and Mixtral on its LPU inference hardware, exposed through an OpenAI-compatible API that you can add to LibreChat as a custom endpoint.

Get an API key

Create a key from the Groq console. Add it to your .env file:

GROQ_API_KEY=your-api-key

Configuration

Add the endpoint under endpoints.custom in your librechat.yaml:

    - name: "groq"
      apiKey: "${GROQ_API_KEY}"
      baseURL: "https://api.groq.com/openai/v1/"
      models:
        default: [
          "llama3-70b-8192",
          "llama3-8b-8192",
          "llama2-70b-4096",
          "mixtral-8x7b-32768",
          "gemma-7b-it",
          ]
        fetch: false
      titleConvo: true
      titleModel: "mixtral-8x7b-32768"
      modelDisplayLabel: "groq"

Notes

• A temperature of 0 is converted to 1e-8. If you hit issues, use a float greater than 0 and up to 2.
• Groq is free but rate limited to 10 queries per minute and 100 per hour.