Apple MLX

Apple MLX serves models locally on Apple silicon through an OpenAI-compatible API, so you can point LibreChat at your own machine.

Configuration

The local MLX server doesn't authenticate requests, so the API key is just a placeholder. Point baseURL at your running server. Add the endpoint under endpoints.custom in your librechat.yaml:

    - name: "MLX"
      apiKey: "mlx"
      baseURL: "http://localhost:8080/v1/" 
      models:
        default: [
          "Meta-Llama-3-8B-Instruct-4bit"
          ]
        fetch: false # fetching list of models is not supported
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      modelDisplayLabel: "Apple MLX"
      addParams:
            max_tokens: 2000
            "stop": [
              "<|eot_id|>"
            ]

Notes

• The MLX server runs one model at a time. To serve more than one model, run a separate instance on a different port and add another endpoint with its own baseURL.
• The API is strict about unrecognized parameters, so keep addParams limited to values the server accepts, such as max_tokens and stop.