Ollama

Ollama runs open models locally and exposes an OpenAI-compatible API, so you can point LibreChat at your own machine. Download models with ollama run <model> and browse what's available in the Ollama Library.

Configuration

Ollama ignores the API key but still expects the field to be present, so set it to any placeholder. Point baseURL at your Ollama server. Add the endpoint under endpoints.custom in your librechat.yaml:

    - name: "Ollama"
      apiKey: "ollama"
      # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
      baseURL: "http://localhost:11434/v1/" 
      models:
        default: [
          "llama2",
          "mistral",
          "codellama",
          "dolphin-mixtral",
          "mistral-openorca"
          ]
        # fetching list of models is supported but the `name` field must start
        # with `ollama` (case-insensitive), as it does in this example.
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      modelDisplayLabel: "Ollama"

Notes

Set titleModel to "current_model" so title generation reuses the conversation's model instead of loading a second one. This keeps Ollama to a single loaded model per conversation.
The default array above is a sample list of popular models. With fetch: true, LibreChat pulls the full list from your server.

Ollama -> llama3

Once stop was removed from the default parameters, the issue below should no longer occur.

If llama3 keeps generating without stopping, add an addParams block with the stop sequences:

    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/"
      models:
        default: [
          "llama3"
        ]
        fetch: false # fetching list of models is not supported
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      modelDisplayLabel: "Ollama"
      addParams:
          "stop": [
              "<|start_header_id|>",
              "<|end_header_id|>",
              "<|eot_id|>",
              "<|reserved_special_token"
          ]

If you only run llama3 with Ollama, setting stop at the config level via addParams is fine. When you run several models, add stop sequences from the frontend through conversation parameters and presets instead, and omit addParams:

    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://host.docker.internal:11434/v1/" 
      models:
        default: [
          "llama3:latest",
          "mistral"
          ]
        fetch: false # fetching list of models is not supported
      titleConvo: true
      titleModel: "current_model"
      modelDisplayLabel: "Ollama"

Set the stop sequences in conversation parameters (and save them as a preset):

Ollama

Configuration

Notes

On this page