Web Search Configuration

The webSearch configuration allows you to customize the web search functionality within LibreChat, including search providers, content scrapers, and result rerankers.

Overview

The web search feature consists of three main components:

1. Search Providers: Services that perform the initial web search
2. Scrapers: Services that extract content from web pages
3. Rerankers: Services that reorder search results for better relevance

Example

webSearch:
  # Search Provider Configuration
  serperApiKey: "${SERPER_API_KEY}"
  searxngInstanceUrl: "${SEARXNG_INSTANCE_URL}"
  searxngApiKey: "${SEARXNG_API_KEY}"
  searchProvider: "serper" # Options: "serper", "searxng"
 
  # Scraper Configuration
  firecrawlApiKey: "${FIRECRAWL_API_KEY}"
  firecrawlApiUrl: "${FIRECRAWL_API_URL}"
  scraperType: "firecrawl" # Options: "firecrawl", "serper"
 
  # Reranker Configuration
  jinaApiKey: "${JINA_API_KEY}"
  jinaApiUrl: "${JINA_API_URL}"
  cohereApiKey: "${COHERE_API_KEY}"
  rerankerType: "jina" # Options: "jina", "cohere"
 
  # General Settings
  scraperTimeout: 7500 # Timeout in milliseconds for scraper requests (default: 7500)
  safeSearch: 1 # Options: 0 (OFF), 1 (MODERATE - default), 2 (STRICT)

Search Providers

searchProvider

serperApiKey

Note: Get your API key from Serper.dev

searxngInstanceUrl

searxngApiKey

Note: This is optional and only needed if your SearXNG instance requires authentication.

Scrapers

firecrawlApiKey

Note: Get your API key from Firecrawl.dev

firecrawlApiUrl

Note: This is optional and only needed if you’re using a custom Firecrawl instance.

scraperType

firecrawlOptions

Subkeys:

formats

includeTags

excludeTags

headers

waitFor

timeout

maxAge

Note: If you do not need extremely fresh data, enabling this can speed up your scrapes by 500%.

mobile

skipTlsVerification

blockAds

removeBase64Images

parsePDF

storeInCache

zeroDataRetention

location

onlyMainContent

changeTrackingOptions

Example:

webSearch:
  firecrawlApiKey: "${FIRECRAWL_API_KEY}"
  firecrawlOptions:
    formats: ["markdown", "rawHtml"]
    includeTags: ["main", "article", ".content"]
    excludeTags: ["nav", "footer", ".ads"]
    waitFor: 2000
    timeout: 10000
    mobile: false
    blockAds: true
    onlyMainContent: true
    location:
      country: "US"
      languages: ["en"]

Note: For detailed information about Firecrawl scraper options and defaults, see the Firecrawl API Documentation.

Rerankers

jinaApiKey

Note: Get your API key from Jina.ai

jinaApiUrl

Note: This is optional and only needed if you’re using a custom Jina instance.

cohereApiKey

Note: Get your API key from Cohere Dashboard

rerankerType

General Settings

scraperTimeout

safeSearch

Note: Safe search levels align with standard search API conventions. MODERATE filtering is enabled by default to provide reasonable content filtering while maintaining search effectiveness.

Notes

• API keys can be configured in two ways:
  1. Set the environment variables specified in the YAML configuration
  2. If environment variables are not set, users will be prompted to provide the API keys via the UI
• The configuration supports multiple services for each component (providers, scrapers, rerankers)
• If a specific service type is not specified, the system will try all available services in that category
• Safe search provides three levels of content filtering: OFF (0), MODERATE (1), and STRICT (2)
• Never put actual API keys in the YAML configuration - only use environment variable names

Setting Up SearXNG

SearXNG is a privacy-focused meta search engine that you can self-host. For more information, see the official SearXNG documentation.

Here are the steps to set up your own SearXNG instance for use with LibreChat:

Using Docker Desktop

1. Search for the official SearXNG image

   • Open Docker Desktop
   • Search for searxng/searxng in the Images tab
   • Click Run on the official image to automatically pull down and run a container of the image

2. Running the container

   • Expand the Optional Settings dropdown in the proceeding panel that appears once the download completes
   • Set your desired configuration details (port number, container name, etc.)
   • Click Run to start the container

3. Configure SearXNG for LibreChat

   • Navigate to the Files tab in Docker Desktop
   • Go to /etc/searxng/settings.yaml
   • Open the file editor
   • Navigate to the formats section
   • Add json as an acceptable format so that LibreChat can communicate with your instance
   • Save the file

4. Restart the container

   • Restart the container for the changes to take effect

Video Guide:

Here’s a video to guide you through the process from start to finish in about a minute:

Note: In this example, the instance URL is http://localhost:55011 (port number found under the container name in the top left at the end of the video)

Configuring LibreChat to use SearXNG

You can configure SearXNG in LibreChat within the UI or through librechat.yaml.

UI Configuration

1. Open the tools dropdown in the chat input bar

2. Click on the gear icon next to Web Search

3. Select SearXNG from the Search Provider dropdown

4. Enter your configuration details (e.g. instance URL, scraper type, etc.) and click save

5. Click on the Web Search option in the tools dropdown

6. The Web Search badge should now be enabled, meaning your queries can now utilize the web search functionality