Building a Modern Site Search Engine with Laravel, Elasticsearch, and AI

I built SiteSearch to provide search across multiple websites. That means getting each site's content into an index, keeping it up to date, and making sure a query searches the right site's pages.

I've worked on search and crawling for a long time. Back in 2007, I was pulling together content from the social web. The stack has changed. Getting useful text out of a page is still a big part of the job.

This version uses Laravel, Elasticsearch, and OpenAI. Here's how the pieces fit together.

Start with the page

The crawler uses a sitemap when one is available and falls back to following URLs. It extracts the title, description, body text, and metadata, then saves a page record in Laravel.

Different sites have different structures. A navigation menu, footer, or repeated sidebar can add a lot of words without adding much useful content to search. The text you put into the index matters before you write a single relevance rule.

The crawl has a page limit and a delay between requests. Laravel records the crawl status and completion time so the application can show whether it finished or failed.

A separate index for each site

Each site has its own Elasticsearch index name. A page's UUID identifies the document inside that index. A search resolves the site first and sends the query to that site's index.

That keeps the search scope clear. The application still has to authorize which site someone can access; an index name isn't an access policy by itself.

The document contains the original page fields and AI-generated metadata. OpenAI generates additional keywords and summaries from the extracted content. Those give the search another representation of the page, alongside its original text.

This implementation uses generated metadata in a keyword query. Embedding-based retrieval is a different approach, which I wrote about in the PostgreSQL hybrid search post.

Weighting the fields

A term in a title shouldn't necessarily have the same influence as one buried in the body. The search uses Elasticsearch's multi_match query to search several fields with different weights.

Here's a reduced query example, assuming the fields have already been mapped and indexed:

{
  "query": {
    "multi_match": {
      "query": "website security",
      "fields": ["title^3", "description^2", "ai_keywords^2.5", "text"],
      "type": "best_fields",
      "tie_breaker": 0.3
    }
  }
}

Those weights are choices to test against the content. They aren't universal settings for good search. The Elasticsearch multi-match documentation explains how field boosts and the tie breaker affect scoring.

Highlights provide snippets from the matching text so someone can see why a page appeared. The title and URL then get them back to the original page.

Keeping the index current

Page changes need to trigger enrichment and indexing. Doing expensive AI work in the request that saves the page would make that request depend on another service, so queues are useful for the processing work.

There are also two different failure cases. If AI enrichment fails, the original page content can still be indexed. If Elasticsearch fails, the indexing job needs to be retried. Tracking those separately makes it easier to see which part needs attention.

There are a couple timing decisions in that pipeline. Refreshing after every document makes changes searchable sooner but adds indexing work. Caching queries reduces repeated searches but gives you another thing to invalidate when a page changes. I'd look at how often the pages change before choosing those settings.

What to look at next

Search logs can show queries with no results, popular searches, and which results people click. Those give you something concrete to investigate when adjusting the index or ranking.

If you want to see the product, it's at sitesearch.app. I also wrote up Meilisearch with Laravel Scout and search with Cloudflare Vectorize as other approaches to the same general problem.