Making Blog Search Smarter with LLMs and Open WebUI

2025-09-29T00:00:00+00:00

We recently released LLM Expanded Search for our blog’s vector search. It builds on what we covered in our earlier posts about AI-powered search and vector search basics. Here’s how we built it with our internal AI setup (Open WebUI running an OpenAI-compatible API), why it makes search better, and what’s coming next.

What “LLM Expanded Search” actually does

Here’s the basic idea: when you search for something, we first ask an LLM to come up with related terms and phrases. Then we search for all of those terms, not just your original query.

Your search gets expanded by an open-source LLM through our AI portal (Open WebUI with an OpenAI-compatible API)
Those extra terms give our vector index more ways to find posts that match what you’re looking for
We combine the results, remove duplicates, and sort by relevance before showing the best matches with snippets and links

This really helps with short or vague searches where regular vector search might miss the relevant context — for example, “S3” refers to Amazon S3, which is a cloud object storage system, so whereas “S3” doesn’t provide enough context for a useful vector search. An LLM can expand this short search and include context about cloud object storage in general, as well as give enough context to return results about S3.

How it works

The frontend is pretty straightforward: our search bar has two options, “Search” (just hit Enter) and “LLM Expanded Search” (Shift/Ctrl/Command+Enter).

When you use expanded search, here’s what happens:

We call our Open WebUI endpoint with a prompt that asks for 8–15 related terms
We turn both your original query and the expanded terms into embeddings
We search our vector store with all these terms and combine the results
Caching and rate limiting keep things fast and cheap

Here’s a simple example of how we expand queries:

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.getenv("OPENAI_BASE_URL"),   # e.g., http://openwebui.local/api/v1
    api_key=os.getenv("OPENAI_API_KEY")      # token managed in your environment
)

def expand_query(raw_query: str) -> list[str]:
    messages = [
        {
            "role": "system",
            "content": (
                "You expand a short search query into a concise, comma-separated list of "
                "synonyms and closely related phrases (8–15 items). No explanations."
            )
        },
        {"role": "user", "content": raw_query}
    ]
    res = client.chat.completions.create(
        model=os.getenv("OPENAI_MODEL", "local-llm"),
        messages=messages,
        temperature=0.2,
        max_tokens=200,
    )
    text = res.choices[0].message.content
    return [t.strip() for t in text.split(",") if t.strip()]

After that, we embed the original query and the expanded terms, search the vector index, then sort by score and drop duplicates so each post appears once. Finally, we render concise snippets.

For example, after a similarity search you can rank and de-duplicate like this:

# given: results = [(doc, score), ...]
valid = [(d, float(s)) for d, s in results if float(s) > 0.05]
valid.sort(key=lambda x: x[1], reverse=True)  # highest score first

seen = set()
unique = []
for doc, score in valid:
    src = doc.metadata.get("source", "")
    if src not in seen:
        unique.append((doc, score))
        seen.add(src)

# unique now holds top ranked, de‑duplicated posts

Why we chose Open WebUI

A few reasons made Open WebUI the right choice:

It’s open source and works great self-hosted
The OpenAI-compatible API means we can drop it into existing code
We can use whatever models and inference backends we want
It’s easy to experiment with different prompts and workflows

What’s next: Moving more into Open WebUI

We’re looking into moving more of the search pipeline directly into Open WebUI workflows:

Query expansion (LLM)
Vector retrieval (custom tool that hits our index)

This would give us tighter integration, fewer network calls, and simpler deployment, and make it easier to try new approaches.

What you’ll notice when using it

Short searches work way better, you get more relevant results and fewer dead ends
It’s still experimental, so sometimes results might drift into related topics. Stick with regular “Search” if you want more exact matches
We cache common terms to keep things smooth

Give it a try at our blog. Just use the search bar in our header: press Enter for regular search, or Shift/Ctrl/Command+Enter for LLM Expanded Search.

Want to know more about why we built this? Check out the announcement and vector search posts linked above.

If you’re interested in setting up LLM-expanded vector search or running something similar self-hosted with Open WebUI, we’d love to help out.

Vector Search: The Future of Finding What Matters

2025-07-01T00:00:00+00:00

Photo by Ann H on Pexels

In a world flooding with data in several different formats like images, documents, text, and videos, traditional search methods are starting to not be modern anymore. Today, the vector search technique is revolutionizing how we retrieve and understand information. If you wonder how Spotify can recommend the perfect song or how Google can find almost perfectly accurate image matches, vector search is kind of the wizard behind the curtain. Let’s see how it has become a game changer.

What Is Vector Search?

At its core, vector search is a method of finding similar items in a dataset by representing them as vectors — essentially, lists of numbers in a multi-dimensional space. Unlike keyword-based search, which relies on exact matches or predefined rules, vector search focuses on semantic similarity. This means it can understand the meaning or context behind data, not just the words or pixels on the surface.

Imagine you’re searching for a cozy cabin in the woods. A traditional search might get stuck on the exact words in this query, missing a listing for something similar like a snug retreat nestled in a forest. Vector search, however, can connect the dots because it understands that cozy and snug or woods and forest are conceptually close.

You can try vector search on this blog! There’s a search bar in the header at the top of the page, and you can see our announcement post here.

How Does It Work?

The magic of vector search happens in three key steps:

Embedding Generation

Once you have access to the data sources that you want to apply vector search, you will need to prepare and ingest the data into your system. This phase will handle data clean-up and extraction from different data sources and formats. For multi-modality support you may need to include multiple data formats like text, image, audio, and video.

In the embedding generation phase, as depicted above, raw data (like text, images, or audio) is transformed into vectors using suitable embedding machine learning (ML) models. These models (e.g. all-MiniLM-L6-v2, all-mpnet-base-v2 for text, or ResNet for images) analyze the data and spit out a numerical representation, a vector, that captures its essence. For example, the sentence I love sunny hikes might become something like [-0.07571851, -0.02147608, 0.07130147, 0.1087752, 0.02052169, ...], where each number reflects a feature of its meaning.

Storing Vectors

The generated vectors are then stored in a vector database or an index optimized for fast retrieval. Think of it as a massive, multi-dimensional map where every point represents an item or a chunk of an item. There are several options for your choice like FAISS, Chroma, Milvus, and pg-vector.

Similarity Search

When you use this system with a query like find me green hoodies, it converts your input into a vector too. Then, it searches the vector store for the closest points using mathematical distance metrics like Cosine Similarity or Euclidean Distance. The closer the vectors, the more similar the items.

The result? You get matches that feel right, even if they don’t share exact keywords or pixel patterns.

A Simple Vector Search Implementation

Here, to demonstrate how vector search works, I implemented a simple vector search based on this architecture without using third-party libraries. The best practice is surely using well-known implementations and tools to achieve production ready applications; this is just for demonstration.

Below is a very simple vector store implementation:

import numpy as np
from typing import List, Tuple

class SimpleVectorStore:
    def __init__(self):
        """Initialize"""

        self.vectors = []
        self.ids = []

    def add_vectors(self, vectors: List[List[float]], ids: List[str]):
        """Add vectors to vector store"""
        if len(vectors) != len(ids):
            raise ValueError("Number of vectors must match number of IDs")

        vectors = np.array(vectors)
        self.vectors.extend(vectors)
        self.ids.extend(ids)

        self.vectors = np.array(self.vectors)

    def cosine_similarity(self, v1: np.ndarray, v2: np.ndarray) -> float:
        """Calculate cosine similarity between two vectors"""

        # zero vectors
        if np.all(v1 == 0) or np.all(v2 == 0):
            return 0.0

        dot_product = np.dot(v1, v2)
        norm_v1 = np.linalg.norm(v1)
        norm_v2 = np.linalg.norm(v2)
        return dot_product / (norm_v1 * norm_v2)

    def search(self, query_vector: List[float], k: int = 5) -> List[Tuple[str, float]]:
        """Search for the k most similar vectors"""

        if len(self.vectors) == 0:
            return []

        query_vector = np.array(query_vector)
        similarities = [
            (self.vectors[i], self.ids[i], self.cosine_similarity(query_vector, vec))
            for i, vec in enumerate(self.vectors)
        ]

        # sort and return
        similarities.sort(key=lambda x: x[2], reverse=True)
        return similarities[:k]

Now, let’s store some sample vectors (so-called embeddings) to this simple vector store and perform an example search. In real-world applications this represents the raw data in vector form, but I used dummy embedding values in this example. I also excluded the embedding model phase when creating these embedding values to simplify the code.

if __name__ == "__main__":
    # vector store instance
    vector_store = SimpleVectorStore()

    # example embeddings
    sample_embeddings = [
        [1.0, 0.0, 0.0], # Vector 1
        [0.0, 1.0, 0.0], # Vector 2
        [1.0, 1.0, 0.0], # Vector 3
        [0.0, 0.0, 1.0], # Vector 4
    ]
    sample_ids = ["doc1", "doc2", "doc3", "doc4"]

    # add sample vectors to vector store
    vector_store.add_vectors(sample_embeddings, sample_ids)

    # query vector
    query = [1.0, 0.5, 0.0]

    # search for top k = 2 results
    top_k = 2
    results = vector_store.search(query, k=top_k)

    print("Query vector:", query)
    print(f"Top {top_k} similar vectors:")
    for vector, doc_id, similarity in results:
        print(f"DocId: {doc_id}, Vector: {vector}, Similarity: {similarity:.4f}")

The output of this search is like below:

Query vector: [1.0, 0.5, 0.0]
Top 2 similar vectors:
DocId: doc3, Vector: [1. 1. 0.], Similarity: 0.9487
DocId: doc1, Vector: [1. 0. 0.], Similarity: 0.8944

Now compare the vectors in the results with the query vector. As you can see the bigger similarity scores are the closer ones and are more similar to the query vector.

Why Vector Search is Powerful

Vector search offers several advantages over traditional search methods. First of all, it has a degree of semantic understanding by focusing on contextual meaning rather than keywords: vector search handles synonyms, misspellings, and contextual nuances. For instance, searching car could return results about automobiles or vehicles (assuming you have a well-trained model).

Multimodal capability is another feature which is commonly used today. Vector search isn’t limited to text. It can process images, audio, video, or even combinations of these. For example, you could search for sunset photos using a text query and retrieve visually similar images.

Vector search can incorporate user preferences or behavior like past searches into the vector space and deliver personalized results.

Real-World Examples

Vector search is already everywhere:

Retrieval-Augmented Generation (RAG)

Regardless of the industry that a RAG application is designed for, vector search is essential in the retrieval (“R”) phase: it is used to find relevant data for a query to build a context for the LLM.
Ecommerce:

Vector search is used for recommending products based on user queries or visual similarity (e.g. “find shoes like these” using an image). Example: Searching for “red sneakers” might return visually similar items even if the product description doesn’t mention “red sneakers”.
Content Discovery:

Used for powering recommendation engines for streaming platforms (e.g. Netflix, Spotify) by finding movies, songs, or articles similar to a user’s interests. Example: Suggesting a sci-fi movie based on a user’s love for “Star Wars”.
Customer Support:

Used for enabling chatbots to retrieve relevant knowledge base articles or FAQs by understanding the intent behind a user’s question. Example: A query like “how to reset my device” could pull up guides even if phrased differently.
Image and Video Search:

Allows users to search for visually similar images or videos, such as finding artwork or stock footage that matches a specific style. Example: Uploading a photo of a beach to find similar vacation destinations.
Enterprise Search:

Helps employees find documents, emails, or internal resources by understanding the context of their queries. Example: Searching “project timeline” could retrieve relevant spreadsheets or emails.
Healthcare:

Used for matching patient records, medical images, or research papers based on semantic similarity to aid diagnosis or treatment planning. Example: Finding studies related to a specific disease even if terminology varies.

Challenges Ahead

Although computational cost is one of the challenges of performing similarity search on large datasets I won’t mention it as a major challenge.

The main challenges in vector search seem to revolve around interpretability and data quality, as is common in ML. Unlike keyword search, where matches are explicit, vector search operates in a black box, making it harder to explain why certain results were returned.

As for data quality, the adage applies: garbage in, garbage out. If the input data is noisy or incomplete, the embeddings may not accurately represent the content. Data extraction and creating the embeddings of certain data formats (e.g. tables) is another challenge. Searching for accurate table data values is an especially challenging task. At End Point we are using advanced methods and LLM services to overcome these challenges when preparing quality embeddings.

Wrapping Up

Vector search isn’t just a tech buzzword — it’s a fundamental shift in how we interact with data. By moving beyond rigid keywords to a world of meaning and similarity, it’s unlocking possibilities we’re only beginning to explore. Whether you’re a developer building the next big app or just a curious soul, vector search is worth keeping an eye on. It’s not about finding exactly what you typed — it’s about finding exactly what you meant.

Building Ecommerce Search Using Algolia

2023-10-12T00:00:00+00:00

A common request that developers receive when embarking on a new website project is for the website to have “Google-like search.” For many years, this meant writing custom code to replicate the intelligent and user-friendly aspects of Google search, which was no easy feat. However, now we have many search-as-a-service offerings that do the hard work for us and make this process much easier.

In this blog post, we’ll dive into one of these search-as-a-service platforms, Algolia. We recently worked on an ecommerce website and used Algolia in an interesting way, both as a search engine and as a lightweight backend database to hold product data managed in Salesforce. Algolia worked beautifully, offering users fast and accurate search results, and also allowing us to launch the site within a relatively short time frame.

We will look at how to load Algolia with data, configure search options, and connect the search to the frontend using Algolia’s Vue library.

Loading the index with data

To start using Algolia’s search, you need to load up an index with data. You have the option of manually uploading a JSON file, or using Algolia’s API to programmatically load records. For our backend, we chose to use Algolia’s JavaScript API client in some lightweight TypeScript scripts that are triggered by cron. These scripts allowed us to sync inventory data between Salesforce and the index in Algolia.

Using the Algolia JavaScript client is quite simple. Regardless of where your data comes from—be it in a database, a platform like Salesforce, or elsewhere—once it is in JSON format, you can load it into Algolia with a few lines of code:

import * as algolia from "algoliasearch";

const products = [{
    name: "Fender F-5 Acoustic",
    make: "Fender",
    model: "F-5",
    category: "Guitars",
    status: "Used",
    objectID: "Fender-001"
  }, {
    name: "Fender Player Jaguar",
    make: "Fender",
    model: "Jaguar (Player)",
    category: "Guitars",
    status: "New",
    objectID: "Fender-002"
  }];


const index = algolia.default(process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_API_KEY).initIndex("store_products");
await index.saveObjects(products);

Note the objectID property, which is used by Algolia as a primary key. If the objectID does not exist, a new record will be created. If it does exist, the record will be updated. This makes it easy to run a data sync process using a single saveObjects command, without having to worry about differentiating between create and update operations.

Configuring the index

Once you have your index loaded, you’ll want to configure it. Algolia does a good job of walking you through this process using a built-in tutorial when you first load your index. Basically, you will be selecting the searchable properties/attributes from your JSON data, setting how results are ranked and sorted, and adjusting more advanced aspects of search like typo tolerance, stop words, etc.

An important feature we utilized on our recent project is faceting. Faceting allows users to easily drill down and refine search by categories, and is also easy to develop using the handy frontend libraries that Algolia provides (more on that in the next section). This feature is powerful and can be used to both refine search and drive homepage category/subcategory links. When you configure your index, you can select which attributes of your data should be used for faceting.

Setting up search on the frontend

We used Nuxt to build the frontend of the website, and we leveraged Algolia’s Vue InstantSearch library for the UI. This library really speeds along development, as it wraps all of the search-related functionality in simple widgets, providing the search bar, results, refinements, filtering, and more.

The ais-instant-search widget is the parent widget. It serves the search state to its children, which allows you to show the search bar, search hits, hierarchical menus, etc. Here is a simple example of the ais-instant-search widget with a search bar and hits (pulled directly from Algolia’s Vue docs):

<template>
  <ais-instant-search :search-client="searchClient" index-name="demo_ecommerce">
    <ais-search-box />
    <ais-hits>
      <template v-slot:item="{ item }">
        <h2>{{ item.name }}</h2>
      </template>
    </ais-hits>
  </ais-instant-search>
</template>


<script>
import algoliasearch from 'algoliasearch/lite';
import 'instantsearch.css/themes/algolia-min.css';


export default {
  data() {
    return {
      searchClient: algoliasearch(
        '[Your app ID]',
        '[Your API key]'
      ),
    };
  },
};
</script>


<style>
body {
  font-family: sans-serif;
  padding: 1em;
}
</style>

I mentioned faceting above when discussing how to configure your index. Once you have selected the attributes in your JSON data that can be used for faceting (e.g., category, subcategory), you can feed those attributes to the ais-hierarchical-menu widget for display on the frontend.

Here is a bit of sample code from the website we built, which offers expandable category refinement via ais-hierarchical-menu.

<div class="sidebar-segment">
  <p class="sidebar-segment-title">Category</p>
  <ais-hierarchical-menu
    :limit="100"
    :attributes="categoryAttrs"
    :sort-by="hierarchicalMenuSort"
  >
    <div slot-scope="{ items, refine, createURL }">
      <hierarchical-menu-list
        :items="items"
        :refine="refine"
        :create-url="createURL"
      />
    </div>
  </ais-hierarchical-menu>
</div>

Above, we are using the limit property to set the maximum number of items to 100. The attributes property, which targets the JSON attributes in your index data that represent your categories, is set to the following:

categoryAttrs: ['categories.lvl0', 'categories.lvl1', 'categories.lvl2'],

These represent the three levels of categories in our data (zero-based), and are used to build the hierarchical menu.

Finally, the sort-by attribute points to a simple function that uses the JavaScript localeCompare method to provide alphanumeric sorting:

function hierarchicalMenuSort(a, b) {
  return a.name.localeCompare(b.name, undefined, {
    numeric: true,
    sensitivity: 'base',
  })
}

Wrapping up: seeing Algolia in action

If you’d like to see Algolia in action on the site that we built, head over to eiffeltrading.com. If you search the site, you’ll see autocomplete, fast results (thanks to both Algolia and Nuxt), and other aspects of good search that we have all come to expect from modern ecommerce sites.

Next time you are faced with a build involving full-text search, consider search-as-a-service offerings like Algolia. They could save you time and headaches over rolling your own search functionality. Sometimes it’s good to let others do the hard work!

Have questions or feedback on the topic? Let us know in the comments section below.

Full-text search on a budget: Xapian

2021-08-19T00:00:00+00:00

Over the years I’ve seen and implemented different full-text search applications using various technologies: plain SQL, PostgreSQL, Elasticsearch, Solr, and most recently Xapian.

While Solr and Elasticsearch are very well known, Xapian, despite the fact that it’s available and packaged in all the major GNU/Linux distributions, doesn’t seem to be so popular, at least not among project managers.

But Xapian is fast, advanced, can be configured to do faceted searches (so the user can filter the search results), and my favorite, is fast to build and has virtually no maintenance overhead.

Its main feature is that it’s not a stand-alone application, like Solr or Elasticsearch, but instead it’s a library written in C++ which has bindings for all the major languages (as advertised on its homepage). It has also great documentation.

Now, being in the e-commerce business, my typical use-case is that the client’s shop needs something faster and better than a search using a SQL query against the products table. And beware, even implementing a non-trivial SQL-based search can burn more hours than setting up Xapian.

With Xapian you can prototype very quickly, without losing hours wading through obscure options, setting up services and configuring firewalls. And yet, the prototype will allow you to build more advanced features once you need them.

I’m a Perl guy, so I will show you some Perl code, but the procedure is the same for the other languages. Even the documentation can be built specifically for your language!

Typically, to add a search engine to your site you need two pieces: an indexer to which you feed the data (from static files or databases or even fetching remote pages or whatever you need) and the search itself in the site.

Both the indexer and the search code need to load the Xapian library and point to the same Xapian database, which is usually a directory (or a file pointing to a directory).

Indexing

Now, stripped down to the minimum, this is what a typical indexer’s code looks like:

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian (':all');
use JSON qw/encode_json/;

my $dblocation = "xapiandb";
my $xapian = Search::Xapian::WritableDatabase->new($dblocation, DB_CREATE_OR_OPEN);
my $indexer = Search::Xapian::TermGenerator->new;
$indexer->set_database($xapian);
$indexer->set_stemmer(Search::Xapian::Stem->new("english"));
my @entries = ({ uri => '/blog/1', title => 'T1', text => 'Marco loves pizza' },
               { uri => '/blog/2', title => 'T2', text => 'They love chapati' });
foreach my $data (@entries) {
    my $doc = Search::Xapian::Document->new;
    my $qterm = 'Q' . $data->{uri};
    $doc->add_term($qterm);
    $doc->set_data(encode_json({ uri => $data->{uri}, title => $data->{title} }));
    $indexer->set_document($doc);
    $indexer->index_text($data->{text});
    $xapian->replace_document_by_term($qterm, $doc);
}

This code will create a xapiandb directory with the Xapian database, indexing the blog posts in the @entries array. In a real script, they would come from the database.

Still, there are a couple of things worth noting in this minimal code.

We set the stemmer for the given language, so the text passed to the indexer via the index_text call is parsed.

Then we store the data structure we want to retrieve later with set_data. The best thing to do is probably to serialize it with JSON, in this case I’m excluding the full text, which we don’t need in the output (but it would be wise to add a teaser).

Also, we use a Q prefix to produce an unique term to update the entry when it already exists.

Of course the indexer will need to grow if you need more power and more structured data (like filtering or searching a specific field), but at this point we want just to show something to our hypothetical client.

The database can be inspected very easily. Xapian comes with a tool called delve (or xapian-delve):

$ xapian-delve xapiandb -a -v -1
All terms in database (termfreq):
Q/blog/1 1
Q/blog/2 1
Zchapati 1
Zlove 2
Zmarco 1
Zpizza 1
Zthey 1
chapati 1
love 1
loves 1
marco 1
pizza 1
they 1

And you can also try a search from the command line with quest:

$ quest -d xapiandb "loves NOT chapati"
Parsed Query: Query((Zlove@1 AND_NOT Zchapati@2))
Exactly 1 matches
MSet:
1: [0.0953102]
{"title":"T1","uri":"/blog/1"}

$ quest -d xapiandb "pizza OR chapati"
Parsed Query: Query((Zpizza@1 OR Zchapati@2))
Exactly 2 matches
MSet:
1: [0.405465]
{"title":"T1","uri":"/blog/1"}
2: [0.405465]
{"uri":"/blog/2","title":"T2"}

As the example above shows, it should be clear that:

the search works as you would expect (with logical operators) out of the box
the stemming works, searching for “loves” and “love” is the same.
the results give us back the JSON we stored in the index.

Searching

So let’s call it done and move to the next part, the searcher.

Now, while the indexer is a single script, the search needs to be plugged into the live code of your site. For the purposes of this article, I will provide a script instead, which does basically the same thing as quest. Plugging it into the web application is left as an exercise for the reader. I would also suggest to put both the indexing and searching code in a single shared module, keeping the logic in a single location.

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian ':all';
use JSON;

my ($cgi) = join(' ', @ARGV);
my $dblocation = "xapiandb";
my $database = Search::Xapian::Database->new($dblocation);
my $enquire = Search::Xapian::Enquire->new($database);
my $qp = Search::Xapian::QueryParser->new;
$qp->set_database($database);
$qp->set_stemmer(Search::Xapian::Stem->new("english"));
$qp->set_stemming_strategy(STEM_SOME);
$qp->set_default_op(OP_AND);
my $query = $qp->parse_query($cgi, FLAG_PHRASE|FLAG_BOOLEAN|FLAG_WILDCARD);
$enquire->set_query($query);

# fetch the first 50 results
my $mset = $enquire->get_mset(0, 50);
print "Total results: " . $mset->get_matches_estimated . "\n";

my $json_pretty = JSON->new->pretty(1)->utf8(1)->canonical(1);
foreach my $m ($mset->items) {
    my $data = decode_json($m->get_document->get_data);
    # decode and reencode the json in a human-readable fashion
    print $json_pretty->encode($data);
}

If you’re wondering what those constants are and where to look for more, they are in the module’s documentation, in plain sight (we asked for them when loading the module with the :all argument).

Most of the code shown here is boilerplate, but that could change once you build up. Notably we set the stemmer for the current language and the query parser options, so we can use wildcard (e.g. piz*), the AND/OR operators, and quoting.

Let’s see the script in action.

Wildcard:

$ ./search.pl 'piz'
Total results: 0
$ ./search.pl 'piz*'
Total results: 1
{
   "title" : "T1",
   "uri" : "/blog/1"
}

Operators:

$ ./search.pl 'pizza OR chapati'
Total results: 2
{
   "title" : "T1",
   "uri" : "/blog/1"
}
{
   "title" : "T2",
   "uri" : "/blog/2"
}
$ ./search.pl 'pizza AND chapati'
Total results: 0

Quoting (beware here the double quotes to escape the shell):

$ ./search.pl '"loves chapati"'
Total results: 0
$ ./search.pl '"love chapati"'
Total results: 1
{
   "title" : "T2",
   "uri" : "/blog/2"
}

The whole thing already looks pretty good. Way better (and way faster to code and to execute) than a home-baked SQL search.

As already noted, this is just scratching the surface. Xapian can do much more: filtering, range queries, facets, sorting, even spelling corrections!

I don’t doubt that Solr & co. have their use-cases, but for the common scenario of a small/mid-sized e-shop or site, I think that this solution is more affordable and maintainable than having a whole separate application (like a Solr server) to maintain, upgrade and secure. Don’t forget that here we haven’t done a single HTTP request. We didn’t have to manage daemons, opening/closing ports, and the like. We didn’t have to configure a schema and a tokenizer in a separate application (and keep that aligned with the handling code). It’s all there in our (Perl) code in two files (as already noted, the logic should live in a single module).

We just installed a library (there is a very good chance that it’s already installed) and a Perl module.

The Xapian database lives on the disk and your code has full control over it. Also it’s normally your GNU/Linux distribution taking care of the security upgrades.

If your client is on a budget, building a full-text search Xapian can be the right choice, and you can scale it up on the go, as more features are required.

Mobile-friendly sites or bust!

2015-03-23T00:00:00+00:00

A few weeks ago, Google announced that starting on April 21 it will expand its “use of mobile-friendliness as a ranking signal” which “will have a significant impact in our search results”.

The world of search engine optimization and online marketing is aflutter about this announcement, given that even subtle changes in Google’s ranking algorithm can have major effects to improve or worsen any particular site’s ranking. And the announcement was made less than two months in advance of the announced date of the change, so there is not much time to dawdle.

Google has lately been increasing its pressure on webmasters (is that still a real term‽) such as with its announcement last fall of an accelerated timetable for sunsetting SSL certificates with SHA-1 signatures. So far these accelerated changes have been a good thing for most people on the Internet.

In this case, Google provides an easy Mobile-Friendly Site Test that you can run on your sites to see if you need to make changes or not:

So get on it and check those sites! I know we have a few that we can do some work on.

Elasticsearch: Give me object!

2013-04-30T00:00:00+00:00

I’m currently working on a project where Elasticsearch is used to index copious amounts of data with sometimes deeply nested JSON. A recurring error I’ve experienced is caused by a field not conforming to the type listed in the mapping. Let’s reproduce it on a small scale.

Assuming you have Elasticsearch installed, let’s create an index and mapping:

$ curl -XPUT 'http://localhost:9200/test' -d '
{
    "mappings": {
        "item": {
            "properties": {
                "state": {
                    "properties": {
                        "name": {"type": "string"}
                    }
                }
            }
        }
    }
}
'
{"ok":true,"acknowledged":true}

Since we’ve defined properties for the “state” field, Elasticsearch will automatically treat it as an object.* Let’s now add some documents:

$ curl -XPUT 'http://localhost:9200/test/item/1' -d '
{
    "state": {
        "name": "North Carolina"
    }
}
'
{"ok":true,"_index":"test","_type":"item","_id":"1","_version":1}

Success! Let’s now get into trouble:

$ curl -XPUT 'http://localhost:9200/test/item/2' -d '
{
    "state": "California"
}
'
{"error":"MapperParsingException[object mapping for [state] tried to parse as object, but got EOF, has a concrete value been provided to it?]","status":400}

The solution: check any non-objects in your data against your mapping schema and you’ll be sure to find a mismatch.

One thing to note is that the explicit creation of the mapping is unnecessary since Elasticsearch creates it using the first added document. Try this:

$ curl -XPUT 'http://localhost:9200/test2/item/1' -d '
{
    "state": {
        "name": "North Carolina"
    }
}
'
{"ok":true,"_index":"test2","_type":"item","_id":"1","_version":1}
$ curl -XGET 'http://localhost:9200/test2/_mapping'
{
    "test2": {
        "item": {
            "properties": {
                "state": {
                    "dynamic":"true",
                    "properties": {
                        "name": {"type":"string"}
                    }
                }
            }
        }
    }
}

So, this stays true to the statement: “Elasticsearch is schema-less, just toss it a typed JSON document and it will automatically index it.” You can throw your car keys at Elasticsearch and it will index, however, as noted above, just be sure to keep throwing nothing but car keys.

*Anything with one or more nested key-value pairs is considered an object in Elasticsearch. For more on the object type, see here.

Make your code search-friendly

2012-04-12T00:00:00+00:00

Here’s something about coding style that you may not have considered: is your code “search-friendly”? That is, does the format of your code help or hinder someone who might be searching it for context while debugging, extending, or just learning how it works?

Seriously Contrived Example (from Perl):

my $string = q{Your transaction could not be} .
   q{ processed due to a charge} .
   q{ card error.};
return $string;

Now someone’s going to experience this error and wonder where it occurs. So armed with grep, or ack, or git-grep, they set off into the wilderness:

$ git grep 'could not be processed'
$ git grep 'charge card error'
$ git grep -e 'transaction.*charge.*error'
$ alsdkjgalkghkf

(The last simulates pounding the keyboard with both fists.) I would suggest humbly that “strings you emit as a line should appear as a line in your code”, if for no other reason than that it makes it so much easier for you or others to find them. Thus:

my $string = <<'MSG';
Your transaction could not be processed due to a charge card error.
MSG
return $string;

Interchange Search Caching with “Permanent More”

2012-01-02T00:00:00+00:00

Most sites that use Interchange take advantage of Interchange’s “more lists”. These are built-in tools that support an Interchange “search” (either the search/scan action, or result of direct SQL via [query]) to make it very easy to paginate results. Under the hood, the more list is a drill-in to a cached “search object”, so each page brings back a slice from the cache of the original search. There are extensive ways to modify the look and behavior of more lists and, with a bit of effort, they can be configured to meet design requirements.

Where more lists tend to fall short, however, is with respect to SEO. There are two primary SEO deficiencies that get business stakeholders’ attention:

There is little control over the construction of the URLs for more lists. They leverage the scan actionmap and contain a hash key for the search object and numeric data to identify the slice and page location. They possess no intrinsic value in identifying the content they reference.
The search cache by default is ephemeral and session-specific. This means all those results beyond page 1 the search engine has cataloged will result in dead links for search users who try to land directly on the more-listed pages.

It is the latter issue that I wish to address because there is—and has been for some time now—a simple mechanism called “permanent more” to remedy the default behavior.

You can leverage “permanent more” by adding the boolean mv_more_permanent, or the shorthand pm, to your search conditions. E.g.:

Link:

    <a href="[area search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]">All Foos</a>

Loop:

    [loop search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]
    ...loop body with [more-list]...
    [/loop]

Query:

    [query
        list=1
        more=1
        ml=10
        <b>pm=1</b>
        sql="SELECT * FROM products WHERE category LIKE '%Foo%'"
    ]
    ...same as loop but with 10 matches/page...
    [/query]

If the initial search is defined with the “permanent more” setting, it will produce the following adjustments:

The hash key used to store and identify the search cache is deterministic based on the search conditions. Many searches for Interchange are category driven. Thus, all end users who wish to browse a category end up clicking identical links, which create duplicate search caches, belonging uniquely to them. With permanent more, they all share the same cache, with the same identifier. As long as the search conditions don’t change, neither does the cache identifier. Even as the cache is refreshed with new executions of the search, the object remains in the same location. Thus, the results a search engine produced this morning reference links still valid now, tomorrow, or next week, provided they reference the same search conditions.
The cached search object has no session affinity. Any link referencing the cache with the correct hash key has access to the content.

Taken together, “permanent more” removes (for the most part, addressed later) dead links from more lists cataloged by search engines. There are, however, other benefits to “permanent more” beyond those intended as described above:

As stated in passing, standard Interchange search caching produces duplicate search objects for common search conditions. For a busy site, these caches can have an impact on storage. Typically, maintenance is implemented to clean up cache files for all such files whose age exceeds by some amount the session duration (standard is 48 hours). With permanent more, duplicate caches are eliminated. A cache location is reused by all users with the same search requirements, keeping data-storage requirements for caches to the minimum necessary. As searches change, ophaned caches can still easily be cleaned up as they will immediately start to age with no more access to them necessary for storage.
For the same reason that “permanent more” resolves search-engine links, it also resolves content management for individual sites using a reverse proxy for caching. Because most (and certainly the easiest) caching keys are based off of URL, the deterministic nature of the hash keys for “permanent more” allows assurance that the cached content in the proxy accurately reflects the search content over time, and that all users will hit the cached resource and not generate new, unique links with varying hash keys.

One shortcoming of “permanent more” to be aware of is the impact of changing data underneath the search. Even if search conditions do not change, the count and order of matching record sets may. So, e.g., enough products may be removed from a given category to cause the last page of a more list to become empty, which would cause any specific link into that page to become dead. More minor, but still a possibility, is the introduction or removal of products so that a particularly searched-for term has been “bumped” to another page within the search cache since the last time the search engine crawled the more lists. For searches backed by particularly volatile data, “permanent more” may not be sufficient to address search-engine or caching demands.

Finally, “permanent more” should be avoided for any search features that may cache data sensitive to an individual user. This is unlikely to happen as, under most circumstances, the configuration of the search itself will change based on the unique characteristics of the user executing the search (e.g., a username included in a query to review order history). However, it is still possible that context-sensitive information could be stored in the search object and, if so, all other users with access to the more lists would have access to that information.

Sunspot, Solr, Rails: Working with Results

2011-12-12T00:00:00+00:00

Having worked with Sunspot and Solr in several large Rails projects now, I’ve gained some knowledge about working with result sets optimally. Here’s a brief explanation on working with results or hits from a search object.

MVC Setup

When working with Sunspot, searchable fields are defined in the model:

class Thing < ActiveRecord::Base
  searchable do
    text :field1, :stored => true
    text :field2
    string :field3, :stored => true
    integer :field4, :multiple => true
  end
end

The code block above will include field1, field2, field3, and field4 in the search index of things . A keyword or text search on things will search field1 and field2 for matches. field3 and field4 may be used for scoping, or limiting the search result set based to specific values of field3 or field4.

In your controller, a new search object is created with the appropriate scoping and keyword values, shown below. Pagination is also added inside the search block.

class ThingsController < ApplicationController
  def index
    @search = Sunspot.search(Thing) do
      #fulltext search
      fulltext params[:keyword]

      #scoping
      if params.has_key?(:field3)
        with :field3, params[:field3]
      end 
      if params.has_key?(:field4)
        with :field3, params[:field4]
      end

      paginate :page => params[:page], :per_page => 25
    end
    @search.execute!
  end
end

In the view, one can iterate through the result set, where results is an array of Thing instances.

<% @search.results.each do |result| -%>
<h2><%= result.field3 %></h2>
<%= result.field1 %>
<% end -%>

Working with Hits

The above code works. It works nicely until you display many results on one page where instantiation of things is not expensive. But the above code will call the query below for every search, and subsequently instantiate Ruby objects for each of the things found. This can become sluggish when the result set is large or the items themselves are expensive to instantiate.

# development.log
Thing Load (0.9ms)  SELECT "things".* FROM "things" WHERE "things"."id" IN (6, 12, 7, 13, 8, ...)

An optimized way to work with search results sets is working directly with hits. @search.hits is an array of Sunspot::Search::Hits, which represent the raw information returned by Solr for a single returned item. Hit objects provide access to stored field values, identified by the :stored option in the model’s searchable definition. The model definition looks the same. The controller may now look like this:

class ThingsController < ApplicationController
  def index
    search = Sunspot.search(Thing) do
      #fulltext search
      fulltext params[:keyword]

      #scoping
      if params.has_key?(:field3)
        with :field3, params[:field3]
      end 
      if params.has_key?(:field4)
        with :field3, params[:field4]
      end
    end
    search.execute!

    @hits = search.hits.paginate :page => params[:page], :per_page => 25
  end
end

And working with the data in the view may look like this:

<% @hits.each do |result| -%>
<h2><%= hit.stored(:field3) %></h2>
<%= hit.stored(:field1) %>
<% end -%>

In some cases, you may want to introduce an additional piece of logic prior pagination, which is the case with the most recent Rails application I’ve been working on:

    ...
    search.execute!

    filtered_results = []

    search.hits.each do |hit|
      if hit.stored(:field3) == "some arbitrary value"
        filtered_results << hit
      elsif hit.stored(:field1) == "some other arbitrary value"
        filtered_results << hit
      end
    end
   
    @hits = filtered_results.paginate :page => params[:page], :per_page => 25

Sunspot and Solr are rich with functionality and features that can add value to a Rails application, but it’s important to identify areas of the application where database calls can be minimized and lazy loading can be optimized for better performance. The standard log file and database log file are good places to start looking.

Creativity with fuzzy string search

2010-08-10T00:00:00+00:00

PostgreSQL provides a useful set of contrib modules for “fuzzy” string searching; that is, searching for something that sounds like or looks like the original search key, but that might not exactly match. One place this type of searching shows up frequently is when looking for peoples’ names. For instance, a receptionist at the dentist’s office doesn’t want to have to ask for the exact spelling of your name every time you call asking for an appointment, so the scheduling application allows “fuzzy” searches, and the receptionist doesn’t have to get it exactly right to find out who you really are. The PostgreSQL documentation provides an excellent introduction to the topic in terms of the available modules; This blog post also demonstrates some of the things they can do.

The TriSano application was originally written to use soundex search alone to find patient names, but that proved insufficient, particularly because common-sounding last names with unusual spellings would be ranked very poorly in the search results. Our solution, which has worked quite well in practice, involved creative use of PostgreSQL’s full-text search combined with the pg_trgm contrib module.

A trigram is a set of three characters. In the case of pg_trgm, it’s three adjacent characters taken from a given input text. The pg_trgm module provides easy ways to extract all possible trigrams from an input, and compare them with similar sets taken from other inputs. Two strings that generate similar trigram lists are, in theory, similar strings. There’s no particular reason you couldn’t use two, four, or some other number of characters instead of trigrams, but you’d trade sensitivity and variability. And as the name implies, pg_trgm only supports trigrams.

Straight trigram search didn’t buy us much on top of soundex, so we got a bit more creative. A trigram is just a set of three characters, which looks pretty much just like a word, so we thought we’d try using PostgreSQL’s full text search on trigram data. Typically full text search has a list of “stop words”: un-indexed words judged too common and too short to contribute meaningfully to an index. Our words would all be three characters long, so we had to create a new text search configuration using a dictionary with an empty stop word list. With that text search configuration, we could index trigrams effectively.

This search helped, but wasn’t quite good enough. We finally borrowed a simplified version of a data mining technique called “boosting”, which involves using multiple “weak” classifiers or searchers to create one relatively good result set. We combined straightforward trigram, soundex, and metaphone searches with a normal full text search of the unmodified name data and a full text search over the trigrams generated from the names. The data sizes in question aren’t particularly large, so this amount of searching hasn’t proven unsustainably taxing on processor power, and it provides excellent results. The code is on github; feel free to try it out.

Update: One of the comments suggested a demonstration of the results, which of course makes perfect sense. So I resurrected some of the scripts I used when developing the technique. In addition to the scripts used to install the fuzzystrmatch and pg_trgm modules and the name_search.sql script linked above, I had a script that populated the people table with a bunch of fake names. Then, it’s easy to test the search mechanism like this:

select * from search_for_name('John Doe')
as a(id integer, last_name text, first_name text, sources text[], rank double precision);

 id  |  last_name  | first_name |                     sources                     |        rank        
-----+-------------+------------+-------------------------------------------------+--------------------
 167 | Krohn       | Javier     | {trigram_fts,name_trgm,trigram_fts,trigram_fts} |  0.281305521726608
 228 | Jordahl     | Javier     | {trigram_fts,name_trgm,trigram_fts}             |  0.237995445728302
  59 | Pesce       | Dona       | {trigram_fts}                                   |  0.174265757203102
 185 | Finchum     | Dona       | {trigram_fts}                                   |  0.174265757203102
 104 | Rumore      | Dona       | {trigram_fts}                                   |  0.174265757203102
 250 | Dumond      | Julio      | {name_trgm,trigram_fts,trigram_fts}             |   0.16849160194397
 200 | Dedmon      | Javier     | {name_trgm,trigram_fts,trigram_fts}             |  0.163729697465897
 230 | Dossey      | Malinda    | {name_trgm,trigram_fts}                         |  0.158055320382118
  50 | Dress       | Darren     | {name_trgm,trigram_fts}                         |  0.153293430805206
 136 | Doshier     | Neil       | {name_trgm,trigram_fts}                         |  0.148531511425972
 165 | Donatelli   | Lance      | {name_trgm,trigram_fts}                         |  0.132845237851143
 280 | Dollinger   | Clinton    | {name_trgm,trigram_fts}                         |  0.132845237851143
 273 | Dimeo       | Milagros   | {name_trgm,trigram_fts}                         | 0.0866267532110214
  49 | Dawdy       | Christian  | {name_trgm,trigram_fts}                         | 0.0866267532110214
 298 | Elswick     | Jami       | {trigram_fts}                                   | 0.0845221653580666

This isn’t all the results it returned, but it gives an idea what the results look like. The rank value ranks results based on the rankings given by each of the underlying search methods, and the sources column shows which of the search methods found this particular entry. Some search methods may show up twice, because that search method found multiple matches between the input text and the result record. These results don’t look particularly good, because there isn’t really a good match for “John Doe” in the data set. But if I horribly misspell “Jamie Elswick”, the search does a good job:

select * from search_for_name('Jomy Elswik') as a(id integer, last_name text,                                                 
first_name text, sources text[], rank double precision)

 id  |  last_name  | first_name |                     sources                     |        rank        
-----+-------------+------------+-------------------------------------------------+--------------------
 298 | Elswick     | Jami       | {trigram_fts,name_trgm,trigram_fts,trigram_fts} |  0.480943143367767
 312 | Elswick     | Kurt       | {name_trgm,trigram_fts}                         |  0.381967514753342
 228 | Jordahl     | Javier     | {trigram_fts,name_trgm,trigram_fts}             |  0.197063013911247
 403 | Walberg     | Erik       | {trigram_fts}                                   |  0.145491883158684
 309 | Hammaker    | Erik       | {trigram_fts}                                   |  0.145491883158684

Acts As Xapian — It Just Works

2008-08-26T00:00:00+00:00

I just recently started listening to the podcast done by the guys at RailsEnvy. It’s an excellent resource for keeping up on what’s new in the Rails world and it’s how I found out about the new acts_as_xapian search plugin for Rails. The podcast mentioned this blog post which contains a very thorough rundown of all the different full-text search options currently available for rails. The timing of this article couldn’t have been better since I was in the market for a new solution.

I was approaching a deadline on a client project here at End Point and I was having lots of trouble with my existing search solution which was acts_as_ferret. Setting up ferret was relatively easy and I was very impressed with the Lucene syntax that it supported. It seemed like a perfect a solution at first but then came “the troubles.”

Ferret is extremely fragile. The slightest problem and your server will just crash. What was causing the crash? Unfortunately the server logs won’t give you much help there. You will receive some cryptic message coming from the C++ library if you’re lucky. Note that I skipped the suggested Drb server setup since this was a development box.

After a while I would notice something wrong in my model code that might have caused an error while updating the search index. Unfortunately this was impossible to verify since I could not predictably reproduce the error. So in the end, I think there may have been issues with my model fields but ferret was of no help in tracking these problems down. The final straw came when the client started testing and almost immediately crashed the server after doing a search.

Enter acts_as_xapian. Jim Mulholland’s excellent tutorial was pretty much all I needed to get it up and running on my Mac. Documentation for acts_as_xapian is a bit thin. It consists primarily of the afore mentioned tutorial and a very detailed README. The mailing list is starting to become more active, however, and you are likely to get a response there to any thoughtful questions you might have.

One major difference with xapian (vs. ferret) is that it does not rebuild your index automatically with each model update. When you modify an ActiveRecord instance it will update the acts_as_xapian_jobs table with the id and model type of your record so that the index can be updated later. The index is then updated via a rake command that you can easily schedule via cron. You can also rebuild the entire index using a different rake command but that shouldn’t really be necessary.

I was a bit concerned about the lack of a continuously updated index but I came to realize that it has some significant advantages. The biggest advantage is that it’s much faster to update your model records since you are not waiting for the re-indexing to complete on the same thread. It also means you can skip the step of setting up a separate Drb server for ferret in your production environment.

With xapian you can index “related fields” in other models by constructing a pseudo-attribute in your model that returns the value of the associated model as a text string. Ferret allows you to do this as well, but unlike ferret, xapian gives excellent feedback about any mistakes you might have made while constructing them. If you have a nil exception somewhere in one of these related fields, xapian will complain and tell you exactly what line it’s bombing out.

I was also able to setup paging for my search results with paginating_find which I prefer to will_paginate (just a personal preference—nothing wrong with will_paginate). There is also a cool feature that will suggest other possible terms (“Did you mean?”) if your search returns no results. So far the only disappointment has been the lack of an obvious way to do searches on specific fields.

If you are in the market for a new full-text search solution for Rails, you should really give xapian a try.