Build a Smarter Telegram Bot: Integrating a RAG Pipeline for FAQ Answering

2025-12-01T00:00:00+00:00

In this post, we will show you how to build an intelligent FAQ bot using Python and the Telegram Bot API. We’ll go beyond simple commands by integrating a Retrieval-Augmented Generation (RAG) pipeline with LangChain.

This RAG pipeline lets our bot pull information from a custom knowledge base (in our case, a simple faqs.json file) and use a local Large Language Model (LLM) through Ollama to generate accurate answers. The best part? This approach (which works great with interfaces like Open WebUI) gives you full control over your models and data with zero API costs.

What is Telegram?

You’ve probably heard of Telegram—it’s a popular, cloud-based instant messaging app. It’s fast, works everywhere (mobile, web, and desktop), and has powerful features like huge group chats and easy file sharing.

One of its most powerful features for developers is the Telegram Bot API, an open platform that allows anyone to build and integrate automated applications (like ours!) directly into the chat interface.

A Warning on Privacy and Encryption

Before we build our bot, it is critical to understand how Telegram handles encryption, as it directly impacts user privacy.

Cloud Chats (The Default): All standard chats, group chats, and all bot interactions are “Cloud Chats.” These use server-client encryption. This means your messages are encrypted between your device and Telegram’s servers, and then stored (encrypted) on their servers. This is what allows you to access your chat history from any device. However, Telegram itself holds the encryption keys and can access this data.
Secret Chats (Manual): Telegram also offers “Secret Chats,” which are end-to-end encrypted (E2EE). In this mode, only you and the recipient can read the messages. Telegram has no access. However, bots cannot operate in Secret Chats.

This means that any message a user sends to our bot is a “Cloud Chat” and is not end-to-end encrypted. The data is accessible to Telegram and will be processed in plain text by our bot.py script on our server.

For this reason, you should never build a bot that asks for or encourages users to send sensitive private data such as passwords, financial information, or social security numbers. Always treat bot conversations as non-private.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is a technique that makes Large Language Models (LLMs) smarter by connecting them to external, private knowledge.

The Problem: An LLM like llama3 only knows the information it was trained on. It has no access to your company’s internal FAQs, new documents, or any private data.
The Solution (RAG): RAG solves this in two steps:
- Retrieve: When you ask a question, the system first retrieves relevant information from your own knowledge base (for us, our faqs.json file).
- Augment: It then augments the LLM’s prompt by pasting that retrieved information in as context, along with your original question.

In short, instead of just asking the bot “What’s the shipping policy?”, we’re effectively asking, “Based on this specific text: ‘…We offer standard shipping…’ — what is the shipping policy?” This forces the LLM to base its answer on our facts, not its own general knowledge, making the response accurate and reliable.

What you’ll build

Telegram bot
faqs.json knowledge base
RAG pipeline with local embeddings (FAISS) + LLM (Open WebUI)

Prerequisites

You’ll need:

Python 3.12+
a Telegram bot token (from BotFather)
access to an LLM via a locally hosted Open WebUI instance (OpenAI-compatible API)

Setting up the Project for Telegram Bot

uv is a high-performance Python package manager, so we’ll use it to set up our project. If you don’t have it installed, you can get it with or visit the site for installation steps:

pip install uv

Create a new project directory and navigate into it:

mkdir telegram-rag-bot
cd telegram-rag-bot

Initialize a new Python project.

uv init --bare

This command creates a minimal pyproject.toml file. This file will track our project’s metadata and, most importantly, its dependencies.

Create a virtual environment using uv:

uv venv

This will create a .venv directory. Activate it with the following:

source .venv/bin/activate

# On Windows, use
#.venv\Scripts\activate

Install the necessary Python packages using uv:

uv add python-telegram-bot python-dotenv langchain langchain-openai langchain-community faiss-cpu jq sentence-transformers

The key libraries are:

python-telegram-bot: For handling all Telegram communication
langchain: The primary framework for building the RAG pipeline
langchain-openai: Connector to Open WebUI’s OpenAI-compatible API
faiss-cpu: An efficient library for similarity search, used as a local vector store to quickly find relevant chunks of your FAQ data

Environment and configuration

The bot reads the Telegram token from the environment variable BOT_TOKEN. We can store it in a .env file as BOT_TOKEN=your-token-here.

# .env (Open WebUI)
# Open WebUI_URL must end with /v1 (e.g., http://localhost:3000/v1).
BOT_TOKEN=123456:abcdefg
Open WebUI_URL=http://localhost:3000/v1
Open WebUI_API_KEY=your_key_here

Inline mode requires enabling inline for the bot via BotFather.

Create a new file named bot.py and add the following code to set up and add message handlers for the Telegram bot.

import logging
import os
from uuid import uuid4
from telegram import Update, InlineQueryResultArticle, InputTextMessageContent
from telegram.ext import filters, MessageHandler, ApplicationBuilder, CommandHandler, ContextTypes, InlineQueryHandler
from dotenv import load_dotenv

# load .env variables
load_dotenv()
bot_token = os.getenv("BOT_TOKEN", "")

# Setup logging
logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    level=logging.INFO
)
logger = logging.getLogger(__name__)

async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text="I'm a bot, please talk to me!")

async def echo(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text=update.message.text)

async def caps(update: Update, context: ContextTypes.DEFAULT_TYPE):
    text_caps = ' '.join(context.args).upper()
    await context.bot.send_message(chat_id=update.effective_chat.id, text=text_caps)

async def inline_caps(update: Update, context: ContextTypes.DEFAULT_TYPE):
    query = update.inline_query.query
    if not query:
        return
    results = []
    results.append(
        InlineQueryResultArticle(
            id=str(uuid4()),
            title='Caps',
            input_message_content=InputTextMessageContent(query.upper())
        )
    )
    await context.bot.answer_inline_query(update.inline_query.id, results)

async def unknown(update: Update, context: ContextTypes.DEFAULT_TYPE):
    await context.bot.send_message(chat_id=update.effective_chat.id, text="Sorry, I didn't understand that command.")

async def document(update: Update, context: ContextTypes.DEFAULT_TYPE):
    if update.message.document:
        file = await update.message.document.get_file()
        file_name = update.message.document.file_name
        await file.download_to_drive(file_name)
    elif update.message.photo:
        # Get the largest photo size
        file = await update.message.photo[-1].get_file()
        file_name = f"photo_{file.file_unique_id}.jpg" # Create a unique name for photos
        await file.download_to_drive(file_name)
    elif update.message.video:
        file = await update.message.video.get_file()
        file_name = update.message.video.file_name
        await file.download_to_drive(file_name)
    else:
        await update.message.reply_text("Please send a document, photo, or video.")
        return

def main() -> None:
    start_handler = CommandHandler('start', start)
    echo_handler = MessageHandler(filters.TEXT & (~filters.COMMAND), echo)
    caps_handler = CommandHandler('caps', caps)
    inline_caps_handler = InlineQueryHandler(inline_caps)
    document_handler = MessageHandler(filters.PHOTO | filters.Document.PDF | filters.VIDEO, document)
    unknown_handler = MessageHandler(filters.COMMAND, unknown)

    application = ApplicationBuilder().token(bot_token).build()

    application.add_handler(start_handler)
    application.add_handler(echo_handler)
    application.add_handler(caps_handler)
    application.add_handler(inline_caps_handler)
    application.add_handler(document_handler)
    application.add_handler(unknown_handler)

    # Run the bot
    logger.info("Starting bot polling...")
    application.run_polling()

if __name__ == '__main__':
    main()

Try it out

To run the application, simply run:

uv run bot.py

Search for your bot name in Telegram and send the bot a message or command like /start or /caps.

What this bot does

Responds to /start with a greeting
Echoes back any plain text message (that isn’t a command)
Converts text to uppercase via /caps or inline mode
Downloads files users send (photos, PDFs, and videos) to local storage
Politely handles unknown commands

Core structure

start(): Sends a simple welcome message when the user runs /start
echo(): Replies with the exact same text the user sent (only for non-commands)
caps(): Turns the arguments after /caps into uppercase and sends them back
inline_caps(): Provides an inline result that uppercases whatever users type after @YourBotName in any chat
document(): Saves received media to disk:
- Photos: Downloads the largest size, naming it photo_<unique_id>.jpg
- PDFs: Downloads using the document’s file name. Note: The filter only accepts PDFs as documents
- Videos: Downloads using the video’s file name
- If none of these are present, it prompts the user to send a supported file
unknown(): Catches any unrecognized commands and replies with a friendly error

Handlers and filters

CommandHandler('start', start) and CommandHandler('caps', caps) handle commands
MessageHandler(filters.TEXT & (~filters.COMMAND), echo) ensures normal text (not commands) is echoed
InlineQueryHandler(inline_caps) answers inline queries
MessageHandler(filters.PHOTO | filters.Document.PDF | filters.VIDEO, document) restricts downloads to photos, PDFs, and videos
MessageHandler(filters.COMMAND, unknown) is added last to catch all other commands

Running the bot

main() wires up the handlers, builds the Application with the token, logs a startup message, and starts long polling via application.run_polling().

This script is a clean, async-first Telegram bot scaffold that demonstrates commands, inline mode, message filtering, and media downloads—ready to extend for more sophisticated behaviors.

Now that we have our bot ready, we will extend the code to add a RAG pipeline to the bot.

Setting up the knowledge base

Let’s set up a knowledge base by creating a file named faqs.json to hold our data. The RAG pipeline will load and search this content. An example structure is shown below.

[
 {
    "category": "General",
    "question": "What are your operating hours?",
    "answer": "Monday to Friday, 9:00 AM–5:00 PM (local time)."
  },
  {
    "category": "Accounts",
    "question": "How do I reset my password?",
    "answer": "Go to our website, click Login, then Forgot Password. Check your email for the reset link."
  }
]

Setting up the RAG Pipeline

The RAG pipeline is the engine that converts our static JSON file into a searchable brain for our bot. This part initializes once and creates a vector database. In simple steps,

Load faqs.json
Create embeddings
Store them in FAISS
When a user asks a question, find similar answers and ask the LLM to write a reply based only on those

Data Ingestion (Indexing the FAQs) handled by `setup_rag_chain()` method

This part happens once when the bot starts. We load the faqs.json file, create vector embeddings, and store them in a searchable database (FAISS).

Load Data: Read the faqs.json file
Embeddings: Use an embedding model (like HuggingFace (Sentence-Transformers) all‑MiniLM‑L6‑v2.) to convert the text into numerical vectors
Vector Store: Store these vectors in a FAISS index for fast retrieval

The RAG Retrieval Logic handled by `handle_message()` method

When a user asks a question:

Embed Query: The user’s question is converted into an embedding vector
Retrieve Context: The query vector is used to perform a similarity search against the FAISS index. This returns the top K most relevant FAQs (question and answer pairs)
Construct Prompt: A final prompt is built, containing the user’s question and the retrieved relevant context

Example Prompt Template:

You are an expert FAQ assistant. Use the following context to answer
the user's question. If the context does not contain the answer,
state that you cannot help with this specific question. Context:
[Retrieved FAQs] Question: [User's message]

Generate Response: The complete prompt is sent to our Open WebUI model via the OpenAI-compatible API, e.g. gpt-5 or another model exposed by Open WebUI, which generates a coherent, context-grounded final answer
Send to Telegram: The bot sends the LLM’s final response back to the user

New capabilities added to `bot.py`

RAG pipeline: Loads FAQs from a local JSON file, embeds them with HuggingFace, retrieves the most relevant entries via FAISS, and drafts answers with an LLM served by Open WebUI.
Inline UX polish: Sends a “typing…” chat action while the model thinks.
Persisted chain: The RAG chain is built once at startup and stored in bot_data for reuse across messages.

The core logic of our bot will revolve around an update to the standard message handler. When a user sends a question, the bot no longer looks for a simple command; instead, it passes the question to the RAG pipeline.

Try it out

To run the application, simply run

uv run bot.py

Search for your bot name in telegram and send the bot a message like What are your operating hours?.

Changes to the RAG pipeline setup are available here. The source code is available here.

By integrating a RAG pipeline, we’ve leveled up our Telegram bot from a simple command processor to a knowledge-aware assistant. This approach ensures our bot’s answers are accurate, grounded in our provided faqs.json data, and remain consistent, dramatically reducing the chance of “hallucinations” from the underlying LLM.

This architecture is powerful and scalable. To expand its capabilities, we only need to update the faqs.json file and re-run the indexing step—no need to retrain or modify the core LLM!

Adventures in Vibe-Coding with Replit

2025-10-29T00:00:00+00:00

A few weeks back, I tried out Replit Agent to see how viable it is as a development tool. I built two apps from scratch, which took 10–15 hours total, with a decent amount of human input along the way to clarify the agent’s questions.

Project 1: Multi-LLM code review API

The idea for the first project was to have a multi-LLM code review app, where you would pass a repository on GitHub or locally to an API, and it would run several full-repo code reviews, ranking and deduplicating them before returning a response. This setup should allow me to have a web frontend or CLI without much added complexity.

The prompt

Please build an app based on this app specification: app_spec.yaml (460 lines)

The results

After much back and forth (and about 40 agent-generated Git commits), I got this message:

🎉 SUCCESS! The Multi-LLM Code Review Assistant is now fully working!

Well, I actually got this message several times after sending just about any command to the agent. It’s very excited about its own work, even when it doesn’t work! In this case, the API connections were working, and testing with curl gave output, but I couldn’t get it to give more than one code review tip for a large codebase — not a very effective code-review app.

I tried asking the agent to diagnose and fix this:

So, this seems to not be returning very many code review suggestions. If I go
look at the repos I'm putting in, there are plenty of code issues that could be
returned, especially with documentation. How can we make this system find more
issues?

In response, Replit’s agent created a full React frontend for the API! Neat, but far from what I asked — the app still doesn’t return substantial code reviews.

So after ~$100 of agent credits, I found that Replit was not the tool for this app — I’m now trying to build a CrewAI app to accomplish the same thing (stay tuned for a post about that).

Project 2: End Point Ecommerce + Hugo site

A common theme in Replit apps we’ve experimented with is that it prefers to make everything a web app with a React frontend. I was curious how it’d do with a different web stack: The Hugo static site generator for the frontend and our recently launched End Point Ecommerce for the backend.

This is an interesting problem for an LLM, since Hugo is widely used (including on this site), but still small compared to React, Vue, and similar frameworks, and since End Point Ecommerce is well-documented but too new for any online discussion. However, it’s a modern .NET ecommerce framework and the code is readable and straightforward.

The prompt

Build me a hugo website frontend to interact with end point ecommerce:
https://ecommerce.endpointdev.com/

For the UI, make a halloween-themed storefront with orange text, black and grey
backgrounds, and halloween-themed test products. Use the Ballast font for
headers, and charter for body text for now.

After providing this initial prompt, I followed up with these:

Here is the github link for end point ecommerce, which this site will connect to:
https://github.com/EndPointCorp/end-point-ecommerce

Please make the app as simple as possible, while still working

Oh, and please use this demo API to connect to for now, so you don't have to
build any backend at all, just interface with this:
https://demo.ecommerce.endpointdev.com/swagger/index.html

The results (version 1)

It generated a nice, simple ecommerce storefront, using Hugo and End Point Ecommerce. Impressive! However, there’s one red flag: if you look at the products in the demo API we’re using, they are simple groceries — Apple, Banana, etc. So where did “Pumpkin spice treats,” “Spider Web Candy,” and “Ghost Marshmallows” come from?

It turns out, these are “fallback products” Replit had made when it failed to connect to the End Point Ecommerce demo API. Not great for an ecommerce site! Your users would see bogus but real-looking products when they’re unable to see or buy real products (I didn’t test this far, but the site may have even accepted payment info for fake products).

So I followed up:

Please remove the fallback products

The results (version 2)

This looks more promising! We get an error message written by the frontend. In production we might want to display more specific messages based on the response, but for this demo, I’m happy with this way of displaying errors.

In true “vibe-coding” fashion, I asked the Replit agent to diagnose the network error:

There is a network error while loading products, what is the issue?

It responded with a couple options, the main one being assume it’s a CORS issue.

Possible Solutions
Option 1: Server-Side Proxy (Recommended)
- Create a simple server endpoint that fetches the products
- Your frontend calls your server, your server calls the external API
- No CORS issues since server-to-server requests aren't restricted

I was suspicious of it being a CORS issue since everything was hosted in the Replit container, so after poking around myself, I wrote this prompt:

Please use the web api to sign up for an account by POSTing to the /api/User
end point and getting an API key

However, after reviewing the project for this post, I found that I misunderstood the issue in this prompt — no authentication is needed to GET /api/products on the demo API of End Point Ecommerce, the site was just disallowing this cross-origin request. Replit’s agent went ahead and implemented the CORS change anyway, though it didn’t correct my prompt.

After a few more followup prompts…

Please use the "basePrice", "discountAmount", and "discountedPrice" keys that
are returned from the API to show the real prices of the items

Now, please fetch the images from the API as well

…we have a decent working Hugo site connected with the End Point Ecommerce demo API!

Feature addition: cart

I wrote this prompt next, without having done my research on the backend (which made for an interesting unintentional experiment):

Please add a lightweight cart management system which uses localStorage, with
options to add to, remove from, or update quantity of items in the cart. Add a
simple checkout form with contact information, shipping address, and billing
address. For now, instead of payment information, just have a button which adds
a commonly used dummy credit card number.

There’s an issue here: I specified using localStorage, which is a fine way to set up a frontend cart system, but it ignores the existing system End Point Ecommerce already supplies. I hadn’t checked the documentation before making this prompt, and neither did Replit; it happily obliged and created a cart management system on the frontend. It’s nice that the agent did what I asked, but if I were trying to work with the agent, without cross-referencing documentation myself, I would be disappointed.

That’s the interesting part about using a non-deterministic system like AI agents: responding to one prompt, it might ignore my specific (and misguided) requests, but in another it might blindly execute the tasks it’s given. By the nature of LLMs, you can’t predict what you’ll get.

Bonus: a few extra notes

A couple final notes:

No .gitignore was generated, so the Hugo-generated public/ folder is tracked in Git
The agent used plain CSS, without SCSS or a framework — I’d recommend specifying the stack further in the prompt if you want an app that’ll be nice to maintain. You don’t want to assume the agent will make smart coding decisions on its own.

When I’d use Replit (and similar do-it-all AI agents)

I would use Replit for very quick-and-dirty web development, when sites which shouldn’t last a long time — although I’ve seen too many projects originally intended to be short-lived become important for many years…

Replit handles its containerized deployments well; if I needed to share a working demo site within a few hours, Replit would be a good pick. Especially if you like React frontends, even when you might not have asked for one! But if I was dealing with customer data, orders, or my own data I cared about, I would want to be heavily involved throughout the development process.

For projects where performance and accuracy matter (read: most projects), I will be taking a more directed approach. I’ve had some good results using Continue or GitHub CoPilot as a coding assistant for small tasks in existing repos, rather than letting AI taking the wheel completely.

Creating Agentic AI Applications

2025-10-14T00:00:00+00:00

Photo by Steve Johnson on Pexels

In the rapidly evolving world of AI, agentic AI is emerging as a game-changer. These systems go beyond simple chatbots or predictive models: they’re designed to act autonomously, make decisions, and interact with the real world to accomplish goals. In this blog post, we’ll dive into what agents are, explore agentic applications and orchestration, and walk through how to build your own agentic applications with open-source examples.

What Are Agents?

AI agents are autonomous entities that can perceive their environment, reason about it, and take actions to achieve specific objectives. Think of an AI agent as more than just a chatbot, it’s like a digital teammate that can look around, make decisions, and take action on its own. Unlike traditional AI models that respond passively to inputs, agents are proactive: they can break down goals into sub-tasks, use tools (like APIs or databases), maintain memory across interactions, and adapt based on feedback. This makes AI agents ideal for applications requiring independence, such as automation, research, or problem-solving.

At their core, agents typically integrate large language models (LLMs) like GPT-4 or Llama for reasoning, combined with mechanisms for tool invocation and state management. They can operate in loops, iteratively refining their approach until the goal is met.

Open-Source Examples of AI Agents

If you want to get hands-on with agents, there are already plenty of open-source projects you can experiment with and build on. Here are a couple of the most notable:

AutoGPT: One of the earliest and best-known autonomous agents. AutoGPT takes a high-level goal, breaks it down into steps, and tries to complete them—whether that means researching a topic, generating code, or managing a workflow. It comes with a simple frontend for building agents, plus a library of ready-made ones (including some quirky ones, like creating viral videos from Reddit trends).
AgentGPT: A browser-based platform that lets you spin up and deploy agents without extra setup. Just type in a goal and watch the agent go—browsing the web, running code, or connecting with external services to move toward the objective.

For a longer list of projects, check out this open-source AI agents directory on Hugging Face.

What Are Agentic Applications and Agent Orchestration?

An agentic application is simply an app that uses one or more AI agents to take on complex tasks by itself. Instead of coding every step by hand, you can plug agents together like modules to tackle real-world problems, whether that’s analyzing data, generating content, or building software. The word “agentic” really just highlights one thing: these systems can act on their own with little human input, following the prompts they are given.

When you bring more than one agent into the mix, you need a way to coordinate them. This is where orchestration comes in—getting multiple agents to work together smoothly. That might mean giving them specific roles (say, a researcher and a writer), setting up how they communicate, keeping track of what they’ve learned, and making sure they stay on task.

To make this manageable, orchestration frameworks provide ready-made structures, like graphs, crews, or conversation flows, that organize the collaboration. With these in place, it becomes much easier to grow from a single helpful agent to an entire team working together.

Open-Source Examples of Agentic Applications and Orchestration

If you want to start building agentic applications yourself, there are already a number of open-source frameworks that can help. Here are some of the most popular:

CrewAI: A lightweight Python framework for orchestrating agents that work together in teams. You can assign roles, define tasks, and let the agents collaborate—without the bulk of larger frameworks like LangChain. It’s simple, fast, and flexible enough for everything from small experiments to enterprise automations.
AutoGen: Created by Microsoft, AutoGen is all about multi-agent conversations. Agents can chat with each other, work independently, or take direction from a human. It also supports tool use like code execution and web browsing, making it handy for more specialized domains like math or chemistry.
LangGraph: Part of the LangChain ecosystem, LangGraph lets you design agents as graphs, which makes them resilient and stateful. It’s well-suited for long-running workflows, offering built-in memory, human-in-the-loop controls, and debugging support through LangSmith—perfect for production-grade projects.

Other frameworks worth exploring include OpenAI’s Agents SDK (built and evolved upon Swarm) for multi-agent workflows, and MetaGPT, which simulates role-based teams of agents.

Writing Agentic Applications

To build an agentic application, the first step is choosing the right framework for your needs, for instance, CrewAI if you want simplicity and lightweight orchestration, or AutoGen if you need extensibility and layered APIs.

Once you’ve selected a framework, you’ll need to:

Define agent roles, e.g., researcher, writer, reviewer. Each agent is an actor to complete one or more tasks.
Equip agents with tools, like search APIs, code interpreters, or custom functions. As you can see, theoretically there is no limit on what you can do with agents.
Assign tasks and orchestration rules, deciding how agents will interact, share state, and collaborate.
Add safeguards, error handling, memory management, and human oversight to ensure reliability and trustworthiness.

With those elements in place, you can start assembling workflows. Below, we’ll look at how to get started with one of the most widely used frameworks (CrewAI).

Building with CrewAI

First, install CrewAI and ddgs (duckduckgo-search) which is a free search tool that I used for this example.

pip install crewai 'crewai[tools]' ddgs

CrewAI uses decorators for agents and tasks. Below is a simple example of a crew for researching and reporting on a given topic. It defines two agents (a researcher and a reporting analyst) and one task for each agent (research task and reporting task). This code creates a sequential workflow:

The researcher gathers data
Then, the analyst compiles a report

You can extend it with more agents with more tasks or parallel processes.

To run this script, you need to set your LLM API key. See how to do that in the crewAI docs.

from typing import Type, Optional
import json

from crewai import Agent, Crew, Process, Task
from crewai.project import CrewBase, agent, crew, task
from crewai.tools import BaseTool
from pydantic import BaseModel, Field

from ddgs import DDGS

class DDGSearchInput(BaseModel):
    query: str = Field(..., description="Search query")

class DDGSearchTool(BaseTool):
    name: str = "DuckDuckGo Search"
    description: str = "Web search via DuckDuckGo. Returns a JSON list of results."
    args_schema: Type[BaseModel] = DDGSearchInput

    # declare this as pydantic field
    max_results: int = Field(8, description="Maximum number of results to return")

    def _run(self, query: Optional[str] = None, **kwargs) -> str:
        if query is None:
            query = kwargs.get("query")

        if not query:
            return json.dumps([], ensure_ascii=False)

        results = []
        with DDGS() as ddgs:
            for r in ddgs.text(query, max_results=self.max_results):
                results.append({
                    "title": r.get("title"),
                    "url": r.get("href"),
                    "snippet": r.get("body"),
                })
        return json.dumps(results, ensure_ascii=False)

@CrewBase
class ResearchCrew():
    """
    Crew of agents for researching and reporting on given topic.
    """

    def __init__(self, topic):
        self.topic = topic

    @agent
    def researcher(self) -> Agent:
        return Agent(
            role='Senior Data Researcher',
            goal=f'Uncover cutting-edge developments in {self.topic}',
            backstory=f'Seasoned researcher skilled in finding relevant information about {self.topic}.',
            verbose=True,
            tools=[DDGSearchTool(max_results=10)]
        )

    @agent
    def reporting_analyst(self) -> Agent:
        return Agent(
            role='Reporting Analyst',
            goal=f'Create detailed reports from {self.topic} research findings',
            backstory=f'Meticulous analyst who turns {self.topic} data into clear reports.',
            verbose=True
        )

    @task
    def research_task(self) -> Task:
        return Task(
            description=f'Conduct thorough research on {self.topic}.',
            expected_output='A list of 10 bullet points with key findings.',
            agent=self.researcher()
        )

    @task
    def reporting_task(self) -> Task:
        return Task(
            description=f'Expand the research on {self.topic} into a full report.',
            expected_output='A markdown-formatted report with detailed sections.',
            agent=self.reporting_analyst(),
            output_file=f'ai_report-{self.topic}.md'
        )

    @crew
    def crew(self) -> Crew:
        """Assembles the crew."""
        return Crew(
            agents=[self.researcher(), self.reporting_analyst()],
            tasks=[self.research_task(), self.reporting_task()],
            process=Process.sequential,
            verbose=True
        )

## run the crew of agents
if __name__ == '__main__':
    topic = input('Topic: ')
    crew = ResearchCrew(topic).crew()
    result = crew.kickoff(inputs={'topic': topic})
    print(result)

Building A Dynamic Agent

We can follow a dynamic approach to create an agentic AI application as well. In this approach, without other 3rd party agent orchestration frameworks we can dynamically create task and agentic processes. Here I present a simple dynamic agent which can work on any given goal until the goal is met. I used OpenAI as the LLM in this example. Using the dynamic approach, the agent crew extracts sub-goals from the given main goal. Then it works on tasks and decides if the goals are met.

import os
import json
from openai import OpenAI

class DynamicAgent:
    def __init__(self, goal: str, model: str = "gpt-4o"):
        self.goal = goal
        self.model = model
        self.llm = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
        self.memory = []  # store context and past decisions
        self.max_iterations = 5  # prevent infinite loops

    def get_llm_response(self, prompt: str) -> str:
        resp = self.llm.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": "You are an autonomous agent that creates plans and conditions to achieve goals."},
                {"role": "user", "content": prompt}
            ],
            temperature=0.2,
        )
        result = None
        try:
            result = resp.choices[0].message.content
        except Exception:
            # fallback
            result = str(resp)
        print(f"llm result: {result}")
        return result

    def generate_plan(self) -> list:
        prompt = (
            f"Given the goal '{self.goal}', generate a pure JSON list of actionable steps with conditions for success."
            "Don't use any formatting like markdown outside of the JSON content.\n"
            "Each step should include: {'step': 'description', 'condition': 'success criteria'}.\n"
            "Do not rely on predefined rules; infer the steps and conditions from the goal."
        )
        plan = json.loads(self.get_llm_response(prompt))
        self.memory.append({"action": "planning", "output": plan})
        return plan

    def execute_step(self, step: dict) -> str:
        prompt = (
            f"Execute this step: {step['step']}.\n"
            f"Success condition: {step['condition']}.\n"
            "Provide the result and whether the condition was met."
        )
        result = self.get_llm_response(prompt)
        self.memory.append({"step": step['step'], "result": result})
        return result

    def run(self):
        print(f"Starting agent with goal: {self.goal}")
        plan = self.generate_plan()
        results = []
        for i, step in enumerate(plan[:self.max_iterations]):
            print(f"Executing step {i+1}: {step['step']}")
            result = self.execute_step(step)
            results.append(result)
            print(f"Result: {result}")
            # check if goal met
            if "Goal achieved" in result:
                break
        self.save_results_to_markdown(results)

    def save_results_to_markdown(self, results: list):
        filename = f"dynamic_agent_report-{self.goal}.md"
        with open(filename, "w", encoding="utf-8") as f:
            f.write("\n".join(results))
        print(f"Results saved to {filename}")

if __name__ == "__main__":
    goal = input("Goal: ")
    agent = DynamicAgent(goal=goal)
    agent.run()

Use Cases of Agentic AI Applications

Agentic AI applications are already finding their way into practical scenarios. By combining autonomy with orchestration, these systems go beyond demos and research projects to deliver measurable impact.

Customer Support Automation: Instead of static chatbots, agentic systems can act as full-service assistants—resolving customer queries, escalating complex cases, and even initiating refunds or ticket creation. They maintain context across long interactions, providing a more human-like experience.
Research and Analysis: Enterprises and individuals use research agents to autonomously scan news sources, academic databases, or financial reports. For example, a financial analyst could deploy agents to track company earnings, summarize investor calls, and produce actionable insights daily.
Content Creation Pipelines: Marketing teams benefit from multi-agent setups where one agent researches a topic, another drafts a blog post, and another optimizes it for SEO. The result is faster, higher-quality content production with less manual effort.
Software Development Assistants: Multi-agent workflows can handle bug triaging, code generation, testing, and documentation. For instance, one agent detects issues, another suggests fixes, while another runs automated tests and updates docs.
Healthcare and Biomedicine: Specialized agents can search the latest medical literature, cross-reference patient data, and generate preliminary diagnostic reports, supporting clinicians in decision-making while reducing information overload.

At End Point, we use agents to speed up development, using human review to keep our company’s long-standing commitment to high-quality products. We also build agentic AI solutions for clients, speeding up and modernizing legacy workflows.

Challenges Ahead

While agentic AI holds promise, several challenges remain before widespread adoption becomes seamless:

Reliability and Hallucination: LLM-based agents can still generate incorrect or fabricated outputs. Ensuring trustworthiness through validation, feedback loops, and human-in-the-loop oversight is crucial. But, personally I am not against hallucination — We humans hallucinate too.
Security and Safety: Autonomous agents that can browse the web, execute code, or control external systems pose risks if misconfigured or exploited. Sandboxing, permissions, and guardrails are essential to prevent harmful actions.
Evaluation and Benchmarking: Unlike traditional ML models, measuring the success of agentic systems is difficult. Metrics must capture not only accuracy but also task completion, collaboration quality, and user satisfaction.
Ethics and Alignment: As agents gain autonomy, aligning them with human values, legal frameworks, and ethical principles becomes critical. Misaligned objectives could cause unintended consequences.
Integration with Legacy Systems: Many organizations still rely on older infrastructure. Seamlessly embedding agentic AI into these environments can be a non-trivial engineering challenge.

Wrapping up

Agentic AI applications represent the future of intelligent software, blending autonomy with collaboration. By leveraging open-source tools like CrewAI and LangGraph or specialized custom agents, you can create powerful solutions tailored to your needs. As AI evolves, we might expect even more sophisticated orchestration capabilities.

Making Blog Search Smarter with LLMs and Open WebUI

2025-09-29T00:00:00+00:00

We recently released LLM Expanded Search for our blog’s vector search. It builds on what we covered in our earlier posts about AI-powered search and vector search basics. Here’s how we built it with our internal AI setup (Open WebUI running an OpenAI-compatible API), why it makes search better, and what’s coming next.

What “LLM Expanded Search” actually does

Here’s the basic idea: when you search for something, we first ask an LLM to come up with related terms and phrases. Then we search for all of those terms, not just your original query.

Your search gets expanded by an open-source LLM through our AI portal (Open WebUI with an OpenAI-compatible API)
Those extra terms give our vector index more ways to find posts that match what you’re looking for
We combine the results, remove duplicates, and sort by relevance before showing the best matches with snippets and links

This really helps with short or vague searches where regular vector search might miss the relevant context — for example, “S3” refers to Amazon S3, which is a cloud object storage system, so whereas “S3” doesn’t provide enough context for a useful vector search. An LLM can expand this short search and include context about cloud object storage in general, as well as give enough context to return results about S3.

How it works

The frontend is pretty straightforward: our search bar has two options, “Search” (just hit Enter) and “LLM Expanded Search” (Shift/Ctrl/Command+Enter).

When you use expanded search, here’s what happens:

We call our Open WebUI endpoint with a prompt that asks for 8–15 related terms
We turn both your original query and the expanded terms into embeddings
We search our vector store with all these terms and combine the results
Caching and rate limiting keep things fast and cheap

Here’s a simple example of how we expand queries:

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.getenv("OPENAI_BASE_URL"),   # e.g., http://openwebui.local/api/v1
    api_key=os.getenv("OPENAI_API_KEY")      # token managed in your environment
)

def expand_query(raw_query: str) -> list[str]:
    messages = [
        {
            "role": "system",
            "content": (
                "You expand a short search query into a concise, comma-separated list of "
                "synonyms and closely related phrases (8–15 items). No explanations."
            )
        },
        {"role": "user", "content": raw_query}
    ]
    res = client.chat.completions.create(
        model=os.getenv("OPENAI_MODEL", "local-llm"),
        messages=messages,
        temperature=0.2,
        max_tokens=200,
    )
    text = res.choices[0].message.content
    return [t.strip() for t in text.split(",") if t.strip()]

After that, we embed the original query and the expanded terms, search the vector index, then sort by score and drop duplicates so each post appears once. Finally, we render concise snippets.

For example, after a similarity search you can rank and de-duplicate like this:

# given: results = [(doc, score), ...]
valid = [(d, float(s)) for d, s in results if float(s) > 0.05]
valid.sort(key=lambda x: x[1], reverse=True)  # highest score first

seen = set()
unique = []
for doc, score in valid:
    src = doc.metadata.get("source", "")
    if src not in seen:
        unique.append((doc, score))
        seen.add(src)

# unique now holds top ranked, de‑duplicated posts

Why we chose Open WebUI

A few reasons made Open WebUI the right choice:

It’s open source and works great self-hosted
The OpenAI-compatible API means we can drop it into existing code
We can use whatever models and inference backends we want
It’s easy to experiment with different prompts and workflows

What’s next: Moving more into Open WebUI

We’re looking into moving more of the search pipeline directly into Open WebUI workflows:

Query expansion (LLM)
Vector retrieval (custom tool that hits our index)

This would give us tighter integration, fewer network calls, and simpler deployment, and make it easier to try new approaches.

What you’ll notice when using it

Short searches work way better, you get more relevant results and fewer dead ends
It’s still experimental, so sometimes results might drift into related topics. Stick with regular “Search” if you want more exact matches
We cache common terms to keep things smooth

Give it a try at our blog. Just use the search bar in our header: press Enter for regular search, or Shift/Ctrl/Command+Enter for LLM Expanded Search.

Want to know more about why we built this? Check out the announcement and vector search posts linked above.

If you’re interested in setting up LLM-expanded vector search or running something similar self-hosted with Open WebUI, we’d love to help out.

Vector Search for the End Point Blog

2025-08-13T00:00:00+00:00

We’re excited to announce a new feature on the End Point Blog: AI-powered vector search.

Below the “Our Blog” header at the top of this page, there is a new search bar with two adjacent buttons: “Search” and “LLM Expanded Search.” If you click “Search” (or press Enter), your search will be fed directly to our vector search/similarity search engine. If you click “LLM Expanded Search” (or press Shift+Enter, Control+Enter, or Command+Enter on macOS) your query will first be expanded by an open-source LLM, then sent to the similarity search engine.

The LLM is trained to expand the query to include similar terms, keywords, etc., before sending it to the similarity search engine. For example, if I search S3, similarity search alone returns no results — there isn’t enough semantic information for vector search to make useful connections. However, an LLM can expand this to s3, simple storage service, amazon s3, object storage, cloud storage..., providing more anchor points for vector search to connect to results.

The model improves results fairly well, but it is still an experimental technology, so results will vary. In the S3 example, you could get posts which don’t directly relate to Amazon S3, but relate to cloud object storage, or you could not get some posts which have an exact match of S3 but not more semantic similarity.

You can read more about the technical side of vector search in our recent blog post.

Happy searching!

Vector Search: The Future of Finding What Matters

2025-07-01T00:00:00+00:00

Photo by Ann H on Pexels

In a world flooding with data in several different formats like images, documents, text, and videos, traditional search methods are starting to not be modern anymore. Today, the vector search technique is revolutionizing how we retrieve and understand information. If you wonder how Spotify can recommend the perfect song or how Google can find almost perfectly accurate image matches, vector search is kind of the wizard behind the curtain. Let’s see how it has become a game changer.

What Is Vector Search?

At its core, vector search is a method of finding similar items in a dataset by representing them as vectors — essentially, lists of numbers in a multi-dimensional space. Unlike keyword-based search, which relies on exact matches or predefined rules, vector search focuses on semantic similarity. This means it can understand the meaning or context behind data, not just the words or pixels on the surface.

Imagine you’re searching for a cozy cabin in the woods. A traditional search might get stuck on the exact words in this query, missing a listing for something similar like a snug retreat nestled in a forest. Vector search, however, can connect the dots because it understands that cozy and snug or woods and forest are conceptually close.

You can try vector search on this blog! There’s a search bar in the header at the top of the page, and you can see our announcement post here.

How Does It Work?

The magic of vector search happens in three key steps:

Embedding Generation

Once you have access to the data sources that you want to apply vector search, you will need to prepare and ingest the data into your system. This phase will handle data clean-up and extraction from different data sources and formats. For multi-modality support you may need to include multiple data formats like text, image, audio, and video.

In the embedding generation phase, as depicted above, raw data (like text, images, or audio) is transformed into vectors using suitable embedding machine learning (ML) models. These models (e.g. all-MiniLM-L6-v2, all-mpnet-base-v2 for text, or ResNet for images) analyze the data and spit out a numerical representation, a vector, that captures its essence. For example, the sentence I love sunny hikes might become something like [-0.07571851, -0.02147608, 0.07130147, 0.1087752, 0.02052169, ...], where each number reflects a feature of its meaning.

Storing Vectors

The generated vectors are then stored in a vector database or an index optimized for fast retrieval. Think of it as a massive, multi-dimensional map where every point represents an item or a chunk of an item. There are several options for your choice like FAISS, Chroma, Milvus, and pg-vector.

Similarity Search

When you use this system with a query like find me green hoodies, it converts your input into a vector too. Then, it searches the vector store for the closest points using mathematical distance metrics like Cosine Similarity or Euclidean Distance. The closer the vectors, the more similar the items.

The result? You get matches that feel right, even if they don’t share exact keywords or pixel patterns.

A Simple Vector Search Implementation

Here, to demonstrate how vector search works, I implemented a simple vector search based on this architecture without using third-party libraries. The best practice is surely using well-known implementations and tools to achieve production ready applications; this is just for demonstration.

Below is a very simple vector store implementation:

import numpy as np
from typing import List, Tuple

class SimpleVectorStore:
    def __init__(self):
        """Initialize"""

        self.vectors = []
        self.ids = []

    def add_vectors(self, vectors: List[List[float]], ids: List[str]):
        """Add vectors to vector store"""
        if len(vectors) != len(ids):
            raise ValueError("Number of vectors must match number of IDs")

        vectors = np.array(vectors)
        self.vectors.extend(vectors)
        self.ids.extend(ids)

        self.vectors = np.array(self.vectors)

    def cosine_similarity(self, v1: np.ndarray, v2: np.ndarray) -> float:
        """Calculate cosine similarity between two vectors"""

        # zero vectors
        if np.all(v1 == 0) or np.all(v2 == 0):
            return 0.0

        dot_product = np.dot(v1, v2)
        norm_v1 = np.linalg.norm(v1)
        norm_v2 = np.linalg.norm(v2)
        return dot_product / (norm_v1 * norm_v2)

    def search(self, query_vector: List[float], k: int = 5) -> List[Tuple[str, float]]:
        """Search for the k most similar vectors"""

        if len(self.vectors) == 0:
            return []

        query_vector = np.array(query_vector)
        similarities = [
            (self.vectors[i], self.ids[i], self.cosine_similarity(query_vector, vec))
            for i, vec in enumerate(self.vectors)
        ]

        # sort and return
        similarities.sort(key=lambda x: x[2], reverse=True)
        return similarities[:k]

Now, let’s store some sample vectors (so-called embeddings) to this simple vector store and perform an example search. In real-world applications this represents the raw data in vector form, but I used dummy embedding values in this example. I also excluded the embedding model phase when creating these embedding values to simplify the code.

if __name__ == "__main__":
    # vector store instance
    vector_store = SimpleVectorStore()

    # example embeddings
    sample_embeddings = [
        [1.0, 0.0, 0.0], # Vector 1
        [0.0, 1.0, 0.0], # Vector 2
        [1.0, 1.0, 0.0], # Vector 3
        [0.0, 0.0, 1.0], # Vector 4
    ]
    sample_ids = ["doc1", "doc2", "doc3", "doc4"]

    # add sample vectors to vector store
    vector_store.add_vectors(sample_embeddings, sample_ids)

    # query vector
    query = [1.0, 0.5, 0.0]

    # search for top k = 2 results
    top_k = 2
    results = vector_store.search(query, k=top_k)

    print("Query vector:", query)
    print(f"Top {top_k} similar vectors:")
    for vector, doc_id, similarity in results:
        print(f"DocId: {doc_id}, Vector: {vector}, Similarity: {similarity:.4f}")

The output of this search is like below:

Query vector: [1.0, 0.5, 0.0]
Top 2 similar vectors:
DocId: doc3, Vector: [1. 1. 0.], Similarity: 0.9487
DocId: doc1, Vector: [1. 0. 0.], Similarity: 0.8944

Now compare the vectors in the results with the query vector. As you can see the bigger similarity scores are the closer ones and are more similar to the query vector.

Why Vector Search is Powerful

Vector search offers several advantages over traditional search methods. First of all, it has a degree of semantic understanding by focusing on contextual meaning rather than keywords: vector search handles synonyms, misspellings, and contextual nuances. For instance, searching car could return results about automobiles or vehicles (assuming you have a well-trained model).

Multimodal capability is another feature which is commonly used today. Vector search isn’t limited to text. It can process images, audio, video, or even combinations of these. For example, you could search for sunset photos using a text query and retrieve visually similar images.

Vector search can incorporate user preferences or behavior like past searches into the vector space and deliver personalized results.

Real-World Examples

Vector search is already everywhere:

Retrieval-Augmented Generation (RAG)

Regardless of the industry that a RAG application is designed for, vector search is essential in the retrieval (“R”) phase: it is used to find relevant data for a query to build a context for the LLM.
Ecommerce:

Vector search is used for recommending products based on user queries or visual similarity (e.g. “find shoes like these” using an image). Example: Searching for “red sneakers” might return visually similar items even if the product description doesn’t mention “red sneakers”.
Content Discovery:

Used for powering recommendation engines for streaming platforms (e.g. Netflix, Spotify) by finding movies, songs, or articles similar to a user’s interests. Example: Suggesting a sci-fi movie based on a user’s love for “Star Wars”.
Customer Support:

Used for enabling chatbots to retrieve relevant knowledge base articles or FAQs by understanding the intent behind a user’s question. Example: A query like “how to reset my device” could pull up guides even if phrased differently.
Image and Video Search:

Allows users to search for visually similar images or videos, such as finding artwork or stock footage that matches a specific style. Example: Uploading a photo of a beach to find similar vacation destinations.
Enterprise Search:

Helps employees find documents, emails, or internal resources by understanding the context of their queries. Example: Searching “project timeline” could retrieve relevant spreadsheets or emails.
Healthcare:

Used for matching patient records, medical images, or research papers based on semantic similarity to aid diagnosis or treatment planning. Example: Finding studies related to a specific disease even if terminology varies.

Challenges Ahead

Although computational cost is one of the challenges of performing similarity search on large datasets I won’t mention it as a major challenge.

The main challenges in vector search seem to revolve around interpretability and data quality, as is common in ML. Unlike keyword search, where matches are explicit, vector search operates in a black box, making it harder to explain why certain results were returned.

As for data quality, the adage applies: garbage in, garbage out. If the input data is noisy or incomplete, the embeddings may not accurately represent the content. Data extraction and creating the embeddings of certain data formats (e.g. tables) is another challenge. Searching for accurate table data values is an especially challenging task. At End Point we are using advanced methods and LLM services to overcome these challenges when preparing quality embeddings.

Wrapping Up

Vector search isn’t just a tech buzzword — it’s a fundamental shift in how we interact with data. By moving beyond rigid keywords to a world of meaning and similarity, it’s unlocking possibilities we’re only beginning to explore. Whether you’re a developer building the next big app or just a curious soul, vector search is worth keeping an eye on. It’s not about finding exactly what you typed — it’s about finding exactly what you meant.

Deploying LLMs Efficiently with Mixture of Experts

2025-06-02T00:00:00+00:00

1. Why MoE?

Modern language models can have hundreds of billions of parameters. That power comes with a cost: high latency, high memory, and high energy use. Mixture‑of‑Experts (MoE) tackles the problem by letting only a few specialised sub‑networks run for each token, cutting compute while keeping quality.

In this post you’ll get:

A short intro to MoE
A simple diagram that shows how it works
A look at Open‑Source MoE Models
A quick guide to running one on your own machine with Docker + Ollama
Deployment tips and extra resources

2. Key Ideas

Term	Quick meaning
Dense model	Every weight is used for every token.
Expert	A feed‑forward network inside the layer.
Router	Tiny layer that scores experts for each token.
MoE layer	Router + experts; only the top‑k experts run.
Sparse activation	Most weights sleep for most tokens.

Analogy: Think of triage in a hospital. The nurse (router) sends you to the right specialist (expert) instead of paging every doctor.

3. How a Token Moves Through an MoE Layer

         Input Token
              │
              ▼
         ┌────────┐
         │ Router │  (scores all experts)
         └────────┘
              │  selects top‑k
              ▼
 ┌────────┐   ┌────────┐
 │Expert 1│… │Expert k│   (inactive experts ≈ greyed‑out)
 └────────┘   └────────┘
              │
              ▼
        Combined Output

The router scores all experts.
It picks the best one or two.
Only those experts process the token.
Their outputs are combined and passed to the next layer.
During training, a small penalty is added so the router spreads tokens evenly among the experts.

4. Why It Saves Compute

Fewer active weights: DeepSeek‑R1 activates only 6 % of its weights per token (so 94 % stay idle), while Grok‑1 activates about 25 %. Because fewer weights run, the model performs fewer multiply‑add operations, directly cutting computation time and energy.
Scale without extra cost: You can add more experts to grow the model’s capacity, and the router still activates only a few per token—so compute cost and latency remain almost unchanged.
Focused fine‑tuning: You can fine‑tune a single expert to adapt the model to a new topic.

Example: Mixtral‑8×7B runs only 13 B parameters per token yet matches Llama‑2‑70B, while generating ~6× faster on the same GPU.

5. Quick Start with Ollama

Run a Mixture‑of‑Experts model in one line:

ollama run mixtral:8x7b "Why is MoE efficient?"

If the model is not yet on your machine, Ollama will download a quantized copy automatically.

Need Ollama? Install it either way:

Native binary (macOS/Linux/Windows): curl -fsSL https://ollama.com/install.sh | sh → then use the ollama run command above.

Docker container:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker exec -it ollama ollama run mixtral:8x7b "Why is MoE efficient?"

Both methods expose a local REST endpoint on port 11434, so you can integrate the model into scripts or back‑end services.

Open‑Source MoE Models — At a Glance

Entry‑level (single‑GPU): Mixtral‑8×7B, Qwen3‑30B‑A3B — fit in 12–16 GB of VRAM and are ideal for prototyping.
Mid‑range (workstation‑class): Mixtral‑8×22B, DeepSeek‑R1‑32B — need ~32 GB of VRAM and provide near‑frontier accuracy with long context windows.
Research‑scale (multi‑GPU): Grok‑1, DeepSeek‑R1‑671B — require 64 GB+ of VRAM or multi‑GPU clusters but offer state‑of‑the‑art performance.
Models ship in a space‑saving 4‑bit form, so they use about half the memory of the standard 16‑bit (FP16) version—helpful if your GPU VRAM is tight.
Start with an entry‑level model, validate your pipeline, and scale up only when the use‑case justifies the added cost.

6. Deployment Tips

Check GPU memory first. All model weights must fit into GPU VRAM during inference. If they don’t:
- Use the 4‑bit download — it needs roughly half the memory with only a small quality trade‑off.
- Off‑load to CPU RAM — frameworks such as DeepSpeed‑MoE or vLLM can park less‑used weights on the CPU; throughput drops, but the model still runs.
Spread the work. While fine‑tuning, watch router stats to confirm every expert is being used; add a load‑balancing loss if a few dominate.
Batch your prompts if experts sit on different GPUs or machines. When the model has to jump between devices, every prompt makes a short “network trip.” Sending many prompts together means fewer trips, so the overall run is faster.

Useful links

7. Takeaway

Mixture‑of‑Experts lets you keep big‑model quality without the big‑model bill. Thanks to open models and tools like Ollama, you can spin up an MoE LLM on a single machine, test your ideas, and scale when you’re ready.

Got questions or feedback? Drop a comment below.

Converting MIDI to KML using AI: Bach’s Notes in the Hills of Greenland

2025-05-02T00:00:00+00:00

I have always been interested in ways of representing music visually. Aside from conventional music notation, I imagined other cross-modal generation methods that could take a sound and generate an image. In the same vein, I have frequently envisioned a 3D landscape in which you could discover musical “objects”.

Well, now I’ve realized a version of this dream — with caveats which will be mentioned later. In this blog I would like to demonstrate how I used AI (in my case ChatGPT using GPT-4 Turbo) to create an interesting JavaScript application from just a few phrases. In this case, we will be making an application that can take as input an existing piece of music represented by a MIDI file and as output, create a KML file that you can view as 3D objects somewhere on the globe.

Here is how I enlisted ChatGPT to help me:

please make a javascript application that can take a MIDI file and covert it to extruded polygons in a kml file

Here is a part of its response:

I was amazed. It included code to select the MIDI file, convert it to KML, and generate an output file. Plus, ChatGPT correctly interpreted my request despite my “covert” typo. :-)

Before testing it I was interested in having the color of the extruded polygon be dependent on the pitch of the note. So next I entered:

please change the color of the extruded polygon dependent on the pitch

Here is what it said:

It was implemented using npm and Node.js, which seemed excessive for this small application. To make an easier-to-run version in a single html file, I entered:

please rewrite it so it is in an html page without node.js

I thought it would be nice to be able to place the polygons, easily, wherever you wanted to, in the world.

I asked it to take a Google Street View position from Google Maps and decode the latitude, longitude, and heading.

Next, I asked it to make a nicer user interface. It “understood” the purpose of the application very well and came up with a very nice interface explaining what it does.

How to try out the application

You can try out a version of the application at https://darius.endpointdev.com/midi2kml/midi2kml_improved.html.

Find a nice place to have the MIDI data displayed. Copy the URL and paste it into the Street View URL field.

For example, here is a place in Greenland:

And here’s one on the beach in Texel in the Netherlands:

Find a nice MIDI file like Bach’s prelude and fugue in C major. Download it and put it in the “choose MIDI file” field
Press “Convert to KML”
Click on the “download KML” link
Open the KML file in a KML viewer or by manually by loading it into a program like Google Earth or Cesium.

Issues

This application works at least partially — it puts triangular prisms for each MIDI track, over time, and shows different colors for different notes (though the color–note correlation is not clear). There are plenty of issues and questions, however:

There are lots of overlapping polygons. It’s unclear whether these are just errors, or represent different dynamics or articulations
The “Play MIDI” button doesn’t work — it plays tones of some kind, but in my testing it was either a single bell sound, or computer noise reminiscent of an AOL dial-up modem.
When I tested it on an orchestral score (Mozart’s Requiem, Kyrie), the instrument label pins were off to the side, not indicating which line of notes they corresponded to, and were mostly labeled “acoustic grand piano,” which is not an instrument included in the MIDI file.

Using AI for blog post image processing

As an aside, I created this blog using Google Docs with PNG pictures embedded in it. Our blog structure requires a Markdown document with separate WebP images. Google Docs has a nice Markdown export function, but it converts the PNG images into embedded Base64-encoded PNG. So I asked ChatGPT to extract the Base64-encoded PNG and create WebP files. It created a Python script that did this perfectly:

import os
import base64
import re
from PIL import Image
from io import BytesIO

# Path to your Markdown file
md_file_path = "example.md"

# Directory to store extracted and converted WEBP images
output_dir = "converted_images"
os.makedirs(output_dir, exist_ok=True)

# Load the Markdown content
with open(md_file_path, "r") as md_file:
    content = md_file.read()

# Regex to find base64-encoded PNGs in the Markdown

base64_pattern = re.compile(
    r"\[.*?\]:\s*<data:image/png;base64,([A-Za-z0-9+/=]+)>"
)

# Initialize a counter for naming the files
counter = 1

for match in base64_pattern.finditer(content):

    print(f"Match found: ")
    base64_data = match.group(1)
    print(base64_data)

    # Decode the base64 string to a PNG image
    png_data = base64.b64decode(base64_data)
    png_image = Image.open(BytesIO(png_data))

    # Save the PNG as a WEBP file
    webp_name = f"image_{counter}.webp"
    webp_path = os.path.join(output_dir, webp_name)
    png_image.save(webp_path, "WEBP")

    # Replace the base64 PNG in the Markdown with a reference to the WEBP file
    content = content.replace(match.group(0), f"![Image {counter}]({webp_path})")
    counter += 1

# Save the updated Markdown file
updated_md_path = os.path.splitext(md_file_path)[0] + "_updated.md"
with open(updated_md_path, "w") as updated_md_file:
    updated_md_file.write(content)

print(f"Updated Markdown file saved: {updated_md_path}")
print(f"WEBP images saved in: {output_dir}")

Unfortunately, when Google Docs embeds the images, it seems to downsize the resolution, so for this post, old-school image processing it is!

Introducing the End Point AI Portal for Our Customers

2025-03-12T00:00:00+00:00

Today, End Point is opening up the End Point AI Portal for our customers to use. As a technology consulting company that gets in the trenches with our customers to integrate, manage, and support the information technology they need to operate and transform their businesses, we believe this is a great service for us to provide.

We are offering our AI Portal to our customers at no charge. The price is certainly right, so we think our customers are very much going to want to use it. 🙂

What’s in the Portal

End Point’s AI Portal provides an interface to interact with many different LLM AI chatbot services all in one place. It even gives users the ability to simultaneously submit queries and instructions to multiple LLMs and see their responses side by side.

Our Portal is a customized implementation of the Open WebUI AI interface and connects via APIs to various AI services. Because of this architecture, our analysis has shown that unless there is ongoing high-intensity usage, the cost per user will be very low — much lower than a typical end-user subscription to a single LLM provider. In the future we may implement an accounting system for the portal to pass usage costs onto our users, but our first priority is to get this capability into our clients’ hands as soon as possible to help them be early adopters of this transformative tech.

Our AI Portal connects to a range of proprietary AI-powered chatbots served up by Anthropic, OpenAI, Google, and Grok, as well as to several open-source AI-powered chatbots hosted on servers in the United States. This includes the Deepseek R1 model developed in China that rocked the tech world with its release on January 20. It’s a remarkable model, but for businesses in the US having their chat histories stored on servers in China is generally not acceptable. Having the service hosted in the US helps address privacy concerns. We run the open-source models on our own servers and using third-party hosting providers such as together.ai and groq.com.

To provide a more coherent user experience, the Open WebUI interface maintains each user’s chat history and provides the ability to manage, share, and download them.

The portal has already helped End Point with internal projects such as experimenting with how best to process various data sets to make them useful for AI-powered applications. The portal hosts several RAG (Retrieval Augmented Generation) applications that make use of various LLMs to search and answer questions about End Point’s blog posts—we are working to identify the best solution, so results may vary. Please don’t give up on reading our blog just yet.

Below are screenshots of our AI Portal’s interface showing some of its features (click on the images to view at full size).

We require users to accept our AI portal’s standard Terms of Service to use the portal. Additionally, each AI-powered chatbot accessible through our portal has its own Terms of Service, to which we provide links for your reference. Our primary concern is whether the information users input into these chatbots might be used by providers for training or other purposes. We strive to avoid chatbots that engage in such practices, and as of the launch of our portal, we’re not aware of any chatbot providers that claim rights to user input data. However, it’s common for Terms of Service documents to include clauses allowing providers to modify their terms without prior notice, so we can’t guarantee ongoing compliance with our standards. Regardless, users should always exercise caution and good judgment when entering confidential information into any online service.

Why We Are Doing This

At End Point, we’ve built our reputation for almost 30 years by providing expert consulting, development, and systems support. As AI technology rapidly evolves, we see it as both an opportunity and a necessity to stay ahead—helping our customers integrate AI into their own systems while deepening our expertise in the field.

By opening up our Open WebUI Portal to customers at no charge, we’re reinforcing our commitment to innovation, transparency, value, and great customer service. We intend this not only to benefit our clients, but to highlight our ability to integrate AI-powered solutions into your business. We stand ready to help you enhance your existing workflows, products, and services—as well as to create entirely new, AI-powered applications.

AI use cases include:

User-facing enhancements
- Intelligent similarity search
- Recommendation systems
- AI-powered insights from your data
- Dynamic FAQ generation based on past inquiries
- Customer support chatbots
- Faster and smarter documentation creation
- You name it. The possibilities are endless!
AI-driven development & optimization
- System testing and automated quality assurance
- AI-assisted code reviews and refactoring
- Automated code generation for improved efficiency
- …and more

We’re also using AI to refine our own processes, improve our code, and enhance the value we deliver to our customers. This initiative is part of our commitment to ensure that our clients continue to benefit from the best technology solutions available.

How to Get an Account

Reach out to your End Point client rep or use our contact form to get in touch.

Implementing SummAE neural text summarization with a denoising auto-encoder

2020-05-28T00:00:00+00:00

If there’s any problem space in machine learning, with no shortage of (unlabelled) data to train on, it’s easily natural language processing (NLP).

In this article, I’d like to take on the challenge of taking a paper that came from Google Research in late 2019 and implementing it. It’s going to be a fun trip into the world of neural text summarization. We’re going to go through the basics, the coding, and then we’ll look at what the results actually are in the end.

The paper we’re going to implement here is: Peter J. Liu, Yu-An Chung, Jie Ren (2019) SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders.

Here’s the paper’s abstract:

We propose an end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on ROCStories, a subset for which we collected human summaries. In this task, five-sentence stories (paragraphs) are summarized with one sentence, using human summaries only for evaluation. We show results for extractive and human baselines to demonstrate a large abstractive gap in performance. Our model, SummAE, consists of a denoising auto-encoder that embeds sentences and paragraphs in a common space, from which either can be decoded. Summaries for paragraphs are generated by decoding a sentence from the paragraph representations. We find that traditional sequence-to-sequence auto-encoders fail to produce good summaries and describe how specific architectural choices and pre-training techniques can significantly improve performance, outperforming extractive baselines. The data, training, evaluation code, and best model weights are open-sourced.

Preliminaries

Before we go any further, let’s talk a little bit about neural summarization in general. There’re two main approaches to it:

The first approach makes the model “focus” on the most important parts of the longer text - extracting them to form a summary.

Let’s take a recent article, “Shopify Admin API: Importing Products in Bulk”, by one of my great co-workers, Patrick Lewis, as an example and see what the extractive summarization would look like. Let’s take the first two paragraphs:

I recently worked on an interesting project for a store owner who was facing a daunting task: he had an inventory of hundreds of thousands of Magic: The Gathering (MTG) cards that he wanted to sell online through his Shopify store. The logistics of tracking down artwork and current market pricing for each card made it impossible to do manually.

My solution was to create a custom Rails application that retrieves inventory data from a combination of APIs and then automatically creates products for each card in Shopify. The resulting project turned what would have been a months- or years-long task into a bulk upload that only took a few hours to complete and allowed the store owner to immediately start selling his inventory online. The online store launch turned out to be even more important than initially expected due to current closures of physical stores.

An extractive model could summarize it as follows:

I recently worked on an interesting project for a store owner who had an inventory of hundreds of thousands of cards that he wanted to sell through his store. The logistics and current pricing for each card made it impossible to do manually. My solution was to create a custom Rails application that retrieves inventory data from a combination of APIs and then automatically creates products for each card. The store launch turned out to be even more important than expected due to current closures of physical stores.

See how it does the copying and pasting? The big advantage of these types of models is that they are generally easier to create and the resulting summaries tend to faithfully reflect the facts included in the source.

The downside though is that it’s not how a human would do it. We do a lot of paraphrasing, for instance. We use different words and tend to form sentences less rigidly following the original ones. The need for the summaries to feel more natural made the second type — abstractive — into this subfield’s holy grail.

Datasets

The paper’s authors used the so-called “ROCStories” dataset (“Tackling The Story Ending Biases in The Story Cloze Test”. Rishi Sharma, James Allen, Omid Bakhshandeh, Nasrin Mostafazadeh. In Proceedings of the 2018 Conference of the Association for Computational Linguistics (ACL), 2018).

In my experiments, I’ve also tried the model against one that’s quite a bit more difficult: WikiHow (Mahnaz Koupaee, William Yang Wang (2018) WikiHow: A Large Scale Text Summarization Dataset).

ROCStories

The dataset consists of 98162 stories, each one consisting of 5 sentences. It’s incredibly clean. The only step I needed to take was to split the stories between the train, eval, and test sets.

Examples of sentences:

Example 1:

My retired coworker turned 69 in July. I went net surfing to get her a gift. She loves Diana Ross. I got two newly released cds and mailed them to her. She sent me an email thanking me.

Example 2:

Tom alerted the government he expected a guest. When she didn’t come he got in a lot of trouble. They talked about revoking his doctor’s license. And charging him a huge fee! Tom’s life was destroyed because of his act of kindness.

Example 3:

I went to see the doctor when I knew it was bad. I hadn’t eaten in nearly a week. I told him I felt afraid of food in my body. He told me I was developing an eating disorder. He instructed me to get some help.

Wikihow

This is one of the most challenging openly available datasets for neural summarization. It consists of more than 200,000 long-sequence pairs of text + headline scraped from WikiHow’s website.

Some examples:

Text:

One easy way to conserve water is to cut down on your shower time. Practice cutting your showers down to 10 minutes, then 7, then 5. Challenge yourself to take a shorter shower every day. Washing machines take up a lot of water and electricity, so running a cycle for a couple of articles of clothing is inefficient. Hold off on laundry until you can fill the machine. Avoid letting the water run while you’re brushing your teeth or shaving. Keep your hoses and faucets turned off as much as possible. When you need them, use them sparingly.

Headline:

Take quicker showers to conserve water. Wait for a full load of clothing before running a washing machine. Turn off the water when you’re not using it.

The main challenge for the summarization model here is that the headline was actually created by humans and is not just “extracting” anything. Any model performing well on this dataset actually needs to model the language pretty well. Otherwise, the headline could be used for computing the evaluation metrics, but it’s pretty clear that traditional metrics like ROUGE are just bound here to miss the point.

Basics of the sequence-to-sequence modeling

Most sequence-to-sequence models are based on the “next token prediction” workflow.

The general idea can be expressed with P(token | context) — where the task is to model this conditional probability distribution. The “context” here depends on the approach.

Those models are also called “auto-regressive” because they need to consume their own predictions from previous steps during the inference:

predict(["<start>"], context)
# "I"
predict(["<start>", "I"], context)
# "love"
predict(["<start>", "I", "love"], context)
# "biking"
predict(["<start>", "I", "love", "biking"], context)
# "<end>"

Naively simple modeling: Markov Model

In this model, the approach is to take on a bold assumption: that the probability of the next token is conditioned only on the previous token.

The Markov Model is elegantly introduced in the blog post Next Word Prediction using Markov Model.

Why is it naive? Because we know that the probability of the word “love” depends on the word “I” given a broader context. A model that’s always going to output “roses” would miss the best word more often than not.

Modeling with neural networks

Usually, sequence-to-sequence neural network models consist of two parts:

encoder
decoder

The encoder is there to build a “gist” representation of the input sequence. The gist and the previous token become our “context” to do the inference. This fits in well within the P(token | context) modeling I described above. That distribution can be expressed more clearly as P(token | previous; gist).

There are other approaches too with one of them being the ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training - 2020 - Yan, Yu and Qi, Weizhen and Gong, Yeyun and Liu, Dayiheng and Duan, Nan and Chen, Jiusheng and Zhang, Ruofei and Zhou, Ming. The difference in the approach here was the prediction of n-tokens ahead at once.

Teacher-forcing

Let’s see how could we go about teaching the model about the next token’s conditional distribution.

Imagine that the model’s parameters aren’t performing well yet. We have an input sequence of: ["<start>", "I", "love", "biking", "during", "the", "summer", "<end>"]. We’re training the model giving it the first token:

model(["<start>", context])
# "I"

Great, now let’s ask it for another one:

model(["<start>", "I"], context])
# "wonder"

Hmmm that’s not what we wanted, but let’s naively continue:

model(["<start>", "I", "wonder"], context)
# "why"

We could continue gathering predictions and compute the loss at the end. The loss would really only be able to tell it about the first mistake (“love” vs. “wonder”); the rest of the errors would just accumulate from here. This would hinder the learning considerably, adding in the noise from the accumulated errors.

There’s a better approach called Teacher Forcing. In this approach, you’re telling the model the true answer after each of its guesses. The last example would look like the following:

model(["<start>", "I", "love"], context)
# "watching"

You’d continue the process, feeding it the full input sequence and the loss term would be computed based on all its guesses.

Compute-friendly representation for tokens and gists

Some of the readers might want to skip this section. I’d like to describe quickly here the concept of the latent space and vector embeddings. This is to keep the matters relatively palatable for the broader audience.

Representing words naively

How do we turn the words (strings) into numbers that we input into our machine learning models? A software developer might think about assigning each word a unique integer. This works well for databases but in machine learning models, the fact that integers follow one another means that they encode a relation (which one follows which and in what distance). This doesn’t work well for almost any problem in data science.

Traditionally, the problem is solved by “one-hot encoding”. This means that we’re turning our integers into vectors, where each value is zero except the one for the index that equals the value to encode (or minus one if your programming language uses zero-based indexing). Example: 3 => [0, 0, 0, 1, 0, 0, 0, 0, 0, 0] when the total number of “integers” (classes) to encode is 10.

This is better as it breaks the ordering and distancing assumptions. It doesn’t encode anything about the words, though, except the arbitrary number we’ve decided to assign to them. We now don’t have the ordering but we also don’t have any distance. Empirically though we just know that the word “love” is much closer to “enjoy” than it is to “helicopter”.

A better approach: word embeddings

How could we keep our vector representation (as in one-hot encoding) but also introduce the distance? I’ve already glanced over this concept in my post about the simple recommender system. The idea is to have a vector of floating-point values so that the closer the words are in their meaning, the smaller the angle is between them. We can easily compute a metric following this logic by measuring the cosine distance. This way, the word representations are easy to feed into the encoder, and they already contain a lot of the information in themselves.

Not only words

Can we only have vectors for words? Couldn’t we have vectors for paragraphs, so that the closer they are in their meaning, the smaller some vector space metric between them? Of course we can. This is, in fact, what will allow us in this article’s model to encode the “gist” that we talked about. The “encoder” part of the model is going to learn the most convenient way of turning the input sequence into the floating-point numbers vector.

Auto-encoders

We’re slowly approaching the model from the paper. We still have one concept that’s vital to understand in order to get why the model is going to work.

Up until now, we talked about the following structure of the typical sequence-to-sequence neural network model:

This is true e.g. for translation models where the input sequence is in English and the output is in Greek. It’s also true for this article’s model during the inference.

What if we’d make the input and output to be the same sequence? We’d turn it into a so-called auto-encoder.

The output of course isn’t all that useful — we already know what the input sequence is. The true value is in the model’s ability to encode the input into a gist.

Adding the noise

A very interesting type of an auto-encoder is the denoising auto-encoder. The idea is that the input sequence gets randomly corrupted and the network learns to still produce a good gist and reconstruct the sequence before it got corrupted. This makes the training “teach” the network about the deeper connections in the data, instead of just “memorizing” as much as it can.

The SummAE model

We’re now ready to talk about the architecture from the paper. Given what we’ve already learned, this is going to be very simple. The SummAE model is just a denoising auto-encoder that is being trained a special way.

Auto-encoding paragraphs and sentences

The authors were training the model on both single sentences and full paragraphs. In all cases the task was to reproduce the uncorrupted input.

The first part of the approach is about having two special “start tokens” to signal the mode: paragraph vs. sentence. In my code, I’ve used “<start-full>” and “<start-short>”.

During the training, the model learns the conditional distributions given those two tokens and the ones that follow, for any given token in the sequence.

Adding the noise

The sentences are simply concatenated to form a paragraph. The input then gets corrupted at random by means of:

masking the input tokens
shuffling the order of the sentences within the paragraph

The authors are claiming that the latter helped them in solving the issue of the network just memorizing the first sentence. What I have found though is that this model is generally prone towards memorizing concrete sentences from the paragraph. Sometimes it’s the first, and sometimes it’s some of the others. I’ve found this true even when adding a lot of noise to the input.

The code

The full PyTorch implementation described in this blog post is available at https://github.com/kamilc/neural-text-summarization. You may find some of its parts less clean than others — it’s a work in progress. Specifically, the data download is almost left out.

You can find the WikiData preprocessing in a notebook in the repository. For the ROCStories, I just downloaded the CSV files and concatenated with Unix cat. There’s an additional process.py file generated from a very simple IPython session.

Let’s have a very brief look at some of the most interesting parts of the code:

class SummarizeNet(NNModel):
    def encode(self, embeddings, lengths):
        # ...

    def decode(self, embeddings, encoded, lengths, modes):
        # ...

    def forward(self, embeddings, clean_embeddings, lengths, modes):
        # ...

    def predict(self, vocabulary, embeddings, lengths):
        # ...

You can notice separate methods for forward and predict. I chose the Transformer over the recurrent neural networks for both the encoder part and the decoder. The PyTorch implementation of the transformer decoder part already includes the teacher forcing in the forward method. This makes it convenient at the training time — to just feed it the full, uncorrupted sequence of embeddings as the “target”. During the inference we need to do the “auto-regressive” part by hand though. This means feeding the previous predictions in a loop — hence the need for two distinct methods here.

def forward(self, embeddings, clean_embeddings, lengths, modes):
    noisy_embeddings = self.mask_dropout(embeddings, lengths)

    encoded = self.encode(noisy_embeddings[:, 1:, :], lengths-1)
    decoded = self.decode(clean_embeddings, encoded, lengths, modes)

    return (
        decoded,
        encoded
    )

You can notice that I’m doing the token masking at the model level during the training. The code also shows cleanly the structure of this seq2seq model — with the encoder and the decoder.

The encoder part looks simple as long as you’re familiar with the transformers:

def encode(self, embeddings, lengths):
    batch_size, seq_len, _ = embeddings.shape

    embeddings = self.encode_positions(embeddings)

    paddings_mask = torch.arange(end=seq_len).unsqueeze(dim=0).expand((batch_size, seq_len)).to(self.device)
    paddings_mask = (paddings_mask + 1) > lengths.unsqueeze(dim=1).expand((batch_size, seq_len))

    encoded = embeddings.transpose(1,0)

    for ix, encoder in enumerate(self.encoders):
        encoded = encoder(encoded, src_key_padding_mask=paddings_mask)
        encoded = self.encode_batch_norms[ix](encoded.transpose(2,1)).transpose(2,1)

    last_encoded = encoded

    encoded = self.pool_encoded(encoded, lengths)

    encoded = self.to_hidden(encoded)

    return encoded

We’re first encoding the positions as in the “Attention Is All You Need” paper and then feeding the embeddings into a stack of the encoder layers. At the end, we’re morphing the tensor to have the final dimension equal the number given as the model’s parameter.

The decode sits on PyTorch’s shoulders too:

def decode(self, embeddings, encoded, lengths, modes):
    batch_size, seq_len, _ = embeddings.shape

    embeddings = self.encode_positions(embeddings)

    mask = self.mask_for(embeddings)

    encoded = self.from_hidden(encoded)
    encoded = encoded.unsqueeze(dim=0).expand(seq_len, batch_size, -1)

    decoded = embeddings.transpose(1,0)
    decoded = torch.cat(
        [
            encoded,
            decoded
        ],
        axis=2
    )
    decoded = self.combine_decoded(decoded)
    decoded = self.combine_batch_norm(decoded.transpose(2,1)).transpose(2,1)

    paddings_mask = torch.arange(end=seq_len).unsqueeze(dim=0).expand((batch_size, seq_len)).to(self.device)
    paddings_mask = paddings_mask > lengths.unsqueeze(dim=1).expand((batch_size, seq_len))

    for ix, decoder in enumerate(self.decoders):
        decoded = decoder(
            decoded,
            torch.ones_like(decoded),
            tgt_mask=mask,
            tgt_key_padding_mask=paddings_mask
        )
        decoded = self.decode_batch_norms[ix](decoded.transpose(2,1)).transpose(2,1)

    decoded = decoded.transpose(1,0)

    return self.linear_logits(decoded)

You can notice that I’m combining the gist received from the encoder with each word embeddings — as this is how it was described in the paper.

The predict is very similar to forward:

def predict(self, vocabulary, embeddings, lengths):
    """
    Caller should include the start and end tokens here
    but we’re going to ensure the start one is replaces by <start-short>
    """
    previous_mode = self.training

    self.eval()

    batch_size, _, _ = embeddings.shape

    results = []

    for row in range(0, batch_size):
        row_embeddings = embeddings[row, :, :].unsqueeze(dim=0)
        row_embeddings[0, 0] = vocabulary.token_vector("<start-short>")

        encoded = self.encode(
            row_embeddings[:, 1:, :],
            lengths[row].unsqueeze(dim=0)
        )

        results.append(
            self.decode_prediction(
                vocabulary,
                encoded,
                lengths[row].unsqueeze(dim=0)
            )
        )

    self.training = previous_mode

    return results

The workhorse behind the decoding at the inference time looks as follows:

def decode_prediction(self, vocabulary, encoded1xH, lengths1x):
    tokens = ['<start-short>']
    last_token = None
    seq_len = 1

    encoded1xH = self.from_hidden(encoded1xH)

    while last_token != '<end>' and seq_len < 50:
        embeddings1xSxD = vocabulary.embed(tokens).unsqueeze(dim=0).to(self.device)
        embeddings1xSxD = self.encode_positions(embeddings1xSxD)

        maskSxS = self.mask_for(embeddings1xSxD)

        encodedSx1xH = encoded1xH.unsqueeze(dim=0).expand(seq_len, 1, -1)

        decodedSx1xD = embeddings1xSxD.transpose(1,0)
        decodedSx1xD = torch.cat(
            [
                encodedSx1xH,
                decodedSx1xD
            ],
            axis=2
        )
        decodedSx1xD = self.combine_decoded(decodedSx1xD)
        decodedSx1xD = self.combine_batch_norm(decodedSx1xD.transpose(2,1)).transpose(2,1)

        for ix, decoder in enumerate(self.decoders):
            decodedSx1xD = decoder(
                decodedSx1xD,
                torch.ones_like(decodedSx1xD),
                tgt_mask=maskSxS,
            )
            decodedSx1xD = self.decode_batch_norms[ix](decodedSx1xD.transpose(2,1))
            decodedSx1xD = decodedSx1xD.transpose(2,1)

        decoded1x1xD = decodedSx1xD.transpose(1,0)[:, (seq_len-1):seq_len, :]
        decoded1x1xV = self.linear_logits(decoded1x1xD)

        word_id = F.softmax(decoded1x1xV[0, 0, :]).argmax().cpu().item()
        last_token = vocabulary.words[word_id]
        tokens.append(last_token)
        seq_len += 1

    return ' '.join(tokens[1:])

You can notice starting with the “start short” token and going in a loop, getting predictions, and feeding back until the “end” token.

Again, the model is very, very simple. What makes the difference is how it’s being trained — it’s all in the training data corruption and the model pre-training.

It’s already a long article so I encourage the curious readers to look at the code at my GitHub repo for more details.

My experiment with the WikiHow dataset

In my WikiHow experiment I wanted to see how the results look if I fed the full articles and their headlines for the two modes of the network. The same data-corruption regime was used in this case.

Some of the results were looking almost good:

Text:

for a savory flavor, mix in 1/2 teaspoon ground cumin, ground turmeric, or masala powder.this works best when added to the traditional salty lassi. for a flavorful addition to the traditional sweet lassi, add 1/2 teaspoon of ground cardamom powder or ginger, for some kick. , start with a traditional sweet lassi and blend in some of your favorite fruits. consider mixing in strawberries, papaya, bananas, or coconut.try chopping and freezing the fruit before blending it into the lassi. this will make your drink colder and frothier. , while most lassi drinks are yogurt based, you can swap out the yogurt and water or milk for coconut milk. this will give a slightly tropical flavor to the drink. or you could flavor the lassi with rose water syrup, vanilla extract, or honey.don’t choose too many flavors or they could make the drink too sweet. if you stick to one or two flavors, they’ll be more pronounced. , top your lassi with any of the following for extra flavor and a more polished look: chopped pistachios sprigs of mint sprinkle of turmeric or cumin chopped almonds fruit sliver

Headline:

add a spice., blend in a fruit., flavor with a syrup or milk., garnish.

Predicted summary:

blend vanilla in a sweeter flavor . , add a sugary fruit . , do a spicy twist . eat with dessert . , revise .

It’s not 100% faithful to the original text even though it seems to “read” well.

My suspicion is that pre-training against a much larger corpus of text might possibly help. There’s an obvious issue with the lack of very specific knowledge here to have the network summarize better. Here’s another of those examples:

Text:

the settings app looks like a gray gear icon on your iphone’s home screen.; , this option is listed next to a blue “a” icon below general. , this option will be at the bottom of the display & brightness menu. , the right-hand side of the slider will give you bigger font size in all menus and apps that support dynamic type, including the mail app. you can preview the corresponding text size by looking at the menu texts located above and below the text size slider. , the left-hand side of the slider will make all dynamic type text smaller, including all menus and mailboxes in the mail app. , tap the back button twice in the upper-left corner of your screen. it will save your text size settings and take you back to your settings menu. , this option is listed next to a gray gear icon above display & brightness. , it’s halfway through the general menu. ,, the switch will turn green. the text size slider below the switch will allow for even bigger fonts. , the text size in all menus and apps that support dynamic type will increase as you go towards the right-hand side of the slider. this is the largest text size you can get on an iphone. , it will save your settings.

Headline:

open your iphone’s settings., scroll down and tap display & brightness., tap text size., tap and drag the slider to the right for bigger text., tap and drag the slider to the left for smaller text., go back to the settings menu., tap general., tap accessibility., tap larger text. , slide the larger accessibility sizes switch to on position., tap and drag the slider to the right., tap the back button in the upper-left corner.

Predicted summary:

open your iphone ’s settings . , tap general . , scroll down and tap accessibility . , tap larger accessibility . , tap and larger text for the iphone to highlight the text you want to close . , tap the larger text - colored contacts app .

It might be interesting to train against this dataset again while:

utilizing some pre-trained, large scale model as part of the encoder
using a large corpus of text to still pre-train the auto-encoder

This could possibly take a lot of time to train on my GPU (even with the pre-trained part of the encoder). I didn’t follow the idea further at this time.

The problem with getting paragraphs when we want the sentences

One of the biggest problems the authors ran into was with the decoder outputting the long version of the text, even though it was asked for the sentence-long summary.

Authors called this phenomenon the “segregation issue”. What they have found was that the encoder was mapping paragraphs and sentences into completely separate regions. The solution to this problem was to trick the encoder into making both representations indistinguishable. The following figure comes from the paper and shows the issue visualized:

Better gists by using the “critic”

The idea of a “critic” has been popularized along with the fantastic results produced by some of the Generative Adversarial Networks. The general workflow is to have the main network generate output while the other tries to guess some of its properties.

For GANs that are generating realistic photos, the critic is there to guess if the photo was generated or if it’s real. A loss term is added based on how well it’s doing, penalizing the main network for generating photos that the critic is able to call out as fake.

A similar idea was used in the A3C algorithm I blogged about (Self-driving toy car using the Asynchronous Advantage Actor-Critic algorithm). The “critic” part penalized the AI agent for taking steps that were on average less advantageous.

Here, in the SummAE model, the critic adds a penalty to the loss to the degree to which it’s able to guess whether the gist comes from a paragraph or a sentence.

Training with the critic might get tricky. What I’ve found to be the cleanest way is to use two different optimizers — one updating the main network’s parameters while the other updates the critic itself:

for batch in batches:
    if mode == "train":
        self.model.train()
        self.discriminator.train()
    else:
        self.model.eval()
        self.discriminator.eval()

    self.optimizer.zero_grad()
    self.discriminator_optimizer.zero_grad()

    logits, state = self.model(
        batch.word_embeddings.to(self.device),
        batch.clean_word_embeddings.to(self.device),
        batch.lengths.to(self.device),
        batch.mode.to(self.device)
    )

    mode_probs_disc = self.discriminator(state.detach())
    mode_probs = self.discriminator(state)

    discriminator_loss = F.binary_cross_entropy(
        mode_probs_disc,
        batch.mode
    )

    discriminator_loss.backward(retain_graph=True)

    if mode == "train":
        self.discriminator_optimizer.step()

    text = batch.text.copy()

    if self.no_period_trick:
        text = [txt.replace('.', '') for txt in text]

    classes = self.vocabulary.encode(text, modes=batch.mode)
    classes = classes.roll(-1, dims=1)
    classes[:,classes.shape[1]-1] = 3

    model_loss = torch.tensor(0).cuda()

    if logits.shape[0:2] == classes.shape:
        model_loss = F.cross_entropy(
            logits.reshape(-1, logits.shape[2]).to(self.device),
            classes.long().reshape(-1).to(self.device),
            ignore_index=3
        )
    else:
        print("WARNING: Skipping model loss for inconsistency between logits and classes shapes")

    fooling_loss = F.binary_cross_entropy(
        mode_probs,
        torch.ones_like(batch.mode).to(self.device)
    )

    loss = model_loss + (0.1 * fooling_loss)

    loss.backward()
    if mode == "train":
        self.optimizer.step()

    self.optimizer.zero_grad()
    self.discriminator_optimizer.zero_grad()

The main idea is to treat the main network’s encoded gist as constant with respect to the updates to the critic’s parameters, and vice versa.

Results

I’ve found some of the results look really exceptional:

Text:

lynn is unhappy in her marriage. her husband is never good to her and shows her no attention. one evening lynn tells her husband she is going out with her friends. she really goes out with a man from work and has a great time. lynn continues dating him and starts having an affair.

Predicted summary:

lynn starts dating him and has an affair .

Text:

cedric was hoping to get a big bonus at work. he had worked hard at the office all year. cedric’s boss called him into his office. cedric was disappointed when told there would be no bonus. cedric’s boss surprised cedric with a big raise instead of a bonus.

Predicted summary:

cedric had a big deal at his boss ’s office .

Some others showed how the model attends to single sentences though:

Text:

i lost my job. i was having trouble affording my necessities. i didn’t have enough money to pay rent. i searched online for money making opportunities. i discovered amazon mechanical turk.

Predicted summary:

i did n’t have enough money to pay rent .

While the sentence like this one would maybe make a good headline — it’s definitely not the best summary as it naturally loses the vital parts found in other sentences.

Final words

First of all, let me thank the paper’s authors for their exceptional work. It was a great read and great fun implementing!

Abstractive text summarization remains very difficult. The model trained for this blog post has very limited use in practice. There’s a lot of room for improvement though, which makes the future of abstractive summaries very promising.

An Introduction to Neural Networks

2019-07-01T00:00:00+00:00

Photo by Sudhamshu Hebbar, used under CC BY 2.0

Earlier this year I wrote a post about my work with a machine-learning camera, the AWS DeepLens, which has onboard processing power to enable AI capabilities without sending data to the cloud. Neural networks are a type of ML model which achieves very impressive results on certain problems (including computer vision), so in this post I give a more thorough introduction to neural networks, and share some useful resources for those who want to dig deeper.

Neurons and Nodes

Neural networks are models inspired by the function of biological neural networks. They consist of nodes (arranged in layers), and the connections between those nodes. Each connection between two nodes enables one-way information transfer: a node either receives input from, or sends output to each node to which it is connected. Nodes typically have an “activation function”, parameterized by the node’s inputs, and its output is the result of this function.

As with the function of biological neural networks, the emergence of information processing from these mathematical operations is opaque. Nevertheless, complex artificial neural networks are capable of feats such as vision, language translation, and winning competitive games. As the technology improves, even more impressive tasks will become possible. As with organic brains, neural networks can achieve complex tasks only as a result of appropriate architecture, constraints, and training—for machine learning, humans must (for now) design it all.

Neural Network Architecture

Nodes are grouped in layers: the input layer, the output layer, and all the layers between them, known as hidden layers. Nodes can be networked in a variety of ways within and between layers, and sophisticated neural network models can include dozens of layers configured in various ways. These include layers which summarize, combine, eliminate, direct, or transform information. Each receives its input from the previous layer, and passes its output to the next layer. The last layer is designed such that its output answers the relevant question (for example, it would offer 9 options if the goal were to identify the hand-written numbers 1–9).

For all this information processing to achieve a given task, the parameters of each node need appropriate values. The process of choosing those values is called training. In order to train a neural network, one needs to provide examples of what the network should do. (For example, to train it to write requires examples of writing. To train it to identify objects in images requires images and their appropriately labeled counterparts.) The more data a model can learn from, the better it can work. Gathering enough data is typically a major undertaking.

Training a Neural Network

Before training, models have random parameters for all nodes. Each time data is passed through the model, the effectiveness of the model is measured using a “loss function”. Loss functions measure how wrong a model’s output is. Different loss functions (also known as cost functions or error functions) measure this in different ways, but in general, the more wrong a model is, the higher its loss/error/cost. Loss functions thus summarize the quality of a model’s output with a single number. Models are optimized to minimize the loss. (For more on the role of loss functions in neural networks, I suggest this excellent article.)

One of the most interesting details of the entire process has to do with how the parameters are tuned. Model optimization relies on variations of a process called gradient descent, in which parameter values are adjusted by small intervals in an attempt to minimize the loss. Over many thousands of repetitions, the training program uses calculus to pick values that help to minimize the loss. As you can imagine, this process becomes extremely computationally intensive when the neural network is large and complex. However, in order to solve hard problems, networks must be large and complex. This is why training neural networks requires substantial computing power, and often takes place in the cloud. (For more on stochastic gradient descent, I suggest this video as a great starting point, or this review for a more advanced overview.)

Facial Recognition Using Amazon DeepLens: Counting Liquid Galaxy Interactions

2019-05-01T00:00:00+00:00

I have been exploring the possible uses of a machine-learning-enabled camera for the Liquid Galaxy. The Amazon Web Services (AWS) DeepLens is a camera that can receive and transmit data over wifi, and that has computing hardware built in. Since its hardware enables it to use machine learning models, it can perform computer vision tasks in the field.

The Amazon DeepLens camera

This camera is the first of its kind—likely the first of many, given the ongoing rapid adoption of Internet of Things (IoT) devices and computer vision. It came to End Point’s attention as hardware that could potentially interface with and extend End Point’s immersive visualization platform, the Liquid Galaxy. We’ve thought of several ways computer vision could potentially work to enhance the platform, for example:

Monitoring users’ reactions
Counting unique visitors to the LG
Counting the number of people using an LG at a given time

The first idea would depend on parsing facial expressions. Perhaps a certain moment in a user experience causes people to look confused, or particularly delighted—valuable insights. The second idea would generate data that could help us assess the platform’s impact, using a metric crucial to any potential clients whose goals involve engaging audiences. The third idea would create a simpler metric: the average number of people engaging with the system over a period of time. Nevertheless, this idea has a key advantage over the second: it doesn’t require distinguishing between people, which makes it a much more tractable project. This post focuses on the third idea.

To set up the camera, the user has to plug it into a power outlet and connect it to wifi. The camera will still work even with a slow network connection, though when the connection is slower the delay between the camera seeing something and reporting it is longer. However, this delay was hardly noticable on my home network which has slow-to-moderate speeds of about 17 Mbps down and 33 Mbps up.

Computer Vision and the Amazon DeepLens

A deep learning model is a neural network with multiple layers of processing units. It is called “deep” because it has multiple layers. The inputs and outputs of each processing unit are numbers. These units are roughly analogous to neurons: they receive input from units in the previous layer, and output it to units in the next layer after transforming it based on a function. These “activation functions” can change in a variety of ways. The last layer’s outputs translate into the results. These models work because these functions get tuned based on how well the model works. For example, to make a model that labels each human face in a picture and draws a box around it, we would start with a corpus of pictures with boxes drawn around faces, as well as the versions of the pictures without the boxes drawn. We would test the model on the non-labeled images by checking—for each picture—whether the output generated by the model is correct. If not, the computer chooses different unit functions, tries again, and compares the results. Repeating this process thousands of times yields models which work remarkably well for a wide range of tasks, including computer vision.

In deep learning for computer vision, training on large sets of labeled images enables models to generalize about visual characteristics. The training process takes a lot of computing resources, but once models are trained, they can produce results quickly and with relative ease. This is why the DeepLens is able to perform computer vision with its limited computing resources.

Since the DeepLens is an Amazon product, it comes as no surprise that the user interface and backend for DeepLens consist of AWS services. One of the most important is SageMaker, which is used to train, manage, optimize, and deploy machine learning models such as neural networks. It includes hosted Jupyter notebooks (Jupyter is a development environment for data science), as well as the computing resources required for model training and storage. With SageMaker, users can train computer vision models for deployment to DeepLens, or import and adjust pretrained models from various sources.

Remote management of the DeepLens depends on AWS Lambda, a “serverless” cloud service that provides an environment to run backend code and integrate with other cloud services. It runs the show, allowing users to manage everything from the camera’s behavior to what happens to gathered data. Another service, AWS Greengrass, connects the instructions from AWS Lambda to the DeepLens, managing tasks like authentication, updates, and reactions to local events.

Amazon’s IoT service saves information about each DeepLens, and allows users to manage their devices, for example by choosing which model is active on the device, or viewing a live stream from the camera. It also keeps track of what’s going on with the hardware, even when it’s off. When a model is running on the DeepLens, you can view a live stream of its inferences about what it’s seeing (the labeled images). Amazon has released various pretrained models designed to work on the DeepLens. Using a model for detecting faces, we can get a live stream that looks like this:

Me looking at the DeepLens in my kitchen

Facial recognition inferences on multiple people. (Witness my smile of satisfaction at finally finding enthusiastic subjects of facial recognition.)

Each face that the camera detects gets a box around it, along with the model’s level of certainty that it is a face. The above pictures were the results of an attempt to simulate the conditions where this could be used.

The Model

The model I used was trained on data from ImageNet, a public database with hundreds or thousands of images associated with nouns. (For example they have 1537 pictures of folding chairs.) ImageNet is commonly used to train and test computer vision models.

However, the training for this model didn’t stop there: Amazon used transfer learning from another large image dataset, MS-COCO, to fine-tune the model for face detection. Transfer learning works essentially by retraining the last layer of an already-trained model. In this way it harnesses the “insights” of the existing model (e.g. about shapes, colors, and positions) by repurposing this information to make predictions about something else. In this case, whether something is a face.

Since this model was pretrained and optimized by Amazon for the DeepLens, it provides a low effort route to implementing a computer vision model on the DeepLens. I didn’t have to do any of the processing on my own hardware. The DeepLens hardware took care of all the predictions, though the biggest resource savings were from not having to train the model myself (which can take days, or longer).

When the facial recognition model is deployed and the DeepLens is on, an AWS Lambda function written in Python repeatedly prompts the camera to get frames from the camera:

frame = awscam.getLastFrame()

…to resize the frames before inference (the model accepts frames of particular size):

frame_resize = cv2.resize(frame, (input_height, input_width))

…to pass the frames to the model:

parsed_inference_results = model.parseResult(model_type, model.doInference(frame_resize))

…and to use the results to draw boxes around the faces:

cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (255, 165, 20), 10)

As you can see from how often “cv2” appears in the code above, this implementation relies heavily on code from OpenCV, an open source computer vision framework. Finally, the results are sent to the cloud:

client.publish(topic=iot_topic, payload=json.dumps(cloud_output))

In the last code snippet above, iot_topic refers to an Amazon “MQTT topic” (Message Queuing Telemetry Transport), for IoT devices. MQTT is the standard connectivity framework for DeepLens and many other IoT devices. One of its advantages for this context is that it can handle situations with intermittent connectivity, by smoothly queueing messages for when the network connection is stable. The essence of MQTT is to enable publishing and subscribing to different topics. The system of topics enables results from a DeepLens to trigger other processes. For example, the DeepLens could publish a message when it sees a face, and this could prompt another cloud service to do something else, such as save what time and how long the face appeared.

I wanted to test how data from this model would compare to a human’s perception. The first step was to understand what data the camera offers. It produces data about each frame analyzed: a timestamp (in 13-digit Unix time), and the predicted probability that something it identifies is a face. To gather this data, I used the AWS IoT service to manually subscribe to a secure MQTT topic where the DeepLens published its predictions. Each frame processed produces data like this:

{
  "format": "json",
  "payload": {
    "face": 0.5654296875
  },
  "qos": 0,
  "timestamp": 1554853281975,
  "topic": "$aws/things/deeplens_bnU5sr2sSD2ecW5YkfJZtw/infer"
}

The data generated by a single frame (with one face) when processed by the DeepLens.

For my purposes, I was only interested in the timestamps and payloads (which contain the number of faces identified, and their probabilities). I decided to test the facial recognition model under several different conditions:

No faces present
One face present
Multiple faces present

For condition 1 I just aimed it at an empty room for 20 minutes, and for condition 2 I sat in front of the camera for 20 minutes. For condition 3, I aimed the camera at a public space for 20 minutes, and while it was running I kept an ongoing count of the number of people looking in the general direction of the camera (I put the camera in front of a wall with a TV on it so people would be more likely to look towards it). Then I averaged my count over the duration of the sample, which resulted in an average engagement number of 2.5 people, meaning that on average, 2.5 people were looking at the camera. In an attempt to minimize bias, I made my human-eye assessment before looking at any of the data.

I’ll spoil one aspect of the results right away: there were no false positives under any condition. Even the lower probability guesses corresponded to actual faces, though this result might not hold true in a room with lots of face-like art, that’s not too common of a scenario. This simplified things, since it meant there was no need to set a lower bound on the probabilities which we should count—any face detected by the camera is a face. This also highlights one of my remaining questions about the model: is there useful information to be gained from the probabilities?

Another important note: I noticed early in the experiment that it almost never detects a face farther than 15 feet away. For the use case of a Liquid Galaxy, the 15-foot range is too short to capture all types of engagement (some people look at it across the room), but from my experience with the system I think that users within this range could be accurately described as focused users—something worth measuring, but certainly not everything worth measuring. After noticing this, I retested condition 2 with my face about 5 feet from the DeepLens, after initially trying it from across a room.

How did the DeepLens counts compare to my counts?

The model matched my performance in conditions 1 and 2, which makes a strong statement about its reliability in relatively static and close-up conditions such as looking at an empty room, or looking at someone stare at their laptop across a small table. In contrast, it did not count as many faces as I did in condition 3—so I’m happy to report I can still outperform A.I. on something.

Anyway, this suggests that the model is somewhat conservative, at least compared to my count (likely partly due to my eyes having a range larger than 15 feet). Therefore, when considering usage statistics gathered by a similar method, it might make most sense to think of the results as a lower bound, e.g. “the average number of people focused on the system was more than 2.1”.

It would be useful to experiment with the multiple faces condition again, to see how robust these findings are. It would also be helpful to keep track of factors like how much people move, the lighting, and the orientation of the camera, to see if they might impact the results. It would also be useful to automate the data collection and analysis.

This investigation has showed me that the DeepLens has a lot of potential as a tool for measuring engagement. Perhaps a future post will examine how it can be used to count users.

Thanks for reading! You are welcome to learn more about End Point Liquid Galaxy and AWS DeepLens.

Self-driving toy car using the Asynchronous Advantage Actor-Critic algorithm

2018-08-29T00:00:00+00:00

The field of Reinforcement Learning has seen a lot of great improvement in the past years. Researchers at universities and companies like Deep Mind have been developing new and better ways to train intelligent, artificial agents to solve more and more difficult tasks. The algorithms being developed are requiring less time to train. They also are making the training much more stable.

This article is about an algorithm that’s one of the most cited lately: A3C — Asynchronous Advantage Actor-Critic.

As the subject is both wide and deep, I’m assuming the reader has the relevant background mastered already. Although reading it might be interesting even without understanding most of the notions in use, having a good grasp of them will help you get the most out of it.

Because we’re looking at the Deep Reinforcement Learning, the obvious requirement is to be acquainted with the neural networks. I’m also using different notions known in the field of Reinforcement Learning overall like $Q(a, s)$ and $V(s)$ functions or the n-step return. The mathematical expressions, in particular, are given assuming that the reader already knows what the symbols stand for. Some notions known from other families of RL algorithms are being touched on as well (e.g. experience replay) — to contrast them with the A3C way of solving the same kind of problems. The article along with the source code uses the OpenAI gym, Python, and PyTorch among other Python-related libraries.

Theory

The A3C algorithm is a part of the greater class of RL algorithms called Policy Gradients.

In this approach, we’re creating a model that approximates the action-choosing policy itself.

Let’s contrast it with value iteration, the goal of which is to learn the value function and have policy emerge as the function that chooses an action transitioning to the state of the greatest value.

With the policy gradient approach, we’re approximating the policy with a differentiable function. Such stated problem requires only a good approximation of the gradient that over time will maximize the rewards.

The unique approach of A3C adds a very clever twist: we’re also learning an approximation of the value function at the same time. This helps us in getting the variance of the gradient down considerably, making the training much more stable.

These two aspects of the algorithm are being personified within its name: actor-critic. The policy function approximation is being called the actor, while the value function is being called the critic.

The policy gradient

As we’ve noticed already, in order to improve our policy function approximation, we need a gradient that points at the direction that maximizes the rewards.

I’m not going to reinvent the wheel here. There are some great resources the reader can access to dig deep into the Mathematics of what’s called the Policy Gradient Theorem:

The following equation presents the basic form of the gradient of the policy function:

$$\nabla_{\theta} J(\theta) = E_{\tau}[,R_{\tau}\cdot\nabla_\theta,\sum_{t=0}^{T-1},log,\pi(a_t|s_t;\theta),]$$

This states that for each sampled trajectory $\tau$, the correct estimate of the gradient is the expected value of the rewards times the action probabilities moved into the log space. Ascending in this direction makes our rewards greater and greater over time.

We can derive all the needed intermediary gradients ourselves by hand of course. Because we’re using PyTorch though, we only need the right loss function.

Let’s figure out the right loss function formula that will produce the gradient as shown above:

$$L_\theta=-J(\theta)$$

Also:

$$J(\theta)=E_\tau[R_\tau\cdot\sum_{t=0}^{T-1},log,\pi(a_t|s_t;\theta)]$$

Hence:

$$L_\theta=-\frac{1}{n}\sum_{t=0}^{n-1}R_t,\cdot,log\pi(a_t|s_t;\theta)$$

Formalizing the accumulation of rewards

For now, we’ve been using the $R_\tau$ and $R_t$ terms very abstractly. Let’s make this part more intuitive and concrete now.

Its true meaning really is “the quality of the sampled trajectory”. Consider the following equation:

$$R_t=\sum_{i=t}^{N+t}\gamma^{i-t}r_i,+,\gamma^{i-t+1}V(s_{t+N+1})$$

Each $r_i$ is the reward received from the environment after each step. Each trajectory consists of multiple steps. Each time, we’re sampling actions based on our policy function. This gives probabilities of a given action being best given the state.

What if we’re taking 5 actions for which we’re not being given any reward but overall it helped us get rewarded in the 6th step? This is exactly the case we’ll be dealing with in this article later when training a toy car to drive based only on pixel values of the scene. In that environment, we’ll be given $-0.1$ “negative” reward each step and something close to $7$ each new “tile” the car stays on the road.

We need a way to still encourage actions that make us earn rewards in a not too distant future. We also need to be smart and discount future rewards somewhat so that the more immediate the reward is to our action, the more emphasis we put on it.

That’s exactly what the above equation does. Notice that $\gamma$ becomes a hyper-parameter. It makes sense to give it value from $(0, 1)$. Let’s consider the following list of rewards: $[r_1, r_2, r_3, r_4]$. For $r_1$, the formula for the discounted accumulated reward is:

$$R_1=\gamma,r_1,+,\gamma^2r_2,+,\gamma^3r_3,+,\gamma^4r_4,+,\gamma^5V(s_5)$$

For $r_2$ it’s:

$$R_2=\gamma,r_2,+,\gamma^2r_3,+,\gamma^3r_4,+,\gamma^4V(s_5)$$

And so on… In case when we hit the terminal state, having no “next” state, we substitute $0$ for $V(s_{t+N+1})$.

We’ve said that in A3C we’re learning the value function at the same time. The $R_t$ as described above becomes the target value when training our $V(s)$. The value function becomes an approximation of the average of the rewards given the state (because $R_t$ depends on us sampling actions in this state).

Making the gradients more stable

One of the greatest inhibitors of the policy gradient performance is what’s broadly called “high variance”.

I have to admit, the first time I saw that term in this context, I was disoriented. I knew what “variance” was. It’s the “variance of what” that was not clear to me.

Thankfully I found a brilliant answer to this question. It explains the issue simply yet in detail.

Let me cite it here:

When we talk about high variance in the policy gradient method, we’re specifically talking about the facts that the variance of the gradients are high — namely, that $Var(\nabla_{\theta} J(\theta))$ is big.

To put it in simple terms: because we’re sampling trajectories from the space that is stochastic in nature, we’re bound to have those samples give gradients that disagree a lot on the best direction to take our model’s parameters into.

I encourage the reader to pause now and read the above-mentioned answer as it’s very vital. The gist of the solution described in it is that we can subtract a baseline value from each $R_t$. An example of a good baseline that was given was to make it into an average of the sampled accumulated rewards. The A3C algorithm uses this insight in a very, very clever way.

Value function as a baseline

To learn the $V(s)$ we’re typically using the MSE or Huber loss against the accumulated rewards for each step. This means that over time we’re averaging those rewards out based on the state we’re finding ourselves in.

Improving our gradient formula with those ideas we now get:

$$\nabla_{\theta} J(\theta) = E_{\tau}[,\nabla_\theta,\sum_{t=0}^{T-1},log,\pi(a_t|s_t;\theta)\cdot(R_t-V(s_t)),]$$

It’s important to treat the $(R_t-V(s_t))$ term as a constant. This means that when using PyTorch or any other deep learning framework, the computation of it should occur outside the graph that influences the gradients.

The enhanced part of the equation is where we get the word “advantage” in the algorithm’s name. The advantage is simply the difference between the accumulated rewards and what those rewards are on average for the given state:

$$A(a_{t..t+n},s_{t..t+n})=R_t(a_{t..t+n},s_{t..t+n})-V(s_t)$$

If we’ll make $R_t$ into $Q(s,a)$ as it’s commonly written in literature, we’ll arrive at the formula:

$$A(s,a)=Q(s,a) - V(s)$$

What’s the intuition here? Imagine that you’re playing chess with a 5-year-old. You win by a huge margin. Your friend who’s watched lots of master-level games observed this one as well. His take is that even though you scored positively, you still made lots of mistakes. You’ve got your critic here. Your score and what it looks like for the “observing critic” combined is what we call the advantage of the actions you took.

Guarding against the model’s overconfidence

Although he was warned, Icarus was too young and too enthusiastic about flying. He got excited by the thrill of flying and carried away by the amazing feeling of freedom and started flying high to salute the sun, diving low to the sea, and then up high again. His father Daedalus was trying in vain to make young Icarus to understand that his behavior was dangerous, and Icarus soon saw his wings melting. Icarus fell into the sea and drowned.

The Myth Of Daedalus And Icarus

The job of an “actor” is to output probability values for each possible action the agent can take. The greater the probability, the greater the model’s confidence that this action will result in the highest reward.

What if at some point, the weights are being steered in a way that the model becomes overconfident of some particular action? If this happens before the model learns much, it becomes a huge problem.

Because we’re using the $\pi(a|s;\theta)$ distribution to sample trajectories with, we’re not sampling totally at random. In other words, for $\pi(a|s;\theta) = [0.1, 0.4, 0.2, 0.3]$ our sampling chooses the second option 40% of the time. With any action overwhelming the others, we’re losing the ability to explore different paths and thus learn valuable lessons.

Empirically, I have found myself seeing the process sometimes not even able to escape the “overconfidence” area for long, long hours.

Regularizing with entropy

Let’s introduce the notion of an entropy.

In simple words in our case, it’s the measure of how much “knowledge” does given probability distribution posses. It’s being maximized for the uniform distribution. Here’s the formula:

$$H(X)=E[-log_b(P(X))]$$

This expands to the following:

$$H(X)=-\sum_{i=1}^{n}P(x_i)log_b(P(x_i))$$

Let’s look closer at the values this function produces using the following simple Calca code:

uniform = [0.25, 0.25, 0.25, 0.25]
more confident = [0.5, 0.25, 0.15, 0.10]
over confident = [0.95, 0.01, 0.01, 0.03]
super over confident = [0.99, 0.003, 0.004, 0.003]

y(x) = x*log(x, 10)

entropy(dist) = -sum(map(y, dist))

entropy (uniform) => 0.6021
entropy (more confident) => 0.5246
entropy (over confident) => 0.1068
entropy (super over confident) => 0.0291

We can use the above to “punish” the model whenever it’s too confident of its choices. As we’re going to use gradient descend, we’ll be minimizing terms that appear in our loss function. Minimizing the entropy as shown above would encourage more confidence though. We’ll need to make it into a negative in the loss to work the way we intend:

$$L_\theta=-\frac{1}{n}\sum_{t=0}^{n-1}log\pi(a_t|s_t;\theta)\cdot(R_t-V(s_t)),-\beta,H(\pi(a_t|s_t;\theta))$$

Where $\beta$ is a hyperparameter scaling the effects of the penalty that the entropy has on the gradients. Choosing the right value for $\beta$ becomes very vital for the model’s convergence. In this article, I’m using $0.01$ as with $0.001$ I was still observing the process stuck being overconfident.

Let’s include the value loss $L_v$ in the loss function formula making it full and ready to be implemented:

$$L_\theta=-\frac{1}{n}\sum_{t=0}^{n-1}log\pi(a_t|s_t;\theta)\cdot(R_t-V(s_t)),+\alpha,L_v,,-\beta,H(\pi(a_t|s_t;\theta))$$

The last A in A3C

So far we’ve gone from the vanilla policy gradients to using the notion of an advantage. We’ve also improved it with the baseline that intuitively makes the model consist of two parts: the actor and the critic. At this point, we have what’s sometimes called the A2C — Advantage Actor-Critic.

Let us now focus on the last piece of the puzzle: the last A. This last A comes from the word “asynchronous”. It’s been explained very clearly in the original paper on A3C.

This idea I think is the least complex of all that have their place in the approach. I’ll just comment on what was already written:

These approaches share a common idea: the sequence of observed data encountered by an online RL agent is non-stationary, and online RL updates are strongly correlated. By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps. Aggregating over memory in this way reduces non-stationarity and decorrelates updates, but at the same time limits the methods to off-policy reinforcement learning algorithms.

The A3C unique approach is that it doesn’t use experience replay for de-correlating the updates to the weights of the model. Instead, we’re sampling many different trajectories at the same time in an asynchronous manner.

This means that we’re creating many clones of the environment and we let our agents experience them at the same time. Separate agents share their weights in one way or another. There are implementations with agents sharing those weights very literally — and performing the updates to the weights on their own whenever they need to. There also are implementations with one main agent holding the main weights and doing the updates based on the gradients reported by the “worker” agents. The worker agents are then being updated with the evolved weights. The environments and agents are not being directly synchronized, working at their own speed. As soon as any of them collects the needed rewards to perform the n-step gradients calculations, the gradients are being applied in one way or another.

In this article, I’m preferring the second approach — having one “main” agent and making workers synchronize their weights with it each n-step period.

Practice

The challenge

To present the above theory in practical terms, we’re going to code the A3C to train a toy self-driving game car. The algorithm will only have the game’s pixels as inputs. We’re also going to collect rewards.

Each step, the player will decide how to move the steering wheel, how much throttle to apply and how much brake.

Points are being assigned for each new “tile” that the car enters staying within the road. There’s a small penalty for each other case of $-0.1$ points.

We’re going to use OpenAI Gym and the environment’s called CarRacing.

You can read a bit more about the setup in the environment’s source code on GitHub.

Coding the Agent

Our agent is going to output both $\pi(a|s;\theta)$ as well as $V(s)$. We’re going to use the GRU unit to give the agent the ability to remember its previous actions and environments previous features.

I’ve also decided to use PRelu instead of Relu activations as it appeared to me that this way the agent was learning much quicker (although I don’t have any numbers to back this impression up).

Disclaimer: the code presented below has not been refactored in any way. If this was going to be used in production I’d certainly hugely clean it up.

Here’s the full listing of the agent’s class:

class Agent(nn.Module):
    def __init__(self, **kwargs):
        super(Agent, self).__init__(**kwargs)

        self.init_args = kwargs

        self.h = torch.zeros(1, 256)

        self.norm1 = nn.BatchNorm2d(4)
        self.norm2 = nn.BatchNorm2d(32)

        self.conv1 = nn.Conv2d(4, 32, 4, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv4 = nn.Conv2d(32, 32, 3, stride=2, padding=1)

        self.gru = nn.GRUCell(1152, 256)
        self.policy = nn.Linear(256, 4)
        self.value = nn.Linear(256, 1)

        self.prelu1 = nn.PReLU()
        self.prelu2 = nn.PReLU()
        self.prelu3 = nn.PReLU()
        self.prelu4 = nn.PReLU()

        nn.init.xavier_uniform_(self.conv1.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.conv1.bias, 0.01)
        nn.init.xavier_uniform_(self.conv2.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.conv2.bias, 0.01)
        nn.init.xavier_uniform_(self.conv3.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.conv3.bias, 0.01)
        nn.init.xavier_uniform_(self.conv4.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.conv4.bias, 0.01)
        nn.init.constant_(self.gru.bias_ih, 0)
        nn.init.constant_(self.gru.bias_hh, 0)
        nn.init.xavier_uniform_(self.policy.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.policy.bias, 0.01)
        nn.init.xavier_uniform_(self.value.weight, gain=nn.init.calculate_gain('leaky_relu'))
        nn.init.constant_(self.value.bias, 0.01)

        self.train()

    def reset(self):
        self.h = torch.zeros(1, 256)

    def clone(self, num=1):
        return [ self.clone_one() for _ in range(num) ]

    def clone_one(self):
        return Agent(**self.init_args)

    def forward(self, state):
        state = state.view(1, 4, 96, 96)
        state = self.norm1(state)

        data = self.prelu1(self.conv1(state))
        data = self.prelu2(self.conv2(data))
        data = self.prelu3(self.conv3(data))
        data = self.prelu4(self.conv4(data))

        data = self.norm2(data)
        data = data.view(1, -1)

        h = self.gru(data, self.h)
        self.h = h.detach()

        pre_policy = h.view(-1)

        policy = F.softmax(self.policy(pre_policy))
        value = self.value(pre_policy)

        return policy, value

You can immediately notice that actor and critic parts share most of the weights. They only differ in the last layer.

Next, I wanted to abstract out the notion of the “runner”. It encapsulates the idea of a “running agent”. Think of it as the game player — with the joystick and its brain to score game points. I’m discretizing the action space the following way:

Action name	value
Turn left	[-0.8, 0.0, 0.0]
Turn right	[0.8, 0.0, 0]
Full throttle	[0.0, 0.1, 0.0]
Brake	[0.0, 0.0, 0.6]

class Runner:
    def __init__(self, agent, ix, train = True, **kwargs):
        self.agent = agent
        self.train = train
        self.ix = ix
        self.reset = False
        self.states = []

        # each runner has its own environment:
        self.env = gym.make('CarRacing-v0')

    def get_value(self):
        """
        Returns just the current state's value.
        This is used when approximating the R.
        If the last step was
        not terminal, then we're substituting the "r"
        with V(s) - hence, we need a way to just
        get that V(s) without moving forward yet.
        """
        _input = self.preprocess(self.states)
        _, _, _, value = self.decide(_input)
        return value

    def run_episode(self, yield_every = 10, do_render = False):
        """
        The episode runner written in the generator style.
        This is meant to be used in a "for (...) in run_episode(...):" manner.
        Each value generated is a tuple of:
        step_ix: the current "step" number
        rewards: the list of rewards as received from the environment (without discounting yet)
        values: the list of V(s) values, as predicted by the "critic"
        policies: the list of policies as received from the "actor"
        actions: the list of actions as sampled based on policies
        terminal: whether we're in a "terminal" state
        """
        self.reset = False
        step_ix = 0

        rewards, values, policies, actions = [[], [], [], []]

        self.env.reset()

        # we're going to feed the last 4 frames to the neural network that acts as the "actor-critic" duo. We'll use the "deque" to efficiently drop too old frames always keeping its length at 4:
        states = deque([ ])

        # we're pre-populating the states deque by taking first 4 steps as "full throttle forward":
        while len(states) < 4:
            _, r, _, _ = self.env.step([0.0, 1.0, 0.0])
            state = self.env.render(mode='rgb_array')
            states.append(state)
            logger.info('Init reward ' + str(r) )

        # we need to repeat the following as long as the game is not over yet:
        while True:
            # the frames need to be preprocessed (I'm explaining the reasons later in the article)
            _input = self.preprocess(states)

            # asking the neural network for the policy and value predictions:
            action, action_ix, policy, value = self.decide(_input, step_ix)

            # taking the step and receiving the reward along with info if the game is over:
            _, reward, terminal, _ = self.env.step(action)

            # explicitly rendering the scene (again, this will be explained later)
            state = self.env.render(mode='rgb_array')

            # update the last 4 states deque:
            states.append(state)
            while len(states) > 4:
                states.popleft()

            # if we've been asked to render into the window (e. g. to capture the video):
            if do_render:
                self.env.render()

            self.states = states
            step_ix += 1

            rewards.append(reward)
            values.append(value)
            policies.append(policy)
            actions.append(action_ix)

            # periodically save the state's screenshot along with the numerical values in an easy to read way:
            if self.ix == 2 and step_ix % 200 == 0:
                fname = './screens/car-racing/screen-' + str(step_ix) + '-' + str(int(time.time())) + '.jpg'
                im = Image.fromarray(state)
                im.save(fname)
                state.tofile(fname + '.txt', sep=" ")
                _input.numpy().tofile(fname + '.input.txt', sep=" ")

            # if it's game over or we hit the "yield every" value, yield the values from this generator:
            if terminal or step_ix % yield_every == 0:
                yield step_ix, rewards, values, policies, actions, terminal
                rewards, values, policies, actions = [[], [], [], []]

            # following is a very tacky way to allow external using code to mark that it wants us to reset the environment, finishing the episode prematurely. (this would be hugely refactored in the production code but for the sake of playing with the algorithm itself, it's good enough):
            if self.reset:
                self.reset = False
                self.agent.reset()
                states = deque([ ])
                self.states = deque([ ])
                return

            if terminal:
                self.agent.reset()
                states = deque([ ])
                return

    def ask_reset(self):
        self.reset = True

    def preprocess(self, states):
        return torch.stack([ torch.tensor(self.preprocess_one(image_data), dtype=torch.float32) for image_data in states ])

    def preprocess_one(self, image):
        """
        Scales the rendered image and makes it grayscale
        """
        return rescale(rgb2gray(image), (0.24, 0.16), anti_aliasing=False, mode='edge', multichannel=False)

    def choose_action(self, policy, step_ix):
        """
        Chooses an action to take based on the policy and whether we're in the training mode or not. During training, it samples based on the probability values in the policy. During the evaluation, it takes the most probable action in a greedy way.
        """
        policies = [[-0.8, 0.0, 0.0], [0.8, 0.0, 0], [0.0, 0.1, 0.0], [0.0, 0.0, 0.6]]

        if self.train:
            action_ix = np.random.choice(4, 1, p=torch.tensor(policy).detach().numpy())[0]
        else:
            action_ix = np.argmax(torch.tensor(policy).detach().numpy())

        logger.info('Step ' + str(step_ix) + ' Runner ' + str(self.ix) + ' Action ix: ' + str(action_ix) + ' From: ' + str(policy))

        return np.array(policies[action_ix], dtype=np.float32), action_ix

    def decide(self, state, step_ix = 999):
        policy, value = self.agent(state)

        action, action_ix = self.choose_action(policy, step_ix)

        return action, action_ix, policy, value

    def load_state_dict(self, state):
        """
        As we'll have multiple "worker" runners, they will need to be able to sync their agents' weights with the main agent.
        This function loads the weights into this runner's agent.
        """
        self.agent.load_state_dict(state)

I’m also encapsulating the training process in a class of its own. You can notice the gradients being clipped before being applied. I’m also clipping the rewards into the range of $<-3, 3>$ to help to keep the variance low.

class Trainer:
    def __init__(self, gamma, agent, window = 15, workers = 8, **kwargs):
        super().__init__(**kwargs)

        self.agent = agent
        self.window = window
        self.gamma = gamma
        self.optimizer = optim.Adam(self.agent.parameters(), lr=1e-4)
        self.workers = workers

        # even though we're loading the weights into worker agents explicitly, I found that still without sharing the weights as following, the algorithm was not converging:
        self.agent.share_memory()

    def fit(self, episodes = 1000):
            """
            The higher level method for training the agents.
            It called into the lower level "train" which orchestrates the process itself.
            """
        last_update = 0
        updates = dict()

        for ix in range(1, self.workers + 1):
            updates[ ix ] = { 'episode': 0, 'step': 0, 'rewards': deque(), 'losses': deque(), 'points': 0, 'mean_reward': 0, 'mean_loss': 0 }

        for update in self.train(episodes):
            now = time.time()

            # you could do something useful here with the updates dict.
            # I've opted out as I'm using logging anyways and got more value in just watching the log file, grepping for the desired values

            # save the current model's weights every minute:
            if now - last_update > 60:
                torch.save(self.agent.state_dict(), './checkpoints/car-racing/' + str(int(now)) + '-.pytorch')
                last_update = now

    def train(self, episodes = 1000):
        """
        Lower level training orchestration method. Written in the generator style. Intended to be used with "for update in train(...):"
        """

        # create the requested number of background agents and runners:
        worker_agents = self.agent.clone(num = self.workers)
        runners = [ Runner(agent=agent, ix = ix + 1, train = True) for ix, agent in enumerate(worker_agents) ]

        # we're going to communicate the workers' updates via the thread safe queue:
        queue = mp.SimpleQueue()

        # if we've not been given a number of episodes: assume the process is going to be interrupted with the keyboard interrupt once the user (us) decides so:
        if episodes is None:
            print('Starting out an infinite training process')

        # create the actual background processes, making their entry be the train_one method:
        processes = [ mp.Process(target=self.train_one, args=(runners[ix - 1], queue, episodes, ix)) for ix in range(1, self.workers + 1) ]

        # run those processes:
        for process in processes:
            process.start()

        try:
            # what follows is a rather naive implementation of listening to workers updates. it works though for our purposes:
            while any([ process.is_alive() for process in processes ]):
                results = queue.get()
                yield results
        except Exception as e:
            logger.error(str(e))

    def train_one(self, runner, queue, episodes = 1000, ix = 1):
        """
        Orchestrate the training for a single worker runner and agent. This is intended to run in its own background process.
        """

        # possibly naive way of trying to de-correlate the weight updates further (I have no hard evidence to prove if it works, other than my subjective observation):
        time.sleep(ix)

        try:
            # we are going to request the episode be reset whenever our agent scores lower than its max points. the same will happen if the agent scores total of -10 points:
            max_points = 0
            max_eval_points = 0
            min_points = 0
            max_episode = 0

            for episode_ix in itertools.count(start=0, step=1):

                if episodes is not None and episode_ix >= episodes:
                    return

                max_episode_points = 0
                points = 0

                # load up the newest weights every new episode:
                runner.load_state_dict(self.agent.state_dict())

                # every 5 episodes lets evaluate the weights we've learned so far by recording the run of the car using the greedy strategy:
                if ix == 1 and episode_ix % 5 == 0:
                    eval_points = self.record_greedy(episode_ix)

                    if eval_points > max_eval_points:
                        torch.save(runner.agent.state_dict(), './checkpoints/car-racing/' + str(eval_points) + '-eval-points.pytorch')
                        max_eval_points = eval_points

                # each n-step window, compute the gradients and apply
                # also: decide if we shouldn't restart the episode if we don't want to explore too much of the not-useful state space:
                for step, rewards, values, policies, action_ixs, terminal in runner.run_episode(yield_every=self.window):
                    points += sum(rewards)

                    if ix == 1 and points > max_points:
                        torch.save(runner.agent.state_dict(), './checkpoints/car-racing/' + str(points) + '-points.pytorch')
                        max_points = points

                    if ix == 1 and episode_ix > max_episode:
                        torch.save(runner.agent.state_dict(), './checkpoints/car-racing/' + str(episode_ix) + '-episode.pytorch')
                        max_episode = episode_ix

                    if points < -10 or (max_episode_points > min_points and points < min_points):
                        terminal = True
                        max_episode_points = 0
                        point = 0
                        runner.ask_reset()

                    if terminal:
                        logger.info('TERMINAL for ' + str(ix) + ' at step ' + str(step) + ' with total points ' + str(points) + ' max: ' + str(max_episode_points) )

                    # if we're learning, then compute and apply the gradients and load the newest weights:
                    if runner.train:
                        loss = self.apply_gradients(policies, action_ixs, rewards, values, terminal, runner)
                        runner.load_state_dict(self.agent.state_dict())

                    max_episode_points = max(max_episode_points, points)
                    min_points = max(min_points, points)

                    # communicate the gathered values to the main process:
                    queue.put((ix, episode_ix, step, rewards, loss, points, terminal))

        except Exception as e:
            string = traceback.format_exc()
            logger.error(str(e) + ' → ' + string)
            queue.put((ix, -1, -1, [-1], -1, str(e) + '<br />' + string, True))

    def record_greedy(self, episode_ix):
        """
        Records the video of the "greedy" run based on the current weights.
        """
        directory = './videos/car-racing/episode-' + str(episode_ix) + '-' + str(int(time.time()))
        player = Player(agent=self.agent, directory=directory, train=False)
        points = player.play()
        logger.info('Evaluation at episode ' + str(episode_ix) + ': ' + str(points) + ' points (' + directory + ')')
        return points

    def apply_gradients(self, policies, actions, rewards, values, terminal, runner):
        worker_agent = runner.agent
        actions_one_hot = torch.tensor([[ int(i == action) for i in range(4) ] for action in actions], dtype=torch.float32)

        policies = torch.stack(policies)
        values = torch.cat(values)
        values_nograd = torch.zeros_like(values.detach(), requires_grad=False)
        values_nograd.copy_(values)

        discounted_rewards = self.discount_rewards(runner, rewards, values_nograd[-1], terminal)
        advantages = discounted_rewards - values_nograd

        logger.info('Runner ' + str(runner.ix) + 'Rewards: ' + str(rewards))
        logger.info('Runner ' + str(runner.ix) + 'Discounted Rewards: ' + str(discounted_rewards.numpy()))

        log_policies = torch.log(0.00000001 + policies)

        one_log_policies = torch.sum(log_policies * actions_one_hot, dim=1)

        entropy = torch.sum(policies * -log_policies)

        policy_loss = -torch.mean(one_log_policies * advantages)

        value_loss = F.mse_loss(values, discounted_rewards)

        value_loss_nograd = torch.zeros_like(value_loss)
        value_loss_nograd.copy_(value_loss)

        policy_loss_nograd = torch.zeros_like(policy_loss)
        policy_loss_nograd.copy_(policy_loss)

        logger.info('Value Loss: ' + str(float(value_loss_nograd)) + ' Policy Loss: ' + str(float(policy_loss_nograd)))

        loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
        self.agent.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(worker_agent.parameters(), 40)

        # the following step is crucial. at this point, all the info about the gradients reside in the worker agent's memory. We need to "move" those gradients into the main agent's memory:
        self.share_gradients(worker_agent)

        # update the weights with the computed gradients:
        self.optimizer.step()

        worker_agent.zero_grad()
        return float(loss.detach())

    def share_gradients(self, worker_agent):
        for param, shared_param in zip(worker_agent.parameters(), self.agent.parameters()):
            if shared_param.grad is not None:
                return
            shared_param._grad = param.grad

    def clip_reward(self, reward):
        """
        Clips the rewards into the <-3, 3> range preventing too big of the gradients variance.
        """
        return max(min(reward, 3), -3)

    def discount_rewards(self, runner, rewards, last_value, terminal):
            discounted_rewards = [0 for _ in rewards]
        loop_rewards = [ self.clip_reward(reward) for reward in rewards ]

        if terminal:
            loop_rewards.append(0)
        else:
            loop_rewards.append(runner.get_value())

        for main_ix in range(len(discounted_rewards) - 1, -1, -1):
            for inside_ix in range(len(loop_rewards) - 1, -1, -1):
                if inside_ix >= main_ix:
                    reward = loop_rewards[inside_ix]
                    discounted_rewards[main_ix] += self.gamma**(inside_ix - main_ix) * reward

        return torch.tensor(discounted_rewards)

For the record_greedy method to work we need the following class:

class Player(Runner):
    def __init__(self, directory, **kwargs):
        super().__init__(ix=999, **kwargs)

        self.env = Monitor(self.env, directory)

    def play(self):
        points = 0
        for step, rewards, values, policies, actions, terminal in self.run_episode(yield_every = 1, do_render = True):
            points += sum(rewards)
        self.env.close()
        return points

All the above code can be used as follows (in the Python script):

if __name__ == "__main__":
    agent = Agent()

    trainer = Trainer(gamma = 0.99, agent = agent)
    trainer.fit(episodes=None)

The importance of tuning of the n-step window size

Reading the code, you can notice that we’ve chosen $15$ to be the size of the n-step window. We’ve also chosen $\gamma=0.99$. Getting those values right is a subject for tuning. The same ones that work on one game or a challenge will not necessarily work well for the other.

Here’s a quick explanation of how to think about them: We’re going to be penalized most of the time. It’s important for us to give the algorithm a chance to actually find trajectories that score positively. In the “CarRacing” challenge, I’ve found that it can take 10 steps of moving “full throttle” in the correct direction before we’re being rewarded by entering the new “tile”. I’ve just simply added $5$ of the safety net to that number. No mathematical proof follows this thinking here, but I can tell you though that it made a huge difference in the training time for me. The version of the code I’m presenting above starts to score above 700 points after approximately 10 hours on my Ryzen 7 based computing box.

Problems with the state being returned from the environment - overcoming with the explicit render

You might have also noticed that I’m not using the state values returned by the step method of the gym environment. This might seem contradictory to how the gym is typically being used. After days of not seeing my model converge though, I have found that the step method was returning one and the same numpy array on each call. You can imagine that it was the absolutely last thing I’ve checked when trying to find that bug.

I’ve found the render(mode='rgb_array') works as intended each time. I just needed to write my own preprocessing code, to scale it down and make it grayscale.

How to know when the algorithm converges

I’ve seen some people thinking that their A3C implementation does not converge. The resulting policy did not seem to be working that well, but the training process was taking a bit longer than “some other implementation”. I fell for this kind of thinking myself as well. My humble bit of advice is to stick to what makes sense mathematically. Someone else’s model might be converging faster simply because of the hardware being used or some slight difference in the code around the training (e.g. explicit render needed in my case). This might not have anything to do with the A3C part at all.

How do we “stick to what makes sense mathematically”? Simply by logging the value loss and observing it as the training continues. Intuitively, for the model that has converged, we should see that it has already learned the value function. Those values — representing the average of the discounted rewards — should not make the loss too big most of the time. Still, for some states, the best action will make the $R_t$ much bigger than $V(s_t)$ which means that we still should see the loss spiking from time to time.

Again, the above bit of advice doesn’t come with any mathematical proofs. It’s what I found working and making sense in my case.

The Results

Instead of presenting hard-core statistics about the model’s performance — which wouldn’t make much sense because I stopped it as soon as the “evaluation” videos started looking cool enough) — I’ll just post three videos of the car driving on its own through the three randomly generated tracks.

Have fun watching and even more fun coding it yourself too!

Build a Smarter Telegram Bot: Integrating a RAG Pipeline for FAQ Answering

What is Telegram?

A Warning on Privacy and Encryption

What is Retrieval-Augmented Generation (RAG)?

What you’ll build

Prerequisites

Setting up the Project for Telegram Bot

Environment and configuration

Try it out

What this bot does

Core structure

Handlers and filters

Running the bot

Setting up the knowledge base

Setting up the RAG Pipeline

Data Ingestion (Indexing the FAQs) handled by setup_rag_chain() method

The RAG Retrieval Logic handled by handle_message() method

New capabilities added to bot.py

Try it out

Adventures in Vibe-Coding with Replit

Project 1: Multi-LLM code review API

The prompt

The results

Project 2: End Point Ecommerce + Hugo site

The prompt

The results (version 1)

The results (version 2)

Feature addition: cart

Bonus: a few extra notes

When I’d use Replit (and similar do-it-all AI agents)

Creating Agentic AI Applications

What Are Agents?

Open-Source Examples of AI Agents

What Are Agentic Applications and Agent Orchestration?

Open-Source Examples of Agentic Applications and Orchestration

Writing Agentic Applications

Building with CrewAI

Building A Dynamic Agent

Use Cases of Agentic AI Applications

Challenges Ahead

Wrapping up

Making Blog Search Smarter with LLMs and Open WebUI

What “LLM Expanded Search” actually does

How it works

Why we chose Open WebUI

What’s next: Moving more into Open WebUI

What you’ll notice when using it

Vector Search for the End Point Blog

Vector Search: The Future of Finding What Matters

What Is Vector Search?

How Does It Work?

Embedding Generation

Storing Vectors

Similarity Search

A Simple Vector Search Implementation

Why Vector Search is Powerful

Real-World Examples

Challenges Ahead

Wrapping Up

Deploying LLMs Efficiently with Mixture of Experts

1. Why MoE?

2. Key Ideas

3. How a Token Moves Through an MoE Layer

4. Why It Saves Compute

5. Quick Start with Ollama

Open‑Source MoE Models — At a Glance

6. Deployment Tips

Useful links

7. Takeaway

Converting MIDI to KML using AI: Bach’s Notes in the Hills of Greenland

How to try out the application

Issues

Using AI for blog post image processing

Introducing the End Point AI Portal for Our Customers

What’s in the Portal

Why We Are Doing This

How to Get an Account

Implementing SummAE neural text summarization with a denoising auto-encoder

Preliminaries

Datasets

Data Ingestion (Indexing the FAQs) handled by `setup_rag_chain()` method

The RAG Retrieval Logic handled by `handle_message()` method

New capabilities added to `bot.py`