Making Blog Search Smarter with LLMs and Open WebUI

2025-09-29T00:00:00+00:00

We recently released LLM Expanded Search for our blog’s vector search. It builds on what we covered in our earlier posts about AI-powered search and vector search basics. Here’s how we built it with our internal AI setup (Open WebUI running an OpenAI-compatible API), why it makes search better, and what’s coming next.

What “LLM Expanded Search” actually does

Here’s the basic idea: when you search for something, we first ask an LLM to come up with related terms and phrases. Then we search for all of those terms, not just your original query.

Your search gets expanded by an open-source LLM through our AI portal (Open WebUI with an OpenAI-compatible API)
Those extra terms give our vector index more ways to find posts that match what you’re looking for
We combine the results, remove duplicates, and sort by relevance before showing the best matches with snippets and links

This really helps with short or vague searches where regular vector search might miss the relevant context — for example, “S3” refers to Amazon S3, which is a cloud object storage system, so whereas “S3” doesn’t provide enough context for a useful vector search. An LLM can expand this short search and include context about cloud object storage in general, as well as give enough context to return results about S3.

How it works

The frontend is pretty straightforward: our search bar has two options, “Search” (just hit Enter) and “LLM Expanded Search” (Shift/Ctrl/Command+Enter).

When you use expanded search, here’s what happens:

We call our Open WebUI endpoint with a prompt that asks for 8–15 related terms
We turn both your original query and the expanded terms into embeddings
We search our vector store with all these terms and combine the results
Caching and rate limiting keep things fast and cheap

Here’s a simple example of how we expand queries:

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.getenv("OPENAI_BASE_URL"),   # e.g., http://openwebui.local/api/v1
    api_key=os.getenv("OPENAI_API_KEY")      # token managed in your environment
)

def expand_query(raw_query: str) -> list[str]:
    messages = [
        {
            "role": "system",
            "content": (
                "You expand a short search query into a concise, comma-separated list of "
                "synonyms and closely related phrases (8–15 items). No explanations."
            )
        },
        {"role": "user", "content": raw_query}
    ]
    res = client.chat.completions.create(
        model=os.getenv("OPENAI_MODEL", "local-llm"),
        messages=messages,
        temperature=0.2,
        max_tokens=200,
    )
    text = res.choices[0].message.content
    return [t.strip() for t in text.split(",") if t.strip()]

After that, we embed the original query and the expanded terms, search the vector index, then sort by score and drop duplicates so each post appears once. Finally, we render concise snippets.

For example, after a similarity search you can rank and de-duplicate like this:

# given: results = [(doc, score), ...]
valid = [(d, float(s)) for d, s in results if float(s) > 0.05]
valid.sort(key=lambda x: x[1], reverse=True)  # highest score first

seen = set()
unique = []
for doc, score in valid:
    src = doc.metadata.get("source", "")
    if src not in seen:
        unique.append((doc, score))
        seen.add(src)

# unique now holds top ranked, de‑duplicated posts

Why we chose Open WebUI

A few reasons made Open WebUI the right choice:

It’s open source and works great self-hosted
The OpenAI-compatible API means we can drop it into existing code
We can use whatever models and inference backends we want
It’s easy to experiment with different prompts and workflows

What’s next: Moving more into Open WebUI

We’re looking into moving more of the search pipeline directly into Open WebUI workflows:

Query expansion (LLM)
Vector retrieval (custom tool that hits our index)

This would give us tighter integration, fewer network calls, and simpler deployment, and make it easier to try new approaches.

What you’ll notice when using it

Short searches work way better, you get more relevant results and fewer dead ends
It’s still experimental, so sometimes results might drift into related topics. Stick with regular “Search” if you want more exact matches
We cache common terms to keep things smooth

Give it a try at our blog. Just use the search bar in our header: press Enter for regular search, or Shift/Ctrl/Command+Enter for LLM Expanded Search.

Want to know more about why we built this? Check out the announcement and vector search posts linked above.

If you’re interested in setting up LLM-expanded vector search or running something similar self-hosted with Open WebUI, we’d love to help out.

Announcing End Point Ecommerce, Our New Open Source Project

2025-08-21T00:00:00+00:00

Today we’re pleased to announce a new open source project: End Point Ecommerce.

End Point Ecommerce is a minimalist ecommerce system that’s quick to set up and easy to understand. It is meant for developers to own, adapt, customize, and extend.

You can learn more about it on our landing page. You can also fork it today on GitHub. There are running demos here:

Are you interested in using End Point Ecommerce for your ecommerce site? Feel free to contact us and we can work together to help you deploy, customize, and maintain it.

Key features

Built with ASP.NET, PostgreSQL, and Authorize.net
Multiple deployment strategies, including Docker Compose
Code base designed to follow Clean Architecture and Domain Driven Design concepts, without being dogmatic about it — simplicity trumps all
Good test coverage with xUnit
Includes a backend admin portal for managing the store, a REST API for building user-facing store frontends, and a simple web frontend
Limited functionality — implements the bare minimum of ecommerce use cases
Product catalog, shopping cart, order submission, email delivery

How we got here

We recently had the opportunity to help one of our clients migrate their ecommerce site. They were using an established ecommerce engine and were having various problems with it. Their requirements were simple, and they didn’t need all the bells and whistles of big ecommerce solutions. They didn’t want to deal with the overhead, bloat, and difficulty of finding maintainers.

During initial investigation and planning, we didn’t find any options that were small and easy to set up, understand, and customize, so we decided to build them a new site from scratch.

We realize that there may be others out there in the same boat that we were. You want to set up a custom ecommerce site but you don’t want to deal with the complexity of bigger off-the-shelf solutions. You want something which you can thoroughly understand quickly, and modify to your heart’s content, as if it were your own code base that you developed from scratch.

Who is this for?

This project is meant for .NET developers who want a solid and simple foundation to build and customize ecommerce sites. To that end, the feature set is very basic, the code base is very straightforward, and the documentation is very developer-oriented. The idea being that developers can quickly get up to speed with how everything works, develop ownership over the code base, and implement their bespoke use cases.

This project is not meant for non-technical folks. There is no installation wizard, no templating engine, no plugin framework, no no-code customization capabilities.

If you want to set up a simple online store, or develop a highly customized one from scratch, think of this as a way of skipping the first few steps of building the foundational architecture and basic functionality.

Please check it out! And do whatever you want with it!

New Cesium KML-CZML Editor Features: Custom Data & Styling, Google 3D Tiles, and More

2025-06-13T00:00:00+00:00

I have made some updates to the Cesium KML-CZML editor I created and maintain.

The most important additions and changes are:

Support for Google 3D tiles
Support for writing many more features, including interpolation and time series data for some properties. There are still no editing capabilities for these properties, but while previously the editor would strip these values from the data, it will now copy them into the output file.
Export to KML and KMZ
Support for custom data and styling using that custom data
Switched frontend framework from Vue to React

Adding support for Google 3D tiles is what caused me to create this major version update. In a nutshell, Cesium has its own way of adding reactivity to Entities and Vue doesn’t always play nice with it. If I add Google 3D tiles to the scene, it looks like that Cesium Entities have some references to the scene and that causes Vue to apply reactive getters and setters to the whole scene.

So, I’ve switched to using React because it’s easier to control which parts should be reactive, as well as when and how you update Cesium Entities and UI components.

The next important piece is the CZML exporter. The main issue here is that I don’t want to strip away features even if I don’t support editing them.

Even if I don’t have the UI to edit certain properties or animations for properties, a user can load a CZML document with a mixture of supported and unsupported features without losing the unsupported features. That means that the exporter should be more robust and feature complete than editor itself.

I will probably separate the exporter as standalone package because it is valuable on its own — I couldn’t find a way to export Cesium entities as CZML within the library.

Custom data & styling

This part is my favorite. I have some experience creating maps, and when you are working with maps you are not focused on polygons or billboards, you are focused on the data, while graphics are just a tool to represent it.

The previous iteration of the KML-CZML editor was more oriented toward importing KML and massaging it to have the same look in Cesium as it would in Google Earth. That puts focus on graphical features. This time I wanted to make it easy to edit data and style things based on the data.

Demo

We’ll load in some demo data from New York elections in Mike4326.geojson.

From here, we’ll click on the “Data table & Conditional Styling” button. Then, in the menu that opens, we’ll set up Color by value so that we can display different regions differently by our selected attribute. Note the “Mike_prc” column which displays a percent value. The app will select a color from the gradient based on this value.The app will select a color from the gradient based on this value.

To group values together, we’ll toggle the “Fixed gradations” switch. Then hit the “Preview” button. By default, there are 9 gradations.

We want fewer gradations than that. Let’s try out 6.

Not much difference, but we’ll go with the 6 gradations. I’ll hit apply, and you can see the colors are now displayed in the “Style” column, while the “Preview” column has been hidden.

Let’s see how it looks on the map.

Very nice! Now let’s add some labels for a different attribute, the election district.

You can see the text color defaults to black, which won’t read well on our colored districts. Let’s change it by hitting the “edit” button next to the color.

I changed it to a light gray. Let’s look at the map now.

Lots of labels! This is quite a dense area, so you would probably want to zoom in further to see the labels more clearly.

See my previous blog post introducing the Cesium CZML-KML Editor for more about the Cesium CZML-KML editor.

Place Labels in Cesium using Vector Data Tiles

2025-02-13T00:00:00+00:00

Cesium is like an IKEA globe: it can do almost anything, but you have to put it together yourself. One major missing feature is labels for cities. Compare the default views of Google Earth and Cesium, and you’ll notice this immediately.

Left: Google Earth default view, with labels and state lines. Right: Cesium default view, with no labels or markers.

If you’re reading this post, you might be able to find any major U.S. city without any clues, but most people can’t. You can overlay imagery that includes labels as part of the image, but this approach has a significant drawback: as soon as you rotate the view away from north-up, the labels become difficult to read.

In Google Earth, this issue is handled using vector data overlays. Labels, borders, and other similar elements are examples of how useful vector overlays can be.

Cesium also supports vector features such as labels, polylines, and polygons. You can create them manually or use CZML or KML to add such features to a scene. If you use KML, Cesium translates it internally, but the result is the same. Vector features have distinct advantages for 3D geographic data, with labels being the most obvious example.

Challenges of Vector Data in Cesium

Displaying vector data in Cesium comes with some challenges. Since we cannot load and display all data at once, we have two options: keep it as vector data or rasterize it into imagery. Either way, we need to tile the data, breaking it into small chunks and generalizing or simplifying it.

Continuing with the example of labels: tiles covering large areas should include only essential labels (e.g., capital cities) to avoid excessive clutter. Similarly, for polylines such as roads or boundaries, we need to prioritize important roads and simplify their geometries.

Most GeoServers and data providers serve vector tiles encoded as GeoJSON, KML, or other binary formats. However, integrating them into Cesium or Google Earth requires additional steps.

There are two big considerations that need to be taken into account when integrating tiled vector data into any application:

How particular geometric features (points, lines, and polygons) are encoded for every tile. Also, what are their attributes and how should they be represented and styled? This involves considering whether a point should have a label or an icon, what color should it be, etc.
How the data is arranged. This includes how many levels it has, how you fetch new tiles, what the area each tile covers is, and how the software you are integrating with (in our case, Cesium) manages rendering, loading, and unloading the data.

In Google Earth, KML covers both issues in one technology. KML describes features and their styling, while for the arrangement of tiles you can use Network Link Regions. To put it simply, Network Link Regions define an area and as soon as the camera gets close enough to that area, Google Earth loads a new chunk of data for that area. You can read more about that in the KML docs.

In Cesium, this process is covered by multiple technologies, and not all of them work together. We have several encoding and styling options (CZML, KML, GeoJSON), but none of them natively support tiling.

On the other hand, Cesium 3D Tiles work well for large 3D meshes and point clouds, but they do not support CZML, KML, or GeoJSON as tile content.

To bridge this gap, Cesium provides the Vector Data tile format and 3D Tiles Styling.

Although they are still experimental, these features work with the standard Cesium distribution without any hacks or undocumented options.

Since I previously implemented city labels in Cesium, I decided to revisit the task to do it with this new standard in mind. In my earlier implementation, I had to rely on undocumented Cesium features to determine which tiles should be loaded and unloaded. More details about that approach can be found in my previous blog post.

Technical Implementation

For this tileset generation, I use two standards:

Vector Data tile content with tileset styling
- Vector Data tile content defines how each tile is encoded
- Tileset styling determines how points, lines, and polygons are rendered
Implicit tiling
- Implicit tiling defines tile arrangement and availability

Both standards use a mix of JSON and binary-encoded features. In a nutshell, to encode a point with a label you have to write a vctr file, a subtree file and a tileset.json file.

Each vctr file consists of the following sections:

Header: Defines technical data, like the size of the sections in bytes and their offsets
Feature Table: Defines how to read points, lines, or polygons from the binary data section In our case we just specify how many points we have
Batch Table: Defines data specific attributes, in our case that will be a name for a point
Binary data with positions and indices: Coordinates of the points optimized for storage size

Subtree files encode which child tiles are available, which of them have content, and what are the available subtrees.

tileset.json puts all that information together, defining what region the whole tileset covers, specifying that we are using implicit tiling, and defining URL templates for fetching vctr and subtree files.

Add a style which tells Cesium that it should create a label for every point with its name attribute. And congratulations, we have our first set of labels!

Managing Uneven Label Distribution

But as soon as we zoom in we hit a problem: we have too many labels too close to each other.

In order to avoid having the globe polluted with messy clumps of labels when zooming in, we have to discern how the visibility of 3D tile content is determined by Cesium. A key concept for handling this is assigning a geometricError value to each tile. The exact value might be complex to calculate, but the general rule is: the larger the geometricError, the bigger the distance from which a tile should be visible.

So as you zoom in closer, labels from tiles with bigger geometricError will appear sooner. Furthermore, for implicit tiling the geometricError of children tiles should be half of what their parent tiles’ geometricError values are. Practically speaking, this means that the geometricError value assigned to the root tile cascades throughout the system and changes how soon all of the labels become visible. To have some of the labels appear later than the others we have to push them down the tile tree.

The first thing I tried was to set a minimum tree depth for a label based on city size, but the end result was pretty much the same:

The root cause of the points clumping up is not small towns appearing too early, it’s how the populated places themself are distributed. “The Bronx” and “Manhattan” are just big and too close to each other and to “New York City”. If I try to push “Manhattan” deep enough so it doesn’t overlap with “New York City” based just on its size we won’t be able to see Boston or Baltimore or Hamilton, although they have plenty of space to be shown at this zoom level. Hence, we need a way to cluster points and determine point depth based on how much space there is between neighbours.

Clustering Labels

Clustering is a well-established problem in computational geometry, and there are plenty of libraries to solve it, but if I used one of them I would also have to integrate it with the tree structure of the tiles. So instead I’ve implemented a straightforward approach that is effective, although not 100% correct mathematically:

For every level of the tree, calculate a distance threshold. The threshold gets smaller and smaller with every level of the tree. Let’s follow the same pattern as the tiles’ geometricError for this threshold
For every point to be inserted into the tree, calculate the distance to a closest point in the tree
1. If the distance is more than the threshold, just insert the point. If the distance is smaller than the threshold, let’s calculate how many levels we should skip to get the threshold small enough to insert the point at this level
2. If an inserted point is more important than an existing point, then put the inserted point at the current level and push the old one down the tree instead

And so, there you have it:

Next Steps

Further steps I have in mind to expand on this work with vector tiles include:

Enhancing styling options
Encoding polylines
Encoding polygons

Notes

The project is available on GitHub under the Apache-2.0 license: https://github.com/EndPointCorp/cesium-vector-data-tiles.

Place data is from GeoNames.org under the CC-BY license.

Screenshots use Bing Imagery, ArcGIS World Imagery, and Google Earth.

Exploring Geodatabase Files

2024-08-14T00:00:00+00:00

One of our clients recently provided us with a dataset of real estate properties that they manage, and asked us to generate content based off of the points and polygons in the dataset.

We will walk through the process of extracting polygons, placemarks, and other info from a geodatabase file and converting them into separate KML files using the ogr2ogr command-line tool, adding some logic to the data selection to limit the subset of features. We will also explore the GDB file using the GDAL Python library to export the data as JSON for use in other scripts.

Prerequisites

Basic understanding of geospatial data
Installed versions of the GDAL/OGR library
A geodatabase file (.gdb, .gdb.zip, or .shp)

A first look into the contents of the GDB file

ogrinfo example.gdb.zip

This command will list all layers that are available in the dataset. Add a specific layer as a parameter to the same command, and it will output all the fields, their types, and values for every feature in the layer.

ogrinfo example.gdb.zip a_layer_name

The parameter -so can be used to omit the values from the output and get only the layer field names and types:

#$> ogrinfo example.gdb.zip Land_Points -so
INFO: Open of `example.gdb.zip'
      using driver `OpenFileGDB' successful.

Layer name: Land_Points
Geometry: Point
Feature Count: 219
Extent: (-122.699891, -23.590280) - (139.763352, 59.622821)
Layer SRS WKT:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[2],
        ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["unknown"],
        AREA["World"],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
FID Column = OBJECTID
Geometry Column = Shape
asset_type: Integer (0.0)
asset_id: String (15.0)
property_code: String (255.0)
full_address_text: String (255.0)
land_name: String (255.0)
land_address1: String (255.0)
land_city: String (255.0)
land_sate_code: String (50.0)
land_postal_code: String (10.0)
land_country_code: String (50.0)
division_name: String (255.0)
region_name: String (255.0)
market_name: String (255.0)
submarket_name: String (255.0)
supplemental_portfolio_name: String (255.0)
ownership_name: String (255.0)
land_held_for_sale_acre: Real (0.0)
land_held_for_sale_hect: Real (0.0)
land_held_for_development_acre: Real (0.0)
land_held_for_development_hect: Real (0.0)
total_land_acre: Real (0.0)
total_land_hectare: Real (0.0)
buildable_area_sf: Real (0.0)
buildable_area_sm: Real (0.0)
buildable_area_tsubo: Real (0.0)
land_latitude: Real (0.0)
land_longitude: Real (0.0)
geocoded: Integer (0.0) DEFAULT 0
globalid: String (0.0) NOT NULL
created_user: String (255.0)
created_date: DateTime (0.0)
last_edited_user:  String (255.0)
last_edited_date: DateTime (0.0)

This command shows you information for a single layer, which can be helpful if you are only looking for certain values. However, the geographic data is not much to look at in the terminal. To visualize it, we can convert the data to KML, which applications like Google Earth or Cesium can render while keeping the information as text that can be read:

ogr2ogr -f "KML" example_output.kml example.gdb.zip layer_name

-f "KML" specifies the output format
example_output.kml is the name of the output KML file
example.gdb.zip is the path to the geodatabase file
layer_name is the geodatabase layer to export

This command will export into a KML file:

All the geometries in the layer, be it placemarks or polygons in our case
All the other fields of information as extended data, which will show up for each feature as a balloon table when visualized in Google Earth

The `ogr2ogr` `-sql` option

The default balloon popup was not the outcome we wanted for this KML file. We first used sed to remove all the extended data from the KML files, but after looking into it a bit further, we found an ogr2ogr option, -sql, that made the data filtering easier. This option lets us add a query to the command just like getting the data from a SQL database.

1. Extract property names

To create a KML file with just the names of the properties, look up the layers and fields in them using ogrinfo and find the points layer that has the names—in this case, Layer_Points. Then, add the desired SQL query to the command.

ogr2ogr -f "KML" output_names.kml example.gdb.zip -sql "SELECT name FROM Layer_Points"

2. Extract polygons only

The polygon geometries are stored in Layer_Polygons. They can be selected using the special OGR field OGR_GEOMETRY that refers to the geometry of the selected layer:

ogr2ogr -f "KML" output_polygons.kml example.gdb.zip -sql "SELECT OGR_GEOMETRY FROM Layer_Polygons"

3. Create KML with pins and names

To create a KML file with property names as placemarks with pins, we just select the name. The point placemark seems to be included with all data:

ogr2ogr -f "KML" output_pins.kml input.gdb layer_name -sql "SELECT name FROM layer_name"

4. Create KML with pins only, no names

To get the points with nothing else, we use the SQLite MakePoint function, which selects a list of points from the KML.

ogr2ogr -f "KML" output_pins.kml input.gdb layer_name -dialect SQLite -sql "SELECT MakePoint(land_longitude, land_latitude) AS geom FROM Land_Points"

5. Extract limited properties

For extracting a limited number of properties, we can add the WHERE clause to the SQL command:

ogr2ogr -f "KML" output_pins.kml input.gdb layer_name -sql "SELECT name FROM layer_name"

We can also use the ogr2ogr -where flag to use that part of the query only:

ogr2ogr -f "KML" limited_output_polygons.kml input.gdb layer_name -where "ID IN (1, 2, 3)"

6. Run it all in a Python script

There is a Python GDAL library, which I will cover the basics of later, but first, here is a simplified example using Python subprocess to run the ogr2ogr commands we tested in the terminal.

import subprocess

gdb_file = "example.gdb.zip"
column_name = "p_code"
elem_str = "'a1', 'a2', 'b1', 'b3'"

sql_land_labels = f"SELECT land_name FROM Land_Points WHERE {column_name} IN ({elem_str})"
subprocess.run(['ogr2ogr', '-f', 'KML', 'land_labels.kml', gdb_file, '-sql', sql_land_labels])

sql_land_points = f"SELECT MakePoint(land_longitude, land_latitude) AS geom FROM Land_Points WHERE {column_name} IN ({elem_str})"
subprocess.run(['ogr2ogr', '-f', 'KML', 'land_points.kml', gdb_file, '-dialect', 'SQLite', '-sql', sql_land_points])

sql_land_polygons = f"SELECT OGR_GEOMETRY FROM Land_Polygons WHERE {column_name} IN ({elem_str})"
subprocess.run(['ogr2ogr', '-f', 'KML', 'land_pols.kml', gdb_file, '-sql', sql_land_polygons])

This script will create the three different KML files that we usually need for our presentations. We used variables for the gdb_file, column_name, and elem_str command line parameters to make the script easy to use for selecting other data. We also use them in other scripts to join, apply custom styling, and add regions to the KMLs, which will be covered in another blog post.

7. Extract all data as JSON from one layer using the Python GDAL library

I probably should have started here, but I only used the Python library later on in the process to join the gdp.zip file data with data from other sources (spreadsheets, emails, etc.).

First, install the GDAL library:

pip install gdal

Then you can use the following Python script, changing as necessary the gdb_file, layer_name, and the unique field chosen to structure the data. The code is explained in the comments.

#!/bin/python

import json
from osgeo import ogr

# Open the Geodatabase file
driver = ogr.GetDriverByName('OpenFileGDB')
gdb_file = 'example.gdb.zip'
data_source = driver.Open(gdb_file, 0)

# Get the layer
layer_name = 'Building_Points'
layer = data_source.GetLayerByName(layer_name)
if layer:
    # Get the layer definition
    layer_defn = layer.GetLayerDefn()

    # Get the number of fields in the layer
    num_fields = layer_defn.GetFieldCount()

    # Initialize an empty list to store field names
    field_names = []

    # Iterate over each field and add its name to the list
    for i in range(num_fields):
        field_defn = layer_defn.GetFieldDefn(i)
        field_name = field_defn.GetName()
        field_names.append(field_name)

    print("Field names:", field_names)
else:
    print(f"Layer '{layer_name}' not found.")

all_info = {}

# Iterate over all features and organize all field names under a unique one for the dictionary structure
for feature in layer:
    if feature is not None:
        globalid = feature.GetField('globalid')
        all_info[globalid] = {}
        for fn in field_names:
            all_info[globalid][fn] = feature.GetField(fn)

# Close the data source
data_source = None

# Save information to a JSON file
output_file = 'info.json'
with open(output_file, 'w') as json_file:
    json.dump(all_info, json_file, indent=4)
print(f"Information saved to {output_file}")

The script will create a JSON file with all the information on the layer.

Conclusion

Following these steps, we can efficiently manage and visualize geospatial data using ogr2ogr, SQL, and KML, and in some cases, JSON. These methods allow for a high degree of customization and can be tailored to specific project requirements.

Additional Resources

The Perl and Raku Conference 2024

2024-08-02T00:00:00+00:00

Next Generation of Perl

I attended The Perl and Raku Conference in Las Vegas, NV, which took place June 25–28, 2024. It was HOT outside (over 40 °C/110 °F) but we stayed cool inside at the Alexis Park Resort.

Curtis Poe (Ovid) got things started with the keynote encouraging us to Party Like It’s 19100+e^iπ, and reminded us that Vegas is lexically scoped (what happens in Vegas stays in Vegas)! More importantly he reminded us that Perl is about people, not just the technology. The Perl community has been meeting all over the world since 1999, with this being the 25th anniversary of the first The Perl Conference (aka YAPC::NA).

Ovid Keynote

Meeting in person with people who you interact with primarily through digital channels, code commits, and MetaCPAN documentation really highlighted the importance of the community. On the first day, I messed up timezones, showed up an hour before registration opened, and witnessed the conference organizers and core members arrive and greet each other with hugs. I also enjoyed visiting with one of the very welcoming board members of The Perl and Raku Foundation (TPRF).

Many of the speakers and attendees put a “Hallway++” sticker on their badge which simply meant “talk to me, I’m here to meet and get to know people”. At breakfast one morning, I had the privilege of sitting with Jason Crome, the core maintainer of Dancer, a framework that I have used extensively. It was amazing to be able to pick the brain of one of the people who has intimate knowledge of the software.

Dancing with Cromedome

The Perl community is large and diverse, which is reflected in the Science Perl Committee and the all-Perl Koha Library Software in use at over 4,000 libraries and with its own annual conference. It was cool to hear about the Glue Photo Project, making algorithmic music, and gaming with the TinyNES.

Every community will experience conflict and this one is no different. The impact of Sawyer’s resignation at TPRC 2023 could be felt at this conference, and in response the community is focused on making things better with a new standards of conduct.

It wouldn’t be a conference in 2024 without talk of AI. We had some Musings on Generative AI and an introduction to PerlGPT, A Code Llama LLM Fine-Tuned For Perl.

There were, of course, a lot of talks about actual Perl code!

One of the things I enjoy about attending conferences is discovering things that I wasn’t looking for. Chad Granum gave a lightning talk on goto::file and I noticed the use of line directives which can be extremely helpful in debugging evaled code. For example, let’s say you are evaling subs into a hashref and then calling them like so:

my $sub1 = "sub {\n  print 'foo';\n  print 'bar';\n print 'baz';\n}";
my $sub2 = "sub {\n  print 'foo';\n  print 'bar';\n warn 'baz';\n}";
my $Sub = {
  sub1 => eval $sub1,
  sub2 => eval $sub2,
};
$Sub->{sub1}->();
$Sub->{sub2}->();

Any warn or die output will give you a line number, but no context as to which sub it originated from:

baz at (eval 2) line 4.

Making use of a line directive when evaling the subs like this:

my $sub1 = "sub {\n  print 'foo';\n  print 'bar';\n print 'baz';\n}";
my $sub2 = "sub {\n  print 'foo';\n  print 'bar';\n warn 'baz';\n}";
my $Sub = {
  sub1 => eval qq(#line 1 "sub1"\n$sub1),
  sub2 => eval qq(#line 1 "sub2"\n$sub2),
};
$Sub->{sub1}->();
$Sub->{sub2}->();

Will now result in a much more friendly:

baz at sub2 line 4.

And in true open source fashion, this has been turned into a pull request for Interchange!

The Las Vegas Sphere

I always enjoy hearing from others in the community about CPAN modules that are in their toolbox. I learned about DBIx::QuickDB, which you can use to spin up a database server on the fly, which removes the need for a running database server for tests, and also enables running concurrent tests which require a database server that would otherwise conflict. Combine this with a DBIx::Class schema and DBIx::Class::Fixtures and you have a very nice way to run some tests against fixed data:

use DBIx::QuickDB PSQL_DB  => {driver => 'PostgreSQL'};
my $dbh = PSQL_DB->connect;
$schema = Some::Schema->connect( sub { $dbh }, { on_connect_do => ["..."] } );
$schema->deploy();
my $fixtures = DBIx::Class::Fixtures->new({ config_dir => '...' });
$fixtures->populate({ no_deploy => 1, schema => $schema, directory => '...' });

ok($schema->resultset('Foo')->count >= 1, 'database populated');

Damian Conway gave a keynote on The Once and Future Perl, showing how far Perl has come as a language and how its rich history can be leveraged into the future: “if you can envisage what you could have done better in the past, then you can probably think of ways to make the future brighter!”.

He showed off his new Multi::Dispatch module which you can use now to write incredibly extensible (and beautiful) code. Here it is in action with a simple Data::Dumper clone in 5 lines of code:

use v5.26;
use Multi::Dispatch;

multi dd :before :where(VOID) (@data)   { say &next::variant }
multi dd ($k, $v)                       { dd($k) . ' => ' . dd($v) }
multi dd ($data :where(ARRAY))          { '[' . join(', ', map {dd($_)}                 @$data) . ']' }
multi dd ($data :where(HASH))           { '{' . join(', ', map {dd($_, $data->{$_})} keys %$data) . '}' }
multi dd ($data)                        { '"' . quotemeta($data) . '"' }

say dd ['foo', { bar => "baz" }];

With Smartmatch (given/when) scheduled for deprecation in 5.42 he has written a drop-in replacement that uses Multi::Dispatch: Switch::Right, which addresses the issues with the original implementation that was in core.

Also on display was the new class syntax introduced into core in 5.38 with use feature 'class'. Here’s an example of how it looks:

use v5.40;
use feature 'class';

class Point {
  field $x :param = 0;
  field $y :param = 0;

  method describe () {
      say "A point at ($x, $y)\n";
   }
}

Point->new(x => 5, y => 10)->describe;

If you are waiting for Perl 7, Damian is here to tell you that the future is now; you don’t have to wait for Perl 7 (or 17), it’s Perl 5 with Multi::Dispatch and use feature 'class'!

The last day of the conference provided an opportunity to go deeper into learning the new class syntax with a workshop on building a roguelike adventure game from scratch.

So what’s next? The London Perl & Raku Workshop is taking place on October 26, 2024 and Perl 5.42 is just around the corner!

—JAPH (Just Another Perl Hacker)

An OpenStreetMap Editor for Adding Public Transit Data Using GTFS

2024-05-08T00:00:00+00:00

You can think of OpenStreetMap (OSM) as Wikipedia for maps. While it may not be as well known as Google Maps or Apple Maps, OSM becomes indispensable when you need data, rather than just a visual representation on your phone. The only other real option for data retrieval is to consult a local agency, so if you’re in search of a comprehensive and global cartographic dataset, OSM is the go-to choice.

OSM also excels in providing navigation for pedestrians and cyclists. For the past decade, I’ve navigated the US, Canada, and Europe using OSM through the OsmAnd app, a dedicated Android application for OpenStreetMap.

Overall, my experience has been quite positive, except for one significant weakness: public transportation. Specifically, I’ve been missing the convenient access to timetables for buses, trams, or any other form of public transit.

A significant part of the challenge in making public transportation data readily available on OSM and its associated applications is the fact that the OSM data model isn’t particularly well suited for this type of information. While OSM can store the location of bus and train stops, the actual timetables change so often that keeping them up to date in OSM is essentially impossible.

Enter the General Transit Feed Specification (GTFS), a widely adopted standard for managing public transportation data. I wanted to explore ways to integrate OSM with GTFS effectively.

My OpenStreetMap editor for GTFS data

A simple approach for addressing this challenge would be to overlay an OSM map with stops from GTFS. While this would be relatively straightforward, it doesn’t leverage OSM’s biggest strength—its vibrant community. One of OSM’s most significant advantages is that it dynamically incorporates user contributions: as soon as the community accesses useful data, members enhance the details and accuracy of its mapping data. Therefore, I’ve been working on truly integrating GTFS data with OSM, creating a cohesive mapping experience rather than settling for a disjointed overlay.

Although the implementation is a bit trickier, I believe the better approach for handling this challenge involves merging GTFS stops with those already present in OSM and incorporating GTFS stop codes into OSM data. Hence, I’ve developed a single-page online editor. This tool facilitates the creation, editing, and deletion of stops within OSM. It also highlights stops based on their alignment with stops specified in a given GTFS file.

Editing the routes wasn’t a priority, yet I found it necessary to match GTFS routes and trips with OSM routes and their variants. Displaying trips shows which part of a street a route uses and also which side of a street bus stops are located on.

I’ve deployed the app on my GitHub, where you can try it out: https://kiselev-dv.github.io/osm-gtfs/.

How to use the app

To start, download a GTFS data dump. They are usually published by local transit companies. As an example, I’ll use data from Halifax, Nova Scotia.

Open the GTFS ZIP file in the editor. The editor should parse stops from GTFS.

Next, query OSM data and select “OSM Tag with GTFS stop code.” After OSM data is loaded you will get OSM tags statistics which might help you to see if there are some popular tag combination for GTFS data in the area. OSM doesn’t have strict data model, but as a rule of thumb for that type of data ref or gtfs:ref should work. If there are multiple agencies sharing a stop and each of them publishes its own GTFS data, you can use something like gtfs:<agency_name>:ref. By default the editor will substitute the most popular tag containing ref or gtfs.

Now you can match stops from OSM with stops from GTFS. Matches are color coded:

Blue: matching stops
Red: stops which are present in GTFS but not mapped in OSM (stops to add)
Black: stops mapped in OSM but not matched with GTFS stop (probably outdated stops which are no longer in use and may be deleted, or stops not used by the agency whose GTFS data you are using)

Adding stops to OSM

There are two main cases for adding or matching a stop:

There is no stop in OSM at all and we want to create a new one
There is a stop in OSM and we want to edit it and add GTFS data

Select one of the GTFS stops. On the left panel there will be two buttons: “Create” and “Assign”.

The “Create” button will switch the map into satellite mode, create a new OSM stop with GTFS code, and add a stop name at the point where you click. When creating a stop or editing its location, pay attention to which side of the road it should be on. To simplify that, you can highlight one of the routes in the “Routes on this stop” section.

The “Assign” button allows you to select one of the existing OSM stops, and set its GTFS code.

Demo videos

You can see a video on my YouTube channel with the latest updates to the editor, along with a live editing session.

I have also created a YouTube playlist with more demos.

An important note

An important note is that I do not upload results directly into OSM. Instead, I generate an XML file for an OSM changeset which you can view and edit in JOSM or another editor. I do this partly so I do not have to deal with authentication, but more importantly, because I see editing public transport relations in OSM as an advanced topic and it makes sense for editors to demonstrate a higher level of sophistication than general users. But I might change my mind about this if I get feedback that it would be desirable.

My app is built using React and Leaflet. You can find the code on my GitHub: https://github.com/kiselev-dv/osm-gtfs

Cesium Labels

2023-05-24T00:00:00+00:00

Improving place labels in CesiumJS, the open source JavaScript library for 3D globes and maps, has been a longstanding request from some of our VisionPort clients. The display of labels embedded in imagery has not been up to their expectations. The names look upside down as the globe is rotated and there’s no option to change the language:

Cesium has vector labels which allow you to anchor some text to a point on a map which will always be aligned with the camera. However, there is no ready-to-use solution to display city names in particular and no way to load them according to a specific zoom level.

To improve performance when displaying labels in Cesium it would make sense to load labels as a tile tree and only show some of the top of the tree at different zoom levels in a manner similar to how KML Regions with NetworkLinks work. Unfortunately, although Cesium supports KML, it doesn’t support KML Regions and NetworkLinks updates on Region change.

Another off-the-shelf solution that might conceivably work would be to use Cesium 3D tiles, but unfortunately, tiles do not support 2D Billboards.

Calculating regions and their level of detail is complicated, but Cesium already does most of that work for us. Cesium already calculates visible tiles and their corresponding levels of detail for ImageryProviders. An ImageryProvider is supposed to load imagery for a given tile’s coordinates and zoom level, which is almost what we want, except that we want to render some 3D primitives for labels, not 2D images for the earth surface.

We can read what tiles are going to be rendered by Cesium and store them in a variable:

const tilesToRender = viewer.scene.globe._surface._tilesToRender;

Then, we get the TMS coordinates out of the visible tiles on screen. Then we calculate the difference between the currently visible tiles and the new ones. We grab all the new tiles that we need from that list, and then we create the Entities with labels and some other styling properties.

Backend and data source

To source the labels we use GeoNames. They have a nice dataset for cities with their population included in the data. We take the city labels they provide and store them into a quadtree. Each node in the quadtree has 10 labels associated with it, starting with the largest population at the top of the data structure and going down. Based on the altitude we traverse a certain distance down the quadtree returning all cities from each node that corresponds to the tile requested.

Here is a video demonstrating how labels display with this new feature we’ve developed:

Future Plans

Currently, the entity creation is done on the frontend, but we are planning to move this to the backend sometime soon. There is also some crowding of the labels when zooming out since surface tiles aren’t immediately updated. The tiles can likely be filtered to remove unneeded ones more aggressively.

Running On Your System

We have released this code on our GitHub under the Apache license for everyone to use.

To get this up and running on your local system, start with a Git clone:

git clone https://github.com/EndPointCorp/tiled-city-labels.git

Then within the repo there’s a README to follow. We will give instructions here, but if they differ from the README follow those instead:

cd tiled-city-labels
docker build -t cesium-labels .
docker run -p 48088:48088 --rm cesium-labels

Then in a new terminal window still inside the project:

cd demo/
python -m SimpleHTTPServer 8000

Now you can navigate to http://localhost:8000/ in your browser and see the city labels up and running. To customize this to your own system, you can use the mixin demo/js/CitiesDataSource.js and add that to your own project. You can also change the port/server used in the mixin by editing the fetch command inside the queryData function.

Formatting SQL code with pgFormatter within Vim

2022-04-26T00:00:00+00:00

Photo by Garrett Skinner

Sometimes a little, seemingly simple tip can make a world of difference. I’ve got enough gray hair these days that it would be pretty easy for me to start thinking I’d seen an awful lot, yet quite frequently when I watch a colleague working in a meeting or a tmux session or somewhere, I learn some new and simple thing that makes my life demonstrably easier.

Luca Ferrari recently authored a post about using pgFormatter in Emacs; essentially the same thing works in Vim, my editor of choice, and it’s one of my favorite quick tips when working with complicated queries. I don’t especially want to get involved an editor war, and offer the following only in the spirit of friendly cooperation for the Vim users out there.

As Luca mentioned, pgFormatter is a convenient way to make SQL queries readable, automatically. It’s easy enough to feed it some SQL, and get a nice-looking result as output:

$ pg_format < create_outbreaks.sql
INSERT INTO outbreak                      
SELECT                              
    nextval('outbreak_id'::regclass),
    extract('year' FROM now())::text || '-' || nextval('outbreak_number_seq')::text, --number
    (                        
        SELECT  
            first_name
        FROM person TABLESAMPLE BERNOULLI (10)
LIMIT 1), -- name
    NOW() - interval '1 day' * random() * 100, (
        SELECT
            id
        FROM "user" TABLESAMPLE BERNOULLI (10)
...

In my perfect world I might quibble with some of its formatting decisions, such as the lack of indent on the LIMIT 1 line above. But in practice the results are good enough for my tastes that I haven’t bothered to investigate whether I can improve them. I just use it, and it’s good enough for me.

And because Vim lets me highlight a region, pipe it through an external program, and replace the region with that program’s output, it’s easy to use it simply by selecting a section of code and typing :!pg_format like this:

On Shapefiles and PostGIS

2022-04-02T00:00:00+00:00

Partial map of the voyage of the Endurance, from “South”, by Ernest Shackleton

The shapefile format is commonly used in geospatial vector data interchange, but as it’s managed by a commercial entity, Esri, and as GIS is a fairly specialized field, and perhaps because the format specification is only “mostly open”, these files can sometimes be confusing to the newcomer. Perhaps these notes can help clarify things.

Though the name “shapefile” would suggest a single file in filesystem parlance, a shapefile requires at least three different files, including filename extensions .shp, .shx, and .dbf, stored in the same directory, and the term “shapefile” often refers to that directory, or to an archive such as a zipfile or tarball containing that directory.

QGIS

QGIS is an open-source package to create, view, and process GIS data. One good first step with any shapefile, or indeed any GIS data, is often to take a look at it. Simply tell QGIS to open the shapefile directory. It may help to add other layers, such as one of the world map layers QGIS provides by default, to see the shapefile data in context.

GDAL

Though QGIS can convert between GIS formats itself, I prefer working in a command-line environment. The GDAL software suite aims to translate GIS data between many available formats, including shapefiles. I most commonly use its ogr2ogr command-line utility, along with the excellent accompanying manpage.

In short, a typical ogr2ogr command tells the utility where to find the input data and where to put the converted output, optionally with various reformatting and processing options. You’ll find some examples below.

PostGIS

Much of our (ok, my) GIS work has involved PostGIS, an extension to the PostgreSQL database for handling GIS data. It’s been convenient for me to process GIS data using the same language and tools I use to process other data. It uses GDAL’s libraries internally.

Examples

Import Shapefile data into PostGIS

The example below comes from a customer’s project we recently worked on. They provided us a set of several shapefiles, which I first arranged in a directory structure. This code imports each of them into a PostGIS database, in the shapefiles schema.

The other arguments to ogr2ogr specify the output format (“PostgreSQL”), the destination database name, and the directory which stores the shapefile. ogr2ogr expects the destination and source arguments in that order, as two positional arguments, so here the destination is PG:dbname=destdb, and the source file name comes from the the $i script variable.

for i in `find . -name "*shp"`; do
    j=$(basename $i)
    k=${j/.shp/}
    ogr2ogr -f PostgreSQL -nln shapefiles.${k} PG:dbname=destdb $i
done

Export PostGIS data as KML

This example creates a KML file from PostGIS query results. The arguments provide the query to use to fetch the data, the output format (“KML”), the output file name, and the source database. This will create a KML file containing a set of unstyled placemarks, with names from the property_code column, and geometry data from the outline_geom column in the properties table of our database.

In this project, outline_geom contained GIS “linestrings”, data types consisting of a series of lines, which ogr2ogr translated into KML polygons. Had outline_geom contained points, for instance, the KML result would also have been points. In other words, ogr2ogr automatically chooses the correct KML object type based on the GIS object type in the input data.

ogr2ogr -sql "select property_code, outline_geom from properties" -f KML outlines.kml PG:dbname=properties

Note that though the examples above use PostGIS, ogr2ogr can take shapefile input and produce KML output directly without the PostGIS intermediary. We used PostGIS in these cases for other purposes, such as to filter the output and limit the attributes stored in the KML result.

By default, ogr2ogr puts all the attributes from the shapefile into ExtendedData elements in the KML, but in our case we didn’t want those. We also didn’t want all the entries in the shapefile in our resulting KML. To skip the PostGIS step, we might do something like this:

ogr2ogr -f kml output.kml shapefile_directory/

What tools do you use for shapefile processing? Please let us know!

CZML-KML Editor Geometry Editing

2021-04-13T00:00:00+00:00

We are happy to introduce a new feature for the Cesium CZML-KML Editor: polygons and polylines geometry editing. You can now edit geometries for existing entities and move entered points during the creation process. Here is a video with a short summary of the editing process:

See our previous blog post introducing the Cesium CZML-KML Editor for further reference.

Announcing the Cesium KML-CZML Editor

2020-12-21T00:00:00+00:00

End Point’s immersive technology team is happy to present a great new tool for the rapidly growing Cesium community: Cesium KML-CZML Editor. The editor gives users the ability to visually and dynamically edit KML and CZML in its Cesium browser window. Updates made with it can be exported at any time to CZML, the native markup language for Cesium.

The Cesium KML-CZML Editor addresses an important but hitherto unaddressed need of the Cesium community: It provides an intuitive interface for making adjustments to fix the many inconsistencies with how KML created for (and often by) Google Earth appears on 3D maps rendered with Cesium. It is a powerful tool for converting and adapting KML for Google Earth into CZML that displays nicely in Cesium. The editor also works as a visual editor for creating and editing CZML, regardless of whether you’re converting from KML.

The inconsistencies with how Cesium displays KML created for Google Earth are due to occasional differences between how Cesium and Google Earth render KML when various attributes aren’t specifically set within a given instance of code. The situation is similar to how web browsers sometimes interpret given instances of HTML differently. Just as with HTML, KML doesn’t require every attribute to be defined in a given instance of markup code.

On the left side we have the editor toolbar, and on the right side the Cesium globe showing loaded or created entities the way they would appear in Cesium.

The Editor Toolbar consist of the following areas:

File Import: you can load KML, KMZ, CZML and GeoJSON formatted data, and you can load multiple files to combine them into one CZML document.
Creation tools.
A list of uploaded or created entities (will appear after upload or creation of a new entity).
The actual editor for entity properties (will appear after entity selection).

Let’s start with a very basic example of highlighting a building by marking it with a pin in Google Earth:

Now let’s move the camera about in different ways, especially noticing how the pin appears when zooming in and out. The pin stays close to the top of the building.

Sorry, your browser doesn’t support embedded videos.

Then let’s open the same KML in Cesium with a 3D building tileset loaded, to compare apples to apples:

Sorry, your browser doesn’t support embedded videos.

It’s easy to see that the green frame which is placed around the geographic position stays around the building, but the icon point of rotation in Cesium doesn’t match the one in Google Earth and the pin falls through the roof and lifts up unfortunately high above the building.

KML doesn’t enforce certain aspects of its representation. If certain attributes are not specified or can’t be implemented exactly, then the KML browsers have the freedom to set different default values for them.

In this example, Cesium and Google Earth display different pixel offsets for the pin icon, because in KML the offset is specified as a ratio of the image, but Cesium specifies the offset in pixels. Cesium converts the pixel offset of the pin from the ratio specified, but with some errors. Cesium reverses the X axis, and in some cases Cesium also calculates the pin icon image size incorrectly. So here it calculates the X offset as -4, which moves the pin icon to the left. To fix that, we can reverse the X axis back to 4 or even move the pin icon 8 pixels further to the right.

🡆

The next issue is how the Z coordinate (altitude) is interpreted. Is it above sea level, above land according to some averaged model of the land, or on the land directly below a point?

Note that 3D tilesets in Cesium are absolutely positioned. This means that if you show 3D buildings and want your pin to stick to a building, it’s better to use absolute elevation. Hence, it makes sense to set Height Reference to NONE, which will force Cesium to use the absolute pin position.

Another option in cases like this is to use relative elevation. This can be helpful if you don’t use 3D tiles, but instead use 3D terrain data, for example, with Cesium World Terrain or Maptiler. It’s especially helpful if you will be switching 3D terrain on and off. In this case, a relatively positioned pin will follow the elevation regardless of whether 3D terrain is turned on or off.

To set it up, adjust Altitude and set Height Reference to RELATIVE_TO_GROUND:

Here are the results for both options:

Sorry, your browser doesn’t support embedded videos.

As you can see, now we have the pin-point of the marker stuck to the desired location.

These are the basics for how to match pins created in Google Earth with their Cesium representations, but in Cesium there are many more options for rendering pins. You can read the relevant Cesium documentation (in Cesium terminology Pins are called Billboards) or interactively explore a live version of this editor to change color, rotation, translucency by distance, and other properties.

Now let’s look at a polygon created in KML for Google Earth:

Then let’s open it in Cesium:

Well, we can select a polygon, but we don’t see it. This happens because the 3D terrain data covers the polygon. Just changing the Height reference helps, but it’s still not what we want:

Cesium has some default assumptions about polygons and rectangles and when to clamp them to the ground.

As commented in the Cesium SandCastle Clamp to Terrain demo code:

Corridors, polygons and rectangles will be clamped automatically if they are filled with a constant color and have no height or extruded height.

NOTE: Setting height to 0 will disable clamping.

So to get a polygon actually clamped to ground, we have to delete the Height property, not just set it to 0:

Now let’s add a 3D tileset:

And again, this is probably not exactly what you wanted. By default, Cesium doesn’t know what type of 3D tileset you would prefer, or whether you want the polygons to cover the tiles, or the tiles to cover the polygons. If you have a fairly large polygon covering a whole city, and a fairly low resolution 3D tileset, you probably want to highlight the whole area. But in the example above, we want our polygon to stick to the ground but not the buildings.

Next, let’s use the Classification Type field, setting it to “TERRAIN” to select what the clamped polygon applies to:

Now, let’s open some exported data. Here’s a typical set of exported points generated with geojson.io:

Let’s choose a point and change its icon and its icon’s vertical alignment (so it doesn’t fall through the terrain) and dimensions:

Here we have selected one icon and updated its properties to make it appear as we want, but changing all the icons one by one would take forever, especially for large data sets.

Click “COPY TO OTHER PINS”:

On the left is a list with all the entities to which the changed properties are applicable and on the right is the list of the properties that were changed for the model entity selected. By default, all the properties are checked to be copied to applicable entities but in this example we only want to change the vertical alignment for all the pins, but want to keep the icon and icon dimension of the other pins unchanged. So we uncheck the image, width and height properties:

Below we see how our scene looks after the update is applied. All the pins are above the ground, but the icons and dimensions are not changed:

After you have all the data styled the way you want, download your CZML!

The Cesium KML-CZML Editor code is Apache 2 licensed. It is available here.

EMSA: Electronic Messaging Staging Area

2020-10-30T00:00:00+00:00

Photo by Lee Roberts, CC BY-SA 2.0

End Point has been involved with developing disease surveillance software for over a decade. One component of that work is EMSA, short for “Electronic Messaging Staging Area”.

The EMSA system we’re currently using was developed by a consortium of states led by Utah, Kansas, and Nevada. It is a PHP web application and can be hosted in the cloud or on-premises as you would any other PHP application.

EMSA:

Is an ingest system for HL7 or CSV-formatted lab reports from doctor offices, labs, and other official reporting groups.
Allows exporting XML output to your disease surveillance system.
Is a web console for managing messages.
Allows you to see messages that are malformed.
Has configurable rules for disease types and message routing; some messages may go directly into a surveillance system while others may be stored for later.
Has configurable “automated workflows” to validate message information and perform tasks.
Can be connected to a BI (Business Intelligence) tool for reporting, or the database can be queried directly.
Works with integration packages like Rhapsody to bring in data from other systems.

End Point has also made custom modifications to EMSA allowing address validation for incoming messages.

EMSA can be deployed along with EpiTrax, the open source tracking system, or as a standalone application connected to a different disease surveillance system.

Benefits of EMSA

Aside from all the features mentioned above, EMSA as an open-source solution brings other benefits:

Reduced cost: Other than development and consulting to help set it up, there are no ongoing licensing costs.
Designed to work as part of the open-source EpiTrax disease surveillance system.
Participating in the open source community that is growing in the healthcare industry.

Help is available!

Interested in talking to us more about EMSA for your disease tracking? Reach out today.

References

Emily Roberts, Rachelle Boulton, Josh Ridderhoff, Theron Jeppson. “Automated Processing of Electronic Data for Disease Surveillance.” Online Journal of Public Health Informatics, 10(1):e2, May 22, 2018. https://doi.org/10.5210/ojphi.v10i1.8903

Fix for cdparanoia segmentation fault

2019-08-27T00:00:00+00:00

Photo by Alberto Cabrera, cropped, CC BY 2.0

It had been a while since I last needed to rip a CD into audio files (and compress them), but the need recently arose. This particular disc was a language learning supplement to a book, and a CD was a reasonable way to distribute that.

(Even though audio file downloads and streaming have become a far more common way to distribute audio than physical CDs, electronic formats don’t preserve our same rights to resell, lend, and make backups. But that is a topic for another blog post!)

I was ripping the CD with abcde (A Better CD Encoder), a text-based CD ripping, tagging, and compressing front-end I have used often in the past. Unexpectedly I got an error, as shown in this terminal output:

Executing customizable pre-read function... done.
Getting CD track info... Querying the CD for audio tracks...
Grabbing entire CD - tracks:  01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
abcde: attempting to resume from /home/user/Music/abcde.2a0e162a..
.
Grabbing track 28: Track 28...
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector  131363 (track 28 [0:00.00])
          to sector  148524 (track 28 [3:48.61])

outputting to /home/user/Music/abcde.2a0e162a/track28.wav

 (== PROGRESS == [ ! !!--!V V   ! ! !   >       | 143137 00 ] == :-) 0 ==)   /bin/abcde: line 3560: 21658 Segmentation fault      (core dumped) nice $READNICE $CDROMREADER -"$CDPARANOIACDROMBUS" "$CDROM" "${READTRACKNUMS:-$UTRACKNUM}" "$FILEARG" 1>&2
[ERROR] abcde: The following commands failed to run:
readtrack-28: cdparanoia  returned code 139
Finished. Not cleaning /home/user/Music/abcde.2a0e162a.

The command that abcde was running when it died is:

cdparanoia -d /dev/cdrom 28 /home/user/Music/abcde.2a0e162a/track28.wav

Getting a segmentation fault suggests that the software has a bug, not necessarily that there is anything wrong with the disc, drive, or USB interface or drivers. But when troubleshooting, it is good (and easy) to start by testing on different hardware and operating system, if possible, to narrow things down.

I was using Fedora 29 with an LG Blu-Ray USB drive. To rule out several of those possible factors, I tried ripping the disc again on another computer running Ubuntu 19.04 with a Samsung DVD USB drive. It crashed the same way on the same track.

While looking online for ideas, I found several people mention bugfixes to cdparanoia in the wild from the open source community, which have not resulted in a new release.

In fact, to my surprise, the popular cdparanoia can safely be classified as abandoned, since the last software release was almost 11 years ago, on 2008-09-11. More tellingly, the latest commit to the Subversion repository was on 2010-06-11.

In the meantime, another project called libcdio contains a port of cdparanoia that was originally forked to make it work on platforms other than Linux. More importantly for my purpose here, it is actively maintained and has had many bugfixes that can be seen in the libcdio-paranoia Git repository.

Its executable is called cd-paranoia to distinguish it from the original cdparanoia, and abcde has an option (here specified in an environment variable) to use libcdio instead of the default cdparanoia. I invoked abcde this way:

CDROMREADERSYNTAX=libcdio abcde -d /dev/cdrom -o mp3,opus,flac -V -x -j 3

and then the processing worked fine.

An aside: You may notice my command above compresses into the opus format. If you are not yet familiar with this Opus audio codec, I recommend you take a look! Quoting their website, it is an:

… open, royalty-free, highly versatile audio codec. Opus is unmatched for interactive speech and music transmission over the Internet, but is also intended for storage and streaming applications. It is standardized by the Internet Engineering Task Force (IETF) as RFC 6716 which incorporated technology from Skype’s SILK codec and Xiph.Org’s CELT codec.

It is supported by many more players than I suspected given its youth, and I am very impressed with its quality vs. data size. If you are writing your own software that includes audio, Opus is a great choice, since you can include a playback library that supports it if needed.

Anyway, back to cdparanoia: Sometimes it takes a bit of detective work to find the better-maintained free software, but often it is out there. Thanks to those who have kept working on cd-paranoia!

LinuxFest Northwest 2019

2019-05-03T00:00:00+00:00

I’m sitting in an airport, writing this in an attempt to stay awake. My flight is scheduled to depart at 11:59 PM, or 2:59 AM in the destination time zone which I’m still used to. This is the first red eye flight I’ve attempted, and I’m wondering why I’ve done this to myself.

I have dedicated a good portion of my life to free, open source software. I’ll occasionally travel to conferences, sitting on long flights and spending those valuable weekends in talks about email encryption and chat bots. I’ve also done this to myself. But even with all this I have zero regrets.

This little retrospective comes courtesy of my experience at LinuxFest Northwest this last weekend in Bellingham, Washington.

Specifically I think it was some of the talks, painting things in broad strokes, that did it. I attended Jon “maddog” Hall’s beard-growing Fifty Years of Unix, and later sat in on the Q&A, which was a bit less technical than expected. So I didn’t ask about the “2038 problem.” But that’s okay.

I felt a little guilty, on one hand, doing these general interest sessions instead of something on a much more specific topic, like ZFS, which would have arguably had a more direct benefit. On the other hand, doing those general interest talks helps me stay grounded, I suppose, helps me keep perspective.

I did attend some more specialized talks, naturally. LFNW was a packed conference, often times there were a number of discussions I would have liked to attend happening at the same time. I’m hoping recordings will become available, or at least slides or other notes will appear. Some of the other talks I attended included, in no particular order:

Audio Production on Linux
Like many other End Pointers, I dabble in a little bit of music. Unlike those other End Pointers, I’ve got no talent for it. Still, I try, and so I listened in on this one to find out a little more about how Jack works. I also caught wind of PipeWire, a project that’s aiming to supplant both PulseAudio and Jack. Neat!
Using GIS in Postgres
I’ve got a couple PostGIS-related projects coming up. So while this talk covered a few of the basics and some query types and didn’t touch much on indexing and such, it was still good to see. Plus it got me in place for mine.
Chat Ops
Internally we have a few scripts that connect in to our chat system and write status messages. But it’s all one-way communication, like alerts from our monitoring system. This talk proposed writing bots that can watch for and process messages written by others, and take actions. We could, for example, acknowledge alerts or set systems into downtime without having to switch out of the chat system. I want to look into options like that for sure.
We can fix email server encryption!
End Point (thankfully) is on the verge of turning down its internal email systems. But the current trouble with server-to-server email encryption was still valuable to learn about!

Yeah, I did one, too: Get It Back, A New PostgreSQL Admin’s Guide to Redundancy and Recovery. I talked about the different methods of performing backups of a Postgres database, and the advantages and drawbacks of each. Plus a little about replication. The slides are up here, but admittedly only about half the material is there, the rest would have been me talking at the slides. I’ll write it up in a subsequent post, promise. Afterward I stuck around to sit in on the Consult the Experts panel.

Plus my talk was attended by two other End Pointers, Greg Davidson and Greg Hanson. No pressure there.

But seriously, that was an added bonus. Being a distributed company much of the time our coworkers are right at our fingertips in one sense, but in another also out of reach. Getting that face time was very nice. LFNW had a couple general social activities, too, on Friday and Saturday. And of course I attended both, talking about Nagios and Haskell and Docker and anything else that came to mind. Florida to Washington was a long hike, but it was well worth it.

All that said, I do have one lingering regret: By the time I figured out that there were tie-dye shirts, a trademark of this conference, and where they were, they were all gone. Oh well. Next year.

Switching from Google Maps to Leaflet

2019-03-23T00:00:00+00:00

Photo: RadSat HD

It’s no news for anyone who has Google Maps running on their websites that Google started charging for using their API. We saw it coming when, back in 2016, they started requiring a key to add a map using their JavaScript API. And on June 11, 2018, they did a major upgrade to their API and billing system.

The consequence? Any website with more than 25,000 page loads per day will have to pay. And if you are using a dynamic map (a map with custom styling and/or content) you only have roughly 28,000 free monthly page loads. We must create a billing account, even if we have a small website with a couple of daily visitors, hand credit card information to Google, and monitor our stats to make sure we won’t be charged. And if we don’t do that, our map will be dark and will have a “For development only” message in the background.

So what are your options? You can either pay or completely remove Google Maps from your websites. Even enterprise weather websites like The Weather Channel or Weather Underground have now replaced their Google Maps API calls with an alternative like Leaflet or MapBox (in some cases, they even gained some functionality in the process).

I have a personal weather website, and when I heard big changes were coming, I started to move away from Google Maps as well. My choice at that moment was Leaflet: It has everything you may need to build a robust tile-based map, add layers, markers, animations, custom tiles, etc. And it’s BSD-licensed open source and free.

Creating a basic map

Google Map conversion to Leaflet can be almost seamless if the same tiles are used.

Google Maps API and Leaflet share a similar way of doing most things, but they have some key differences we need to take into account. As a general rule, Google used the “google.maps” prefix to name most classes and interfaces, while Leaflet uses the “L” prefix instead.

First thing we need to do is to remove the Google Maps API reference from our website(s). So we need to replace the reference:

<script type="text/javascript" src="https://maps.googleapis.com/maps/api/js?key=[your_api_key]"></script>

With the references to the Leaflet map JavaScript and stylesheet URIs.

<script src="https://unpkg.com/leaflet@1.0.2/dist/leaflet.js"></script>
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.0.2/dist/leaflet.css" />

Now let’s take a look at the code needed to create a Google Map vs. a Leaflet map.

Google:

var map = new google.maps.Map(document.getElementById("map"), {
	center: new google.maps.LatLng(40.7401, -73.9891),
	zoom: 12,
	mapTypeId: google.maps.MapTypeId.ROADMAP
});

Leaflet:

var map = new L.Map("map", {
	center: new L.LatLng(40.7401, -73.9891),
	zoom: 12,
	layers: new L.TileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png")
});

Quite similar, isn’t it? The main difference is that, in Leaflet, we need to provide a tile layer for the base map because there isn’t one by default. There are a lot of excellent tile layers available to use at no cost. Here are some of them:

Bright: https://a.tiles.mapbox.com/v3/mapbox.world-bright/{z}/{x}/{y}.png
Topographic: https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png
Black and white: https://stamen-tiles-{s}.a.ssl.fastly.net/toner/{z}/{x}/{y}{r}.png

You can browse other free tile layer providers for Leaflet on this link. And of course, if you want to pay there’s a lot of affordable paid tiles out there too.

Adding a marker

Adding a marker is quite straightforward as well. It even looks easier on Leaflet than Google.

Google:

var marker = new google.maps.Marker({
	position: new google.maps.LatLng(40.7401, -73.9891),
	map: map,
	title: "End Point Corporation"
});

Leaflet:

var marker = new L.Marker(new L.LatLng(40.7401, -73.9891));
marker.bindPopup("End Point Corporation");
map.addLayer(marker);

And that’s it: we have a working Leaflet map with a marker that displays a text when we click on it.

Screenshot of the Leaflet example. Code below, if you want to try it live:

<head>
  <title>Leaflet map example — End Point Corporation</title>
  <script src="https://unpkg.com/leaflet@1.0.2/dist/leaflet.js"></script>
  <link rel="stylesheet" href="https://unpkg.com/leaflet@1.0.2/dist/leaflet.css" />
</head>
<body>
  <style>
    body { margin: 0 };
    #map { height: 100% };
  </style>
  <div id="map"></div>
  <script type="text/javascript">
    var endPointLocation = new L.LatLng(40.7401, -73.9891);
    var map = new L.Map("map", {
      center: endPointLocation,
      zoom: 12,
      layers: new L.TileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png")
    });
    var marker = new L.Marker(endPointLocation);
    marker.bindPopup("End Point Corporation");
    map.addLayer(marker);
  </script>
</body>

Layers and controls

From this point, we can start doing more complex things if we need to:

Display images on the map: ImageOverlay.
Display a custom tile layer: TileLayer.
Draw polygons, rectangles, circles: Polygon - Rectangle - Circle.
Display GeoJSON data on the map: GeoJSON.

You can browse the Leaflet API reference for further details.

Plugins and tools

There is some extended functionality in Google Maps that is not available in Leaflet by default unless we use a plugin. For example, if we want to add the “fullscreen” button to the top right corner, just as Google has it, or if we want to let the user draw polygons on top of the map, we’ll need to download and add the reference to the required plugins. Here is a list of the ones I’ve already used:

“Fullscreen” button plugin: Leaflet.fullscreen.
Vector drawing and editing plugin: Leaflet.draw.
Heatmap plugin: Leaflet.heat.

You can find more plugins at the Leaflet GitHub account. And of course, you can (and should!) contribute to improve them.

There is also some alternatives to additional services offered by Google like geocoding or routing. They might have some limitations involved, so it would be wise to take a look at their usage policies first.

Geocoding API: Nominatim.
Routing: Project ORSM (free version has limited use).

More services can be found at switch2osm.org/other-uses.

Putting it all together

I’ve been using Leaflet for almost a year now in an interactive weather map originally made with the Google Maps API. Of course, I’ve had some minor hiccups along the way, but having full control of the source code and resources allows you to add functionality, fix things or even rewrite whatever you need.

The Leaflet source code is well organized, modularized and easy to understand. I’ve created custom grid layers using different tile sources, with different coordinate systems, animations with frame transitions, custom controls, clickable polygons, popups with dynamic content from AJAX calls, and more. And all works smoothly. So I recommend that you go ahead and start using Leaflet right away.

Example of a fully-functional Leaflet map with custom controls, overlays, animations and polygons.

And this is the repository with my weather map source code: RadSat HD. Feel free to leave any comments or suggestions!

Start basic application with Vue.js 2 and Drupal 8

2018-04-05T00:00:00+00:00

Introduction

The purpose of creating this post is to show how fast can you build web applications with Vue.js on the front-end and Drupal on the back-end side.

Let’s call our project “Awesome Nerds”.

What do we need?

Debian/Ubuntu system
Internet
30 minutes
Vagrant & VirtualBox
Git
Vim
Yarn

Step by step

Here’s what we’re going to do:

Install Vagrant, VirtualBox, and Git
Setup a new Drupal 8 project that will be our back-end project
Setup a new Vue.js project that will be our front-end application
Let’s code

Install Vagrant, VirtualBox, and Git

Open your console and run:

$ sudo apt-get install software-properties-common - getting some common libraries

$ sudo apt-add-repository ppa:ansible/ansible

$ sudo apt-get update

$ sudo apt-get install ansible - installing Ansible

$ wget ‘https://releases.hashicorp.com/vagrant/2.0.2/vagrant_2.0.2_x86_64.deb’ && dpkg -i vagrant_2.0.2_x86_64.deb - installing Vagrant

$ sudo apt-get install dkms

$ deb https://download.virtualbox.org/virtualbox/debian <mydist> contrib

$ wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -

$ wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -

$ sudo apt-get update

$ sudo apt-get install virtualbox-5.2 git vim nfs-kernel-server - installing VirtualBox

$ vagrant plugin install vagrant-vbguest

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

$ sudo apt-get update && sudo apt-get install yarn - installing Yarn

Now we can continue to the back-end project installation.

Install and setup back-end project

Create a new folder for the project:

$ mkdir awesome_nerds && cd awesome_nerds

$ mkdir frontend backend

We need to clone the DrupalVM repository. DrupalVM is a Drupal setup that helps to encapsulate services with Vagrant. Run:

$ cd backend

$ git clone git@github.com:geerlingguy/drupal-vm.git .

Let’s name our project:

$ vim default.config.yml

Look and set these two settings so they look like this:

vagrant_hostname: awesomenerds.backend                                   
vagrant_machine_name: awesomenerds_backend

Quit Vim and run:

$ vagrant up

It will take a while to set up everything, you can get a coffee or browse some memes or go to the next chapter and start creating our front-end project.

When it’s ready we will have a running Drupal 8 setup with MySQL, PHP 7, and Apache (you can configure this stack in default.config.yml if you prefer nginx for example).

Drupal project files are in the drupal directory and that’s the only folder that you would want to add to a project Git repository.

Setup new Vue.js project

We will use a minimal project skeleton from https://github.com/vuejs-templates/webpack.

Run:

$ cd awesome_nerds/frontend

$ git clone https://github.com/vuejs-templates/webpack .

$ yarn install -g vue-cli

$ vue init webpack awesome_nerds

Name the project “awesome_nerds” (yes!) and just hit enter to install with defaults.

When you run:

$ yarn run dev

you will get a fresh Vue.js application running on http://localhost:8080.

Let’s code!

Now we are ready for development. It can be really rapid, both Vue.js 2 and Drupal 8 are impressively good and it’s just a matter of finding a good idea for your new start-up.

In my next post I will continue and code a simple social application using the REST API of Drupal and our Vue.js front-end.

Thank you and good luck!

Vue in Ecommerce: Routing and Persistence

2018-03-01T00:00:00+00:00

I recently wrote about Vue in Ecommerce and pointed to a handful of references to get started. Today, I’ll talk about using vue-router in a small ecommerce application, combined with vuex-persist for state storage.

I forked this Vue Shop on GitHub from Matheus Azzi. It was a great starting point for to see how basic component organization and state management might look in a Vue ecommerce application, but it is a single page ecommerce app with no separate page for a product detail, checkout, or static pages, so here I go into some details on routing and persistence in a Vue ecommerce application.

Vue Router

In looking through the documentation, I don’t see a great elevator pitch on what it is that Vue Router does. If you are new to routing, it’s a tool to map the URL request to the Vue component. Since I’m coming from the Rails perspective, I’m quite familiar with the Ruby on Rails routing from URL pattern matching, constraints, resources to Ruby on Rails controllers and actions. Vue routing via vue-router has some similar elements.

When you create a basic Vue application via vue-cli, you are given the option to include vue-router:

vue init webpack myapp

? Project name myapp
? Project description A Vue.js project
? Author Steph <steph@endpoint.com>
? Vue build standalone
? Install vue-router? (Y/n)

If you select Yes here, the main differences you’ll see are that an application with vue-router installed will call <router-view/> to render the view for the current router, instead of a <HelloWorld/> component, and that a vue-router app will include src/router/index.js with basic routing configuration to your <HelloWorld/> component.

Without Routing

// App.vue without vue-router
<template>
  <div id="app">
    <img src="./assets/logo.png">
    <HelloWorld/>
  </div>
</template>

<script>
import HelloWorld from './components/HelloWorld'

export default {
  name: 'App',
  components: {
    HelloWorld
  }
}
</script>

With Routing

// App.vue with vue-router
<template>
  <div id="app">
    <img src="./assets/logo.png">
    <router-view/>
  </div>
</template>

<script>
export default {
  name: 'App'
}
</script>


// src/router/index.js
import Vue from 'vue'
import Router from 'vue-router'
import HelloWorld from '@/components/HelloWorld'

Vue.use(Router)

export default new Router({
  routes: [
    {
      path: '/',
      name: 'HelloWorld',
      component: HelloWorld
    }
  ]
})

Moving on, building out an ecommerce app with some basic page set up, not including user accounts, might look like the setup below, which includes basic components and routes for the Home, Cart, Checkout, Receipt, and About page. This simplified set-up doesn’t even include a product detail page, but one could accomplish that with a bit of URL matching via parameters.

// src/router/index.js
import Vue from 'vue'
import Router from 'vue-router'

import Home from '@/components/pages/Home'
import Receipt from '@/components/pages/Receipt'
import About from '@/components/pages/About'
import Cart from '@/components/pages/Cart'
import Checkout from '@/components/pages/Checkout'

Vue.use(Router)

export default new Router({
  mode: 'history',
  routes: [
    { path: '/', component: Home },
    { path: '/receipt', component: Receipt },
    { path: '/about', component: About },
    { path: '/cart', component: Cart },
    { path: '/checkout', component: Checkout }
  ]
})

History Mode

One thing you also might notice is the ‘history’ mode setting above, overriding the default hash mode. You can read about that here, but it leverages history.pushState to URL navigation. The history mode override must be combined with server configuration to ensure that non-static asset URL requests (i.e. all of our routes) hit the Vue index.html app in our Vue application, and then renders the component associated with the requested route. I’m using Apache for my Vue application, so I followed the instructions on that documentation for vue-router history mode configuration.

Using <router-link/>

One thing you also will want to use throughout your templates is <router-link/>, instead of hard-coding rigid URL paths. For example, instead of linking to the cart via <a href="/cart">Cart</a>, you’ll want to link via <router-link to="cart">Cart</router-link>. This will be resolve to the proper link upon rendering depending on your routing configuration.

Vuex PersistedState

If you are still reading, you’ve now seen the basic configuration for vue-router in an ecommerce application running on Vue. The ecommerce application I forked uses vuex, a state management pattern and library to store the state of our application. What you’ll notice with a shopping cart application, though, is that you need to persist the state of your cart for sessions. Without further setup, every page refresh on your application will result in losing the state of the cart.

To address this behavior, there are a few options to persist your state in Vue. I chose vuex-persist, which stores the state (for all modules or specific modules) in localStorage. After installing vuex-persist, I modified the following to include the shoppingCart module in my stored state:

// src/store/index.js
import Vue from 'vue'
import Vuex from 'vuex'
import VuexPersistence from 'vuex-persist'
// import other stuff

Vue.use(Vuex)
const vuexLocal = new VuexPersistence({
  storage: window.localStorage,
  reducer: state => ({
    shoppingCart: state.shoppingCart
  })
})

export default new Vuex.Store({
  // ...
  plugins: [vuexLocal.plugin]
})

It’s as simple as that to persist the state of your shopping cart using all the vuex-persist defaults, and there are some alternative Promise-based stores supported if you are interested.

What’s Next?

What’s next in building out your ecommerce application? From here, one might consider adding user authentication & authorization support, as well as support for retrieving and rendering products in the product list view. I hope to address those in upcoming blog posts in this series on Vue in ecommerce.

Vue and Ecommerce: An Introduction

2018-02-19T00:00:00+00:00

I speak the domain specific language of ecommerce, Ruby on Rails, JavaScript, and jQuery. Lately, I’ve been getting up to speed on Vue.js. I’ve been working on writing a small ecommerce site using Vue.js, because for me, creating an application addressing familiar paradigm in a new technology is a great way to learn.

Vue.js, initially released about 3 years ago, is a lightweight JS framework that can be adopted incrementally. In the case of my small example site, Vue.js serves the frontend shop content and connects to a decoupled backend that can be run on any platform of choice. On my path to get up to speed on Vue, I found great resources that I wanted to write about before I get into the details of my ecommerce app. Here they are:

First, I started with the Vue.js documentation. The documentation is great as a starting point to understand some of the terminology. I often don’t love the documentation that comes with technologies, but I found the Essentials section in the Vue documentation to be a great launching point.
After I worked through some of the Vue.js documentation, I did a simple search for “vue jsfiddle” and experimented with a few of those fiddles. Having come from a jQuery background (with mustache), I was familiar with how templating and logic inside a template might work, but less familiar with the vue instance and binding, so I targeted jsfiddles with this in mind.
Next, I looked for resources with Vue + ecommerce specifically. Here’s where I found some good stuff:
- Scotch.io Course on Vue + Ecommerce: This online course has a great amount of overlap with the main documentation of Vue, but some of the examples are ecommerce specific, which serves as a good introduction.
- Vue + Third Party Integration for Ecommerce: This Medium article talks about integrating third party tools used with Vue (Stripe, AWS S3, Heroku) to build an ecommerce site. While this post doesn’t go into the nitty gritty details of a Vue application, it does provide many code examples for checkout and payment integration, although noting that vue-stripe-elements is now the module of choice for Stripe integration in Vue.
- Vue API App + Magento Backend: This is a front-end open source Vue application designed to pair with a Magento backend, capitalizing on the popularity of Magento.
- Moltin: This company is providing backend microservices for ecommerce, with Vue.js support (and other technologies) to build out your store.
- Vue + Ecommerce with Vuex: Here’s another Medium article which features vuex, the state management library in Vue, essentially going through examples of state management to manage a shopping cart.
- CodePen Vue + Ecommerce Example: Here’s a standalone CodePen example of a Vue shopping cart. While I find this a little hard to parse because there’s a significant amount of code in the example, you can download and experiment with it.
- Stripe Integration Example: Here’s a GitHub repo that demonstrates Stripe integration in serverless Vue shop.
- Basic Vue Shop on GitHub: Why reinvent the wheel when someone has already built out product listing and cart functionality? After reviewing all the examples above, this was my choice for a starting point in building out a Vue shop, but it is a basic starter point, without the use of vue-router.

Vue Libraries / Dependencies

You might have noticed I mentioned a few Vue libraries and tools. Here are additional resources for getting up to speed on those dependencies:

Installing Node.js and npm, if you aren’t already familiar with it. Here is a great comparison if you are coming from the Rails space.
Already mentioned, vuex state management in Vue.
vue-router. In an ecommerce site, you would possibly have routing for the index, product detail, cart, checkout, user account, order detail pages.
vue-cli: simple command line interface for scaffolding Vue projects. Think “rails generate …” for Vue.
vue-resource, or a comparable module (vue-axios) for making API and AJAX requests.

Incremental Adoption

While I am writing a Vue ecommerce shop based on the shop-vue GitHub repo mentioned above, in the ecommerce space, we commonly take on incremental ecommerce updates rather than “The Big Rewrite”, as noted in Greg’s article. Components that could be introduced incrementally into existing ecommerce applications might include:

Dynamic marketing elements on homepage or product listing page, e.g. promo tools, “Customers Also Looked at”, “Products Recently Viewed”
Cart + Checkout process (standalone component)
Admin elements for data management and processing
User account section

Sunsetting Piggybak, A Ruby on Rails Ecommerce Gem

2018-02-14T00:00:00+00:00

Hi there! I’m Steph, the original creator and not-very-good maintainer of Piggybak, a modular ecommerce platform written in Ruby on Rails. With the help of End Point, I created Piggybak after building a custom Ruby on Rails ecommerce application for one of our clients here at End Point.

My goal with Piggybak was to create a lightweight, modular ecommerce platform in the form of a Ruby on Rails gem that could be combined with other Ruby on Rails gems for quick setup. Over the years, End Point has had much success working in Interchange, an ecommerce framework written in Perl. The web stack has evolved greatly over the years, as has the capacity for modularity and the ability to decouple front-end and back-end architecture.

Fast forward about 4 years after Piggybak was released, and we’ve decided to retire it. Not only did I leave the maintenance up to End Point after I left to work as an in-house software engineer for the last couple of years, but I was also in a position to evaluate using Piggybak as the base for a custom solution.

While I think there are some great Ruby on Rails gems to help support your ecommerce application (see below), one of the main things I realized was that the modularity in Piggybak often doesn’t suit the needs for the target audience of custom Ruby on Rails ecommerce platforms.

These days, here’s my oversimplified categorization of those looking for ecommerce solutions, divided into two audiences:

Audience #1: Boilerplate Saas with Theme Customization
Those sellers where boilerplate ecommerce solutions work, with simple product and variant modeling. Shopify, Magento, BigCommerce, WooCommerce can be decent popular options for that audience, but there are so many other options out there.

Audience #2: Custom Ecommerce in Ruby on Rails
Companies going this route have custom enough needs where a small team can develop and maintain a custom solution, using Ruby on Rails efficiently that don’t require them to depend on a pre-existing data model or business rule.

Not only did Piggybak suffer from a lack of community involvement, but it also didn’t suit the needs of the two audiences listed above. Because it was complex enough written in the form of a Rails gem (engine), it required a developer with some knowledge to install and customize further. And because it defined assumptions around the product-variant data model and business rules for inventory and payment management, it wasn’t necessarily a good fit for custom ecommerce needs.

Other Ruby on Rails Gems

While I’m sad to sunset Piggybak, I still believe Rails offers great options for ecommerce needs, including these popular Ruby on Rails gems:

Rails_admin, active_admin (no longer maintained), administrate
Carrierwave: File attachment management. Has decent third party support.
Devise. User authentication marches on.
Cancancan. User authorization.
Kaminari. Pagination.
RSpec, FactoryBot, Capybara, the testing stack I am most familiar with.
ActiveShipping. Shipping for ecommerce.
ActiveMerchant. Payment gateways for ecommerce.

If you are wondering if there are any Ruby on Rails ecommerce gems still out there, you can look at Solidus, Spree, and RoR-E. End Point has a long history with Spree, and experience with the other two platforms, but again, the audience of those businesses choosing a custom path may not want to be tied to the data models and business rules adopted by these existing platforms.

A Collaborative Timezone Utility

2017-10-30T00:00:00+00:00

Try It Out Yourself

The code for this project is hosted on GitHub and can be cloned from here.

At End Point Corporation, our team is spread out across 10 time zones. This gives us the advantage of being able to work around the clock on projects. When one co-worker leaves for day, another can take over. Consider this scenario. It’s Monday evening and Martin needs to continue installing software on that Linux cluster, but it’s already 6pm and his wife is going to murder him if he’s not ready to go out for their anniversary dinner. Let’s see who can take over… Ah, yes, Sanjay in Bangalore can continue with the maintenance. Tuesday morning, the client wakes up to be surprised that 16 hours of work was completed in a day. With respect to software development, the same efficiencies can be realized by parallelizing tasks across time-zones. Code reviews and further development can be continued after normal business hours.

With all the blessings of a distributed engineering team, collaborating with co-workers can be, occasionally, challenging. Some of these challenges stem from complexities of our system of time. Every co-worker may be operating in a timezone that is different than yours. Time-zones have an associated offset relative to Coordinated Universal Time (UTC). These offsets are usually in whole hour increments but they may be any real-valued number.

For example, Eastern Standard Time (EST) has an offset of -5 (five hours behind UTC) and Indian Standard Time (IST) has an offset of 5.5 (five and half hours ahead of UTC). Furthermore, these UTC offsets can be completely arbitrary. In 1995, Kiribati, an island nation in the Pacific, changed its UTC offset from -10 to +14 so that all of its outlying islands can share the same time. To further complicate things, some regions may not observe daylight savings time (DST) while other regions do. In fact, in the United States, Indiana started observing DST on April 2, 2006. Some states like Arizona and Hawaii do not observe DST. Other countries, like Australia, have a similar situation where it’s left to local governments to decide whether DST is observed or not. Moreover, although DST usually accounts for adding or subtracting an hour of time, it isn’t always one hour. This has historically changed from time to time.

Now you may begin to imagine the headaches that arise when you need to coordinate with anything involving multiple time-zones. To make all of this easier, you can use a utility that we wrote to do all the time conversions for you. First, you have to add each co-worker’s information to a configuration file stored at ~/.timezoner. This configuration file will describe all of your co-worker’s contact information and their associated IANA time-zone. As an example, this is what the configuration file looks like:

# Timezone            Email                  Name              OfficePhone        MobilePhone
America/New_York      "edward@example.com"   "Edward Teach "   "n/a"              "+1 731 555 1234"
America/New_York      "henry@dexample.com"   "Henry Morgan"    "+1 646 555 5678"  "+1 954 555 5678"
America/New_York      "john@example.com"     "John Auger"      "n/a"              "+1 902 555 1234"
America/Denver        "sam@example.com"      "Samuel Bellamy"  "+1 347 535 1234"  "+1 994 555 5678"
America/Los_Angeles   "william@example.com"  "William Kidd"    "+1 330 555 5678"  "+1 305 555 1234"
America/Los_Angeles   "israel@example.com"   "Israel Hands"    "+1 507 555 1234"  "+1 208 555 5678"

Now when I need to coordinate a meeting, I can run the utility with the -T option to see each team member’s local time.

With the -U option, you can display each contact separated in groups based on UTC offset.

Let us know what you think and if you found this tool helpful.

FOSDEM 2017: experience, community and good talks

2017-02-13T00:00:00+00:00

In case you happen to be short on time: my final overall perspective about FOSDEM 2017 is that it was awesome… with very few downsides.

If you want the longer version, keep reading cause there’s a lot to know and do at FOSDEM and never enough time, sadly.

This year I actually took a different approach than last time and decided to concentrate on one main track per day, instead of (literally) jumping from one to the other. While I think that overall this may be a good approach if most of the topics covered in a track are of your interest, that comes at the cost of missing one of the best aspects of FOSDEM which is “variety” in contents and presenters.

Day 1: Backup & Recovery

For the first day I chose the Backup & Recovery track which hosted talks revolving around three interesting and useful projects: namely REAR (Relax and Recovery), DRLM, a wrapper and backup management tool based on REAR and Bareos, which is a backup solution forked from Bacula in 2010 and steadily proceeding and improving since then. Both REAR and DLRM were explained and showcased by some of the respective projects main contributors and creators. As a long time system administrator, I particularly appreciated the pride in using Bash as the main “development platform” for both projects. As Johannes Meixner correctly mentioned, using bash facilitates introduces these tools into your normal workflow with knowledge that you’ll most likely already have as a System Administrator or DevOps, thus allowing you to easily “mold” these scripts to your specific needs without spending weeks to learn how to interact with them.

During the Day 1 Backup & Recovery track there were also a few speeches from two Bareos developers (Jörg Steffens and Stephan Dühr) that presented many aspects of their great project, ranging from very introductory topics, to providing a common knowledge ground for the audience, up to more in depth topics like software capabilities extension through Python Plugins, or a handful of best practices and common usage scenarios. I also enjoyed the speech about automated testing in REAR, presented by Gratien D’haese, which showed how to leverage common testing paradigms and ideas to double-check a REAR setup for potential unexpected behaviors after updates or on new installations or simply as a fully automated monitoring tool to do sanity checks on the backup data flow. While this testing project was new, it’s already functional and impressive to see at work.

Day 2: Cloud Microservices

On the second day I moved in a more “cloudy” section of the FOSDEM where most of the conferences revolved around Kubernetes, Docker and more in general the microservices landscape. CoreOS (the company behind the open source distribution) was a major contributor and I liked their Kubernetes presentation by Josh Wood and Luca Bruno which respectively explained the new Kubernetes Operators feature and how containers work under the hood in Kubernetes.

Around lunch time there was a “nice storm of lightning talks” which kept most of the audience firmly on their seats, especially since the Microservices track room didn’t have a free seat for the entire day. I especially liked the talk from Spyros Trigazis about how CERN created and is maintaining a big OpenStack Magnum (the container integrated version of OpenStack) cloud installation for their internal use.

Then it was Chris Down’s turn and, while he’s a developer from Facebook, his talk gave the audience a good perspective on the future of CGROUPs in the Linux kernel and how they are already relatively safe and usable, even if not yet officially marked as production ready. While I already knew and used “sysdig” in past as a troubleshooting and investigation tool, it was nice to see one of the main developers, Jorge Salamero, using it and showing alternative approaches such as investigating timeout issues between Kubernetes Docker containers by just sysdig and its many modules and filters. It was really impressive seeing how easy it is to identify cross-containers issues and data flow.

Atmosphere

There were a lot of Open Source communities with “advertising desks” and I had a nice talk with a few interesting developers from the CoreOS team or from FSFE (Free Software Foundation Europe). Grabbing as many computer stickers as possible is also mandatory at FOSDEM, so I took my share and my new Thinkpad is way more colorful now. In fact, on a more trivial note, this year the FOSDEM staff decided to sell on sale all the laptops that were used during the video encoding phase for the streaming videos before the upload. These laptops were all IBM Thinkpad X220 and there were only a handful of them (~30) at a very appealing price. In fact, this article is being written from one of those very laptops now as I was one of the lucky few which managed to grab one before they were all gone within an hour or so. So if you’re short of a laptop and happen to be at FOSDEM next year, keep your eyes open cause I think they’ll do it again!

So what’s not to like in such a wonderful scenario? While I admit that there was a lot to be seen and listened to, I sadly didn’t see any “ground-shaking” innovation this year at FOSDEM. I did see many quality talks and I want to send a special huge “thank you” to all the speakers for the effort and high quality standards that they keep for their FOSDEM talks—but I didn’t see anything extraordinarily new from what I can remember.

Bottom line is that I still have yet to find someone who was ever disappointed at FOSDEM, but the content quality varies from presenter to presenter and from year to year, so be sure to check the presentations you want to attend carefully before hand.

I think that the most fascinating part of FOSDEM is meeting interesting, smart, and like-minded people that would be difficult to reach otherwise.

In fact, while a good share of the merit should be attributed to the quality of the content presented, I firmly believe that the community feeling that you get at FOSDEM is hard to beat and easy to miss when skipped even for one year.

I’ll see you all next year at FOSDEM then.

Malaysia Open Source Community Meetup Quarter 4 2015 (MOSCMY Q4 2015)

2016-12-09T00:00:00+00:00

After a year, finally I decided to publish this post to all of you!

On November 26th 2015 I had a chance to give a talk in a local open source conference here in Malaysia. The organizer requested me to specifically deliver a talk on “remote work”. This meetup was organized by Malaysian Development Corporation (MDEC) with the sponsorship of Microsoft Malaysia. Microsoft recently started to become more “open source friendly” given that they are in the effort of pushing their cloud based product, Azure. The full schedule of the event can be referred here.

The conference was divided into two sessions; where the morning session was held in Petronas Club, Tower One of Kuala Lumpur City Centre (KLCC) and the other session was held in Microsoft Malaysia’s office in Tower Two KLCC. Generally the morning session was for non parallel track (including my track) and the afternoon sessions were two parallel sessions slot.

Morning Session

The morning session started with a talk by Dinesh Nair, as the Director of Developer Experience and Evangelism, Microsoft Malaysia. The second session in the morning was delivered by Mr Izzat M. Salihuddin, from Multimedia Development Corporation (MDeC), Malaysia. He spoke on the behalf of MDeC explaining the effort by MDeC as a government wing to realize the local cloud infrastructure. One of the challenges that being mentioned by Mr Izzat was the readiness of physical infrastructure as well as the broadband connectivity for the public. The final slot in the morning was delivered by me in which I explained much of the way of how End Pointers do their job, open source software that we used, as well as how we accomplish our job remotely. The morning session was adjourned with a lunch break.

Just in case if you are wondering, this is me delivering the talk on that day

Afternoon Session

The afternoon session was a parallel track session, where I chose to attend a talk on Ubuntu’s Juju service. The talk was delivered by Mr Khairul Aizat Kamarudzaman from Informology. Mr Aizat’s slides for his talk could be read here. Later, Mr Sanjay shared his Asterisk skills in which the server is hosted on the Azure platform. Mr Sanjay showed to us how make phone call from the computer to the mobile phone. Asterisk is different from Skype because it is using an open protocol (SIP) and with open clients. Mr Sanjay showed a demo on his implementation, in which it looks like the setup is to compete with the typical PABX phone system.

For the next slot I decided to enter the slot on TCPTW kernel patch which was delivered by Mr Faisal from Nexoprima. As far as I understood, Mr Faisal reintroduced his own patch for the Linux kernel in order to handle TCP TIME_WAIT issue which was happened due to extremely busy HTTP requests. Since connection in TIME_WAIT state hold a local port for 1 minute and in many distro the default ports are up to 30,000, the effort put to search for free port(s) will use intensive CPU and it could was CPU cycle to purge tons of TIME_WAIT connections.

Mr Faisal gave his talks on TCPTW kernel patch

Mr Faisal’s TCPTW patch for CentOS 7 could be viewed here. His presentation slides could be viewed here.

Before went back home, I decided to enter a talk on “Dockerizing IOT Service” by Mr Syukor. In this talk Mr Syukor gave a bit theoretical background on Docker and how it can be used on Raspberry Pi board. You can view his slides here. My personal thought is that Raspberry Pi is versatile enough to run any modern operating system and Docker should not be much an issue.

Happy 10th birthday, Git!

2015-04-08T00:00:00+00:00

Git’s birthday was yesterday. It is now 10 years old! Happy birthday, Git!

Git was born on 7 April 2005, as its creator Linus Torvalds recounted in a 2007 mailing list post. At least if we consider the achievement of self-hosting to be “birth” for software like this. :)

Birthdays are really arbitrary moments in time, but they give us a reason to pause and reflect back. Why is Git a big deal?

Even if Git were still relatively obscure, for any serious software project to survive a decade and still be useful and maintained is an accomplishment. But Git is not just surviving.

Over the past 5–6 years, Git has become the standard version control system in the free software / open source world, and more recently, it is becoming the default version control system everywhere, including in the proprietary software world. It is amazing to consider how fast it has overtaken the older systems, and won out against competing newer systems too. It is not unreasonable these days to expect anyone who does software development, and especially anyone who claims to be familiar with version control systems, to be comfortable with Git.

So how did I get to be friends with Git, and end up at this birthday celebration?

After experimenting with Git and other distributed version control systems for a while in early and mid-2007, I started using Git for real work in July 2007. That is the earliest commit date in one my personal Git repositories (which was converted from an earlier CVS repository I started in 2000). Within a few weeks I was in love with Git. It was so obviously vastly superior to CVS and Subversion that I had mostly used before. It offered so much more power, control, and flexibility. The learning curve was real but tractable and it was so much easier to prevent or repair mistakes that I didn’t mind the retraining at all.

So I’m sounding like a fanboy. What was so much better?

First, the design. A distributed system where all commits were objects with a SHA-1 hash to identify them and the parent commit(s). Locally editable history. Piecemeal committing thanks to the staging power of the Git index. Cheap and quick branching. Better merging. A commit log that was really useful. Implicit rename tracking. Easy tagging and commit naming. And nothing missing from other systems that I needed.

Next, the implementation. Trivial setup, with no political and system administrative fuss for client or server. No messing with users and permissions and committer identities, just name & email address like we’re all used to. An efficient wire protocol. Simple ssh transport for pushes and/or pulls of remote repositories, if needed. A single .git directory at the repository root, rather than RCS, CVS, or .svn directories scattered throughout the checkout. A simple .git/config configuration file. And speed, so much speed, even for very large repositories with lots of binary blobs.

The speed is worth talking about more.

The speed of Git mattered, and was more than just a bonus. It proved true once again the adage that a big enough quantitative difference becomes a qualitative difference. Some people believed that speed of operations wasn’t all that important, but once you are able to complete your version control tasks so quickly that they’re not at all bothersome, it changes the way you work.

The ease of setting up an in-place repository on a whim, without worrying about where or if a central repository would ever be made, let alone wasting any time with access control, is a huge benefit. I used to administer CVS and Subversion repositories and life is so much better with the diminished role a “Git repository administrator” plays now.

Cheap topic branches for little experiments are easy. Committing every little thing separately makes sense if I can later reorder or combine or split my commits, and craft a sane commit before pushing it out where anyone else sees it.

Git subsumed everything else for us.

RCS, despite its major limitations, stuck around because CVS and Subversion didn’t do the nice quick in-place versioning that RCS does. That kind of workflow is so useful for a system administrator or ad-hoc local development work. But RCS has an ugly implementation and is based on changing single files, not sets. It can’t be promoted to real distributed version control later if needed. At End Point we used to use CVS, Subversion, and SVK (a distributed system built on top of Subversion), and also RCS for those cases where it still proved useful. Git replaced them all.

Distributed is better. Even for those who mostly use Git working against a central repository.

The RCS use case was a special limited subset of the bigger topic of distributed version control, which many people resisted and thought was overkill, or out of control, or whatever. But it is essential, and was key to fixing a lot of the problems of CVS and Subversion. Getting over the mental block of Git not having a single sequential integer revision number for each commit as Subversion did was hard for some people, but it forces us to confront the new reality: Commits are their own objects with an independent identity in a distributed world.

When I started using Git, Mercurial and Bazaar were the strongest distributed version control competitors. They were roughly feature-equivalent, and were solid contenders. But they were never as fast or compact on disk, and didn’t have Git’s index, cheap branching, stashing, or so many other niceties.

Then there is the ecosystem.

GitHub arrived on the scene as an unnecessary appendage at first, but its ease of use and popularity, and social coding encouragement, quickly made it an essential part of the Git community. It turned the occasional propagandistic accusation that Git was antisocial and would encourage project forks, into a virtue, by calling everyone’s clone a “fork”.

Over time GitHub has played a major role in making Git as popular as it is. Bypassing the need to set up any server software at all to get a central Git repository going removed a hurdle for many people. GitHub is a centralized service that can go down, but that is no serious risk in a distributed system where you generally have full repository mirrors all over the place, and can switch to other hosting any time if needed.

I realize I’m gushing praise embarrassingly at this point. I find it is warranted, based on my nearly 8 years of using Git and with good familiarity with the alternatives, old and new.

Thanks, Linus, for Git. Thanks, Junio C Hamano, who has maintained the Git open source project since early on.

Presumably someday something better will come along. Until then let’s enjoy this rare period of calm where there is an obvious winner to a common technology question and thus no needless debate before work can begin. And the current tool stability means we don’t have to learn new version control skills every few years.

To commemorate this 10-year birthday, Linux.com interviewed Linus and it is a worthwhile read.

You may also be interested to read more in the Wikipedia article on Git or the Git wiki’s history of Git.

Finally, an author named Stephen has written an article called The case for Git in 2015, which revisits the question of which version control system to use as if it were not yet a settled question. It has many good reminders of why Git has earned its prominent position.

Impressions from Open Source work with Elixir

2015-03-26T00:00:00+00:00

Some time ago I started working on the Elixir library that would allow me to send emails as easily as ActionMailer known from the Ruby world does.

The beginnings were exciting—I got to play with a very clean and elegant new language which Elixir is. I also quickly learned about the openness of the Elixir community. After hacking some first draft-like version and posting it on GitHub and Google groups—I got a very warm and thorough code review from the language’s author José Valim! That’s just impressive and it made me even more motivated to help out the community by getting my early code into a better shape.

Coding the ActionMailer like library in a language that was born 3 years ago doesn’t sound like a few hours job—there’s lots of functionality to be covered. An email’s body has to be somehow compiled from the template but also the email message has to be transformed to the form in which the SMTP server can digest and relay it. It’s also great if the message’s body can be encoded with „quoted printable”—this makes even the oldest SMTP server happy. But there’s lots more: connecting with external SMTP servers, using the local in-Elixir implementation, ability to test etc…

Fortunately Elixir’s built on top of the Erlang’s „Virtual Machine”—BEAM—which makes you able to use its libraries—a lot of them. For the huge part of the functionality I needed to cover I chose the great gen_smtp library. This allowed me to actually send emails to SMTP servers and have them properly encoded. With the focus on developer’s productivity, Elixir made me come up with the nice set of other features that you can check out here.

This serves as a shout out blog post for the Elixir ecosystem and community. The friendliness that it radiates with makes open source work like this very rewarding. I invite you to make your contributions as well—you’ll like it.

Making Blog Search Smarter with LLMs and Open WebUI

What “LLM Expanded Search” actually does

How it works

Why we chose Open WebUI

What’s next: Moving more into Open WebUI

What you’ll notice when using it

Announcing End Point Ecommerce, Our New Open Source Project

Key features

How we got here

Who is this for?

New Cesium KML-CZML Editor Features: Custom Data & Styling, Google 3D Tiles, and More

Custom data & styling

Demo

Place Labels in Cesium using Vector Data Tiles

Challenges of Vector Data in Cesium

Technical Implementation

Managing Uneven Label Distribution

Clustering Labels

Next Steps

Notes

Exploring Geodatabase Files

Prerequisites

A first look into the contents of the GDB file

The ogr2ogr -sql option

1. Extract property names

2. Extract polygons only

3. Create KML with pins and names

4. Create KML with pins only, no names

5. Extract limited properties

6. Run it all in a Python script

7. Extract all data as JSON from one layer using the Python GDAL library

Conclusion

Additional Resources

The Perl and Raku Conference 2024

An OpenStreetMap Editor for Adding Public Transit Data Using GTFS

My OpenStreetMap editor for GTFS data

How to use the app

Adding stops to OSM

Demo videos

An important note

Cesium Labels

Backend and data source

Future Plans

Running On Your System

Formatting SQL code with pgFormatter within Vim

On Shapefiles and PostGIS

QGIS

GDAL

PostGIS

Examples

Import Shapefile data into PostGIS

Export PostGIS data as KML

CZML-KML Editor Geometry Editing

Announcing the Cesium KML-CZML Editor

EMSA: Electronic Messaging Staging Area

EMSA:

Benefits of EMSA

Help is available!

References

Fix for cdparanoia segmentation fault

LinuxFest Northwest 2019

Switching from Google Maps to Leaflet

Creating a basic map

Adding a marker

Layers and controls

Plugins and tools

Putting it all together

Start basic application with Vue.js 2 and Drupal 8

Introduction

What do we need?

Step by step

Install Vagrant, VirtualBox, and Git

Install and setup back-end project

Setup new Vue.js project

Let’s code!

Vue in Ecommerce: Routing and Persistence

Vue Router

Without Routing

With Routing

History Mode

The `ogr2ogr` `-sql` option