A Preact Web App Without npm Build

2025-10-15T00:00:00+00:00

Modern JavaScript frameworks such as React and Preact usually require a full “build pipeline” with Node.js, npm, and bundlers like Webpack or Vite. That’s fine for large web apps, but sometimes you just need a small, self-contained browser interface inside a project that isn’t mainly about the web — say, a Java program that processes transit data and simply needs a page to display results.

This post explores how to use Preact and its companion library @preact/signals without any npm build process at all. By taking advantage of browser-native import maps and writing a bit of code directly in JavaScript instead of JSX, it shows how to build a lightweight, modern UI that runs straight from static HTML, CSS, and JS files — no build tools required.

The Scenario

This might look like a strange problem to solve but sometimes the web UI part just isn’t a main character in your project.

For instance, I have a GTFS data processor whose main goal is to preprocess public transport stop timetables. It’s written in Java and I want a simple browser UI to check its output. So I want the web UI part of the project to be just static HTML, CSS, and JavaScript files.

I also don’t want to run npm or yarn inside what would otherwise be a pure Maven-managed project. But at the same time, I would like to use a modern JS UI framework such as Preact.

The Main Issues

To pull off this trick, there are two main challenges:

JSX
Imports

JSX

JSX is basically JavaScript templates. When you write something like:

function Greeting() {
  return (
    <div className="greeting">
      Hello world
    </div>
  )
}

function App() {
  return (<Greeting/>)
}

During project build Babel will transform it into:

function Greeting() {
  // Particular function for creating elements may vary.
  return h('div', {className: "greeting"}, 'Hello world')
}

function App() {
 return h(Greeting, null);
}

This is a pretty straightforward transformation, and no one is stopping you from importing the element creation function (h in the example above) and composing what would be Babel output yourself.

Imports

Now for the more interesting issue: imports.

Normally, with a build process, you could just write:

import { h } from 'preact'
import { signal } from '@preact/signals'

What’s more, some packages will try to resolve their own dependencies by importing them by global name. For instance, @preact/signals will try to import @preact/signals-core, and there isn’t a ready-made UMD (Universal Module Definition) version of @preact/signals-core available.

The Solution: Browser Import Maps

I would like to be able to just map library names to URLs of ES modules.

There is a relatively new browser feature that does exactly this: an import map. It’s a special type of <script> block containing a JSON object with your mapping.

<script type="importmap">
  {
    "imports": {
      "preact": "https://unpkg.com/preact@latest/dist/preact.mjs",
      "preact/hooks": "https://unpkg.com/preact@latest/hooks/dist/hooks.mjs",
      "preact/compat": "https://unpkg.com/preact@latest/compat/dist/compat.mjs",
      "@preact/signals-core": "https://unpkg.com/@preact/signals-core@latest/dist/signals-core.mjs",
      "@preact/signals": "https://unpkg.com/@preact/signals@latest/dist/signals.mjs"
    }
  }
</script>

And that’s it!

With the import map in place, your browser can resolve all those imports directly from a CDN like unpkg.com. You can then use Preact, hooks, and signals exactly as if you were running a fully bundled app.

Conclusion

This approach is ideal for small, embedded, or hybrid projects where a full JavaScript toolchain would be overkill. It lets you:

Build modern UI components with Preact
Avoid npm, yarn, Babel, and bundlers entirely
Keep your project clean and language-native (e.g., Maven for Java, without extra layers)
Run everything as plain static assets, right from your browser

Everything works, and nothing extra gets in the way.

How to write end-to-end & component tests with Cypress in Vue.js

2022-12-10T00:00:00+00:00

Is writing tests painful for you? In this tutorial, I explain how to handle UI testing with Cypress and hope to convince you that writing tests is not always so tedious and expensive, but can be fun instead.

Cypress is a purely JavaScript-based front-end testing tool built for the modern web. it can test anything that runs in a browser and has built-in support for testing modern frameworks such as Vue.js, React, and Angular. See the full list of front-end frameworks Cypress supports.

As an example we are going to use a to-do app built using Vue. We will learn:

How to install and set up Cypress.
How to create a simple to-do app with Vue 3.
How to write end-to-end tests.
How to write component tests.

How to install and set up Cypress

First let’s create a new Vue project using the Vue CLI.

Install Vue CLI if you don’t have it in your machine:
```
npm install -g @vue/cli
```
Create a project (pick the Vue 3,babel,eslint preset):
```
vue create todo-app
```
cd into the todo-app project and install Cypress:
```
npm install cypress --save-dev
```
No dependencies, extra downloads, or changes to your code are required!
Edit package.json. In the scripts section, add a command, "cypress:open": "cypress open". See the example below.
```
"scripts": {
    "cypress:open": "cypress open"
}
```
Launch Cypress:
```
npm run cypress:open
```

When Cypress opens up you should be able to view the Cypress launchpad as shown in this screenshot:

We will use the Launchpad to configure both E2E Testing and Component Testing. Cypress will automatically generate configuration files as you configure the tests in the Launchpad. If you get stuck, please visit the docs for Launchpad.

How to create a simple to-do app with Vue 3

Open the todo-app project in your favourite editor and add the following single-file components:

src/components/BaseTextInput.vue

BaseTextInput will contain a text input and a button for adding new to-do items.

<template>
    <form class="add-todo-form">
        <input type="text" class="input" placeholder="Add a new todo" v-model="todo">
        <input type="submit" value="Add" @click="emitNewTodoEvent">
    </form>
</template>

<script>
export default {
    data() {
        return {
            todo: ''
        }
    },
    methods: {
        emitNewTodoEvent(e) {
            e.preventDefault();
            this.$emit('newTodo', this.todo);
            this.todo = '';
        }
    }
}
</script>

<style scoped>
.input {
    width: 100%;
    padding: 8px 10px;
    border: 1px solid #32485F;
}

.add-todo-form {
    width: 100%;
    display: flex;
    align-items: center;
}

input[type="submit"] {
    margin-left: 5px;
    padding: 8px 10px;
    border: 1px solid #32485F;
    background-color: #32485F;
    color: #fff;
    font-weight: bold;
    cursor: pointer;
}

input[type="submit"]:hover {
    background-color: #00C185;
}
</style>

src/components/TodoListItem.vue

TodoListItem displays a to-do along with a button for removing that to-do.

<template>
    <li>
      {{ todo.text }}
      <button @click="$emit('remove', todo.id)">
        X
      </button>
    </li>
  </template>

  <script>
  export default {
    props: {
      todo: {
        type: Object,
        required: true
      }
    }
  }
  </script>

src/components/TodoList.vue

TodoList component lists all to-dos.

<template>
  <div>
    <BaseTextInput
      @newTodo="addTodo"
    />
    <ul v-if="todos.length">
      <TodoListItem
        v-for="todo in todos"
        :key="todo.id"
        :todo="todo"
        @remove="removeTodo"
      />
    </ul>
    <p class="empty-state-message" v-else>
      Nothing left in the list. Add a new todo in the input above.
    </p>
  </div>
</template>

<script>
import BaseTextInput from './BaseTextInput.vue';
import TodoListItem from './TodoListItem.vue';

let nextTodoId = 1;

export default {
  components: {
    BaseTextInput, TodoListItem
  },
  data () {
    return {
      todos: []
    }
  },
  methods: {
    addTodo (todo) {
      const trimmedText = todo.trim();
      if (trimmedText) {
        this.todos.unshift({
          id: nextTodoId++,
          text: trimmedText
        });
      }
    },
    removeTodo (idToRemove) {
      this.todos = this.todos.filter(todo => {
        return todo.id !== idToRemove;
      });
    }
  }
}
</script>

Edit src/App.vue to include code that will render the Todo App.

<template>
  <div id="app">
    <h1>My Todo App!</h1>
    <TodoList/>
  </div>
</template>

<script>
import TodoList from './components/TodoList.vue';

export default {
  components: {
    TodoList
  }
}
</script>

<style>
*, *::before, *::after {
  box-sizing: border-box;
}

#app {
  max-width: 400px;
  margin: 0 auto;
  line-height: 1.4;
  font-family: 'Avenir', Helvetica, Arial, sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  color: #00C185;
}

h1 {
  text-align: center;
}
</style>

Then, run:

npm run serve

and visit http://localhost:8080 to view your Todo App!

How to write end-to-end tests

End-to-end (E2E) testing is used to test an application flow from start to finish. Tests are designed to use the application the same way that a user would.

In our example we are going to test the whole Todo app with the following test cases:

The user should see a message when the to-do list is empty.
The user should be able to view a list of to-dos.
The user should be able to add a new to-do.
The user should be able to remove an existing to-do.

Cypress is built on top of Mocha and Chai. If you’re familiar with writing tests in JavaScript, then writing tests in Cypress will be a breeze.

Visit the official Cypress docs to learn more about how to start testing a new project in Cypress.

Without futher ado lets start testing our Todo app.

Inside the cypress folder that was added during installation create a file, cypress/e2e/todo.cy.js, and add the following tests.

Please visit the folder structure docs to learn more about how to organise tests in Cypress and understand the generated Cypress folder structure.

/* eslint-disable no-undef */

// describe() function is used to group tests
describe('Todo tests', () => {
  it('should display empty state message', () => {
    // cy.visit() used to visit a remote url
    // learn more about it here: https://docs.cypress.io/api/commands/visit#Syntax
    cy.visit('http://localhost:8080');

    // cy.get() command Gets one or more DOM elements by selector or alias
    // learn more about Cypress commands/api here: https://docs.cypress.io/api/table-of-contents
    cy.get('.empty-state-message')
    .contains('Nothing left in the list. Add a new todo in the input above.')
    .should('be.visible');
  });

  it('should add todo', () => {
    cy.visit('http://localhost:8080');

    // add todo by typing in the input and pressing enter
    cy.get('input[type="text"]').type('new todo{enter}');

    // check if the todo is added
    cy.get('ul').contains('new todo').should('be.visible');
    cy.get('ul').find('li').should('have.length', 1);


    // add another todo by typing in the input and pressing 'add' button
    cy.get('input[type="text"]').type('more todo');
    cy.get('input[type=submit]').click();

    // check if the todo is added
    cy.get('ul').contains('more todo').should('be.visible');
    cy.get('ul').find('li').should('have.length', 2);
  });

  it('should delete todo', () => {
    cy.visit('http://localhost:8080');

    // delete the first todo
    cy.get(':nth-child(1) > button').click();

    // check if the todo is deleted
    // cy.should() command is used to assert that the todo list has only one todo
    // learn more about Cypress assertions here: https://docs.cypress.io/guides/references/assertions
    cy.get('ul').find('li').should('have.length', 1);
  });
});

To make Vue available at http://localhost:8080, run:

npm run serve

Then, to launch Cypress, run:

npm run cypress:open

On the Cypress launcher select “End2End Tests” and click on the todo.cy.js spec to run your tests.

How to write component tests

Cypress Component Test Runner executes your component tests in the browser as a user would by simulating real interactions. Since it runs in the browser, you get to debug your components using your favourite developer tools.

To demonstrate how to write component tests using Cypress let’s write tests for the following components:

BaseTextInput.vue
TodoListItem.vue

Component tests for BaseTextInput.vue

Here we are going to assert the following;

Text input is rendered with placeholder text “Add a new todo”.
When the “add” button is clicked a newTodo event is emitted with a payload containing text that was typed in the text input.

First create a spec, src/components/BaseTextInput.cy.js, then add the following code:

/* eslint-disable no-undef */
import BaseTextInput from './BaseTextInput.vue';

describe('<BaseTextInput />', () => {
  it('renders base text input component', () => {
    // Renders the component in DOM.
    // cy.mount() is a custom command.
    // Learn more about cypress custom commands here:
    // https://docs.cypress.io/api/commands/mount#Creating-a-New-cy-mount-Command
    cy.mount(BaseTextInput);

    // Asserts that the text input is rendered with the correct placeholder
    cy.get('input').should('have.attr', 'placeholder', 'Add a new todo');

    // asserts that when the 'add' button is clicked, an event is emitted
    // with the payload containing the value of the text input
    cy.get('input[type="text"]').type('new todo');
    cy.get('input[type=submit]').click().then(() => {
      // Cypress.vueWrapper provides access to the Vue Test Utils.
      // With this wrapper you can access any Vue Test Utils API.
      // Learn more about Vue Test Utils here: https://vue-test-utils.vuejs.org/
      // e.g. cy.vueWrapper().emitted() returns all the events emitted by the BaseTextInput component
      cy.wrap(Cypress.vueWrapper.emitted()).should('have.property', 'newTodo');
      expect(Cypress.vueWrapper.emitted().newTodo[0]).to.deep.equal(['new todo']);
    });
  })
});

To run the test, launch Cypress using cypress:open. This time select “component testing”, then click on BaseTextInput.cy.js to run tests.

Component tests for TodoListItem.vue

Here we are going to assert the following:

A Todo list item is rendered with correct text.
When the “remove” button is clicked, a “remove” event is emitted with payload containing the ID of the Todo to be removed.

First create a spec, src/components/TodoListItem.cy.js, and add the following code:

/* eslint-disable no-undef */
import TodoListItem from './TodoListItem.vue';

describe('<TodoListItem />', () => {
  it('renders todo list item', () => {
    const text = 'new todo';
    const id = 1;
    // mount the component with props
    // see: https://test-utils.vuejs.org/guide/
    cy.mount(TodoListItem, {props: {todo: { id, text }}});

    // asserts that the todo list item is rendered with the correct text
    cy.get('li').contains(text);

    // asserts that when 'x' button is clicked, 'remove' event is emitted with the correct payload
    cy.get('button').click().then(() => {
      cy.wrap(Cypress.vueWrapper.emitted()).should('have.property', 'remove');
      expect(Cypress.vueWrapper.emitted().remove[0]).to.deep.equal([id]);
    })
  })
});

To run the test, launch Cypress using cypress:open. Select “component testing” again, then click on TodoListItem.cy.js to run tests.

The final source code with all the tests can be found in my GitHub todo-app repo.

Next steps

This article is meant to provide you with knowledge on how to set up and run tests using Cypress. However, you may need to further learn Cypress and its API to write better and efficient tests. I recommend you continue learning with:

Migrating a Node.js app database from MongoDB to PostgreSQL

2022-02-25T00:00:00+00:00

Recently I worked on a web app designed for tracking, editing, and reporting on archaeological survey data. When I joined the project, it used React on the front-end, Node.js on the back-end, GraphQL as the API interface, and MongoDB as the database. For the most part, this architecture and choice of technology worked well, and I was quickly able to make meaningful contributions to the app both in the user interface and in the backend. However, after a while, things started to get difficult and we began to consider why we were using MongoDB, and what might be better.

MongoDB vs. PostgreSQL

First, why was MongoDB difficult to use in our case? The biggest difficulty was using their query language for certain aspects of our database. Part of the database involved representing artifacts found at a site using a specific type hierarchy defined by the state government. Representing that hierarchy within Mongo was difficult, and querying it even more so. It was hard to find how to query a recursive structure using the MongoDB query language.

As I worked on this, scouring the documentation and Q&A websites to find ways to query this data efficiently, I began to realize that it was a problem that could be easily solved with PostgreSQL’s recursive queries.

Now, a single recursive table isn’t really a good reason to completely switch the database used by an application, but it got us thinking: Why were we using MongoDB? What benefits was it offering?

I found a relevant article published on the official MongoDB website entitled Comparing MongoDB vs PostgreSQL. That article is generally very positive in its portrayal of PostgreSQL, so I was interested to hear what they had to say for themselves. Here are some of the main points in favor of MongoDB that the article discusses:

MongoDB is horizontally scalable by default: This seems cool, but we were working on a new app that was not expected to need to scale much, and a single PostgreSQL instance has been shown to scale more than enough for our anticipated needs.
MongoDB supports fancy new serverless and mobile paradigms: Again, cool, but this was a fairly small-scale project using a pretty normal Linux/Node.js stack. No need for serverless support.
MongoDB’s document structure makes it easy for developers to change the structure of data on the fly: While this can certainly be nice, it can also bring its own issues. If data is expected to fit a certain structure and one developer changes that structure, this will likely cause problems for other developers whether or not there is a schema. Additionally, our project had a small team of 1-3 developers working on it and changing the schema as needed just wasn’t a problem.

The MongoDB vs. PostgreSQL article also mentions “resilience” as a strength of MongoDB, citing its ability to easily be replicated and broken into shards across datacenters and regions. However, research regarding the historical reliability of PostgreSQL and MongoDB showed that MongoDB’s fairly short history is littered with data loss problems. Anecdotally, a Google search for “MongoDB data loss” returns 1.26 million results (an average of 0.14 million results per year since its initial release) while “PostgreSQL data loss” returns 1.57 million (0.06 million per year since release). More specifically, there are fascinating in-depth reviews of MongoDB’s reliability, such as those conducted by Jepsen as recently as 2020 (see #1, #2, #3, #4, #5).

PostgreSQL hasn’t been perfect in the 25 years since its initial release, and MongoDB has improved a lot in recent years, but it’s clear that Postgres is the winner when it comes to data reliability.

After reviewing all of these pros and cons of MongoDB, it became clear that it wasn’t really offering anything that we needed, and our data, which was strictly organized and relational by nature, would fit much better in a relational database. We decided that it would be better to switch to PostgreSQL.

Mongoose and Sequelize

Our database was only being used within the backend GraphQL server, using Mongoose, an object modeling library for Node.js. Mongoose uses a schema/model system, where every MongoDB collection contains documents fitting a schema defined within Mongoose model definitions, for example:

import mongoose from "mongoose";
const { Schema } = mongoose;

const blogPostSchema = new Schema({
  title: String, // String is shorthand for {type: String}
  author: String,
  body: String,
  comments: [{ body: String, date: Date }],
  date: { type: Date, default: Date.now },
  hidden: Boolean,
  meta: {
    votes: Number,
    favs: Number,
  },
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Once a model is defined it can easily be used to query and create documents:

const new_post = new BlogPost({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

const posts = await BlogPost.find({ author: "Ronald McClure" });

Mongoose provides a lot of functions for querying and editing documents in a collection on the Model class, as well as a rich Query class to make manual queries, among other things. Their documentation provides helpful guides and a great API reference.

Because we generally liked the Mongoose interface and didn’t want to have to change our existing code too drastically, we searched for a similar library that could work with PostgreSQL and settled upon Sequelize.

Sequelize has a very similar interface to Mongoose, where you start by defining tables (instead of “collections”) as models in JavaScript:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // Model attributes are defined here
  title: DataTypes.STRING,
  author: DataTypes.STRING,
  body: DataTypes.STRING,
  // This will need to be modeled differently
  // comments: [{ body: DataTypes.STRING, date: Date }],
  date: {
    type: DataTypes.DATETIME,
    defaultValue: DataTypes.NOW
  },
  hidden: DataTypes.BOOLEAN,
  votes: DataTypes.INTEGER,
  favs: DataTypes.INTEGER
}, {
  // Other model options go here
});

// Object creation is similar
const new_post = BlogPost.build({
  title: "On the international epidemic of counterfeit at-home brogal treatments",
  author: "Ronald McClure",
  body: "...",
  // ...
});
await new_post.save();

Note that this model, while mimicking the Mongoose schema as much as possible, still requires some changes to fit the relational model. First, the comments field, which was an array of documents in the Mongoose object will need to be represented in another table, which can then be linked via foreign key.

While this seems annoying in some ways — creating a new table will require creating another Sequelize model, setting up the relations correctly, etc. — it makes a lot of sense for this example (a blog) and most other situations. We don’t just want blog posts to have associated comments, but we likely also want those comments to be tied to an author and to have their own IDs, which would allow lookup by author (for example so that on a user’s page they can see all comments they’ve made on all posts) and by individual comment (so it’s easier to make links directly to a single comment).

So, rather than having comments be a subfield of a blog post, we want them to be defined by some kind of relation. In MongoDB this can be done in a few ways, most obviously by creating a comments collection making the comments list contain ObjectIDs:

const mongoose = require("mongoose");
const { Schema } = mongoose;

const Comment = mongoose.model(
  "Comment",
  new Schema({
    body: String,
    date: Date,
  })
);

const blogPostSchema = new Schema({
  // ...
  comments: [{ type: Schema.Types.ObjectId, ref: "Comment" }],
  // ...
});

const BlogPost = mongoose.model("BlogPost", blogPostSchema);

Mongoose also has a way to automatically fill in those referenced objects with the real thing as needed, using their populate functionality:

BlogPost.findOne({ author: "Ronald McClure" })
  .populate("comments")
  .exec(function (err, post) {
    if (err) return handleError(err);
    post.comments.forEach((comment) => console.log(comment));
  });

With the call to .populate('comments'), the comments array is filled with objects instead of just the object IDs. Sequelize has similar functionality, but uses the concept of associations which model the relational structure of SQL more closely:

const { Sequelize, DataTypes } = require('sequelize');
const sequelize = new Sequelize('postgres://user:pass@example.com:5432/dbname');

const BlogPost = sequelize.define('BlogPost', {
  // ... fields here
}, {});

const Comment = sequelize.define('Comment', {
  body: DataTypes.TEXT,
  date: DataTypes.DATE,
}, {});

Comment.belongsTo(BlogPost;
BlogPost.hasMany(Comment);

These belongsTo and hasMany associations use foreign key/primary key references on the models, and Sequelize also provides a similar interface for filling in the referenced data:

const post = await BlogPost.findOne({
  where: {
    author: "Ronald McClure",
  },
  include: Comment,
});

Both libraries also have ways to lazy-load related objects, only fetching them as necessary. For MongoDB, this involves writing your own queries on referenced fields or using MongoDB’s $lookup operator, while Sequelize allows custom queries using the foreign key fields, as well as a set of nice methods to load associations on demand.

Modeling and Migrating Data

Having weighed for this situation the pros and cons of PostgreSQL and MongoDB as well as Mongoose and Sequelize, the question then comes: How do we actually move the data from one system to another? Unfortunately, as far as I know there is no one tool that can be used to migrate data between MongoDB and PostgreSQL. However, using a combination of mongoexport and some basic SQL files, it was not too difficult to migrate data.

The first step and perhaps one of the more difficult ones is to create an SQL schema that works for your data. If your MongoDB data is already in a normalized and relational structure, it might not be too hard, but if you have a lot of subdocuments and arrays that need to be expanded into rows, things might be more difficult. My recommendation for how to approach this is:

Model your data in SQL. If you’re doing a migration from MongoDB, you likely have an idea of what the application needs are and how data should be organized.
Test your model with some dummy data. This will probably be a bit of a back-and-forth iterative process as you iron out issues with relations, decide what kind of constraints will be necessary, etc.
Use mongoexport to export your data as appropriate. For many collections, it may be easy to simply export it as CSV and use Postgres’s COPY command to import it into the appropriate table. However, many things won’t be so simple and might require writing your own script that reads the exported CSV or JSON and writes it to the database, altering data as needed. This might be necessary for example to change MongoDB’s object IDs to match a datatype in PostgreSQL, such as an identity column (defined in CREATE TABLE), serial, or UUID.
Use your set of import commands and scripts to import all of your data into a test database.
Modify your application to use Sequelize instead of Mongoose.
Find and update all cases in your code where a Mongoose model or instance was used and update it to use Sequelize instead.

Caveats and Gotchas

One of the biggest problems we came across after our migration was the differences in query syntax between Mongoose and Sequelize. Compare the following queries:

// Mongoose
post = await BlogPost.findOne({ author: 'Ronald McClure' });

// Sequelize
post = await BlogPost.findOne({ author: 'Ronald McClure' });

They look the same, they both run with no errors, and they both return the one post we created. But say we change which author we are querying for:

// Mongoose
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

// Sequelize
post = await BlogPost.findOne({ author: 'Darlene Roberts' });

Mongoose’s query returns nothing, as we expect, but Sequelize… returns the same post as the “Ronald McClure” query returned? Astute readers might notice what the issue is from earlier examples: Sequelize’s find methods take an options object which doesn’t use its fields as WHERE parameters by default. The correct way to write this query with Sequelize is:

post = await BlogPost.findOne({ where: { author: "Darlene Roberts" } });

It’s a bit of a silly mistake, but one that caused pain several times in the process of migration. Some queries seemed to work just fine, but just wouldn’t return the right results!

Building a search suggestions feature with Node.js and Vue

2021-11-27T00:00:00+00:00

Photo by Kasper Rasmussen on Unsplash

The backstory

Some time ago, I worked on a project to improve the usability of a search component for our clients. Similar to Google and other search interfaces, the user was presented with a number of suggested search terms as they typed into the search box. We wanted to add keyboard support and give the component a visual facelift. When the customer used the up, down, Esc, or Enter or Return keys, the component would allow them to choose a particular search term, clear their search, or navigate to the results for their chosen search term.

This is what the new and improved UI looked like:

As developers, it can sometimes feel like we’re stuck when working on older, well established projects. We gaze longingly at newer and shinier tools. Part of my objective while building this feature was to prove the viability of a newer approach (Node.js and Vue) to the other engineers on the project as well as the client.

The feature existed already but we wanted to improve the UX and performance. Having added several Vue-powered features to this site in the past, I was very comfortable with the idea and have written about that previously. It would also be very easy to roll back if needed since this project was limited in scope, and very easy to compare the new solution with the code it replaced.

Picking a route and configuring it

This project was running on Interchange, nginx, and MySQL, but our approach would work in other stacks (e.g. with Apache). One key concept is that we’re using nginx as a reverse proxy to route requests to our various apps, serve static files, etc. Using nginx in this way allows us to stitch together the different services for a given project. In this case I created an endpoint for the search suggestions endpoint in our nginx config file. This enabled requests made to /suggestions/ to be passed along to our Node.js app. You can read more about reverse proxying with nginx if you like.

# Node.js powered endpoint for search suggestions
location /suggestions/ {
    proxy_pass http://0.0.0.0:8741/;
}

If your project is running on Apache or another platform there will likely be a similar configuration option to route the requests to your Node.js app as I did.

Node.js time

I wrote a little app with the MySQL driver for Node.js and Express. The app connects to MySQL and uses the nifty built-in connection pooling. It accepts POST requests from the site and responds with an array of search suggestion objects in JSON format.

Enter Vue

For the front-end part of the feature I created Vue components for the search input and for the display of the results. As the customer types, results are fetched and displayed. Using the arrow keys navigates up and down and Esc clears out the search. Once the customer has the search they want they can either press Enter or click on the Search button.

It’s worth mentioning that this search feature works without JavaScript. When you enter a search term and press Enter or click the Search button, you will get results. However, if you do have JavaScript enabled (and if our suggestions app is up and running) you’ll get suggestions as you type. This is a good thing.

Performance wins

The new endpoint returns results in less than 100ms — a 3x or 4x improvement over the Perl script it replaced. We get a response faster, meaning the experience is much more smooth for the user!

Managing the Node.js process

We used pm2 and systemd to manage the Node.js processes and ensure they are started up when the server is rebooted. In my experience this has been very stable and has not required any babysitting by our operations folks.

Great success!

Have you done anything similar in your projects? Do you have vintage/legacy app that could benefit from learning some new tricks? Let us know!

An Introduction to TypeScript

2021-02-09T00:00:00+00:00

TypeScript is a programming language defined as a superset of JavaScript. It adds static type information to JavaScript code using type annotations. These annotations permit strong type-checking at compile-time, kind of like a very strict linter. They are only used for static analysis. TypeScript is transpiled to JavaScript for execution in the browser or Node.js and the type annotations are stripped out. It is still possible to use standard JavaScript type information at run-time, such as that obtained using the typeof and instanceof operators.

Adding type annotations brings many benefits. Most importantly, TypeScript can tell us when we’re doing something dumb that’s likely to cause a type-related bug. But more than that, it powers editors’/IDEs’ context sensitive tool tips so when you hover or start typing the editor can supply helpful information so you can get your job done quicker. This is particularly useful to new developers as it saves them the trouble of reading all the sources to figure out the variable types from context, especially when debugging.

JavaScript is a fairly dynamic language but types still exist in JavaScript, whether we like it or not. Because it’s so dynamic it assumes that you the programmer know the type of every object you are using on every line of code and will do nothing to help you get it right. The type is specified by the context. The trouble is that in any non-trivial codebase it becomes impossible to be absolutely 100% certain about the type of some particular object that’s being passed around without reading the context, which is often a ton of code.

By eagerly annotating our JavaScript code with types we can eliminate entire classes of bugs from our codebase. TypeScript will never let you play loosey-goosey with integers and strings, for example. If you want a variable to be able to hold an int or a string you must explicitly declare it as type “integer” or “string”.

TypeScript lowers the total cost of ownership of a codebase by saving time. Programmers no longer have to guess or infer types and spend less time reading documentation and fixing bugs. Because fewer classes of errors are possible, less testing is required. Entire classes of serious problems never make it to the users because developers never commit them to the codebase. New developers on-board faster and make fewer mistakes. When multiple developers collaborate, their intentions with respect to types are immediately known to all, including the TS compiler, without reading any docs.

TypeScript is open source software maintained by Microsoft: It is free to use, and the specification and tools are provided under the Apache open source license. Microsoft has cultivated a strong developer community with high levels of engagement.

Strictness

Note that TypeScript has several settings for controlling the strictness of its type checking. To obtain maximum benefit we recommend using the --strict option which turns on all strict-checking options. This blog post assumes that strict mode is enabled. For more information see the TypeScript Compiler Options.

Declaring Variables and Functions

let someBool: boolean;

TypeScript will complain if you try to do this:

someBool = "hello world!";  // this is a TypeScript error; someBool
                            // cannot hold a string.

Another way to declare types is to assign directly to constants:

const someBool = true;

TypeScript knows through inference that someBool is of type boolean because it’s a const assigned from a boolean type expression. Later TypeScript will complain if you try to assign a non-boolean typed value to someBool.

Functions

Function declaration is similar to variable declaration:

function someFunc(someBool: boolean): string {
  return "Hello, world!";
}

This useless function takes one boolean type argument and returns a string type value:

const someFunc = (someBool: boolean): string => "Hello, world!";

TypeScript requires parentheses around annotated lambda arguments, even for a single argument.

Assertions

TypeScript supports “type assertions”, which is a way of telling the compiler that you are very certain a particular value is of a particular type and that it should assume you are right. These are sometimes called “casts” in other languages.

Generally speaking, type assertions are a red flag. There’s usually a better, more strongly-type way to write the code that avoids their usage. But sometimes they are convenient, particularly when interfacing with untyped JavaScript APIs. Of course it would be better to declare types for those APIs, but this is beyond the scope of some projects.

Type assertions use the as keyword:

let someBool: boolean;
someBool as unknown as string = "hello world";  // Not an error (but probably a mistake)

Because this is case is so egregious, and likely to be a programmer error, TS forces you to double cast it. A single as is usually sufficient.

Interfaces

Ok, now the real fun begins. So far we’ve only looked at primitive types like string and boolean. What about objects, arrays, and all that?

TypeScript “interfaces” allow us to declare the properties and behaviors that objects should have. There are required properties, optional properties, etc.

Interface declaration looks somewhat like object usage:

interface myInterface {
  someBool: boolean;
  someString: string;
}

Now when we use it, both someBool and someString are required. Hence, the following attempt to assign an empty object to a const of type myInterface is a TypeScript error:

const myObj: myInterface = {};  // TypeScript error!

The object we assign must have both required properties:

const myObj: myInterface = {
  someBool: true,
  someString: "Hello"
};  // This works.

Optional fields are annotated using the ? character:

interface myInterface {
  thisIsRequired: boolean;
  thisIsOptional?: string;
}

const myObj: myInterface = {
  thisIsRequired: true
};

Union Types

Sometimes you want to say that “this thing is of this type… or that type.” Enter “union types”:

type boolOrString = boolean | string;

let myVar: boolOrString;

myVar = true;
myVar = 'hello!';

This is 100% valid. myVar can hold either a boolean or a string value. However, TypeScript will force you to disambiguate which type it currently holds before you can do anything unsafe with it. For example, if we have a function that takes a union type:

function upperBool(arg: boolean | string): string {
  if (typeof arg == 'string') {
    return arg.toUpperCase();
  }
  else if (typeof arg == 'boolean') {
    return arg.toString().toUpperCase();
  }
}

It looks like our function could fall through and return undefined, and theoretically in JavaScript it could, but any such usage would trigger an error in TypeScript (assuming you are using --strictNullChecks, part of the --strict group of checks we recommend). TypeScript “knows” that we’ve handled both possible types of arg and so the if/else definitely does not fall through.

Intersection Types

An intersection type combines two or more interfaces into a single compound interface:

interface This {
  aProperty: string;
}

interface That {
  bProperty: boolean;
}

type Both = This & That;

const both: Both = {
  aProperty: 'funny',
  bProperty: true
};

Classes

TS has classes. I won’t get deep into TS classes, but I want to mention them. They can transpile to ES6 classes or ES2015 prototypal code. They are very powerful. They are, however, optional, as classes are still somewhat “new” to many JavaScript developers.

Enums

TypeScript supports enum declarations. An enum is a type that can only hold one of a specific set of possible values.

Generics

Generics are a mechanism for declaring “complex” types like arrays and promises. Creating new generics is somewhat advanced, but a basic understanding is required to use TS effectively.

const arrayOfStrings: string[];
const anotherArrayOfStrings: Array<string>;

Both of these syntaxes are equivalent. They both declare “an array of strings”.

TypeScript includes a significant number of built-in generic types and you will need them to use the language effectively.

let promiseMeAString: Promise<string>;

This declares a promise that is expected to resolve to a string type value.

Indexable Types

Indexable types allow the developer to specify the key and value types expected when using the square-bracket operator [] to access object or array properties. This is appropriate when the key set is large or arbitrary and not known in advance. In this case it’s not possible to define each possible key value.

This interface describes an object that holds a collection of string type keys with boolean values:

interface stringKeysBoolVals {
  [key: string]: boolean
}

The syntax is a little funny in that it includes a “placeholder” field, which I’ve set to key. This field is essentially a comment. TS ignores it. You can set it to anything that makes sense, e.g. index.

This constructs an object acting as a collection of arbitrary key/value pairs using the interface we just declared:

const myCollection: stringKeysBoolVals = {
  "value1": true,
  "value2": false,
  "value3": true
};

This usage is invalid because neither the key type of int nor the value-type of string is compatible with the declared interface:

const invalidCollection: stringKeysBoolVals = {
  0: "foobar"
};

A Note about Tooling

To get the most out of TypeScript it’s critical that your editor integrates with it and supports real-time type checking and error highlighting. Yes, you can run the TS compiler and check for errors at the command line, but this is a much slower development process.

Also, most editors will be able to use TypeScript annotations to provide context-sensitive help and tell you about types and declarations, which is really helpful. Modern APIs are too complicated to memorize; don’t try.

Conclusion

I hope this helps bring to light some of the wonderful features of TypeScript that help us write more reliable, more maintainable, and more readable code.

The official TypeScript website is a great place to learn more.

Testing to defend against nginx add_header surprises

2020-05-29T00:00:00+00:00

These days when hosting websites it is common to configure the web server to send several HTTP response headers with every single request for security purposes.

For example, using the nginx web server we may add these directives to our http configuration scope to apply to everything served, or to specific server configuration scopes to apply only to particular websites we serve:

add_header Strict-Transport-Security max-age=2592000 always;
add_header X-Content-Type-Options    nosniff         always;

(See HTTP Strict Transport Security and X-Content-Type-Options at MDN for details about these two particular headers.)

The surprise (problem)

Once upon a time I ran into a case where nginx usually added the expected HTTP response headers, but later appeared to be inconsistent and sometimes did not. This is distressing!

Troubleshooting leads to the (re-)discovery that add_header directives are not always additive throughout the configuration as one would expect, and as every other server I can think of typically does.

If you define your add_header directives in the http block and then use an add_header directive in a server block, those from the http block will disappear.

If you define some add_header directives in the server block and then add another add_header directive in a location block, those from the http and/or server blocks will disappear.

This is even the case in an if block.

In the nginx add_header documentation we find the reason for the behavior explained:

There could be several add_header directives. These directives are inherited from the previous level if and only if there are no add_header directives defined on the current level.

This nginx directive has always behaved this way. Various people have warned about it in blog posts and online discussions for many years. But the situation remains the same, a trap for the unwary.

I have tried to imagine the rationale behind this behavior. Response headers often are set in groups, so the programmer who created this feature may have decided that any new scope’s add_header directives should start with a clean slate, unaffected by those set elsewhere. Hmm. The need for exclusive grouping of response headers is rare in my experience, and adding headers to the existing stack of tentative response headers is far more commonly what I want.

So while this behavior may make sense somewhere, it has not ever done so for me or anyone I have talked to about it. For us it is simply misbehavior, silent and easy to overlook when making later seemingly unrelated configuration adjustments.

Dangers

It often has security implications when headers you thought were being added to every response are not. Consider more fine-tuned and consequential security-related headers such as Content-Security-Policy, Vary for cache object separation, CORS headers Access-Control-*, etc.

Headers such as these are especially important when they need to be added based on logic spread across various configuration blocks, and that is exactly when nginx add_headers doesn’t work as expected.

Another pitfall is omitting the always option to add_header. Without that, the header will only be added to success responses (2XX and 3XX, but see the docs for specifics). We usually want security-related headers to be added even to 4XX and 5XX error responses.

Workaround using include

My first instinct was to work around the problems caused by this behavior by putting the standard add_header list in a file that I include everywhere. In some cases that works.

But despite the nginx include documentation saying that directive is allowed in “Context: any”, include is not allowed in an if block and will result in the fatal startup error:

“include” directive is not allowed here

So the only recourse in those cases is to repeat all needed add_header directives in every if block that uses add_header. Gross.

Repeating configuration manually means almost surely having the add_header directives in different configuration areas drift over time. So if we have to repeat ourselves, at least let’s do it with automation, such as by using configuration templating and preprocessing.

That is what I have most recently done. And we can still use native nginx include directives everywhere those are allowed.

nginx Headers More module

Many people have run into exactly this problem, and some of them developed a separate nginx module ngx_headers_more to solve most of these problems.

By using its more_set_headers directive, you get the expected additive behavior with previously-declared headers, regardless of the block scope:

Directives inherited from an upper level scope (say, http block or server blocks) are executed before the directives in the location block.

Note that although more_set_headers is allowed in location if blocks, it is not allowed in the server if blocks …

Fortunately I have not needed to use this in an if block in the server scope, so that one remaining limitation doesn’t pose a problem for me.

It also has options to set a header only for responses of a certain HTTP content type or status code.

The more_clear_headers directive allows the * wildcard for clearing all headers with the same prefix at once, such as Access-Control-*.

Installing ngx_headers_more

Because “Headers More” is a separate module, not part of standard nginx, it is not usually available without some extra work.

You can build it from source and install it manually, but of course that isn’t good to do on a production machine since it won’t get updated on its own.

You can use the OpenResty server built around nginx, which “Headers More” is part of. But you may not want all of that if you’re not writing a Lua web application.

Many Linux distributions and 3rd-party package repositories have prebuilt packages for “Headers More” which you can use:

Alpine
- nginx-mod-http-headers-more
Debian & Ubuntu
- nginx-extras
- libnginx-mod-http-headers-more-filter
RHEL/CentOS
- GetPageSpeed & Webtatic repos nginx-module-headers-more
- Aeris repo nginx-more

Search the excellent pkgs.org to find what you need if it isn’t already available through your package manager.

Apache

Apache httpd is still alive and well — actually better than ever. So depending on your situation, you may want to use that instead.

Apache’s Header directive has intuitive (to me) default behavior for setting response headers across the whole configuration, and many ways to deal with a possibly already-existing header:

add another header, or set exclusively (replace), or set only if this header doesn’t already exist
append to or merge into an existing header (for headers that accept multiple values)
edit an existing header with a regular expression search-and-replace
unset a header if one was previously set

I don’t know a way to have Apache clear a group of headers with a wildcard, or all headers at once, so they need to be individually cleared by name if that’s what you want.

Доверяй, но проверяй (Trust, but verify)

nginx was written by Igor Sysoev. Despite my disagreement with this one feature’s behavior, overall I find that nginx is excellent. Because of its open source release, excellent performance, and wide use, it has provided much-needed competition to Apache and Microsoft IIS. Thank you, Igor and all other contributors!

In the relevant spirit, since Igor is Russian, I close with the Russian proverb Доверяй, но проверяй: Trust, but verify.

Let us code (and configure) defensively, yet also test to avoid being surprised by missing headers.

We can manually test various HTTP responses are as we expect using curl -v or other HTTP clients to exercise various requests.

Even better, we can add to our automated test suite to confirm these HTTP response headers appear everywhere we expect, for static files and API endpoints backed by different application servers, and for various success and error responses.

Here is a test adapted from one I put together for one of our clients. It uses JavaScript in Node.js, the Jest test framework, and the Axios HTTP client. It ensures the security headers example I showed at the beginning of this article keeps working, even as we make nginx configuration changes over time:

const axios = require('axios');

const http = axios.create({
  baseURL: 'https://your.dom.ain',
});

describe('Check security headers', () => {
  const verifs = [
    { header: 'strict-transport-security', expect: (x) => x.toMatch(/max-age=\d{3,}/) },
    { header: 'x-content-type-options',    expect: (x) => x.toEqual('nosniff')        },
  ];

  const locs = [
    { path: '/robots.txt',                status: 200 },  // static
    { path: '/feed/endpoint/of/interest', status: 200 },  // API backend in PHP
    { path: '/api/other/auth/endpoint',   status: 403 },  // API backend in Perl
    { path: '/never/gonna/give/you/up!',  status: 404 },
    { path: '/api/dies/for/testing',      status: 500 },
  ];

  // throw no exceptions for non-success HTTP response status
  const conf = { validateStatus: () => true };

  for (const l of locs) {
    test(`${l.status} ${l.path}`, async () => {
      const res = await http.get(l.path, conf);
      expect(res.status).toBe(l.status);
      for (const v of verifs) {
        v.expect(expect(res.headers[v.header]));
      }
    });
  }
});

Here I run just this one test rather than the whole suite:

% jest -w 6 ./__tests__/webserver/security-headers.test.js
Determining test suites to run...
testing on https://https://your.dom.ain

 PASS  webserver/security-headers.test.js
  Check security headers
    ✓ 200 /robots.txt (55ms)
    ✓ 200 /feed/endpoint/of/interest (408ms)
    ✓ 403 /api/other/auth/endpoint (18ms)
    ✓ 404 /never/gonna/give/you/up! (6ms)
    ✓ 500 /api/dies/for/testing (12ms)

Test Suites: 1 passed, 1 total
Tests:       5 passed, 5 total
Snapshots:   0 total
Time:        2.721s, estimated 3s
Ran all test suites matching /.\/__tests__\/webserver\/security-headers.test.js/i.

This can also be extended to ensure that certain headers do not exist, or do not contain details that you do not want exposed:

the Server header should not reveal the nginx (see server_tokens) or Apache (see ServerTokens) version numbers
the X-Powered-By header should be absent, not exposing the fact that you are using PHP, and the version number — see the expose_php directive for php.ini
or with the Java Wildfly server, both of those headers are sent by default! — see instructions on how to omit them by editing XML or using jboss-cli

Add to the verifs array in the code above:

    { header: 'server',                    expect: (x) => x.not.toMatch(/\d/)         },
    { header: 'x-powered-by',              expect: (x) => x.toBeUndefined()           },

Now if (when) I forget about the nginx add_headers behavior, make changes, and inadvertently break things? Instead of it being unnoticed, my test suite will alert me so I can fix it before it goes into production!

Salesforce Integration with Node.js

2020-03-27T00:00:00+00:00

Photo by Dylan Wooters, 2020

Salesforce is huge. It is currently the dominant customer relationship management (CRM) provider, accounting for around 20% of market share. Businesses are using Salesforce not only as a traditional CRM solution, but also for novel purposes. Salesforce can serve as a backend database and admin portal for custom apps, or as a reporting tool that pulls data from various systems.

This growth leads to increasing demand for Salesforce integrations. The term “Salesforce integration” may conjure up images of expensive enterprise software or dense API documentation, but it doesn’t have to be that way. You can work with Salesforce easily using Node.js and the npm package JSforce. An example of a project that might benefit from this kind of Node.js integration is an e-commerce website where order data is loaded to and from Salesforce for order fulfillment, tracking, and reporting.

In this post we’ll cover how to connect to Salesforce using JSforce, the basics of reading and writing data, as well as some advanced topics like working with large amounts of data and streaming data with Socket.IO.

Setting Up

You’ll first want to install Node.js on your local machine, if you haven’t done so already.

Next, create your Node app. This will vary with your requirements. I often use Express to build a REST API for integration purposes. Other times, if I am routinely loading data into Salesforce, I will create Node scripts and schedule them using cron. For the purposes of this post, we will create a small Node script that can be run on the command line.

Create a new directory for your project, and within that directory, run npm init to generate your package.json file. Then install JSforce with npm install jsforce.

Finally, create a file named script.js, which we will run on the command line for testing. To test the script at any time, simply navigate to your app’s directory and run node script.js.

At the top of the script, require jsforce, as well as the Node IO libraries fs and path. Then define an asynchronous function that will serve as our script body. This is where all of your Salesforce code will go.

var jsforce = require('jsforce');
var fs = require('fs');
var path = require('path');

run();
async function run() {
    // salesforce code goes here...
}

Connecting to Salesforce

I usually store my Salesforce credentials and instance URL as a JSON object in a separate file, which I gitignore. This ensures that sensitive data does not appear in Git. Below is the content of my salesforce-creds.json file. You’ll want to add your Salesforce username and password and update the instance URL, if necessary.

{
    "username": [your username],
    "password": [your password],
    "url": "https://na111.salesforce.com"
}

To connect to Salesforce simply retrieve the credentials from the file and use them with the JSforce Connection class to login. Be sure to wrap all JSforce code in a try-catch block, to catch any errors coming back from Salesforce.

let creds = JSON.parse(fs.readFileSync(path.resolve(__dirname, './salesforce-creds.json')).toString());
let conn = new jsforce.Connection({
    loginUrl: creds.url
});
try {
    await conn.login(creds.username, creds.password);
    console.log('Connected to Salesforce!');
    // now you can use conn to read/write data...
    await conn.logout();
} catch (err) {
    console.error(err);
}

Reading, Writing, and Deleting Data

Once connected, the easiest way to query data from Salesforce is to use the JSforce query function, and pass in an SOQL statement. This offers the most flexibility, as you can run queries for child and parent objects. Using SOQL, we can query all accounts and their contacts (children) in a single statement. Note, however, that there are limitations on relationship queries. You can only go down one level, from parent to child, but you can go up multiple levels from child to parent.

Writing and deleting data is simple with JSforce using the sobject class and the corresponding create/update/delete function. In the example below, we will query for accounts and contacts using SOQL, and then isolate and update a specific contact using sobject().update.

let soql = `select id, name,
    (SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c from Contacts)
    FROM Account`;
let accounts = await conn.query(soql);
let cooper = accounts.records
    .filter(x => x.Name === 'Twin Peaks Sheriff Dept.')[0].Contacts.records
    .filter(y => y.FirstName === 'Dale' && y.LastName === 'Cooper')[0];
console.log(cooper);

// Console output:
// {
//     attributes: {
//         type: 'Contact',
//         url: '/services/data/v42.0/sobjects/Contact/0033h000001sDzDAAU'
//     },
//     Id: '0033h000001sDzDAAU',
//     FirstName: 'Dale',
//     LastName: 'Cooper',
//     Email_Verified__c: true,
//     Enrollment_Status__c: 'Pending'
// }

cooper.Enrollment_Status__c = 'Accepted';
let ret = await conn.sobject('Contact').update(cooper);
if (ret.success) {
    console.log('Contact updated in Salesforce.');
}

Working with Large Amounts of Data

You may need to read and write large amounts of data, for example if you are using Salesforce for reporting and loading data to and from other systems.

Event-driven Querying

The record limit for standard promise-style SOQL querying, as in our example above, is 2000 records. To query more than that, it is best to shift to the event-driven style of querying. This will ensure that all records are successfully retrieved from Salesforce. You can use the maxFetch property to set the upper limit of records returned. By default, maxFetch is set to 10,000.

let contacts = [];
let soql = 'SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c from Contact';
let query = await conn.query(soql)
    .on("record", (record) => {
        contacts.push(record);
    })
    .on("end", async () => {
        console.log(`Fetched Contacts. Total records fetched: ${contacts.length}`);
    })
    .on("error", (err) => {
        console.error(err);
    })
    .run({
        autoFetch: true,
        maxFetch: 5000
    });

Loading Data with the Bulk API

Loading a large amount of data into Salesforce is best accomplished through the Bulk API via JSforce. There are a couple good reasons for this approach.

The Bulk API has better performance over other methods when working with large collections of objects.

The standard JSforce sobject create/update/delete functions have a 200 object limit. For operations on large collections, you must divide the total by 200, resulting in many separate API calls. By contrast, the Bulk API only uses a single API call. Since Salesforce imposes API limits, this makes the Bulk API a better choice.

Running a bulk operation is simple using the bulk.load method, which takes three parameters: the Salesforce object type, the operation type, and an array of objects. The method returns an array of objects with success/errors fields, as well as the id of the newly created object, if successful.

If you’re working with thousands of objects, it’s good to set the pollTimeout property manually to one minute or more, to avoid Salesforce connection timeouts. Also note that the possible values for operation type are: ‘insert’, ‘update’, ‘upsert’, ‘delete’, or ‘hardDelete’.

// set poll timeout to one minute for larger datasets
sfConnection.bulk.pollTimeout = 240000;

// normally you will have thousands of Accounts, this is just an example
let accounts = [{
        Name: 'Saul Goodman, LLC'
    },
    {
        Name: 'Los Pollos Hermanos Inc'
    },
    {
        Name: 'Hamlin, Hamlin & McGill'
    }
];
let results = await conn.bulk.load('Account', 'insert', accounts);
console.log(results);

// Console output:
// [
//     { id: '0013h000006bdd2AAA', success: true, errors: [] },
//     { id: '0013h000006bdd3AAA', success: true, errors: [] },
//     { id: '0013h000006bdd4AAA', success: true, errors: [] }
// ]

if (accounts.length === results.filter(x => x.success).length) {
    console.log('All account successfully loaded.');
}

WebSocket Streaming with Socket.io

Say you are building a web application for reporting, and the app contains a dashboard with data on all of your contacts in Salesforce. You want the dashboard to be updated whenever the data in Salesforce changes, and you also want this to happen without refreshing the web page.

To accomplish this, you can stream real-time data from Salesforce using JSforce and the Socket.IO library, which makes working with WebSockets quite simple.

The first step in this process is creating a PushTopic in Salesforce. This is basically a trigger that emits a notification anytime an object is created, updated, etc. in Salesforce. I created a PushTopic for Contacts by running the following Apex code in the Salesforce developer console.

PushTopic pushTopic = new PushTopic();
pushTopic.Name = 'UserChange';
pushTopic.Query = 'SELECT Id, FirstName, LastName, Email_Verified__c, Enrollment_Status__c FROM Contact';
pushTopic.ApiVersion = 48.0;
pushTopic.NotifyForOperationCreate = true;
pushTopic.NotifyForOperationUpdate = true;
pushTopic.NotifyForOperationUndelete = true;
pushTopic.NotifyForOperationDelete = true;
pushTopic.NotifyForFields = 'Referenced';
insert pushTopic;

Then, back in your Node app, install Express and Socket.IO.

Next, you’ll want to create a very basic Express server that will listen for updates from the Salesforce PushTopic, and emit them to your reporting site. Start by installing Express and Socket.IO.

npm install express
npm install socket.io

Then delete the run function in your script.js file, which contained the code from the samples above, and replace it with the following:

async function run() {
    // listen with express
    server.listen(3000, function() {
        console.log('listening on *:3000');
    });

    // connect to Salesforce
    let creds = JSON.parse(fs.readFileSync(path.resolve(__dirname, './salesforce-creds.json')).toString());
    let conn = new jsforce.Connection({
        loginUrl: creds.url
    });
    try {
        await conn.login(creds.username, creds.password);
    } catch (err) {
        console.error(err);
    }

    // when the client connects, emit streaming updates from salesforce to client
    io.on("connection", (socket) => {
        console.log('A socket connection was made!');
        let eventHandler = (message) => {
            console.log('New streaming event received from Salesforce:', message);
            socket.emit('UserChange', message);
        };
        conn.streaming.topic('UserChange').subscribe(eventHandler);
    });
}

Here is a step-by-step description of what is occurring in the code sample above:

The Express server is set to listen for connections on port 3000.
We connect to Salesforce and login.
Socket.IO is set to listen for incoming connections from clients.
A function called eventHandler that emits Salesforce streaming messages to the client is defined.
When a connection is made, eventHandler is attached to the Salesforce streaming topic as a callback, using the live Salesforce connection.

If you follow the nice little tutorial from Socket.IO and create the sample chat webpage, you can actually test the Salesforce streaming updates. In the chat page, add this script, which will log messages coming back from Salesforce.

<script>
    var socket = io();
    socket.on('UserChange', function(msg) {
        console.log(msg);
    });
</script>

Then update a contact in Salesforce, changing the contact’s first name. If everything works correctly, you should see the client connect via Socket.IO in the Node logs, and also see a streaming message from Salesforce logged in the browser’s console window.

Summary

Node.js and JSforce provide a straightforward and elegant way to interact with Salesforce. Whether you have an existing Node API that needs to work with Salesforce, or you are building a new application that is powered by Salesforce data, consider the recipes above as stepping stones for completing your project.

GraphQL Server Libraries

2019-07-12T00:00:00+00:00

Photo by Jon Flobrant on Unsplash

This post is a followup to my previous post, GraphQL — An Alternative to REST. Please check that out for an introduction to GraphQL and what makes it different from other API solutions. I’ve collected a list of some of the currently-maintained GraphQL libraries for a few different languages, along with some examples (most of which aren’t fully functional on their own, they’d need more configuration) so you can see what it might be like to use GraphQL in your project. I’ll be focusing on the ways each of these libraries implement GraphQL and what you’d need to do to start a project with each of them, so if you have questions about GraphQL itself, please check out my other blog post.

Apollo Server (JavaScript/TypeScript)

Apollo GraphQL has libraries for both a GraphQL server and client (which I’ll discuss later). Apollo Server can be used both as a standalone server as well as with libraries like Express. Apollo Server is the server library I have the most experience with—I wrote a server last year using Express and Apollo Server, along with a client that used Apollo Client. I’m a fan of the flexibiliy of Apollo, but it takes more work to set up than some of the alternatives.

Setting up Apollo Server as a standalone can be done fairly simply following the directions on their website. However, I’m going to go over the basics of integrating with Express. There are two main parts to writing a server with Apollo: your GraphQL schema and your resolvers. These stay more or less the same whether you’re using Apollo as a standalone server or combining it with Express. You do have to have a database set up separately; I’ll show examples with MongoDB, but you could easily swap it out with PostgreSQL or another database. I’ll show an example resolver along with its GraphQL schema for a blog post. The schema follows the GraphQL schema rules and might look like the following:

const typeDefs = [gql`
  type Post {
    id: String!
    body: String!
  }

  query {
    post(id: String!): Post
  }
`];

Now for the resolver. Resolvers are functions that take information from the query (like arguments) and return the relevant data, usually from a database. For our blog post, a resolver might look like this:

const resolvers = {
  post: (root, args, context, info) => {
    return Post.findById(args.id);
  }
};

Simple! We just get the data from the database and return it—as long as the property names match those of our schema, Apollo will automatically format it according to the frontend’s request and return it to them.

OK, next we create the server:

import express from 'express';
import { ApolloServer } from 'apollo-server-express';

const PORT = 3000;

const app = express();
const server = ApolloServer({
  typeDefs,
  resolvers
});

server.applyMiddleware(app);

app.listen(PORT, () => {
  console.log(
    `Server running at http://localhost:${PORT}/graphql`
  );
});

And that’s it! Note that these examples are missing a few things like imports, and we didn’t add authentication of any kind, but this is the general format for creating a server with Apollo.

Prisma

Prisma is a cool library developed by the same people as Graph.cool that does much of the work for you in enabling GraphQL access to your database.

Prisma offers configuration for existing databases, but unfortunately I had trouble getting it to work on my Ubuntu system—I ran into issues getting the Docker container to connect to my local Postgres and MongoDB databases. However, following the quick guide found here on the Prisma website, I was able to get a GraphQL server up and running inside a Docker container with a new database. The process was simple:

First, you have to install the Prisma command line utility:

npm install -g prisma

You also need to have Docker installed. Documentation for Docker can be found here.

Next, you need to configure Prisma. Create a directory for your Prisma server, and create a new file named docker-compose.yml:

mkdir hello-world
cd hello-world
touch docker-compose.yml

Then, paste the following into it:

version: '3'
services:
  prisma:
    image: prismagraphql/prisma:1.34
    restart: always
    ports:
      - '4466:4466'
    environment:
      PRISMA_CONFIG: |
        port: 4466
        databases:
          default:
            connector: mongo
            uri: mongodb://prisma:prisma@mongo
  mongo:
    image: mongo:3.6
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: prisma
      MONGO_INITDB_ROOT_PASSWORD: prisma
    ports:
      - '27017:27017'
    volumes:
      - mongo:/var/lib/mongo
volumes:
  mongo: ~

I used Mongo here, but Prisma’s site has guides for PostgreSQL and MySQL as well. It’s important to make sure now that you don’t have any conflicts with currently running databases—on my machine I had already had a MongoDB server running on port 27017. I fixed this by just stopping my local MongoDB server, but I’m sure you could configure the Docker containers to work with different ports as well. Running Ubuntu, I just ran sudo service mongodb stop and then the Prisma Docker containers worked just fine. When I was done, I ran sudo service mongodb start to start it up again.

Next, you’ll start the Prisma containers and initialize the Prisma server configuration:

docker-compose up -d
prisma init --endpoint http://localhost:4466

The final step is to deploy the service:

prisma deploy

If all goes well, you’ll see a message that includes a URL to the Prisma Admin, which is a browser tool to interact with your GraphQL endpoints. I used it for a little while when I was testing Prisma out, and it seems to work well and is easy to use.

All in all, Prisma seems like a great way to start if you don’t want to handle the messy details of setup. However, I did have issues getting it to play nice with my already-existing databases (including both PostgreSQL and MongoDB). However, it is still relatively new, so I would expect support to get better over time.

Graphene (Python)

Graphene is a GraphQL framework for Python. It has integrations for a few different server frameworks (a list can be found here), but I’ll show examples from graphene-django, since Django is fairly common and something that we use fairly often at End Point.

Because you’re also setting up a Django project, the tutorial for graphene-django is a little more involved, so I’ll just share the relevant GraphQL sections so you can compare to the other libraries in this post. The most important part, the schema, is defined in Python with a similar format to Django models:

import graphene

from graphene_django.types import DjangoObjectType

from app.models import Category, Ingredient

class CategoryType(DjangoObjectType):
    class Meta:
        model = Category

class IngredientType(DjangoObjectType):
    class Meta:
        model = Ingredient

class Query(object):
    all_categories = graphene.List(CategoryType)
    all_ingredients = graphene.List(IngredientType)

    def resolve_all_categories(self, info, **kwargs):
        return Category.objects.all()

    def resolve_all_ingredients(self, info, **kwargs):
        # We can easily optimize query count in the resolve method
        return Ingredient.objects.select_related('category').all()

As you can see, the format for defining your GraphQL schema is quite different from some other libraries, but you have the advantage of it looking similar to Django’s model definitions. You’ll also need a higher-level Query definition:

import graphene

import cookbook.ingredients.schema


class Query(cookbook.ingredients.schema.Query, graphene.ObjectType):
    # This class will inherit from multiple Queries
    # as we begin to add more apps to our project
    pass

schema = graphene.Schema(query=Query)

Now that we have a schema defined, we need to add a few things to settings.py:

INSTALLED_APPS = [
    ...
    # This will also make the `graphql_schema` management command available
    'graphene_django',
]

GRAPHENE = {
    'SCHEMA': 'cookbook.schema.schema'
}

The last piece needed to use your GraphQL schema is in urls.py:

from graphene_django.views import GraphQLView

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^graphql$', GraphQLView.as_view(graphiql=True)),
]

And finally we run the server:

$ python manage.py runserver

Now you should be able to use your GraphQL schema at http://localhost:8000/graphql just like with any other GraphQL server.

Graphene for Django seems like a good solution in that it uses a similar format to other aspects of Django, like the model definitions. However, its format (especially for schema definition) is rather different-looking from the GraphQL standard used by most other libraries, and it seems like it might make it more work to keep your frontend and backend in sync.

Graph.cool

I won’t discuss Graph.cool in detail here, because I went over it in my previous blog post. However, it still merits mention here as an option for your GraphQL server. Essentially, Graph.cool lets you define a GraphQL schema and then handles the work of setting up a database and even hosting for you. If you just want to get a basic GraphQL server set up for testing, or if you don’t need too many features beyond data storage and retrieval, Graph.cool is a great choice.

Additional links

For server libraries in other languages, these seem like good options:

Thanks for reading! Keep an eye out next week for a second post which will cover GraphQL client libraries.

GraphQL — An Alternative to REST

2019-05-11T00:00:00+00:00

GraphQL has become more and more popular recently as an alternative to traditional RESTful APIs since it was released as open source by Facebook in 2015. According to the GraphQL website, it is “a query language for APIs and a runtime for fulfilling those queries with your existing data”. In this blog post, I’ll go over some of what makes GraphQL different from other API solutions, and then show how to get a GraphQL API up and running so you can try it out yourself!

GraphQL is designed to fit on top of your database layer. With the help of libraries like Apollo GraphQL, it can be used with many different databases. Some of the main differences between GraphQL and more traditional RESTful APIs include:

GraphQL uses one endpoint. Most traditional APIs use an endpoint for each type of data; in my example, you’d probably have one each for users (/user), posts (/post) and comments (/comment). Each of these would return some JSON with the data you want. GraphQL, on the other hand, lives at one endpoint (usually /graphql) and changes what it returns based on what you ask for, as detailed in the next point.
You can get multiple types in one request. For instance, if you want to get information about an author plus all of their posts, instead of making a request for the author and a request for posts, you do just one request for the author and specify that you’d like their posts as well:

query {
  user(id: "12345") {
    name
    posts {
      title
      body
    }
  }
}

You decide which parts of the data you want. Traditional REST APIs give you data based on which endpoint you’re querying (/post/:id, /user/:id, etc.), and the format of the data is generally the same. For instance, no matter which id you ask for at /posts/:id, you’ll always get something that looks like this back:

{
  "id":"123",
  "name":"Smash Mouth",
  "joined":"1994"
}

But what if we don’t need to know when they joined right now? Another example that better illustrates this problem (and GraphQL’s solution) is getting a list of blog posts. The home page of a blog usually shows several of the most recent posts, but you can also view a list of just post names that will go farther back. With a traditional REST API, you would have to create separate endpoints for each of these scenarios, for example /first_posts and /posts, or add GET parameters or similar. With GraphQL, you can just specify exactly what you want. The query to replace /first_posts might look like this:

query {
  posts {
    title
    body
    author {
      name
    }
  }
}

The data returned might look like:

{
  "data": {
    "posts": [
      {
        "title": "All Star",
        "body": "Somebody once told me the world is gonna roll me\n I ain't the sharpest tool in the shed\n She was looking kind of dumb with her finger and her thumb\n In the shape of an \"L\" on her forehead\n Well the years start coming and they don't stop coming\n Fed to the rules and I hit the ground running\n Didn't make sense not to live for fun\n Your brain gets smart but your head gets dumb\n So much to do, so much to see\n So what's wrong with taking the back streets?\n You'll never know if you don't go\n You'll never shine if you don't glow\n Hey now, you're an all-star, get your game on, go play\n Hey now, you're a rock star, get the show on, get paid\n And all that glitters is gold\n Only shooting stars break the mold\n Only shooting stars break the mold",
        "author": {
          "name": "Smash Mouth"
        }
      },
      {
        "title": "Less-interesting Stuff",
        "body": "Sed ut perspiciatis, unde omnis iste natus error sit voluptatem\n accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore\n veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam\n voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur\n magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui\n dolorem ipsum, quia dolor sit amet consectetur adipisci[ng] velit, sed quia non-numquam\n [do] eius modi tempora inci[di]dunt, ut labore et dolore magnam aliquam quaerat\n voluptatem. Ut enim ad minima veniam, quis nostrum[d] exercitationem ullam corporis\n suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure\n reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel\n illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?",
        "author": {
          "name": "Some Latin-speaking Guy"
        }
      }
    ]
  }
}

That’s great for /first_posts, but we probably don’t need to bloat the response with the entire post body if we’re just using the names for a list, right? Let’s try again, but this time we’ll remove body from the request:

query {
  posts {
    title
    author {
      name
    }
  }
}

Now we just get the title and author, exactly like we want. Notice also that these queries are pulling in data from a related model, the User model, via the author property.

{
  "data": {
    "posts": [
      {
        "title": "All Star",
        "author": {
          "name": "Smash Mouth"
        }
      },
      {
        "title": "Less-interesting Stuff",
        "author": {
          "name": "Some Latin-speaking Guy"
        }
      }
    ]
  }
}

Although you can get similar functionality by using GET parameters or creating extra endpoints, with GraphQL, you can just make that one simple change without needing to write extra code.

Try it!

OK, so there are some of the things that sets GraphQL apart. However, reading about it is only so helpful—now I’ll show how to get a GraphQL backend up and running quickly using Graph.cool, a Node.js backend service that does most of the work of setting up a server, including setting up a database based on your schema. (Graphcool was retired in mid-2020, so you should instead check out their more flexible server option called Prisma.)

First, you need to install graphcool by running:

npm install -g graphcool

You’ll also need to sign up for a (free) account, but you can do that later. Once graphcool is installed, create a directory to use for your code and initialize it.

$ mkdir gql-blog
$ cd gql-blog
$ graphcool init

Open the file types.graphql and uncomment the lines describing a model called Post, as well as the related line in User; aftewards, it should look something like this:

type User @model {
  # id is of type ID, must be not null (!), and must be unique
  id: ID! @isUnique
  name: String
  dateOfBirth: DateTime

  # posts contains an array of Post objects
  posts: [Post!]! @relation(name: "UserPosts")
}


type Post @model {
  id: ID! @isUnique
  title: String!

  # Graphcool relations have to be defined both ways so the service knows
  # whether it's a one-to-many, many-to-many, or many-to-one relation
  author: User! @relation(name: "UserPosts")
}

Note that there are a few extra descriptors here that aren’t part of the core GraphQL schema; @isUnique, @relation, @model, and ID are all part of the Graphcool service’s additions. However, they are allowed within GraphQL’s spec and provide some extra functionality to the API. GraphQL’s website has more information about the type system.

Now that you have a basic schema, it’s time to run your server. In the directory where you ran graphcool init, run graphcool deploy. First, it’ll open a browser window and ask you to log in or create an account. Once you do that, you should see a prompt asking you which server cluster you want to deploy to. Pick any of them (but note that local requires a bit more setup before working). Push enter on the next few prompts to use the default options and you’ll get a big wall of text telling you that your server is up and running! To try it out, find the line near the end that looks like this:

Here are your GraphQL Endpoints:

  Simple API:        https://api.graph.cool/simple/v1/bbd987478jjhhds902k2l

Copy and paste that URL into a browser and you’ll have access to the GraphQL Playground, where you can run queries and mutations (like queries, but for creating/updating data). You need some data to play with, so copy the following into the Playground one at a time and run them to create a User and a Post. Note that you’ll need to look at the ID returned by the first mutation and paste it into the second mutation so that the two are linked.

mutation {
  createUser(
    name: "SmashMouth"
  ) {
    id
    name
  }
}

mutation {
  createPost(
    title: "All Star",
    authorId: "USER_ID_HERE"
  ) {
    id
    title
  }
}

Once those are finished, you can run queries to get the data you want. Here’s an example of a query to return the author you created earlier plus any posts associated with it:

query {
  User(id: "USER_ID_HERE") {
    id
    posts {
      id
    }
  }
}

You’ll notice that this query only returns the ID of the user and their posts. That’s because we only asked for the ID—if you want to get the name of the user and the title of the post, you’ll have to add that to the query:

query {
  User(id: "USER_ID_HERE") {
    id
    name
    posts {
      id
      title
    }
  }
}

Pretty cool, huh? You get exactly what you ask for. Next, try getting your post (you’ll need its ID) and asking for its author along with it..

On the far right of the GraphQL Playground, there’s a tab labeled Schema that shows the entire schema of your app. If you look in there, you’ll see that there are far more types defined than were in the types.graphql file we edited earlier—this is because graphcool automatically generates them for you. That’s the main reason why it’s easier to use than other GraphQL libraries. In particular, you might try getting all posts or all users.

If you want to keep messing around with GraphQL, I’d suggest trying to add your own new Comment type to types.graphql, emulating the way User and Post are related to each other. Keep in mind that you’ll have to run graphcool deploy and reload your browser to see your changes show up in the Playground.

For more information and thorough documentation, see the GraphQL website.

If you want to try GraphQL without installing anything, visit graphql.nodaljs.com which lets you try it on their own server.

Extensible Binary Encoding with CBOR

2019-03-18T00:00:00+00:00

CBOR is a relatively new IETF draft standard extensible binary data format. Compared to similar formats like MessagePack and BSON, CBOR was developed from the ground up with clear goals:

unambiguous encoding of most common data formats from Internet standards

code compactness for encoder and decoder

no schema description needed

reasonably compact serialization

applicability to constrained and unconstrained applications

good JSON conversion

extensibility

—RFC 7049 Appendix E, Copyright © 2013 IETF Trust, Bormann & Hoffman

In the context of data storage and messaging, most developers can relate to CBOR as a binary drop-in replacement for JSON. While CBOR doesn’t share the human readability of JSON, it can efficiently and unambiguously encode types of data that JSON struggles with. CBOR can also be extended with tags to optimize serialization beyond its standard primitives.

Encoding Binary Data

JSON is a ubiquitous data format for web and beyond, for many good reasons, but encoding blobs of binary data is an area where JSON falters. For example, if you are designing a JSON protocol to wrap the storage or transfer of arbitrary objects, your options are:

Require that all input data can be represented as JSON. When possible this is potentially a reasonable solution, but limits the types of data that can be encoded. Notable exceptions include most popular image encodings, excluding SVG.
Base64 encode any binary data values to a string. This can encode any binary data, but increases the size of the data by a minimum of 1/3, incurs encoding and decoding cost, and requires magic to indicate that the string is Base64 encoded.
Encode the bytes as an array of numbers or a hex string. These are probably not things you should do, but it seemed worth mentioning that these techniques increase the size of the data by anywhere from 2x to 5x and also require magic to indicate that the data is really binary.

With CBOR, binary blobs of any length are supported out of the box and are encoded 1:1. Encoding and decoding CBOR byte strings is extremely fast, even in higher-level languages. In the case of JavaScript, CBOR byte strings can be transcoded to and from the very fast and efficient Uint8Array.

For example, let’s try encoding a simple object with two fields: “name” and “data”. An abstract view of this object:

name: "Strawberry Pie"
data: <00 01 02 03 04 05 06 07 08 09>

A C struct of this raw data would be absolute minimum of 24 bytes.

If you Base64 the data into JSON, your output might look like this:

{"name":"Strawberry Pie","data":"AAECAwQFBgcICQ=="}

Length: 51 bytes.

This illustrates the “magic” problem with Base64 encoding binary data in JSON. Unless you have a JSON schema or special protocol fields, the decoder has no indication that the data in this object needs to be Base64 decoded, because it looks like a string. It is not self-describing.

The CBOR representation of the same input data, as a hex string:

a2646e616d656e5374726177626572727920506965696a7065675f646174614a00010203040506070809

Whoa, there. What do all these numbers and letters mean? I miss my JSON! The Node cbor package has a handy cbor2comment tool to annotate this hex string for us.

  a2                -- Map, 2 pairs
    64              -- String, length: 4
      6e616d65      -- {Key:0}, "name"
    6e              -- String, length: 14
      5374726177626572727920506965 -- {Val:0}, "Strawberry Pie"
    64              -- String, length: 4
      64617461 -- {Key:1}, "data"
    4a              -- Bytes, length: 10
      00010203040506070809 -- {Val:1}, 00010203040506070809

Length: 35 bytes.

Now let’s benchmark JSON against CBOR using real JPEG data. For this test I used a modified version of cbor-js, a library compatible with both Node and browsers, and encoded a 910,226 byte JPEG of strawberry rhubarb pie.

|                           | JSON      | CBOR    |
| :------------------------ | --------: | ------: |
| Median encoding time (ms) | 3.983     | 0.538   |
| Median decoding time (ms) | 3.151     | 0.006   |
| Encoded size (bytes)      | 1,213,676 | 910,262 |

As the numbers show, CBOR is both faster and more concise for this particular data. Also, CBOR pie tastes better.

Food always looks good in pictures!

Optimizing CBOR with Tags

In the case of encoding homogeneous numeric arrays, CBOR encoders can struggle with optimizing the packing of the data. For example, if you have an array of floating point numbers in a higher-level language like Python or JavaScript, the CBOR encoder implementation won’t necessarily determine how many bits are required to encode the numbers, defaulting to the largest available. Additionally, each value in the array will be individually described as a floating point number. This increases the cost and size of the data considerably.

CBOR has an answer to this problem. The Draft Typed Array Tags spec includes tags specifying typed arrays which happen to match JavaScript TypedArray flavors.

For example, let’s say you have a Float32Array with a few values:

const floats = new Float32Array([1.234567, 2.345678, 3.456789]);

JSON has a really funny way of encoding a Float32Array, lookit all those bytes:

{"0":1.2345670461654663,"1":2.3456780910491943,"2":3.456789016723633}

Length: 69 bytes.

If we were to help the encoder by sending it a regular Array it would still be pretty verbose, but the precision we weren’t using is truncated, so the overall length will vary wildly depending on the values in the Array:

[1.234567,2.345678,3.456789]

Length: 28 bytes.

The well-meaning Node cbor library can encode the Float32Array directly, but doesn’t try to optimize for size:

  83                -- Array, 3 items
    fb              -- Float, next 8 bytes
      3ff3c0c960000000 -- [0], 1.2345670461654663
    fb              -- Float, next 8 bytes
      4002c3f2e0000000 -- [1], 2.3456780910491943
    fb              -- Float, next 8 bytes
      400ba78100000000 -- [2], 3.456789016723633

Length: 28 bytes.

Look at all that wasted precision, 64 bits for each 32-bit float, this just won’t do! The CBOR spec allows you to “tag” data for special treatment. According to the list of registered CBOR tags, “IEEE 754 binary32, little endian, Typed Array” is tag 85. The consecutive bytes of the three numbers follow.

  d8                --  next 1 byte
    55              -- Tag #85
      4c            -- Bytes, length: 12
        4b069e3f971f1640083c5d40 -- 4b069e3f971f1640083c5d40

Length: 15 bytes.

The contained 12 byte string is equivalent to the values in the ArrayBuffer underneath our Float32Array.

The CBOR decoder will spit out the tag alongside its associated blob, so our optimization is self-described. Now we need to ensure that a byte string with this tag is correctly converted to the typed array, minding endianness. In a mature JavaScript implementation, this is both easy and very fast. Because we encoded the values of the ArrayBuffer underneath a Float32Array, we can construct a new Float32Array from the ArrayBuffer.

const floats = new Float32Array(bytes);  // assuming platform and bytes are same endianness!

This is a very scalable way to read and write arrays of numeric values into JavaScript.

When to Use CBOR

As King Crimson teaches us, “it doesn’t mean you should / just because you can.”

I’ve found CBOR useful as an alternative to JSON for large (>2kB) chunks of raw data or numeric arrays. In many other cases JSON is equivalent or superior, because most languages have native JSON encoding and decoding in their standard libraries. CBOR encoders are not always so optimized, but the gap is closing with wider adoption. Run your own tests and benchmarks on real data against real libraries before deciding to use CBOR.

Many readers will recognize that schemaful formats such as protocol buffers are an endgame for structured data. If the infrastructure demands make sense for your application, this is also a good way to go. If you’re working with an application that already uses JSON, the difference in development and maintenance costs of porting it to CBOR or protobuf-likes should be measured against the size and performance gains of each approach.

It’s tempting to let numbers alone decide what format to use, but don’t underestimate the value of JSON’s human readability for structured data. The value of human readability diminishes quickly when dealing with large numeric arrays or binary blobs, so once again CBOR is an appealing choice for these data.

img.bi, a secret encrypted image sharing service tool

2015-07-30T00:00:00+00:00

After a fairly good experience with dnote installed on our own servers as an encrypted notes sharing service, my team decided that it would have been nice to have a similar service for images.

We found a nice project called img.bi that is based on NodeJS, Python, Redis and a lot of client-side JavaScript.

The system is divided into two components: the HTML/JS frontend and a Python FastCGI API.

Unfortunately the documentation is a still in its very early stage and it’s lacking a meaningful structure and a lot of needed information.

Here’s an overview of the steps we followed to setup img.bi on our own server behind nginx.

First of all we chose that we wanted to have as much as possible running and confined to a regular user, which is always a good idea with such young and potentially vulnerable tools. We chose to use the imgbi user.

Then since we wanted to keep as clean as possible the root user environment (and system status), we also decided to use pyenv. To be conservative we chose the latest Python 2.7 stable release, 2.7.10.

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo $SHELL -l
pyenv install -l  | grep 2\\.7
pyenv install 2.7.10
pyenv global 2.7.10
pyenv version
which python
python --version

In order to use img.bi, we also needed NodeJS and following the same approach we chose to use nvm and install the latest NodeJS stable version:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.25.4/install.sh | bash
nvm install stable
nvm list
nvm use stable
nvm alias default stable
node --version

As a short note to the usage of the bad practice of blindly using:

curl -o- https://some_obscure_link_or_not | bash

We want to add that we do not endorse this practice as it’s dangerous and exposes your system to many security risks. On the other hand, though, it’s true that cloning the source via Git and compile/installing it blindly is not much safer, so it’s always up to how much you trust the peer review on the project you’re about to use. And at least with an https URL you should be talking to the destination you want, whereas an http URL is much more dangerous.

Furthermore going through the entire Python and NodeJS installation as a regular user, was far beyond the scope of this post and the steps proposed here assumes that you’re doing everything as the regular user, except where specifically stated differently.

Anyway after that we updated pip and then installed all the needed Python modules:

pip install --upgrade pip
pip install redis m2crypto web.py bcrypt pysha3 zbase62 pyutil flup

Then it’s time to clone the actual img.bi code from the GitHub repo, install a few missing dependencies and then use the bower and npm .json files to add the desired packages:

git clone https://github.com/imgbi/img.bi.git
cd img.bi/
npm install -g bower grunt grunt-cli grunt-multiresize
npm install -g grunt-webfont --save-dev
npm install
bower install

We also faced an issue which made Grunt fail to start correctly. Grunt was complaining about an “undefined property” called “prototype”. If you happen to have the same problem just type

cd node_modules/grunt-connect-proxy/node_modules/http-proxy
npm install eventemitter3@0.1.6
cd -

That’ll basically install the eventemitter3 NodeJS package module locally to the grunt-connect-proxy module so to overcome the compatibility issues which in turn causes the error mentioned above.

You should use your favourite editor to change the file config.json, which basically contains all your local needed configuration. In particular our host is not exposed on the I2P or Tor network, so we “visually” disabled those options.

# lines with "+" needs to be replace the ones starting with a "-"
-  "name": "img.bi",
+  "name": "img.bi - End Point image sharing service",

-  "maxSize": "3145728",
+  "maxSize": "32145728",

-  "clearnet": "https://img.bi",
+  "clearnet": "https://imgbi.example",

-  "i2p": "http://imgbi.i2p",
+  "i2p": "http://NOTAVAILABLE.i2p",

-  "tor": "http://imgbifwwqoixh7te.onion",
+  "tor": "http://NOTAVAILABLE.onion",

Save and close the file. At this point you should be able to run “grunt” to build the project but if it fails on the multiresize task, just run

grunt --force

to ignore the warnings.

That’s about everything you need for the frontend part, so it’s now time to take care of the API.

cd
git clone https://github.com/imgbi/img.bi-api.git
cd /home/imgbi/img.bi-api/

You now need to edit the two Python files which are the core of the API.

# edit code.py expired.py
-upload_dir = '/home/img.bi/img.bi-files'
+upload_dir = '/home/imgbi/img.bi-files'

Verify that you’re not having any Python import related error, due to missing modules or else, by running the Python code.py file directly.

./code.py

If that’s working okay, just create a symlink in the build directory in order to have the API created files available to the frontend

ln -s /home/imgbi/img.bi-files /home/imgbi/img.bi/build/download

And then it’s time to spawn the actual Python daemon:

spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234

The expired.py file is used by a cronjob which periodically checks if there’s any image/content that should be removed because its time has expired. First of all let’s call the script directly and if there’s no error, let’s create the crontab:

python /home/imgbi/img.bi-api/expired.py

crontab -e

@reboot spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234
30 4 * * * python /home/imgbi/img.bi-api/expired.py

It’s now time to install nginx and Redis (if you still haven’t done so), and then configure them. For Redis you can just follow the usual simple, basic installation and that’ll be just okay. Same is true for nginx but we’ll add our configuration/vhost file content here as an example /etc/nginx/sites-enabled/imgbi.example.conf for everyone who may need it:

upstream imgbi-fastcgi {
  server 127.0.0.1:1234;
}

server {
  listen 80;
  listen [::]:80;
  server_name imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;
  rewrite ^ https://imgbi.example/ permanent;
}

server {
  listen 443 ssl spdy;
  listen [::]:443 ssl spdy;
  server_name  imgbi.example;
  server_name  imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;

  client_max_body_size 4G;

  include include/ssl-wildcard-example.inc;

  add_header Strict-Transport-Security max-age=31536000;
  add_header X-Frame-Options SAMEORIGIN;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";

  location / {
    root /home/imgbi/img.bi/build;
  }

  location /api {
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;

    fastcgi_param SCRIPT_NAME "";
    fastcgi_param PATH_INFO $uri;
    fastcgi_param REQUEST_URI $request_uri;
    fastcgi_param DOCUMENT_URI $document_uri;
    fastcgi_param DOCUMENT_ROOT $document_root;
    fastcgi_param SERVER_PROTOCOL $server_protocol;

    fastcgi_param GATEWAY_INTERFACE CGI/1.1;
    fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

    fastcgi_param REMOTE_ADDR $remote_addr;
    fastcgi_param REMOTE_PORT $remote_port;
    fastcgi_param SERVER_ADDR $server_addr;
    fastcgi_param SERVER_PORT $server_port;
    fastcgi_param SERVER_NAME $server_name;
    fastcgi_param HTTPS on;

    fastcgi_pass imgbi-fastcgi;
    fastcgi_keep_conn on;
  }
}

Well, that should be enough to get you started and at least have all the components in place. Enjoy your secure image sharing now.

Web Development, Big Data and DevOps—OSI Days 2014, India

2015-01-12T00:00:00+00:00

This is the second part of an article about the conference Open Source India, 2014 was held at Bengaluru, India. The first part is available here. The second day of the conference started with the same excitement level. I plan to attend talks covering Web, Big Data, Logs monitoring and Docker.

Web Personalisation

Jacob Singh started the first talk session with a wonderful presentation along with real-world cases which explained the importance of personalisation in the web. It extended to content personalisation for users and A/B testing (comparing two versions of a webpage to see which one performs better). The demo used the Acquia Lift personalisation module for the Drupal CMS which is developed by his team.

MEAN Stack

Sateesh Kavuri of Yodlee spoke about the MEAN stack which is a web development stack equivalent to popular LAMP stack. MEAN provides a flexible compatibility to web and mobile applications. He explained the architecture of MEAN stack.

He also provided an overview of each component involved in MEAN Stack.

MongoDB — NoSQL database with dynamic schema, in-built aggregation, mapreduce, JSON style document, auto-sharding, extensive query mechanism and high availability.

ExpressJS — A node.js framework to provide features to web and mobile applications.

AngularJS — seamless bi-directional model with extensive features like services and directives.

Node.js — A server side JavaScript framework with event based programming and single threaded (non blocking I/O with help of request queue).

Sails.js — MEAN Stack provisioner to develop applications quickly.

Finally he demonstrated a MEAN Stack demo application provisioned with help of Sails.js.

Moving fast with high performance Hack and PHP

Dushyant Min spoke about the way Facebook optimised the PHP code base to deliver better performance when they supposed to handle a massive growth of users. Earlier there were compilers HipHop for PHP (HPHPc) or HPHPi (developer mode) to convert the php code to C++ binary and executed to provide the response. After sometime, Facebook developed a new compilation engine called HipHop Virtual Machine which uses Just-In-Time (JIT) compilation approach and converts the code to HipHop ByteCode (HHBC). Both Facebook’s production and development environment code runs over HHVM.

Facebook also created a new language called Hack which is very similar to PHP which added static typing and many other new features. The main reason for Hack is to get the fastest development cycle to add new features and release frequent versions. Hack also uses the HHVM engine.

HHVM engine supports both PHP and Hack, also it provides better performance compare to Zend engine. So Zend Engine can be replaced with HHVM without any issues in the existing PHP applications to get much better performance. It is simple as below:

Also PHP code can be migrated to Hack by changing the <?php tag to <?hh and there are some converters (hackficator) available for code migration. Both PHP and Hack provide almost the same performance on the HHVM engine, but Hack has some additional developer-focussed features.

Application Monitoring and Log Management

Abhishek Dwivedi spoke about a stack to process the logs with various formats, myriad timestamp and no context. He explains a stack of tools to process the logs, store and visualize in a elegant way.

ELK Stack = Elasticsearch, LogStash, Kibana. The architecture and the data flow of ELK stack is stated below:

Elasticsearch — Open source full text search and analytics engine

Log Stash — Open source tool for managing events and logs which has following steps to process the logs

Kibana — seamlessly works with Elasticsearch and provides elegant user interface with various types of graphs

Apache Spark

Prajod and Namitha presented the overview of Apache Spark which is a real time data processing system. It can work on top of Hadoop Distributed FileSystem (HDFS). Apache Spark performs 100x faster in memory and 10x faster in disk compare to Hadoop. It fits with Streaming and Interactive scale of Big Data processing.

Apache Spark has certain features in processing the data to deliver the promising performance:

Multistep Directed Acyclic Graph
Cached Intermediate Data
Resilient Distributed Data
Spark Streaming — Adjust batch time to get the near real time data process
Implementation of Lambda architecture
Graphx and Mlib libraries play an important role

Online Data Processing in Twitter

Lohit Vijayarenu from Twitter spoke about the technologies used at Twitter and their contributions to Open Source. Also he explained the higher level architecture and technologies used in the Twitter microblogging social media platform.

The Twitter front end is the main data input for the system. The Facebook-developed Scribe log servers gather the data from the Twitter front end application and transfers the data to both batch and real time Big Data processing systems. Storm is a real time data processing system which takes care of the happening events at the site. Hadoop is a batch processing system which runs over historical data and generates result data to perform analysis. Several high level abstraction tools like PIG are used write the MR jobs. Along with these frameworks and tools at the high level architecture, there are plenty of Open Source tools used in Twitter. Lohit also strongly mentioned that in addition to using Open Source tools, Twitter contributes back to Open Source.

Docker

Neependra Khare from Red Hat given a talk and demo on Docker which was very interactive session. The gist of Docker is to build, ship and run any application anywhere. It provides good performance and resource utilization compared to the traditional VM model. It uses the Linux core feature called containerization. The container storage is ephemeral, so the important data can be stored in persistent external storage volumes. Slides can be found here.

Liquid Galaxy and its Very Own Street View App

2013-11-09T00:00:00+00:00

Liquid Galaxy does Street View!

Peruse-a-Rue is the combination of a Node.js server with a Maps API browser client, all wrapped up in one neat bundle. The result of this marriage is a highly compelling immersive Street View experience.

Everything from a single screen kiosk to a cylindrical Liquid Galaxy to an endless display wall can be configured, with bezel offsets, portrait or landscape. A touchscreen control interface is optional, and a Space Navigator can drive the display.

Testing Peruse-a-Rue on the desktop

By leveraging the Connect framework for Node, the entire application is served on a single port. Any number of browser windows can be synchronized, thanks to the scalability of websockets. When integrated with the Squid caching of the Liquid Galaxy project, redundant downloading is eliminated; each screen shares retrieved tile data with its peers.

The Peruse-a-Rue touchscreen interface

Since NPM installs dependencies automatically, deployment is a breeze. Every Liquid Galaxy is a git checkout and an npm install away from running the server. Peruse-a-Rue supports any operating system that can run Node.js (as a server) or Google Chrome (as a client). I’ve even tested the server on a Raspberry Pi and BeagleBone Black, and it runs perfectly!

It runs on this thing, too

Peruse-a-Rue is hosted here. If you’re interested in the project or want to contribute, drop us a line.

Happy Perusing!

A Preact Web App Without npm Build

The Scenario

The Main Issues

JSX

Imports

The Solution: Browser Import Maps

Conclusion

How to write end-to-end & component tests with Cypress in Vue.js

How to install and set up Cypress

How to create a simple to-do app with Vue 3

How to write end-to-end tests

How to write component tests

Component tests for BaseTextInput.vue

Component tests for TodoListItem.vue

Next steps

Migrating a Node.js app database from MongoDB to PostgreSQL

MongoDB vs. PostgreSQL

Mongoose and Sequelize

Modeling and Migrating Data

Caveats and Gotchas

Building a search suggestions feature with Node.js and Vue

The backstory

Picking a route and configuring it

Node.js time

Enter Vue

Performance wins

Managing the Node.js process

Great success!

An Introduction to TypeScript

Strictness

Declaring Variables and Functions

Functions

Assertions

Interfaces

Union Types

Intersection Types

Classes

Enums

Generics

Indexable Types

A Note about Tooling

Conclusion

Testing to defend against nginx add_header surprises

The surprise (problem)

Dangers

Workaround using include

nginx Headers More module

Installing ngx_headers_more

Apache

Доверяй, но проверяй (Trust, but verify)

Salesforce Integration with Node.js

Setting Up

Connecting to Salesforce

Reading, Writing, and Deleting Data

Working with Large Amounts of Data

Event-driven Querying

Loading Data with the Bulk API

WebSocket Streaming with Socket.io

Summary

GraphQL Server Libraries

Apollo Server (JavaScript/TypeScript)

Prisma

Graphene (Python)

Graph.cool

Additional links

GraphQL — An Alternative to REST

Try it!

Extensible Binary Encoding with CBOR

Encoding Binary Data

Optimizing CBOR with Tags

When to Use CBOR

img.bi, a secret encrypted image sharing service tool

Web Development, Big Data and DevOps—​OSI Days 2014, India

Web Personalisation

MEAN Stack

Moving fast with high performance Hack and PHP

Application Monitoring and Log Management

Apache Spark

Online Data Processing in Twitter

Docker

Web Development, Big Data and DevOps—OSI Days 2014, India