Writing a WebSocket-Controlled State Machine

2024-07-20T00:00:00+00:00

This article was co-authored by Jacob Minshall

We recently developed a state machine to control a piece of software for a client. The client wanted to have an API to interact with the state machine, triggering state changes while it was running. Depending on the current state’s requirements, the state machine could either wait for a WebSocket message to proceed to another state or transition to the next state without outside input. WebSockets allow for two way communication so the clients can also have visibility into the state machine’s current state.

To start, we looked for a simple way to implement a state machine within our TypeScript/Node.js based project. The typescript-fsm library on GitHub was a good solution for us. What made us consider this package was the simplicity of the library: the entire source file is around 100 lines of code.

We did end up making some custom changes to the library that won’t be shown here; for example, we wanted to broadcast state change messages via WebSockets to any connected clients. With such a simple library, it was a breeze adding that code. The code in this post will still run with the vanilla library, it just won’t notify you when the state changes.

Here’s a simple example of a state machine that manages a car’s door (borrowed from the typescript-fsm README, with some modifications). It can be opened or closed, but it can also be in the middle of opening and closing as well. If the door is closed, you can’t close it again. If it’s opening, then trying to close it will have to wait until the door has finished opening.

As a side note, state machines lend themselves well to diagrams. These help developers and clients see and understand the code at a high level. PlantUML has worked well for us. Our door diagram is below:

Let’s break down the door state machine first, then the WebSocket control after.

// Use our customized version of typescript-fsm
import { t, StateMachine } from "./stateMachine";

enum States { closing = 0, closed, opening, opened, broken };
enum Events { open = 100, openComplete, close, closeComplete, break };

We start by creating the basic states and events. These are each just enums that will be used when checking the current state and moving from one state to the next.

// lets define the transitions that will govern the state-machine
const transitions = [
  /* fromState        event                 toState         callback */
  t(States.closed,    Events.open,          States.opening, onOpen),
  t(States.opening,   Events.openComplete,  States.opened,  justLog),
  t(States.opened,    Events.close,         States.closing, onClose),
  t(States.closing,   Events.closeComplete, States.closed,  justLog),
];

Here the transitions are defined. This is a list of tuples that define the state machine. First is the starting state where an event can occur. Next is the events that will cause a state transition to happen, and then the state that will be transitioned to. Finally there’s the callback that is called as part of the transition. It’s not shown here, but one state may have multiple transitions, for instance if you wanted the closed state could have an event called “lock” that moves to the state locked, that way while in the closed state you the door would respond to either the “open” event or the “lock” event.

// initialize the state machine
const door = new StateMachine<States, Events>(
   States.closed,   // initial state
   transitions,     // array of transitions 
);

This is how the state machine itself is created. The enums are passed to the machine so it knows what states and events there are, then the transitions and starting state are supplied to the constructor, and the new state machine is returned. Multiple doors can be created; each door object would have a “dispatch” method attached to it to send the events that are happening to that specific door.

// transition callbacks - async functions
async function onOpen() {
    console.log("onOpen...");
    // actually open the door
    
    // waiting 5 seconds and then pretending the door has been opened
    setTimeout(() => {door.dispatch(Events.openComplete);}, 5000);
}

async function onClose() {
    console.log("onClose...");
    // actually close the door
    
    // waiting 5 seconds and then pretending the door has been closed
    setTimeout(() => {door.dispatch(Events.closeComplete);}, 5000);
}

// synchronous callbacks are also ok
function justLog() { 
    console.log(`new state is: ${States[door.getState()]}`);
}

These are basic callbacks to use as an example. When opening or closing the door, the script will wait 5 seconds and signal a completion of either event, changing the state automatically. This means some of the state transitions are internal to the state machine rather than external. Once the door has begun opening, the user is no longer in control of when it has finished opening, they just have to wait for it to be in the opened state. A door will never start closing on its own with this implementation, so some outside source will need to initiate the closing/opening process.

Make sure you have the ws package installed via NPM: npm install ws. You also need the @types/ws package for WebSocket type definitions: npm i --save-dev @types/ws.

import WebSocket from 'ws';

console.log("creating websocket server");
let ws = new WebSocket.Server({port: 9091})

ws.on("connection", (socket: WebSocket) => {
    console.log("you are connected to the websocket");

    // handle a message coming in on this websocket
    socket.on("message", async (message: string) => {

This creates the WebSocket, puts it on port 9091, and then starts to define what to do when it gets a message.

        // message is JSON
        let msg = JSON.parse(message);

        // if there's an event inside the message handle it
        if (msg.event) {
            let event: number = msg.event;

            // check if the state can happen according to transitions available
            if (door.can(event)) {
                // dispatch that event to kick off the state transition
                await door.dispatch(event);
                // return to the websocket the current state
                // if open is sent, opening will be the new state
                // that is returned
                socket.send(JSON.stringify({
                    current_state: door.getState()
                }))

Here the message sent by the client is parsed into JSON. If there’s an event requested, we check if the door can handle that event. This means that when the door is in the opened state and the open event is sent we won’t try to dispatch that event. The dispatch call is the way the state machine requests a state change. Internally, it also checks to see if the event is possible.

            } else {
                console.log(`error, bad event sent ${event}`)
                socket.send(JSON.stringify({
                    error: `bad event sent ${event}`,
                    current_state: door.getState()
                }))
            }
        } else if (msg.current_state) {
            socket.send(JSON.stringify({
                current_state: door.getState()
            }))
        }
    });
});

Another option in the API is a simple check of the current state. If the message sent has a field that has the key current_state then the current state is returned on the socket.

The car door example is a simple use case; the state machine that we wrote for our client has 15 different states and 36 different transitions. Adding and changing states has proven fairly easy and the states themselves provide nice ways to encapsulate fairly complicated behaviors. This encapsulation has eased the cognitive load required to work on the codebase, so we will definitely keep this in our tool bag for future projects.

Building Ecommerce Search Using Algolia

2023-10-12T00:00:00+00:00

A common request that developers receive when embarking on a new website project is for the website to have “Google-like search.” For many years, this meant writing custom code to replicate the intelligent and user-friendly aspects of Google search, which was no easy feat. However, now we have many search-as-a-service offerings that do the hard work for us and make this process much easier.

In this blog post, we’ll dive into one of these search-as-a-service platforms, Algolia. We recently worked on an ecommerce website and used Algolia in an interesting way, both as a search engine and as a lightweight backend database to hold product data managed in Salesforce. Algolia worked beautifully, offering users fast and accurate search results, and also allowing us to launch the site within a relatively short time frame.

We will look at how to load Algolia with data, configure search options, and connect the search to the frontend using Algolia’s Vue library.

Loading the index with data

To start using Algolia’s search, you need to load up an index with data. You have the option of manually uploading a JSON file, or using Algolia’s API to programmatically load records. For our backend, we chose to use Algolia’s JavaScript API client in some lightweight TypeScript scripts that are triggered by cron. These scripts allowed us to sync inventory data between Salesforce and the index in Algolia.

Using the Algolia JavaScript client is quite simple. Regardless of where your data comes from—be it in a database, a platform like Salesforce, or elsewhere—once it is in JSON format, you can load it into Algolia with a few lines of code:

import * as algolia from "algoliasearch";

const products = [{
    name: "Fender F-5 Acoustic",
    make: "Fender",
    model: "F-5",
    category: "Guitars",
    status: "Used",
    objectID: "Fender-001"
  }, {
    name: "Fender Player Jaguar",
    make: "Fender",
    model: "Jaguar (Player)",
    category: "Guitars",
    status: "New",
    objectID: "Fender-002"
  }];


const index = algolia.default(process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_API_KEY).initIndex("store_products");
await index.saveObjects(products);

Note the objectID property, which is used by Algolia as a primary key. If the objectID does not exist, a new record will be created. If it does exist, the record will be updated. This makes it easy to run a data sync process using a single saveObjects command, without having to worry about differentiating between create and update operations.

Configuring the index

Once you have your index loaded, you’ll want to configure it. Algolia does a good job of walking you through this process using a built-in tutorial when you first load your index. Basically, you will be selecting the searchable properties/attributes from your JSON data, setting how results are ranked and sorted, and adjusting more advanced aspects of search like typo tolerance, stop words, etc.

An important feature we utilized on our recent project is faceting. Faceting allows users to easily drill down and refine search by categories, and is also easy to develop using the handy frontend libraries that Algolia provides (more on that in the next section). This feature is powerful and can be used to both refine search and drive homepage category/subcategory links. When you configure your index, you can select which attributes of your data should be used for faceting.

Setting up search on the frontend

We used Nuxt to build the frontend of the website, and we leveraged Algolia’s Vue InstantSearch library for the UI. This library really speeds along development, as it wraps all of the search-related functionality in simple widgets, providing the search bar, results, refinements, filtering, and more.

The ais-instant-search widget is the parent widget. It serves the search state to its children, which allows you to show the search bar, search hits, hierarchical menus, etc. Here is a simple example of the ais-instant-search widget with a search bar and hits (pulled directly from Algolia’s Vue docs):

<template>
  <ais-instant-search :search-client="searchClient" index-name="demo_ecommerce">
    <ais-search-box />
    <ais-hits>
      <template v-slot:item="{ item }">
        <h2>{{ item.name }}</h2>
      </template>
    </ais-hits>
  </ais-instant-search>
</template>


<script>
import algoliasearch from 'algoliasearch/lite';
import 'instantsearch.css/themes/algolia-min.css';


export default {
  data() {
    return {
      searchClient: algoliasearch(
        '[Your app ID]',
        '[Your API key]'
      ),
    };
  },
};
</script>


<style>
body {
  font-family: sans-serif;
  padding: 1em;
}
</style>

I mentioned faceting above when discussing how to configure your index. Once you have selected the attributes in your JSON data that can be used for faceting (e.g., category, subcategory), you can feed those attributes to the ais-hierarchical-menu widget for display on the frontend.

Here is a bit of sample code from the website we built, which offers expandable category refinement via ais-hierarchical-menu.

<div class="sidebar-segment">
  <p class="sidebar-segment-title">Category</p>
  <ais-hierarchical-menu
    :limit="100"
    :attributes="categoryAttrs"
    :sort-by="hierarchicalMenuSort"
  >
    <div slot-scope="{ items, refine, createURL }">
      <hierarchical-menu-list
        :items="items"
        :refine="refine"
        :create-url="createURL"
      />
    </div>
  </ais-hierarchical-menu>
</div>

Above, we are using the limit property to set the maximum number of items to 100. The attributes property, which targets the JSON attributes in your index data that represent your categories, is set to the following:

categoryAttrs: ['categories.lvl0', 'categories.lvl1', 'categories.lvl2'],

These represent the three levels of categories in our data (zero-based), and are used to build the hierarchical menu.

Finally, the sort-by attribute points to a simple function that uses the JavaScript localeCompare method to provide alphanumeric sorting:

function hierarchicalMenuSort(a, b) {
  return a.name.localeCompare(b.name, undefined, {
    numeric: true,
    sensitivity: 'base',
  })
}

Wrapping up: seeing Algolia in action

If you’d like to see Algolia in action on the site that we built, head over to eiffeltrading.com. If you search the site, you’ll see autocomplete, fast results (thanks to both Algolia and Nuxt), and other aspects of good search that we have all come to expect from modern ecommerce sites.

Next time you are faced with a build involving full-text search, consider search-as-a-service offerings like Algolia. They could save you time and headaches over rolling your own search functionality. Sometimes it’s good to let others do the hard work!

Have questions or feedback on the topic? Let us know in the comments section below.

Upgrade Vue to TypeScript

2023-08-28T00:00:00+00:00

It’s important to keep your code up to date so that time can be dedicated to improving an application instead of version-related mishaps. This is especially true for web development as the landscape changes so quickly.

I recently upgraded a Vue project to exclusively use Vuex. This was a great opportunity to also upgrade the project from JavaScript to TypeScript. This article will cover the steps I took.

Some of the changes can be difficult to understand if you are not familiar with TypeScript. I recommend reading the TypeScript Handbook to become more familiar.

Several features of Vue, originally written in JavaScript without types, are hard to convert to TypeScript. These include this.$parent, this.$refs, and this.$emit. These allow you to access the parent component, child components, and emit events respectively. We will make changes to these features along with adding types to the global state handler provided by Vuex.Store.

Installation

Before you begin, make sure the necessary dependencies are installed:

~$ vue add typescript

Also make sure that Vuex is installed:

~$ yarn add vuex@next

Convert your component files

There are several changes that must be made to component files, such as App.vue.

Add lang="ts" to the <script> tag.
Replace the default export with a class that extends Vue and uses the @Component decorator.
Replace props with class properties marked with the @Prop decorator.
Replace data with individual class properties.
Add a typed $refs class property.
Replace computed with getter and setter class properties.
Lifecycle hooks simply become class methods, so they can be copied directly.
Replace methods with individual class methods.
Replace this.$emit with class methods using the @Emit decorator.

Let’s look at each change separately.

Add `lang="ts"` to the `<script>` tag

This is the easiest change. Simply add lang="ts" to the <script> tag.

-<script>
+<script lang="ts">

Replace the default export with a class that extends `Vue` and uses the `Component` decorator

This change can be complicated to visualise, but the @Component decorator provided by vue-class-component makes it easy to convert a component incrementally by accepting existing properties of the default export as arguments to allow backwards compatibility.

First, replace the default export with a class component:

-export default {
-  name: 'App',
-  components: {
-    HelloWorld
-  },
-  props: ['msg'],
-}
+import Component from 'vue-class-component'
+import { Vue } from 'vue-property-decorator'
+
+@Component({
+  components: {
+    HelloWorld
+  },
+  props: ['msg'],
+})
+export default class App extends Vue {}

Note that the name property is no longer necessary. The name of the component is now the name of the class.

Also, to illustrate the capability of the @Component decorator, the props property is passed as an argument. This allows the component to be used as before with no other changes. Next, we will replace props with class properties marked by the @Prop decorator.

Replace `props` with class properties marked by the `@Prop` decorator

The @Prop decorator is used to mark class properties as props. It accepts an optional argument to specify the type of the prop. If no argument is provided, the type is inferred from the default value.

import Component from 'vue-class-component'
import { Prop, Vue } from 'vue-property-decorator'

@Component
export default class App extends Vue {
  @Prop() msg!: string
  @Prop({ type: Number }) count!: number // type can also be specified explicitly
  @Prop(Boolean) disabled!: boolean // shorthand for { type: Boolean }
}

Now, when a prop is referenced in a function or the template, it will be typed correctly.

Replace `data` with individual class properties

The data property is replaced with individual class properties.

export default {
  name: 'App',
  data() {
    return {
      count: 0,
      msg: 'Hello World',
      complex: { /* ... */ },
    },
  },
}

Becomes:

@Component
export default class App extends Vue {
  count = 0
  msg: string = 'Hello World'
  complex: ComplexType = { /* ... */ }
}

As before, types can be specified explicitly or inferred from the default value.

Add a typed `$refs` class property

The $refs property is typed by adding a class property with the same name and type.

@Component
export default class App extends Vue {
  $refs!: {
    input: HTMLInputElement
  }
}

Now, references to $refs.input will be typed correctly.

Replace `computed` with getter and setter class properties

The computed property is replaced with getter and setter class properties.

export default {
  name: 'App',
  computed: {
    count() {
      return this.$store.state.count
    },
    msg() {
      return this.$store.state.msg
    },
  },
}

Becomes:

@Component
export default class App extends Vue {
  get count() {
    return this.$store.state.count
  }
  set count(value: number) {
    this.$store.commit('setCount', value)
  }
  get msg() {
    return this.$store.state.msg
  }
  set msg(value: string) {
    this.$store.commit('setMsg', value)
  }
}

Now the getters and setters can accept and return the correct types.

Lifecycle hooks become class methods

Lifecycle hooks such as created and mounted are represented as class methods.

export default {
  name: 'App',
  created() {
    this.$store.commit('setCount', 0)
    this.$store.commit('setMsg', 'Hello World')
  },
}

Becomes:

@Component
export default class App extends Vue {
  created() {
    this.$store.commit('setCount', 0)
    this.$store.commit('setMsg', 'Hello World')
  }
}

Replace `methods` with individual class methods

The methods property is replaced with individual class methods. Not much about the methods needs to change.

export default {
  name: 'App',
  methods: {
    increment() {
      this.$store.commit('increment')
    },
    decrement() {
      this.$store.commit('decrement')
    },
  },
}

Becomes:

@Component
export default class App extends Vue {
  increment() {
    this.$store.commit('increment')
  }
  decrement() {
    this.$store.commit('decrement')
  }
}

Replace `this.$emit` with class methods using the `@Emit` decorator

The @Emit decorator is used to mark class methods as emitters. It accepts an optional argument to specify the event name. If no argument is provided, the event name is the name of the method.

export default {
  name: 'App',
  methods: {
    setValue(value) {
      this.$emit('set-value', value)
    },
  },
}

Becomes:

@Component
export default class App extends Vue {
  @Emit('set-value') setValue(value: string) {
    return value
  }
}

This would be called from the parent component in the same way:

<template>
  <app @set-value="onSetValue" />
</template>

Now that your components have been converted, let’s look at the store.

Convert your store

State in Vuex is managed using the Vuex.Store object. Below is a basic example:

import Vue from 'vue'
import Vuex from 'vuex'

Vue.use(Vuex)

export default new Vuex.Store({
  state: {
    count: 0,
    msg: 'Hello World',
  },
  mutations: {
    setCount(state, value) {
      state.count = value
    },
    setMsg(state, value) {
      state.msg = value
    },
    increment(state) {
      state.count++
    },
    decrement(state) {
      state.count--
    },
  },
  getters: {
    count: state => state.count,
    msg: state => state.msg,
  },
  actions: {
    increment(context) {
      context.commit('increment')
    },
    decrement(context) {
      context.commit('decrement')
    },
  },
})

Inside a component, access to the store can be typed using the @State, @Mutation, @Getter, and @Action decorators.

import { Component } from 'vue-property-decorator';
import { Mutation, Getter, Action } from 'vuex-class';

@Component
export default class MyComponent extends Vue {
    @State count!: number
    @State msg!: string
    @Mutation setCount!: (value: number) => void
    @Mutation setMsg!: (value: string) => void
    @Mutation increment!: () => void
    @Mutation decrement!: () => void
    @Getter count!: number
    @Getter msg!: string
    @Action increment!: () => void
    @Action decrement!: () => void
}

Parts of the store can also be namespaced using the modules key in the store object:

import Vue from 'vue'
import Vuex from 'vuex'

Vue.use(Vuex)

export default new Vuex.Store({
  modules: { ExampleModule },
});

Where a module is an object with the same structure as the store object. See the Vuex module documentation for more information.

The namespaced state can then be typed and accessed from components:

import { Component } from 'vue-property-decorator';
import { namespace } from 'vuex-class';

const Store = {
  ExampleModule: namespace('ExampleModule'),
};

@Component
export default class ImportExample extends Vue {
  @Store.ExampleModule.Action('import') import!: ({ file, config }: { file: File; config: ExampleConfig }) => Promise<boolean>;
}

Note that this adds types to the store access points within components, but does not fully statically type the store itself. For more information on how to do this, refer to this excellent article: Vuex + TypeScript.

Conclusion

In conclusion, despite the number of steps needed to convert your Vue components and store, the process is relatively simple. The result is a fully typed Vue application that is easier to maintain and refactor. You won’t regret the ability to trace the flow of data through your application. Additionally, currently developed packages for Vue such as Vuetify and Vuelidate are fully typed and will work seamlessly with your new Vue application.

Type guards in TypeScript

2022-10-12T00:00:00+00:00

TypeScript has a very rich type system. Sometimes too rich. For example, interfaces and types perform the same function with slightly different syntax.

This post is a short intro to a powerful feature in TypeScript: type guards. This feature also exists in many strongly typed languages. While TypeScript is not strongly typed, it replicates this excellent feature.

The feature

Normally, a function can be annotated with boolean to indicate its return type. However, when the intention is to narrow the type of object being tested from a supertype to one of its subtypes, the annotation parameterName is Type can be used instead. This allows the compiler to infer the type of the argument at other points in the code after the test has been passed.

The following code demonstrates a use of this feature:

type Animal = {
  canWoof: boolean;
};

type Dog = {
  canWoof: true;
  woof: () => void;
};

const isDog = (animal: Animal): animal is Dog => {
  if (animal.canWoof) {
    return true;
  }

  return false;
};

In this example, Animal is a supertype of Dog. The function isDog() takes an Animal object and uses a simple test to determine whether the object satisfies the type Dog. If the test passes, the annotation kicks in, allowing more powerful inference by the compiler as can be seen in the following example.

An example

Since isDog() has been annotated with animal is Dog, all following statements in the scope can be typed accordingly.

const dogFunction = (animal: Animal) => {
  if (!isDog(animal)) {
    return;
  }

  animal.woof(); // <- this is valid since the type has been inferred above
};

This would not be possible if the return type of isDog() were boolean, but after a statement returning from the function after failing the isDog() test, animal.woof() can be called safely.

This is also true of union types such as:

type Cat = {
  canWoof: false;
  meow: () => void;
};

type Dog = {
  canWoof: true;
  woof: () => void;
};

type Animal = Cat | Dog;

Again:

const dogFunction = (animal: Animal) => {
  // animal could still be Cat | Dog
  animal.meow(); // <- uh oh, 'meow' does not exist on type 'Animal'

  if (!isDog(animal)) {
    animal.meow(); // <- definitely Cat

    return;
  }

  animal.woof(); // <- definitely Dog because of return above
};

In these examples, if the annotation animal is Dog were replaced with boolean these type inferences would not be possible, greatly reducing the power of TypeScript to provide suggestions and type checking.

Filtering a list

A list of ambiguous objects can be filtered to be of a single type as follows:

const ambiguousList: Animal[] = [new Cat(), new Dog()];
const dogList: Dog[] = ambiguousList.filter(isDog);

This is now valid as far as the compiler is concerned. The array is now of type Dog.

Limitations from an example in the wild: lodash.isEmpty

The isUndefined function from Lodash has the following signature:

isUndefined(value: any): value is undefined;

This employs the typeguard correctly, allowing the value to be disambiguated. On the other hand, the isEmpty function has the following signature:

isEmpty(value?: any): boolean;

Despite the fact that an undefined value should be captured by the isEmpty function, it means that the following code will not compile:

type Container = {
  escape: () => void;
};

const runEscapeIfNotEmpty = (container?: Container) => {
  if (isEmpty(container)) {
    return;
  }

  container.escape(); // <-- this will complain that `container` could be `undefined`
};

To correct this, the compiler would need to be able to rule out undefined and null as values. Sadly, there is no is not operator in TypeScript, otherwise the following might be possible:

isEmpty(value?: any): value is not undefined | null;

It’s worthwhile to keep the limitations of this feature in mind when designing guard functions.

Conclusion

In conclusion, this feature should be taken advantage of whenever possible. It can greatly simplify code, making it clearer and easier to read.

However, there are limitations, such as the inability to detect when a variable does not satisfy a type.

Refer to the official TypeScript documentation for more.

On the Importance of Explicitly Converting Strings to Numbers

2022-01-11T00:00:00+00:00

Recently a valued colleague approached me with a JavaScript problem. This individual is new to programming and is working on a self-taught course.

The assignment was fairly simple: Take a list of space-delimited integers and find the maximum and minimum values. If you are an experienced developer you can probably already guess where this is going. His code:

function highAndLow(numbers) {
  const myArr = numbers.split(" ");
  let lowNum = myArr[0];
  let highNum = myArr[0];
  for (let i = 0; i < myArr.length; i++) {
    if (myArr[i] > highNum) {
      highNum = myArr[i];
    } else if (myArr[i] < lowNum) {
      lowNum = myArr[i];
    }
  }
  return highNum + ' ' + lowNum;
}

console.log(highAndLow("8 3 -5 42 -1 0 0 -9 4 7 4 -4"));

This produced the output:

"8 -1"

These are clearly not the maximum or minimum values.

After looking at it for a few moments I recognized a classic JavaScript pitfall: failure to explicitly convert stringy numbers to actual number types.

You see, JavaScript tries to be clever. JavaScript tries to get it right. JavaScript tries to say “the thing you are doing looks like something that you would do with numbers so I’m going to automatically convert these stringy numbers to number-numbers for you.”

The problem is that JavaScript is not clever; it is in fact very dumb about this. The further problem is when developers come to trust and rely on automatic conversion. Careers have been ruined that way.

In this case the naive programmer would say “Well, I’m comparing the things with a mathematical operator (< and >) so JavaScript should treat the values as numbers, right?” Wrong. JavaScript compares them alphabetically, not numerically. Except that even the “alphabetical” comparison kind of sucks but that’s another topic. JavaScript doesn’t even attempt to convert to numbers in this case.

Repeat after me:

Always explicitly convert stringy numbers to actual numbers even if the language claims to do it automatically.

I don’t care if it’s JavaScript, Perl, some fancy Python package, it doesn’t matter.

Do not trust automatic type conversion.

You will get it wrong. It will get it wrong. There will be tears.

Fixing this program is as simple as changing one line to explicitly convert the numbers from strings.

const myArr = numbers.split(" ").map(n => Number.parseInt(n, 10));

Number.parseInt(n, 10) is the “one true way” to turn a string-number into a number-number in JavaScript. Never omit the 10; it is technically optional but you will regret it if you omit it, trust me. If you are reading base 10 numbers, tell JavaScript so explicitly. Otherwise it will again try to be clever but be not-clever and probably screw up the conversion by guessing the wrong radix.

It’s good that the developer caught this error visually, also, because they did not include a unit test. Errors like this slip through the cracks all. the. time.

Even TypeScript would not catch this. This function is perfectly legal TypeScript. There’s nothing illegal about comparing strings with < or >. TypeScript could only catch this if the developer provided additional type information up front, for example:

Now that we’ve told the compiler “This is a string” and “This is a number”, now it can helpfully tell us “Hey, you’re trying to mix strings and numbers in a not-good way”.

So it all comes back to the mantra of “Always explicitly convert strings to numbers. Always.” And if you’re bothering to use TypeScript, go the extra step and actually tell it what the types are. Don’t make it guess; it might guess wrong. Explicit is better than implicit.

Some things never change.

Salesforce data migration: promoting data changes from a sandbox to production

2021-11-24T00:00:00+00:00

Photo by Dylan Wooters, 2021

Intro

End Point recently completed a new e-commerce website built using Nuxt, Node.js, and Algolia and backed by Salesforce data. As part of the project, the client also wanted to re-categorize their products. They had already updated the data in a Salesforce sandbox and needed a way to move their changes to production in conjunction with the launch of the new website.

This data migration initially seemed like a simple request, but it ended up being one of the more challenging parts of the project. While there are clear instructions on how to migrate code changes from a sandbox to production (via change sets), there is very little information on how to migrate data.

Here I will outline the solution we developed, which involved writing a custom migration script using the JSForce library, testing using additional sandboxes, and planning for a rollback. This can serve as a blueprint for developers faced with the same task.

Why not use Data Loader to migrate?

The Salesforce Data Loader is a clunky yet reliable application that allows you to load large amounts of data directly into a Salesforce org. It will work to migrate data if the requirements are not complex. For example, if you only need to load data into a few objects and there are no connected objects or parent-child relationships. We chose not to use Data Loader for a few important reasons:

The data we were migrating was complex. Three levels of hierarchical categories were stored in Salesforce as separate custom objects, attached to another custom object representing products. To make things worse, the category objects were also all attached to each other in a parent-child hierarchy. To load this data using the Data Loader and CSV files was very time-consuming and error-prone.
We needed a fast and reproducible migration process. We had a strict cutover time for the launch of the new website, and the migration therefore had to run quickly.

We did end up using Data Loader as part of our rollback process, but more on that in step 6.

Our solution

1. Export JSON data from the sandbox

For our source data, I exported JSON from the Salesforce sandbox using the JSForce CLI, specifically the query function. Having the source data in JSON was easier than parsing CSV files exported directly from Salesforce. I committed the query commands to our Git repo for easy access during the actual production migration.

Here is a basic example of how to export data using the JSForce CLI.

jsforce -c dylan@endpointdev.com \
  -e "query('SELECT Id, Name, CreatedDate FROM Account')" > sandbox-export.json

2. Write a custom migration script

We were already using JSForce in our project to sync Salesforce data with the Algolia search engine, so we decided to also use the library to migrate data. I won’t dive into the specifics of how to use JSForce in this article, since we have another post that does that well.

My basic approach to writing the script started with asking the question “What would we do if we had direct access to the Salesforce database”? Once I had all of the set-based INSERT and UPDATE operations mapped out, I then translated that into procedural TypeScript code.

Here is a basic example from the migration script, where we are reading data from the JSON exported from the sandbox and loading it into Salesforce production.

import * as fs from 'fs';
import * as path from 'path';
import * as jsforce from 'jsforce';

const openConnection = async() => {
  let conn = new jsforce.Connection({
    loginUrl : process.env.instanceURL
  });
  await conn.login(process.env.username as string, process.env.password as string);
  console.log('info', `Connected to Salesforce.`);
  return conn;
}

const getJson = (filename: string) => {
  return JSON.parse(
    fs.readFileSync(
      path.resolve(__dirname, 'sandbox-data/' + filename)
    ).toString()
  );
};

const segmentData = getJson('new-segments.json');
const connection = await openConnection();

for (let i = 0; i < segmentData.records.length; i++) {
  let segment = segmentData.records[i];
  // Ignore Marine Equipment segment since it already exists
  if (segment.Name != 'Marine Equipment') {
    await connection.sobject('Segment__c').create({
      Name: segment.Name,
      Description__c: segment.Description__c,
      Meta_Description__c: segment.Meta_Description__c,
      Meta_Keywords__c: segment.Meta_Keywords__c,
      SubHeader__c: segment.SubHeader__c
    });
  }
}

3. Come up with a plan for testing data changes

After the migration script runs, you’ll need a way to verify that the data was loaded successfully and according to the requirements. For us, this meant working with our client to develop a smoke testing plan. After running our script, we switched a test version of the website to run against the migrated data in a Salesforce sandbox, and then used the plan to test all of the website functionality to make sure everything worked and the data looked correct. You’ll want to develop a plan based on your specific use case, whether that means verifying changes in Salesforce or an integrated system/website.

4. Test the migration script against a copy of production

Once you have the migration script and test plan completed, you’ll want to run the script against a copy of Salesforce production. The easiest way to do this is to create a new Full sandbox in Salesforce. However, if you’re short on Full sandbox licenses, you can create a Partial Copy, choosing only the options that are targeted in your script. Both sandboxes can be created by navigating to the Setup page in Salesforce, and then going to Platform Tools > Environments > Sandboxes.

After the script runs successfully, you can evaluate the data in the target sandbox to make sure everything looks correct. In our case, this involved pointing a test version of the new site to the migrated data.

One important caveat in this step is the refresh interval for the sandbox. Full sandboxes have a refresh interval of 30 days, and partials have an interval of 5 days. Essentially what this means is that you can’t easily press a button and refresh the sandbox after running the migration script, in case you need to make a change or fix an error. As a result, it’s best to have the data exported from Salesforce production and understand how to rollback the script updates, which I’ll explain more later on.

5. Export data from Salesforce production

Obtain a copy of the Salesforce production data by using the Data Export feature. This can be used as source data for a rollback, if errors occur in either testing the migration script or the real run against production.

To export data from Salesforce, first make a list of all the objects that will be affected by the migration. Then, navigate to the Salesforce Setup page, and go to Administration > Data > Data Export. Click the “Export Now” button and choose the applicable objects (or choose “Include All Data”), and hit “Start Export”. The data export will take a bit, but you should get an email from Salesforce once it is complete.

6. Perform a test rollback

After you test the migration script, if you find that the data is incorrect, you can perform a rollback using the Salesforce Data Loader. Even if the script runs successfully, you’ll want to test a rollback in case an error occurs during the real run. Here’s how I did it:

Download and install the Data Loader for your platform
Delete all the data for each affected object. To do this, log into Salesforce and open the Developer Console, then click on the “Debug” menu and choose “Open execution anonymous window”. Use the following code snippet to delete all data, replacing the Yourobject__c with the name of the target object. Again, you’ll need to do this for each object that was affected by your script.

List<Yourobject__c> SobjLst = [select id from Yourobject__c];
delete SobjLst;

Finally, re-insert the data using the CSV backups from step 4. In the Data Loader, choose the Insert method, and log into Salesforce. Be sure to use an account with the proper write permissions. Then go through the Data Loader wizard, choosing the applicable CSV backup file and load the data back into Salesforce. Repeat for each affected object.

Not the prettiest application, at least on a Mac. Circled in red is the Insert button.

This process turned out to be quite cumbersome for our project, due to the complexity of the data. It ended up taking several rounds of manual work in Excel (vlookups) to align all of the connected object ids correctly. It took me back to my first job after college. Hopefully your attempt will be easier! If it proves to be challenging, check out the online version of the Data Loader. It has expanded features that make it easier to import objects with relationships. Note, however, that the online version is only free for up to 10,000 records.

7. Run the script against production

Starting the “livestream” of our website launch in Slack!

Finally it’s time for the main event. Before you run your script against Salesforce production, be sure to export both the data from the sandbox and the data from Production (steps 1 and 4). This will ensure that the most recent data gets migrated and will set you up for a rollback if something goes wrong.

After your migration script runs without errors, use the testing plan from step 3 to verify that the data is correct. Once you’ve verified everything, give yourself a pat on the back for successfully navigating the arcane and treacherous world of a Salesforce data migration!

Summary

If your Salesforce data model is complex, consider using the steps above to migrate data from your sandbox to production. It will ensure a successful migration, especially if need a fast and repeatable process. If your data is simple and you are migrating a few objects, check out the Salesforce Data Loader application or DataLoader.io.

For more info about the tech we used in this project, check out:

Algolia
Nuxt

An Introduction to TypeScript

2021-02-09T00:00:00+00:00

TypeScript is a programming language defined as a superset of JavaScript. It adds static type information to JavaScript code using type annotations. These annotations permit strong type-checking at compile-time, kind of like a very strict linter. They are only used for static analysis. TypeScript is transpiled to JavaScript for execution in the browser or Node.js and the type annotations are stripped out. It is still possible to use standard JavaScript type information at run-time, such as that obtained using the typeof and instanceof operators.

Adding type annotations brings many benefits. Most importantly, TypeScript can tell us when we’re doing something dumb that’s likely to cause a type-related bug. But more than that, it powers editors’/IDEs’ context sensitive tool tips so when you hover or start typing the editor can supply helpful information so you can get your job done quicker. This is particularly useful to new developers as it saves them the trouble of reading all the sources to figure out the variable types from context, especially when debugging.

JavaScript is a fairly dynamic language but types still exist in JavaScript, whether we like it or not. Because it’s so dynamic it assumes that you the programmer know the type of every object you are using on every line of code and will do nothing to help you get it right. The type is specified by the context. The trouble is that in any non-trivial codebase it becomes impossible to be absolutely 100% certain about the type of some particular object that’s being passed around without reading the context, which is often a ton of code.

By eagerly annotating our JavaScript code with types we can eliminate entire classes of bugs from our codebase. TypeScript will never let you play loosey-goosey with integers and strings, for example. If you want a variable to be able to hold an int or a string you must explicitly declare it as type “integer” or “string”.

TypeScript lowers the total cost of ownership of a codebase by saving time. Programmers no longer have to guess or infer types and spend less time reading documentation and fixing bugs. Because fewer classes of errors are possible, less testing is required. Entire classes of serious problems never make it to the users because developers never commit them to the codebase. New developers on-board faster and make fewer mistakes. When multiple developers collaborate, their intentions with respect to types are immediately known to all, including the TS compiler, without reading any docs.

TypeScript is open source software maintained by Microsoft: It is free to use, and the specification and tools are provided under the Apache open source license. Microsoft has cultivated a strong developer community with high levels of engagement.

Strictness

Note that TypeScript has several settings for controlling the strictness of its type checking. To obtain maximum benefit we recommend using the --strict option which turns on all strict-checking options. This blog post assumes that strict mode is enabled. For more information see the TypeScript Compiler Options.

Declaring Variables and Functions

let someBool: boolean;

TypeScript will complain if you try to do this:

someBool = "hello world!";  // this is a TypeScript error; someBool
                            // cannot hold a string.

Another way to declare types is to assign directly to constants:

const someBool = true;

TypeScript knows through inference that someBool is of type boolean because it’s a const assigned from a boolean type expression. Later TypeScript will complain if you try to assign a non-boolean typed value to someBool.

Functions

Function declaration is similar to variable declaration:

function someFunc(someBool: boolean): string {
  return "Hello, world!";
}

This useless function takes one boolean type argument and returns a string type value:

const someFunc = (someBool: boolean): string => "Hello, world!";

TypeScript requires parentheses around annotated lambda arguments, even for a single argument.

Assertions

TypeScript supports “type assertions”, which is a way of telling the compiler that you are very certain a particular value is of a particular type and that it should assume you are right. These are sometimes called “casts” in other languages.

Generally speaking, type assertions are a red flag. There’s usually a better, more strongly-type way to write the code that avoids their usage. But sometimes they are convenient, particularly when interfacing with untyped JavaScript APIs. Of course it would be better to declare types for those APIs, but this is beyond the scope of some projects.

Type assertions use the as keyword:

let someBool: boolean;
someBool as unknown as string = "hello world";  // Not an error (but probably a mistake)

Because this is case is so egregious, and likely to be a programmer error, TS forces you to double cast it. A single as is usually sufficient.

Interfaces

Ok, now the real fun begins. So far we’ve only looked at primitive types like string and boolean. What about objects, arrays, and all that?

TypeScript “interfaces” allow us to declare the properties and behaviors that objects should have. There are required properties, optional properties, etc.

Interface declaration looks somewhat like object usage:

interface myInterface {
  someBool: boolean;
  someString: string;
}

Now when we use it, both someBool and someString are required. Hence, the following attempt to assign an empty object to a const of type myInterface is a TypeScript error:

const myObj: myInterface = {};  // TypeScript error!

The object we assign must have both required properties:

const myObj: myInterface = {
  someBool: true,
  someString: "Hello"
};  // This works.

Optional fields are annotated using the ? character:

interface myInterface {
  thisIsRequired: boolean;
  thisIsOptional?: string;
}

const myObj: myInterface = {
  thisIsRequired: true
};

Union Types

Sometimes you want to say that “this thing is of this type… or that type.” Enter “union types”:

type boolOrString = boolean | string;

let myVar: boolOrString;

myVar = true;
myVar = 'hello!';

This is 100% valid. myVar can hold either a boolean or a string value. However, TypeScript will force you to disambiguate which type it currently holds before you can do anything unsafe with it. For example, if we have a function that takes a union type:

function upperBool(arg: boolean | string): string {
  if (typeof arg == 'string') {
    return arg.toUpperCase();
  }
  else if (typeof arg == 'boolean') {
    return arg.toString().toUpperCase();
  }
}

It looks like our function could fall through and return undefined, and theoretically in JavaScript it could, but any such usage would trigger an error in TypeScript (assuming you are using --strictNullChecks, part of the --strict group of checks we recommend). TypeScript “knows” that we’ve handled both possible types of arg and so the if/else definitely does not fall through.

Intersection Types

An intersection type combines two or more interfaces into a single compound interface:

interface This {
  aProperty: string;
}

interface That {
  bProperty: boolean;
}

type Both = This & That;

const both: Both = {
  aProperty: 'funny',
  bProperty: true
};

Classes

TS has classes. I won’t get deep into TS classes, but I want to mention them. They can transpile to ES6 classes or ES2015 prototypal code. They are very powerful. They are, however, optional, as classes are still somewhat “new” to many JavaScript developers.

Enums

TypeScript supports enum declarations. An enum is a type that can only hold one of a specific set of possible values.

Generics

Generics are a mechanism for declaring “complex” types like arrays and promises. Creating new generics is somewhat advanced, but a basic understanding is required to use TS effectively.

const arrayOfStrings: string[];
const anotherArrayOfStrings: Array<string>;

Both of these syntaxes are equivalent. They both declare “an array of strings”.

TypeScript includes a significant number of built-in generic types and you will need them to use the language effectively.

let promiseMeAString: Promise<string>;

This declares a promise that is expected to resolve to a string type value.

Indexable Types

Indexable types allow the developer to specify the key and value types expected when using the square-bracket operator [] to access object or array properties. This is appropriate when the key set is large or arbitrary and not known in advance. In this case it’s not possible to define each possible key value.

This interface describes an object that holds a collection of string type keys with boolean values:

interface stringKeysBoolVals {
  [key: string]: boolean
}

The syntax is a little funny in that it includes a “placeholder” field, which I’ve set to key. This field is essentially a comment. TS ignores it. You can set it to anything that makes sense, e.g. index.

This constructs an object acting as a collection of arbitrary key/value pairs using the interface we just declared:

const myCollection: stringKeysBoolVals = {
  "value1": true,
  "value2": false,
  "value3": true
};

This usage is invalid because neither the key type of int nor the value-type of string is compatible with the declared interface:

const invalidCollection: stringKeysBoolVals = {
  0: "foobar"
};

A Note about Tooling

To get the most out of TypeScript it’s critical that your editor integrates with it and supports real-time type checking and error highlighting. Yes, you can run the TS compiler and check for errors at the command line, but this is a much slower development process.

Also, most editors will be able to use TypeScript annotations to provide context-sensitive help and tell you about types and declarations, which is really helpful. Modern APIs are too complicated to memorize; don’t try.

Conclusion

I hope this helps bring to light some of the wonderful features of TypeScript that help us write more reliable, more maintainable, and more readable code.

The official TypeScript website is a great place to learn more.

Writing a WebSocket-Controlled State Machine

Building Ecommerce Search Using Algolia

Loading the index with data

Configuring the index

Setting up search on the frontend

Using faceting for search refinement and filtering

Wrapping up: seeing Algolia in action

Upgrade Vue to TypeScript

Installation

Convert your component files

Add lang="ts" to the <script> tag

Replace the default export with a class that extends Vue and uses the Component decorator

Replace props with class properties marked by the @Prop decorator

Replace data with individual class properties

Add a typed $refs class property

Replace computed with getter and setter class properties

Lifecycle hooks become class methods

Replace methods with individual class methods

Replace this.$emit with class methods using the @Emit decorator

Convert your store

Conclusion

Type guards in TypeScript

The feature

An example

Filtering a list

Limitations from an example in the wild: lodash.isEmpty

Conclusion

On the Importance of Explicitly Converting Strings to Numbers

Salesforce data migration: promoting data changes from a sandbox to production

Intro

Why not use Data Loader to migrate?

Our solution

1. Export JSON data from the sandbox

2. Write a custom migration script

3. Come up with a plan for testing data changes

4. Test the migration script against a copy of production

5. Export data from Salesforce production

6. Perform a test rollback

7. Run the script against production

Summary

An Introduction to TypeScript

Strictness

Declaring Variables and Functions

Functions

Assertions

Interfaces

Union Types

Intersection Types

Classes

Enums

Generics

Indexable Types

A Note about Tooling

Conclusion

Add `lang="ts"` to the `<script>` tag

Replace the default export with a class that extends `Vue` and uses the `Component` decorator

Replace `props` with class properties marked by the `@Prop` decorator

Replace `data` with individual class properties

Add a typed `$refs` class property

Replace `computed` with getter and setter class properties

Replace `methods` with individual class methods

Replace `this.$emit` with class methods using the `@Emit` decorator