Generate PDF with Chrome, Puppeteer, and Serverless Stack

2022-01-02T00:00:00+00:00

Function as a Service (FaaS) solutions are becoming more and more mainstream today. The FaaS model allows developers to not have to worry about managing infrastructure and instead focus on writing the application logic. In the FaaS model, developers write individual functions that run specific tasks that are deployed together on a FaaS platform, including but not limited to Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP).

These functions don’t always run like a traditional application server does, waiting for a request. Instead, these FaaS platforms only spin up instances of these functions whenever there is traffic and will shut them back down once there are no more requests after a given period of time. This helps make FaaS a really cheap platform while traffic is low. It is a good approach for on-demand tasks that are part of the application but not necessarily the most common path in a customer’s everyday journey within the application.

The tools

In this tutorial, we’re going to be looking at implementing PDF download with Chrome, Puppeteer, and Serverless Stack, but first, let’s have a brief introduction of the tools that we are going to be using.

1. Serverless Stack

Serverless Stack is a framework for building full-stack serverless apps. The bigger player in this space is the Serverless framework. Serverless Stack is giving the Serverless framework (yes, I know, it can be very confusing at times) a challenge to the throne. The latter has been around for many years now and has been the authority for building serverless apps. There are pros and cons to both of them.

Serverless Stack’s biggest advantage is the live Lambda debugging, a faster development process and overall better development experience. With the Serverless framework, developers have to either constantly deploy to the cloud to test any change, or spend some time setting up serverless-offline for simulated Lambda and API gateway environments on their local machines.

But with the Serverless framework, developers are not tied to using a specific platform, as it can be used to deploy functions on AWS, Azure, and Google Cloud Platform. Serverless Stack, on the other hand, only allows deploying to AWS since its internals are built on top of AWS CDK.

2. Chrome and Puppeteer

Chrome will be used in this tutorial as a headless browser, controlled using Puppeteer. Puppeteer is a Node.js package that provides an API to make Chrome or Chromium navigate to a specific URL, take screenshots, click buttons, type in input fields, as well as take PDF screenshots. Note that the PDF screenshot feature is specific to Chrome and not Chromium, so make sure to double check that you are using the right browser.

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Now, let’s get to the meat of this tutorial. We are going to implement PDF generation with Chrome and Puppeteer, deployed on AWS Lambda through Serverless Stack. Here is what we will do:

Use Puppeteer in Chrome to navigate to a receipt page that we implemented.
Instruct Chrome to take a PDF screenshot of the page.
Stream the PDF file back to the user.
Deploy this implementation on AWS Lambda using Serverless Stack and demonstrate the development experience throughout.

PDF generation is a great use case for FaaS. It represents something that doesn’t happen very often and is triggered on demand by the user. Instead of letting our monolith application handle that intermittent traffic, we can delegate the task to FaaS in the cloud which will do this job without worrying us about scaling the infrastructure to cater to this particular use case. Then our application can just focus on handling the regular web traffic for general use.

Prerequisite: Sample receipt pages

First, let’s create some demo pages for PDF screenshots. I’m not going to go in-depth on these since it’s not the topic of the day. I have deployed these simple HTML pages that are styled like receipts on this Surge instance here. You can find the code on GitHub here.

Getting started

Let’s generate a new Serverless Stack project.

$ npx create-serverless-stack@latest pdf-generator
$ cd pdf-generator
$ npm install

Congratulations, we have just created a new project with Serverless Stack! Here we get a basic template of a working Serverless Stack app. You can go here for more information on how to get started with Serverless Stack.

First, let’s look at the files we get from bootstrapping the project and customize them wherever we see fit.

1. sst.json

{
  "name": "pdf-generator",
  "region": "us-east-1",
  "main": "stacks/index.js"
}

This file is the entry point to the Serverless Stack app, and here we can define the region of choice, the name of the app, and the main file that Serverless Stack will use.

2. stacks/index.js

 // stacks/index.js
 app.setDefaultFunctionProps({
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

stacks/index.js is the main file declared in sst.json. When building the project, Serverless Stack will use this file as an entry point to our application. This file is pretty simple:

It configures the Lambda functions to use the Node.js 12.x runtime.
It registers a stack called my-stack. This is the name of the CloudFormation stack on AWS. We definitely want to give it a better name than my-stack here, so let’s rename that to pdf-generator.

3. stacks/MyStack.js

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      routes: {
        "GET /": "src/lambda.handler",
      },
    });

    // Show the endpoint in the output
    this.addOutputs({
      "ApiEndpoint": api.url,
    });
  }
}

MyStack.js is where we declare the resources we need inside a given CloudFormation stack. This can be anything from a cluster of Lambda functions, API Gateway endpoints, DynamoDB tables, S3 buckets, etc. The full list of resources that we can create with Serverless Stack is listed here in the documentation. For our project we just need one API endpoint with API gateway, so what we are provided with here is already sufficient.

Deploying the project

Now, we can try to deploy this project to AWS. Ensure that the AWS Access Key ID and AWS Secret Access Key are set in the development environment, and then run npx sst start.

# export AWS credentials
export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<secret access key>

# or if the credentials are set in ~/.aws/credentials
export AWS_PROFILE=<AWS profile name>

# Start the app in development mode
npx sst start

After a few minutes, we will have the Cloudformation stack deployed on our AWS account. Keep in mind that the first deployment will take a bit longer than subsequent deployments. When the deployment is done, we should be able to see that it is watching for file changes from our local machine.

threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::ApiGatewayV2::Route | ApiRouteGET8AC7D3F8
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::Lambda::Permission | ApiRouteGETthreadlightlypdfgeneratorpdfgeneratorApiRouteGET7230D6CDPermissionE4542537
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::CloudFormation::Stack | threadlightly-pdf-generator-pdf-generator

 ✅  threadlightly-pdf-generator-pdf-generator


Stack threadlightly-pdf-generator-pdf-generator
  Status: deployed
  Outputs:
    ApiEndpoint: https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com


==========================
 Starting Live Lambda Dev
==========================

Transpiling Lambda code...
Debug session started. Listening for requests...

Let’s head to the API URL given in the output and we should get a response back like this:

Hello, World! Your request was received at 30/Nov/2021:06:21:13 +0000.

Developing with Serverless Stack

Let’s try updating one of the files provided to see how we can test the changes we have made and will make.

 // stacks/MyStack.js
 const api = new sst.Api(this, "Api", {
   routes: {
-    "GET /": "src/lambda.handler",
+    "GET /downloads/receipt": "src/lambda.handler",
   },
 });

 // stacks/index.js
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

Here we did two things:

We updated the API endpoint to GET /downloads/receipt instead of just GET /.
We made the Lambda function return the URL given in the query string.

Once we save these changes we should see Serverless Stack automatically reloading our code and since we changed the API endpoint path, Serverless Stack will need to make changes to our infrastructure (i.e. the API Gateway resources), and whenever it detects that it has to change the infrastructure of our application, it will immediately prompt us to redeploy.

Rebuilding code...
Rebuilding infrastructure...
Done building code
Press ENTER to redeploy infrastructure

Once we hit ENTER, Serverless Stack will automatically update our infrastructure for us: delete our old endpoint, create a new endpoint on GET /downloads/receipt, and hook up our src/lambda.js file to handle that endpoint.

When it’s ready, try hitting the API gateway URL again, but this time append /downloads/receipt?url=https://google.com to it.

// https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://google.com
Hello! You've requested to print the receipt at page https://google.com

Great, now we have learned how to make changes to our Serverless Stack application during development. Let’s go ahead and add Puppeteer and Chrome to our Lambda function.

 // stacks/MyStack.js
 import * as sst from "@serverless-stack/resources";
+import { LayerVersion } from "@aws-cdk/aws-lambda";
+
+const layerArn = "arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:25";

 // Create a HTTP API
 const api = new sst.Api(this, "Api", {
   routes: {
-     "GET /downloads/receipt": "src/lambda.handler",
+     "GET /downloads/receipt": {
+       function: {
+         handler: "src/lambda.handler",
+         // Increase the timeout for generating screenshots
+         timeout: 15,
+         // Load Chrome in a Layer
+         layers: [LayerVersion.fromLayerVersionArn(this, "Layer", layerArn)],
+         // Exclude bundling it in the Lambda function
+         bundle: { externalModules: ["chrome-aws-lambda"] },
+       }
+     },
   },
 });

Here, we are configuring our AWS Lambda function to use a Lambda layer that includes Chrome in our Lambda functions. You can think of these Lambda layers as being similar to NPM packages that you pull for extending your projects except that this is a 3rd party package for extending AWS Lambda functions instead.

With this layer, your AWS Lambda function will boot with the Chrome binary already included. This layer is maintained here. Make sure you’re using the ARN from the right region. In this tutorial, we are using us-east-1 per Serverless Stack’s default, so we are going to pick the ARN for us-east-1.

Now, let’s update our Lambda handler to:

Use Puppeteer to instruct Chrome to navigate to the URL passed in the parameter.
Take a screenshot of the web page and save a PDF out of that. Puppeteer provides a handy page.pdf API to do just that.
Ultimately, stream that PDF file back to the user.

Here’s the code for that, with some comments to help you out:

+import chrome from "chrome-aws-lambda";
+
+const puppeteer = chrome.puppeteer;
+
 export async function handler(event) {
-  return {
-    statusCode: 200,
-    headers: { "Content-Type": "text/plain" },
-    body: `Hello! You've requested to print the receipt at page ${event.queryStringParameters.url}`,
-  };
+  let browser
+  let response
+  try {
+    const { url } = event.queryStringParameters;
+
+    browser = await puppeteer.launch({
+      args: chrome.args,
+      executablePath: await chrome.executablePath,
+    });
+
+    const page = await browser.newPage();
+    // Use A5 size at 150dpi
+    const width = 874
+    const height = 1240
+
+    await page.setViewport({ width, height });
+
+    // Navigate to the url
+    await page.goto(url, { waitUntil: 'networkidle2' });
+
+    // Take the screenshot
+    await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+
+    response = {
+      statusCode: 200,
+      body: JSON.stringify({message: 'Screenshot taken'})
+    };
+  } catch(err) {
+    response = {
+      statusCode: 500,
+      body: JSON.stringify({message: `An error occured. ${err.message}`})
+    }
+  } finally {
+    await browser && browser.close()
+  }
+  return response
 }

Let’s also not forget to install Puppeteer and Chrome.

$ npm install puppeteer puppeteer-core chrome-aws-lambda

Now, give Serverless Stack some time to reload our new changes. Once we have the API endpoint ready to go, let’s head to the browser and hit our API endpoint, passing the URL to the sample receipt page that we did earlier as a query param.

// Visit https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://unwieldy-key.surge.sh/index.html
// Response:
{"message":"Screenshot taken"}

Great! We are getting back a 200 status code and a message saying “Screenshot taken”. That means our code works and there were no exceptions. We’re halfway there.

Now, if we check the project folder, we should now see a new file called receipt.pdf there. This is the receipt.pdf captured by Puppeteer. We can open this file and verify that this is the demo receipt page that I deployed earlier that we passed to the endpoint.

We can also hit the API endpoint with the second sample receipt URL (i.e. url=https://unwieldy-key.surge.sh/receipt-2.html) and we should see the receipt.pdf file gets replaced with the receipt from the new URL.

Note that in a real-world application, the URL you would want to screenshot might need some form of authentication, so you are going to have to figure that out for your application. In this case the sample receipt page is accessible publicly, so we don’t run into this problem, but chances are your users’ receipt page will not be accessible publicly like this.

One way you could solve this is to create a special user account that has permission to visit these authenticated pages for this purpose. Then, prior to visiting the receipt page, you’d program Puppeteer to first log in as the special user and then head to the receipt page URL to be screenshot.

However, we are only halfway there. We can see the receipt.pdf file now only because the receipt.pdf file was made available to us by Serverless Stack, which automatically downloads that file to our local machine in development mode. In a real AWS Lambda execution, the file will remain in the filesystem of that Lambda execution context instead, and the user will not see it.

Hence we have to read and stream that file back to the user instead of returning a JSON message saying “Screenshot taken”. Getting a 200 status code is a great start, but that still does not give our users the actual PDF that they wanted, so let’s update our Lambda handler to do just that:

 // src/lambda.js
+import * as fs from "fs";

 // Take the screenshot
 await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+const buffer = fs.readFileSync('receipt.pdf')

 response = {
   statusCode: 200,
-  body: JSON.stringify({message: 'Screenshot taken'})
+  headers: {'Content-type' : 'application/pdf'},
+  body: buffer.toString('base64'),
+  isBase64Encoded : true,
 };

The changes this time are pretty simple:

Import the fs module from Node.js, which will give us access to readFileSync API that can read files on the disk.
Read the generated receipt.pdf file as a Buffer.
Return the file’s Buffer as a base64 string in the response body.
Set the content-type to be application/pdf. Setting the content-type appropriately will tell the browser what to do with the file. Some browsers will display the PDF in an in-browser PDF reader, while others will download the file.

That’s it! Now let’s test the change. Hit the same URL again.

Bingo, the endpoint now returns the PDF to the user! Now we can integrate this endpoint with our main application to delegate these on-demand PDF generation to managed FaaS infrastructure.

Conclusion

FaaS platforms are a really great way to offload intermittent and on-demand traffic from our main application. This ensures our application runs smoother and can serve our general customers better, while at the same time handling these on-demand tasks really well.

Caveats

Building a full-blown application on the stack is also possible, but in my personal experience, it comes with its own complexities and tradeoffs that may or may not be worth the engineering time.

For example, databases are an issue with going full-blown development on FaaS. Because FaaS can scale indefinitely, the number of connections to the database is a limitation.

You can switch to FaaS-friendly databases such as DynamoDB or FaunaDB, but these are NoSQL databases and require a different mindset in structuring the application data than the simpler relational databases and that can cost tremendous engineering time. Relational databases are easier to work with for most types of applications.

Nonetheless, more and more tools are being developed to tackle these complexities, such as PlanetScale and Serverless Aurora v2 (still in preview). All in all, it is definitely a trend worth following closely.

Jamstack Conf Virtual 2020: Thoughts & Highlights

2020-06-16T00:00:00+00:00

Welcome to Jamstack Conf Virtual 2020

Last week I attended Jamstack Conf Virtual 2020. It had originally been slated to take place in London, UK but was later transformed into a virtual event in light of the COVID-19 pandemic. The conference began at 2pm London time (thankfully I double-checked this the night before!)—6am for those of us in the Pacific Time Zone.

Up early for #jamstackconf 😎☕️ pic.twitter.com/ydjvrHWCZH
— Greg Davidson (@syncopated) May 27, 2020

Before getting too much further I wanted to mention that if you are not familiar with the Jamstack, You can read more about it at jamstack.org.

To virtually participate in the conference we used an app called Hopin. I had not heard of it before but was impressed with how well it worked. There were over 3000 attendees from 130+ countries one of the times I checked. Phil Hawksworth was the Host/MC for the event and did a great job. There were virtual spaces for the stage, sessions, expo (vendors), and networking. If you opted to, the networking feature paired you with a random attendee for a video chat. I’m not sure what I expected going into it but I thought it was fun. I met a fellow developer from the Dominican Republic. The experience was very similar though more serendipitous than the hallway track or lunch line at an in-person conference.

Keynote

Matt Biilmann opened the conference with a keynote address about the challenges we face as a developer community trying to improve access to accurate, timely and locally relevant information to a global audience. Many billions of users with all kinds of devices and varying levels of connectivity. He moved on to share how Netlify is trying to enable developers to “build websites instead of infrastructure” and “ensure all the best practices become common practices” through features like git-based deployments, build plugins, and edge handlers (more on those later).

State of the Jamstack Survey results

Laurie Voss walked us through the results of the Jamstack Survey 2020. There were some interesting findings and surprises. Later on I read Laurie’s post (which Matt had mentioned in his talk) and found that very interesting as well.

Fireside chat with Harper Reed

Frances Berriman chatted with Harper Reed and asked him about his application of Jamstack principles from years ago when he led the technology team for Barack Obama’s election campaign. He described the need to “get far with very limited resources” and spoke about experimenting with serving HTML from Google App Engine and Amazon S3. Using pre-built HTML allowed them to scale very efficiently and he opted to use message queues rather than interacting with the database to keep things very quick for users.

Another benefit Harper noted was how quickly and easily new team members could be onboarded. Folks who knew HTML, CSS and JavaScript would be up to speed and productive in no time. He admitted it’s more complicated today 😀. Speed is another benefit—he just loves when he comes across a super-fast web site (often mostly static).

Lightning Launch: Netlify ⚡

David Calavera gave a demo of Netlify’s new Edge Handlers feature which lets developers add logic to their code at the edge (e.g. the CDN servers geographically closest to the user). He demonstrated how Edge Handlers make it possible to examine the headers of the request to tailor the response to that unique request. Check out the video of his talk to watch him live code an example. I believe Cloudflare Workers and Fastly’s Edge Compute are operating in a similar problem space. I plan to look into each of these offerings more thoroughly in future.

Lightning Launch: Prismic ⚡

Renaud Bressand from Prismic demoed Slicemachine—a new feature from Prismic which lets you combine nuxt/Vue components with content managed in Prismic. It looked like a very compelling way of enabling better collaboration between developers and content creators.

Lightning Launch: RedwoodJS ⚡

Tom Preston-Werner demoed his latest project RedwoodJS. I had heard Tom speak about this on a few podcasts recently and it was nice to see him demo it for us. It looks interesting and feels reminiscent of Rails. RedwoodJS simplifies wiring up your database to a GraphQL API (they use Prisma for this) and integrating it into a React application. Tom is also the creator of Jekyll—a Jamstack-style tool which has been around for many years. It was nice to see several speakers give him some recognition for his work on that project.

The COVID Tracking Project: 0 to 2M API requests in 3 months

Erin Kissane gave an inspiring talk about her work on The COVID Tracking Project. She described it as an “interim public data rescue project”. Erin and some friends, journalists and volunteers worked together to create the site. They started by scraping COVID data from each state’s web site and storing it in a Google Sheet. They used Gatsby, Contentful, and Sass modules and hosted the site on Netlify. Using the Jamstack approach allowed the site to remain performant and continue to be responsive under some huge traffic spikes. Over time, they iterated on the design and continue to improve the site daily. I highly recommend checking out the video of the talk when you get a chance.

Jamstack for emerging markets

Christian Nwamba described some of the challenges of building sites for users in Nigeria (low power devices, spotty connectivity, unreliable power). He shared that 55% of the most visited sites in Nigeria are global companies (Google, Facebook/Instagram, Netflix, Stack Overflow etc.). Christian reviewed a large, popular banking site in Nigeria and noted its many shortcomings.

To demonstrate how the bank might do better he built an app for transferring money between friends & family built in the Jamstack style and using serverless functions. The most interesting thing I picked up from this was his method of using serverless functions to protect the app secrets (API keys, etc.). The front end of the application did not need to concern itself with this—the serverless functions kept the secrets safe and acted as a proxy between the frontend and the backend APIs. Be sure to take a look at Christian’s code if you are interested.

Managing diabetes with Jamstack

Jamie Bradley taught us about diabetes and the Jamstack app he built to help himself and others manage it. He built HeySugar with Gatsby and Sanity, hosted it on Netlify. He’s making it easier for others to deploy their own instances as well.

Selling tickets without servers. Or frameworks. Or cookies.

Jan van Hellemond has volunteered for years at a very popular conference in Europe (Fronteers I think). In previous years tickets sold out in 6 minutes one year, and 30 seconds in another! Their ticket vendor was struggling to handle to the load and this caused them to oversell early bird tickets. Jan built a Jamstack site to sell their tickets and was very pleased with the performance. He used simple, single-purpose vanilla JavaScript components and simple, single-purpose API handlers (serverless functions).

Jan prerendered as much as possible and seeded a relational database with the tickets available for sale. As the tickets were sold, they were marked sold with a database update. Webhooks were used as customers stepped through the checkout flow. Jan joked about deploying to production on a Friday afternoon because of how safe and simple the deployment process was. There were no services to restart, etc. because “it’s just files”. It was cool to see a practical example and that Jan used the basic building blocks of the web (HTML, JavaScript, CSS, old school links) successfully without reaching for a large JavaScript framework.

The business side of the Jamstack

Ana Rossetto shared how her company has been having great success with the Jamstack. Previously, the agency had been building projects for clients with Drupal. She walked through a project she and the team built to encourage people to buy from small, locally-owned businesses. She was impressed with what they were able to create in a very short amount of time.

Build plugin authors’ session

After the main talks there were several sessions. In the Hopin app, I was able to peek into some of these and the presenters and attendees were chatting (both text chat and video). This was very much like the experience of peeking into conference rooms at a physical venue and choosing whether to stay and participate or move on. After some wandering I chose to attend a session with three Netlify build plugin authors.

Peter Müller built Subfont with a friend. He demonstrated how subsetting web fonts (i.e. only loading the characters & glyphs you really need) can dramatically improve frontend performance. He compared the WebPageTest results for Google Fonts with and without Subfont and Subfont was seconds faster! I have subset and locally hosted webfonts in several client projects here at End Point. It takes time and is a manual process. Peter’s plugin makes this excellent performance improvement relatively painless.

David Darnes demoed his build plugin to turn Ghost content into Markdown files. Very interesting and it seemed flexible enough to work with other tools as well.

Gleb Bahmutov presented the build plugin he created to run Cypress tests as part of your Netlify build process. It was amazing to see how simple it was (single devDependency and single line in the netlify.toml config file).

Videos from the conference

Netlify has already put all of the Jamstack Conf Virtual 2020 talks on YouTube so head over there and check those out if you’d like to. Thanks very much to the team at Netlify for organizing the conf, all of the speakers, vendors and attendees!

Facial Recognition Using Amazon DeepLens: Counting Liquid Galaxy Interactions

2019-05-01T00:00:00+00:00

I have been exploring the possible uses of a machine-learning-enabled camera for the Liquid Galaxy. The Amazon Web Services (AWS) DeepLens is a camera that can receive and transmit data over wifi, and that has computing hardware built in. Since its hardware enables it to use machine learning models, it can perform computer vision tasks in the field.

The Amazon DeepLens camera

This camera is the first of its kind—likely the first of many, given the ongoing rapid adoption of Internet of Things (IoT) devices and computer vision. It came to End Point’s attention as hardware that could potentially interface with and extend End Point’s immersive visualization platform, the Liquid Galaxy. We’ve thought of several ways computer vision could potentially work to enhance the platform, for example:

Monitoring users’ reactions
Counting unique visitors to the LG
Counting the number of people using an LG at a given time

The first idea would depend on parsing facial expressions. Perhaps a certain moment in a user experience causes people to look confused, or particularly delighted—valuable insights. The second idea would generate data that could help us assess the platform’s impact, using a metric crucial to any potential clients whose goals involve engaging audiences. The third idea would create a simpler metric: the average number of people engaging with the system over a period of time. Nevertheless, this idea has a key advantage over the second: it doesn’t require distinguishing between people, which makes it a much more tractable project. This post focuses on the third idea.

To set up the camera, the user has to plug it into a power outlet and connect it to wifi. The camera will still work even with a slow network connection, though when the connection is slower the delay between the camera seeing something and reporting it is longer. However, this delay was hardly noticable on my home network which has slow-to-moderate speeds of about 17 Mbps down and 33 Mbps up.

Computer Vision and the Amazon DeepLens

A deep learning model is a neural network with multiple layers of processing units. It is called “deep” because it has multiple layers. The inputs and outputs of each processing unit are numbers. These units are roughly analogous to neurons: they receive input from units in the previous layer, and output it to units in the next layer after transforming it based on a function. These “activation functions” can change in a variety of ways. The last layer’s outputs translate into the results. These models work because these functions get tuned based on how well the model works. For example, to make a model that labels each human face in a picture and draws a box around it, we would start with a corpus of pictures with boxes drawn around faces, as well as the versions of the pictures without the boxes drawn. We would test the model on the non-labeled images by checking—for each picture—whether the output generated by the model is correct. If not, the computer chooses different unit functions, tries again, and compares the results. Repeating this process thousands of times yields models which work remarkably well for a wide range of tasks, including computer vision.

In deep learning for computer vision, training on large sets of labeled images enables models to generalize about visual characteristics. The training process takes a lot of computing resources, but once models are trained, they can produce results quickly and with relative ease. This is why the DeepLens is able to perform computer vision with its limited computing resources.

Since the DeepLens is an Amazon product, it comes as no surprise that the user interface and backend for DeepLens consist of AWS services. One of the most important is SageMaker, which is used to train, manage, optimize, and deploy machine learning models such as neural networks. It includes hosted Jupyter notebooks (Jupyter is a development environment for data science), as well as the computing resources required for model training and storage. With SageMaker, users can train computer vision models for deployment to DeepLens, or import and adjust pretrained models from various sources.

Remote management of the DeepLens depends on AWS Lambda, a “serverless” cloud service that provides an environment to run backend code and integrate with other cloud services. It runs the show, allowing users to manage everything from the camera’s behavior to what happens to gathered data. Another service, AWS Greengrass, connects the instructions from AWS Lambda to the DeepLens, managing tasks like authentication, updates, and reactions to local events.

Amazon’s IoT service saves information about each DeepLens, and allows users to manage their devices, for example by choosing which model is active on the device, or viewing a live stream from the camera. It also keeps track of what’s going on with the hardware, even when it’s off. When a model is running on the DeepLens, you can view a live stream of its inferences about what it’s seeing (the labeled images). Amazon has released various pretrained models designed to work on the DeepLens. Using a model for detecting faces, we can get a live stream that looks like this:

Me looking at the DeepLens in my kitchen

Facial recognition inferences on multiple people. (Witness my smile of satisfaction at finally finding enthusiastic subjects of facial recognition.)

Each face that the camera detects gets a box around it, along with the model’s level of certainty that it is a face. The above pictures were the results of an attempt to simulate the conditions where this could be used.

The Model

The model I used was trained on data from ImageNet, a public database with hundreds or thousands of images associated with nouns. (For example they have 1537 pictures of folding chairs.) ImageNet is commonly used to train and test computer vision models.

However, the training for this model didn’t stop there: Amazon used transfer learning from another large image dataset, MS-COCO, to fine-tune the model for face detection. Transfer learning works essentially by retraining the last layer of an already-trained model. In this way it harnesses the “insights” of the existing model (e.g. about shapes, colors, and positions) by repurposing this information to make predictions about something else. In this case, whether something is a face.

Since this model was pretrained and optimized by Amazon for the DeepLens, it provides a low effort route to implementing a computer vision model on the DeepLens. I didn’t have to do any of the processing on my own hardware. The DeepLens hardware took care of all the predictions, though the biggest resource savings were from not having to train the model myself (which can take days, or longer).

When the facial recognition model is deployed and the DeepLens is on, an AWS Lambda function written in Python repeatedly prompts the camera to get frames from the camera:

frame = awscam.getLastFrame()

…to resize the frames before inference (the model accepts frames of particular size):

frame_resize = cv2.resize(frame, (input_height, input_width))

…to pass the frames to the model:

parsed_inference_results = model.parseResult(model_type, model.doInference(frame_resize))

…and to use the results to draw boxes around the faces:

cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (255, 165, 20), 10)

As you can see from how often “cv2” appears in the code above, this implementation relies heavily on code from OpenCV, an open source computer vision framework. Finally, the results are sent to the cloud:

client.publish(topic=iot_topic, payload=json.dumps(cloud_output))

In the last code snippet above, iot_topic refers to an Amazon “MQTT topic” (Message Queuing Telemetry Transport), for IoT devices. MQTT is the standard connectivity framework for DeepLens and many other IoT devices. One of its advantages for this context is that it can handle situations with intermittent connectivity, by smoothly queueing messages for when the network connection is stable. The essence of MQTT is to enable publishing and subscribing to different topics. The system of topics enables results from a DeepLens to trigger other processes. For example, the DeepLens could publish a message when it sees a face, and this could prompt another cloud service to do something else, such as save what time and how long the face appeared.

I wanted to test how data from this model would compare to a human’s perception. The first step was to understand what data the camera offers. It produces data about each frame analyzed: a timestamp (in 13-digit Unix time), and the predicted probability that something it identifies is a face. To gather this data, I used the AWS IoT service to manually subscribe to a secure MQTT topic where the DeepLens published its predictions. Each frame processed produces data like this:

{
  "format": "json",
  "payload": {
    "face": 0.5654296875
  },
  "qos": 0,
  "timestamp": 1554853281975,
  "topic": "$aws/things/deeplens_bnU5sr2sSD2ecW5YkfJZtw/infer"
}

The data generated by a single frame (with one face) when processed by the DeepLens.

For my purposes, I was only interested in the timestamps and payloads (which contain the number of faces identified, and their probabilities). I decided to test the facial recognition model under several different conditions:

No faces present
One face present
Multiple faces present

For condition 1 I just aimed it at an empty room for 20 minutes, and for condition 2 I sat in front of the camera for 20 minutes. For condition 3, I aimed the camera at a public space for 20 minutes, and while it was running I kept an ongoing count of the number of people looking in the general direction of the camera (I put the camera in front of a wall with a TV on it so people would be more likely to look towards it). Then I averaged my count over the duration of the sample, which resulted in an average engagement number of 2.5 people, meaning that on average, 2.5 people were looking at the camera. In an attempt to minimize bias, I made my human-eye assessment before looking at any of the data.

I’ll spoil one aspect of the results right away: there were no false positives under any condition. Even the lower probability guesses corresponded to actual faces, though this result might not hold true in a room with lots of face-like art, that’s not too common of a scenario. This simplified things, since it meant there was no need to set a lower bound on the probabilities which we should count—any face detected by the camera is a face. This also highlights one of my remaining questions about the model: is there useful information to be gained from the probabilities?

Another important note: I noticed early in the experiment that it almost never detects a face farther than 15 feet away. For the use case of a Liquid Galaxy, the 15-foot range is too short to capture all types of engagement (some people look at it across the room), but from my experience with the system I think that users within this range could be accurately described as focused users—something worth measuring, but certainly not everything worth measuring. After noticing this, I retested condition 2 with my face about 5 feet from the DeepLens, after initially trying it from across a room.

How did the DeepLens counts compare to my counts?

The model matched my performance in conditions 1 and 2, which makes a strong statement about its reliability in relatively static and close-up conditions such as looking at an empty room, or looking at someone stare at their laptop across a small table. In contrast, it did not count as many faces as I did in condition 3—so I’m happy to report I can still outperform A.I. on something.

Anyway, this suggests that the model is somewhat conservative, at least compared to my count (likely partly due to my eyes having a range larger than 15 feet). Therefore, when considering usage statistics gathered by a similar method, it might make most sense to think of the results as a lower bound, e.g. “the average number of people focused on the system was more than 2.1”.

It would be useful to experiment with the multiple faces condition again, to see how robust these findings are. It would also be helpful to keep track of factors like how much people move, the lighting, and the orientation of the camera, to see if they might impact the results. It would also be useful to automate the data collection and analysis.

This investigation has showed me that the DeepLens has a lot of potential as a tool for measuring engagement. Perhaps a future post will examine how it can be used to count users.

Thanks for reading! You are welcome to learn more about End Point Liquid Galaxy and AWS DeepLens.