Generate PDF with Chrome, Puppeteer, and Serverless Stack

2022-01-02T00:00:00+00:00

Function as a Service (FaaS) solutions are becoming more and more mainstream today. The FaaS model allows developers to not have to worry about managing infrastructure and instead focus on writing the application logic. In the FaaS model, developers write individual functions that run specific tasks that are deployed together on a FaaS platform, including but not limited to Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP).

These functions don’t always run like a traditional application server does, waiting for a request. Instead, these FaaS platforms only spin up instances of these functions whenever there is traffic and will shut them back down once there are no more requests after a given period of time. This helps make FaaS a really cheap platform while traffic is low. It is a good approach for on-demand tasks that are part of the application but not necessarily the most common path in a customer’s everyday journey within the application.

The tools

In this tutorial, we’re going to be looking at implementing PDF download with Chrome, Puppeteer, and Serverless Stack, but first, let’s have a brief introduction of the tools that we are going to be using.

1. Serverless Stack

Serverless Stack is a framework for building full-stack serverless apps. The bigger player in this space is the Serverless framework. Serverless Stack is giving the Serverless framework (yes, I know, it can be very confusing at times) a challenge to the throne. The latter has been around for many years now and has been the authority for building serverless apps. There are pros and cons to both of them.

Serverless Stack’s biggest advantage is the live Lambda debugging, a faster development process and overall better development experience. With the Serverless framework, developers have to either constantly deploy to the cloud to test any change, or spend some time setting up serverless-offline for simulated Lambda and API gateway environments on their local machines.

But with the Serverless framework, developers are not tied to using a specific platform, as it can be used to deploy functions on AWS, Azure, and Google Cloud Platform. Serverless Stack, on the other hand, only allows deploying to AWS since its internals are built on top of AWS CDK.

2. Chrome and Puppeteer

Chrome will be used in this tutorial as a headless browser, controlled using Puppeteer. Puppeteer is a Node.js package that provides an API to make Chrome or Chromium navigate to a specific URL, take screenshots, click buttons, type in input fields, as well as take PDF screenshots. Note that the PDF screenshot feature is specific to Chrome and not Chromium, so make sure to double check that you are using the right browser.

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Now, let’s get to the meat of this tutorial. We are going to implement PDF generation with Chrome and Puppeteer, deployed on AWS Lambda through Serverless Stack. Here is what we will do:

Use Puppeteer in Chrome to navigate to a receipt page that we implemented.
Instruct Chrome to take a PDF screenshot of the page.
Stream the PDF file back to the user.
Deploy this implementation on AWS Lambda using Serverless Stack and demonstrate the development experience throughout.

PDF generation is a great use case for FaaS. It represents something that doesn’t happen very often and is triggered on demand by the user. Instead of letting our monolith application handle that intermittent traffic, we can delegate the task to FaaS in the cloud which will do this job without worrying us about scaling the infrastructure to cater to this particular use case. Then our application can just focus on handling the regular web traffic for general use.

Prerequisite: Sample receipt pages

First, let’s create some demo pages for PDF screenshots. I’m not going to go in-depth on these since it’s not the topic of the day. I have deployed these simple HTML pages that are styled like receipts on this Surge instance here. You can find the code on GitHub here.

Getting started

Let’s generate a new Serverless Stack project.

$ npx create-serverless-stack@latest pdf-generator
$ cd pdf-generator
$ npm install

Congratulations, we have just created a new project with Serverless Stack! Here we get a basic template of a working Serverless Stack app. You can go here for more information on how to get started with Serverless Stack.

First, let’s look at the files we get from bootstrapping the project and customize them wherever we see fit.

1. sst.json

{
  "name": "pdf-generator",
  "region": "us-east-1",
  "main": "stacks/index.js"
}

This file is the entry point to the Serverless Stack app, and here we can define the region of choice, the name of the app, and the main file that Serverless Stack will use.

2. stacks/index.js

 // stacks/index.js
 app.setDefaultFunctionProps({
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

stacks/index.js is the main file declared in sst.json. When building the project, Serverless Stack will use this file as an entry point to our application. This file is pretty simple:

It configures the Lambda functions to use the Node.js 12.x runtime.
It registers a stack called my-stack. This is the name of the CloudFormation stack on AWS. We definitely want to give it a better name than my-stack here, so let’s rename that to pdf-generator.

3. stacks/MyStack.js

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // Create a HTTP API
    const api = new sst.Api(this, "Api", {
      routes: {
        "GET /": "src/lambda.handler",
      },
    });

    // Show the endpoint in the output
    this.addOutputs({
      "ApiEndpoint": api.url,
    });
  }
}

MyStack.js is where we declare the resources we need inside a given CloudFormation stack. This can be anything from a cluster of Lambda functions, API Gateway endpoints, DynamoDB tables, S3 buckets, etc. The full list of resources that we can create with Serverless Stack is listed here in the documentation. For our project we just need one API endpoint with API gateway, so what we are provided with here is already sufficient.

Deploying the project

Now, we can try to deploy this project to AWS. Ensure that the AWS Access Key ID and AWS Secret Access Key are set in the development environment, and then run npx sst start.

# export AWS credentials
export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<secret access key>

# or if the credentials are set in ~/.aws/credentials
export AWS_PROFILE=<AWS profile name>

# Start the app in development mode
npx sst start

After a few minutes, we will have the Cloudformation stack deployed on our AWS account. Keep in mind that the first deployment will take a bit longer than subsequent deployments. When the deployment is done, we should be able to see that it is watching for file changes from our local machine.

threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::ApiGatewayV2::Route | ApiRouteGET8AC7D3F8
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::Lambda::Permission | ApiRouteGETthreadlightlypdfgeneratorpdfgeneratorApiRouteGET7230D6CDPermissionE4542537
threadlightly-pdf-generator-pdf-generator | CREATE_COMPLETE | AWS::CloudFormation::Stack | threadlightly-pdf-generator-pdf-generator

 ✅  threadlightly-pdf-generator-pdf-generator


Stack threadlightly-pdf-generator-pdf-generator
  Status: deployed
  Outputs:
    ApiEndpoint: https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com


==========================
 Starting Live Lambda Dev
==========================

Transpiling Lambda code...
Debug session started. Listening for requests...

Let’s head to the API URL given in the output and we should get a response back like this:

Hello, World! Your request was received at 30/Nov/2021:06:21:13 +0000.

Developing with Serverless Stack

Let’s try updating one of the files provided to see how we can test the changes we have made and will make.

 // stacks/MyStack.js
 const api = new sst.Api(this, "Api", {
   routes: {
-    "GET /": "src/lambda.handler",
+    "GET /downloads/receipt": "src/lambda.handler",
   },
 });

 // stacks/index.js
   runtime: "nodejs12.x"
 });

-new MyStack(app, "my-stack");
+new MyStack(app, "pdf-generator");

Here we did two things:

We updated the API endpoint to GET /downloads/receipt instead of just GET /.
We made the Lambda function return the URL given in the query string.

Once we save these changes we should see Serverless Stack automatically reloading our code and since we changed the API endpoint path, Serverless Stack will need to make changes to our infrastructure (i.e. the API Gateway resources), and whenever it detects that it has to change the infrastructure of our application, it will immediately prompt us to redeploy.

Rebuilding code...
Rebuilding infrastructure...
Done building code
Press ENTER to redeploy infrastructure

Once we hit ENTER, Serverless Stack will automatically update our infrastructure for us: delete our old endpoint, create a new endpoint on GET /downloads/receipt, and hook up our src/lambda.js file to handle that endpoint.

When it’s ready, try hitting the API gateway URL again, but this time append /downloads/receipt?url=https://google.com to it.

// https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://google.com
Hello! You've requested to print the receipt at page https://google.com

Great, now we have learned how to make changes to our Serverless Stack application during development. Let’s go ahead and add Puppeteer and Chrome to our Lambda function.

 // stacks/MyStack.js
 import * as sst from "@serverless-stack/resources";
+import { LayerVersion } from "@aws-cdk/aws-lambda";
+
+const layerArn = "arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:25";

 // Create a HTTP API
 const api = new sst.Api(this, "Api", {
   routes: {
-     "GET /downloads/receipt": "src/lambda.handler",
+     "GET /downloads/receipt": {
+       function: {
+         handler: "src/lambda.handler",
+         // Increase the timeout for generating screenshots
+         timeout: 15,
+         // Load Chrome in a Layer
+         layers: [LayerVersion.fromLayerVersionArn(this, "Layer", layerArn)],
+         // Exclude bundling it in the Lambda function
+         bundle: { externalModules: ["chrome-aws-lambda"] },
+       }
+     },
   },
 });

Here, we are configuring our AWS Lambda function to use a Lambda layer that includes Chrome in our Lambda functions. You can think of these Lambda layers as being similar to NPM packages that you pull for extending your projects except that this is a 3rd party package for extending AWS Lambda functions instead.

With this layer, your AWS Lambda function will boot with the Chrome binary already included. This layer is maintained here. Make sure you’re using the ARN from the right region. In this tutorial, we are using us-east-1 per Serverless Stack’s default, so we are going to pick the ARN for us-east-1.

Now, let’s update our Lambda handler to:

Use Puppeteer to instruct Chrome to navigate to the URL passed in the parameter.
Take a screenshot of the web page and save a PDF out of that. Puppeteer provides a handy page.pdf API to do just that.
Ultimately, stream that PDF file back to the user.

Here’s the code for that, with some comments to help you out:

+import chrome from "chrome-aws-lambda";
+
+const puppeteer = chrome.puppeteer;
+
 export async function handler(event) {
-  return {
-    statusCode: 200,
-    headers: { "Content-Type": "text/plain" },
-    body: `Hello! You've requested to print the receipt at page ${event.queryStringParameters.url}`,
-  };
+  let browser
+  let response
+  try {
+    const { url } = event.queryStringParameters;
+
+    browser = await puppeteer.launch({
+      args: chrome.args,
+      executablePath: await chrome.executablePath,
+    });
+
+    const page = await browser.newPage();
+    // Use A5 size at 150dpi
+    const width = 874
+    const height = 1240
+
+    await page.setViewport({ width, height });
+
+    // Navigate to the url
+    await page.goto(url, { waitUntil: 'networkidle2' });
+
+    // Take the screenshot
+    await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+
+    response = {
+      statusCode: 200,
+      body: JSON.stringify({message: 'Screenshot taken'})
+    };
+  } catch(err) {
+    response = {
+      statusCode: 500,
+      body: JSON.stringify({message: `An error occured. ${err.message}`})
+    }
+  } finally {
+    await browser && browser.close()
+  }
+  return response
 }

Let’s also not forget to install Puppeteer and Chrome.

$ npm install puppeteer puppeteer-core chrome-aws-lambda

Now, give Serverless Stack some time to reload our new changes. Once we have the API endpoint ready to go, let’s head to the browser and hit our API endpoint, passing the URL to the sample receipt page that we did earlier as a query param.

// Visit https://<yourapigatewayurl>.execute-api.us-east-1.amazonaws.com/downloads/receipt?url=https://unwieldy-key.surge.sh/index.html
// Response:
{"message":"Screenshot taken"}

Great! We are getting back a 200 status code and a message saying “Screenshot taken”. That means our code works and there were no exceptions. We’re halfway there.

Now, if we check the project folder, we should now see a new file called receipt.pdf there. This is the receipt.pdf captured by Puppeteer. We can open this file and verify that this is the demo receipt page that I deployed earlier that we passed to the endpoint.

We can also hit the API endpoint with the second sample receipt URL (i.e. url=https://unwieldy-key.surge.sh/receipt-2.html) and we should see the receipt.pdf file gets replaced with the receipt from the new URL.

Note that in a real-world application, the URL you would want to screenshot might need some form of authentication, so you are going to have to figure that out for your application. In this case the sample receipt page is accessible publicly, so we don’t run into this problem, but chances are your users’ receipt page will not be accessible publicly like this.

One way you could solve this is to create a special user account that has permission to visit these authenticated pages for this purpose. Then, prior to visiting the receipt page, you’d program Puppeteer to first log in as the special user and then head to the receipt page URL to be screenshot.

However, we are only halfway there. We can see the receipt.pdf file now only because the receipt.pdf file was made available to us by Serverless Stack, which automatically downloads that file to our local machine in development mode. In a real AWS Lambda execution, the file will remain in the filesystem of that Lambda execution context instead, and the user will not see it.

Hence we have to read and stream that file back to the user instead of returning a JSON message saying “Screenshot taken”. Getting a 200 status code is a great start, but that still does not give our users the actual PDF that they wanted, so let’s update our Lambda handler to do just that:

 // src/lambda.js
+import * as fs from "fs";

 // Take the screenshot
 await page.pdf({path: 'receipt.pdf', width: width + "px", height: height + "px", printBackground: true});
+const buffer = fs.readFileSync('receipt.pdf')

 response = {
   statusCode: 200,
-  body: JSON.stringify({message: 'Screenshot taken'})
+  headers: {'Content-type' : 'application/pdf'},
+  body: buffer.toString('base64'),
+  isBase64Encoded : true,
 };

The changes this time are pretty simple:

Import the fs module from Node.js, which will give us access to readFileSync API that can read files on the disk.
Read the generated receipt.pdf file as a Buffer.
Return the file’s Buffer as a base64 string in the response body.
Set the content-type to be application/pdf. Setting the content-type appropriately will tell the browser what to do with the file. Some browsers will display the PDF in an in-browser PDF reader, while others will download the file.

That’s it! Now let’s test the change. Hit the same URL again.

Bingo, the endpoint now returns the PDF to the user! Now we can integrate this endpoint with our main application to delegate these on-demand PDF generation to managed FaaS infrastructure.

Conclusion

FaaS platforms are a really great way to offload intermittent and on-demand traffic from our main application. This ensures our application runs smoother and can serve our general customers better, while at the same time handling these on-demand tasks really well.

Caveats

Building a full-blown application on the stack is also possible, but in my personal experience, it comes with its own complexities and tradeoffs that may or may not be worth the engineering time.

For example, databases are an issue with going full-blown development on FaaS. Because FaaS can scale indefinitely, the number of connections to the database is a limitation.

You can switch to FaaS-friendly databases such as DynamoDB or FaunaDB, but these are NoSQL databases and require a different mindset in structuring the application data than the simpler relational databases and that can cost tremendous engineering time. Relational databases are easier to work with for most types of applications.

Nonetheless, more and more tools are being developed to tackle these complexities, such as PlanetScale and Serverless Aurora v2 (still in preview). All in all, it is definitely a trend worth following closely.

Convenient Reporting with Jasper

2020-04-28T00:00:00+00:00

Business Intelligence (BI) reporting is a huge problem space in custom software. There’s a wide range of business needs for looking at past and predictive behavior. Building a reporting tool can be a very cost effective way to get this data, especially compared to writing individual queries or manually generating reports.

I’ve been working with Jasper in the Java project space and wanted to write about some research I’ve collected on the topic.

JasperReports takes .jrxml files as input and outputs a .jasper report. Possible output targets include:

Screen
Printer
PDF
HTML
Excel files
RTF
ODT
CSV
XML

Jasper history

June 2001: Teodor Danciu began working on JasperReports.
September 2001: Jasper was registered on SourceForge.
November 2001: JasperReports 0.1.5 was released.
2004: Panscopic teamed up with Teodor Danciu, acquired ownership of the product and changed its name to Jaspersoft.
2005: JasperReports 1.0 was released.
2007: Brian Gentile became CEO of the company.
2014: TIBCO acquired Jaspersoft for ~$185 million.

Best reporting tools

Let’s compare some popular reporting tools:

JasperReports is a free and open source Java-based reporting tool, which supports lots of possible outputs, as mentioned earlier. Generating reports can be difficult if you’re less technical. More technical aspects can be more difficult as well; embedding JasperReports into a project is not necessarily simple, but once it’s done, the tool will be reliable.
Crystal Reports supports many inputs, including Access, Excel, XML, ODBC, and JDBC. It also has good multi-language support. It’s easy to embed into a .NET project, but software updates are unstable. The process can be very slow and there is no control of data-level security. A trial version is offered, but if one wants to use it long-term, then the price is a one-time payment of $495 or more, for larger companies.
Domo is another a popular reporting tool. It provides a trial version, and a 5 user plan costs $5700/year*.
Zoho analytics is an easy-to-use BI reporting tool, priced between $22–$445, depending on the number of users and data.
Host analytics is a great tool for finance automation. Pricing not publicized.
Tableau is an excellent reporting tool, with a thriving community online, but its quote-based price is high.
Pentaho is a Java-based reporting tool, which provides data integration, online analytical processing and reporting, among other features. Pentaho offers a 30-day trial period. Contract pricing isn’t disclosed.

So, if you are writing software and already use Java, or using Java reporting is an option, JasperReports is a great choice. It supports a variety of outputs, is free to use, and open source.

Installing JasperReports Server

To install JasperReports Server, you need a computer with a fully functional JRE (Java Runtime Environment). The server might be Tomcat or GlassFish. An RDBMS is also needed. since JasperReports has its own database, this could be PostgreSQL, Oracle, MySQL, DB2, or SQL Server. JasperReports prefers PostgreSQL and Tomcat, so these will be included with an automatic install. You may choose to use your existing Tomcat/PostgreSQL or have it install them as well.

Manual installation is also possible, as described here. At my first encounter with Jasper I installed Tomcat with the installer and used it for generating JasperReports, while the application I was working with was running WildFly (formerly JBoss), using a MySQL database. Needless to say, this was unnecessary, but I was not aware of that at the time. JasperServer can be configured to work with JBoss and MySQL as well.

The core of JasperReports is the JasperReports Library, which was already integrated into the project I was working with and is integrated into JasperReports Server as well as into popular IDEs, like TIBCO Jaspersoft Studio or iReport Designer.

Database

JasperReports provides example databases for imaginary companies, like FoodMart:

As we can see these are normal tables, having a primary key and some other fields.

.jrxml

jrxml, which stands for Jasper XML, contains report definitions in XML format. This type of file can be edited as code or visually to be compiled into .jasper files. The community provides samples that can be used and understood. Among them we can find a JFreeChart sample, which contains a jrxml file, a preview HTML and an actual PDF. The .jrxml file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jasperReport PUBLIC "-//JasperReports//DTD Report Design//EN" "http://jasperreports.sourceforge.net/dtds/jasperreport.dtd">

<jasperReport name="JFreeChartReport" pageWidth="595" pageHeight="842" columnWidth="515" leftMargin="40" rightMargin="40" topMargin="50" bottomMargin="50" scriptletClass="JFreeChartScriptlet">
  <variable name="Chart" class="net.sf.jasperreports.engine.JRRenderable" calculation="System"/>
  <title>
    <band height="742">
      <line>
        <reportElement x="0" y="0" width="515" height="1"/>
        <graphicElement/>
      </line>
      <staticText>
        <reportElement x="0" y="10" width="515" height="30"/>
        <textElement textAlignment="Center">
          <font size="22"/>
        </textElement>
        <text><![CDATA[JFreeChart Sample]]></text>
      </staticText>
      <textField>
        <reportElement x="0" y="50" width="515" height="50"/>
        <textElement textAlignment="Center">
          <font size="12"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA["This sample uses JFreeChart Version 1.0.0-pre2\n" + "Written by David Gilbert (david.gilbert@object-refinery.com) and others.\n" + "(C)opyright 2000-2004, by Object Refinery Limited and Contributors."]]></textFieldExpression>
      </textField>
      <image scaleImage="Clip" hAlign="Center" hyperlinkType="Reference">
        <reportElement x="0" y="110" width="515" height="300"/>
        <graphicElement/>
        <imageExpression class="net.sf.jasperreports.engine.JRRenderable"><![CDATA[$V{Chart}]]></imageExpression>
        <hyperlinkReferenceExpression><![CDATA["http://www.jfree.org/jfreechart"]]></hyperlinkReferenceExpression>
      </image>
    </band>
  </title>
</jasperReport>

It starts with the xml tag, specifying that this file should be interpreted as XML. Then comes the DOCTYPE and finally the jasperReport node, which contains the actual report nodes. A variable is defined, called Chart, which is used later in the inner XML of the image node. A hyperlink is defined for the image. The preview for this report looks like this:

Don’t worry about the broken images; this is just the preview, the actual result looks like this:

Nice, isn’t it?

Data source

It’s nice to generate reports, but in many cases the content is not fully known at programming time. It’s quite possible that we need to provide some input for the template. For this purpose, the interface of JRDataSource was defined to be iterated with a .next() method and is readable via the .getFieldValue() method. To make sure that we can read fields, another interface, JRField was defined as well. We therefore will need to use implementations of these classes, possibly writing our own if the currently available implementations are not fulfilling our needs.

Let’s consider a datasource sample, also taken from the community. It has this .jrxml template:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jasperReport PUBLIC "-//JasperReports//DTD Report Design//EN" "http://jasperreports.sourceforge.net/dtds/jasperreport.dtd">

<jasperReport name="DataSourceReport" pageWidth="595" pageHeight="842" columnWidth="515" leftMargin="40" rightMargin="40" topMargin="50" bottomMargin="50">
  <style name="Arial_Normal" isDefault="true" fontName="Arial" fontSize="12" isBold="false" isItalic="false" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <style name="Arial_Bold" isDefault="false" fontName="Arial" fontSize="12" isBold="true" isItalic="false" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica-Bold" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <style name="Arial_Italic" isDefault="false" fontName="Arial" fontSize="12" isBold="false" isItalic="true" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica-Oblique" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <parameter name="ReportTitle" class="java.lang.String"/>
  <parameter name="DataFile" class="java.lang.String"/>
  <field name="id" class="java.lang.Integer"/>
  <field name="name" class="java.lang.String"/>
  <field name="street" class="java.lang.String"/>
  <field name="the_city" class="java.lang.String">
    <fieldDescription>me.me.city</fieldDescription>
  </field>
  <variable name="CityNumber" class="java.lang.Integer" incrementType="Group" incrementGroup="CityGroup" calculation="Count">
    <variableExpression><![CDATA[Boolean.TRUE]]></variableExpression>
  </variable>
  <group name="CityGroup" minHeightToStartNewPage="60">
    <groupExpression><![CDATA[$F{the_city}]]></groupExpression>
    <groupHeader>
    <band height="20">
      <textField evaluationTime="Group" evaluationGroup="CityGroup" bookmarkLevel="1">
        <reportElement mode="Opaque" x="0" y="5" width="515" height="15" backcolor="#c0c0c0" style="Arial_Bold"/>
        <box leftPadding="10" bottomBorder="1Point"/>
        <textFieldExpression class="java.lang.String"><![CDATA["  " + String.valueOf($V{CityNumber}) + ". " + String.valueOf($F{the_city})]]></textFieldExpression>
        <anchorNameExpression><![CDATA[String.valueOf($F{the_city})]]></anchorNameExpression>
      </textField>
    </band>
    </groupHeader>
    <groupFooter>
    <band height="20">
      <staticText>
        <reportElement x="400" y="1" width="60" height="15" style="Arial_Bold"/>
        <textElement textAlignment="Right"/>
        <text><![CDATA[Count :]]></text>
      </staticText>
      <textField>
        <reportElement x="460" y="1" width="30" height="15" style="Arial_Bold"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.Integer"><![CDATA[$V{CityGroup_COUNT}]]></textFieldExpression>
      </textField>
    </band>
    </groupFooter>
  </group>
  <title>
    <band height="70">
      <line>
        <reportElement x="0" y="0" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField isBlankWhenNull="true" bookmarkLevel="1">
        <reportElement x="0" y="10" width="515" height="30" style="Arial_Normal"/>
        <textElement textAlignment="Center">
          <font size="22"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$P{ReportTitle}]]></textFieldExpression>
        <anchorNameExpression><![CDATA["Title"]]></anchorNameExpression>
      </textField>
      <textField isBlankWhenNull="true">
        <reportElement x="0" y="40" width="515" height="20" style="Arial_Normal"/>
        <textElement textAlignment="Center">
          <font size="14"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$P{DataFile}]]></textFieldExpression>
      </textField>
    </band>
  </title>
  <pageHeader>
    <band height="20">
      <rectangle>
        <reportElement x="0" y="5" width="515" height="15" forecolor="#333333" backcolor="#333333"/>
        <graphicElement/>
      </rectangle>
      <staticText>
        <reportElement mode="Opaque" x="0" y="5" width="55" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <textElement textAlignment="Center"/>
        <text><![CDATA[ID]]></text>
      </staticText>
      <staticText>
        <reportElement mode="Opaque" x="55" y="5" width="205" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <text><![CDATA[Name]]></text>
      </staticText>
      <staticText>
        <reportElement mode="Opaque" x="260" y="5" width="255" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <text><![CDATA[Street]]></text>
      </staticText>
    </band>
  </pageHeader>
  <detail>
    <band height="15">
      <textField bookmarkLevel="2">
        <reportElement x="0" y="0" width="50" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.Integer"><![CDATA[$F{id}]]></textFieldExpression>
        <anchorNameExpression><![CDATA[$F{name} + " (" + $F{id} + ")"]]></anchorNameExpression>
      </textField>
      <textField isStretchWithOverflow="true">
        <reportElement positionType="Float" x="50" y="0" width="200" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{name}]]></textFieldExpression>
      </textField>
      <textField isStretchWithOverflow="true">
        <reportElement positionType="Float" x="250" y="0" width="265" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" rightBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{street}]]></textFieldExpression>
      </textField>
    </band>
  </detail>
  <pageFooter>
    <band height="40">
      <line>
        <reportElement x="0" y="10" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField>
        <reportElement x="200" y="20" width="80" height="15"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.String"><![CDATA["Page " + String.valueOf($V{PAGE_NUMBER}) + " of"]]></textFieldExpression>
      </textField>
      <textField evaluationTime="Report">
        <reportElement x="280" y="20" width="75" height="15"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[" " + String.valueOf($V{PAGE_NUMBER})]]></textFieldExpression>
      </textField>
    </band>
  </pageFooter>
  <lastPageFooter>
    <band height="60">
      <textField bookmarkLevel="1">
        <reportElement x="0" y="10" width="515" height="15"/>
        <textElement textAlignment="Center"/>
        <textFieldExpression class="java.lang.String"><![CDATA["There were " +
          String.valueOf($V{REPORT_COUNT}) +
          " address records on this report."]]></textFieldExpression>
        <anchorNameExpression><![CDATA["Summary"]]></anchorNameExpression>
      </textField>
      <line>
        <reportElement x="0" y="30" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField>
        <reportElement x="200" y="40" width="80" height="15"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.String"><![CDATA["Page " + String.valueOf($V{PAGE_NUMBER}) + " of"]]></textFieldExpression>
      </textField>
      <textField evaluationTime="Report">
        <reportElement x="280" y="40" width="75" height="15"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[" " + String.valueOf($V{PAGE_NUMBER})]]></textFieldExpression>
      </textField>
    </band>
  </lastPageFooter>
</jasperReport>

As we can see, there are fields defined like id, name, street, and the_city. We also have a group called CityGroup, so when the items from the data source are iterated through, their group is known via the_city. It’s worth looking at how the paging works. The key is evaluationTime, which is telling the engine to not evaluate a given element at iteration time, but rather when an event occurs. evaluationTime="Report" means that we need to evaluate the value when the Report event occurs. At that time $V{PAGE_NUMBER} already has the value equal to the number of pages. Let’s see the preview:

Again, we don’t need to worry about the missing image icons, since this is only a preview and this is the actual result:

Since we have a few interfaces that we need to respect, we can easily integrate Hibernate with our Jasper reports as data source, we just need to make sure we are using the field and data source interfaces they defined. Here we have a few examples, notably the following:

List cats = session.find("from eg.Cat");

Map parameters = new HashMap();
parameters.put("Title", "The Cat Report");

InputStream reportStream = this.class.getResourceAsStream("/the-cat-report.xml");
JasperDesign jasperDesign = JasperManager.loadXmlDesign(reportStream);
JasperReport jasperReport = JasperManager.compileReport(jasperDesign);

JRBeanCollectionDataSource ds = new JRBeanCollectionDataSource(cats);
JasperPrint jasperPrint = JasperManager.fillReport(jasperReport, parameters, ds);

JasperManager.printReportToPdfFile(jasperPrint, "the-cat-report.pdf");

We gather a List of Cat instances, then define some parameters, like Title, followed by the creation of an InputStream object which will be used to generate a JasperDesign object, which is then the input for the instantiation of JasperReport. Now, we define cats to be our data source and call fillReport, passing the jasperReport object we have just created, the parameters, which contain the title and the data source. Finally, we print the report to a PDF. Note that you can use a compiled .jasper file as input for getResourceAsStream.

TIBCO Jaspersoft Studio

This IDE is excellent for compiling the jrxml files into jasper files. I use it as a desktop application. There are three tabs:

Design
Source
Preview

In the Design tab one can interactively design the report, not worrying about source code, XML, and the like, which enables non-programmers to work on report creation as well. Handling the variables, parameters and fields needs some algorithmic understanding, but nonetheless, code is generated interactively in a point-and-click manner. The designer cannot do everything we need, but the Source tab comes to the rescue if needed. Sometimes I do not even need to use the Source tab.

There’s still one problem: testing is a high-cost operation in the project I am using Jasper for. We need to build the .jasper files, but that’s low-cost. The higher cost is that we need to build the application that actually generates the Jasper reports in order to test and then deploy it with JBoss. Then the actual test in the application occurs, so a test takes more than a minute. Luckily there is a Preview tab, where I can more or less see whether it’s a good idea to invest time into building and deploying, or if we need to do some tweaking first.

Long story short

For more information, see Jaspersoft’s website.

JasperReports is free and open source. However, you have to pay a fee to consult the help documentation. It’s easily a competitor in quality with other reporting tools, which are less reasonably priced. When I first had to work with Jasper reports, I didn’t know anything about it, but was able to complete the tasks at hand in a few hours, which shows that getting into the Jasper universe isn’t too hard. One may have difficulty understanding .jrxml files at first, but trust me, it’s worth it.

If you are already using Java and want to generate reports, Jasper is a good candidate. It can periodically generate reports for you and even send them attached to an email to you. I couldn’t write better closing words than a poster I found:

* Domo plan pricing was found at yurbi.com.

MediaWiki extensions and wfLoadExtensionMessages

2014-05-09T00:00:00+00:00

Image by Susan Drury

Upgrading MediaWiki can be a challenging task, especially if you use a lot of extensions. While the core upgrade process usually goes smoothly, it’s rare you can upgrade a major version or two without having to muddle with your collection of extensions. Extensions are bits of code that extend what MediaWiki can do. Only a few are packaged with and maintained alongside MediaWiki itself—the great majority are written by third-party developers. When the MediaWiki API changes, it is up to those developers to update their extension so it works with the new version of MediaWiki. This does not always happen. Take for example one of the more common errors seen on a MediaWiki upgrade since 1.21 was released:

[Tue May 06 11:21:52 2014] [error] [client 12.34.56.78] PHP Fatal error:  Call to undefined function wfLoadExtensionMessages() in /home/beckett/mediawiki/extensions/PdfExport/PdfExport.php on line 83, referer: http://test.ziggy.com/wiki/Main_Page

This is because the wfLoadExtensionMessages function, which many extensions use, has been deprecated since MediaWiki version 1.16 and was finally removed in 1.21, resulting in the error seen above. Luckily, this function has been a no-op since 1.16, so it is safe to comment it out and/or make a dummy function in your LocalSettings.php file (see below).

Sadly, the release notes for 1.21 make no mention of this fairly major change. Let’s walk through as if we didn’t know anything about it and see how we could solve the given error with the help of git. For this example, we’ll use the Pdf Export extension, which allows you to export your wiki pages into PDF form. A pretty handy extension, and one which completely fails to work in MediaWiki version 1.21 or better.

First, let’s verify that wfLoadExtensionMessages does not exist at all in version 1.21 of MediaWiki. For these examples, I’ve checked out the MediaWiki code via git, and am relying on the fact that lightweight git tags were made for all the versions we are interested in.

$ git clone https://github.com/SemanticMediaWiki/SemanticMediaWiki.git mediawiki
$ cd mediawiki
$ git grep wfLoadExtensionMessages 1.21.0
1.21.0:HISTORY:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php

A nice feature of git grep is the ability to simply use a tag after the search string. In this case, we see that the only mention of wfLoadExtensionMessages in the entire codebase is an old mention of it in the history file. Let’s see what version that bug is from:

$ git grep -n wfLoadExtensionMessages 1.21.0
1.21.0:HISTORY:5280:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php
$ git show 1.21.0:HISTORY | head -5280 | tac | grep '===' -m1
=== Bug fixes in 1.12 ===

That message is from way back in version 1.12, and doesn’t concern us. Let’s take a look at what tags exist in the 1.20 branch so we can scan the latest one:

$ git tag | grep '^1\.20'
1.20.0
1.20.0rc1
1.20.0rc2
1.20.1
1.20.2
1.20.3
1.20.4
1.20.5
1.20.6
1.20.7
1.20.8

Now we can peek inside version 1.20.8 and see what that function did before it was removed. By using the -A and -B (after and before) arguments to grep, we can see the entire function in context:

$ git grep wfLoadExtensionMessages 1.20.0
1.20.0:HISTORY:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php
1.20.0:includes/GlobalFunctions.php:function wfLoadExtensionMessages() {
$ git show 1.20.8:includes/GlobalFunctions.php | grep -B6 -A2 LoadExtensionMessages
/**
 * Load an extension messages file
 *
 * @deprecated since 1.16, warnings in 1.18, remove in 1.20
 * @codeCoverageIgnore
 */
function wfLoadExtensionMessages() {
    wfDeprecated( __FUNCTION__, '1.16' );
}

Thus wfLoadExtensionMessages was basically a no-op in MediaWiki version 1.20, with the caveat that it will write a deprecation warning to your error log (or, in modern versions, the debug log unless $wgDevelopmentWarnings is set). Next we want to find the last time this function did something useful—which should be version 1.15 according to the comment above. Thus:

$ git show 1.15.0:includes/GlobalFunctions.php | grep -A4 LoadExtensionMessages
function wfLoadExtensionMessages( $extensionName, $langcode = false ) {
    global $wgExtensionMessagesFiles, $wgMessageCache, $wgLang, $wgContLang;

    #For recording whether extension message files have been loaded in a given language.
    static $loaded = array();

So, it’s a pretty safe bet that unless you are upgrading from 1.15.0 or earlier, it should be completely safe to remove it. When was 1.16.0 released? There are no dates in the HISTORY file (shame), but the date it was tagged should be a good guess:

$ git show 1.16.0 | grep -m1 Date
Date:   Wed Jul 28 07:11:03 2010 +0000

So what should you do with extensions that are still using this deprecated function? There are two quick solutions: comment it out inside the extension, or add a dummy function to your version of MediaWiki.

Changing the extension itself is certainly quick and easy. To get the PdfExport extension to work, we only have to comments out two calls to wfLoadExtensionMessages inside of the file PdfExport.php, and one inside of PdfExport_body.php. The diff:

$ git difftool -y -x "diff -u1"
--- /tmp/7YqvXv_PdfExport.php 2014-05-08 12:45:03 -0400
+++ PdfExport.php             2014-05-08 12:34:39 -0400
@@ -82,3 +82,3 @@
   if ($img_page > 0 || $img_page === false) {
-        wfLoadExtensionMessages('PdfPrint');
+        //wfLoadExtensionMessages('PdfPrint');
                $nav_urls['pdfprint'] = array(
@@ -92,3 +92,3 @@
 function wfSpecialPdfToolbox (&$monobook) {
-          wfLoadExtensionMessages('PdfPrint');
+          //wfLoadExtensionMessages('PdfPrint');
           if (isset($monobook->data['nav_urls']['pdfprint']))
--- /tmp/7gO8Hz_PdfExport_body.php   2014-05-08 12:45:03 -0400
+++ PdfExport_body.php               2014-05-08 12:34:44 -0400
@@ -44,3 +44,3 @@
            // For backwards compatibility
-             wfLoadExtensionMessages('PdfPrint');
+             //wfLoadExtensionMessages('PdfPrint');

A better way is to add a dummy function to LocalSettings.php. This ensures that any extension we add in the future will continue to work unmodified. Just throw this at the bottom on your LocalSettings.php:

function wfLoadExtensionMessages() { }

Probably the best overall solution is to not only add that to your LocalSettings.php, but to try to get the extension changed as well. You can notify the author, or try to fix it yourself and release a new version if the extension has been abandoned. You might also look to see if the extension has been superseded by a different extension, as sometime happens.

While there may be other compatibility issues when upgrading MediaWiki, for some extensions (such as PdfExport), this is the only change needed to make it work again on newer versions of MediaWiki!

Little Spree Big Performance Problems

2013-08-05T00:00:00+00:00

Recently I worked on an online food store serving an area with very little infrastructure. As a result, the orders tended to be really big with lots of products.

The website worked in the following environment:

Ruby 1.9.2
Spree 0.60
Heroku, Bamboo stack
PostgreSQL 9.2.4

H12 timeout errors

The performance problems started when we migrated Bamboo to Cedar on Heroku and replaced Thin webserver with Unicorn. We started getting a lot of Heroku Request timeout errors - H12:

The problems happened mostly when logging in to admin dashboard or during the checkout for the certain orders. H12 errors occur when a HTTP request takes longer than 30 seconds to complete. For example, if a Rails app takes 35 seconds to render the page, the HTTP router returns a 503 after 30 seconds and abandons the incomplete Rails request for good. The Rails request will keep working and logging the normal errorless execution. After completion, the request will indefinitely hang in the application dyno.

We started debugging H12: we set Unicorn timeout to 20 seconds to prevent the runaway requests and installed the rack-timeout gem with the timeout of 10 seconds to raise an error on a slow request. It all came down to a trivial database timeout!

The application source has not changed during the transition from Bamboo to Cedar, but apparently Cedar/Unicorn is much more sensitive to the troubled code. Below is the list of performance bottlenecks and solutions in Spree. Some of them exist in version 0.60 only, but a lot of them are still present in Spree 1.x, which means that your application may have them too.

Issue #1: Real-time reports

Let’s take a closer look at the earlier database timeout code. It came from Admin Dashboard and admin/overview_controller.rb.

The “Best Selling variants" report was being calculated real-time right in the web process:

def best_selling_variants
  li = LineItem.includes(:order).
      where("orders.state = 'complete'").
      sum(:quantity, :group => :variant_id, :limit => 5)
  ...
end

Record counts in the database are large enough to crash the application on Heroku with the database timeout:

irb(main):001:0> LineItem.count
=> 162279
irb(main):002:0> Order.count
=> 13027
irb(main):003:0> Variant.count
=> 14418

Other reports on the Dashboard experience the same problem. They would cause the timeout in turns when logging into Admin:

top_grossing_variants
best_selling_taxons
last_five_orders
biggest_spenders

Solution

Fortunately, in Spree 1.x the internal reporting system has been replaced with Jirafe:

If switching to Spree 1.x is not an option, another way is to move the calculation into a background job, using, for example, delayed_job gem and Heroku Scheduler Addon:

task :statistics => :environment do
  Delayed::Job.enqueue StatisticsJob.new
end

Issue #2: Large numbers

It is an established fact that humans eat a lot! Think about an order of a thousand Heineken 6-pack cans…

Or even something like this:

or this:

Spree, both 0.60 and 1.x, proved to have a huge problem if an order has a lot of line items and/or a large quantity of single line items. The potentially dangerous code can be found all over the place. Consider the following example:

class InventoryUnit < ActiveRecord::Base
  def self.destroy_units(order, variant, quantity)
    variant_units = order.inventory_units.group_by(&:variant_id)[variant.id].sort_by(&:state)
    quantity.ceil.times do
      inventory_unit = variant_units.shift
      inventory_unit.destroy
    end
  end
end

Now imagine what will happen with the order from the first screenshot. We have 15100 inventory units for that one. They will be meticulously destroyed from the inventory one by one in a loop after the checkout. This method was born to crash the application!

Solution

A simple mindful refactoring was enough to solve the problem for me. There is no need to call “destroy” in the loop for every single inventory unit because we can use the efficient “destroy_all” method. I’m sure this can be optimized further, but it was enough to get rid of the timeout:

def self.destroy_units(order, variant, quantity)
  variant_units = order.inventory_units.
    group_by(&:variant_id)[variant.id].
    sort_by(&:state)
  variant_units = variant_units.shift(quantity.ceil)
InventoryUnit.
    where(:order_id => order.id,:variant_id => variant.id).
    order('state asc').limit(quantity.ceil).
    destroy_all
end

Issue #3: Real-time emails

All emails in Spree are sent in real-time.

class Order < ActiveRecord::Base
  def finalize!
    ...
    OrderMailer.confirm_email(self).deliver
    ...
  end
end

Why is it bad? Let’s look at the following example from my application:

def confirm_email(order)
  attachments["invoice.pdf"] = {
    'Content-type' => 'application/pdf',
    :content => OrderInvoice.new.to_pdf(order)}

  mail(:subject  => 'Order #' + order.number,
       :from   =>    Spree::Config[:order_from],
       :to => order.email)
end

In my application “confirm_email” was overridden and generated the PDF invoice. The invoice listed all the products in the order and had about 200 lines in it. Again, it lead to the H12 timeout error.

Solution

All emails should be sent in the background rather than in the web request. First, because network operations can take a long time, and second, because generating an email can also be slow. For example, sending in the background can be accomplished using delayed_job gem:

OrderMailer.delay.confirm_email(self)

Issue #4: Lazy-loading

Ecommerce objects are usually complicated with a lot of associations. This is totally fine as long as you eager-load the associations that will be used most with the loaded object later on. In most cases, Spree does not preload associations for its orders. For example, in spree/base_controller.rb:

@order = Order.find_by_number! params[:order_id]

As the result, here is what I see in server console while loading the order display page on the frontend:

…And five more screens like this! The associations of the order — line items, variants and products — generate an additional chain of queries to the database during lazy-loading.

Solution: Eager-loading

If I modify the line from the controller like this…

@order = Order.where(:number => params[:order_id])
         .includes(:line_items => {:product => :taxons, :variant => [:product, :option_values]})
         .includes(:adjustments).first

…SQL queries will shrink down to this:

Eager-loading did the trick. Of course, not everything needs to be loaded eagerly, and the solution varies from case to case. It worked like charm in my case.

Issue #5: Dangerous code all over the place

There a lot of places in the Spree source code that are not optimized for performance. We don’t need to look far for an example, because there is another killer method right near the “destroy_units” one we inspected earlier!

class InventoryUnit < ActiveRecord::Base
  def self.create_units(order, variant, sold, back_order)
    shipment = order.shipments.detect {|shipment| !shipment.shipped? }
    sold.ceil.times {
      order.inventory_units.create(:variant => variant, :state => "sold", :shipment => shipment)
    }
    back_order.ceil.times {
      order.inventory_units.create(:variant => variant, :state => "backordered", :shipment => shipment)
    }
  end
end

And then:

order.line_items.each do |line_item|
  back_order = determine_backorder(order, variant, quantity)
  sold = quantity - back_order
  create_units(order, line_item.variant,  sold, back_order)
end

I received a timeout either in the “sold” or the “backorder” loop, but there wasn’t a time when I didn’t receive a timeout!

Solution

def self.create_units(order, variant, sold, back_order)
  shipment = order.shipments.detect {|shipment| !shipment.shipped? }
  values = sold.ceil.times.to_a.map { "(#{order.id},#{variant.id},'sold',#{shipment.id})" } +
    back_order.ceil.times.to_a.map { "(#{order.id},#{variant.id},'backordered',#{shipment.id})" }
  values.in_groups_of(500, false) do |group|
    InventoryUnit.connection.execute("INSERT INTO inventory_units(order_id, variant_id, state, shipment_id) VALUES #{group.join(',')}")
  end
end

Another example: every line item has the after_create and after_save callbacks. The callback invokes the Order.update! method. Order.update! method calls update_totals method…twice throughout the method.

def update_totals
  self.payment_total = payments.completed.map(&:amount).sum
  self.item_total = line_items.map(&:amount).sum
  self.adjustment_total = adjustments.map(&:amount).sum
  self.total = item_total + adjustment_total
end

Now imagine the order from the second screenshot with a lot of line items. During the checkout “update_totals” will be called each time the line item is saved. Typically, this line would produce a timeout, because the “line_items” association was, of course, not preloaded:

line_items.map(&:amount).sum

I couldn’t list every circumstance like that, because it would require a lot of context to explain the catch, but I can still say many times …

No long-running tasks in the web request!

Be it Spree, Heroku, or any other context, environment or platform, please, never do the following in the web process:

(!!!) Heavy database usage (slow or numerous queries, N+1 queries)
Sending an email
Accessing a remote API (posting to Twitter, querying Flickr, etc.)
Rendering an image or PDF
Heavy computation (computing a fibonacci sequence, etc.)

Say “No” to all these things to ensure a much happier life for your application!

Generating PDF documents in the browser

2013-03-15T00:00:00+00:00

What you will learn

How to generate PDF documents in the browser with JavaScript
How to generate them out of normal HTML
How to open those PDFs in new windows
How to render them inline inside the DOM

Introduction

Every once in a while as web developers we face the challenge of providing PDF documents for the data we have persisted in the database.

The usual approach is to:

generate the document with a library / DSL in the language of the backend
or—generate normal html view and use a utility like wkhtmltopdf

That works nice, but what if you’re developing an SPA app which only consumes JSON data from its backend? Imagine a scenario when the backend isn’t capable of producing responses other than JSON data. What could you do there?

The solution

Thanks to some very bright folks behind the jsPDF library we have both above mentioned options right inside the browser.

I strongly encourage you to visit their website. There is a nice live coding editor set up which reflects in real time the PDF your code is producing.

Using the DSL

The whole process looks like this:

# 1. create jsPDF object:
doc = new jsPDF()

# 2. put something interesting in there:
doc.setFontSize(22)
doc.text(20, 20, 'This is a title')
doc.setFontSize(16)
doc.text(20, 30, 'This is some normal sized text underneath.')

# 3. choose some useful outlet:
doc.output('dataurlnewwindow', {})

Where is the documentation?

One thing I didn’t like about the experience which was really great otherwise, was the fact that you could browse the Internet for hours without luck, looking for some useful docs for the library.

For all of you who don’t know the shortcut yet, here it is:

git clone https://github.com/MrRio/jsPDF.git

And in the doc directory you will find nicely generated docs out of the project sourcesvery neat!

Can I generate it just from HTML?

Yes you can. On the project’s website the authors are warning us that this feature is still experimental. But I have to say that last time I used it, all went well and the code produces nice PDF to this day (yay!).

doc.fromHTML $('.report_web').get(0), 10, 10,
  'width': 170,
  'elementHandlers': 
    'LI': (el, renderer) =>
      if renderer.y > 250
        doc.addPage()
        renderer.y = 10
      else
        renderer.y += 10
      false
    'H1': (el, renderer) =>
      doc.setFontSize(14)
      doc.setFontStyle('bold')
      doc.text($(el).text(), 10, renderer.y)
      doc.setFontSize(12)
      doc.setFontStyle('normal')
      true
doc.output 'dataurlnewwindow', {}

Take a look at the code above. #fromHtml method takes the DOM element, margins, and a hash of options. From my experience the elementHandlers attribute seems to be mandatory (I got errors without it).

Every element handler is just a pair of capitalized tag name and the function which takes DOM element and the renderer context.

The renderer variable contains data of the current paragraph, the pdf internal object, settings object and x and y. You have to remember to change x and y variables yourself after creating some content.

One last thing to remember is that the handler can return either true or false. If it returns true, then the engine will not create content out of the given DOM element on its own. Otherwise, after your handler finishes its execution, the engine will generate the content for you. So you can use those handlers to munge a few things in the output document or you can take over the whole control and do it your way.

Open PDF output in a new window

You may have notice in one of above examples, that we were simply creating new windows with PDF documents there.

To recap, opening PDF documents in new window:

doc.output 'dataurlnewwindow', {}

Render PDFs in the DOM

Nesting the PDF document inside the DOM seems way cooler. To do so:

data = doc.output 'dataurlstring', {}
$('#report').html "<iframe src='#{data}'></iframe>"

You will notice that the #output method takes a string which tells it how you’d like to generate the PDF. If you specify the ‘dataurlstring’ it will return a string containing the url data you use in your iframe. That is how the last example works.

There is much more to jsPDF than shown in this short post. If you’re interested, you can browse the docs included in the projects repository. There are a lot of useful methods for creating almost any type of document you would want.

Appending one PDF to another using PDF Toolkit

2011-11-22T00:00:00+00:00

Ever need to manipulate PDFs? Prefer the command line? Us too. Imagine you have a contract in PDF format. When people print, sign, and re-scan the contract, that’s good documentation of the signature, but the clarity of the original machine-readable text is lost and the the file’s size is unnecessarily large. One solution is to append the scanned signature page to the original contract document.

There are many PDF editors out there which address this need. One command line solution that works well is PDF Labs’s PDF Toolkit. Let’s look at how we would use PDF Toolkit to append one document to another.

pdftk contract.pdf scanned_contract.pdf cat output original_and_signed_contract.pdf

With this command we now have both contracts in their entirety. What we really want is to just take the signature page and append it. Let’s revise our command a bit to only take the signature page using what PDF Toolkit calls handles.

pdftk A=contract.pdf B=scanned_contract.pdf cat A B5 output contract_with_signature_attached.pdf

We’ve assigned each document to a handle (A and B), which allows us to define the order of the output as well as the pages we want to select for the output. With the argument B5 PDF Toolkit knows we only want the fifth page of the scanned_contract.pdf. Ranges are also supported, so we could write something like B4-5 too.

Unfortunately, the scanned contract was scanned upside down, so let’s rotate 180 degrees by adding the -endS argument.

pdftk A=contract.pdf B=scanned_contract.pdf cat A B5-endS output contract_with_signature_attached.pdf

One notable issue I encountered while rotating individual pages was the inability to rotate and append only the first page. When specifying an option like B1-endS, the entire “B” document would be rotated and appended instead of just the first page. One other gotcha to remember: escape spaces and special characters when providing the names of documents. For example, if our document was named “scanned contract.pdf” we would need to do this:

pdftk contract.pdf scanned\ contract.pdf cat output signed_contract.pdf

The PDF Toolkit is licensed under GNU General Public License (GPL) Version 2. PDF Labs’s website provides a host of other examples including how to encrypt, password-protect, and repair PDFs.

Building Xpdf on Ubuntu

2011-08-31T00:00:00+00:00

It may happen that you need to use Xpdf, even though it no longer ships with Ubuntu and is considered … outdated? buggy? insecure? In any case, it still renders some PDFs that Poppler-based viewers such as Evince don’t, or allows some troublesome PDFs to print as fonts and line art instead of a rasterized mess.

Here’s how I built and installed xpdf 3.02 on Ubuntu 11.04 (Natty Narwhal) x86_64:

sudo apt-get install libfreetype6-dev libmotif-dev
wget ftp://ftp.foolabs.com/pub/xpdf/xpdf-3.02.tar.gz  # now 3.03 is current
tar xzpf xpdf-3.02.tar.gz
cd xpdf-3.02
./configure --with-freetype2-library=/usr/lib/x86_64-linux-gnu \
    --with-freetype2-includes=/usr/include/freetype2 \
    --with-Xm-library=/usr/lib \
    --with-Xm-includes=/usr/include/Xm
make
# see lots of warnings!
sudo make install

That’s it. Not as nice as the old native Debian/Ubuntu packages, but gets the job done.

Generate PDF with Chrome, Puppeteer, and Serverless Stack

The tools

1. Serverless Stack

2. Chrome and Puppeteer

Implementing PDF screenshot and download with Serverless Stack, Chrome, and Puppeteer

Prerequisite: Sample receipt pages

Getting started

1. sst.json

2. stacks/index.js

3. stacks/MyStack.js

Deploying the project

Developing with Serverless Stack

Conclusion

Caveats

Further readings

Convenient Reporting with Jasper

Jasper history

Best reporting tools

Installing JasperReports Server

Database

.jrxml

Data source

TIBCO Jaspersoft Studio

Long story short

MediaWiki extensions and wfLoadExtensionMessages

Little Spree Big Performance Problems

H12 timeout errors

Issue #1: Real-time reports

Solution

Issue #2: Large numbers

Solution

Issue #3: Real-time emails

Solution

Issue #4: Lazy-loading

Solution: Eager-loading

Issue #5: Dangerous code all over the place

Solution

No long-running tasks in the web request!

Generating PDF documents in the browser

What you will learn

Introduction

The solution

Using the DSL

Where is the documentation?

Can I generate it just from HTML?

Open PDF output in a new window

Render PDFs in the DOM

Appending one PDF to another using PDF Toolkit

Building Xpdf on Ubuntu