Convenient Reporting with Jasper

2020-04-28T00:00:00+00:00

Business Intelligence (BI) reporting is a huge problem space in custom software. There’s a wide range of business needs for looking at past and predictive behavior. Building a reporting tool can be a very cost effective way to get this data, especially compared to writing individual queries or manually generating reports.

I’ve been working with Jasper in the Java project space and wanted to write about some research I’ve collected on the topic.

JasperReports takes .jrxml files as input and outputs a .jasper report. Possible output targets include:

Screen
Printer
PDF
HTML
Excel files
RTF
ODT
CSV
XML

Jasper history

June 2001: Teodor Danciu began working on JasperReports.
September 2001: Jasper was registered on SourceForge.
November 2001: JasperReports 0.1.5 was released.
2004: Panscopic teamed up with Teodor Danciu, acquired ownership of the product and changed its name to Jaspersoft.
2005: JasperReports 1.0 was released.
2007: Brian Gentile became CEO of the company.
2014: TIBCO acquired Jaspersoft for ~$185 million.

Best reporting tools

Let’s compare some popular reporting tools:

JasperReports is a free and open source Java-based reporting tool, which supports lots of possible outputs, as mentioned earlier. Generating reports can be difficult if you’re less technical. More technical aspects can be more difficult as well; embedding JasperReports into a project is not necessarily simple, but once it’s done, the tool will be reliable.
Crystal Reports supports many inputs, including Access, Excel, XML, ODBC, and JDBC. It also has good multi-language support. It’s easy to embed into a .NET project, but software updates are unstable. The process can be very slow and there is no control of data-level security. A trial version is offered, but if one wants to use it long-term, then the price is a one-time payment of $495 or more, for larger companies.
Domo is another a popular reporting tool. It provides a trial version, and a 5 user plan costs $5700/year*.
Zoho analytics is an easy-to-use BI reporting tool, priced between $22–$445, depending on the number of users and data.
Host analytics is a great tool for finance automation. Pricing not publicized.
Tableau is an excellent reporting tool, with a thriving community online, but its quote-based price is high.
Pentaho is a Java-based reporting tool, which provides data integration, online analytical processing and reporting, among other features. Pentaho offers a 30-day trial period. Contract pricing isn’t disclosed.

So, if you are writing software and already use Java, or using Java reporting is an option, JasperReports is a great choice. It supports a variety of outputs, is free to use, and open source.

Installing JasperReports Server

To install JasperReports Server, you need a computer with a fully functional JRE (Java Runtime Environment). The server might be Tomcat or GlassFish. An RDBMS is also needed. since JasperReports has its own database, this could be PostgreSQL, Oracle, MySQL, DB2, or SQL Server. JasperReports prefers PostgreSQL and Tomcat, so these will be included with an automatic install. You may choose to use your existing Tomcat/PostgreSQL or have it install them as well.

Manual installation is also possible, as described here. At my first encounter with Jasper I installed Tomcat with the installer and used it for generating JasperReports, while the application I was working with was running WildFly (formerly JBoss), using a MySQL database. Needless to say, this was unnecessary, but I was not aware of that at the time. JasperServer can be configured to work with JBoss and MySQL as well.

The core of JasperReports is the JasperReports Library, which was already integrated into the project I was working with and is integrated into JasperReports Server as well as into popular IDEs, like TIBCO Jaspersoft Studio or iReport Designer.

Database

JasperReports provides example databases for imaginary companies, like FoodMart:

As we can see these are normal tables, having a primary key and some other fields.

.jrxml

jrxml, which stands for Jasper XML, contains report definitions in XML format. This type of file can be edited as code or visually to be compiled into .jasper files. The community provides samples that can be used and understood. Among them we can find a JFreeChart sample, which contains a jrxml file, a preview HTML and an actual PDF. The .jrxml file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jasperReport PUBLIC "-//JasperReports//DTD Report Design//EN" "http://jasperreports.sourceforge.net/dtds/jasperreport.dtd">

<jasperReport name="JFreeChartReport" pageWidth="595" pageHeight="842" columnWidth="515" leftMargin="40" rightMargin="40" topMargin="50" bottomMargin="50" scriptletClass="JFreeChartScriptlet">
  <variable name="Chart" class="net.sf.jasperreports.engine.JRRenderable" calculation="System"/>
  <title>
    <band height="742">
      <line>
        <reportElement x="0" y="0" width="515" height="1"/>
        <graphicElement/>
      </line>
      <staticText>
        <reportElement x="0" y="10" width="515" height="30"/>
        <textElement textAlignment="Center">
          <font size="22"/>
        </textElement>
        <text><![CDATA[JFreeChart Sample]]></text>
      </staticText>
      <textField>
        <reportElement x="0" y="50" width="515" height="50"/>
        <textElement textAlignment="Center">
          <font size="12"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA["This sample uses JFreeChart Version 1.0.0-pre2\n" + "Written by David Gilbert (david.gilbert@object-refinery.com) and others.\n" + "(C)opyright 2000-2004, by Object Refinery Limited and Contributors."]]></textFieldExpression>
      </textField>
      <image scaleImage="Clip" hAlign="Center" hyperlinkType="Reference">
        <reportElement x="0" y="110" width="515" height="300"/>
        <graphicElement/>
        <imageExpression class="net.sf.jasperreports.engine.JRRenderable"><![CDATA[$V{Chart}]]></imageExpression>
        <hyperlinkReferenceExpression><![CDATA["http://www.jfree.org/jfreechart"]]></hyperlinkReferenceExpression>
      </image>
    </band>
  </title>
</jasperReport>

It starts with the xml tag, specifying that this file should be interpreted as XML. Then comes the DOCTYPE and finally the jasperReport node, which contains the actual report nodes. A variable is defined, called Chart, which is used later in the inner XML of the image node. A hyperlink is defined for the image. The preview for this report looks like this:

Don’t worry about the broken images; this is just the preview, the actual result looks like this:

Nice, isn’t it?

Data source

It’s nice to generate reports, but in many cases the content is not fully known at programming time. It’s quite possible that we need to provide some input for the template. For this purpose, the interface of JRDataSource was defined to be iterated with a .next() method and is readable via the .getFieldValue() method. To make sure that we can read fields, another interface, JRField was defined as well. We therefore will need to use implementations of these classes, possibly writing our own if the currently available implementations are not fulfilling our needs.

Let’s consider a datasource sample, also taken from the community. It has this .jrxml template:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jasperReport PUBLIC "-//JasperReports//DTD Report Design//EN" "http://jasperreports.sourceforge.net/dtds/jasperreport.dtd">

<jasperReport name="DataSourceReport" pageWidth="595" pageHeight="842" columnWidth="515" leftMargin="40" rightMargin="40" topMargin="50" bottomMargin="50">
  <style name="Arial_Normal" isDefault="true" fontName="Arial" fontSize="12" isBold="false" isItalic="false" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <style name="Arial_Bold" isDefault="false" fontName="Arial" fontSize="12" isBold="true" isItalic="false" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica-Bold" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <style name="Arial_Italic" isDefault="false" fontName="Arial" fontSize="12" isBold="false" isItalic="true" isUnderline="false" isStrikeThrough="false" pdfFontName="Helvetica-Oblique" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <parameter name="ReportTitle" class="java.lang.String"/>
  <parameter name="DataFile" class="java.lang.String"/>
  <field name="id" class="java.lang.Integer"/>
  <field name="name" class="java.lang.String"/>
  <field name="street" class="java.lang.String"/>
  <field name="the_city" class="java.lang.String">
    <fieldDescription>me.me.city</fieldDescription>
  </field>
  <variable name="CityNumber" class="java.lang.Integer" incrementType="Group" incrementGroup="CityGroup" calculation="Count">
    <variableExpression><![CDATA[Boolean.TRUE]]></variableExpression>
  </variable>
  <group name="CityGroup" minHeightToStartNewPage="60">
    <groupExpression><![CDATA[$F{the_city}]]></groupExpression>
    <groupHeader>
    <band height="20">
      <textField evaluationTime="Group" evaluationGroup="CityGroup" bookmarkLevel="1">
        <reportElement mode="Opaque" x="0" y="5" width="515" height="15" backcolor="#c0c0c0" style="Arial_Bold"/>
        <box leftPadding="10" bottomBorder="1Point"/>
        <textFieldExpression class="java.lang.String"><![CDATA["  " + String.valueOf($V{CityNumber}) + ". " + String.valueOf($F{the_city})]]></textFieldExpression>
        <anchorNameExpression><![CDATA[String.valueOf($F{the_city})]]></anchorNameExpression>
      </textField>
    </band>
    </groupHeader>
    <groupFooter>
    <band height="20">
      <staticText>
        <reportElement x="400" y="1" width="60" height="15" style="Arial_Bold"/>
        <textElement textAlignment="Right"/>
        <text><![CDATA[Count :]]></text>
      </staticText>
      <textField>
        <reportElement x="460" y="1" width="30" height="15" style="Arial_Bold"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.Integer"><![CDATA[$V{CityGroup_COUNT}]]></textFieldExpression>
      </textField>
    </band>
    </groupFooter>
  </group>
  <title>
    <band height="70">
      <line>
        <reportElement x="0" y="0" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField isBlankWhenNull="true" bookmarkLevel="1">
        <reportElement x="0" y="10" width="515" height="30" style="Arial_Normal"/>
        <textElement textAlignment="Center">
          <font size="22"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$P{ReportTitle}]]></textFieldExpression>
        <anchorNameExpression><![CDATA["Title"]]></anchorNameExpression>
      </textField>
      <textField isBlankWhenNull="true">
        <reportElement x="0" y="40" width="515" height="20" style="Arial_Normal"/>
        <textElement textAlignment="Center">
          <font size="14"/>
        </textElement>
        <textFieldExpression class="java.lang.String"><![CDATA[$P{DataFile}]]></textFieldExpression>
      </textField>
    </band>
  </title>
  <pageHeader>
    <band height="20">
      <rectangle>
        <reportElement x="0" y="5" width="515" height="15" forecolor="#333333" backcolor="#333333"/>
        <graphicElement/>
      </rectangle>
      <staticText>
        <reportElement mode="Opaque" x="0" y="5" width="55" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <textElement textAlignment="Center"/>
        <text><![CDATA[ID]]></text>
      </staticText>
      <staticText>
        <reportElement mode="Opaque" x="55" y="5" width="205" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <text><![CDATA[Name]]></text>
      </staticText>
      <staticText>
        <reportElement mode="Opaque" x="260" y="5" width="255" height="15" forecolor="#ffffff" backcolor="#333333" style="Arial_Bold"/>
        <text><![CDATA[Street]]></text>
      </staticText>
    </band>
  </pageHeader>
  <detail>
    <band height="15">
      <textField bookmarkLevel="2">
        <reportElement x="0" y="0" width="50" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.Integer"><![CDATA[$F{id}]]></textFieldExpression>
        <anchorNameExpression><![CDATA[$F{name} + " (" + $F{id} + ")"]]></anchorNameExpression>
      </textField>
      <textField isStretchWithOverflow="true">
        <reportElement positionType="Float" x="50" y="0" width="200" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{name}]]></textFieldExpression>
      </textField>
      <textField isStretchWithOverflow="true">
        <reportElement positionType="Float" x="250" y="0" width="265" height="15"/>
        <box leftBorder="Thin" bottomBorder="Thin" rightBorder="Thin" leftPadding="10" rightPadding="10"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[$F{street}]]></textFieldExpression>
      </textField>
    </band>
  </detail>
  <pageFooter>
    <band height="40">
      <line>
        <reportElement x="0" y="10" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField>
        <reportElement x="200" y="20" width="80" height="15"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.String"><![CDATA["Page " + String.valueOf($V{PAGE_NUMBER}) + " of"]]></textFieldExpression>
      </textField>
      <textField evaluationTime="Report">
        <reportElement x="280" y="20" width="75" height="15"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[" " + String.valueOf($V{PAGE_NUMBER})]]></textFieldExpression>
      </textField>
    </band>
  </pageFooter>
  <lastPageFooter>
    <band height="60">
      <textField bookmarkLevel="1">
        <reportElement x="0" y="10" width="515" height="15"/>
        <textElement textAlignment="Center"/>
        <textFieldExpression class="java.lang.String"><![CDATA["There were " +
          String.valueOf($V{REPORT_COUNT}) +
          " address records on this report."]]></textFieldExpression>
        <anchorNameExpression><![CDATA["Summary"]]></anchorNameExpression>
      </textField>
      <line>
        <reportElement x="0" y="30" width="515" height="1"/>
        <graphicElement/>
      </line>
      <textField>
        <reportElement x="200" y="40" width="80" height="15"/>
        <textElement textAlignment="Right"/>
        <textFieldExpression class="java.lang.String"><![CDATA["Page " + String.valueOf($V{PAGE_NUMBER}) + " of"]]></textFieldExpression>
      </textField>
      <textField evaluationTime="Report">
        <reportElement x="280" y="40" width="75" height="15"/>
        <textElement/>
        <textFieldExpression class="java.lang.String"><![CDATA[" " + String.valueOf($V{PAGE_NUMBER})]]></textFieldExpression>
      </textField>
    </band>
  </lastPageFooter>
</jasperReport>

As we can see, there are fields defined like id, name, street, and the_city. We also have a group called CityGroup, so when the items from the data source are iterated through, their group is known via the_city. It’s worth looking at how the paging works. The key is evaluationTime, which is telling the engine to not evaluate a given element at iteration time, but rather when an event occurs. evaluationTime="Report" means that we need to evaluate the value when the Report event occurs. At that time $V{PAGE_NUMBER} already has the value equal to the number of pages. Let’s see the preview:

Again, we don’t need to worry about the missing image icons, since this is only a preview and this is the actual result:

Since we have a few interfaces that we need to respect, we can easily integrate Hibernate with our Jasper reports as data source, we just need to make sure we are using the field and data source interfaces they defined. Here we have a few examples, notably the following:

List cats = session.find("from eg.Cat");

Map parameters = new HashMap();
parameters.put("Title", "The Cat Report");

InputStream reportStream = this.class.getResourceAsStream("/the-cat-report.xml");
JasperDesign jasperDesign = JasperManager.loadXmlDesign(reportStream);
JasperReport jasperReport = JasperManager.compileReport(jasperDesign);

JRBeanCollectionDataSource ds = new JRBeanCollectionDataSource(cats);
JasperPrint jasperPrint = JasperManager.fillReport(jasperReport, parameters, ds);

JasperManager.printReportToPdfFile(jasperPrint, "the-cat-report.pdf");

We gather a List of Cat instances, then define some parameters, like Title, followed by the creation of an InputStream object which will be used to generate a JasperDesign object, which is then the input for the instantiation of JasperReport. Now, we define cats to be our data source and call fillReport, passing the jasperReport object we have just created, the parameters, which contain the title and the data source. Finally, we print the report to a PDF. Note that you can use a compiled .jasper file as input for getResourceAsStream.

TIBCO Jaspersoft Studio

This IDE is excellent for compiling the jrxml files into jasper files. I use it as a desktop application. There are three tabs:

Design
Source
Preview

In the Design tab one can interactively design the report, not worrying about source code, XML, and the like, which enables non-programmers to work on report creation as well. Handling the variables, parameters and fields needs some algorithmic understanding, but nonetheless, code is generated interactively in a point-and-click manner. The designer cannot do everything we need, but the Source tab comes to the rescue if needed. Sometimes I do not even need to use the Source tab.

There’s still one problem: testing is a high-cost operation in the project I am using Jasper for. We need to build the .jasper files, but that’s low-cost. The higher cost is that we need to build the application that actually generates the Jasper reports in order to test and then deploy it with JBoss. Then the actual test in the application occurs, so a test takes more than a minute. Luckily there is a Preview tab, where I can more or less see whether it’s a good idea to invest time into building and deploying, or if we need to do some tweaking first.

Long story short

For more information, see Jaspersoft’s website.

JasperReports is free and open source. However, you have to pay a fee to consult the help documentation. It’s easily a competitor in quality with other reporting tools, which are less reasonably priced. When I first had to work with Jasper reports, I didn’t know anything about it, but was able to complete the tasks at hand in a few hours, which shows that getting into the Jasper universe isn’t too hard. One may have difficulty understanding .jrxml files at first, but trust me, it’s worth it.

If you are already using Java and want to generate reports, Jasper is a good candidate. It can periodically generate reports for you and even send them attached to an email to you. I couldn’t write better closing words than a poster I found:

* Domo plan pricing was found at yurbi.com.

Web Projects for a Rainy Day

2020-03-25T00:00:00+00:00

Image by Yellowstone NPS on Flickr

With the COVID-19 quarantine disrupting life for many of us, I thought I’d put together a list of things you can do with your website on a rainy day. These are things to keep your business moving even if you’re at home and some of your projects are stuck waiting on things to reopen. If you’re looking for some useful things to do to fill your days over the next few months, this post is for you!

Major Version Updates

Make a list of your entire stack, from OS to database to development frameworks. Note the current version and research the current supported versions. I find Wikipedia pages to be fairly reliable for this (e.g. en.wikipedia.org/wiki/CentOS). Ok, so what things need to be updated, or will need to be in the next year? Start on those now and use some downtime to get ahead of your updates.

Sample of a client’s stack review

Software	Purpose	Our version	Release date	End of support	Next update	Newest version	Notes
CentOS	OS for e-commerce server	7	July 2014	June 2024	Not imminent	8	https://wiki.centos.org/About/Product
Nginx	Web server	1.16.0	March 2020	Unclear	Not imminent	1.16.1	https://nginx.org/
PostgreSQL	Database server	9.5.20	January 2016	Feb 2020	Medium term, to version 11	12	https://www.postgresql.org/support/versioning/
Rails	App framework for store	5.1	February 2017	Current	Long Term, to version 6	6	https://rubygems.org/gems/rails/versions
Spree	Ecommerce and admin gem	3.3	April 2017	Current	Long Term, to version 4	4	https://rubygems.org/gems/spree/versions
Elasticsearch	Search platform for product import/search	5.6.x	September 2017	March 2019	Immediate, to version 6.8	7.4	https://www.elastic.co/support/eol
WordPress	Info site	5.2.3	September 2019	Unclear	5.2.4 shipped recently	5.2	https://codex.wordpress.org/Supported_Versions

Content Cleanup & SEO Review

Everyone’s website gets cluttered with outdated content. Take a look at your pages, review, and update what needs to be changed. Pay attention to search engine optimization (SEO) concerns as you go through it. Make sure your content has headers, accurate keywords, and good meta-descriptions. Research SEO best practices if you need a refresher.

Nowadays, reducing repeated content has huge benefits for SEO so we recommend any content review includes a review of duplication. If you have a small site, you can go through your content and SEO manually. Larger projects can utilize tools on the market such as Siteliner or WPOptimize.

While you’re taking a dive into content, don’t forget to review your Google Analytics and understand what content is being used and what isn’t. Google has added many new features to Analytics and Ads, so it’s a good idea to refresh yourself on the updated documentation and new features.

Reporting

A lot of clients with big ecommerce data sets or other applications that collect data benefit from a separate reporting or business analytics tool. A rainy day can be a good time to think about what reports you want on last year’s business, what data will help you plan for the future. End Point has worked with a few different reporting tools that easily add on to your database, like Pentaho or Jasper and those can be really useful.

Documentation

I wouldn’t be a good project manager if I didn’t throw this one in the list. Documentation is so, so important, yet really we can always do more. End Point uses a few different tools, including wikis running MediaWiki and Google Docs, for keeping track of project details. Now’s a good time to set up a nice documentation system or do a big review and make sure everything is updated and back in order. Maybe dream of a vacation you might be able to take when this is over and make sure everything’s ready for you to do that.

Disaster Recovery Tests

For anyone with business-critical infrastructure, you need to ensure you know how to get everything back up and running in the case of a major failure, either with on-premises or cloud hosting. Now’s a good time to clarify with your hosting vendor things like: What are your backups like? What is your disaster recovery plan? What is the timeline for recovering the application in the event of a major failure? If you can, take time to do a simulation and make sure all the pieces are there if they need to be. Simply said, we also need to test our backups in order to ensure that they actually work.

Redesign

If you’ve been meaning to refresh your website, a rainy day is prime time to do it. Designers and developers are looking for projects and you’ll have extra time on your hands to oversee the process, spend time reviewing and testing, and get things done just the way you want.

Automated Testing

Good developers want an automated test suite as part of their application. Not all applications were built with this from the beginning and many didn’t have the budget or time to get it done. With extra time on your hands, this can be a great time to start building your test suite or to improve the coverage of your existing one.

Unit tests in particular are a good place to start. Unit tests are great not only because they help validate software correctness and protect against regressions, but also because they require a well-factored and modular system. This means that, while writing your unit tests, you will often be forced to go back to your application’s code base and refactor it to make it testable, to make it better. Investing in creating a solid unit test suite is a great bang for your buck. You can also look at implementing continuous integration—having a pipeline to let multiple developers deploy code throughout the day and configure your automated tests into the workflow.

Versioning & Deployment Tools

When you’re cleaning house, take a look at your Git version control repository and make sure everything important is in there. We have a few projects that have a main project in Git, but sometimes the smaller projects and one-offs can go astray. This is a good time to get everything organized into one repository, or make sure external repositories are connected and integrated.

Automated DevOps deployment tools can also be nice to work on. Tools like Ansible and Chef can take a lot of time to set up and test, but they have some great time-saving and accuracy advantages down the line. Our in-house security experts also recommend tools like AIDE and OSSEC which automate monitoring file changes daily.

Security Audit and Monitoring

Taking a few days to review your personal security and that of your application is something you should do regularly and now’s a good time to plan for it. Charlie’s got a great security post that’s a good top-level review. For application security, End Point uses some tools for vulnerability scanning. We also have a checklist of basic security items that include password handling, PII data, and other common security holes. For certain projects/clients we must also take HIPAA or PCI DSS compliance into account. Also, don’t neglect to review your TLS status and ensure that web applications run on TLS 1.2 and are TLS 1.3 ready. This also may relate to the underlying operating systems—whether they are able to support the latest TLS version natively.

Optimization and Performance:

Most of the time new features have higher priority than improving the performance of an existing system. It could be the right time to review core functionalities and list out the areas that need improvement in serving a better experience to customers by optimization. The areas can be focused on optimizing code, database queries, image size, data compression over network, adding cache, CDN, and so on. We’ve been moving quite a few clients to the Cloudflare DNS and CDN service and we’ve been really happy with it. Optimization work will definitely influence the customer retention rate which helps to increase profitability long term.

Refactoring

Along the same lines as optimization, code refactoring can have long term gains in performance and ease of future development. Think of this like house cleaning. It is always easier to find any item in the house when things are arranged in an orderly manner. Similarly, the organized and clean code base will play a vital role in future code changes and development, helping to reduce the chances of unexpected bugs, save time making changes at one place and improving code readability. Disciplined refactoring delivers readable, reusable, non-redundant code. Refactoring can be applied to your databases and user interfaces as well.

Want to get started on some background projects for your website? Talk to us today.

Analyzer Reports with Geo Map Option in Pentaho 5 BI Server

2014-09-12T00:00:00+00:00

The “Geo Map” option in Analyzer Reports provides a feature to visualize data with geographic locations. We will learn how to design a Mondrian schema and configure Pentaho to make use of the “Geo Map” feature in the Analyzer Reports. This article will show us how to set this feature up step by step.

Enable Geo Map feature on Geographic fields in Mondrian Schema

The Mondrian schema has two main categories called Dimensions and Measures. The Dimensions are defined as levels in the Mondrian schema. The Geographic fields should have two additional annotations to use Geo Map. The two annotations are:

Data.Role — defines the type of level generally; for this type of node, this must be set to ‘Geography’.
Geo.Role — defines the geographical classification in a hierarchy. These can be either predefined roles (‘country’, ‘state’, ‘city’, ‘postalcode’) or custom roles.

Sample Level with Annotation:

<Level name="Country Name" visible="true" column="country" type="String" uniqueMembers="false" levelType="Regular" hideMemberIf="Never">
  <Annotations>
    <Annotation name="Data.Role"><![CDATA[Geography]]></Annotation>
    <Annotation name="Geo.Role"><![CDATA[country]]></Annotation>
  </Annotations>
</Level>

Geographic fields and datasets in database

I have created a sample table with the fields containing geographic locations for dimensions and aggregated value for measures. The sample population table contains Pentaho-defined geographic locations ‘country’, ‘state’, ‘city’ and aggregated population count for those geographic fields.

‘Population’ table design and datasets:

Here we create a sample population table with geographic fields and the population count in a PostgreSQL database.

CREATE TABLE population (
   id INT PRIMARY KEY   NOT NULL,
   country      TEXT    NOT NULL,
   state        TEXT    NOT NULL,
   city         TEXT   NOT NULL,
   count        INT    NOT NULL
);

Next we load population data into the table for 4 cities of 2 states in USA. (Population data for more geographic locations in USA are available at USA Population.)

# SELECT * FROM population;
 id | country |   state    |     city      |  count 
----+---------+------------+---------------+---------
  1 | USA     | California | Los Angeles   | 3857800
  2 | USA     | California | San Francisco |  825863
  3 | USA     | New York   | Hilton        |    5974
  4 | USA     | New York   | Johnsburg     |    2390

Download the sql dump file with table schema and datasets.

Design a Mondrian Schema with Geographic Support

Pentaho provides a tool called “Schema Work Bench” to design a Mondrian schema for a specific table’s data. We can create a new Mondrian schema for the table by selecting File -> New -> Schema. The picture below depicts the hierarchy level of the Mondrian schema elements.

Publish the Mondrian schema to Pentaho

The publish process requires the JDBC datasource to have access to database. Create a JDBC datasource in the manage datasources wizard with necessary input values.

Once the JDBC datasource has been created in Pentaho server, the Mondrian schema can be published from the Schema Work Bench.

Download the Mondrian schema xml to view the schema, cube, table, dimension, hierarchy, level, annotations, measures elements and corresponding attribute values.

The Mondrian schema xml can be imported directly into Pentaho server to create an analysis datasource.

Create a Analyzer Report with Geo Map

Add the necessary geographic fields under “Rows” and population count under “Measure” to create a basic analyzer report.

Change the report type to “Geo Map” through the right top corner options to view the visualized data. Congratulations, you’re done!

Dimensional Modeling

2013-05-08T00:00:00+00:00

People occasionally bring up the question, “What exactly is a data warehouse?” Though answers to this question vary, in short a data warehouse exists to analyze data and its behavior over large swaths of time or many different data sources. There’s more to it, though, than simply cramming historical data into the same old database. There are a number of defining characteristics, including the following:

Query patterns and behavior
Data retention policy
Database structure

Query Behavior

Data warehouses are sometimes called “OLAP” databases, which stands for “on-line analytical processing”, in contrast to the more common “OLTP”, or “on-line transaction processing” databases that manage data for an online storefront, a bug tracker, or a blog. A typical OLTP database supports applications that issue short, simple queries, and expect quick answers and support for many simultaneous transactions. The average OLAP query, by contrast, is generally read-only, but can be quite complex, and might take minutes or hours to complete. Such queries will often include heavy-duty statistical processing and data mining, involving terabytes of data.

Data Retention

In a typical e-commerce database, eventually it becomes helpful to archive away older data that the front-end applications won’t need anymore. This helps performance, simplifies backups, and has the nice side-effect of leaving less data available for nefarious black hats that come snooping around. But it doesn’t make sense simply to discard much of this data, because it contains valuable information: customer behavior, supplier response time, etc. Often this deleted data remains alive, in a different form perhaps, in a data warehouse, which can contain data spanning many years.

Database structure

Because query patterns in data warehouses differ so much from OLTP databases, it makes sense to structure the OLAP database to support its queries better. OLTP databases typically follow an “entity” model, hence the ubiquitous (if often not particularly useful) entity-relationship diagram. In such a design, tables represent objects stored in the database, such as an order, a user, or a product. OLAP databases, on the other hand, are commonly “dimensionally” modeled, which results in something called a “star schema”. In a star schema, the database contains large “fact” tables, full of foreign keys pointing to a set of “dimension” tables; when diagrammed, this looks like star with the fact table in the center, or for the more mechanically oriented, a wheel with the fact table at the hub and dimension tables at the end of each spoke. Rather than modeling a particular entity, the fact table generally describes business processes, such as customer conversion or shipping efficiency.

Dimensional modeling is an interesting topic full of its own rules of thumb, which often differ quite dramatically from typical entity modeling. For instance, whereas many database modelers complain about the use of surrogate keys in OLTP databases, it’s recommended in dimensional modeling. Dimensional databases generally don’t need OUTER joins, and rarely contain NULL values. As a result, business intelligence applications designed for data warehousing can make certain assumptions about how they’ll be asked to query the database. These assumptions obviously place some limits on the types of queries the database can process effectively. However it is through these assumptions that the system gains most of its efficiency.

Biggest among the limitations of an OLAP database is what’s called the “grain”. The grain describes exactly what information the fact table contains, and at what level of detail; it should be made clear during the first stages of warehouse design and widely understood by all involved. Queries that require information that isn’t part of the grain, or at finer levels of detail, must find a different fact table to use. But for queries which depend only on the available data, the fact table can be very efficient, as the database can partition it easily, and scan it unencumbered by simultaneous writes from other transactions, and filtered by simple conditions and INNER joins to the various dimension tables.

Data warehouses differ from the traditional database in several other ways, but this covers some of the basics. Dimensional modeling alone is a well-developed field of study with numerous intricacies, where experience and careful training are important for developing a useful final model. But the analytical power of such databases has been proven.

A comparison of Jaspersoft iReport and Pentaho Reporting

2011-10-02T00:00:00+00:00

I’ve recently been involved in reporting projects using both Jaspersoft’s iReport and Pentaho’s Reporting, so this seemed a good opportunity to compare the two. Both are Java-based, open source reporting systems which claim to build “pixel-perfect” documents (“pixel-perfect” means that when you put something somewhere on a report design, it doesn’t move around. That this isn’t taken for granted is a rant for another time). I have more experience with Pentaho than with Jaspersoft, and once reviewed a book on Pentaho; I’ll try to give the two a fair evaluation, but in the end I can’t promise my greater experience with Pentaho won’t affect my conclusions one way or the other. Both suites are available in open source and commercial flavors; I’ll consider only the open source versions here.

First let me point out that Jaspersoft and Pentaho both produce business intelligence software suites. The two suites exist in both community (open source) and enterprise (commercial) forms, are well worth comparing in their entirety, but I’m focusing principally on the report builder component of each, because that’s where my recent experience has led me. These report builder packages allow users to build complex and visually interesting data-based documents from many different kinds of data sources. A “document” could be a simple form letter, an address book, or a complex dashboard of business metrics. In each case, users build documents by dragging and dropping various components into report “bands”, and then modifying components’ properties. “Bands” are horizontal sections of the page with different roles. The page header and footer bands, for instance, are (obviously) printed at the top and bottom of each page. “Detail” bands print once per row returned by the query that underlies the report. Both iReport and Pentaho Reporting will group and filter query results for you, if you want, and include header and footer bands for each group. Both products allow users to publish finished reports to a server where other users can view them, schedule them to run periodically, or modify them for new purposes, and both provide a Java-based reporting library to embed reporting in other applications.

iReport Query Dialog

Reports in both products are based on queries. These queries may be SQL using JDBC data sources, or they can come from other more obscure data sources, such as MDX queries, XQuery, scripts in various languages, web services, and more. Both also provide a query editor, at least for SQL queries, and in fact, both use the same query editor. I’ve only used it rarely, and only in Pentaho; both products also allow users to type in queries free-form, which I much prefer. In Pentaho, the data source for the query is embedded in the report itself, in the form of a JDBC URL, a JNDI name, or something else appropriate for that data source, so if you publish the report to a server, you’re responsible for ensuring that JNDI name or JDBC URL or whatever makes sense on that server. Jasper, on the other hand, prompts the user with a list of available data sources when publishing a report to a server. Jasper’s method seems more friendly, but Pentaho’s choice may have its advantages here, because each report is self-contained, whereas Jasper’s reports have metadata outside the single report file to describe the associated data source.

The Pentaho Report Builder

The component libraries available in each product are fairly similar. Users can select from basic drawing components such as lines and circles, static formatted text labels, and of course various numeric, textual, and date components to display query results. Both products also include complex charting components, to display visualizations of many different kinds. Though the charting functionality of both products, at least at the basic level I’ve used, is quite similar, I found iReport’s charting dialogs quite helpful for making complex charts much easier to create than they would otherwise be. In Pentaho, after adding a chart component the user can open a special properties window, but the window offers few clues beyond the usual tooltips and occasionally meaningful property names to help the user know what to do. In contrast, iReport describes many of the properties involved in clear language.

iReport’s chart selection dialog

This brings me to the topic of documentation, which, in fact, I found lacking for both produts. Yes, these are open source projects, and yes, documentation isn’t always as fun to write as code, so yes, open source projects sometimes end up with lousy documentation. The enterprise versions come with documentation in one form or another, and there are several books published on the different suites’ components, including reporting. But the documentation available free-of-charge on the web left me unimpressed. Here Pentaho was particularly frustrating, largely because Pentaho’s reporting has changed a great deal from version to version, especially with its 3.5 release a couple years ago, so much of the help available in forums and wikis is completely out of date. Jasper documentation was more difficult to find, in general, but more accurate and up-to-date when I found it.

Both products save their results in fairly comprehensible formats, which is helpful if you ever need to modify them by hand, without the help of the report building tool. iReport’s files are in a single XML file; Pentaho creates a ZIP-ed archive of several XML files. That fact has come in quite handy several times, both with Pentaho and with iReport, typically because it’s faster for me to edit XML by hand than click on each of 20 components and tweak properties one-by-one, because (in Pentaho) I needed to modify a data source, or because the tool failed to figure out what columns to expect from a query and I wanted to enter them manually.

One major difference between the two products stems from what is available in their open source “community editions”. Jasper looks a bit more polished to me, the documentation is somewhat more consistent, and its selection of sample reports and sample code is more comprehensive—but the community edition does little more than support iReport. Pentaho’s community edition, in addition to the reporting functions I’ve discussed, also offers ad-hoc web based reporting and a powerful MDX analysis package. Jaspersoft offers those features only in its enterprise edition. That may not be a big deal in some places, but it is the deciding factor in others. Having access to Pentaho’s code has made possible a number of things we certainly couldn’t have done otherwise.

I’ve been trying to keep this “a comparison” of Pentaho Reporting and iReport, rather than a “showdown” or “shootout” or even “Jasper vs. Pentaho”, but at some point, conclusions must be drawn. I’ll draw here somewhat on evidence not mentioned above, because, well, the post was getting a bit long as it was, and I didn’t want to describe everything. I admit to liking iReport’s interface better, because to me it seems to make better use of screen real estate. Google was much better able to answer questions for iReport than for Pentaho. But, although both products sometimes seem to make simple things hard, Pentaho seemed to do this less than iReport, and the various magic incantations I needed to get things working were fewer in Pentaho. In the end, the much greater capability of Pentaho’s open source offering over Jaspersoft’s clinches it for me. I’ll take open source over closed any day.

Automatically building Pentaho metadata

2009-11-04T00:00:00+00:00

Every so often I’ll hear of someone asking for a way to allow their users to write queries against their database without having to teach everyone SQL. There are various applications to do this: BusinessObjects and Cognos, are two common commercial examples, among many others. Pentaho and JasperReports provide similar capabilities in the open-source world. These tools allow users to write reports by selecting fields from a user-friendly list, adding suitable constraints, and making other formatting and filtering choices, all without needing to understand SQL.

Those familiar with these packages know that in order to provide those nice, readable field names and simple, meaningful field groupings, the software generally needs some sort of metadata file. This file maps actual database fields to readable descriptions, specifies relationships between tables, and translates database field types to data types the reporting software understands. Typically to create such a file, an administrator spends a few hours in front of a vendor-supplied GUI application dragging graphical representations of their tables and columns around, defining joins and entering friendly descriptions.

For the TriSano project’s data warehouse, we needed a way to make regular modifications to the metadata file we gave to our Pentaho instance, in order to allow users to write reports that included data from the custom-built forms TriSano allowed them to create. To this end, we dove into the Pentaho APIs and developed a system to modify the metadata file automatically, adding tables and relationships whenever users create a new custom form.

TriSano is a Ruby on Rails application, running on JRuby, and the ability to use Java objects natively within JRuby was critical to interfacing correctly with Pentaho, a Java application. Within JRuby, our script can create Pentaho objects at will. Interested parties are encouraged to browse the source code of the TriSano script for the many details required to make this work.

In short, the script makes a new Pentaho metadata file entirely from scratch, using only information from a small number of purpose-built database tables, and database structure information taken directly from the PostgreSQL catalogs. It creates a schema file, populates it with descriptions of each of the actual database tables our users are interested in, assigns friendly names to each of the database objects with which users will interact, and divides up the results into user-defined groupings meaningful to their business.

I’m not familiar with a commercial reporting package that allows for modification of the underlying metadata without user intervention; doing something like this without the benefit of open-source software would have been daunting indeed.

Pentaho Reporting 3.5 for Java Developers

2009-10-28T00:00:00+00:00

I was recently asked to review a copy of Will Gorman’s Pentaho Reporting 3.5 for Java Developers, and came to a few important realizations. Principal among these is that my idea of what “reporting” means includes far too little. Reporting includes much more than just creating a document with some graphs on it—any document or presentation including information from a “data source” comprises a “report”. This includes the typical sheaf of boardroom presentation slides, but also includes dashboards within an application, newsletters, or even form letters. I recently discovered that a local church group uses a simple reporting application to print its membership directory. In short, it’s not just the analysts and managers that can use reporting.

The book gives the reader a tour of Pentaho’s newest Pentaho Reporting system, which consists of a desktop application where users define reports, and a library by which developers can integrate those reports into their own applications. So as an example, not only can Pentaho Reporting publish weekly sales printouts, but it can also produce real-time inventory information in a J2EE-based web application or even a Swing application running on a user’s desktop. Gorman describes, step-by-step, the process of building a report, integrating advanced data sources, working with graphics and visualizations, and building custom reporting components such as user-defined functions and custom report controls. Although this step-by-step description combined with Java’s native verbosity occasionally make it difficult to read cover to cover, it provides a valuable reference and starting point for users wanting to implement advanced reporting features.

Shortly after having grasped the idea that “reporting”, as a concept, was broader than I’d originally assumed, I got the idea that I’d like to know how to use Pentaho to replace a reporting system I worked on a while ago, which used Jython to gather data from a JMX server. Pentaho Reporting allows users to create data sources in languages other than Java, via the Apache Bean Scripting Framework, suggesting that my old Jython code might work mostly unchanged via Bean Scripting. Disappointingly, the book doesn’t give an example of this technique, but since the Bean Scripting integration is still considered experimental (along with, apparently, several other data source types, contributed by the Pentaho community). Perhaps I’ll figure this out one day, and make it the subject of a future blog post.

This fairly slight omission notwithstanding, I enjoyed the book and the ideas it suggested about places I might apply more interesting reporting. Thanks to PacktPub for the opportunity to review it.

MDX

2009-07-14T00:00:00+00:00

Recently I’ve been working with Mondrian, an open source MDX engine. MDX stands for “multi-dimensional expressions”, and is a query language used in analytical databases. In MDX, data are considered in “cubes” made up of “dimensions”, which are concepts analogous to “tables” and “columns”, respectively, in a relational database. And in MDX, much as in SQL, queries written in a special query language tell the MDX engine to return a data set by describing filters in terms of the various dimensions.

But MDX and SQL return data sets in very different ways. Whereas a SQL query will return individual rows (unless aggregate functions are used), MDX always aggregates rows. In MDX, dimensions aren’t simple fields that contain arbitrary values; they’re hierarchical objects that can be queried at different levels. And finally, in MDX only certain dimensions can be returned in a query. These dimensions are known as “Measures”.

Without an example this doubtless makes little sense at first glance. In my case, the underlying data come from a public health application. Among other responsibilities, public health departments have as their task to prevent the spread of disease. Some diseases, such as tuberculosis or swine flu, are of particular interest because of their virulence, their mortality, or other characteristics. Health care providers are legally required to report cases of these diseases to various public health organizations, where the data are analyzed to identify and control outbreaks. The cube in question describes cases of these reportable conditions. Dimensions include the particular disease, the patient’s gender and race, the health department jurisdiction the patient lives in, and a few other characteristics. Among the available measures are the count of cases, the average age of each patient, and the average duration of the local public health department’s investigation into the case.

You’ll note that each of the measures describes groups of cases: a count of cases, the average from a group of values, etc. MDX will tell me the number of cases that meet a criterion, for instance, but not the names of each patient involved. As I said before, MDX only returns aggregates, not individual rows. Each measure’s definition includes an aggregate function used to calculate the final value for that measure based on a group of rows in a database.

The cube also uses hierarchical dimensions. As an example, public health data categorizes cases by age group rather than by age. Groups include ‘< 1 year’ and ‘1-4 years’ at the young end, ‘85+ years’ at the older end, and five year increments for everything in between. So the age dimension hierarchy would include two levels: one for the age group, and one for the specific age. In some instances, the jurisdiction dimension might also be a hierarchy, with the public health department at the top level, and subdivisions such as county, zip code, or neighborhood in levels of increasing specificity underneath.

At this point, the SQL-oriented reader says, “Well, you can do all this in SQL,” and that is perfectly true. In fact, the major duty of an MDX engine is generally to translate MDX queries into SQL queries (or more often, sets thereof). The advantage of MDX is that sometimes it’s simply easier to express a particular set of dimensions and measures in MDX than in the corresponding set of SQL queries. Better still, there are nice applications that speak MDX and allow you to browse interactively through MDX cubes without knowing either MDX or SQL. And finally, when the data set gets really large, which is common in OLAP databases, the MDX engine knows about optimizations it can make to speed things up.

A simple MDX query might look like this:

SELECT
  NON EMPTY {[Measures].[Quantity]} ON COLUMNS,
  NON EMPTY
      {([Markets].[All Markets], [Customers].[All Customers],
        [Product].[All Products], [Time].[All Years],
        [Order Status].[All Status Types])} ON ROWS
FROM [SteelWheelsSales]

These data come from a cube that ships as a sample with the open source Pentaho business intelligence software suite. [SteelWheelsSales] represents the cube name; other bracketed expressions are measure and dimension names. “ON ROWS” and “ON COLUMNS” describe the “axis” on which the particular measure or dimensions should be displayed. The “ROWS” and “COLUMNS” axes exist by default, and others can be defined at will. The query above gives a result set like this one:

This image shows what a more complex Mondrian MDX session might look like. The cube describes sales data from a sample business. Users can easily “slice and dice” data to view trends over time, variations across sales regions or product lines, or mixtures thereof. In this case, the rows describe various combinations of dimension values, and each cell contains the one measure this query asks for, aggregated across the rows that match the corresponding dimensions.

For more on MDX query syntax or MDX in general, see this Microsoft library MDX reference.

Subverting PostgreSQL Aggregates for Pentaho

2009-07-14T00:00:00+00:00

In a recent post I described MDX and a project I’m working on with the Mondrian MDX engine. In this post I’ll describe a system I implemented to overcome one of Mondrian’s limitations.

Each Mondrian measure has an associated aggregate function defined. For instance, here’s a measure from the sample data that ships with Pentaho:

<Measure name="Quantity" column="QUANTITYORDERED" aggregator="sum" />

The schema defines the database connection properties and the table this cube deals with elsewhere; this line says there’s a column called QUANTITYORDERED which Mondrian can meaningfully aggregate with the sum() function. Mondrian knows about six aggregates: count, avg, sum, min, max, and distinct-count. And therein lies the problem. In this case, the client wanted to use other aggregates such as median and standard deviation, but Mondrian didn’t provide them[1].

Mondrian uses the aggregator attribute of the measure definition to generate SQL statements exactly as you might expect. In the case of the measure above, the SQL query involving that measure would read “sum(QUANTITYORDERED)”. In our case, Mondrian is backed by a PostgreSQL database, which offers a much richer set of aggregates (such as stddev() for the standard deviation, one of the numbers we need), but Mondrian doesn’t know how to get to them.

Measures can be defined in terms of SQL expressions, rather than simple column names, but this doesn’t immediately help. If I wanted the standard deviation of the quantity ordered, I might try something like this:

<Measure name="Quantity">
    <KeyExpression><SQL dialect="postgres">
        stddev(quantityordered)
    </SQL></KeyExpression>
</Measure>

Here, Mondrian would complain that the measure was defined without an aggregator attribute. And if I define one, such as sum, the resulting SQL becomes “sum(stddev(quantityordered))”, which is illegal and makes PostgreSQL complain about nested aggregates.

But PostgreSQL’s function overloading can help here. Although Mondrian’s generated SQL will always include a call to a “count()” function if the aggregator is defined as “count”, but there’s no reason we can’t make PostgreSQL use some other count() function. For instance, let’s defined a new “count()” function that isn’t an aggregate, but simply returns whatever argument it is passed. Then we can use it to wrap whatever function we want, including arbitrary aggregate functions.

Consider an attempt to get Mondrian to use the stddev() aggregate. It returns a DOUBLE PRECISION type, so our fake count function must simply accept a DOUBLE PRECISION variable and return it:

CREATE FUNCTION count(DOUBLE PRECISION) RETURNS DOUBLE PRECISION AS $$
    SELECT $1
$$ LANGUAGE SQL IMMUTABLE;

Then we define a measure like this:

<Measure name="Quantity Std. Dev" aggregator="count">
    <KeyExpression><SQL dialect="postgres">
        stddev(quantityordered)
    </SQL></KeyExpression>
</Measure>

The resulting SQL is “count(stddev(quantityordered))”, but in this case PostgreSQL uses our new count() function, and we get exactly the return value we want.

There’s a catch: if we have a double precision column “foo” in a table “bar”, and write:

SELECT count(foo) FROM bar;

…it uses our new count function, and rather than returning the number of rows in bar, it returns the value for foo from each row in bar.

To get around this problem, we can define a new data type. We’ll write a function to create that datatype from another data type, and rewrite our count function to accept only that data type, and return the original data type, like this:

CREATE TYPE dp_cust AS (dp DOUBLE PRECISION);

CREATE FUNCTION make_dpcust(a DOUBLE PRECISION) RETURNS dp_cust IMMUTABLE AS $$
DECLARE
    dpc dp_cust;
BEGIN
    dpc.dp := a;
    RETURN dpc;
END;
$$ LANGUAGE plpgsql;

DROP FUNCTION count(double precision);

CREATE FUNCTION count(dp_cust) RETURNS DOUBLE PRECISION IMMUTABLE AS $$
    SELECT $1.dp
$$ LANGUAGE sql;

Now our count() function will only be called when we’re dealing with the dp_cust type, and we can control precisely when that happens, because the only way we make dp_cust values will be with the make_dpcust function. Our measure now looks like this:

<Measure name="Quantity Std. Dev" aggregator="count">
    <KeyExpression><SQL dialect="postgres">
        make_dpcust(stddev(quantityordered))
    </SQL></KeyExpression>
</Measure>

With this new data type and our custom count() function we can use whatever PostgreSQL aggregate we want as a measure aggregate in Mondrian.

[1] Note that the Mondrian developers already recognize this as a shortcoming worth removing. Allowing user-defined aggregates is on the Mondrian roadmap.