Migrating from Universal Analytics to GA4

2023-06-01T00:00:00+00:00

Image: Migration by Aivar Ruukel, 2014. Attribution 2.0 Generic (CC BY 2.0), obtained from Flickr

Most public-facing websites rely on Google Analytics to track their traffic, analyze the user’s characteristics and behavior, and run reports based on that information to improve marketing strategies, engage their public, and ultimately, increase the user’s loyalty.

Universal Analytics vs. Google Analytics 4

Until 2020, Google relied on Universal Analytics, a system that offered a set of reports that were mainly based on page visits and content visualization. But in October 2020, Google announced that the new Google Analytics 4 (GA4) was launched, using an event-centered approach for metrics. That allows several improvements in the way the data is collected and analyzed, like taking into account several platforms and devices as a source for the data (for example, combining website traffic with mobile app usage and activity on social networks).

Another improvement is privacy: Among the new features, anonymous IP addresses are now the default setting for GA4. The user’s IP address will be still registered and used to group data when doing the initial collection, but it won’t be retained after, and only the location metadata will be used for reporting.

At the present, both versions co-exist—but not for long! On July 1, GA4 will be the one option available, and Universal Analytics will disappear completely. That means we have a month (or less, depending on when you’re reading this!) to add a GA4 property and export the existing data from the Universal Analytics property.

One thing to consider is that the historical data will still be available under our Universal Analytics account for at least six months, but no new traffic will be processed.

Migrating from UA to GA4

First, a few considerations:

We should do this process on the Google Analytics website. We cannot do it through the mobile app.
We should repeat this process for every account that we need to migrate.
We need access to our website’s source code in case we need to update the existing GA tag.

After navigating to the Analytics website, we need to do the following steps:

Go to the Settings item on the lower left.
Select the existing Universal Analytics property.
Click the GA4 Setup Assistant link.
Create a new GA4 property with the assistant’s help.

If you already have a Google tag on the website that can be reused for GA4, you won’t need to make any changes to your source code, and you will be all set. If your tag is not compatible, or Google can’t detect it properly, the assistant will provide you a new tag. In that case, you will need to install the JavaScript snippet that contains the tag on your website.

The snippet will look similar to the one below:

<!-- Global Site Tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=xxxxxxxxxxxxxxx"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
  gtag('config', 'xxxxxxxxxxxxxxx');
</script>

After finishing the wizard, you will have the new GA4 property connected to your account. Now you can navigate to it and start the second Setup Assistant, that will allow you to customize your property and import the existing data from Universal Analytics:

Click on the “Set up conversions” option on the main screen of the assistant.
Choose the option “Import from Universal Analytics”.
Select which goals should be imported into events on the GA4 property.
Click “Import selected conversions”.

That’s it! Once the import finishes, the existing goals will appear as conversion events on the GA4 property. If the Google tag was set up properly, the traffic information should start flowing to the new property, showing real-time information about your traffic, and the conversion events should be registering specific user actions:

We have done several migrations to GA4 for clients that were using Analytics during the past months. If you haven’t done it yet and need any assistance with the process, don’t hesitate to contact us!

3 Immediate Benefits of Google Analytics for Business Owners

2021-04-30T00:00:00+00:00

Image from Google’s marketing platform blog

Where is your traffic coming from? What drew the traffic to your website? Which parts of your website are most visited? How do visits change over time? And how can the answers to these questions help you?

Answering such questions and doing something about it is called search engine optimization (SEO).

To help you with this is Google Analytics, a web analytics service that lets you track and understand your website traffic. It is a valuable tool for businesses of all sizes that are looking to grow.

Here are three ways Google Analytics can benefit your business:

Determining Site Improvements to Strengthen Website Flow

This is a great way to generate more “conversions” — visitors to your website taking a desired action. Are visitors behaving the way you expected them to? Can you observe any bottlenecks in audience flow?

Bottlenecks include traffic getting stuck on one page, when you want them to be going to a different one, like a contact page. Understanding how traffic gets stuck might point you toward the need to refresh certain web pages, which could in turn lead to more conversions.

For example, we observed that our “Deployment Automation” Expertise subpage has had a 100% bounce rate over the past three months. This is concerning because it means that the content may not be engaging or there may not be a clear visitor navigation path, the end goal being a contact submission. Analytics helped us start looking at how to strengthen this subpage.

Image from Google’s marketing platform blog

Understanding your Audience

Who is coming to your site, and how are they finding you? What referral sites, partner sites, media, and blog posts are directing the most traffic to your page? How can you leverage that?

In reviewing your inbound traffic, you will see some combination of the following types of traffic:

Direct: Traffic from directly typing the URL into the browser address bar.
Organic: Traffic from people who navigate to your website through search engines after seeing you in search results. Having a strong online presence, especially strong SEO, will help more visitors arrive on your website without the need to pay for them.
Referral: Traffic that comes to your website after being “referred” from a different website. This is when other websites link to your webpage. More backlinks and referral traffic typically leads to significant SEO benefits.
Paid: This traffic arrives from paid search campaigns on platforms such as Google Ads.
Email: Traffic from links in emails.
Social: Traffic that comes from posts on social media networks like Facebook, LinkedIn, and Twitter.
Other: All the traffic which doesn’t fit in any other category.

We recommend reviewing each type of traffic to get a better understanding of their flow through your website, and noting any trends you find within individual web traffic sources and mediums.

Data-Driven Decision Making: Stop Relying on Assumptions and Rely on Data

One great challenge to businesses is overconfidence in how much you understand about your audience. Google Analytics, and other similar analytics tools, can transform your work culture from being based on opinions and assumptions to being based on hard data. Google Analytics provides data in an organized and impactful format, and using analytics data in tandem with sales efforts can lead to more conversions and revenue for your business.

Alternatives

With Google having access to so much data and being one of the two major advertisers on the web, many people are looking for alternatives that allow them more control over their customer data, separation from Google’s advertising platforms, and a slimmer data footprint for compliance with privacy laws such as CCPA (California) and GDPR (European Union).

There have always been various options for web visitor analytics. Google Analytics was originally created by a company called Urchin Software, which Google acquired in 2005. Some current alternatives include:

Cloudflare web analytics, a new service offered by the popular CDN (Content Distribution Network) that simply shows visitor data already flowing through their systems.
GoatCounter, a SaaS or self-hosted open source application, which aims to provide simple counters rather than collecting personal data, thus avoiding any need for a privacy notice.
Matomo, formerly known as Piwik, a fully-featured SaaS or on-premises paid package with a limited open source version.
Open Web Analytics, a customizable open source analytics framework.

We at End Point have found success with these core ideas and several of these services. We are happy to provide a free consultation to discuss your website needs.

Using Google Analytics to understand and grow your business

2016-01-13T00:00:00+00:00

Google Analytics, a web analytics service offered by Google, is a very handy tool for understanding your audience. It allows you to understand where traffic comes from and what resonates with your audience, which has led to Google Analytics being the most widely used web analytics service on the internet. If you understand your website’s traffic, you then have the ability to focus your website and content to optimize engagement and growth.

With Google Analytics, you have the ability to see traffic from all channels. This will lead to clear insights, and will help you understand what’s working and what’s not.

Organic — traffic from search engines which is not paid for
Paid Search — visitors that clicked on one of your paid advertisements (also known as Pay-Per-Click or PPC)
Direct — visitors that typed your website address directly into the browser (includes bookmarks)
Social — traffic from sites that are considered to be social networking sites
Referral — visitors that arrived from 3rd party referrals
Email — visitors that are directed from an email
Display — visitors directed from video and display advertising

It will be helpful to walk through an example. Say you launch an email marketing campaign, and want to understand how your audience responded to your content. First, you can check how many people clicked the ad to come to your website. From there, you can see how they navigated the page. Who came to the webpage? Did they take the actions you were hoping they would take? How long did they spend viewing the page? Where did they click? Did this click lead to a conversion/sale?

Prior to coming to End Point, I was working as an Analytic Strategist at a digital media agency. One of my biggest clients was luxury jewelry company Tiffany & Co. My responsibilities included analyzing seasonal and promotional trends to develop forecasts, broken out by channel. I would also evaluate the marketing effectiveness of initiatives by keeping a close eye on the user experience and navigational behavior, and provide recommendations throughout the customer journey.

Many jewelry promotions are seasonal, so we were constantly making changes to optimize the page and ensure traffic was navigating the website as we hoped. I would make sure to keep in mind the 4 P’s—Product, Price, Place, and Promotion. I wanted to ensure that the layout of your site, price-point, and specials were always in line with what the audience is most looking for during the season.

This experience of understanding client goals, and helping them achieve those goals through the success of their websites, has proven to be a valuable skill since joining End Point’s team. Beyond having the ability to help clients with their websites, this knowledge has also been helpful internally. I keep a close eye on where our website traffic is coming from, including seeing which blog posts resonate with different audiences. I have made recommendations for adjustments to poor-performing pages, and taken time to analyze pages with high bounce rates, to see whether the bounce rates are due to the page not being well-constructed or due to traffic not being properly directed. I also take note of any industry changes, and make website adjustments to accommodate those changes. For example, End Point has a plethora of skill in many programming languages, and if we see one language gaining popularity we will make a point to highlight it more prominently.

I look forward to continuing to use Google Analytics to strengthen our company websites and to help our clients strengthen theirs. Please don’t hesitate to reach out if you would like to learn more.

Install Pentaho BI Server 4.8 Community Edition with PostgreSQL Repository

2013-11-07T00:00:00+00:00

Pentaho BI server community edition can be installed through an archive file available from SourceForge.

Prerequisites

Java 6
PostgreSQL

Download Pentaho BI Server installation file (biserver-ce-4.8.0-stable.zip) from SourceForge: http://sourceforge.net/projects/pentaho/files/Business%20Intelligence%20Server/4.8.0-stable

Unzip the archive file and navigate inside biserver-ce to set sh files to executable mode:

$ unzip biserver-ce-4.8.0-stable.zip
$ cd biserver-ce
$ find . -type f -iname '*.sh' -exec chmod a+x {} \;

Pentaho community edition uses hsql database as default. Need to create two databases in Postgres for Pentaho. Find the SQL files to create databases under biserver-ce/data/postgresql. database_name, user_name and password are configurable through SQL files. Fix two errors before creating database using SQL files. Comment two lines in below files tables as commented.

create_quartz_postgresql.sql

ALTER TABLE qrtz_fired_triggers
    ALTER TRIGGER_NAME  TYPE VARCHAR(200),
    ALTER TRIGGER_GROUP TYPE VARCHAR(200),
    ALTER INSTANCE_NAME TYPE VARCHAR(200),
    ALTER JOB_NAME      TYPE VARCHAR(200),
    ALTER JOB_GROUP     TYPE VARCHAR(200),
    ADD COLUMN PRIORITY INTEGER NULL;
--    ADD COLUMN PRIORITY INTEGER NOT NULL;

migrate_quartz_postgresql.sql

ALTER TABLE qrtz_fired_triggers
    ALTER TRIGGER_NAME  TYPE VARCHAR(200),
    ALTER TRIGGER_GROUP TYPE VARCHAR(200),
    ALTER INSTANCE_NAME TYPE VARCHAR(200),
    ALTER JOB_NAME      TYPE VARCHAR(200),
    ALTER JOB_GROUP     TYPE VARCHAR(200);
--    ADD COLUMN PRIORITY INTEGER NOT NULL;
--    ALTER COLUMN PRIORITY SET NULL;

Modify database name, username & password if necessary and create databases for configuration and scheduling by running below commands.

$ psql -U postgres -a -f create_quartz_postgresql.sql
$ psql -U postgres -a -f create_repository_postgresql.sql
$ psql -U postgres -a -f create_sample_datasource_postgresql.sql
$ psql -U postgres -a -f migrate_quartz_postgresql.sql
$ psql -U postgres -a -f migration.sql

Verify databases quartz (scheduling) and hibernate (configuration) and their tables.

Now database name, username, password and driver should be configured in following places in files. By default hsql drivers, settings enabled in config files so comment hsql configurations and enable Postgres settings.

biserver-ce/tomcat/webapps/pentaho/META-INF/context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/pentaho" docbase="webapps/pentaho/">
        <Resource name="jdbc/Hibernate" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="hibuser" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/hibernate"
                validationQuery="select 1" />
               
        <Resource name="jdbc/Quartz" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="pentaho_user" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/quartz"
                validationQuery="select 1"/>
</Context>

biserver-ce/pentaho-solutions/system/applicationContext-spring-security-jdbc.xml

        <bean id="dataSource"
                class="org.springframework.jdbc.datasource.DriverManagerDataSource">
                <property name="driverClassName" value="org.postgresql.Driver" />
                <property name="url" value="jdbc:postgresql://localhost:5432/hibernate" />
                <property name="username" value="hibuser" />
                <property name="password" value="password" />
        </bean>

biserver-ce/pentaho-solutions/system/applicationContext-spring-security-hibernate.properties

jdbc.driver=org.postgresql.Driver
jdbc.url=jdbc:postgresql://localhost:5432/hibernate
jdbc.username=hibuser
jdbc.password=password

hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

biserver-ce/pentaho-solutions/system/hibernate/hibernate-settings.xml

<config-file>system/hibernate/postgresql.hibernate.cfg.xml</config-file>

biserver-ce/pentaho-solutions/system/hibernate/postgresql.hibernate.cfg.xml

<property name="connection.driver_class">org.postgresql.Driver</property>
    <property name="connection.url">jdbc:postgresql://localhost:5432/hibernate</property>
    <property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
    <property name="connection.username">hibuser</property>
    <property name="connection.password">password</property>

biserver-ce/pentaho-solutions/system/simple-jndi/jdbc.properties

Hibernate/type=javax.sql.DataSource
Hibernate/driver=org.postgresql.Driver
Hibernate/url=jdbc:postgresql://localhost:5432/hibernate
Hibernate/user=hibuser
Hibernate/password=password
Quartz/type=javax.sql.DataSource
Quartz/driver=org.postgresql.Driver
Quartz/url=jdbc:postgresql://localhost:5432/quartz
Quartz/user=pentaho_user
Quartz/password=password

Specify the pentaho solutions path, server url and port in web.xml of tomcat webapp

biserver-ce/tomcat/webapps/pentaho/WEB-INF/web.xml

        <context-param>
                <param-name>solution-path</param-name>
                <param-value>/opt/avr-new/biserver-ce/pentaho-solutions</param-value>
        </context-param>

        <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Pentaho can be run tomcat custom ports by modifying the ports in server.xml and web.xml

biserver-ce/tomcat/conf/server.xml

    <Connector URIEncoding="UTF-8" port="9090" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

biserver-ce/tomcat/webapps/pentaho/WEB-INF/web.xml

         <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Let’s start the Pentaho BI server and try out its great features. Commands to start and stop the BI server:

biserver-ce$ ./start-pentaho.sh
biserver-ce$ ./stop-pentaho.sh

Troubleshooting:

biserver-ce$ tail -f tomcat/logs/catalina.out
biserver-ce$ tail -f tomcat/logs/pentaho.out

Install Pentaho BI Server 5 Enterprise Edition with PostgreSQL repository

2013-11-07T00:00:00+00:00

Pentaho provides different ways to install Pentaho BI server. Each method has its own flexibility in installation.

Installer mode—Easy to install BA & DI server & tools in one flow with default PostgreSQL repo & default Tomcat server. (New Postgres installed on port 5432.)
Archive mode—BA server installed with own BA repository & default Tomcat server. Necessary tools need to be installed manually.
Manual mode—BA server installed with own BA repository & own application server (Tomcat or JBoss). Necessary tools need to installed manually.

We have a Postgres instance running on our server and are good with Tomcat as application server so Archive mode of installation is suitable for us. Pentaho installation requires two things be installed before starting with Pentaho installation.

Java 7
PostgreSQL

Archive mode installation files can be accessible only to license purchased users. Download biserver-ee-5.x-dist.zip from Pentaho customer portal with account credentials here: https://support.pentaho.com/forums/20413716-Downloads

Unzip the archive file and you can see the installation files inside extracted directory.

$ unzip biserver-ee-5.x-dist.zip
$ cd biserver-ee-5.x;ls
install.bat  installer.jar  install.sh  license.txt  README.txt

In remote servers Pentaho can be installed on console mode with -console’. Accept the license and provide the installation path to install Pentaho BI server.

$ java -jar installer.jar -console

Find biserver-ee directory under the installation path and set sh files to executable mode.

$ cd biserver-ee;
$ find . -type f -iname '*.sh' -exec chmod a+x {} \;

Let’s create repository databases by running queries in SQL files located at biserver-ee/data/postgresql.

quartz, hibernate and jackrabbit databases will be created by executing these SQL files. Database names, usernames and password can be changed by modifying in SQL files if required.

$ cd biserver-ee/data/postgresql
$ psql -U postgres -p 5432 -a -f create_jcr_postgresql.sql
$ psql -U postgres -p 5432 -a -f create_quartz_postgresql.sql
$ psql -U postgres -p 5432 -a -f create_repository_postgresql.sql
$ psql -U postgres -p 5432 -a -f pentaho_mart_postgresql.sql

Pentaho uses postgresql as default database and files are configured to use postgresql. So just verify the database_name, username and password to work with installed postgresql and databases created.

biserver-ee/pentaho-solutions/system/quartz/quartz.properties

org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

biserver-ee/pentaho-solutions/system/hibernate/hibernate-settings.xml

    <config-file>system/hibernate/postgresql.hibernate.cfg.xml</config-file>

biserver-ee/pentaho-solutions/system/hibernate/postgresql.hibernate.cfg.xml

    <property name="connection.driver_class">org.postgresql.Driver</property>
    <property name="connection.url">jdbc:postgresql://localhost:5432/hibernate</property>
    <property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
    <property name="connection.username">pentaho_user</property>
    <property name="connection.password">password</property>

There are more occurrences in this file. Carefully do the necessary changes in all the places.

biserver-ee/pentaho-solutions/system/jackrabbit/repository.xml
biserver-ee/pentaho-solutions/system/jackrabbit/repository/workspaces/default/workspace.xml
biserver-ee/tomcat/webapps/pentaho/META-INF/context.xml

<Resource name="jdbc/PDI_Operations_Mart" auth="Container" type="javax.sql.DataSource"
            factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
            maxWait="10000" username="pentaho_user" password="password"
            driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/hibernate"
            validationQuery="select 1"/>

Download postgresql and h2 drivers then place it under biserver-ee/tomcat/lib

postgresql-9.x.jdbc4.jar

h2-1.2.x.jar

Change Tomcat port on these two files to run Pentaho on different port.

Specify the Pentaho solutions path, server URL and port in web.xml of Tomcat webapp.

biserver-ee/tomcat/webapps/pentaho/WEB-INF/web.xml

        <context-param>
                <param-name>solution-path</param-name>
                <param-value>$INSTALLATION_PATH/biserver-ee/pentaho-solutions</param-value>
        </context-param>

        <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Pentaho can be configured to run on different ports by changing ports on Tomcat server.xml and web.xml

biserver-ee/tomcat/biserver-ee/tomcat/server.xml

    <Connector URIEncoding"UTF-8" port"9090" protocol"HTTP/1.1"
               connectionTimeout"20000"
               redirectPort"8443" />

- biserver-ee/tomcat/webapps/pentaho/WEB-INF/web.xml
         <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:9090/pentaho/</param-value>
        </context-param>

Install license

A license needs to be installed to use Pentaho. Navigate to the license-installer directory in installation path. Feed license file to install_license.sh, separating more than one license file with spaces.

$ ./install_license.sh install ../license/Pentaho\ BI\ Platform\ Enterprise\ Edition.lic

Install plugins

Archive mode of installation installs only BI Server. Necessary plugins can to be installed manually. Here install the plugins for reporting, analyzer and dashboard. Plugins are available at the same place where download BI server. Download these three files and place at any path on server:

Reporting—pir-plugin-ee-5.x-dist.zip
Analyzer—pdd-plugin-ee-5.0.0.1-dist.zip
Dashboard—paz-plugin-ee-5.0.0.1-dist.zip

All the plugins installed through same procedure.

Unzip the plugins and navigate to extracted directory
run installer on console, accept the license and provide $INSTALLATION_PATH/biserver-ee/pentaho-solutions/system as location to install plugins

$ java -jar installer.jar -console

Let’s start the BI server

biserver-ee$ ./start-pentaho.sh

Install the licenses for the plugins by login as admin user (default—Admin:password) or install through the command line:

Administration -> License -> install licenses for plugin by click +

Troubleshooting:

biserver-ee$ tail -f tomcat/logs/catalina.out
biserver-ee$ tail -f tomcat/logs/pentaho.log

If the pentaho.xml is present at tomcat/conf/Catalina directory, delete it. It will be generated again when you start the BA Server.

Start and stop the BI server:

biserver-ee$ ./start-pentaho.sh
biserver-ee$ ./stop-pentaho.sh

Conversion Tracking via JavaScript

2013-01-08T00:00:00+00:00

Most analytics conversion tracking is done these days with JavaScript or invisible pixel requests on the page that indicates a user has reached a conversion event, such as the receipt page. For example, Google Analytics conversion code might look like this on the receipt page:

_gaq.push(['_setAccount', 'UA-XXXXX-X']);
_gaq.push(['_trackPageview']);
_gaq.push(['_addTrans',
   '1234',           // transaction ID - required
   'Womens Apparel', // affiliation or store name
   '28.28',          // total - required
   '1.29',           // tax
   '15.00',          // shipping
   'San Jose',       // city
   'California',     // state or province
   'USA'             // country
]);
_gaq.push(['_addItem',
   '1234',           // transaction ID - necessary to associate item with transaction
   'DD44',           // SKU/code - required
   'T-Shirt',        // product name
   'Olive Medium',   // category or variation
   '11.99',          // unit price - required
   '1'               // quantity - required
]);
_gaq.push(['_trackTrans']);

But what happens when a single page with various JavaScript-driven UI updates drives a user to a conversion event via AJAX? All of the conversion events have to be triggered via JavaScript after the conversion event. After some experimentation and verification with Google’s Developer Tools (Network) or Firebug’s Net Tool, I’ve implemented the following tracking services and code in JavaScript upon a conversion event for our Ruby on Rails client Mobixa:

MSN Conversion

tagid = /* tag id */
domainid = /* domain id */
actionid = /* action id */
jQuery('<iframe src="//flex.atdmt.com/mstag/tag/' + tagid
  + '/analytics.html?dedup=1&domainId=' + domainid + '&type=1&actionid='
  + actionid + '" frameborder="0" scrolling="no" width="1" height="1"
  style="visibility:hidden;display:none"><0/iframe>')
  .appendTo('#hidden_tracking');

Google Ad Services

google_conversion_id = /*conversion id*/
google_conversion_label = /* conversion label */
var image = new Image(1,1);
image.src = 'http://www.googleadservices.com/pagead/conversion/' +
  google_conversion_id + '/?label=' + google_conversion_label +
  '&value=1&guid=ON&script=0';

Floodlight Conversion

jQuery('<img height="1" width="1" src="http://sa.jumptap.com/a/conversion?
  event=Purchase" />').appendTo('#hidden_tracking');

AdParlor Conversion

adid = /* Ad Parlor id */
jQuery('<img height="1" width="1" alt="AP_pixel"
  src="http://fbads.adparlor.com/conversion.php?adid="'
  + adid + '  />').appendTo('#hidden_tracking');

AdKnowledge

adknowledgeid = /* adknowledge id */
jQuery('<iframe src="https://www.lynxtrack.com/track.frame.php?g='
  + adknowledgeid + '&o=' + /* order number */ + '&s=' + /* total */
  + '" height="1" width="1" frameborder="0">  <script
  language="JavaScript" src="https://www.lynxtrack.com/trackjs/g-'
  + adknowledgeid + '/o-' + /* order number */ + '/s-' + /* total */
  + '/track.js"> </script><noscript>
  <img src="https://www.lynxtrack.com/track.php?g=' + adknowledgeid
  + '&o=' + /* order number */ + '&s=' + /* total */ + '" width="1"
  height="1" border="0"></noscript></iframe>')
  .appendTo('#hidden_tracking');

Google Analytics Ecommerce Conversion

_gaq.push(['_addTrans',
    /* order number */,
    /* affiliation */,
    /* total */,
    /* tax */,
    /* shipping */,
    /* city */,
    /* state */,
    /* country */
]);
$.each(purchased_items, function(i, item) {
    gaq.push(['_addItem',
        /* order number */,
        /* item sku */,
        /* item name */,
        /* item category */,
        /* item price */,
        /* item quantity */
    ]);
});
_gaq.push(['_trackTrans']);

Conclusion

Here are a few important takeaways in working with conversion tracking via JavaScript:

Most of the above conversion tracking calls have a specific ID that is provided by the marketing service.
Images and iframes are appended to a div with an id of hidden_tracking on the page to trigger the conversion request. Tracking did not appear to work if the images or iframes were appended to the body element. Also in the case of Google Ad Services conversion, the image itself did not have to be appended to the page; a request alone was enough.
It’s important and extremely helpful to use Google Developer Tools or Firebug to verify these requests go through during development.
Google Analytics tracking does not look much different from non-AJAX conversion tracking, but much of the other tracking code differs from what you might see in on-page tracking events.

Cannot parse Cookie header in Ruby on Rails

2012-09-07T00:00:00+00:00

Yesterday I resolved a client emergency for a Ruby on Rails site that continues to leave me scratching my head, even with follow-up investigation. In short, the emergency came up after an email marketing campaign was sent out in the morning, and resulted in server (HTTP 500 Status Code) errors for every customer that clicked on the email links. Despite the fact that Rails exception emails are sent to the client and me, the errors were never reaching the exception email code, so I was unaware of the emergency until the client contacted me.

Upon jumping on the server, I saw this in the production log repeatedly:

ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):
ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):
ArgumentError (cannot parse Cookie header: invalid %-encoding (...)):

The URLs that the production log was complaining about had a bunch of Google Analytics tracking variables:

utmcmd=Email
utmcct=customeremail
utmccn=New Site Sale 70% off
etc.

After a user visits the site, these variables are typically stored as cookies for Google Analytics tracking. Upon initial investigation, the issue appeared to be triggered from any Google campaign variable that contained a ‘%’ character.

After follow-up investigation today, the more complete story looks like this:

Email blast sent
User clicks on link in email. That link goes to the email marketing company first for tracking, then is redirected to the website.
According to the email marketing campaign (after chatting with them today), Google Analytics tacks on their own tracking here, which is the source of the non-parseable URLs.
Rack receives the request and tries to parse the query, utilizing the Ruby URI module:

def self.decode_www_form_component(str, enc=Encoding::UTF_8)
    if TBLDECWWWCOMP_.empty?
      tbl = {} 
      256.times do |i|
        h, l = i>>4, i&15 
        tbl['%%%X%X' % [h, l]] = i.chr
        tbl['%%%x%X' % [h, l]] = i.chr
        tbl['%%%X%x' % [h, l]] = i.chr
        tbl['%%%x%x' % [h, l]] = i.chr
      end  
      tbl['+'] = ' '
      begin
        TBLDECWWWCOMP_.replace(tbl)
        TBLDECWWWCOMP_.freeze
      rescue
      end  
    end  
    raise ArgumentError, "invalid %-encoding (#{str})" unless /\A[^%]*(?:%\h\h[^%]*)*\z/ =~ str
    str.gsub(/\+|%\h\h/, TBLDECWWWCOMP_).force_encoding(enc)
  end

The argument error on line 18 trickles up the pipeline in rack, and is not handled elegantly, so a rack-originated server (HTTP 500 Status Code) error is triggered. Again, the ‘%’ character in the URL appears to be the problem here likely based on the regexp match on line 18 — the error is not triggered when the Google variable does not contain a ‘%’ character.
Customer sees server error page and is unhappy :(

At the time of the emergency we tried solving the problem on multiple avenues:

Investigated removal of Google Analytics tracking URLs from email blast links, but this wouldn’t help all the customers who already received the emails.
Remove CGI parameters or sanitize them via nginx.
Make web application changes to ignore or handle the ArgumentError.

Ultimately, I ended up added a begin/rescue statement to the rack code to skip escaping URLs where decode_www_form_component was raising an error:

def unescape(s, encoding = Encoding::UTF_8)
  begin
    URI.decode_www_form_component(s, encoding)
  rescue
    Rails.logger.warn "DECODING on #{s.inspect} with #{encoding.inspect} FAILING."
  end 
end

While this is a reasonable fix, I’m still puzzled for a number of reasons:

The email marketing company (via chatting) claims that Google Analytics is tacking on the tracking variables, however, there is only one redirect from the email marketing company to the website. I don’t understand the mechanism for which Google Analytics tracking variables are added to the URL, and if this process can be cleaned up to ensure proper URL encoding.
I’m not sure if the issue happens immediately upon a customer landing on the site, or after a cookie is stored.
At the moment, I’m not able to reproduce this issue in development mode, which makes it difficult to troubleshoot on my development instance.
When I use the URI module directly in a console, no ArgumentError is raised:

>> URI.decode_www_form_component("url_with_google_campaign_variables")
>> #happy dance

My best advice at this point is to tell the client not to use ‘%’ character in the Google Campaign ID, but I’m still putting all the pieces together in the virtual code map in my head. I think a fix is more likely needed on the Ruby and rack side to handle URL parameters with the ‘%’ character, and to elegantly handle situations where the URI.decode_www_form_components method dies.

An Introduction to Google Website Optimizer

2012-04-17T00:00:00+00:00

On End Point’s website, Jon and I recently discussed testing out use of Google Website Optimizer to run a few A/B tests on content and various website updates. I’ve worked with a couple of clients who use Google Website Optimizer, but I’ve never installed it from start to finish. Here are a few basic notes that I made during the process.

What’s the Point?

Before I get into the technical details of the implementation, I’ll give a quick summary of why you would want to A/B test something. A basic A/B test will test user experiences of content A versus content B. The goal is to decide which of the two (content A or content B) leads to higher conversion (or higher user interactivity that indirectly leads to conversion). After testing, one would continue to use the higher converting content. An example of this in ecommerce may be product titles or descriptions.

A/B tests in Google Website Optimizer

I jumped right into the Google Website Optimizer sign-up and wanted to set up a simple A/B test to test variations on our home page content. Unfortunately, I found right away that basic A/B tests in Google Website optimizer require two different URLs to test. In test A, the user would be see index.html, and in test B, the user would see index_alt.html. This is unfortunate because for SEO and technical reasons, I didn’t want to create an alternative index page.


Test A: Keep existing text	Test B: Remove some paragraph text in first section

Multivariate Testing

Rather than implement a basic A/B tests in Google Website optimizer, I decided to implement a multivariate test with just the two options (A and B). The basic setup required this:

Copy provided JavaScript into my test page just above </head>
Wrap <script>utmx_section("stephs_test")</script>…</noscript> around the section of text that will be modified by the test.
Copy provided JavaScript into my converting test page just above </head>

Google Website Optimizer will verify the code and provide a user interface to enter the variations of the test text.

I used multivariate testing to test the homepage changes described above. After a couple of weeks of testing, my test results were inconclusive:

Example multivariate test result in Google Website Optimizer

A Limitation with Multivariate Testing

One thing we wanted to test was a site-wide CSS change. Unfortunately, the multivariate testing in place is designed to test on page content only rather than global CSS changes. You could potentially come up with a “creative” hack and set a cookie inside the variation to specify which layout option you would use. And then the page would always look at that cookie while rendering to apply the special CSS behavior. However, this requires a bit of customization and development.

A comparison of Jaspersoft iReport and Pentaho Reporting

2011-10-02T00:00:00+00:00

I’ve recently been involved in reporting projects using both Jaspersoft’s iReport and Pentaho’s Reporting, so this seemed a good opportunity to compare the two. Both are Java-based, open source reporting systems which claim to build “pixel-perfect” documents (“pixel-perfect” means that when you put something somewhere on a report design, it doesn’t move around. That this isn’t taken for granted is a rant for another time). I have more experience with Pentaho than with Jaspersoft, and once reviewed a book on Pentaho; I’ll try to give the two a fair evaluation, but in the end I can’t promise my greater experience with Pentaho won’t affect my conclusions one way or the other. Both suites are available in open source and commercial flavors; I’ll consider only the open source versions here.

First let me point out that Jaspersoft and Pentaho both produce business intelligence software suites. The two suites exist in both community (open source) and enterprise (commercial) forms, are well worth comparing in their entirety, but I’m focusing principally on the report builder component of each, because that’s where my recent experience has led me. These report builder packages allow users to build complex and visually interesting data-based documents from many different kinds of data sources. A “document” could be a simple form letter, an address book, or a complex dashboard of business metrics. In each case, users build documents by dragging and dropping various components into report “bands”, and then modifying components’ properties. “Bands” are horizontal sections of the page with different roles. The page header and footer bands, for instance, are (obviously) printed at the top and bottom of each page. “Detail” bands print once per row returned by the query that underlies the report. Both iReport and Pentaho Reporting will group and filter query results for you, if you want, and include header and footer bands for each group. Both products allow users to publish finished reports to a server where other users can view them, schedule them to run periodically, or modify them for new purposes, and both provide a Java-based reporting library to embed reporting in other applications.

iReport Query Dialog

Reports in both products are based on queries. These queries may be SQL using JDBC data sources, or they can come from other more obscure data sources, such as MDX queries, XQuery, scripts in various languages, web services, and more. Both also provide a query editor, at least for SQL queries, and in fact, both use the same query editor. I’ve only used it rarely, and only in Pentaho; both products also allow users to type in queries free-form, which I much prefer. In Pentaho, the data source for the query is embedded in the report itself, in the form of a JDBC URL, a JNDI name, or something else appropriate for that data source, so if you publish the report to a server, you’re responsible for ensuring that JNDI name or JDBC URL or whatever makes sense on that server. Jasper, on the other hand, prompts the user with a list of available data sources when publishing a report to a server. Jasper’s method seems more friendly, but Pentaho’s choice may have its advantages here, because each report is self-contained, whereas Jasper’s reports have metadata outside the single report file to describe the associated data source.

The Pentaho Report Builder

The component libraries available in each product are fairly similar. Users can select from basic drawing components such as lines and circles, static formatted text labels, and of course various numeric, textual, and date components to display query results. Both products also include complex charting components, to display visualizations of many different kinds. Though the charting functionality of both products, at least at the basic level I’ve used, is quite similar, I found iReport’s charting dialogs quite helpful for making complex charts much easier to create than they would otherwise be. In Pentaho, after adding a chart component the user can open a special properties window, but the window offers few clues beyond the usual tooltips and occasionally meaningful property names to help the user know what to do. In contrast, iReport describes many of the properties involved in clear language.

iReport’s chart selection dialog

This brings me to the topic of documentation, which, in fact, I found lacking for both produts. Yes, these are open source projects, and yes, documentation isn’t always as fun to write as code, so yes, open source projects sometimes end up with lousy documentation. The enterprise versions come with documentation in one form or another, and there are several books published on the different suites’ components, including reporting. But the documentation available free-of-charge on the web left me unimpressed. Here Pentaho was particularly frustrating, largely because Pentaho’s reporting has changed a great deal from version to version, especially with its 3.5 release a couple years ago, so much of the help available in forums and wikis is completely out of date. Jasper documentation was more difficult to find, in general, but more accurate and up-to-date when I found it.

Both products save their results in fairly comprehensible formats, which is helpful if you ever need to modify them by hand, without the help of the report building tool. iReport’s files are in a single XML file; Pentaho creates a ZIP-ed archive of several XML files. That fact has come in quite handy several times, both with Pentaho and with iReport, typically because it’s faster for me to edit XML by hand than click on each of 20 components and tweak properties one-by-one, because (in Pentaho) I needed to modify a data source, or because the tool failed to figure out what columns to expect from a query and I wanted to enter them manually.

One major difference between the two products stems from what is available in their open source “community editions”. Jasper looks a bit more polished to me, the documentation is somewhat more consistent, and its selection of sample reports and sample code is more comprehensive—but the community edition does little more than support iReport. Pentaho’s community edition, in addition to the reporting functions I’ve discussed, also offers ad-hoc web based reporting and a powerful MDX analysis package. Jaspersoft offers those features only in its enterprise edition. That may not be a big deal in some places, but it is the deciding factor in others. Having access to Pentaho’s code has made possible a number of things we certainly couldn’t have done otherwise.

I’ve been trying to keep this “a comparison” of Pentaho Reporting and iReport, rather than a “showdown” or “shootout” or even “Jasper vs. Pentaho”, but at some point, conclusions must be drawn. I’ll draw here somewhat on evidence not mentioned above, because, well, the post was getting a bit long as it was, and I didn’t want to describe everything. I admit to liking iReport’s interface better, because to me it seems to make better use of screen real estate. Google was much better able to answer questions for iReport than for Pentaho. But, although both products sometimes seem to make simple things hard, Pentaho seemed to do this less than iReport, and the various magic incantations I needed to get things working were fewer in Pentaho. In the end, the much greater capability of Pentaho’s open source offering over Jaspersoft’s clinches it for me. I’ll take open source over closed any day.

PostgreSQL log analysis / PGSI

2011-08-19T00:00:00+00:00

Image by “exfordy” on Flickr

End Point recently started working with a new client (a startup in stealth mode, cannot name names, etc.) who is using PostgreSQL because of the great success some of the people starting the company have had with Postgres in previous companies. One of the things we recommend to our clients is a regular look at the database to see where the bottlenecks are. A good way to do this is by analyzing the logs. The two main tools for doing so are PGSI (Postgres System Impact) and pgfouine. We prefer PGSI for a few reasons: the output is better, it considers more factors, and it does not require you to munge your log_line_prefix setting quite as badly.

Both programs work basically the same: given a large number of log lines from Postgres, normalize the queries, see how long they took, and produce some pretty output.If you only want to look at the longest queries, it’s usually enough to set your log_min_duration_statement to something sane (such as 200), and then run a daily tail_n_mail job against it. This is what we are doing with this client, and it sends a daily report that looks like this:

Date: Mon Aug 29 11:22:33 2011 UTC
Host: acme-postgres-1
Minimum duration: 2000 ms
Matches from /var/log/pg_log/postgres-2011-08-29.log: 7

[1] (from line 227)
2011-08-29 08:36:50 UTC postgres@maindb [25198]
LOG: duration: 276945.482 ms statement: COPY public.sales 
(id, name, region, item, quantity) TO stdout;

[2] (from line 729)
2011-08-29 21:29:18 UTC tony@quadrant [17176]
LOG: duration: 8229.237 ms execute dbdpg_p29855_1: SELECT 
id, singer, track FROM album JOIN artist ON artist.id = 
album.singer WHERE id < 1000 AND track <> 1

However, the PGSI program was born of the need to look at all the queries in the database, not just the slowest-running ones; the cumulative effect of many short queries can have much more of an impact on the server than a smaller number of long-running queries. Thus, PGSI looks not only at how long a query takes to run, but how many times it has run in a certain period, as well as how often it runs. All of this information is put together to give a score to each normalized query, known as the “system impact”. Like the costs on a Postgres explain plan, this is a unit-less number and of little importance in and of itself—the important thing is to compare it to the other queries to see the relative impact. We also have that report emailed out, it looks similar to this (this is a text version of the HTML produced):

Log file: /var/log/pg_log/postgres-2011-08-29.log

 * SELECT (24)
 * UPDATE (1)

Query System Impact : SELECT

 Log activity from 2011-08-29 11:00:01 to 2011-08-29 11:15:01

   +----------------------------------+
   |   System Impact: | 0.15          |
   |   Mean Duration: | 1230.95 ms    |
   | Median Duration: | 1224.70 ms    |
   |     Total Count: | 411           |
   |   Mean Interval: | 4195 seconds  |
   |  Std. Deviation: | 126.01 ms     |
   +---------------------------------+

 SELECT *
  FROM albums
  WHERE track <> ? AND artist = ?
  ORDER BY artist, track

At this point you may be wondering how we get all the queries into the log. This is done by setting log_min_duration_statement to 0. However, most (but not all!) clients do not want full logging 24 hours a day, as this creates some very large log files. So the solution we use is to analyze a slice of the day, only. It depends on the client, but we try for about 15 minutes during a busy time of day. Thus, the sequence of events is:

Turn on “full logging” by dropping log_min_duration_statement to zero1. Some time later, set log_min_duration_statement back to what it was (e.g. 200)1. Extract the logs from the time it was set to zero to when it was flipped back.1. Run PGSI against the log subsection we pulled out1. Mail the results out

All of this is run by cron. The first problem is how to update the postgresql.conf file and have Postgres re-read it, all automatically. As covered previously, we use the modify_postgres.pl script for this.

The exact incantation looks like this:

0 11 * * * perl bin/modify_postgres_conf --quiet \
  --pgconf /etc/postgresql/9.0/main/postgresql.conf \
  --change log_min_duration_statement=0
15 11 * * * perl bin/modify_postgres_conf --quiet \
  --pgconf /etc/postgresql/9.0/main/postgresql.conf \
  --change log_min_duration_statement=200 --no-comment
## The above are both one line each, but split for readability here

This changes log_min_duration_statement to 0 at 11AM, and then back to 200 15 minutes later. We use the –quiet argument as this is run from cron so we don’t want any output from modify_postgres_conf on success. We do want a comment when we flip it to 0, as this is the temporary state and we want people viewing the postgresql.conf file at that time to realize it (or someone just doing a “git diff”). We don’t want a comment when we flip it back, as the timestamp in the comment would cause git to think the file had changed.

Now for the tricky bit: extracting out just the section of logs that we want and sending it to PGSI. Here’s the recipe I came up with for this client:

16 11 * * * tac `ls -1rt /var/log/pg_log/postgres*log \
  | tail -1` \
  | sed -n '/statement" changed to "200"/,/statement" changed to "0"/ p' \
  | tac \
  | bin/pgsi.pl --quiet > tmp/pgsi.html && bin/send_pgsi.pl
## Again, the above is all one line

What does this do? First, it finds the latest file in the /var/log/pg_log directory that starts with ‘postgres’ and ends with ‘log’. Then it uses the tac program to spool the file backwards, one line at a time (‘tac’ is the opposite of ‘cat’). Then it pipes that output to the sed program, which prints out all lines starting with the one where we changed the log_min_duration_statement to 200, and ending with the one where we changed it to 0 (the reverse of what we actually did, as we are reading it backwards). Finally, we use tac again to put the lines back in the correct order, pipe the output to pgsi, write the output to a temporary file, and then call a quick Perl script named send_pgsi.pl which mails the temporary HTML file to some interested parties.

Why do we use tac? Because we want to read the file backwards, so as to make sure we get the correct slice of log files as delimited by the log_min_duration_statement changes. If we simply started at the beginning of the file, we might encounter other similar changes that were made earlier and not by us.

All of this is not foolproof, of course, but it does not have to be, as it is very easy to run manually is something (for example the sed recipe) goes wrong, as the log file will still be there. Yes, it’s also possible to grab the ranges in other ways (such as perl), but I find sed the quickest and easiest. As tempting as it was to write Yet Another Perl Script to extract the lines, sometimes a few chained Unix programs can do the job quite nicely.

Virtual Page Tracking and Goals with Google Analytics

2011-04-06T00:00:00+00:00

Sometimes I come across websites that don’t use RESTful URLs or have too unique (with an order number) URLs during checkout and I need to implement Goal Tracking in Google Analytics on these user interactions. I’ve also had to implement Goal Tracking in a non-ecommerce web application where tabbed on-page browsing guides users through a 3-step process. Examples of situations that pose challenges to traditional page tracking in Google Analytics include:

Throughout Interchange’s checkout, URLs are posts to “/process”, which makes the user interactions difficult to distinguish.
Throughout Spree’s checkout, URLs are posts to “/order/:id/edit”, which are distinct and can be difficult to aggregate.
In a Sinatra application we developed recently, the single page URL is “/locate.html”, but tabbed browsing occurs through three unique steps.

Google Analytics Tagging

To add Goal Tracking by URL, pages must first be tagged as “virtual pages”. To implement virtual page tracking in Google, it’s as simple as including a new virtual page URL in the _trackPageview action:

<script type="text/javascript">

  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXXXXX-1']);
- _gaq.push(['_trackPageview']);
+ _gaq.push(['_trackPageview', '/cart/basket']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>

In the cases I described above, here are the tags used for each:

Interchange	Spree	Sinatra App
On a three page checkout process, the following virtual pages were tagged: Shopping Basket: /cart/basket Checkout: /cart/checkout Receipt: /cart/receipt	On a multi-page checkout process, the following virtual pages were tagged: Shopping Basket: /checkout/cart Registration (optional): /checkout/register Address: /checkout/address Delivery: /checkout/delivery Payment: /checkout/payment Confirm: /checkout/confirm Receipt: /checkout/complete	On a three page process, the following virtual pages were tagged: Step 1: /step1 Step 2: /step2 Step 3: /step3

Goal Definition

After tagging is complete, you’ll need to define a Goal in Google Analytics to begin tracking it.

First, you’ll want to name your goal. Here I’ve named the goal “Purchase”. In this case, the Goal is a URL Destination based on our new tags.

The Goal URL is set to “/checkout/complete” for the Spree example described above. In this case, I have it set to a Regular Expression Match, but an Exact Match will work as well. I haven’t assigned a Goal Value and I’ve left the match case insensitive for now.

Next, I add the steps involved leading up to the goal URL. This will allow you to visualize a goal funnel and see where users may be abandoning checkout. In this example, each of the steps of the funnel correspond to a checkout step (shopping basket, optional register page, individual checkout steps, receipt/complete page).

Note that this screenshot was taken from the new Google Analytics interface. The old Google Analytics interface may appear slightly different, but contains similar content.

Looking at the Data

With virtual page tracking and goal tracking, you can see how many goals have been completed during a certain time frame:

You can also compare regular segments of traffic: e.g. users that found your site through search versus referrals:

You can compare advanced segments of visitors: e.g. marketing campaign #1 versus marketing campaign #2. This type of comparison may give you more actionable metrics to determine which marketing campaign to put more resources into.

Last but not least, you can visualize the funnel of conversion to see where you might be losing customers. This might also be actionable in that you may be able to A/B test checkout process (or another type of conversion) changes to improve conversion.

Referral Tracking with Google Analytics

2011-03-25T00:00:00+00:00

It’s pretty easy to use Google Analytics to examine referral traffic, including using custom referral tracking codes. Here’s how:

Once you have referrers or affiliates that plan to link to your site, you can ask that those affiliates append a unique tracking ID to the end of the URL. For example, I’ll use the following referral ID’s to track metrics from Milton and Roger’s websites to End Point’s site.

After you’ve seen some traffic build up from those affiliates, you must create two Custom Advanced Segments in Google Analytics:


Follow the link to create an Advanced Segment.	The New Advanced Segment page.

Once you’ve landed on the New Advanced Segment page, you create a custom segment by dragging “Landing Page” from the “Content” tab to define the criteria, and set it to contains your unique referral identifier.


Roger’s Referral Traffic	Milton’s Referral Traffic

That’s it! You now have custom Advanced Segments defined to track referral or affiliate data. You can select the Advanced Segments from any metrics page:

All traffic compared to referral traffic from Milton and Roger’s sites.

Traffic from Milton’s website only.

You can also examine conversion driven from the affiliate. For example, how does conversion driven by one affiliate compare to the entire site’s conversion? On our site, conversion is measured by contact form submission — but on ecommerce sites, you can measure conversion in the form of purchases relative to different affiliates.

Roger’s Referral conversion versus conversion of the entire site. Roger’s doing pretty good!

One potential disadvantage to this method for affiliate tracking is that you are creating duplicate content in Google by introducing additional URLs. You may want to use the rel=“canonical” tag on the homepage to minimize duplicate content in search engine indexes. A very similar alternative to this method to bypass adding a referral ID would be to create custom segments defined by Source and Referral Path, however, the method described in this article is valuable for sites that may have a redirect between the referral site and the landing URL (http://www.miltonsblog.com/ links to http://www.endpointcorp.com/?ref=milton redirects to http://www.endpoint.com/?ref=milton retains the referral information).

Google Analytics is a great tool that allows you to measure analytics such as the ones shown in this post. It’s fairly standard for our all of our clients to request Google Analytics installation. Google announced last week that a new Google Analytics platform will be rolled out soon, which includes a feature update to multiple segments that will allow us to examine traffic from multiple affiliates without showing “All Visits”.

Note that the data presented in this article is fictitious. I don’t think Milton and Roger will be linking to End Point’s site any time soon!

In Our Own Words

2011-02-02T00:00:00+00:00

What do our words say about us?

Recently, I came across Wordle, a Java-based Google App Engine application that generates word clouds from websites and raw text. I wrote a cute little rake task to grab text from our blog to plug into Wordle. The rake task grabs the blog contents, uses REXML for parsing, and then lowercases the results. The task also applies a bit of aliasing since we use postgres, postgreSQL and pg interchangeably in our blog.

task :wordle => :environment do
   data = open('http://blog.endpoint.com/feeds/posts/default?alt=rss&max-results=999', 'User-Agent' => 'Ruby-Wget').read
   doc = REXML::Document.new(data)
   text = ''
   doc.root.each_element('//item') do |item|
     text += item.elements['description'].text.gsub(/<\/?[^>]*>/, "") + ' '
     text += item.elements['title'].text.gsub(/<\/?[^>]*>/, "") + ' '
   end
   text = text.downcase \
     .gsub(/\./, ' ')   \
     .gsub(/^\n/, '')   \
     .gsub(/ postgres /, ' postgresql ') \
     .gsub(/ pg /, ' postgresql ')
   file = File.new(ENV['filename'], "w")
   file.puts text
   file.close
 end

So, you tell me: Do you think we write like engineers? How well does this word cloud represent our skillset?

New Year Bug Bites

2011-01-06T00:00:00+00:00

Happy New Year! And what would a new year be without a new year bug bite? This year we had one where figuring out the species wasn’t easy.

On January 2nd one of our ecommerce clients reported that starting with the new year a number of customers weren’t able to complete their web orders because of credit card security code failures. Looking in the Interchange server error logs we indeed found a significant spike in the number of CVV2 code verification failures (Payflow Pro gateway error code “114”) starting January 1st.

We hadn’t made any programming or configuration changes on the system in the recent days. We double-checked to make sure: nope, no code changes. So it had to be a New Year’s bug and presumably something with the Payflow Pro gateway or banks further upstream. We checked error logs for other customers to see if they were being similarly impacted, but they weren’t. Our client contacted PayPal (the vendor for Payflow Pro) and they reported there were no problems with their system. The failures must indeed be card failures or a problem with the website according to them. We further checked our code looking for what we could possibly have done that might be the cause, double-checking our Git repository (which showed no recent changes) and reexamining our checkout code for possible year-based logic flaws.

Our client’s top-notch customer service group got on the phone with a customer who’d gotten a security code failure and got PayPal tech support on another line. The customer service rep tried to place the customer’s order on the website using the customer’s credit card info and once again got the CVV2 error. She then did the credit card transaction using the swipe machine in the office, and lo and behold the order went through! What was going on??!

It turned out that despite the Payflow Pro gateway returning CVV2 verification errors what was really happening was that the year of the credit card was coming into the Payflow Pro gateway as “2012”—not as “2011” as entered into the checkout form. We knew all along that it was possible that the 114 error code responses were possibly misleading because payment gateway error codes are notorious this way. (Payment gateways blame the banks, saying they can only pass along what the banks give them. Some banks’ credit card validations don’t actually even care about the years being correct, but just that they not be in the past; but I digress…)

Previously we’d reviewed the checkout pages and the dropdown menus to verify that the dropdown menus weren’t off, but nevertheless it very much sounded like this rather stupid problem could very well be the culprit. So we checked and checked again. What we found is that sometimes on the checkout form the year dropdown menu was mangled such that the values associated with the displayed years were YYYY+1.

The oddly intermittent behavior of the problem, the process of elimination and the all around hair pulling this loss of business was causing made somebody in the marketing group at our client realize that they are in fact still running an Omniture Test & Target A/B test on the checkout pages that they thought had been discontinued. To quote David Christensen (thanks, David!): “The Omniture system works by replacing select content for randomly chosen users in an effort to track user behavior/response to proposed site changes. Alternate site content is created and dynamically replaced for these users as they use the site, such as the specific content on the checkout page in this instance.”

We verified that the Omniture A/B test’s JavaScript replacement code was alternately mangling and not mangling the year dropdown on the checkout form as mentioned. Our client took out the A/B test and the “security code errors” dropped back to a normal low level.

This was a difficult and expensive problem—not only was there business lost because of the problem, but there were a lot of resources put into troubleshooting it. We’ve come away from this episode with some lessons learned and with plenty of food for thought. I’ll leave it to commentators to opine away on this, including the End Point folks who scratched this itch: David Christensen, Jeff Boes, Mark Johnson, and Jon Jensen.

More Code and SEO with the Google Analytics API

2010-02-22T00:00:00+00:00

My latest blog article inspiration came from an SEOmoz pro webinar on Actionable Analytics. This time around, I wrote the article and it was published on SEOmoz’s YOUmoz Blog and I thought I’d summarize and extend the article here with some technical details more appealing to our audience. The article is titled Visualizing Keyword Data with the Google Analytics API.

In the article, I discuss and show examples of how the number of unique keywords receiving search traffic has diversified or expanded over time and that our SEO efforts (including writing blog articles) are likely resulting in this diversification of keywords. Some snapshots from the articles:

[The unique keyword (keywords receiving at least one search visit) count per month (top) compared to the number of articles available on our blog at that time (bottom).]

I also briefly examined how unique keywords receiving at least one visit overlapped between each month and saw about 10-20% of overlapping keywords (likely the short-tail of SEO).

[The keyword overlap per month, where the keywords receiving at least one visit in consecutive months are shown in the overlap section.]

Now, on to things that End Point’s audience may find more interesting. Something that might appeal more to our developer-types is the code written to use the Google Analytics API to generate the data used for this article. I researched a bit and tried writing my own ruby code (gem-less) to pull from the Google API, followed by using the Gattica gem, and finally the garb gem. After wrestling with the former two options, I settled on the garb gem, which had decent documentation here to get me up and running with a Google Analytics report quickly. Here’s an example of the code required to create your first Google Analytics API report:

#!/usr/bin/ruby

require 'rubygems'
require 'garb'

# set email, password, profile_id
Garb::Session.login(email, password)
profile = Garb::Profile.first(profile_id)

report = Garb::Report.new(profile,
        :limit => 100,
        :start_date => Date.today - 30,
        :end_date => Date.today)
report.dimensions :keyword
report.metrics :visits
report.results.each do |result|
  puts "#{result.keyword}:#{result.visits}"
end

If you aren’t familiar with the Google Analtyics API, possible dimensions and metrics are documented here. There are some Google Analytics API limitations on metric and dimension combinations, but I think if you get creative you’d be able to overcome most of those limitations (assuming you won’t be exceeding the limit of 1,000 API requests per day).

Why should you care about the Google Analytics API? Well, the API allowed me to programmatically aggregate the keyword counts in monthly increments for the SEOmoz article. One thing I consider to be pretty lame is the inability to select more than 3 custom segments and exclude the “All Visits” segment to allow a better visual comparison of the segments. In the data below, I have 3 defined custom segments. I would prefer to compare about 10 custom segments of End Point’s blog keyword groupings (e.g., “Rails Keywords”, “Postgres Keywords”), but Google Analytics limits the selected segments and includes “All Visits” when you select more than one custom segment.

Another thing I consider to be lame is the inability to merge Google Analytics profiles. Recently, End Point combined its corporate blog GA profile with its main website GA profile to better track conversion between the sites:

[Dead metrics from migrated profile.]

With the Google Analytics API, we could compute different aggregates of data, compare more than a few custom data segments, and combine two google profiles if they have merged. Of course, these things wouldn’t necessarily be easy, but working with the gem proved to be simple, so in theory this all could be done and in the meantime we’ll keep our dead profile around.

Again, please read the original article here if you are interested. :)

Safari 4 Top Sites feature skews analytics

2010-02-13T00:00:00+00:00

Safari version 4 has a new “Top Sites” feature that shows thumbnail images of the sites the user most frequently visits (or, until enough history is collected, just generally popular sites).

Martin Sutherland describes this feature in details and shows how to detect these requests, which set the X-Purpose HTTP header to “preview”.

The reason this matters is that Safari uses its normal browsing engine to fetch not just the HTML, but all embedded JavaScript and images, and runs in-page client JavaScript code. And these preview thumbnails are refreshed fairly frequently—possibly several times per day per user.

Thus every preview request looks just like a regular user visit, and this skews analytics which see a much higher than average number of views from Safari 4 users, with lower time-on-site averages and higher bounce rates since no subsequent visits are registered (at least as part of the preview function).

The solution is to simply not output any analytics code when the X-Purpose header is set to “preview”. In Interchange this is easily done if you have an include file for your analytics code, by wrapping the file with an [if] block such as this:

[tmp x_purpose][env HTTP_X_PURPOSE][/tmp]
[if scratch x_purpose eq 'preview']
<!-- skip analytics for browser previews -->
[else]
(normal Google Analytics, Omniture SiteCatalyst, or other analytics code)
[/else]
[/if]

In Ruby on Rails you’d check request.env[“HTTP_X_PURPOSE”].

In PHP you’d check $_SERVER[“HTTP_X_PURPOSE”].

In Django you’d check request.META[“HTTP_X_PURPOSE”] or the equivalent request.META.get(“HTTP_X_PURPOSE”) (from the HttpRequest class).

And so on.

I confirmed the analytics tracking code was omitted by waiting for Safari to make its preview request and inspecting the response with the Fiddler proxy, on Windows. The same can be done for Safari on Mac OS X with a suitable Mac OS X HTTP proxy.

End Point Blogging Stats Year In Review

2010-01-04T00:00:00+00:00

Many of the blogs I follow recently published a list of most popular articles from 2009, so I thought we should too. End Point started the blog in July of 2008, and has since then backported our older technical articles and company news to the blog, which is why you may see older articles in the archives. In 2009, we published just over 150 articles.

Here’s a breakdown of article categories (the categorization was a bit difficult: the “Tips, Community, Etc.” was a fallback category for any articles that didn’t have more than a few similar articles):

In 2009, the top 10 articles with the highest number of unique visitors were:

The top 10 “Top Pages”, according to a tool provided by SEOmoz, which is calculated using SEOmoz popularity metrics based on external links and quality of external linking pages to the url, are:

It would probably be more interesting to normalize the data above by the total number of days published to get an idea of which articles were the most impacting. This is likely an example of normalization that search engines employ in popularity algorithms.

The lamest articles, ordered by the number of unique visitors ascending, were:

Phew—none of my articles were on that list :)

I should probably note that those aren’t really lame articles. They likely didn’t get a ton of traffic because they were published around the time we started our blog that received little traffic at the time and because they probably weren’t marketed through social media as some of our recent blog articles have been.

But what does this all mean?

This past year, I’ve approached blogging as a sales or marketing tool to get traffic from potential clients or to inform existing clients of the variety of services we provide. So, I was interested in comparing traffic between our blog and main site. The graph below shows the total page views for the main site and blog:

And I was interested in looking at the amount of referral traffic (page views again) our main site receives from the blog:

It’s pretty cool that in the last few months, our blog traffic has been 2 to 3 times as much as our main web site. And also interesting that 1/4 to 1/3 of the referral traffic to our main site came from our blog in the last few months. Referral traffic makes up roughly one third of our main site traffic, so we get about 10% of total traffic from blog referral traffic—which is fairly effective.

There are many other metrics I’m interested in, specifically segmenting our traffic into categories similar to the blog categories in the pie chart above to examine which topics bring us the most traffic. I hope we continue to blog about topics that bring us unique (and converting) traffic. But I’ll save discussion of those metrics for another blog article, that hopefully won’t make the “2010 Lamest Blog Articles” list :)

WordPress Plugin for Omniture SiteCatalyst

2009-11-18T00:00:00+00:00

A couple of months ago, I integrated Omniture SiteCatalyst into an Interchange site for one of End Point’s clients, CityPass. Shortly after, the client added a blog to their site, which is a standalone WordPress instance that runs separately from the Interchange ecommerce application. I was asked to add SiteCatalyst tracking to the blog.

I’ve had some experience with WordPress plugin development, and I thought this was a great opportunity to develop a plugin to abstract the SiteCatalyst code from the WordPress theme. I was surprised that there were limited Omniture WordPress plugins available, so I’d like to share my experiences through a brief tutorial for building a WordPress plugin to integrate Omniture SiteCatalyst.

First, I created the base wordpress file to append the code near the footer of the wordpress theme. This file must live in the ~/wp-content/plugins/ directory. I named the file omniture.php.

  <?php /*
    Plugin Name: SiteCatalyst for WordPress
    Plugin URI: https://www.endpointdev.com/
    Version: 1.0
    Author: Steph Powell
    */
    function omniture_tag() {
    }
    add_action('wp_footer', 'omniture_tag');
  ?>

In the code above, the wp_footer is a specific WordPress hook that runs just before the tag. Next, I added the base Omniture code inside the omniture_tag function:

...

function omniture_tag() {
?>
<script type="text/javascript">
<!-- var s_account = 'omniture_account_id'; -->
</script>
<script type="text/javascript" src="/path/to/s_code.js"></script>
<script type="text/javascript"><!--
s.pageName='' //page name
s.channel='' //channel
s.pageType='' //page type
s.prop1='' //traffic variable 1
s.prop2='' //traffic variable 2
s.prop3='' //traffic variable 3
s.prop4= '' //traffic variable 4
s.prop5= '' //traffic variable 5
s.campaign= '' //campaign variable
s.state= '' //user state
s.zip= '' //user zip
s.events= '' //user events
s.products= '' //user products
s.purchaseID= '' //purchase ID
s.eVar1= '' //conversion variable 1
s.eVar2= '' //conversion variable 2
s.eVar3= '' //conversion variable 3
s.eVar4= '' //conversion variable 4
s.eVar5= '' //conversion variable 5
/************* DO NOT ALTER ANYTHING BELOW THIS LINE ! **************/
var s_code=s.t();if(s_code)document.write(s_code)
--></script>
<?php
}

...

To test the footer hook, I activated the plugin in the WordPress admin. A blog refresh should yield the Omniture code (with no variables defined) near the tag of the source code.

After verifying that the code was correctly appended near the footer in the source code, I determined how to track the WordPress traffic in SiteCatalyst. For our client, the traffic was to be divided into the home page, static page, articles, tag pages, category pages and archive pages. The Omniture variables pageName, channel, pageType, prop1, prop2, and prop3 were modified to track these pages. Existing WordPress functions is_home, is_page, is_single, is_category, is_tag, is_month, the_title, get_the_category, the_title, single_cat_title, single_tag_title, the_date were used.

...

<script type="text/javascript"><!--
<?php
if(is_home()) {    //WordPress functionality to check if page is home page
        $pageName = $channel = $pageType = $prop1 = 'Blog Home';
} elseif (is_page()) {    //WordPress functionality to check if page is static page
        $pageName = $channel = the_title('', '', false);
        $pageType = $prop1 = 'Static Page';
} elseif (is_single()) { //WordPress functionality to check if page is article
        $categories = get_the_category();
        $pageName = $prop2 = the_title('', '', false);
        $channel = $categories[0]->name;
        $pageType = $prop1 = 'Article';
} elseif (is_category()) {    //WordPress functionality to check if page is category page
        $pageName = $channel = single_cat_title('', false);
        $pageName = 'Category: ' . $pageName;
        $pageType = $prop1 = 'Category';
} elseif (is_tag()) {     //WordPress functionality to check if page is tag page
        $pageName = $channel = single_tag_title('', false);
        $pageType = $prop1 = 'Tag';
} elseif (is_month()) {     //WordPress functionality to check if page is month page
        list($month, $year) = split(' ', the_date('F Y', '', '', false));
        $pageName = 'Month Archive: ' . $month . ' ' . $year;
        $channel = $pageType = $prop1 = 'Month Archive';
        $prop2 = $year;
        $prop3 = $month;
}
echo "s.pageName = '$pageName' //page name\n";
echo "s.channel = '$channel' //channel\n";
echo "s.pageType = '$pageType'  //page type\n";
echo "s.prop1 = '$prop1' //traffic variable 1\n";
echo "s.prop2 = '$prop2' //traffic variable 2\n";
echo "s.prop3 = '$prop3' //traffic variable 3\n";
?>
s.prop4 = '' //traffic variable 4

...

The plugin allows you to freely switch between WordPress themes without having to manage the SiteCatalyst code and to track the basic WordPress page hierarchy. Here are example outputs of the SiteCatalyst variables broken down by page type:

Homepage

s.pageName = 'Blog Home' //page name
s.channel = 'Blog Home' //channel
s.pageType = 'Blog Home'  //page type
s.prop1 = 'Blog Home' //traffic variable 1
s.prop2 = '' //traffic variable 2
s.prop3 = '' //traffic variable 3

Tag Page

s.pageName = 'chocolate' //page name
s.channel = 'chocolate' //channel
s.pageType = 'Tag'  //page type
s.prop1 = 'Tag' //traffic variable 1
s.prop2 = '' //traffic variable 2
s.prop3 = '' //traffic variable 3

Category Page

s.pageName = 'Category: Food' //page name
s.channel = 'Food' //channel
s.pageType = 'Category'  //page type
s.prop1 = 'Category' //traffic variable 1
s.prop2 = '' //traffic variable 2
s.prop3 = '' //traffic variable 3

Static Page

s.pageName = 'About' //page name
s.channel = 'About' //channel
s.pageType = 'Static Page'  //page type
s.prop1 = 'Static Page' //traffic variable 1
s.prop2 = '' //traffic variable 2
s.prop3 = '' //traffic variable 3

Article

s.pageName = 'Hello world!' //page name
s.channel = 'Test Category' //channel
s.pageType = 'Article'  //page type
s.prop1 = 'Article' //traffic variable 1
s.prop2 = 'Hello world!' //traffic variable 2
s.prop3 = '' //traffic variable 3

A followup step to this plugin would be to use the wp_options table in WordPress to manage the Omniture account id, which would allow admin to set the Omniture account id through the WordPress admin without editing the plugin code. I’ve uploaded the plugin to a GitHub repository here.

Update: This plugin is included in the WordPress plugin registry and can be found at https://wordpress.org/extend/plugins/omniture-sitecatalyst-tracking/.

Using the new-style Google Analytics pageTracker functions in Interchange

2009-06-05T00:00:00+00:00

For a while now there have been two different ways to setup the JavaScript calls to report traffic back to Google Analytics. The older method uses functions names that mention “urchin,” while the newer method uses a function named “pageTracker”. This post describes an approach for using the new method at a standard Interchange store.

You can see an example of the new method of reporting a page view here. Nothing Interchange-related is required for normal page tracking, but you may want to use a variable for the Google Account Number, of which more below.

If you have your Google Analytics account setup to treat the website as an E-commerce site, then you can also add the order tracking tags to your receipt page, so that it sends order data over to Google Analytics at the time of conversion. The order tracking tags can be viewed here. This gist shows the typical Interchange tags you might want to use to transmit the order specifics. Of course you might need to change the field used for the category for the products since not everyone uses the prod_group field from the products table to hold this information.

As you can see, both normal and the order-conversion scripts need to be modified to contain the individual Google Analytics account number for the website. I tend to set up an Interchange variable such as GOOGLE_ANALYTICS_ID in the variable.txt file or catalog.cfg.