Integrating Contentful with NuxtJS

2022-10-07T00:00:00+00:00

Some time ago, I had the opportunity to collaborate on a cool NuxtJS project. I’m still somewhat new to Vue.js and its related frameworks, meaning I’m discovering exciting new tools and third-party services that can be integrated with them every time a new requirement appears. And there is a particular concept that I heard of, but never worked with until this project: using a Headless CMS to deliver content.

Essentially, a headless CMS permits creating a custom content model, making it accessible through one (or several) APIs that we can query, allowing us to choose whatever presentation layer we prefer to handle the display. This approach decouples the content management part (the “body”) of a project from the design, templates, and frontend logic (the “head”), becoming particularly useful when we have several application types that will interact with the same data, such as a website, a mobile app, or an IoT device.

With that in mind, let’s have a quick look at Contentful: It’s a headless CMS that is offered under the concept of content-as-a-service (CaaS), meaning the content is delivered on demand from a cloud platform to the consumer by implementing an API or web service.

Pricing

For individual or small websites, the free option should be sufficient. It has a limit of 5 users and a size limit of 50MB for assets, and the technical support area is disabled. The next option (Medium, $489/month) also includes an additional role (author), additional locales, and the possibility to create up to ten different user accounts. The asset size is also extended up to 1000MB.

You can review the full pricing details on Contentful’s website.

Creating content

In order to start creating content, there are two essential steps:

We will need to set up a new space. A space is an area where the content will be grouped into a single project.
We need to define the model for our content. The model is the type and structure that our content will have. For the integration below, we will need a new “Page” model, that will contain two fields to save the information that we need for a static page: title and content. We can also include a publishDate field, just to know when each page was created.

Integration

With our space created and our content model ready, it’s time to add Contentful to our NuxtJS app. The integration process is quite simple, thanks to the JavaScript client library provided by Contentful. The library is based on the axios client, allowing it to run on the client as well as on the server, for SSR. If we have npm set up, we can add it to our project by running:

npm install --save contentful

The most efficient way to use it across our app and have it ready for client and server-side rendering is to declare a new plugin. All we need to do is create a new file named contentful.js under our project’s plugins folder:

const contentful = require('contentful')

const config = {
  space: process.env.CONTENTFUL_SPACE_ID,
  accessToken: process.env.CONTENTFUL_API_ACCESS_TOKEN,
}

module.exports = {
  createClient() {
    return contentful.createClient(config)
  },
}

Next, we need to add the new environment variables to our project’s .env file. The values that we need to provide are our space ID and the access token for querying the API:

CONTENTFUL_SPACE_ID={our_space_id}
CONTENTFUL_API_ACCESS_TOKEN={our_access_token}

We’re all set! Now, we have our plugin ready to use. One neat extra step that we did for this particular project is creating a ContentfulPage component that will automatically pull the contents from Contentful based on the given entry ID. By doing that, we can simply use the component in all the static pages that we have on our website.

First, let’s create the component, containing a simple wrapper for the template section, and an entryId property that we will use to query the API. We can save it under ~/components/ContentfulPage.vue:

<template>
  <div :id="entryId">
    <p v-if="$fetchState.pending">Loading...</p>
    <div v-else>
      <h1>
        {{ page.fields.title }}
      </h1>
      <div class="page-content" v-html="$md.render(page.fields.content)" />
    </div>
  </div>
</template>

<script>
  import { createClient } from '~/plugins/contentful'
  const contentful = createClient()

  export default {
    name: 'ContentfulPage',
    props: {
      entryId: {
        type: String,
        required: true,
      },
    },

    data() {
      return {
        page: {},
      }
    },

    async fetch() {
      this.page = await contentful.getEntry(this.entryId)
    },
  }
</script>

This component will asynchronously fetch the entry from Contentful, and display a “loading” legend while it does so. Once the query is complete, the content will be shown inside the div element with the page-content class. The component expects the returned page to have at least two attributes: title and content.

With the new component added to our project, we are ready to create a page (for example, index.vue) that uses it to render our content like this:

<template>
  <div>
    <contentful-page entry-id="{index_entry_id}" />
  </div>
</template>

<script>
  import ContentfulPage from '~/components/ContentfulPage'
  export default {
    components: {
      ContentfulPage,
    },
  }
</script>

All we need to do is get the entry ID from the content we created in Contentful, pass it to the entry-id parameter of the component, and that’s it! Our content will be fetched from the API and displayed to the user. It will also work when rendering on the client, as well as for SSR.

Alternatives

There are several alternatives to Contentful out there: Prismic CMS or GraphCMS — which is based entirely in GraphQL — are the most popular. There are also downloadable products, like SilverStrap CMS. Their pricing plans are varied, but all of them offer a free community plan for starters or small websites.

Have you used any other headless CMS not listed here? We would love to hear your comments!

Advanced WordPress customizations

2020-11-27T00:00:00+00:00

Photo by Chris Harrison on Flickr, CC BY 2.0, cropped

WordPress is the most popular CMS, allowing you to create a professional-looking website at a relatively low cost. It also has a really strong community behind it, creating great content and supporting developers across the world.

But being popular also means being the main target of hacker attacks, and that’s why it’s crucial to keep the CMS, the theme, and all the plugins updated. When the requirements go beyond what WordPress offers on the surface, we need to find an efficient way to add our custom logic into the CMS without interfering with version upgrades, keeping the focus on security.

Custom CSS and JavaScript

A pretty common scenario in WordPress consists of installing a theme that fits most of our requirements and writing an extra layer of functionality over it to get that custom look and user experience we are looking for. Updating the theme files means that we cannot easily upgrade or change the theme without backing up our changes, and then restoring them after the upgrade, so that’s definitely not a good approach.

To make our way around this issue, some themes offer a section to add custom JavaScript or CSS rules. But sometimes we need change themes in the middle of our developing process, so I usually rely on plugins to make my frontend changes. One simple, straightforward plugin I usually install to add custom styles and frontend scripts is Simple Custom CSS and JS.

It has several customization options I usually need for SEO purposes, including the possibility to create several independent code segments and load each one in a different section (header/footer) to improve loading speed. We can also include our custom content as embedded source/styles or as external references. It also includes an editor with syntax highlighting, and allows adding custom content to the WordPress admin section.

Custom PHP code

Another thing I usually need when customizing a WordPress website is the ability to run my own PHP code inside a WordPress page or post. That is not allowed by the CMS by default, but like most things in WordPress, there is a plugin that will make our lives easier: It’s called Insert PHP Code Snippet.

You can create your own PHP routines as snippets that can be inserted with shortcodes into a WordPress page, posts, template, or whenever you can add a shortcode. This will allow running any custom backend code anywhere on the website.

To run the code on a page or post, all that needs to be done is pasting the shortcode on the content with the “PHP” button that appears on the button bar on the editor:

Custom hooks

When our logic needs to be fired up from an event on the CMS, or if we need to make changes to the default WordPress behavior or data, we will need to use the add_action() function or the add_filter() function.

add_action() allows us to execute a PHP function on specific points of the WordPress execution, for example when a post is created or commented, or when a user is created. A full list of actions is available here.
add_filter() allows us to update information associated with WordPress, for example, to set a custom CSS class for the body or to replace the login button text. A full list of filters is available here.

The following example sends an email to the webmaster when a new comment is created, using the add_action() function:

function email_comment() {
  wp_mail('webmaster@website.com', 'New comment', 'New comment posted on the website');
}
add_action('comment_post','email_comment');

Here’s another example that allows to perform a string replacement across the website by using the add_filter() function:

function replace_text($text) {
  return str_replace('Text to search', 'Text to replace with', $text);
}
add_filter('gettext', 'replace_text');

We can add these code segments using the PHP Snippet plugin (recommended) or using the functions.php file included in our theme. This last option is not recommended since it has the difficulty we discussed above about missing our custom content after upgrading the theme.

Conclusion

WordPress is a great CMS with an elegant backend and a big list of themes and plugins we can use. But before we start adding custom code, we need to make sure it will persist after upgrading everything else to the latest version.

The techniques we saw in this post are meant for websites that only need some small custom changes or improvements. If we need to include a complete layer of logic into a website, it’s always recommended to write a custom plugin from scratch, if there is nothing out there that serves the purpose. But that’s material for another blog post!

At End Point, we build web solutions in WordPress and many other technologies, keeping security and high standards in mind. Contact us if you want to find out more.

Craft: A CMS for developers

2020-10-31T00:00:00+00:00

As a software engineer, I thrive and thoroughly enjoy working on fully custom software products, applications conceived to model and help in the execution of some business process and that are built from the ground up by a team of developers.

Such projects are often complex and expensive, though, and for some clients, they can be overkill. Some clients come up with requirements that are better served by off-the-shelf software solutions. One group of such solutions are content management systems (CMS). As a rule of thumb, if a client wants a website whose main purpose is to showcase some content, their brand or image, and custom business logic requirements are limited, then chances are that a CMS will fit the bill nicely.

Lately we’ve been using the Craft CMS for a client that meets the aforementioned criteria, and I gotta say, I’ve been pleasantly surprised by the developer experience it offers.

Unlike most of the technology and products we discuss in our blog, Craft CMS is not Open Source or Free Software. The source code is readily available in GitHub for anybody to use, study, and modify, but commercial use of it is restricted and certain features are exclusive to a so-called “Pro” edition. Learn more by reading their license and their pricing structure.

The features that we will discuss in this article are all part of the no-charge “Solo” edition of Craft CMS 3 that can be used for noncommercial websites.

In this article I’m going to talk through a few of the key aspects of Craft that make me think that it’s really a CMS made for developers. Let’s get started:

Craft is easy to get up and running

Craft is just a PHP application. And it is as typical as modern PHP applications go, capable of being initially set up with Composer and of running on top of a MySQL database (it also supports Postgres!) and the Apache web server. It can all be done via console too, if that’s how you roll.

If you already have a box with Apache, PHP, MySQL and Composer, it all amounts to little more than creating a MySQL database for Craft, composer installing the Craft package, sorting out some permissions, running php craft setup, following the prompts, and finally, configuring a virtual host in Apache to serve the web directory from inside where Craft was installed.

All of this is explained in Craft’s official documentation.

Craft is easy to put in containers

For ease of development and project bootstrapping, I’ve created a containerized setup with Docker and Docker Compose that encapsulates some infrastructure tailored to my development needs. You can get the relevant files here.

If you want to follow along, clone that repo, and you’ll end up with this file structure (as shown by the tree command):

.
├── apache_config
│   └── 000-default.conf
├── docker-compose.yml
├── Dockerfile
└── README.md

1 directory, 4 files

This setup includes two containers: one for running Apache and Craft, and another for running MySQL. The apache_config/000-default.conf contains some Apache VirtualHost configuration for serving the site. docker-compose.yml defines the whole infrastructure: both containers, a network that they use to talk to each other, and a volume to persist MySQL database files. The Dockerfile is the definition of the image for the container that runs Apache and Craft.

Feel free to explore the files; I’ve made sure to comment them so that they are easy to understand and modify as you see fit.

Note: If you want to run this setup, be sure to change the ServerAdmin value in apache_config/000-default.conf, and the USER, UID, and GID values in docker-compose.yml under services > web > build > args according to your environment and user account information.

If you have Docker and Docker Compose installed in your machine, you can go to the directory just created by the git clone and:

Run docker-compose up to set up the infrastructure. You will see Docker and Docker Compose creating the image defined in Dockerfile and the containers defined in docker-compose.yml. Then the logs of the various containers will start showing. It you want to run this in the background, use docker-compose up -d instead and it will give you control of the terminal immediately after it’s done.
Run docker-compose exec web bash to connect to the web container. This is the container that has Craft’s code and is running Apache. You’ll be “logged into” the container and be placed in /var/www. This is the directory where we will install Craft.

Once in there, run composer create-project craftcms/craft ./install to install Craft with Composer. In other words, it will download all of the files that Craft needs to run. You should see something like this at the end:

> @php craft setup/welcome

   ______ .______          ___       _______ .___________.
  /      ||   _  \        /   \     |   ____||           |
 |  ,----'|  |_)  |      /  ^  \    |  |__   `---|  |----`
 |  |     |      /      /  /_\  \   |   __|      |  |
 |  `----.|  |\  \----./  _____  \  |  |         |  |
  \______|| _| `._____/__/     \__\ |__|         |__|

     A       N   E   W       I   N   S   T   A   L   L
               ______ .___  ___.      _______.
              /      ||   \/   |     /       |
             |  ,----'|  \  /  |    |   (----`
             |  |     |  |\/|  |     \   \
             |  `----.|  |  |  | .----)   |
              \______||__|  |__| |_______/


Generating an application ID ... done (CraftCMS--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
Generating a security key ... done (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)

Welcome to Craft CMS! Run the following command if you want to setup Craft from your terminal:

    /var/www/install/craft setup

After that’s done, use this bit of bash black magic:
```
(shopt -s dotglob; mv -v ./install/* .)
```
This command moves all the files that composer just downloaded into /var/www/install out from there and into /var/www. Then rmdir install to remove the install directory because we no longer need it.
Now do php craft setup to use Craft’s CLI to set up the site, its configuration, and its database structure. Just follow the prompts. When it asks you for database configuration, choose mysql as the database driver, set mysql as the database server name (because that’s the name we’ve given it in our docker-compose.yml), and use the environment variables defined in docker-compose.yml around lines 14 to 17 for the rest of the values. You can change these to whatever you want as long as you make sure that it coincides with how you defined your MySQL database container in the docker-compose.yml file. The prompts should look something like this:
```
Which database driver are you using? [mysql,pgsql,?]: mysql
Database server name or IP address: [127.0.0.1] mysql
Database port: [3306]
Database username: [root] craft
Database password:
Database name: craft_demo
Database table prefix:
Testing database credentials ... success!
Saving database credentials to your .env file ... done
```
Next, sort out some Craft file permission requirements with this command:
```
chmod -R o+w config storage web/cpresources
```
These are directories that Craft needs write access to.
Now start up Apache with sudo service apache2 start.

And that’s it! Open a browser to localhost or 127.0.0.1 and you should see your Craft 3 homepage:

You can start playing with the control panel or the templates/index.twig right away.

Craft’s design makes sense

When it comes to content modeling, Craft offers a set of abstractions that make sense. The main concepts to understand are sections and entries. Entries are the main pieces of content. An “article” in a news site or a “post” in a blog. Sections are the way Craft groups entries together. They are useful when your site has multiple streams of content. You can, for example, have a site where you publish news, opinion pieces, and random thoughts. With Craft, that would translate neatly into three separate sections, each one with its own type of entries.

Craft also allows you to set up custom fields for every type of entry. For example, the entries on your news section may need to include a link to the original source of the news, while your opinion pieces need a short description instead. You can configure your entries using custom fields so that they include the data that makes sense for your use case.

Let’s see what that looks like in concrete terms. Now that we have a Craft instance running in localhost, go to localhost/admin in your browser. You should see Craft’s control panel. Click on the “Settings” option in the navigation bar to the left of the screen, then select the “Sections” item under “Content”:

Next, click on the “+ New Section” button by the top of the screen and you’ll be shown the section creation form. We will create a “News” section so let’s fill in the form like this:

The “Name” and “Handle” fields are pretty self explanatory. The “Section Type” is a concept we haven’t discussed yet. “Channel” is the most appropriate for a news section, which is a stream of multiple entries with the same structure.

There are other types: “Single” is a type which you would use for entries that are unique, like a home or contact page. For sections of type “Single”, there’s generally one single entry that fits in them. This is unlike “Channels” which fit multiple entries. The other section type is “Structure”, which also accommodates multiple entries, but rather than a stream of ever-growing content, it’s more appropriate for similar entries that share a certain theme. A “Structure” section type is appropriate for things like services offered or projects in a portfolio.

Learn all about entries, sections, section types, and more in Craft’s official docs.

Now that we’ve filled the form, click the “Save and edit entry types” button. This has created our new “News” section and defined the “entry type” that this section will be able to contain. The control panel now shows this:

In Craft, a section can contain multiple types of entries. For our purposes with the news section, though, just the default one is enough. Click on it, and you’ll see an editor where you can select fields that make up that entry type:

The editor I mentioned before is below the “Field Layout” title. Here’s where we can pick and choose which fields make up the entries for the “News” section. We have a fresh installation of Craft though, so we don’t have any fields. Let’s create a few by going to Settings > Fields.

This is where we can define new fields to be used for our entries throughout the site. Click the “+ New Field” button near the top of the screen and you’ll be presented with the field creation form where you can specify all manner of details. For now, we just care about “Name”, “Handle” and “Field Type”. Let’s create three fields:

One named “Heading” with a type of “Plain Text”.
One named “Body” with a type of “Plain Text”.
One named “Source” with a type of “URL”.

You should end up with something like this in the control panel’s “Fields” page (localhost/admin/settings/fields):

Now, if we go back to our “News” section’s default entry type at http://localhost/admin/settings/sections/1/entrytypes/1 or Settings > Sections > News > Entry Types > News…

You can see how the new fields that we just created are present in the “Field Layout” panel. In order to make these fields available for our “News” entries, we just need to drag them into the box named “Content” inside the greyish area.

Click the “Save” button at the top, and that’s all it takes to set up a “Channel” section, an entry type for it, and a few fields.

Now that we’ve set up the blueprints for them, let’s actually create a few entries in the “News” section. To do so, click on the “Entries” link in the navigation bar to the left which should’ve revealed itself by now, and you’ll see this screen:

If you’re used to CMS back ends, this is pretty familiar. In this screen you can create new entries and browse existing ones.

Click the big red “+ New Entry” button and select “News” in the resulting pop-up menu. You should see a form with the fields that we defined in the “Field Layout” panel during previous steps. Feel free to create a few news entries. I’ve created these two:

Craft gives you complete freedom over your front end

Most CMSs can be thought of as having two components: a front end and a back end. The back end is where content is authored and the front end is where the style and structure in which the content is presented. In Craft, most of the effort has gone into creating a solid, highly customizable back end.

As we’ve just seen, Craft comes out of the box with a back end control panel where site administrators and content creators can author new content. As far as front end goes though, Craft has nothing. For a developer well versed in front end web technologies, this is freeing and transformative.

Craft makes no assumption and makes no decision for you when it comes to developing your site’s look and feel. It gets out of your way and lets you do your job. There’s no concept of “theme”. There’s no obscure framework to learn and integrate into. There’s no proprietary templating language to struggle with. In Craft, you are completely free to write HTML, CSS and JS as you see fit to obtain your desired effect for your site.

You can develop templates using the tried and true Twig templating engine, which, if you have some experience with PHP, you’ve most likely already encountered and worked with. All the content created in the back end is exposed to the Twig templates via objects. Let’s see how.

First we need to specify a template for our sections. Continuing with our example, let’s assign a template to our “News” section. Go to Settings > Sections > News and scroll down to find the “Site Settings” area. In the table there, type news into the “Template” column. Now go to the template directory where craft was installed and create a new news.twig file. The contents can be simple, like this:

<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{{ entry.title }} - My Craft Demo</title>
</head>
<body>
    <h1>{{ entry.heading }}</h1>
    <p>{{ entry.body }}</p>
    <p><a href="{{ entry.source }}">Source</a></p>
</body>
</html>

The only noteworthy aspect of this template is how we are injecting the data that we defined in the back end into this template. We use double curly brackets to reference the entry variable. This is provided to Twig by Craft and contains all the fields that we defined for our entries in the “News” section.

With that done, save and visit any of the entries you created and you’ll see something like this:

As you can see, this entry’s URL is localhost/news/i-just-learned-that-craft-uses-twig. Yours will obviously differ depending on the title (and slug) that you gave them.

What this example lacks in complexity, it more than makes up for in potential. This is a plain old HTML document that we’ve created, with a Twig template, of course. This is the complete freedom that I like about Craft. From this point on, you can do whatever you want in terms of front end development: use whatever CSS or JavaScript framework or library you want, organize your template files in a way that makes sense to you, your team, and your website, etc. The sky is the limit.

That’s good for individual news pages. But now let’s try to link to them from the homepage. To do so, we need to edit the templates/index.twig file. Around line 174, remove the <ul> that’s there along with all its <li>s and put this instead:

{% set entries = craft.entries().section('news').all() %}

<ul>
    {% for entry in entries %}
        <li><a href="{{ entry.url }}">{{ entry.title }}</a></li>
    {% endfor %}
</ul>

Here, we leverage Twig’s templating engine capabilities, sprinkled with some of Craft’s features to obtain a list of all the entries in our “News” section. Then, we iterate over them to render links.

Effectively, Craft enhances what you can do with Twig by exposing an API for accessing the data that exists in the CMS back end.

If you’re familiar with any sort of templating language like those included in most web application frameworks like Ruby on Rails, Symfony, ASP.NET Core MVC, etc., you’ll probably feel right at home with this.

Here’s what the homepage looks like now:

You can click on any of the links and they will take you to the specific entry page.

You can learn more about querying entries in Craft’s official documentation.

Craft is cool 🕶️

So, in conclusion, I’ve found that Craft is a cool tool to have in the toolbox. It is a full-fledged CMS with tons of customization opportunities for how to model and organize the content and data of your site. When it comes to developing the front end, though, it gets out of your way and lets you do your job. That, to me, is a big win.

Drupal — rapid development

2017-05-26T00:00:00+00:00

Here at End Point, we had the pleasure to be a part of multiple Drupal 6, 7 and 8 projects. Most of our clients wanted to use the latest Drupal version, to have a long term support, stable platform.

A few years ago, I already had big experience with PHP itself and other, various PHP frameworks like WordPress, Joomla! or TYPO3. I was happy to use all of them, but then one of our clients asked us for a simple Drupal 6 task. That’s how I started my Drupal journey which continues until now.

To be honest, I had a difficult start, it was different, new and pretty inscrutable for me. After a few days of reading documentation and playing with the system I was ready to do some simple work. Here, I wanted to share my thoughts about Drupal and tell you why I LOVE! it.

Low learning curve

It took, of course, a few months until I was ready to build something more complex, but it really takes a few days only to be ready for simple development. It’s not only about Drupal, but also PHP, it’s much cheaper to maintain and extend a project. Maybe it’s not so important with smaller projects, but definitely important for massive code bases. Programmers can jump in and start being productive really quick.

Great documentation

Drupal documentation is well structured and constantly developed, usually you can find what you need within a few minutes. It’s critical and must have for any other framework and not so common unfortunately.

Big community

The Drupal community is one of the biggest IT communities I have ever encountered. They extend, fix and document the Drupal core regularly. Most of them have their other jobs and work on this project just for fun and with passion.

It’s free

It’s an open source project, that’s one of the biggest pros here. You can get it for free, you can get support for free, you can join the community for free too (:)).

Modules

On the official Drupal website you can find tons of free plugins/modules. It’s a time and money saver, you don’t need to reinvent the wheel for every new widget on your website and focus on fireworks.

Usually you can just go there and find a proper component. E-commerce shop? Slideshow? Online classifieds website? No problem! It’s all there.

PHP7 support

I can often hear from other developers that PHP is slow, well, it’s not the Road Runner, but come on, unless you are Facebook (and I think that they, correct me if I’m wrong, still use PHP :)) it’s just OK to use PHP.

Drupal fully supports PHP7.

With PHP7 it’s much faster, better and safer. To learn more: https://pages.zend.com/rs/zendtechnologies/images/PHP7-Performance%20Infographic.pdf.

In the infographic you can see that PHP7 is much faster than Ruby, Perl and Python when you try to render a Mandelbrot fractal. In general, you definitely can’t say that PHP is slow, same as Drupal.

REST API support

Drupal has the built in, ready to use API system. In a few moments you can spawn a new API endpoint for you application. You don’t need to implement a whole API by yourself, I did it a few times in multiple languages, believe me, it’s problematic.

Perfect for a backend system

Drupal is a perfect candidate for a backend system. Let’s imagine that you want to build a beautiful, mobile application. You want to let editors, other people to edit content. You want to grab this content through the API. It’s easy as pie with Drupal.

Drupal’s web interface is stable and easy to use.

Power of taxonomies

Taxonomies are, really basically, just dictionaries. The best thing about taxonomies is that you don’t need to touch code to play with them.

Let’s say that on your website you want to create a list of states in the USA. Using most of the frameworks you need to ask your developer/technical person to do so. With taxonomies you just need a few clicks and that’s it, you can put in on your website. That’s sweet, not only for non technical person, but for us, developers as well. Again, you can focus on actually making the website attractive, rather than spending time on things that can be automated.

Summary

Of course, Drupal is not perfect, but it’s undeniably a great tool. Mobile application, single page application, corporate website—there are no limits for this content management system. And actually, it is, in my opinion, the best tool to manage your content and it does not mean that you need to use Drupal to present it. You can create a mobile, ReactJS, AngularJS, VueJS application and combine it with Drupal easily.

I hope that you’ve had a good reading and wish to hear back from you! Thanks.

Testing Django Applications

2015-12-14T00:00:00+00:00

This post summarizes some observations and guidelines originating from introducing the pytest unit testing framework into our CMS (Content Management System) component of the Liquid Galaxy. Our Django-based CMS allows users to define scenes, presentations and assets (StreetView, Earth tours, panos, etc) to be displayed on the Liquid Galaxy.

The purpose of this blog post is to capture my Django and testing study points, summarize useful resource links as well as to itemize some guidelines for implementing tests for newcomers to the project. It also provides a comparison between Python’s standard unittest library and the aforementioned pytest. Its focus is on Django database interaction.

Versions of software packages used

This post describes some of our experiences at End Point in designing and working on comprehensive QA/CI facilities for a new system which is closely related to the Liquid Galaxy.

The experiments were done on Ubuntu Linux 14.04:

python (2.7.6) and its corresponding version of unittest
django 1.7 (current recent is 1.9 but our CMS uses still 1.7 version)
pytest-django 2.8.0
pytest 2.7.2 (with py 1.4.30)
virtualenv 13.1.2
factory_boy 2.6.0

Testing Django Applications

We probably don’t need to talk much about the importance of testing. Writing tests along with the application code has become standard over the years. Surely, developers may fall into a trap of their own prejudice when creating testing conditions which would still result in faulty software but the likelihood of buggy software is certainly higher on a code that has no QA measures. If the code works and is untested, it means it works by accident, they say.

As a rule of thumb, unit tests should be very brief testing items seldom interacting with any external services such as the database. Integration tests on the other hand often communicate with external components.

This post will heavily reference an example minimal Django application written for the purpose of experimenting on Django testing. Its README file contains some set up and requirement notes. Also, I am not going to list (m)any code snippets here but rather reference the functional application and its test suite. Hence the points below qualify for more or less assorted little topics or observations.

In order to benefit from this post, it will be helpful to follow the README and interact (run tests that is) with the demo django-testing application.

Basic Django unittest versus pytest basic examples

This pair of test modules shows the differences between Django TestCase (unittest) and pytest-django (pytest) frameworks.

test_unittest_style.py

The base Django TestCase class derives along this tree:

    django.test.TestCase
        django.test.TransactionTestCase
            django.test.SimpleTestCase
                unittest.TestCase

Django adds (among any other aspects) handling of database, the documentation is here, on top of the Python standard unittest library.

test_pytest_style.py

this is a pytest style implementation of the same tests and pytest-django plug-in adds, among other features, Django database handling support.

The advantage of unittest is that it comes with the Python installation—it’s a standard library. That means that one does not have to install anything for writing tests, unlike pytest which is a third-party library and needs to be installed separately. While the absence of additional installation is certainly a plus, it’s dubious whether being a part of Python distribution is a benefit. I seem to recall Guido Van Rossum during Europython 2010 having said the the best thing for pytest is not being part of the Python standard set of libraries for its lively development and evolution would be slowed down by the inclusion.

There are very good talks and articles summarizing advantages of pytest. For me personally, the reporting of error context is supreme. No boiler-plate (no inheritance), using plain Python asserts instead of many assert* methods and flexibility (function, class) are other big plus points

Testing Django applications with py.test (EuroPython 2013) (very good)
pytest presentation from its author
very brief pytest introduction
switch to pytest, rich features descriptions, has some Django touches

As the comment in the test_unittest_style.py file says, this particular unittest-based test module can be run by both Django manage.py (which boils down to unittest lookup discovery on a lower layer) or by py.test (pytest).

It should also be noted, that pytest’s flexibility can bite back if something gets overlooked.

Django database interaction unittest versus pytest (advanced examples)

test_unittest_advanced.py

Since this post concentrates on pytest and since it’s the choice for our LG CMS project (naturally :-), this unittest example just shows how the test (fresh) database is determined and how Django migrations are run at each test suite execution. Just as described in the Django documentation: “If your tests rely on database access such as creating or querying models, be sure to create your test classes as subclasses of django.test.TestCase rather than unittest.TestCase.”

That is true for database interaction but not completely true when using pytest. And “Using unittest.TestCase avoids the cost of running each test in a transaction and flushing the database, but if your tests interact with the database their behavior will vary based on the order that the test runner executes them. This can lead to unit tests that pass when run in isolation but fail when run in a suite.” django.test.TestCase, however, ensures that each test runs inside a transaction to provide isolation. The transaction is rolled back once the test case is over.

test_pytest_advanced.py

This file represents the actual core of the test experiments for this blog / demo app and shows various pytest features and approaches typical for this framework as well as Django (pytest-django that is) specifics.

Django pytest notes (advanced example)

Much like the unittest documentation, the pytest-django recommends avoiding database interaction in unittest and concentrate only on the logic which should be designed in such a fashion that it can be tested without database.

test database name prefixed test_ (just like at the unittest example), the base value is taken from the database section of the settings.py. As a matter of fact, it’s possible to run the test suite after previously dropping the main database, the test suite interacts only with test_ + DATABASE_NAME
migration execution before any database interaction is carried out (similarly to unittest example)
database interaction marked by a Python decorator @pytest.mark.django_db on the method or class level (or stand-alone function level). It’s in fact the first occurrence of this marker which triggers the database set up (its creation and migrations handling). Again analogously to unittest (django.test.TestCase), the test case is wrapped in a database transaction which puts the database back into the state prior to the test case. The database test_ + DATABASE_NAME itself is dropped once the test suite run is over. The database is not dropped if --db-reuse option is used. The production DATABASE_NAME remains untouched during the test suite run (more about this below)
pytest_djangodb_only.py — setup_method — run this module separately and the data created in setup_method end up NOT in the test_ + DATABASE_NAME database but in the standard one (as configured in the settings.py which would be the production database likely)! Also this data won’t be rolled back. When run separately, this test module will pass (but still the production database would be tainted). It may or may not fail on the second and subsequent run depending whether it creates any unique data. When run within the test suite, the database call from the setup_method will fail despite the presence of the class django_db marker. This has been very important to realize. Recommendation: do not include database interaction in the pytest special methods (such assetup_method or teardown_method, etc), only include database interaction in the test case methods
The error message Failed: Database access not allowed, use the "django_db" mark to enable was seen on a database error on a method which actually had the marker. This output is not to be 100% trusted
data model factories are discussed separately below
lastly the test module shows Django Client instance and calling an HTTP resource

pytest setup_method

While the fundamental differences between unittest and pytest were discussed, there is something to be said about Django specific differences of the two. There is different database-related behaviour of unittest setUp method versus the pytest setup_method method. The setUp is included in the transaction and database interactions are rolled back once the test case is over. The setup_method is not included in the transaction. Moreover, interacting with the database from setup_method results in faulty behaviour and difference depending whether the test module is run on its own or as a part of the whole test suite.

The bottom line is: do not include database interaction in setup_method. This setUp, setup_method behaviour was already shown in the basic examples. And more description and demonstration of this behaviour is in the file: pytest_djangodb_only.py. This actually revealed the fact that using django_db database fixture is not supported in special pytest methods and the aforementioned error message is misleading (more references here and here).

When running the whole test suite, this file won’t be collected (its name lacks test_ string). It needs to be renamed to be included in the test suite run.

JSON data fixtures versus factories (pytest advanced example)

The traditional way of interacting with some test data was to perform following steps:

have data loaded in the database
python manage.py dumpdata
the produced JSON file is dragged along the application test code
call_command(“loaddata”, fixture_json_file_name) happens at each test suite run

The load is expensive, the JSON dump file is hard to maintain manually if the original modified copy and the current needs diverge (the file has integer primary keys value, etc). Although even the recent Django testing documentation mentions usage of JSON data fixtures, the approach is considered discouraged and the goal is recommended to achieve by means of loading the data in migrations or using model data factories.

This talk for example compares the both approaches in favour of factory_boy library. A quote from the article: “Factory Boy is a Python port of a popular Ruby project called Factory Girl. It provides a declarative syntax for how new instances should be created. … Using fixtures for complex data structures in your tests is fraught with peril. They are hard to maintain and they make your tests slow. Creating model instances as they are needed is a cleaner way to write your tests which will make them faster and more maintainable.”

The file test_pytest_advanced.py demostrates interaction with factories defined in the module factories.py, the basic very easy-to-use features.

Despite its ease of use, the factory_boy is a powerful library capable of modeling Django’s ORM many-to-many relationships, among other features.

Additional useful links

Django 1.7 testing — version used in the demo application
Django 2.0 testing — latest stable version
Effective Django — testing covered already in the second chapter of the book
Effective Django factory_boy
Django testing — excellent PyCon talk, slides covering pytest, fixtures vs factories, etc

Conclusion

You should have a good idea about testing differences via unittest and pytest in the Django environment. The emphasis has been put on pytest (django-pytest) and some recommended approaches. The demo application django-testing brings functional test cases demonstrating the behaviour and features discussed. The articles and talks listed in this post were extremely helpful and instrumental in gaining expertise in the area and introducing rigorous testing approach into the production application.

Any discrepancy between the behaviour described above and on your own setup may originate from different software versions. In any case, if anything is not clear enough, please let me know in the comments.

Web Development, Big Data and DevOps—OSI Days 2014, India

2015-01-12T00:00:00+00:00

This is the second part of an article about the conference Open Source India, 2014 was held at Bengaluru, India. The first part is available here. The second day of the conference started with the same excitement level. I plan to attend talks covering Web, Big Data, Logs monitoring and Docker.

Web Personalisation

Jacob Singh started the first talk session with a wonderful presentation along with real-world cases which explained the importance of personalisation in the web. It extended to content personalisation for users and A/B testing (comparing two versions of a webpage to see which one performs better). The demo used the Acquia Lift personalisation module for the Drupal CMS which is developed by his team.

MEAN Stack

Sateesh Kavuri of Yodlee spoke about the MEAN stack which is a web development stack equivalent to popular LAMP stack. MEAN provides a flexible compatibility to web and mobile applications. He explained the architecture of MEAN stack.

He also provided an overview of each component involved in MEAN Stack.

MongoDB — NoSQL database with dynamic schema, in-built aggregation, mapreduce, JSON style document, auto-sharding, extensive query mechanism and high availability.

ExpressJS — A node.js framework to provide features to web and mobile applications.

AngularJS — seamless bi-directional model with extensive features like services and directives.

Node.js — A server side JavaScript framework with event based programming and single threaded (non blocking I/O with help of request queue).

Sails.js — MEAN Stack provisioner to develop applications quickly.

Finally he demonstrated a MEAN Stack demo application provisioned with help of Sails.js.

Moving fast with high performance Hack and PHP

Dushyant Min spoke about the way Facebook optimised the PHP code base to deliver better performance when they supposed to handle a massive growth of users. Earlier there were compilers HipHop for PHP (HPHPc) or HPHPi (developer mode) to convert the php code to C++ binary and executed to provide the response. After sometime, Facebook developed a new compilation engine called HipHop Virtual Machine which uses Just-In-Time (JIT) compilation approach and converts the code to HipHop ByteCode (HHBC). Both Facebook’s production and development environment code runs over HHVM.

Facebook also created a new language called Hack which is very similar to PHP which added static typing and many other new features. The main reason for Hack is to get the fastest development cycle to add new features and release frequent versions. Hack also uses the HHVM engine.

HHVM engine supports both PHP and Hack, also it provides better performance compare to Zend engine. So Zend Engine can be replaced with HHVM without any issues in the existing PHP applications to get much better performance. It is simple as below:

Also PHP code can be migrated to Hack by changing the <?php tag to <?hh and there are some converters (hackficator) available for code migration. Both PHP and Hack provide almost the same performance on the HHVM engine, but Hack has some additional developer-focussed features.

Application Monitoring and Log Management

Abhishek Dwivedi spoke about a stack to process the logs with various formats, myriad timestamp and no context. He explains a stack of tools to process the logs, store and visualize in a elegant way.

ELK Stack = Elasticsearch, LogStash, Kibana. The architecture and the data flow of ELK stack is stated below:

Elasticsearch — Open source full text search and analytics engine

Log Stash — Open source tool for managing events and logs which has following steps to process the logs

Kibana — seamlessly works with Elasticsearch and provides elegant user interface with various types of graphs

Apache Spark

Prajod and Namitha presented the overview of Apache Spark which is a real time data processing system. It can work on top of Hadoop Distributed FileSystem (HDFS). Apache Spark performs 100x faster in memory and 10x faster in disk compare to Hadoop. It fits with Streaming and Interactive scale of Big Data processing.

Apache Spark has certain features in processing the data to deliver the promising performance:

Multistep Directed Acyclic Graph
Cached Intermediate Data
Resilient Distributed Data
Spark Streaming — Adjust batch time to get the near real time data process
Implementation of Lambda architecture
Graphx and Mlib libraries play an important role

Online Data Processing in Twitter

Lohit Vijayarenu from Twitter spoke about the technologies used at Twitter and their contributions to Open Source. Also he explained the higher level architecture and technologies used in the Twitter microblogging social media platform.

The Twitter front end is the main data input for the system. The Facebook-developed Scribe log servers gather the data from the Twitter front end application and transfers the data to both batch and real time Big Data processing systems. Storm is a real time data processing system which takes care of the happening events at the site. Hadoop is a batch processing system which runs over historical data and generates result data to perform analysis. Several high level abstraction tools like PIG are used write the MR jobs. Along with these frameworks and tools at the high level architecture, there are plenty of Open Source tools used in Twitter. Lohit also strongly mentioned that in addition to using Open Source tools, Twitter contributes back to Open Source.

Docker

Neependra Khare from Red Hat given a talk and demo on Docker which was very interactive session. The gist of Docker is to build, ship and run any application anywhere. It provides good performance and resource utilization compared to the traditional VM model. It uses the Linux core feature called containerization. The container storage is ephemeral, so the important data can be stored in persistent external storage volumes. Slides can be found here.

Ecommerce in the Django World

2014-12-03T00:00:00+00:00

Mezzanine (http://mezzanine.jupo.org/) is a powerful piece of software written for the Django Framework in Python that functions like a structured content management system similar to Drupal, WordPress, and others. Built on top of this CMS structure is a module which adds features for Ecommerce which is known as Cartridge (http://cartridge.jupo.org/).

Installing Cartridge/Mezzanine

Installing the Mezzanine CMS system from scratch can be accomplished in two methods, either by using the pip python package manager, or you can clone the current source from the git repository from the software maintainers. If you were planning to modify either Mezzanine or Cartridge to customize the setup for your own needs the latter would likely be preferable as you could then easily begin creating custom branches off the original source to track the customization work. However for this example I will show the pip method of installation:

pip install -U cartridge

Once installed, there is a mezzanine-project command which will allow you to create a new blank Mezzanine environment within a new directory, and in this case we will also send an option to instruct the command to install the Cartridge module as well.

mezzanine-project -a cartridge new_cartridge_project

At this point you will have a blank Mezzanine environment with the cartridge module installed, now the Django database must be populated with the model information for the application, and then the Django application server will be started.

cd new_cartridge_project
python manage.py createdb
python manage.py runserver

If all went well, you should see the startup messages for the Django application server which will list the version numbers of the various libraries it will use, and then should be up and listening on the loopback interface. At this point to complete the setup you would just need to point your httpd at this loopback socket to send connections there.

Cartridge Product Models

Products within Cartridge are defined within the python models for the application, there are three primary models to be concerned with in a Product definition, the Product, ProductVariation and ProductImage models specifically.

Product — Defines the primary attributes for a product like name, price, SKU, and can be populated with optional fields for sale prices, etc.
ProductVariation — Defines a variant of a product SKU, these would be most commonly used for things like product sizes and colors.
ProductImage — Defines the image for the picture of each product.

Each of these three models exist within Django as their own data sets, but then they have foreign key references from each Product to all of its Variants and Images. While you can alter these model definitions using the django shell and other methods within python, you can also update and manage the products within the admin interface for the django site.

Adding Products / Tracking Orders

Adding products through the admin interface for Django was relatively easy and quick:

Within this interface you would provide the details for the products name, published status, date range for the product to be on the site, and product description in addition to any Variants or Images for the product. This is where the Mezzanine foundation of Cartridge start to show through, where this process mirrors the creating of a content page within Mezzanine, but has added these attributes of the product to the definition.

In the same way, Product Variants, Discount Codes, and Sales can be created in much the same way within the admin interface. In this way, once you had a basic Cartridge setup in place on your server, within most use cases for setting up a simple web store, Cartridge would remove the need to do any further hacking of python code, and would allow any user familiar with a CMS workflow for creating and managing the objects within the environment to manage the store.

In addition, Cartridge provides order management within the admin interface as well:

From this interface you can update orders, create new orders, print the PDF invoices for the orders and track whether orders have been fulfilled and shipped to the customer.

Conclusions

Like other Ecommerce applications built on top of a Content Management System, Cartridge and Mezzanine work well together to create a stack that allows for both ease of getting started with the system, and a massive degree of customization possible. Each component of Cartridge is defined within Django, and can be rewritten and customized to serve any number of use cases. The setup process for Cartridge was pretty straightforward, but one recommendation I can make is that once you have a working system up, you should place all of the files for Mezzanine and Cartridge into a source code management system like Git. Like other web applications of this type, the complexity of each change to the system can create a possible point of failure. Once you start making changes to the Mezzanine configuration files and the python for the Models and Views for Cartridge, controlling those changes becomes critical. In this way Mezzanine and Cartridge are spanning the normal continuum of software with the polar extremes of ease of use and customization by providing a good example to start with in the setup, and also exposing the full range of customization to the developer.

Drupal Commerce for Fun and Profit

2014-05-06T00:00:00+00:00

Drupal has for years been known as open source content management system (CMS) for creating easy content-driven websites for any topic. In addition to the CMS, there are a whole host of tools built upon this foundation to extend the features using modules. One module in particular I worked with recently for a client was Drupal Commerce, which helps to augment the CMS with an eCommerce platform.

Within Drupal for normal content items, you would create distinct articles, which have a predefined model of fields that would reflect the different aspects of an article, author, title, tags, etc. While Drupal Commerce uses these same functions, it will also give you the capability to also define SKUs, product categories, and rules and procedures to be carried when adding a particular item to your cart. In much the same way as articles are published to the Drupal taxonomy to allow users to browse through the articles, product categories, and individual products are published, and allow the administrator to customize the layout of how those products would display.

Getting Started with Drupal Commerce

One tool that is extremely helpful on getting a Drupal Commerce setup up for yourself is using the Drupal Commerce Kickstart.

This tool, which is maintained by a group of Drupal consultants/enthusiasts who created an embedded install of Drupal with some modules installed and configured to get both a working Drupal install and a copy of the Commerce plugin as well. The installation is quite simple, you extract the install archive into your web server document root, and then you follow an install procedure to configure your database, the name of the site, and any other information needed.

Once the install is complete, you will have a copy of Drupal Commerce installed, and will have a basic product display model on the front page with a few example products available. From here the next steps would be to define your own products and categories and input those model objects.

Creating Products

Internally to Drupal, when referring to articles, there are distinct units for each article and category known as nodes, and a hierarchical taxonomy that is defined by the administrator for how these items fit together. Within Drupal Commerce there is a similar designation, but the items are known as ‘entities’ and could refer to a whole host of possible objects in Drupal Commerce. Here is a chart to show the relationships:

These “entities” extend the original concept of a Drupal node to allow for different products, attributes of products and customers and their attributes. The product reference is a pointer object that allows a product to be added to a node article to be displayed on the front end of the site. This reference refers back to the products that were added, and if those product objects are updated, the content to display those products will stay static, but will be dynamically updated. In this way, the content templates for display products is abstracted from the actual product SKUs themselves.

When defining each product there are a number of relationships that are created which relate to the possible categories that a product would live in, as well as fields and properties implied by that product. Here is a visual representation of these relationships:

Every product entity would by definition be part of the “Product” bundle, and also would fall into a custom bundle for the “T-shirt” product category. In this way, you can create as complex or as simple of relationships between products and categories to reflect the actual state of the business using the tool, and will allow for the product definitions to remain in place, even if you were to redefine the layout for those products to display.

Conclusions

Drupal Commerce is a hulking tool, with a number of learning curves and a bit of terminology to get straight, but once in place and working will give a user a very robust commerce platform. Any Drupal developer familiar with creating custom taxonomies for article display and categorization will be able to use the same skills to create a slick eCommerce site.

Ecommerce Solutions: What are the Options?

2011-02-28T00:00:00+00:00

Lately, I’ve been evaluating ecommerce options for use on a side hobby/business. I’m obviously a developer, so in theory I could use one of End Point’s supported ecommerce frameworks or just write my own framework. But, my bottom line is that I don’t need the feature set offered by some of the ecommerce options out there and I don’t necessarily have the resources to develop a custom solution.

In addition to personal interest, End Pointers constantly encounter potential clients who aim to get a better understanding of the cost of using open source, our preferred ecommerce solution. I put together two infographics on ecommerce options, ongoing cost, feature sets, and the ability to customize. Before anyone flips out about the infographics, note that they represent my broad generalizations regarding the ongoing cost, feature sets and ability to customize. I’m intimately familiar with some of these options and less familiar with a couple of them.

Feature Set versus Ongoing Cost of Ecommerce Solutions

Ability to Customize versus Ongoing Cost of Ecommerce Solutions

Some notes on on the ecommerce solutions shown in the infographics:

Online payment service (Paypal): An online payment collection service like PayPal offers a minimal “ecommerce” feature set and might be suitable for someone looking to simply collect money. It also provides almost no ability to customize. The ongoing cost of PayPal is lower than many of the other options, where a percentage of each sale goes to PayPal.
Online catalog service (Etsy, eBay): An online catalog service such as Etsy or eBay offers very basic ecommerce listing features, little to no ability to customize, but has relatively low ongoing cost. For example, Etsy charges $0.20 to list a single item for four months and takes 3.5% of the sales fee.
Hosted ecommerce (shopify, Big Cartel, Big Commerce, Yahoo Merchant): A hosted ecommerce solution offers the ability to customize the appearance, typically in the form of a custom template language and has a basic ecommerce feature set. Ongoing costs would include the cost of the service, domain registration cost, and payment gateway (e.g. Authorize.NET) fees. Additional cost may apply if development services are required for building a custom template. Shopify’s basic solution costs $29/mo. with a max of 100 skus. Shopify offers several other higher-priced options which include more features and have less limitations. Most other hosted ecommerce solutions are similarly priced.
Open source ecommerce (Interchange, Spree, Magento, Zen Cart, osCommerce, prestaShop): An open-source ecommerce solution tends to have generic ecommerce features, provides the opportunity for a large amount of customization, but is typically more expensive than hosted ecommerce solutions and online catalog services. The software itself is free, but ongoing costs of hosting (server, domain registration, SSL certificate), development (any piece of customization that does not fit into the generic mold), and payment gateway fees apply. One positive about open source ecommerce is that additional plugins or add-ons are produced by members of the community. If the business needs are satisfied by the community-available extensions, cost from additional development or customization may be eliminated or reduced. I’d also probably group in existing open source plugins or modules for open source CMS solutions like WordPress and Drupal into this category: the generic solution is free, but additional resources may be required for customization.
Enterprise ecommerce (ATG, Magento Enterprise): From my experience, I’ve observed that enterprise ecommerce solutions tend to offer a large feature set and a similar ability to customize as open source frameworks. The ongoing cost is high: for example, Magento Enterprise starts at $12,990/year. In addition to the licensing cost, hosting, development, and payment gateway fees may apply.
Custom ecommerce (homegrown): The cost of writing your own ecommerce framework depends on the functionality requirements. The ability to customize is unlimited since the entire solution is custom. The feature set is likely to be proportional to resources spent on the project. Ongoing costs here include hosting, development and payment gateway fees.

Which one do you choose?

It has been my experience that most of End Point’s ecommerce clients need some type of customization: a custom appearance, discount functionality, shipping integration, payment gateway integration, social media integration, and other third-party integration. Choosing an option that allows easy customization tends to benefit our customers in the long run. We’re pretty biased at End Point towards open source ecommerce solutions, but my opinion recently is that with the advancement of web frameworks and web framework tools (e.g. gems in ruby), development of custom solutions can be done efficiently and may be a better option for a site that is outside the realm of standard ecommerce. For businesses not able to pay development or consulting costs, hosted ecommerce solutions are affordable and provide the essentials needed as the business grows.

Spree vs. Magento: Feature List Revisited

2010-07-13T00:00:00+00:00

A little over a month ago, I wrote an article on Spree vs. Magento Features. Recently, a client asked me to describe the features mentioned in that article. I thought this was another great opportunity to expand on my response to the client. So, here I am, revisiting ecommerce features in Spree and Magento. The original article can be referenced to compare availability of these features in Spree and Magento.

Features on a Single Product or Group of Product

Product reviews and/or ratings: functionality to allow customers to review and rate products. See a Backcountry.com product page for an example.
Product QnA: functionality allow customers to ask and answer questions on products. See a Backcountry.com product page for an example.
Product SEO (URL, title, meta data control): functionality to allow site administrators to manage product URLs, product page titles, and product meta data.
Advanced/flexible taxonomy: functionality to build a custom taxonomy / navigation structure for product browsing. For example, build multiple categories and subcategories with complex hierarchy. The taxonomy at Spree’s demo includes two categories of brand and category and subcategories in each.
SEO for taxonomy pages: functionality to allow site administrators to manage taxonomy URLs, taxonomy page titles, and taxonomy meta data.
Configurable product search: functionality to allow the developers and site administrators to adjust parameters used in search, such as products to show per page, and to show products with no on hand stock.
Bundled products for discount: functionality to allow site administrators to create bundles or group products together and then apply a discount to the entire bundle. For example, product X, Y and Z purchased together will yield a $10.00 discount.
Recently viewed products: functionality to show customers products they recently visited. This can be displayed on other product pages or the navigation pages to aid in navigation back to those products if the user would like to revisit the products. See the image below for an example of this functionality in action at Paper Source.

Soft product support/downloads: functionality to sell soft products such as mp3 or pdf files.
Product comparison: functionality to allow customers to compare multiple products, such as a comparison of price or technical features. See Backcountry.com for an example.
Upsell: functionality to encourage the customer to purchase a similar product from a higher price point, or to purchase an add on, may or may not include the functionality to allow site administrators to manage the upsell products.
Cross sell: functionality to encourage the customer to purchase related items, may or may not include the functionality to allow site administrators to manage the cross sell products. See the image below for an example of this functionality at Backcountry.com after adding an item to the cart.

Related items: functionality to display related items on product pages, may or may not include the functionality to allow site administrators to manage the related products.
RSS feed of products: functionality to produce a RSS feed with product detail releases. steepandcheap.com offers a RSS feed, however, they are unique in that they offer a one deal at a time business model, so the RSS feed contains a stream of products for sale. This may or may not include the functionality to allow site administrators to manage the RSS feed contents.
Multiple images per product: functionality to allow site administrators to upload multiple products per image. See a Spree demo product for an example.
Product option selection (variants): functionality to allow site administrators to create and manage variants for products to offer multiple variants per product, such as variants by size and color. See a Spree demo product for an example.
Wish list: functionality to allow customers to create product wish lists. See Amazon.com for a description of their wish list functionality.
Send product email to friend: functionality to allow customers to send emails to their friend to visit a specific product.
Product tagging / search by tagging: functionality to allow site administrators to assign tags to products for navigation or searching. See CityPass’s blog for an example of tag use in a content management system; in ecommerce context, the tags would navigate to a set of products instead of a set of blog articles.
Breadcrumbs: functionality that renders the product navigation hierarchy on navigation and product pages to allow customers to navigate to previous pages visited. See a Spree demo navigation page for an example.

CMS Features

Blogging functionality: functionality to allow site administrators to create, manage and display blog articles.
Static page management: functionality to allow site administrators to create, manage, and display static pages such as “About Us”, “Information”.
Media management: functionality to allow site administrators to create, manage, and display media such as images, video, audio. See the image below for an example of WordPress’s dashboard for content management.

Contact us form: functionality to allow customers to submit a request for contact. See End Point’s contact page for an example.
Polls: functionality to allow site administrators to create and manage basic polls, functionality to allow customers to submit answers to basic polls.

Checkout Support

One page checkout: functionality to allow customers to complete checkout on one page, rather than move forward through checkout through multiple address, payment pages. See Paper Source or Backcountry.com’s checkout processes for examples.
Guest checkout: functionality to checkout without creating a user account. Checkout at the Spree demo without being logged in for an example.
SSL support: functionality to configure use of SSL during checkout. In Spree’s case, the site administrator may turn SSL off during development and on during production.
Discounts: functionality to allow customers to apply discount coupons to orders for a percentage or dollar amount reduction.
Gift certificates: functionality to allow customers to purchase and use gift certificates as credit for purchases.
Saved shopping cart: the functionality to save the products in a customers shopping cart so their shopping cart will be pre-populated on their next visit.
Saved addresses: functionality to allow customers to create and manage addresses to be selected during checkout for billing or shipping rather than requiring the customer to re-enter their address. See the image below for an example of using saved addresses during Backcountry.com’s checkout.

Shipping Support

Real time rate lookup (UPS, USPS, FedEx): the functionality to request rates from UPS, USPS, or FedEx during checkout for more accurate rate pricing rather than using a flat shipping rate.
Order tracking: functionality to allow the site administrators to enter order tracking information and allow the user to review that tracking information, may or may not include sharing this information in a “Your order has been shipped” email.
Multiple shipments per order: functionality to allow site administrators to split orders into multiple packages if specific products can not be shipped at the same time or can not be shipped together. See the image below for an example of Spree’s backend shipping interface.

Complex rate lookup: functionality to calculate ship rates based on weight or price.
Free shipping: functionality to offer free shipping.

Payment Support

Multiple payment gateways: integration of multiple payment gateways such as Authorize.NET, Beanstream, Paypal, SagePay, etc.
Authorize.Net: integration of the Authorize.Net payment gateway; may or may include the use of profiles (Authorize.Net CIM).
Authorize and capture versus authorize only: functionality to allow site administrators to configure whether or not credit cards should be authorized only during checkout completion or authorized and captured. If the credit card is authorized only, site administrators may finalize an order by capturing on the backend interface.
Google Checkout: integration of Google Checkout.
Paypal Express: integration of Paypal Express.

Admin Features

Sales reporting: the functionality to display sales statistics, such as profits on sales or year-over-year sales.
Sales management tools: functionality to allow site administrators to create and manage product sales. For example, the site administrator might create a 50% off sale to cover 25% of the products to begin in a week and end in two weeks.
Inventory management: functionality to allow the site administrator to manage individual inventory units and their current state (on_hand, shipped, backordered) and order assignment.
Purchase order management: functionality to allow the site administrator to create and manage purchase orders. See the image below for an example of a potential backend interface for purchase orders in Spree.

Multi-tier pricing for quantity discounts: functionality to allow customers to buy large quantities of products at a discount, the functionality to allow the site administrator to manage the large quantity product discounts.
Landing page tool: functionality to create custom landing pages that may include targeted content or products, typically used for advertising or marketing.
Batch import and export of products: functionality to allow the site administrator to import and export products via admin interface or script rather than entering each product individually.
Multiple sales reports: See “Sales reporting” above.
Order fulfillment: functionality to allow site administrators to manage fulfillment (inventory selection, shipping) of orders.
Tax Rate Management: functionality to manage tax rates per zone, where zones are defined by states and/or countries. Note that in Spree, zones can only be defined by a combination of states and countries and tax rates can be tied to one or more zones.

User Account Features

User addresses: See “Saved addresses” above.
Feature rich user preferences: integration of various user account tools, such as address management, profile management, order review, etc.
Order tracking history: functionality to allow a customer to lookup their order history, may or may not include order tracking information.

Site Wide Features

Extensibility: functionality to extend the ecommerce core with modular components.
Appearance Theming: functionality to change the appearance of the site.
Ability to customize appearance at category or browsing level: functionality to create and manage custom and varied appearances for product browsing pages. For example, the categories “Jackets” and “Pants” may have different appearances, motivated by marketing or advertising.
Localization: the functionality to translate the ecommerce site to a different language. See the image below for a small example of localization in action in Spree.

Multi-store, single admin support: functionality to manage multiple stores from a single administrative location. An example of this might include http://store1.endpoint.com/, and http://store2.endpoint.com/, where both can be managed at http://admin.endpoint.com/.
Support for multiple currencies: functionality to translate product prices between currencies.
Web service API: functionality to retrieve data from the ecommerce application for third party use. See Spree’s documentation on the Spree API.
System wide SEO: general site-wide SEO functionality including features such as sitemap, googlebase integration, URL management, page title management.
Google Analytics: functionality to allow site administrators to create and manage Google Analytics Ids, functionality to track traffic and conversion on the frontend.
Active community: an active developer community with frequent core and extension contributions.

Most of the features described above are well known to ecommerce developers, but this list might also serve as a good checklist to review with a potential client during the estimate process to make sure expectations of an ecommerce platform are managed, especially with a young platform such as Spree where some features are not yet included in the core.

Spree vs Magento: A Feature List Comparison

2010-06-07T00:00:00+00:00

Note: This article was written in June of 2010. Since then, there have been several updates to Spree. Check out the current Official Spree Extensions or review a list of all the Spree Extensions.

This week, a client asked me for a list of Spree features both in the core and in available extensions. I decided that this might be a good time to look through Spree and provide a comprehensive look at features included in Spree core and extensions and use Magento as a basis for comparison. I’ve divided these features into meaningful broader groups that will hopefully ease the pain of comprehending an extremely long list :) Note that the Magento feature list is based on their documentation. Also note that the Spree features listed here are based on recent 0.10.* releases of Spree.

Features on a Single Product or Group of Product

Feature	Spree	Magento
Product reviews and/or ratings	Y, extension	Y
Product qna	N	N
Product seo (url, title, meta data control)	N	Y
Advanced/flexible taxonomy	Y, core	Y
Seo for taxonomy pages	N	Y
Configurable product search	Y, core	Y
Bundled products for discount	Y, extension	Y
Recently viewed products	Y, extension	Y
Soft product support/downloads	Y, extension	Y, I think so
Product comparison	Y, extension	Y
Upsell	N	Y
Cross sell	N	Y
Related items	Y, extension	Y
RSS feed of products	N	Y
Multiple images per product	Y, core	Y
Product option selection (variants)	Y, core	Y
Wishlist	Y, extension	Y
Send product email to friend	Y, extension	Y
Product tagging / search by tagging	N	Y
Breadcrumbs	Y, core	Y

CMS Features

Features	Spree	Magento
Blogging functionality	Y, extension	Y *extension
Static page management	Y, extension	Y
Media management	N	Y
Contact us form	Y, extension	Y
Polls	Y, extension	Y

Checkout Support

Feature	Spree	Magento
One page checkout	N	Y
Guest checkout	Y, core	Y
SSL Support	Y, core	Y
Discounts	Y, core	Y
Gift Certificates	N	Y
Saved Shopping Cart	N	Y
Saved Addresses	Y, extension	Y

Shipping Support

Feature	Spree	Magento
Real time rate lookup (UPS, USPS, Fedex)	Y, extension	Y
Order tracking	N	Y
Multiple shipments per order	Y, core	Y
Complex rate lookup	Y, extension	Y
Free shipping	Y, extension	Y

Payment Support

Feature	Spree	Magento
Multiple Payment Gateways	Y, core	Y
Authorize.net	Y, core	Y
Authorize and capture versus authorize only	Y, core	Y
Google Checkout	Y, extension	Y
Paypal Express	Y, extension	Y

Admin Features

Feature	Spree	Magento
Sales reporting	Y, core	Y
Sales Management Tools	N	Y
Inventory management	Y, core	Y
Purchase order management	N	Y
Multi-tier pricing for quantity discounts	N	Y
Landing page tool	Y, extension	Y
Batch import and export of products	Y, extension	Y
Multiple Sales reports	Y, core	Y
Order fulfillment	Y, core	Y
Tax Rate Management	Y, core	Y

User Account Features

Feature	Spree	Magento
User addresses	Y, extension	Y
Feature rich user preferences	N	Y
Order tracking history	Y, core	Y

System Wide Features

Feature	Spree	Magento
Extensibility	Y, core	Y
Appearance Theming	Y, core	Y
Ability to customize appearance at category or browsing level	N	Y
Localization	Y, core	Y
Multi-store, single admin support	Y, extension	Y
Support for multiple currencies	N	Y
Web Service API	Y, core	Y
SEO System wide: sitemap, google base, etc	Y, extension	Y
Google Analytics	Y, core	Y
Active community	Y, N/A	Y

The configurability and complexity of each feature listed above varies. Just because a feature is provided within a platform does not guarantee that it will meet the desired business needs. Magento serves as a more comprehensive ecommerce platform out of the box, but the disadvantage may be that adding custom functionality may require more resources (read: more expensive). Spree serves as a simpler base that may encourage quicker (read: cheaper) customization development simply because it’s in Rails and because the dynamic nature of Ruby allows for elegant extensibility in Spree, but a disadvantage to Spree could be that a site with a large amount of customization may not be able to take advantage of community-available extensions because they may not all play nice together.

Rather than focus on the platform features, the success of the development depends on the developer and his/her skillset. Most developers will say that any of the features listed above are doable in Magento, Spree, or Interchange (a Perl-based ecommerce platform that End Point supports) with an unlimited budget, but a developer needs to have an understanding of the platform to design a solution that is easily understood and well organized (to encourage readability and understandability by other developers), develop with standard principles like DRY and MVC-style separation of concerns, and elegantly abstract from the ecommerce core to encourage maintainability. And of course, be able to understand the business needs and priorities to guide a project to success within the given budget. Inevitably, another developer will come along and need to understand the code and inevitably, the business will often use an ecommerce platform longer than planned so maintainability is important.

Please feel free to comment on any errors in the feature list. I’ll be happy to correct any mistakes. Now, off to rest before RailsConf!

New End Point site launched: Rails, jQuery, Flot, blog feed

2009-10-07T00:00:00+00:00

This week we launched a new website for End Point. Not only did the site get a facelift, but the backend content management system was entirely redesigned.

Goodbye Old Site:

Hello New Site:

Our old site was a Rails app with a Postgres database running on Apache and Passenger. It used a custom CMS to manage dynamic content for the bio, articles, and service pages. The old site was essentially JavaScript-less, with the exception of Google Analytics.

Although the new site is still a Rails application, it no longer uses the Postgres database. As developers, we found that it is more efficient to use Git as our “CMS” rather than developing and maintaining a custom CMS to meet our ever-changing needs. We also trimmed down the content significantly, which further justified the design; the entire site and content is now comprised of Rails views and partial views. Also included in the new site is cross browser functioning jQuery and flot. Some of the interesting implementation challenges are discussed below.

jQuery Flot Integration

The first interesting JavaScript component I worked on was using flot to improve interactivity to the site and to decrease the excessive text that End Pointers are known for (for example, this article). Flot is a jQuery data plotting tool that contains functionality for plot zooming, data interactivity, and various configuration display settings (see more flot examples). I’ve used flot before in several experiments but had yet to use it on a live site. For the implementation, we chose to plot our consultant locations over a map of the US to present our locations in an interactive and fun to use way. The tedious part of this implementation was actually creating the datapoints to align with cities. Check out the images below for examples.

Flot has built in functionality for on hover events. When a point on the plot is hovered over, correlating employees are highlighted using jQuery and their information is presented to the right of the map.

When a bio picture is hovered over, the correlating location is highlighted using jQuery and flot data point highlighting.

We also implemented a timeline using flot to map End Point’s history. Check out the images below.

When a point on the plot is hovered over, the history details are revealed in the section below.

The triangle image CSS position is adjusted when a point on the plot is activated.

Dynamic Rails Partial Generation

One component of the old site that was generated dynamically sans-CMS was blog article integration into the site. A cron job ran daily to import new blog article title, link, and content snippets into the Postgres database. We opted for removing dependency on a database with the new site, so we investigated creative ways to include the dynamic blog content. We developed a rake task that is run via cron job to dynamically generate partial Rails views containing blog content. Below is an example and explanation of how the blog RSS feed is retrieved and a partial is generated:

Open URI and REXML are used to retrieve and parse the XML feed.

require 'open-uri'
require 'rexml/document'
...

The feed is retrieved and a REXML object created from the feed in the rake task:

data = open('https://www.endpointdev.com/blog/feed.xml', 'User-Agent' => 'Ruby-Wget').read
doc = REXML::Document.new(data)

The REXML object is iterated through. An array containing the blog links and titles is created.

results = []
doc.root.each_element('//item') do |item|
  author = item.elements['author'].text.match(/\(.+/).to_s.gsub(/\.|\(|\)/,'')
  results << '<a href="' + item.elements['link'].text + '">' + item.elements['title'].text + '</a>'
end

Finally, a Rails dynamic partial is written containing the contents of the results array:

  File.open(#{RAILS_ROOT}/app/views/blog/_index.rhtml", 'w') { |f| f.write(results.inject('') { |s, v| s = s + '<p>' + v  + '</p>'}) }

A similar process was applied for bio and tag dynamic partials. The partials are included on pages such as the End Point service pages, End Point bio pages, and End Point home page.

jQuery Carousel Functionality

Another interesting JavaScript component I worked on for the new site was the carousel functionality for the home page and client page. Carousels are a common “web 2.0” JavaScript component where visible items slide one direction out of view and new items slide into view from the other direction. I initially planned on implementing a simple carousel with a jQuery plugin, such as jCarousel. Other JavaScript frameworks also include carousel functionality such as the YUI Carousel Control or the Prototype UI. I went along planning to implement the existing jQuery carousel functionality, but then was asked, “Can you make it a circular carousel where the left and right buttons are always clickable?” In many of the existing carousel plugins and widgets, the carousel is not circular, so this request required custom jQuery. After much cross-browser debugging, I implemented the following (shown in images for a better explanation):

Step 1: The page loads with visible bios surrounded by empty divs with preset width. The visibility of the bios is determined by CSS use of the overflow, position, and left attributes.

Step 2: Upon right carousel button click, new bios populate the right div via jQuery.

Step 3: To produce the carousel or slider effect, the left div uses jQuery animation functionality and shrinks to a width of 0px.

Step 4: Upon completion of the animation, the empty left div is removed, and a new empty div is created to the right of the new visible bios.

Step 5: Finally, the left div’s contents are emptied and the carousel is in its default position ready for action!

Another request for functionality came from Jon. He asked that we create and use “web 2.0” URLs to load specific content on page load for the dynamic content throughout our site, such as www.endpoint.com/clients#citypass, www.endpoint.com/clients#backcountry.

Upon page load, JavaScript is used to detect if a relative link exists:

if(document.location.href.match('#.+')) {
    var id = document.location.href.match('#.*').toString().replace('#', '');
}

The id retrieved from the code snippet above is used to populate the dynamic page content. Then, JavaScript is used during dynamic page functionality, such as carousel navigation, to update the relative link:

document.location.href = document.location.href.split('#')[0] + '#' + anchor;

Twitter Integration

Another change in the new site was importing existing functionality previously written in Python to update End Point’s Twitter feed automagically. The rake task uses the Twitter4R gem to update the Twitter feed and is run via cron job every 30 minutes. See the explanation below:

The public twitter feed is retrieved using Open URI and REXML.

    data = open('https://twitter.com/statuses/user_timeline/endpoint.xml', 'User-Agent' => 'Ruby-Wget').read
    doc = REXML::Document.new(data)

An array containing all the titles of all tweets is created.

    doc.each_element('statuses/status/text') do |item|
      twitter << item.text.gsub(/ http:\/\/j\.mp.*/, '')
    end

The blog RSS feed is retrieved and parsed. An array of hashes is created to track the un-tweeted blog articles.

    data = open('https://www.endpointdev.com/blog/feed.xml', 'User-Agent' => 'Ruby-Wget').read
    doc = REXML::Document.new(data)
    found_recent = false
    doc.root.each_element('//item') do |item|
      found_recent = true if twitter.include?(item.elements['title'].text)
      blog << { 'title' => item.elements['title'].text, 'link' => item.elements['link'].text } if !found_recent
    end

Using the j.mp api, a short url is generated. A Twitter message is created from the short URL.

      data = open('http://api.j.mp/shorten?version=2.0.1&longUrl=' + blog.last['link'] + '&login=**&apiKey=*****&format=xml')
      ...
      twitter_msg = blog.last['title'] + ' ' + short_url

The twitter4r gem is used to login and update the Twitter status message.

      client = Twitter::Client.new(:login => **, :password => *****)
      begin
        status = client.status(:post, twitter_msg)
      rescue
      end

Google Event Tracking

Finally, since we implemented dynamic content throughout the site, we decided to use Google Event Tracking to track user interactivity. We followed the standard Google Analytics event tracking to add events for events such as the slider carousel user involvement, the team page bio and history hover user involvement:

//pageTracker._trackEvent(category, action, optional_label, optional_value);
pageTracker._trackEvent('Team Page Interaction', 'Map Hover', bio);

We are happy with the new site and we hope that it presents our skillz well!

Spree at RailsConf

2009-05-11T00:00:00+00:00

Last week at RailsConf 2009, the Spree folks from End Point conducted a Birds of a Feather session to discuss Spree, an End Point sponsored open source rails ecommerce platform. Below is some of the dialog from the discussion (paraphrased).

Crowd: “How difficult is it to get Spree up and running from start to finish?”

Spree Crew: “This depends on the level of customization. If a customer simple needs to reskin the site, this shouldn’t take more than a week (hopefully much less than a full week). If the customer needs specific functionality that is not included in core functionality or extensions, you may need to spend some time developing an extension.”

Crowd: “How difficult is it to develop extensions in Spree?”

Spree Crew: “Spree extension work is based on the work of the Radiant community. Extensions are mini-applications: they allow you to drop a pre-built application into spree to override or insert new functionality. Documentation for extensions is available at the spree github wiki. We also plan to release more extensive Spree Guides documentation based on Rails Guides soon.”

Spree Crew: “How did you hear about Spree?”

Crowd: “My client and I found it via search engines. My client thought that Spree looked like a good choice.”

Spree Crew: “What other platforms did you consider before you found spree?”

Crowd: “Magento”, “Substruct”, “My client considered Magento, but I know several people that have developed with Magento and have found it difficult to override core functionality.”

Spree Crew: “What types of functionality were missing from Spree that you’d like to see developed in the future?”

Crowd: “My client wanted checkout split into multiple steps instead of the new single page checkout. I was able to implement this by overriding the Spree checkout library and checkout views.”, “My client needed complex inventory management.”, “My client needed split shipping functionality.”

Crowd: “What is the plan for Spree with regards to CMS development?”

Spree Crew: “There has been some discussion on the integration of a CMS into Spree. No one in the Spree community appears to be currently working on this. Contributions in this area are welcome. Also, Yehuda Katz is giving a talk on mountable apps—the Spree community would like to investigate the implications this has for Spree.”

Crowd: “What are the next steps for localization, especially multilingual product descriptions?”

Spree Crew: “This is on the radar for future Spree development. It is not currently in development, and again, contributions in this area are welcome.”

From the discussion, I took away that some of the desired features for Spree are inventory management, split shipping functionality, cms integration, and improved localization. I hope that the application of Spree continues to contribute to it’s progress. The Spree Crew also hopes to showcase some of the sites referenced above at the spree site.

SEO Ecommerce

2009-04-20T00:00:00+00:00

I recently read an article that discusses Magento SEO problems and solutions. This got me to think about common search engine optimization issues that I’ve seen in e-commerce. Below are some highlighted e-commerce search engine optimization issues. The Spree Demo, Interchange Demo, and Magento Demo are used as references.

Duplicate Home Pages (www, non-www, index.html)

Duplicate home pages can come in the form of a homepage with www and without www, a homepage in the form of http://www.domain.com/ and a homepage with some variation of “index” appended to the url, or a combination of the two. In the Interchange demo, http://demo.icdevgroup.org/i/demo1 and http://demo.icdevgroup.org/i/demo1/index.html are duplicate, http://demo.spreecommerce.com/ and http://demo.spreecommerce.com/products/ in the Spree demo, and finally http://demo.magentocommerce.com/ and http://demo.magentocommerce.com/index.php in the Magento demo.

External links positively influence search engine performance more if they are pointing to one index page rather than being divided between two or three home pages. Since the homepage most likely receives the most external links, this issue can be more problematic than other generated duplicate content. I’ve also seen this happen in several content management systems.

This article provides directions on mod_rewrite use to apply a 301 redirect from the www.domain.com/index.php homepage to www.domain.com. This solution or other redirect solutions can be applied to Spree, Interchange, and other ecommerce platforms.

Irrelevant Product URLs

A search engine optimization best practice is to provide relevant and indicative text in the product urls. In the Interchange demo, the default catalog uses the product sku in the product url (http://demo.icdevgroup.org/i/demo1/os28073.html). In Magento and Spree, product permalinks with relevant text are used in the product url. In wordpress, the author has the ability to set permalinks for articles. I am unsure if Magento gives you the ability to customize product urls. Spree does not currently give you the ability to manage custom product permalinks. However, for all of these ecommerce platforms, these fixes may all be in the works since it is important for ecommerce platforms to implement search engine optimization best practices.

Duplicate product content

I’ve observed several situations where products divided into multiple taxonomies results in duplicate content creation via different user navigation paths. For example, in the Spree demo, the “Ruby Baseball Jersey” can be reached through the Ruby brand page, the Clothing page, or the homepage. The three generated duplicate content urls are http://demo.spreecommerce.com/products/ruby-on-rails-ringer-t-shirt, http://demo.spreecommerce.com/t/brands/ruby/p/ruby-baseball-jersey, and http://demo.spreecommerce.com/t/categories/clothing/shirts/p/ruby-baseball-jersey.

Another example of this can be found in the Interchange demo. The left navigation taxonomy tree provides links to any product url with “?open=X,Y,Z” appended to the url. The “open” query string indicates how the DHTML tree should be displayed. For example, the “Digger Hand Trencher” has a base url of http://demo.icdevgroup.org/i/demo1/os28076.html. Depending on which tree nodes are exploded, the product can be reached at http://demo.icdevgroup.org/i/demo1/os28076.html?open=0,11,13,19, http://demo.icdevgroup.org/i/demo1/os28076.html?open=0,11,13, etc. This standard demo functionality yields a lot of duplicate content.

In Magento, products are the in the form of www.domain.com/product-name, although the article I mentioned above mentions that www.domain.com/category/product.html product urls were generated. Perhaps this was a recent fix, or perhaps the demo is configured to avoid generating this type of duplicate content.

Duplicate product page content is often used to indicate which breadcrumb should display or to track user click-through behavior (for example, did a user click on a “featured product”? a “best seller”? a specific “product advertisement”?). In Interchange, session ids are appended to urls which is another source of duplicate content. Instead of using the url to track user navigation or behavior, several other solutions such as using cookies, using a ‘#’ (hash), or using session data can be used to avoid duplicate content generation.

Performance

Performance should not be overlooked in ecommerce for search engine optimization. In March of 2008, Google wrote about how landing page load time will be incorporated into the Quality Score for Google Adwords—which is also believed to apply to regular search results. And github recently released some data on how performance improvements influenced http://www.github.com/ Googlebot visits.

Keeping a high content to text ratio, consolidation, minification, and gzipping CSS and JavaScript, and minimizing the use of JavaScript-based suckerfish can all improve search engine performance.

The Interchange default catalog has a simple template with minimal css and javascript includes, so the developer is responsible for sticking to best performance practices. The Magento demo appears to have decent content to text ratio, but still requires 5 css files that should be consolidated and minified if they are included on every page. Finally, Spree has undergone some changes in the last month and is moving in the direction of including one consolidated javascript file plus any javascript required for extensions on every page, and the upcoming release of Spree 0.8.0 will have considerable frontend view improvements.

Ecommerce platforms should have decent performance—yslow or this book on high performance website essentials are good resources.

Lacking basic CMS management

Basic CMS management such as the ability to manage and update page titles and page meta data is something that has been overlooked by ecommerce platforms in the past, but appears to have been given more attention recently. An ecommerce solution should also have functionality to create and manage static pages.

The Interchange demo does not have meta description and keyword functionality, however, page titles are equal to product names which is an acceptable default. It’s also very simple to add a static content page (as a developer) and would require just a bit more effort to have this content managed by a database in Interchange. The Spree core is missing some basic CMS management such as page title and meta data management, but this functionality is currently in development. One Spree contributer developed a Spree extension that provides management of simple static pages using a WYSIWYG editor. At the moment, Magento appears to have the most traditional content management system functionality out of the box.

Another area to improve CMS within Ecommerce is to determine a solution to integrate a blog. A quick Google search of “magento add blog” revealed how to set up a wordpress blog in Magento with an extension. One of End Point’s clients, CCI Beauty, also has wordpress integrated into their Interchange setup. Finally, there has been discussion about the development of “Spradiant”, or mixing spree and radiant.

Another missed opportunity in ecommerce platforms is finding a solution to elegantly blend content and product listings to target specific keywords. A “landing page” can have a page title, meta data, and content targeted towards a specific terms. http://www.backcountry.com/store/gear/arcteryx-vests.html and http://www.backcountry.com/store/gear/cargo-pant.html are examples of targeted terms with corresponding products. Going one step farther, search pages themselves can have managed content to attract keywords, such as a page title, and meta data for specific high traffic keywords with the related products. For example, http://www.domain.com/s/ruby_shirt could be a search page for “Ruby Shirt” which contains meaningful content and relevant products.

Mishandled Product Pagination

Finding a search engine optimization solution for pagination can be a difficult problem in ecommerce. When there are less than 100 products for a site, this shouldn’t be an issue because a simple taxonomy can appropriately group the products with low crawl depth. A website with 10,000 products must balance between keeping a low taxonomy depth to minimize crawl depth and ensure that all products are listed and indexable.

For example, products may be divided and fit into three levels of navigation: category, subcategory, and group. If there are 10,000 products, divided into 10 categories, 10 subcategories per category, and 10 product groups per category, 10 products can be shown on each group per page with no pagination. However, product taxonomy is not always so ideal. In some groups there may be 2 products and in others there may be 30. Pagination, or pages with an offset of product listings are generated to accommodate these product listings (for example, http://www.backcountry.com/store/group/61/Sun-Hats-Rain-Hats-Safari-Hats.html, http://www.backcountry.com/store/group/61/Sun-Hats-Rain-Hats-Safari-Hats-p1.html).

A few problems can arise from the pagination solution. First, by web 2.0 standards, the content should be generated via ajax. An SEO friendly ajax solution must be implemented—where the onclick event refreshes the content, but the links are still crawlable via search engine bots. Second, page 1 with no product offset will have 1 level less of crawl depth, therefore it will receive the most link juice from it’s parent page (subcategory). As a result, there must be thoughtful analysis of which products to present on that page: should high traffic pages get the traffic? should popular items be listed on the first page? should low traffic products be listed to try to bump the traffic on those pages? should products with the most “user interaction” (reviews, qna, ratings) be shown on that page? Another problem that comes up is that the page meta data and title will most likely be very similar since the content is a list of similar products. These two pages can essentially be competing for traffic and may be counted as duplicate content if the page titles and meta data are equal.

Interchange uses the more list to handle pagination, but this functionality is not search engine friendly as it generates urls such as http://demo.icdevgroup.org/i/demo1/scan/MM=3ffffa066192cba677e1428d7461ddc9:10:19:10.html?mv_more_ip=1&mv_nextpage=results&mv_arg=, http://demo.icdevgroup.org/i/demo1/scan/MM=3ffffa066192cba677e1428d7461ddc9:20:27:10.html?mv_more_ip=1&mv_nextpage=results&mv_arg=, etc. The Spree demo had some pagination implementation, but upon recent frontend changes, it is no longer included in the demo. The Magento demo was carefully arranged so that product group pages have no more than 9 products to avoid showing any pagination functionality. However, when modifying the number of products displayed per group or using the “Sort By” mechanism, ?limit=Y and &order=X&dir=asc is appended to the url—which can produce a large volume of duplicate content (try filters on this page).

It is difficult to determine which of the above problems is the most problematic. From personal experience, I have been involved in tackling all duplicate content issues, and then moving on to “optimization” opportunities such as enhancing the content management system. At the very least, developers and users of any ecommerce platform should be aware of common search engine optimization issues.

Varnish, Radiant, etc.

2008-10-26T00:00:00+00:00

As my colleague Jon mentioned, the Presidential Youth Debates launched its full debate content this week. And, as Jon also mentioned, the mix of tools involved was fairly interesting:

Radiant, a simple Rails-based CMS
Varnish, a high-performance reverse proxy server
Amazon Simple Storage Service, a scalable storage layer
PostgreSQL, the truly marvelous open-source relational database management system that End Point loves so well

Our use of Postgres for this project was not particularly special, and is simply a reflection of our using Postgres by default. So I won’t discuss the Postgres usage further (though it pains me to ignore my favorite piece of the software stack).

Radiant

Dan Collis-Puro, who has done a fair amount of CMS-focused work throughout his career, was the initial engineer on this project and chose Radiant as the backbone of the site. He organized the content within Radiant, configured the Page Attachments extension for use with Amazon’s S3 (Simple Storage Service), and designed the organization of videos and thumbnails for easy administration through the standard Radiant admin interface. Furthermore, prior to the release of the debate videos, Dan built a question submission and moderation facility as a Radiant extension, through which users could submit questions that might ultimately get passed along to the candidates for the debate.

In the last few days prior to launch, it fell to me to get the new debate materials into production, and we had to reorganize the way we wanted to lay out the campaign videos and associated content. Because the initial implementation relied purely on conventions in how page parts and page attachments are used, accomplishing the reorganization was straightforward and easily achieved; it was not the sort of thing that required code tweaks and the like, managed purely through the CMS. It ended up being quite—dare I say it?—an agile solution. (Agility! Baked right in! Because it’s opinionated software! Where’s my Mac? It just works! Think Same.)

For managing small, simple, straightforward sites, Radiant has much to recommend it. For instance:

the hierarchical management of content/pages is quite effective and intuitive
a pretty rich set of extensions (such as page attachments)
the “filter” option on content is quite handy (switch between straight text, fckeditor, etc.) and helpful
the Radiant tag set for basic templating/logic is easy to use and understand
the general resources available for organizing content (pages, layouts, snippets) enables and readily encourages effective reuse of content and/or presentation logic

That said, there are a number of things for which one quickly longs within Radiant:

In-place editing user interface: an adminstrative mode of viewing the site in which editing tools would show in-place for the different components on a given page. This is not an uncommon approach to content management. The fact that you can view the site in one window/tab and the admin in another mitigates the pain of not having this feature to a healthy extent, but the ease of use undoubtedly suffers nevertheless.
Radiant offers different publishing “states” for any given page (“draft”, “published”, “hidden”, etc.), and only publicly displays pages in the “published” state in production. This is certainly helpful, but it is ultimately insufficient. This is no substitute for versioning of resources; there is no way to have a staging version of a given page, in which the staging version is exposed to administrative users only at the same URL as the published version. To get around this, one needs to make an entirely different page that will replace the published page when you’re ready. While it’s possible to work around the problem in this manner, it clutters up the set of resources in the CMS admin UI, and doesn’t fit well with the hierarchical nature of the system; the staging version of a page can’t have the same children as the published version of the page, so any staging involving more than one level of edits is problematic and awkward. That leaves quite a lot to be desired: any engineer who has ever done all development on a production site (no development sites) and moved to version-controlled systems knows full well that working purely against a live system is extremely painful. Content management is no different.
The page attachments extension, while quite handy in general, has configuration information (such as file size limits and the attachment_fu storage backend to use) hard-coded into its PageAttachment model class definition, rather than abstracting that configuration information into YAML files. Furthermore, it’s all or nothing: you can only use one storage backend, apparently, rather than having the flexibility of choosing different storage backends by the content type of the file attached, or choosing manually when uploading the file, etc. The result in our case is that all page attachments go to Amazon S3, even though videos were the only thing we really wanted to have in S3 (bandwidth on our server is not a concern for simple images and the like).

The in-place editing UI features could presumably be added to Radiant given a reasonable degree of patience. The page attachment criticisms also seem achievable. The versioning, however, is a more fundamental issue. Many CMSes attempt to solve this problem many different ways, and ultimately things tend to get unpleasant. I tend to think that CMSes would do well to learn from version control systems like Git in their design; beyond that, integrate with Git: dump the content down to some intelligent serialized format and integrate with git branching, checkin, checkout, pushing, etc. That dandy, glorious future is not easily realized.

To be clear: Radiant is a very useful, effective, straightforward tool; I would be remiss not to emphasize that the things it does well are more important than the areas that need improvement. As is the case with most software, it could be better. I’d happily use/recommend it for most content management cases I’ve encountered.

Amazon S3

I knew it was only a matter of time before I got to play with Amazon S3. Having read about it, I felt like I pretty much knew what to expect. And the expectations were largely correct: it’s been mostly reliable, fairly straightforward, and its cost-effectiveness will have to be determined over time. A few things did take me by surprise, though:

The documentation on certain aspects, particularly the logging is, fairly uninspiring. It could be a lot worse. It could also be a lot better. Given that people pay for this service, I would expect it to be documented extremely well. Of course, given the kind of documentation Microsoft routinely spits out, this expectation clearly lacks any grounding in reality.
Given that the storage must be distributed under the hood, making usage information aggregation somewhat complicated, it’s nevertheless disappointing that Amazon doesn’t give any interface for capping usage for a given bucket. It’s easy to appreciate that Amazon wouldn’t want to be on the hook over usage caps when the usage data comes in from multiple geographically-scattered servers, presumably without any guarantee of serialization in time. Nevertheless, it’s a totally lame problem. I have reason to believe that Amazon plans to address this soon, for which I can only applaud them.

So, yeah, Amazon S3 has worked fine and been fine and generally not offended me overmuch.

Varnish

The Presidential Youth Debate project had a number of high-profile sponsors potentially capable of generating significant usage spikes. Given the simplicity of the public-facing portion of the site (read-only content, no forms to submit), scaling out with a caching reverse proxy server was a great option. Fortunately, Varnish makes it pretty easy; basic Varnish configuration is simple, and putting it in place took relatively little time.

Why go with Varnish? It’s designed from the ground up to be fast and scalable (check out the architecture notes for an interesting technical read). The time-based caching of resources is a nice approach in this case; we can have the cached representations live for a couple of minutes, which effectively takes the load off of Apache/Rails (we’re running Rails with Phusion Passenger) while refreshing frequently enough for little CMS-driven tweaks to percolate up in a timely fashion. Furthermore, it’s not a custom caching design, instead relying upon the fundamentals of caching in HTTP itself. Varnish, with its Varnish Configuration Language (VCL), is extremely flexible and configurable, allowing us to easily do things like ignore cookies, normalize domain names (though I ultimately did this in Apache), normalize the annoying Accept-Encoding header values, etc. Furthermore, if the cache interval is too long for a particular change, Varnish gives you a straightforward, expressive way of purging cached representations, which came in handy on a number of occasions close to launch time.

A number of us at End Point have been interested in Varnish for some time. We’ve made some core patches: JT Justman tracked down a caching bug when using Edge-Side Includes (ESI), and Charles Curley and JT have done some work to add native gzip/deflate support in Varnish, though that remains to be released upstream. We’ve also prototyped a system relying on ESI and message-driven cache purging for an up-to-date, high-speed, extremely scalable architecture. (That particular project hasn’t gone into production yet due to the degree of effort required to refactor much of the underlying app to fit the design, though it may still come to pass next year—I hope!)

Getting Varnish to play nice with Radiant was a non-issue, because the relative simplicity of the site feature set and content did not require specialized handling of any particular resource: one cache interval was good for all pages. Consequently, rather than fretting about having Radiant issue Cache-Control headers on a per-page basis (which may have been fairly unpleasant, though I didn’t look into it deeply; eventually I’ll need to, though, having gotten modestly hooked on Radiant and less-modestly hooked on Varnish), the setup was refreshingly simple:

The public site’s domain disallows all access to the Radiant admin, meaning it’s effectively a read-only site.
The public domain’s Apache container always issues a couple of cache-related headers:

Header always set Cache-Control "public; max-age=120"
Header always set Vary "Accept-Encoding"

The Cache-Control header tells clients (Varnish in this case) that it’s acceptable to cache representations for 120 seconds, and that all representations are valid for all users (“public”). We can, if we want, use VCL to clean this out of the representation Varnish passes along to clients (i.e. browsers) so that browsers don’t cache automatically, instead relying on conditional GET. The Vary header tells clients that cache (again, primarily concerned with Varnish here) to consider the “Accept-Encoding” header value of a request when keying cached representations.

An entirely separate domain exists that is not fronted by Varnish and allows access to the Radiant admin. We could have it fronted by Varnish with caching deactivated, but the configuration we used keeps things clean and simple.
We use some simple VCL to tell Varnish to ignore cookies (in case of Rails sessions on the public site), to normalize the Accept-Encoding header value to one of “gzip” or “deflate” (or none at all) to avoid caching different versions of the same representation due to inconsistent header values submitted by competing browsers.

Getting all that sorted was, as stated, refreshingly easy. It was a little less easy, surprisingly, to deal with logging. The main Varnish daemon (varnishd) logs to a shared memory block. The logs just sit there (and presumably eventually get overwritten) unless consumed by another process. A varnishlog utility, which can be run as a one-off or as a daemon, reads in the logs and outputs them in various ways. Furthermore, a varnishncsa utility outputs logging information in an Apache/NCSA-inspired “combined log” format (though it includes full URLs in the request string rather than just the path portion, presumably due to the possibility of Varnish fronting many different domains). Neither one of these is particularly complicated, though the varnishlog output is reportedly quite verbose and may need frequent rotation, and when run in daemon mode, both will re-open the log file to which they write upon receiving SIGHUP, meaning they’ll play nice with log rotation routines. I found myself repeatedly wishing, however, that they both interfaced with syslog.

So, I’m very happy with Varnish at this point. Being a jerk, I must nevertheless publicly pick a few nits:

Why no syslog support in the logging utilities? Is there a compelling argument against it (I haven’t encountered one, but admittedly I haven’t looked very hard), or is it simply a case of not having been handled yet?
The VCL snippet we used for normalizing the Accept-Encoding header came right off the Varnish FAQ, and seems to be a pretty common case. I wonder if it would make more sense for this to be part of the default VCL configuration requiring explicit deactivation if not desired. It’s not a big deal either way, but it seems like the vast majority of deployments are likely to use this strategy.

That’s all I have to whine about, so either I’m insufficiently observant or the software effectively solves the problem it set out to address. These options are not mutually exclusive.

I’m definitely looking forward to further work with Varnish. This project didn’t get into ESI support at all, but the native ESI support, combined with the high-performance caching, seems like a real win, potentially allowing for simplification of resource design in the application server, since documents can be constructed by the edge server (Varnish in this case) from multiple components. That sort of approach to design calls into question many of the standard practices seen in popular (and unpopular) application servers (namely, high-level templating with “pages” fitting into an overall “layout”) but could help engineers keep maintain component encapsulation, think through more effectively the URL space, resource privacy and scoping considerations (whether or not a resource varies per user, by context, etc.), etc. But I digress. Shocking.

Integrating Contentful with NuxtJS

Pricing

Creating content

Integration

Other content types

Alternatives

Advanced WordPress customizations

Custom CSS and JavaScript

Custom PHP code

Custom hooks

Conclusion

Craft: A CMS for developers

Craft is easy to get up and running

Craft is easy to put in containers

Craft’s design makes sense

Craft gives you complete freedom over your front end

Craft is cool 🕶️

Drupal — rapid development

Low learning curve

Great documentation

Big community

It’s free

Modules

PHP7 support

REST API support

Perfect for a backend system

Power of taxonomies

Summary

Testing Django Applications

Versions of software packages used

Testing Django Applications

Basic Django unittest versus pytest basic examples

Django database interaction unittest versus pytest (advanced examples)

Django pytest notes (advanced example)

pytest setup_method

JSON data fixtures versus factories (pytest advanced example)

Additional useful links

Conclusion

Web Development, Big Data and DevOps—​OSI Days 2014, India

Web Personalisation

MEAN Stack

Moving fast with high performance Hack and PHP

Application Monitoring and Log Management

Apache Spark

Online Data Processing in Twitter

Docker

Ecommerce in the Django World

Installing Cartridge/Mezzanine

Cartridge Product Models

Adding Products / Tracking Orders

Conclusions

Drupal Commerce for Fun and Profit

Getting Started with Drupal Commerce

Creating Products

Conclusions

Ecommerce Solutions: What are the Options?

Which one do you choose?

Spree vs. Magento: Feature List Revisited

Features on a Single Product or Group of Product

CMS Features

Checkout Support

Shipping Support

Payment Support

Admin Features

User Account Features

Site Wide Features

Spree vs Magento: A Feature List Comparison

Features on a Single Product or Group of Product

CMS Features

Checkout Support

Shipping Support

Payment Support

Admin Features

User Account Features

System Wide Features

New End Point site launched: Rails, jQuery, Flot, blog feed

jQuery Flot Integration

Dynamic Rails Partial Generation

jQuery Carousel Functionality

Twitter Integration

Web Development, Big Data and DevOps—OSI Days 2014, India