Django and Mojolicious: a quick comparison of two popular web frameworks

2025-02-06T00:00:00+00:00

Recently I’ve been working on a project with a Vue front-end and two back-ends, one in Python using the Django framework and one in Perl using the Mojolicious framework. So, it’s a good time to spend some words to share the experience and do a quick comparison.

Previously I wrote a post about Perl web frameworks, and now I’m expanding the subject into another language.

Django was chosen for this project because it’s been around for almost 20 years now and provides the needed maturity and stability to be long-running and low-budget. In this regard, it has proved a good choice so far. Recently it saw a major version upgrade without any problems to speak of. It could be argued that I should have used the Django REST Framework instead of plain Django. However, at the time the decision was made, adding a framework on top of another seemed a bit excessive. I don’t have many regrets about this, though.

Mojolicious is an old acquaintance. It used to have fast-paced development but seems very mature now, and it’s even been ported to JavaScript.

Both frameworks have just a few dependencies (which is fairly normal in the Python world, but not in the Perl one) and excellent documentation. They both follow the model-view-controller pattern. Let’s examine the components.

Views

Both frameworks come with a built-in template system (which can be swapped out with something else), but in this project we can skip the topic altogether as both frameworks are used only as back-end for transmitting JSON, without any HTML rendering involved.

However, let’s see how the rendering looks for the API we’re writing.

use Mojo::Base 'Mojolicious::Controller', -signatures;
sub check ($self) {
    $self->render(json => { status => 'OK' });
}

from django.http import JsonResponse
def status(request):
    return JsonResponse({ "status":  "OK" })

Nothing complicated here, just provide the right call.

Models

Django

Usually a model in context of web development means a database and here we are going to keep this assumption.

Django comes with a comprehensive object-relational mapping (ORM) system and it feels like the natural thing to use. I don’t think it makes much sense to use another ORM, or even to use raw SQL queries (though it is possible).

You usually start a Django project by defining the model. The Django ORM gives you the tools to manage the migrations, providing abstraction from the SQL. You need to define the field types and the relationships (joins and foreign keys) using the appropriate class methods.

For example:

from django.db import models
class User(AbstractUser):
    email = models.EmailField(null=False, blank=False)
    site = models.ForeignKey(Site, on_delete=models.CASCADE, related_name="site_users")
    libraries = models.ManyToManyField(Library, related_name="affiliated_users")
    expiration = models.DateTimeField(null=True, blank=True)
    created = models.DateTimeField(auto_now_add=True)
    last_modified = models.DateTimeField(auto_now=True)

These calls provide not only the SQL type to use, but also the validation. For example, the blank parameter is a validation option specifying whether Django will accept an empty value. It is different from the null option, which directly correlates to SQL. You can see we’re quite far from working with SQL, at least two layers of abstraction away.

In the example above, we’re also defining a foreign key between a site and a user (many-to-one), so each user belongs to one site. We also define a many-to-many relationship with the libraries record. I like how these relationships are defined, it’s very concise.

Thanks to these definitions, you get a whole admin console almost for free, which your admin users are sure to like. However, I’m not sure this is a silver bullet for solving all problems. With large tables and relationships the admin pages load slowly and they could become unusable very quickly. Of course, you can tune that by filtering out what you need and what you don’t, but that means things are not as simple as “an admin dashboard for free” — at the very least, there’s some configuring to do.

As for the query syntax, you usually need to call Class.objects.filter(). As you would expect from an ORM, you can chain the calls and finally get objects out of that, representing a database row, which, in turn, you can update or delete.

The syntax for the filter() call is based on the double underscore separator, so you can query over the relationships like this:

for agent in (Agent.objects.filter(canonical_agent_id__isnull=False)
              .prefetch_related('canonical_agent')
              .order_by('canonical_agent__name', 'name')
              .all()):
    agent.name = "Dummy"
    agent.save()

In this case, provided that we defined the foreign keys and the attributes in the model, we can search/order across the relationship. The __isnull suffix, as you can imagine, results in a WHERE canonical_agent_id IS NOT NULL query, while in the order_by call we sort over the joined table using the name column. Looks nice and readable, with a touch of magic.

Of course things are never so simple, so you can build complex queries with the Q class combined with bytewise operators (&, |).

Here’s an example of a simple case-insensitive search for a name containing multiple words:

from django.db.models import Q

def api_list(request)
    term = request.GET.get('search')
    if term
        words = [ w for w in re.split(r'\W+', term) if w ]
        if words:
            query = Q(name__icontains=words.pop())
            while words:
                query = query & Q(name__icontains=words.pop())
            # logger.debug(query)
            agents = Agent.objects.filter(query).all()

To sum up, the ORM is providing everything you need to stay away from the SQL. In fact, it seems like Django doesn’t like you doing raw SQL queries.

Mojolicious and Perl

In the Perl world things are a bit different.

The Mojolicious tutorial doesn’t even mention the database. You can use any ORM or no ORM at all, if you prefer so. However, Mojolicious makes the DB handle available everywhere in the application.

You could use DBIx::Connector, DBIx::Class, Mojo::Pg (which was developed with Mojolicious), or whatever you prefer.

For example, to use Mojo::Pg in the main application class:

package MyApp;
use Mojo::Base 'Mojolicious', -signatures;
use Mojo::Pg;
use Data::Dumper::Concise;

sub startup ($self) {
    my $config = $self->plugin('NotYAMLConfig');
    $self->log->info("Starting up with " . Dumper($config));
    $self->helper(pg => sub {
                      state $pg = Mojo::Pg->new($config->{dbi_connection_string});
                  });

In the routes you can call $self->pg to get the database object.

The three approaches I’ve mentioned here are different.

DBIx::Connector is basically a way to get you a safe DBI handle across forks and DB connection failures.

Mojo::Pg gives you the ability to do abstract queries but also gives some convenient methods to get the results. I wouldn’t call it a ORM; from a query you usually gets hashes, not objects, you don’t need to define the database layout, and it won’t produce migrations for you, though there is some migration support.

Here’s an example of standard and abstract queries:

sub list_texts ($self) {
    if (my $sid = $self->param('sid')) {
        my $sql = 'SELECT * FROM texts WHERE sid = ? ORDER BY sorting_index';
        @all = $self->pg->db->query($sql, $sid)->hashes->each;
    }
    $self->render(json => { texts => \@all });

The query above can be rewritten with an abstract query, using the same module.

@all = $self->pg->db->select(texts => undef,
                             { sid => $sid },
                             { order_by => 'sorting_index' })->hashes->each;

If it’s a simple, static query, it’s basically a matter of taste; do you prefer to see the SQL or not? The second version is usually nicer if you want to build a different query depending on the parameters, so you add or remove keys to the hashes which maps to query and finally execute it.

Now, speaking of taste, for complex queries with a lot of joins I honestly prefer to see the SQL query instead of wondering if the abstract one is producing the correct SQL. This is true regardless of the framework. I have the impression that it is faster, safer, and cleaner to have the explicit SQL in the code rather than leaving future developers (including future me) to wonder if the magic is happening or not.

Finally, nothing stops you from using DBIx::Class, which is the best ORM for Perl, even if it’s not exactly light on dependencies.

It’s very versatile, it can build queries of arbitrary complexity, and you usually get objects out of the queries you make. It doesn’t come with an admin dashboard, it doesn’t enforce the data types and it doesn’t ship any validation by default (of course, you can implement that manually). The query syntax is very close to the Mojo::Pg one (which is basically SQL::Abstract).

The gain here is that, like in Django’s ORM, you can attach your methods to the classes representing the rows, so the data definitions live with the code operating on them.

However, the fact that it builds an object for each result means you’re paying a performance penalty which sometimes can be very high. I think this is a problem common to all ORMs, regardless of the language and framework you’re using.

The difference with Django is that once you have chosen it as your framework, you are basically already sold to the ORM. With Mojolicious and other Perl frameworks (Catalyst, Dancer), you can still make the decision and, at least in theory, change it down the road.

My recommendation would be to keep the model, both code and business logic, decoupled from the web-specific code. This is not really doable with Django, but is fully doable with the Perl frameworks. Just put the DB configuration in a dedicated file and the business code in appropriate classes. Then you should be able to, for example, run a script without loading the web and the whole framework configuration. In this ideal scenario, the web framework just provides the glue between the user and your model.

Controllers

Routes are defined similarly between Django and Mojolicious. Usually you put the code in a class and then point to it, attaching a name to it so you can reference it elsewhere. The language is different, the style is different, but they essentially do the same thing.

Django:

from django.urls import path
from . import views
urlpatterns = [
    path("api/agents/<int:agent_id>", views.api_agent_view, name="api_agent_view"),
]

The function views.api_agent_view will receive the request with the agent_id as a parameter.

Mojolicious:

sub startup ($self) {
    # ....
    my $r = $self->routes;
    $r->get('/list/:sid')->to('API#list_texts')->name('api_list_texts');
}

The ->to method is routing the request to the Myapp::Controller::API::list_texts, which will receive the request with the sid as parameter.

This is pretty much the core business of every web framework: routing a request to a given function.

Mojolicious has also the ability to chain the routes (pretty much taken from Catalyst). The typical use is authorization:

sub startup ($self) {
    ...
    my $r = $self->routes;
    my $api = $r->under('/api/v1', sub ($c) {
        if ($c->req->headers->header('X-API-Key') eq 'testkey') {
            return 1;
        }
        $c->render(text => 'Authentication required!', status => 401);
        return undef;
    }
    $api->get('/check')->to('API#check')->name('api_check');

So the request to /api/v1/check will first go in the first block and the chain will abort if the API key is not set in the header. Otherwise it will proceed to run the API module’s check function.

Conclusion

I’m Perl guy and so I’m a bit biased toward Mojolicious, but I also have a pragmatic approach to programming. Python is widely used — they teach it in schools — while Perl is seen as old-school, if not dead (like all the mature technologies). So, Python could potentially attract more developers to your project, and this is important to consider.

Learning a new language like Python is not a big leap; it and Perl are quite similar despite the different syntax. I’d throw Ruby in the same basket.

Of course both languages provide high quality modules you can use, and these two frameworks are an excellent example.

How to deploy a containerized Django app with AWS Copilot

2022-06-21T00:00:00+00:00

Generally there are 2 major options at AWS when it comes to deployment of containerized applications. You can either go for EKS or ECS.

EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. ECS (Elastic Container Service), on the other hand, is AWS’s own way to manage your containerized application. You can learn more about EKS and ECS on the AWS website.

For this post we will use ECS.

The chosen one and the sidekick

With ECS chosen, now you have to find a preferably easy way to deploy your containerized application on it.

There are quite a number of resources from AWS that are needed for your application to live on ECS, such as VPC (Virtual Private Cloud), Security Group (firewall), EC2 (virtual machine), Load Balancer, and others. Creating these resources manually is cumbersome so AWS has came out with a tool that can automate the creation of all of them. The tool is known as AWS Copilot and we are going to learn how to use it.

Install Docker

Docker or Docker Desktop is required for building the Docker image later. Please refer to my previous article on how to install Docker Desktop on macOS, or follow Docker’s instructions for Linux and Windows.

Set up AWS CLI

We need to set up the Docker AWS CLI (command-line interface) for authentication and authorization to AWS.

Execute the following command to install the AWS CLI on macOS:

$ curl -O "https://awscli.amazonaws.com/AWSCLIV2.pkg"
$ sudo installer -pkg AWSCLIV2.pkg -target /

For other OSes see Amazon’s docs.

Execute the following command and enter the AWS Account and Access Keys.

$ aws configure

Install AWS Copilot CLI

Now it’s time for the main character: AWS Copilot.

Install AWS Copilot with Homebrew for macOS:

$ brew install aws/tap/copilot-cli

See AWS Copilot Installation for other platforms.

The Django project

Create a Django project by using a Python Docker Image. You can clone my Git project to get the Dockerfile, docker-compose.yaml and requirements.txt that I’m using.

$ git clone https://github.com/aburayyanjeffry/django-copilot.git

Go to the django-pilot directory and execute docker-compose to create a Django project named “mydjango”.

$ cd django-copilot
$ docker-compose run web django-admin startproject mydjango .

Edit mydjango/settings.py to allow all hostnames for its URL. This is required because by default AWS will generate a random URL for the application. Find the following variable and set the value as follows:

ALLOWED_HOSTS = ['*']

The Deployment with AWS Copilot

Create an AWS Copilot “Application”. This is a grouping of services such as web app or database, environments (development, QA, production), and CI/CD pipelines. Execute the following command to create an Application with the name of “mydjango”.

$ copilot init -a mydjango

Select the Workload type. Since this Django is an internet-facing app we will choose the “Load Balanced Web Service”.

Which workload type best represents your architecture?  [Use arrows to move, type to filter, ? for more help]
    Request-Driven Web Service  (App Runner)
  > Load Balanced Web Service   (Internet to ECS on Fargate)
    Backend Service             (ECS on Fargate)
    Worker Service              (Events to SQS to ECS on Fargate)
    Scheduled Job               (Scheduled event to State Machine to Fargate)

Give the Workload a name. We are going to name it “mydjango-web”.

Workload type: Load Balanced Web Service

  What do you want to name this service? [? for help] mydjango-web

Select the Dockerfile in the current directory.

Which Dockerfile would you like to use for mydjango-web?  [Use arrows to move, type to filter, ? for more help]
  > ./Dockerfile
    Enter custom path for your Dockerfile
    Use an existing image instead

Accept to create a test environment.

All right, you're all set for local development.

  Would you like to deploy a test environment? [? for help] (y/N) y

Wait and see. At the end of the deployment you will get the URL of your application. Open it in a browser.

Now let’s migrate some data, create a superuser, and try to log in. The Django app comes with a SQLite database. Execute the following command to get a terminal for the Django app:

$ copilot svc exec

Once in the terminal, execute the following to migrate the initial data and to create the superuser:

$ python manage.py migrate
$ python manage.py createsuperuser

Now you may access the admin page and login by using the created credentials.

You should see the Django admin:

A mini cheat sheet

AWS Copilot commands	Remarks
`copilot app ls`	To list available Applications
`copilot app show -n appname`	To get the details of an Application
`copilot app delete -n appname`	To delete an Application
`copilot svc ls`	To list available Services
`copilot svc show -n svcname`	To get the details of a Service
`copilot svc delete -n svcname`	To delete a Service

The End

That’s all, folks.

AWS Copilot is a tool to automate the deployment of AWS infrastructure for our containerized application needs. It takes away most of the worries about infrastructure and enables us to focus sooner on the application development.

For further info on AWS Copilot visit its website.

GraphQL Server Libraries

2019-07-12T00:00:00+00:00

Photo by Jon Flobrant on Unsplash

This post is a followup to my previous post, GraphQL — An Alternative to REST. Please check that out for an introduction to GraphQL and what makes it different from other API solutions. I’ve collected a list of some of the currently-maintained GraphQL libraries for a few different languages, along with some examples (most of which aren’t fully functional on their own, they’d need more configuration) so you can see what it might be like to use GraphQL in your project. I’ll be focusing on the ways each of these libraries implement GraphQL and what you’d need to do to start a project with each of them, so if you have questions about GraphQL itself, please check out my other blog post.

Apollo Server (JavaScript/TypeScript)

Apollo GraphQL has libraries for both a GraphQL server and client (which I’ll discuss later). Apollo Server can be used both as a standalone server as well as with libraries like Express. Apollo Server is the server library I have the most experience with—I wrote a server last year using Express and Apollo Server, along with a client that used Apollo Client. I’m a fan of the flexibiliy of Apollo, but it takes more work to set up than some of the alternatives.

Setting up Apollo Server as a standalone can be done fairly simply following the directions on their website. However, I’m going to go over the basics of integrating with Express. There are two main parts to writing a server with Apollo: your GraphQL schema and your resolvers. These stay more or less the same whether you’re using Apollo as a standalone server or combining it with Express. You do have to have a database set up separately; I’ll show examples with MongoDB, but you could easily swap it out with PostgreSQL or another database. I’ll show an example resolver along with its GraphQL schema for a blog post. The schema follows the GraphQL schema rules and might look like the following:

const typeDefs = [gql`
  type Post {
    id: String!
    body: String!
  }

  query {
    post(id: String!): Post
  }
`];

Now for the resolver. Resolvers are functions that take information from the query (like arguments) and return the relevant data, usually from a database. For our blog post, a resolver might look like this:

const resolvers = {
  post: (root, args, context, info) => {
    return Post.findById(args.id);
  }
};

Simple! We just get the data from the database and return it—as long as the property names match those of our schema, Apollo will automatically format it according to the frontend’s request and return it to them.

OK, next we create the server:

import express from 'express';
import { ApolloServer } from 'apollo-server-express';

const PORT = 3000;

const app = express();
const server = ApolloServer({
  typeDefs,
  resolvers
});

server.applyMiddleware(app);

app.listen(PORT, () => {
  console.log(
    `Server running at http://localhost:${PORT}/graphql`
  );
});

And that’s it! Note that these examples are missing a few things like imports, and we didn’t add authentication of any kind, but this is the general format for creating a server with Apollo.

Prisma

Prisma is a cool library developed by the same people as Graph.cool that does much of the work for you in enabling GraphQL access to your database.

Prisma offers configuration for existing databases, but unfortunately I had trouble getting it to work on my Ubuntu system—I ran into issues getting the Docker container to connect to my local Postgres and MongoDB databases. However, following the quick guide found here on the Prisma website, I was able to get a GraphQL server up and running inside a Docker container with a new database. The process was simple:

First, you have to install the Prisma command line utility:

npm install -g prisma

You also need to have Docker installed. Documentation for Docker can be found here.

Next, you need to configure Prisma. Create a directory for your Prisma server, and create a new file named docker-compose.yml:

mkdir hello-world
cd hello-world
touch docker-compose.yml

Then, paste the following into it:

version: '3'
services:
  prisma:
    image: prismagraphql/prisma:1.34
    restart: always
    ports:
      - '4466:4466'
    environment:
      PRISMA_CONFIG: |
        port: 4466
        databases:
          default:
            connector: mongo
            uri: mongodb://prisma:prisma@mongo
  mongo:
    image: mongo:3.6
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: prisma
      MONGO_INITDB_ROOT_PASSWORD: prisma
    ports:
      - '27017:27017'
    volumes:
      - mongo:/var/lib/mongo
volumes:
  mongo: ~

I used Mongo here, but Prisma’s site has guides for PostgreSQL and MySQL as well. It’s important to make sure now that you don’t have any conflicts with currently running databases—on my machine I had already had a MongoDB server running on port 27017. I fixed this by just stopping my local MongoDB server, but I’m sure you could configure the Docker containers to work with different ports as well. Running Ubuntu, I just ran sudo service mongodb stop and then the Prisma Docker containers worked just fine. When I was done, I ran sudo service mongodb start to start it up again.

Next, you’ll start the Prisma containers and initialize the Prisma server configuration:

docker-compose up -d
prisma init --endpoint http://localhost:4466

The final step is to deploy the service:

prisma deploy

If all goes well, you’ll see a message that includes a URL to the Prisma Admin, which is a browser tool to interact with your GraphQL endpoints. I used it for a little while when I was testing Prisma out, and it seems to work well and is easy to use.

All in all, Prisma seems like a great way to start if you don’t want to handle the messy details of setup. However, I did have issues getting it to play nice with my already-existing databases (including both PostgreSQL and MongoDB). However, it is still relatively new, so I would expect support to get better over time.

Graphene (Python)

Graphene is a GraphQL framework for Python. It has integrations for a few different server frameworks (a list can be found here), but I’ll show examples from graphene-django, since Django is fairly common and something that we use fairly often at End Point.

Because you’re also setting up a Django project, the tutorial for graphene-django is a little more involved, so I’ll just share the relevant GraphQL sections so you can compare to the other libraries in this post. The most important part, the schema, is defined in Python with a similar format to Django models:

import graphene

from graphene_django.types import DjangoObjectType

from app.models import Category, Ingredient

class CategoryType(DjangoObjectType):
    class Meta:
        model = Category

class IngredientType(DjangoObjectType):
    class Meta:
        model = Ingredient

class Query(object):
    all_categories = graphene.List(CategoryType)
    all_ingredients = graphene.List(IngredientType)

    def resolve_all_categories(self, info, **kwargs):
        return Category.objects.all()

    def resolve_all_ingredients(self, info, **kwargs):
        # We can easily optimize query count in the resolve method
        return Ingredient.objects.select_related('category').all()

As you can see, the format for defining your GraphQL schema is quite different from some other libraries, but you have the advantage of it looking similar to Django’s model definitions. You’ll also need a higher-level Query definition:

import graphene

import cookbook.ingredients.schema


class Query(cookbook.ingredients.schema.Query, graphene.ObjectType):
    # This class will inherit from multiple Queries
    # as we begin to add more apps to our project
    pass

schema = graphene.Schema(query=Query)

Now that we have a schema defined, we need to add a few things to settings.py:

INSTALLED_APPS = [
    ...
    # This will also make the `graphql_schema` management command available
    'graphene_django',
]

GRAPHENE = {
    'SCHEMA': 'cookbook.schema.schema'
}

The last piece needed to use your GraphQL schema is in urls.py:

from graphene_django.views import GraphQLView

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^graphql$', GraphQLView.as_view(graphiql=True)),
]

And finally we run the server:

$ python manage.py runserver

Now you should be able to use your GraphQL schema at http://localhost:8000/graphql just like with any other GraphQL server.

Graphene for Django seems like a good solution in that it uses a similar format to other aspects of Django, like the model definitions. However, its format (especially for schema definition) is rather different-looking from the GraphQL standard used by most other libraries, and it seems like it might make it more work to keep your frontend and backend in sync.

Graph.cool

I won’t discuss Graph.cool in detail here, because I went over it in my previous blog post. However, it still merits mention here as an option for your GraphQL server. Essentially, Graph.cool lets you define a GraphQL schema and then handles the work of setting up a database and even hosting for you. If you just want to get a basic GraphQL server set up for testing, or if you don’t need too many features beyond data storage and retrieval, Graph.cool is a great choice.

Additional links

For server libraries in other languages, these seem like good options:

Thanks for reading! Keep an eye out next week for a second post which will cover GraphQL client libraries.

Client Case Study: Carjojo

2016-12-23T00:00:00+00:00

Carjojo’s site makes use of some of the best tools on the market today for accessing and displaying data. Carjojo is a car buying application that takes data about car pricing, dealer incentives, and rebate programs and aggregates that into a location-specific vehicle pricing search tool. The Carjojo work presented a great opportunity for End Point to utilize our technical skills to build a state-of-the-art application everyone is very proud of. End Point worked on the Carjojo development project from October of 2014 through early 2016, and the final Carjojo application launched in the summer of 2016. This case study shows that End Point can be a technology partner for a startup, enabling the client to maintain their own business once our part of the project is over.

Why End Point?

Reputation in full stack development

End Point has deep experience with full stack development so for a startup getting advice from our team can prove really helpful when deciding what technologies to implement and what timelines are realistic. Even though the bulk of the Carjojo work focused on specific development pieces, having developers available to help advise on the entire stack allows a small startup to leverage a much broader set of skills.

Startup Budget and Timelines

End Point has worked with a number of startups throughout our time in the business. Startups require particular focused attention on budget and timelines to ensure that the minimum viable product can be ready on time and that the project stays on budget. Our consultants focus on communication with the client and advise them on how to steer the development to meet their needs, even if those shift as the project unfolds.

Client Side Development Team

One of the best things about a lot of our clients is their technological knowledge and the team they bring to the table. In the case of Carjojo, End Point developers fit inside of their Carjojo team to build parts that they were unfamiliar with. End Point developers are easy to work with and already work in a remote development environment, so working in a remote team is a natural fit.

Client Side Project Management

End Point works on projects where either the project management is done in-house or by the client. In the case of a project like Carjojo where the client has technical project management resources, our engineers work within that team. This allows a startup like Carjojo insight into the project on a daily basis.

Project Overview

The main goal of the Carjojo project was to aggregate several data sources on car price and use data analytics to provide useful shopper information, and display that for their clients.

Carjojo’s staff had experience in the car industry and leveraged that to build a sizeable database of information. Analytics work on the database provided another layer of information, creating a time- and location-specific market value for a vehicle.

Carjojo kept the bulk of the database collection and admin work in house, as well as provided an in-house designer that closely worked with them on their vision for the project. End Point partnered to do the API architecture work as well as the front end development.

A major component of this project was using a custom API to pull information from the database and display it quickly with high end, helpful infographics. Carjojo opted to use APIs so that the coding work would seamlessly integrate with future plans for a mobile application, which normally require a substantial amount of recoding.

Creating a custom API also allows Carjojo to work with future partners and leverage their data and analytics in new ways as their business grows.

Team

Patrick Lewis: End Point project manager and front end developer. Patrick led development of the AngularJS front end application which serves as the main customer car shopping experience on the Carjojo site. He also created data stories using combinations of integrated Google Maps, D3/DimpleJS charts, and data tables to aid buyers with car searches and comparisons.

Matt Galvin: Front end developer. Matt led the efforts for data-visualization with D3 and DimpleJS. He created Angular services that were used to communicate with the backend, used D3 and DimpleJS to illustrate information graphically about cars, car dealers, incentives, etc., sometimes neatly packaging them into directives for easy re-use when the case fit. He also created a wealth of customizations and extensions of DimpleJS which allowed for rapid development without sacrificing visualization quality.

Josh Williams: Python API development. Josh led the efforts in connecting the database into Django and Python to process and aggregate the data as needed. He also used TastyPie to format the API response and created authentication structures for the API.

Project Specifics

API Tools

Carjojo’s project makes use of some of the best tools on the market today for accessing and displaying data. Django and Tastypie were chosen to allow for rapid API development and to keep the response time down on the website. In most cases the Django ORM was sufficient for generating queries from the data, though in some cases custom queries were written to better aggregate and filter the data directly within Postgres.

To use the location information in the database, some GIS location smarts were tied into Tastypie. Location searches tied into GeoDjango and generated PostGIS queries in the database.

Front End Tools

D3 is standard in data-visualization and is great for doing both simple and complicated graphics. Many of Carjojo’s graphs were bar graphs, pie charts and didn’t really require writing out D3 by hand. We also wanted to make many of them reusable and dynamic (often based on search terms or inputs) with use of Angular directives and services. This could have been done with pure D3, but Dimple makes creating simple D3 graphs easy and fast.

DimpleJS was used a lot in this project. Since Carjojo is data-driven, they wanted to display their information in an aesthetically pleasing manner and DimpleJS allowed us to quickly spin up information against some of the project’s tightest deadlines.

The approach worked well for most cases. However, sometimes Carjojo wanted something slightly different than what DimpleJS does out of the box. One example of DimpleJS customization work can be found here on our blog.

Another thing to note about the data visualizations was that sometimes when the data was plotted and graphed, it brought to light some discrepancies in the back-end calculations and analytics, requiring some back-and-forth between the Carjojo DBA and End Point.

Results

Carjojo had a successful launch of their service in the summer of 2016. Their system has robust user capabilities, a modern clean design, and a solid platform to grow from. The best news for Carjojo is that now the project has been turned back over to them for development. End Point believes in empowering our clients to move forward with their business and goals without us. Carjojo knows that we’ll be here for support if they need it.

Executing Custom SQL in Django Migrations

2016-09-17T00:00:00+00:00

Since version 1.7, Django has natively supported database migrations similar to Rails migrations. The biggest difference fundamentally between the two is the way the migrations are created: Rails migrations are written by hand, specifying changes you want made to the database, while Django migrations are usually automatically generated to mirror the database schema in its current state.

Usually, Django’s automatic schema detection works quite nicely, but occasionally you will have to write some custom migration that Django can’t properly generate, such as a functional index in PostgreSQL.

Creating an empty migration

To create a custom migration, it’s easiest to start by generating an empty migration. In this example, it’ll be for an application called blog:

$ ./manage.py makemigrations blog --empty -n create_custom_index
Migrations for 'blog':
  0002_create_custom_index.py:

This generates a file at blog/migrations/0002_create_custom_index.py that will look something like this:

# -*- coding: utf-8 -*-
# Generated by Django 1.9.4 on 2016-09-17 17:35
from __future__ import unicode_literals

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('blog', '0001_initial'),
    ]

    operations = [
    ]

Adding Custom SQL to a Migration

The best way to run custom SQL in a migration is through the migration.RunSQL operation. RunSQL allows you to write code for migrating forwards and backwards—that is, applying migrations and unapplying them. In this example, the first string in RunSQL is the forward SQL, the second is the reverse SQL.

# -*- coding: utf-8 -*-
# Generated by Django 1.9.4 on 2016-09-17 17:35
from __future__ import unicode_literals

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('blog', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            "CREATE INDEX i_active_posts ON posts(id) WHERE active",
            "DROP INDEX i_active_posts"
        )
    ]

Unless you’re using Postgres for your database, you’ll need to install the sqlparse library, which allows Django to break the SQL strings into individual statements.

Running the Migrations

Running your migrations is easy:

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: blog, sessions, auth, contenttypes, admin
Running migrations:
  Rendering model states... DONE
  Applying blog.0002_create_custom_index... OK

Unapplying migrations is also simple. Just provide the name of the app to migrate and the id of the migration you want to go to, or “zero” to reverse all migrations on that app:

$./manage.py migrate blog 0001
Operations to perform:
  Target specific migration: 0001_initial, from blog
Running migrations:
  Rendering model states... DONE
  Unapplying blog.0002_create_custom_index... OK

Hand-written migrations can be used for many other operations, including data migrations. Full documentation for migrations can be found in the Django documentation.

(This post originally covered South migrations and was updated by Phin Jensen to illustrate the now-native Django migrations.)

Book review: “Two Scoops of Django: Best Practices for Django 1.8”

2016-07-13T00:00:00+00:00

Two Scoops of Django: Best Practices for Django 1.8 is, shockingly, a book about best practices. It’s not a Django library reference, or a book about Django fundamentals, or tips and tricks. It’s a book designed to help web developers, novices and experts alike, avoid common pitfalls in every stage of the web development process, specifically the process of developing with Django.

The book can be used as a reference of best practices and a cover-to-cover guide to best practices. I’ve done both and found it to be enjoyable, accessible, and educational when read cover-to-cover and a valuable reference when setting up a new Django project or doing general Django development. It covers a huge range of material, answering questions like:

Where should I store secret keys?
Should I use virtualenv?
How should I structure my Django project?
When should I use ‘blank’ and ‘null’ in model fields?
Should I use Class-Based Views or Function-Based Views?
How should I structure my URLConfs?
When should I use Forms?
Where should templates be stored?
Should I write custom template tags and filters?
What package should I use to create a REST API?
What core components should I replace?
How can I modify or replace the User and authentication system?
How should I test my app?
How can I improve performance?
How can I keep my app secure?
How do I properly use the logging library?
How do I deploy to a Platform as a Service or my own server(s)?
What can I do to improve the debugging process?
What are some good third-party packages?
Where can I find good documentation and tutorials?
Where should I go to ask more questions?

The question, then, is whether or not this book delivers this information well. For the most part, it does. It’s important to recognize that the book doesn’t cover any of these subjects in great detail, but it does do a great job explaining the “why” behind some of the simple rules that are established and referencing online resources that can be used to go much more in-depth with the subject. It does a great job showing clearly marked bad examples, making it very easy to see whether or not you are or were planning on doing something badly. The writing style is very accessible and straightforward; I read large portions of the book during breakfast or lunch.

Two sections stood out to me as being very helpful for my own projects. First is Chapter 4, Fundamentals of Django App Design, which explained better than any resource I’ve found yet exactly what a Django “app” (as in ./manage.py startapp polls) should be used for. It explains what an app should or shouldn’t have in it, how much an app should do, when you should break into separate apps, and more.

The next section that really helped me was Chapter 6, Model Best Practices, which explained things like what code should and shouldn’t be in a model, how to use migrations and managers, and ModelFields that should be avoided. Perhaps the most useful part of that chapter is a table in the section “When to use Null and Blank,” which makes for an easy and quick reference to which fields go well with the null and blank parameters and when you should use both or neither.

The only real problem I had with Two Scoops of Django was that the illustrations rarely felt necessary or helpful. The majority of them are ice cream-themed jokes that aren’t particularly funny. Overall, I really enjoyed this book and I definitely recommend it for anybody who is or is interested in doing serious Django development.

Testing Django Applications

2015-12-14T00:00:00+00:00

This post summarizes some observations and guidelines originating from introducing the pytest unit testing framework into our CMS (Content Management System) component of the Liquid Galaxy. Our Django-based CMS allows users to define scenes, presentations and assets (StreetView, Earth tours, panos, etc) to be displayed on the Liquid Galaxy.

The purpose of this blog post is to capture my Django and testing study points, summarize useful resource links as well as to itemize some guidelines for implementing tests for newcomers to the project. It also provides a comparison between Python’s standard unittest library and the aforementioned pytest. Its focus is on Django database interaction.

Versions of software packages used

This post describes some of our experiences at End Point in designing and working on comprehensive QA/CI facilities for a new system which is closely related to the Liquid Galaxy.

The experiments were done on Ubuntu Linux 14.04:

python (2.7.6) and its corresponding version of unittest
django 1.7 (current recent is 1.9 but our CMS uses still 1.7 version)
pytest-django 2.8.0
pytest 2.7.2 (with py 1.4.30)
virtualenv 13.1.2
factory_boy 2.6.0

Testing Django Applications

We probably don’t need to talk much about the importance of testing. Writing tests along with the application code has become standard over the years. Surely, developers may fall into a trap of their own prejudice when creating testing conditions which would still result in faulty software but the likelihood of buggy software is certainly higher on a code that has no QA measures. If the code works and is untested, it means it works by accident, they say.

As a rule of thumb, unit tests should be very brief testing items seldom interacting with any external services such as the database. Integration tests on the other hand often communicate with external components.

This post will heavily reference an example minimal Django application written for the purpose of experimenting on Django testing. Its README file contains some set up and requirement notes. Also, I am not going to list (m)any code snippets here but rather reference the functional application and its test suite. Hence the points below qualify for more or less assorted little topics or observations.

In order to benefit from this post, it will be helpful to follow the README and interact (run tests that is) with the demo django-testing application.

Basic Django unittest versus pytest basic examples

This pair of test modules shows the differences between Django TestCase (unittest) and pytest-django (pytest) frameworks.

test_unittest_style.py

The base Django TestCase class derives along this tree:

    django.test.TestCase
        django.test.TransactionTestCase
            django.test.SimpleTestCase
                unittest.TestCase

Django adds (among any other aspects) handling of database, the documentation is here, on top of the Python standard unittest library.

test_pytest_style.py

this is a pytest style implementation of the same tests and pytest-django plug-in adds, among other features, Django database handling support.

The advantage of unittest is that it comes with the Python installation—it’s a standard library. That means that one does not have to install anything for writing tests, unlike pytest which is a third-party library and needs to be installed separately. While the absence of additional installation is certainly a plus, it’s dubious whether being a part of Python distribution is a benefit. I seem to recall Guido Van Rossum during Europython 2010 having said the the best thing for pytest is not being part of the Python standard set of libraries for its lively development and evolution would be slowed down by the inclusion.

There are very good talks and articles summarizing advantages of pytest. For me personally, the reporting of error context is supreme. No boiler-plate (no inheritance), using plain Python asserts instead of many assert* methods and flexibility (function, class) are other big plus points

Testing Django applications with py.test (EuroPython 2013) (very good)
pytest presentation from its author
very brief pytest introduction
switch to pytest, rich features descriptions, has some Django touches

As the comment in the test_unittest_style.py file says, this particular unittest-based test module can be run by both Django manage.py (which boils down to unittest lookup discovery on a lower layer) or by py.test (pytest).

It should also be noted, that pytest’s flexibility can bite back if something gets overlooked.

Django database interaction unittest versus pytest (advanced examples)

test_unittest_advanced.py

Since this post concentrates on pytest and since it’s the choice for our LG CMS project (naturally :-), this unittest example just shows how the test (fresh) database is determined and how Django migrations are run at each test suite execution. Just as described in the Django documentation: “If your tests rely on database access such as creating or querying models, be sure to create your test classes as subclasses of django.test.TestCase rather than unittest.TestCase.”

That is true for database interaction but not completely true when using pytest. And “Using unittest.TestCase avoids the cost of running each test in a transaction and flushing the database, but if your tests interact with the database their behavior will vary based on the order that the test runner executes them. This can lead to unit tests that pass when run in isolation but fail when run in a suite.” django.test.TestCase, however, ensures that each test runs inside a transaction to provide isolation. The transaction is rolled back once the test case is over.

test_pytest_advanced.py

This file represents the actual core of the test experiments for this blog / demo app and shows various pytest features and approaches typical for this framework as well as Django (pytest-django that is) specifics.

Django pytest notes (advanced example)

Much like the unittest documentation, the pytest-django recommends avoiding database interaction in unittest and concentrate only on the logic which should be designed in such a fashion that it can be tested without database.

test database name prefixed test_ (just like at the unittest example), the base value is taken from the database section of the settings.py. As a matter of fact, it’s possible to run the test suite after previously dropping the main database, the test suite interacts only with test_ + DATABASE_NAME
migration execution before any database interaction is carried out (similarly to unittest example)
database interaction marked by a Python decorator @pytest.mark.django_db on the method or class level (or stand-alone function level). It’s in fact the first occurrence of this marker which triggers the database set up (its creation and migrations handling). Again analogously to unittest (django.test.TestCase), the test case is wrapped in a database transaction which puts the database back into the state prior to the test case. The database test_ + DATABASE_NAME itself is dropped once the test suite run is over. The database is not dropped if --db-reuse option is used. The production DATABASE_NAME remains untouched during the test suite run (more about this below)
pytest_djangodb_only.py — setup_method — run this module separately and the data created in setup_method end up NOT in the test_ + DATABASE_NAME database but in the standard one (as configured in the settings.py which would be the production database likely)! Also this data won’t be rolled back. When run separately, this test module will pass (but still the production database would be tainted). It may or may not fail on the second and subsequent run depending whether it creates any unique data. When run within the test suite, the database call from the setup_method will fail despite the presence of the class django_db marker. This has been very important to realize. Recommendation: do not include database interaction in the pytest special methods (such assetup_method or teardown_method, etc), only include database interaction in the test case methods
The error message Failed: Database access not allowed, use the "django_db" mark to enable was seen on a database error on a method which actually had the marker. This output is not to be 100% trusted
data model factories are discussed separately below
lastly the test module shows Django Client instance and calling an HTTP resource

pytest setup_method

While the fundamental differences between unittest and pytest were discussed, there is something to be said about Django specific differences of the two. There is different database-related behaviour of unittest setUp method versus the pytest setup_method method. The setUp is included in the transaction and database interactions are rolled back once the test case is over. The setup_method is not included in the transaction. Moreover, interacting with the database from setup_method results in faulty behaviour and difference depending whether the test module is run on its own or as a part of the whole test suite.

The bottom line is: do not include database interaction in setup_method. This setUp, setup_method behaviour was already shown in the basic examples. And more description and demonstration of this behaviour is in the file: pytest_djangodb_only.py. This actually revealed the fact that using django_db database fixture is not supported in special pytest methods and the aforementioned error message is misleading (more references here and here).

When running the whole test suite, this file won’t be collected (its name lacks test_ string). It needs to be renamed to be included in the test suite run.

JSON data fixtures versus factories (pytest advanced example)

The traditional way of interacting with some test data was to perform following steps:

have data loaded in the database
python manage.py dumpdata
the produced JSON file is dragged along the application test code
call_command(“loaddata”, fixture_json_file_name) happens at each test suite run

The load is expensive, the JSON dump file is hard to maintain manually if the original modified copy and the current needs diverge (the file has integer primary keys value, etc). Although even the recent Django testing documentation mentions usage of JSON data fixtures, the approach is considered discouraged and the goal is recommended to achieve by means of loading the data in migrations or using model data factories.

This talk for example compares the both approaches in favour of factory_boy library. A quote from the article: “Factory Boy is a Python port of a popular Ruby project called Factory Girl. It provides a declarative syntax for how new instances should be created. … Using fixtures for complex data structures in your tests is fraught with peril. They are hard to maintain and they make your tests slow. Creating model instances as they are needed is a cleaner way to write your tests which will make them faster and more maintainable.”

The file test_pytest_advanced.py demostrates interaction with factories defined in the module factories.py, the basic very easy-to-use features.

Despite its ease of use, the factory_boy is a powerful library capable of modeling Django’s ORM many-to-many relationships, among other features.

Additional useful links

Django 1.7 testing — version used in the demo application
Django 2.0 testing — latest stable version
Effective Django — testing covered already in the second chapter of the book
Effective Django factory_boy
Django testing — excellent PyCon talk, slides covering pytest, fixtures vs factories, etc

Conclusion

You should have a good idea about testing differences via unittest and pytest in the Django environment. The emphasis has been put on pytest (django-pytest) and some recommended approaches. The demo application django-testing brings functional test cases demonstrating the behaviour and features discussed. The articles and talks listed in this post were extremely helpful and instrumental in gaining expertise in the area and introducing rigorous testing approach into the production application.

Any discrepancy between the behaviour described above and on your own setup may originate from different software versions. In any case, if anything is not clear enough, please let me know in the comments.

Ecommerce in the Django World

2014-12-03T00:00:00+00:00

Mezzanine (http://mezzanine.jupo.org/) is a powerful piece of software written for the Django Framework in Python that functions like a structured content management system similar to Drupal, WordPress, and others. Built on top of this CMS structure is a module which adds features for Ecommerce which is known as Cartridge (http://cartridge.jupo.org/).

Installing Cartridge/Mezzanine

Installing the Mezzanine CMS system from scratch can be accomplished in two methods, either by using the pip python package manager, or you can clone the current source from the git repository from the software maintainers. If you were planning to modify either Mezzanine or Cartridge to customize the setup for your own needs the latter would likely be preferable as you could then easily begin creating custom branches off the original source to track the customization work. However for this example I will show the pip method of installation:

pip install -U cartridge

Once installed, there is a mezzanine-project command which will allow you to create a new blank Mezzanine environment within a new directory, and in this case we will also send an option to instruct the command to install the Cartridge module as well.

mezzanine-project -a cartridge new_cartridge_project

At this point you will have a blank Mezzanine environment with the cartridge module installed, now the Django database must be populated with the model information for the application, and then the Django application server will be started.

cd new_cartridge_project
python manage.py createdb
python manage.py runserver

If all went well, you should see the startup messages for the Django application server which will list the version numbers of the various libraries it will use, and then should be up and listening on the loopback interface. At this point to complete the setup you would just need to point your httpd at this loopback socket to send connections there.

Cartridge Product Models

Products within Cartridge are defined within the python models for the application, there are three primary models to be concerned with in a Product definition, the Product, ProductVariation and ProductImage models specifically.

Product — Defines the primary attributes for a product like name, price, SKU, and can be populated with optional fields for sale prices, etc.
ProductVariation — Defines a variant of a product SKU, these would be most commonly used for things like product sizes and colors.
ProductImage — Defines the image for the picture of each product.

Each of these three models exist within Django as their own data sets, but then they have foreign key references from each Product to all of its Variants and Images. While you can alter these model definitions using the django shell and other methods within python, you can also update and manage the products within the admin interface for the django site.

Adding Products / Tracking Orders

Adding products through the admin interface for Django was relatively easy and quick:

Within this interface you would provide the details for the products name, published status, date range for the product to be on the site, and product description in addition to any Variants or Images for the product. This is where the Mezzanine foundation of Cartridge start to show through, where this process mirrors the creating of a content page within Mezzanine, but has added these attributes of the product to the definition.

In the same way, Product Variants, Discount Codes, and Sales can be created in much the same way within the admin interface. In this way, once you had a basic Cartridge setup in place on your server, within most use cases for setting up a simple web store, Cartridge would remove the need to do any further hacking of python code, and would allow any user familiar with a CMS workflow for creating and managing the objects within the environment to manage the store.

In addition, Cartridge provides order management within the admin interface as well:

From this interface you can update orders, create new orders, print the PDF invoices for the orders and track whether orders have been fulfilled and shipped to the customer.

Conclusions

Like other Ecommerce applications built on top of a Content Management System, Cartridge and Mezzanine work well together to create a stack that allows for both ease of getting started with the system, and a massive degree of customization possible. Each component of Cartridge is defined within Django, and can be rewritten and customized to serve any number of use cases. The setup process for Cartridge was pretty straightforward, but one recommendation I can make is that once you have a working system up, you should place all of the files for Mezzanine and Cartridge into a source code management system like Git. Like other web applications of this type, the complexity of each change to the system can create a possible point of failure. Once you start making changes to the Mezzanine configuration files and the python for the Models and Views for Cartridge, controlling those changes becomes critical. In this way Mezzanine and Cartridge are spanning the normal continuum of software with the polar extremes of ease of use and customization by providing a good example to start with in the setup, and also exposing the full range of customization to the developer.

Integrate Twilio in Django

2014-11-24T00:00:00+00:00

Twilio

Twilio is a powerful HTTP API that allows you to build powerful voice and SMS apps. The goal of this blog post is to help make building the SMS applications as simple as possible in django.

There is a already Twilio Python help library available. The open source twilio-python library lets us to write Python code to make HTTP requests to the Twilio API.

Installation

The easiest way to install twilio-python library is using pip. Pip is a package manager for Python.

Simply run following command in terminal.

$ pip install twilio

Twilio API Credentails

To Integrate twilio API in django application, we need TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN variables. These variables can be found by logging into your Twilio account dashboard. These variables are used to communicate with the Twilio API.

You’ll need to add them to your settings.py file:

TWILIO_ACCOUNT_SID = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
TWILIO_AUTH_TOKEN = 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'

Create a New App

We are going to interact with people using SMS, so I prefer to create an app named communication. I am assuming you’ve already installed Django.

Run following command in terminal.

$ django-admin.py startapp communcation

We will need to register the new app in our django project.

Add it to your INSTALLED_APPS tuple in your settings.py file:

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',

‘communication’,
...
)

Create the Model

Now we’ll open up communication/models.py to start creating models for our app.

class SendSMS(models.Model):
    to_number = models.CharField(max_length=30)
    from_number = models.CharField(max_length=30)
    sms_sid = models.CharField(max_length=34, default="", blank=True)
    account_sid = models.CharField(max_length=34, default="", blank=True)
    created_at = models.DateTimeField(auto_now_add=True)
    sent_at = models.DateTimeField(null=True, blank=True)
    delivered_at = models.DateTimeField(null=True, blank=True)
    status = models.CharField(max_length=20, default="", blank=True)

and run the syncdb command after defining the model:

$ python manage.py syncdb

It will create the necessary database tables for our app.

Create utils.py file

Create a new file named utils.py and save in communication/utils.py.

Put the following code in communication/utils.py:

from django.conf import settings

import twilio
import twilio.rest

def send_twilio_message(to_number, body):
    client = twilio.rest.TwilioRestClient(
        settings.TWILIO_ACCOUNT_SID, settings.TWILIO_AUTH_TOKEN)

    return client.messages.create(
        body=body,
        to=to_number,
        from_=settings.TWILIO_PHONE_NUMBER
    )

Testing send_twilio_message

Open the shell and run following commands.

>>> from communication.utils import send_twilio_message
>>> sms = send_twilio_message('+15005550006', 'Hello Endpointer,')
>>> print sms.sid
SM97f8ac9321114af1b7fd4463ff8bd038

Having the sid means that everything in the backend is working fine. And we can proceed to work on the front end.

Create Form

Lets create a form to gather the data. Now open/create up communication/forms.py to start creating forms for our app. And paste the following code into it:

class SendSMSForm(forms.ModelForm):

    class Meta:
        model = SendSMS
        fields = ('to_number', 'body')

The View CreateView

class SendSmsCreateView(CreateView):
    model = SendSMS
    form_class = SendSMSForm
    template_name = 'communication/sendsms_form.html'
    success_url = reverse_lazy('send_sms')

    def form_valid(self, form):
        number = self.cleaned_data['to_number']
        body = self.cleaned_data['body']
        # call twilio
        sent = send_twilio_message(number, body)

        # save form
        send_sms = form.save(commit=False)
        send_sms.from_number = settings.TWILIO_PHONE_NUMBER
        send_sms.sms_sid = sent.sid
        send_sms.account_sid = sent.account_sid
        send_sms.status = sent.status
        send_sms.sent_at = now()
        if sent.price:
            send_sms.price = Decimal(force_text(sent.price))
            send_sms.price_unit = sent.price_unit
        send_sms.save()

    return super(SendSmsCreateView, self).form_valid(form)

Defining URLS

The URL configuration tells Django how to match a request’s path to your Python code. Django looks for the URL configuration, defined as urlpatterns, in the urls.py file in your project:

from django.conf.urls import patterns, url

from .views import SendSmsCreateView

urlpatterns = patterns('',
    url(
        regex=r'^communication/send/sms/$',
        view=SendSmsCreateView.as_view(),
        name='send_sms'
    ),
)

Creating the Template

Now that we’ve defined a URL for our list view, we can try it out. Django includes a server suitable for development purposes that you can use to easily test your project:

If you visit the http://127.0.0.1:8000/communication/send/sms/ in your browser, though, you’ll see an error: TemplateDoesNotExist.

This is because we have not defined the template file yet. So now create sendsms_form.html file in templates/communication/ and put the following code in it:

<form action="." class="form-horizontal" method="post" role="form">{% csrf_token %}
    {% for field in form %}
        <div class="form-group">            <label class="col-sm-2 control-label" for="inputEmail3">{{ field.label_tag }}</label>
            <div class="col-sm-10">                {{ field }}
                {{ field.errors }}
            </div>        </div>    {% endfor %}
    <div class="form-group">        <div class="col-sm-offset-2 col-sm-10">            <button class="btn btn-default" type="submit">Submit</button>
        </div>    </div></form>

Now reload the http://127.0.0.1:8000/communication/send/sms/ in your browser. Assuming everything is okay, you should then see the following form:

Fill out the form, and hit the submit button to send your SMS.

CONCLUSION

Congratulations, your SMS is successfully sent. Good luck!

Asynchronous Page Switches with Django

2013-11-19T00:00:00+00:00

Now that the newly rebuilt endpoint.com website is up and running, you may have noticed it does something fancy: internal links within the site are fetched in the background, and the page is replaced dynamically with a script. That eliminates the ‘flicker’ of normal website navigation, and removes the need for the browser to re-parse CSS and JavaScript, making it feel more responsive.

Recently I did some work on a Django project that uses jQuery for some AJAX calls to send information back to the database. It was a fairly simple $.post() call, but it got me thinking about Django’s template inheritance and how it could be used to render parts of templates and update those client-side without having to render the whole thing. The idea being, if your base template is complex and has a number of built-in queries or calculations, for example if you have a dynamic navigation menu, why put extra load on Postgres, on the web server, or have the browser reload the CSS, JS, images, or other resources, to load in what could be otherwise static data into a content column?

The idea’s a little half-baked, just the result of a little after-hours tinkering over a couple evenings. Certainly hasn’t been fleshed out in a production environment, but seems to work okay in a test. I probably won’t develop it much more, but maybe the concepts will help someone looking for something similar.

There’s a few options out there, apparently, between django-ajax-blocks (which seems to do something similar to what I’m about to describe) and Tastypie, which lets you easily build with REST-based frameworks. Django’s usually pretty good about that, having projects available that build functionality on top of it. But not having researched those at the time, I put together this basic technique for doing the same:

Create a replacement template render function that detects AJAX-y requests.
Update your page templates to use a variable in {% extends %}.*
Create a simple XML base template with the blocks you use in your normal base template.
Add a little JavaScript to perform the content switch.

Yes, this works, which was a bit of a surprise to me. It’s also the part I’m least happy with. More on that in a bit.

The details…

Template Render Function

This takes after the handy django.shortcuts.render function. In fact, it leans on it fairly heavily.

def my_render(request, template, context={}, html_base='main.html', xml_base='main.xml'):
    if request.is_ajax():
        context['base_template'] = xml_base
        return render(request, template, context, content_type='application/xml')
    else:
        context['base_template'] = html_base
        return render(request, template, context)

Giving html_base and xml_base as parameters lets views override those. This then injects a new variable, base_template, into the context passed to it with the appropriate base template.

Update Page Templates

Your page templates, assuming they now at the top say {% extends ‘main.html’ %}, replace with {% extends base_template %}. You shouldn’t have to make any other changes to it.

But again, this is the bit I’m least happy about. It takes the selection out of the page template, and puts it in code. That takes away some of the decoupling an MVC environment like this is supposed to provide. Haven’t come up with a way around it, though.

Create XML Base Template

In templates/main.xml (or whatever you want to call it above) create XML nodes for the blocks in your main.html file. Or at least the blocks your pages will replace:

<?xml version="1.0" encoding="UTF-8"?>
<content>
 <title><![CDATA[{% block title %}Django Site!{% endblock %}]]></title>
 <main_content><![CDATA[{% block main_content %}{% endblock %}]]></main_content>
</content>

Like your main.html, you can have defaults for the blocks here, such as a default title.

Why XML? I’d originally envisioned using JSON, but it has escaping rules, of course. Django, so far as I’m aware, will always drop the contents of a block into place verbatim, without an opportunity to escape it into a JSON string. That’s where XML’s CDATA construct came in handy, allowing a segment of HTML to be put right in place. Assuming, of course, “]]>” doesn’t appear in your HTML.

JavaScript Page Switches

That takes care of the back-end Django bit. The last bit involves the front end JavaScript that takes care of the page switch/content replacement. This example leans on jQuery fairly heavily. In essence we’ll: A) take over the link’s click event, B) send off an AJAX-type request for the same href, C) set up a callback to do the actual content switch. Or, to put it another way:

$('a.ajax').click(function (e) {
  // Suppress the default navigate event
  e.preventDefault();
  // Instead, do the GET in the background
  $.get(this.href).done(function (response) {
    // The XML is automatically parsed and can be traversed in script
    var contentNodes = response.firstChild.childNodes;
    for (var i = 0; i < contentNodes.length; i++) {
      // Ignore any textNodes or other non-elements
      if (contentNodes[i].nodeType != 1) continue;

      // Handle each XML element appropriately:
      if (contentNodes[i].nodeName == 'title')
        document.title = contentNodes[i].firstChild.nodeValue;
      if (contentNodes[i].nodeName == 'main_content')
        $('#main_content').html(contentNodes[i].firstChild.nodeValue);
    }
  });
});

JavaScript, I’ll admit, isn’t a language I work in all that often. There’s probably a better way of parsing and handling that, but that seemed to work okay in testing. And, of course, it’s fairly bare bones as far as functionality. But it shouldn’t be difficult to add in a spinner, error handling, etc.

Anyway, it was fun. Even the slightly frustrating update-JS/refresh/try-again cycle. Again it’s still fairly rough, and quite untested at any scale. But maybe the idea will help someone out there.

Getting the Django Admin to sort modified columns

2013-09-28T00:00:00+00:00

A lot of what I’ve worked on at End Point is Ovis, a program to keep track of information about our servers. This information includes current operating system, data center, which client owns or uses it, etc. Ovis is built entirely on the Django Admin, with the most important information displayed on the Servers list page.

A very important part of it is tracking server updates.

Knowing when and by who a server was last updated is nice to see, but to put it to good use, we needed to have a column that would show when it was last updated and who last updated it. We also wanted this column to have links to the relevant pages.

Django has a handy way of using functions to create a column based on data that is not part of the model being listed. The documentation for that is here. This is the code we are using:

def latest_update(self):
    try:
        object = self.update_set.latest()
        if object:
            return "<a href="%s">%s</a> by <a href="%s">%s</a>" % (
                '/admin/ovisapp/update/%s' % object.id,
                object.when_updated.date(),
                '/admin/auth/user/%s' % object.updated_by.id,
                "%s %s." % (object.updated_by.first_name, object.updated_by.last_name[0])
            )
    except:
        return ""

It ends up looking like this:

While this is very useful, it has one very big problem. You are unable to sort your list of objects by this new column. This is because with other columns, it sorts with an easy ORDER BY in the database. But when you are grabbing data from another table, adding links and extra space, words, etc., Django doesn’t know what to sort by.

The main part of the sorting is creating a custom Manager to your model, and annotating the QuerySet with aggregate functions. Like this, which gets the newest update object for each server object:

class ServerManager(Manager):
    def get_query_set(self):
        qs = super(ServerManager, self).get_query_set().annotate(Max('update__when_updated'))
        return qs

Set it as your model’s manager by adding this line to the model definition:

objects = ServerManager()

Now, to get the column to sort by the aggregate function, set the admin_order_field method attribute to point to your new annotated aggregate function, like this:

def latest_update(self):
...
latest_update.admin_order_field = 'update__when_updated__max'

And it will sort by the date!

Selenium Testing File Uploads in Django Admin

2013-05-15T00:00:00+00:00

The Django framework version 1.4 added much better integration with Selenium for in-browser functional testing. This made Test-Driven Development an even more obvious decision for our new Liquid Galaxy Content Management System. This went very well until we needed to test file uploads in the Django admin interface.

A browser’s file upload control has some unique security concerns that prevent JavaScript from setting its value. Trying to do so may raise INVALID_STATE_ERR: DOM Exception 11. Selenium’s WebDriver may sometimes send keystrokes directly into the input element, but this did not work for me within Django’s admin interface.

To work around this limitation, Ryan Kelly developed a Middleware to emulate successful file uploads for automated testing. This middleware inserts additional hidden fields into any forms sent to the client. Setting their value causes a file upload to happen locally on the server. (I used a slightly newer version of this Middleware from another project.)

However, Selenium intentionally will not interact with hidden elements. To work around this, we must send JavaScript to be executed directly in the browser using WebDriver’s execute_script method.

self.browser.execute_script("document.getElementsByName('fakefile_storage')[0].value='placemark_end_point.kml'")

This is a lot of hoops to jump through, but now we have functional tests for file uploads and their post-upload processing. Hopefully the Selenium or Django projects can develop a better-supported method for file upload testing.

Making SSL Work with Django Behind an Apache Reverse Proxy

2013-04-09T00:00:00+00:00

Bouncing Admin Logins

We have a Django application that runs on Gunicorn behind an Apache reverse proxy server. I was asked to look into a strange issue with it: After a successful login to the admin interface, the browser was re-directed to the http (non-SSL) version of the interface.

After some googling and investigation I determined the issue was likely due to our specific server arrangement. Although the login requests were made over https, the requests proxied by Apache to Gunicorn used http (securely on the same host). Checking the Apache SSL error logs quickly affirmed this suspicion. I described the issue in the #django channel on freenode IRC and received some assistance from Django core developer Carl Meyer. As of Django 1.4 there was a new setting Carl had developed to handle this particular scenario.

Enter SECURE_PROXY_SSL_HEADER

The documentation for the SECURE_PROXY_SSL_HEADER variable describes how to configure it for your project. I added the following to the settings.py config file:

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

Because this setting tells Django to trust the X-Forwarded-Proto header coming from the proxy (Apache) there are security concerns which must be addressed. The details are described in the Django documentation and this is the Apache configuration I ended up with:

# strip the X-Forwarded-Proto header from incoming requests
RequestHeader unset X-Forwarded-Proto

# set the header for requests using HTTPS
RequestHeader set X-Forwarded-Proto https env=HTTPS

With SECURITY_PROXY_SSL_HEADER in place and the Apache configuration updated, logins to the admin site began to work correctly.

This is standard practice for web applications that reside behind an HTTP reverse proxy, but if the application was initially set up using only plain HTTP, when HTTPS is later added, it can be easy to be confused and overlook this part of the setup.

RailsAdmin & Django

2012-06-14T00:00:00+00:00

Steph Skardal and Greg Davidson presented on RailsAdmin and Django (in the context of two of our clients, Musica Russica and Providence Plan).

Clients need a browser interface to administrate their Rails apps’ assets and configuration. RailsAdmin is an “engine” (an embedded miniature Rails app) for developing an admin interface, and a relatively young open-source project. It offers a CRUD-capable interface. It provides data export, filtering, pagination, and support for file attachments and a popular plug-in called “PaperTrail”. Musica Russica uses this as their site administration. This admin interface can be configured with many different customizable options: formatting, help text, sorting, etc.

RailsAdmin also provides authorization to limit certain actions (“delete”, “read”, “manage”) to objects (“User”, “Order”) by user.

In contrast, Django, written in Python, dates back to 2005 and is aimed at complex web applications. It provides an automatic admin interface which of course you can extend and customize, and mostly the same feature set (pagination, filtering, etc). Several high-profile web applications (Instagram, Pinterest, and Mozilla’s add-ons site) are powered by Django. Our own work here is represented by the client site for Providence Plan.

Django’s admin interface is (mostly) free of effort, once you have developed models for your application. In addition, there are many open-source apps that can be plugged into a project to manage specific things, such as permissions. Filters and searching can be configured, too: this helps limit the data that a user must consider during an administration operation.

Debugging Celery Tasks in Django Projects

2012-03-27T00:00:00+00:00

I recently had the opportunity to work on a Django project that was using Celery with RabbitMQ to handle long-running server-side processing tasks. Some of the tasks took several hours to complete. The tasks had originally been executed with the at command and others had been managed with cron jobs. The client had started to migrate several of the tasks to use Celery when I joined the project.

As I started to debug the Celery tasks, I quickly discovered there were many moving parts involved and it was not immediately clear which piece of the system was the cause of the problem. I was working in a local development environment that closely matched the production system. The dev environment consisted of a virtualenv created using the same method Szymon Guz wrote about in his article Django and virtualenvwrapper. Django, Celery and their related dependencies were installed with pip and I installed RabbitMQ with Homebrew. After activating my virtualenv and starting up everything up (or so I thought), I jumped in to an IPython shell and began to debug the tasks interactively. Some tasks completed successfully but they finished almost instantaneously which didn’t seem right. The client had experienced the same issue when they excecuted tasks on their development server.

Because I was joining an existing project in progress, the system administration and configuration had already been taken care of by other members of the team. Howerver, in the process of configuring my local development server to mimic the production systems, I learned a few things along the way, described below.

RabbitMQ

RabbitMQ is a message broker; at its most basic it sends and receives messages between sender (publisher) and receiver (consumer) applications. It’s written in Erlang which helps to make it highly parallel and reliable. The RabbitMQ web site is a great place to learn more about the project. For my purposes I needed to create a user and virtual host (vhost) and set up permissions for Celery to communicate with the RabbitMQ server. This was done with the rabbitmqctl command. I issued the following command to start up the server and let the process run in the background.

rabbitmq-server -detached

I also enabled the management plugin which provides both a web-based UI and a command line interface for managing and monitoring RabbitMQ. This is what the web-based UI looks like:

django-celery

Celery works very well with Django thanks in large part to the django-celery module. The django-celery module includes the djcelery app which can be plugged in to the Django admin site for your project. Connecting Django to Celery and RabbitMQ requires a few simple steps:

Add djcelery to the list of INSTALLED_APPS in the settings.py file for the project.
Add the following lines to settings.py:

import djcelery
djcelery.setup_loader()

Create the celery database tables using the syncdb management command
Configure the broker setttings in settings.py:

BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "celery_user"
BROKER_PASSWORD = "celery_password"
BROKER_VHOST = "celery"

Celery

With the RabbitMQ server up and running and Django configured to connect to Celery the last few steps involved starting up the Celery worker and its related monitoring apps. The Celery daemon (celeryd) has lots of options that you can check out by running the following command:

python manage.py celeryd --help

For my purposes, I wanted Celery to broadcast events which the various monitoring applications could then subscribe to. It would also be good to print some helpful debugging info to the logs. I started up the Celery worker daemon with the following command:

python manage.py celeryd -E -loglevel=DEBUG

Because I specified the -E flag, the celeryev application could be used to monitor and manage the Celery worker inside a terminal which was very helpful:

For Django to capture and save Celery task information to the database, the celerycam application needs to be running. This command line app takes a snapshot of Celery every few seconds or at an interval you specify on the command line:

python manage.py celerycam

With celerycam running, the Django admin interface is updated as Celery tasks are executed:

You can also view the detail for a particular task including any error messages from the task code:

With RabbitMQ, celeryd and celerycam ready to go, the Django development server could be started to begin testing and debugging Celery task code. To demonstrate this workflow in action, I wrote a simple Celery task that could be used to simulate how Django, Celery and RabbitMQ all work together.

from celery.decorators import task
import time
import random

# A simple task to demonstrate Celery & djcelery
@task()
def add(x, y):
    delay = random.randint(1,60)
    time.sleep(delay)
    return x + y

Tying it all Together

With everything configured, I was ready to get to work debugging some Celery tasks. I set up a dashboard of sorts in tmux to keep an eye on everything as I worked on the code for a particular task:

Clockwise from the bottom left you’ll see an IPython shell to debug the code interactively, the Django development server log, celeryev, Celery daemon (with debugging info) and the task code in Vim.

When I started developing task-related code I wasn’t sure why my changes were not showing up in the Celery or Djcelery logs. Although I had made some changes, the same errors persisted. When I looked into this further I found that Celery caches the code used for a particular task and re-uses it the next time said task is executed. In order for my new changes to take effect I needed to restart the Celery daemon. As of Celery version 2.5 there is an option to have Celery autoreload tasks. However, the version of Celery used in this client project did not yet support this feature. If you find yourself working with Django, Celery and RabbitMQ, I hope you’ll find this helpful.

Downloading CSV file from Django admin

2012-02-22T00:00:00+00:00

Django has a very nice admin panel. The admin panel is highly extensible and there can be performed really cool enhancements. One of such things is a custom action.

For the purpose of this this article I’ve created a simple Django project with a simple application containing only one model. The file models.py looks like this:

from django.db import models
from django.contrib import admin

class Stat(models.Model):
    code = models.CharField(max_length=100)
    country = models.CharField(max_length=100)
    ip = models.CharField(max_length=100)
    url = models.CharField(max_length=100)
    count = models.IntegerField()

class StatAdmin(admin.ModelAdmin):
    list_display = ('code', 'country', 'ip', 'url', 'count')

admin.site.register(Stat, StatAdmin)

I’ve also added a couple of rows in the database table for this model. The admin site for this model looks like this:

Now I want to be able to select some rows and download a CSV file right from the Django admin panel. The file should contain only the information about selected rows.

This can be done really easy with the admin actions mechanism. Over the table with rows there is the actions menu. There is one default action, it is “Delete selected stats”. To use the action you need to select the rows, select the action from the combo box and press the OK button.

I will add another action there, which will be named “Download CSV file for selected stats”.

Add the action.

First of all I will add the custom action. For this I will enhance the StatAdmin class with the field actions, and add the method, called when someone wants to run this action (all changes from the previous version are highlighted):

class StatAdmin(admin.ModelAdmin):
    actions = ['download_csv']
    list_display = ('code', 'country', 'ip', 'url', 'count')
    def download_csv(self, request, queryset):
        None
    download_csv.short_description = "Download CSV file for selected stats."

In the admin panel for the stats model you can notice that there is the new action. The above code doesn’t do anything useful, so let’s generate the CSV file.

The whole idea is to generate the CSV file, don’t use any disk, do it in memory only and return to the user without any redirection to other page (the file should download automatically after pushing the OK button).

Do it in small steps:

Generate the CSV

For this I will use the CSV module from Python’s standard library and the function now looks like this:

def download_csv(self, request, queryset):
    import csv
    f = open('some.csv', 'wb')
    writer = csv.writer(f)
    writer.writerow(["code", "country", "ip", "url", "count"])
    for s in queryset:
        writer.writerow([s.code, s.country, s.ip, s.url, s.count])

After selecting two rows and running the action, it created a file some.csv in the main project directory with the following content.

code,country,ip,url,count
B,BB,BBB,BBBB,22
C,CC,CCC,CCCC,33

That’s OK, generating the CSV file works, however it shouldn’t be stored on the disk.

Return the file directly into the browser.

I want to send the file to the client right after clicking on the OK button. This is fairly easy, the whole magic is to use proper HTTP headers. I will modify the method to look like this:

def download_csv(self, request, queryset):
    import csv
    from django.http import HttpResponse

    f = open('some.csv', 'wb')
    writer = csv.writer(f)
    writer.writerow(["code", "country", "ip", "url", "count"])

    for s in queryset:
        writer.writerow([s.code, s.country, s.ip, s.url, s.count])

    f.close()

    f = open('some.csv', 'r')
    response = HttpResponse(f, content_type='text/csv')
    response['Content-Disposition'] = 'attachment; filename=stat-info.csv'
    return response

So the main changes are:

closed the file and reopen it for reading
added headers for proper content type and file name.

The result is that when someone clicks on the OK button, the browser automatically starts downloading the stat-info.csv file.

Don’t use disk.

The only thing left: create the file in memory only. For this I will use StringIO module. It is a nice module implementing exactly the same interface as the file, so I can use it instead the file. StringIO operates only on memory without any disk operations.

def download_csv(self, request, queryset):
    import csv
    from django.http import HttpResponse
    import StringIO

    f = StringIO.StringIO()
    writer = csv.writer(f)
    writer.writerow(["code", "country", "ip", "url", "count"])

    for s in queryset:
        writer.writerow([s.code, s.country, s.ip, s.url, s.count])

    f.seek(0)
    response = HttpResponse(f, content_type='text/csv')
    response['Content-Disposition'] = 'attachment; filename=stat-info.csv'
    return response

As you can see the changes are:

added import StringIO
changed opening file to creating new StringIO object
there is no reopening the file, only seek to set the marker at the beginning of the file

Everything is finished now. There is a new action in the admin panel which generates a new CSV file with information about chosen rows and it doesn’t do any browser redirection.

Django and Virtual Environments

2012-02-13T00:00:00+00:00

When you have to work with a bunch of different Python applications, the usual problem is that you have to deal with plenty of different packages in different versions. Each application needs its own set of libraries. Usually the versions of the libraries vary between the applications.

To solve all the problems you could create Python virtual environments. There is a great tool: virtualenv. It can create virtual environments for Python. Using it is not too nice. However there is a wrapper to it, called virtualenvwrapper. It wraps all the virtualenv commands into a couple of shell commands.

Let’s assume that I need to work on two applications written in Django 1.2 and Django 1.3. Each of the applications needs different set of packages in different versions. I will create two virtual environments.

Installing virtualenvwrapper on Ubuntu is pretty easy:

$ sudo apt-get install virtualenvwrapper

After the installation there is a couple of new commands. The basic one is: mkvirtualenv which creates a new environment. Let’s create one.

$ mkvirtualenv django_demo_12

This command automatically switches to the new environment, so you might notice that the prompt changed. The prompt always starts with the name of current virtual environment.

Let’s create another one, called django_demo_13 (to use Django 1.3 there).

(django_demo_12)$ mkvirtualenv django_demo_13

The list of environments is printed by the command workon, when called without arguments.

$ workon
django_demo_12
django_demo_13

As you can see, there are two environments ready to use. You can pass the name of the virtual environment as parameter to the workon command. Now let’s install Django 1.2 on the environment django_demo_12.

First of all switch to the new environment:

$ workon django_demo_12

Now the prompt changed, so you can always be sure which Python virtual environment you are using.

(django_demo_12)$

Now Django should be installed. There is a couple of ways to install it. The one I prefer is to create a text file with names and versions of all needed packages. This file will be named requirements.txt and will contain only this one line so far (other packages will be added later):

Django==1.2.7

To install the packages listed in the file, I will use the command “pip install -r requirements.txt”:

(django_demo_12)$ pip install -r requirements.txt
Downloading/unpacking Django==1.2.7 (from -r requirements.txt (line 1))
Downloading Django-1.2.7.tar.gz (6.4Mb): 6.4Mb downloaded
Running setup.py egg_info for package Django

Installing collected packages: Django
Running setup.py install for Django
changing mode of build/scripts-2.7/django-admin.py from 664 to 775

changing mode of /home/szymon/.virtualenvs/django_demo_12/bin/django-admin.py to 775
Successfully installed Django
Cleaning up...

Now I can check which Django version is installed:

(django_demo_12)$ django-admin.py --version
1.2.7

Now I will create a standard Django project:

(django_demo_12)$ django-admin.py startproject django_demo_12

The only additional thing here is to move the requirements.txt file info the Django project:

(django_demo_12)$ mv requirements.txt django_demo_12

To create application using Django 1.3 the steps are similar. The first thing is to switch to another virtual environment:

(django_demo_12)$ workon django_demo_13

From this moment it will be almost the same as in the previous environment, with the change that the requirements file should contain:

Django==1.3.1

The commands are:

(django_demo_13)$ pip install -r requirements.txt
(django_demo_13)$ django-admin.py startproject django_demo_13
(django_demo_13)$ mv requirements.txt django_demo_13

So now there are two different Python environments, totally separated from each other. When I install something in one of them, it is not installed in the other, so I can have different packages for different Django versions.

The best way to install a package here is to update the requirements.txt file, and run the “pip install -r requirements.txt” once again. Later it will be easier to give the whole code to another programmer, who then could run the command on his computer and it will automatically install all needed packages (each in exactly needed version).

There is one simple command left. Sometimes you just want to remove the virtual environment from the path and use standard python libraries installed in the system. It can be done using this command:

(django_demo_13)$ deactivate

Red Hat SELinux policy for mod_wsgi

2010-10-13T00:00:00+00:00

Using SELinux, you can safely grant a process only the permissions it needs to perform its function, and no more. Linux distributions provide policies to enforce these limits on most software they package, but many aren’t covered. We’ve made allowances for mod_wsgi on RHEL and CentOS 5 by extending Apache httpd’s SELinux policy.

It seems the SELinux policy for Apache httpd is twice as large as any other package’s. The folks at Red Hat have put a lot of work into making sure that attackers who manage to exploit httpd can’t break out to the rest of your system, while still allowing the flexibility to serve most applications. Consult the httpd_selinux man page if messages in audit.log coincide with your error.

File Contexts

If you’ve created files and/or directories in /etc/httpd, make sure they have the proper file contexts so the daemon can read them:

  # restorecon -vR /etc/httpd

httpd can only serve files with an explicitly allowed file context. Configure the context of files and directories within your production code base using the semanage command:

  # semanage fcontext --add --ftype -- --type httpd_sys_content_t "/home/projectname/live(/.*)?"
  # semanage fcontext --add --ftype -d --type httpd_sys_content_t "/home/projectname/live(/.*)?"
  # restorecon -vR /home/projectname/live

View file contexts with ls -Z. Changes should be generally accomplished with semanage and restorecon -vR.

Booleans

The httpd policy provides several boolean options for easy run-time configuration:

httpd_can_network_connect — Allows httpd to make network connections, including the local ones you’ll be making to a database
httpd_enable_homedirs — Allows httpd to access /home/

Booleans are persistently set using the setsebool command with the -P flag:

  # setsebool -P httpd_can_network_connect on

WSGI Socket

When running in daemon mode, httpd and the mod_wsgi daemon communicate via a UNIX socket file. This should usually have a context of httpd_var_run_t. The standard Red Hat SELinux policy includes an entry for /var/run/wsgi.* to use this context, so it makes sense to put the socket there using the WSGISocketPrefix directive within your httpd configuration:

  WSGISocketPrefix run/wsgi

(Note that run/wsgi translates to /etc/httpd/run/wsgi which is symlinked to /var/run/wsgi.)

If socket communication fails, httpd returns a 503 “Temporarily Unavailable” error response.

SELinux Policy Module

In the course of our testing SELinux denials like the following appeared:

  host=example.com type=AVC msg=audit(1262803154.315:1851): avc:  denied  { execmem } for  pid=5337 comm="httpd" scontext=root:system_r:httpd_t:s0 tcontext=root:system_r:httpd_t:s0 tclass=process

Unusual behavior like this is usually best allowed by creating application-specific SELinux policy modules. If you cannot resolve these AVC errors by manipulating file contexts or booleans, collect all the errors into a single file and feed that into the audit2allow utility:

  # yum install policycoreutils
  # mkdir ~/tmp  # if this doesn't exist already
  # audit2allow --module wsgi < ~/tmp/pile_of_auditd_output > ~/tmp/wsgi.te

This will output source for a new policy module. You might review the .te file before compiling. Ours looks like this:

module wsgi 1.0;

require {
      type httpd_t;
      class process execmem;
}

#============= httpd_t ==============
allow httpd_t self:process execmem;

Compile this source into a new policy module and package it:

  # checkmodule -M -m -o ~/tmp/wsgi.mod ~/tmp/wsgi.te
  # semodule_package --outfile ~/tmp/wsgi.pp --module ~/tmp/wsgi.mod

Once created, the module may be installed permanently into any compatible system’s SELinux configuration:

  # semodule --install ~/tmp/wsgi.pp

There’s plenty of room for improvement here. The file contexts we assigned with semanage should be defined in a .fc source file and included within the policy module. And creating a new context just for the WSGI daemon to transition into would restrict it further, allowing only a subset of Apache httpd’s abilities. Writing your own policy like this allows you much finer tuning of your processes’ limits, while allowing their needed functionality.

Safari 4 Top Sites feature skews analytics

2010-02-13T00:00:00+00:00

Safari version 4 has a new “Top Sites” feature that shows thumbnail images of the sites the user most frequently visits (or, until enough history is collected, just generally popular sites).

Martin Sutherland describes this feature in details and shows how to detect these requests, which set the X-Purpose HTTP header to “preview”.

The reason this matters is that Safari uses its normal browsing engine to fetch not just the HTML, but all embedded JavaScript and images, and runs in-page client JavaScript code. And these preview thumbnails are refreshed fairly frequently—possibly several times per day per user.

Thus every preview request looks just like a regular user visit, and this skews analytics which see a much higher than average number of views from Safari 4 users, with lower time-on-site averages and higher bounce rates since no subsequent visits are registered (at least as part of the preview function).

The solution is to simply not output any analytics code when the X-Purpose header is set to “preview”. In Interchange this is easily done if you have an include file for your analytics code, by wrapping the file with an [if] block such as this:

[tmp x_purpose][env HTTP_X_PURPOSE][/tmp]
[if scratch x_purpose eq 'preview']
<!-- skip analytics for browser previews -->
[else]
(normal Google Analytics, Omniture SiteCatalyst, or other analytics code)
[/else]
[/if]

In Ruby on Rails you’d check request.env[“HTTP_X_PURPOSE”].

In PHP you’d check $_SERVER[“HTTP_X_PURPOSE”].

In Django you’d check request.META[“HTTP_X_PURPOSE”] or the equivalent request.META.get(“HTTP_X_PURPOSE”) (from the HttpRequest class).

And so on.

I confirmed the analytics tracking code was omitted by waiting for Safari to make its preview request and inspecting the response with the Fiddler proxy, on Windows. The same can be done for Safari on Mac OS X with a suitable Mac OS X HTTP proxy.

DjangoCon 2009: Portland, Ponies, and Presentations

2009-11-04T00:00:00+00:00

I attended DjangoCon this year for the first time, and found it very informative and enjoyable. I hoped to round out my knowledge of the state of the Django art, and the conference atmosphere made that easy to do.

Presentations

Avi Bryant’s opening keynote was on the state of web application development, and what Django must do to remain relevant. In the past, web application frameworks did things in certain ways due to the constraints of CGI. Now they’re structured around relational databases. In the future, they’ll be arranged around Ajax and other asynchronous patterns to deliver just content to browsers, not presentation. To wit, “HTML templates should die”, meaning we’ll see more Gmail-style browser applications where the HTML and CSS is the same for each user, and JavaScript fetches all content and provides all functionality. During Q&A, he clarified that most of what he said applies to web applications, not content-driven sites which must be SEO-friendly and so arranged much differently. Many of these themes were serendipitously also in Jacob Kaplan-Moss’ “Snakes on the Web” talk, which he gave at PyCon Argentina the same week as DjangoCon.

Ian Bicking’s keynote was on open-source as a philosophy; very abstract and philosophical but also interesting. It has been described as “a free software programmer’s midlife crisis”. Frank Wiles of Revolution Systems gave a barn-burner talk on how Django converted him from Perl to Python, followed by another on Postgres optimization. The latter reflected a theme that all web developers are now expected to do Operations as well, with several talks devoted to simple systems administration concepts.

Deployment

While working on Django projects we’ve been doing this year, I’ve been watching developments around deployment of Python web applications, particularly with Apache. The overwhelming consensus: Without active development, mod_python is going the way of the dodo. Although WSGI is architecturally similar to CGI, the performance difference can be striking. mod_wsgi’s daemon mode running as a separate user is more secure and flexible than mod_python processes running as the Apache user. Given mod_wsgi’s momentum, it makes sense to use it and avoid mod_python for new projects.

Several other tools kept re-appearing in presenters’ demonstrations. Fabric is a remote webapp deployment tool similar to Ruby’s Capistrano. Python’s package index, formerly named “The Cheeseshop”, has been renamed PyPI, the Python Package Index. Though easy_install is the standard tool to install PyPI packages, pip is gaining momentum as its successor. VirtualEnv is a tool to create isolated Python environments, discrete from the system environment. Since the conference I’ve been exploring how these tools may be leveraged for our own development, and may be integrated into our DevCamps multiple-environments system.

Pinax

The Eldarion folks gave three talks on Pinax, and the project came up a lot in conversations. If anything could be said to have “buzz” at the conference, this is it. Pinax is a suite of re-usable Django applications, encompassing functionality often-desired, but not common enough to be included in django.contrib. It may be compared to Drupal and Plone. Its popularity also spurred discussions on what should or should not be included in the Django core, and how all Django developers should make their apps re-usable (some of James Bennett’s favorite topics).

Community

Among others, I was fortunate to spend time with Kevin Fricovsky and the others who launched the new community site DjangoDose during the conference. DjangoDose is a spiritual successor to the now-defunct This Week in Django podcast, and was visible on many laptops in the conference room, aggregating #djangocon tweets.

That’s all I have time to relate now. There was plenty more there and I look forward to following up with people & projects.