A Rusty Web? An Excursion of a Perl Guy into Rust Land

2025-08-05T00:00:00+00:00

In my programming career centered around web applications I’ve always used dynamic, interpreted languages: Perl, JavaScript, Python, and Ruby. However, I’ve always been curious about compiled, strongly typed languages and if they can be useful to me and to my clients. Based on my recent findings, Rust would be my first choice. It’s a modern language, has excellent documentation and it’s quite popular. However, it’s very different from the languages I know.

I read most of the book a couple of years ago, but given that I didn’t do anything with it, my knowledge quickly evaporated. This time I read the book and immediately after that I started to work on a non-trivial project involving downloading XML data from different sources, database operations, indexing and searching documents, and finally serving JSON over HTTP. My goal was to replace at least part of a Django application which seemed to have performance problems. The Django application uses Xapian (which is written in C++) via its bindings to provide the core functionality. Indexing documents would be delegated to a Celery task queue.

Unfortunately Xapian does not have bindings for Rust so far.

My reasoning was: I could use the PostgreSQL full text search feature instead of Xapian, simplifying the setup (updating a row would trigger an index update, instead of delegating the operation to Celery).

After reading the Rust book I truly liked the language. Its main feature is that it (normally) gives you no room for nasty memory management bugs which plague languages like C. Being compiled to machine code, it’s faster than interpreted languages by an order of magnitude. However, having to state the type of variables, arguments, and return values was at first kind of a culture shock, but I got used to it.

When writing Perl, I’m used to constructs like these:

if (my $res = download_url($url)) {
    ...
}

which are not possible any more. Instead you have to use the match construct and extract values from Option (Some/None) and Result (Ok, Err) enumerations. This is the standard way to handle errors and variables which may or may not have values. There is nothing like an undef and this is one of the main Rust features. Instead, you need to cover all the cases with something like this:

match download_url(url.clone()) {
    Ok(res) => {
       ...
    },
    Err(e) => println!("Error {url}: {e}"),
}

Which can also be written as:

if let Ok(res) = download_url(url.clone()) {
    ...
}

You must be consistent with the values you are declaring and returning, and take care of the mutability and the borrowing of the values. In Rust you can’t have a piece of memory which can be modified in multiple places: for example, once you pass a string or a data structure to a function, you can’t use it any more. This is without a doubt a good thing. When in Perl you pass a reference of hash to a function, you don’t know what happens to it. Things can be modified as a side effect, and you are going to realize later at debugging time why that piece of data is not what you expect.

In Rust land, everything feels under control, and the compiler throws errors at you which most of the time make sense. It explains to you why you can’t use that variable at that point, and even suggests a fix. It’s amazing the amount of work behind the language and its ability to analyze the code.

The string management feels a bit weird because it’s normally anchored to the UTF-8 encoding, while e.g. Perl has an abstract way to handle it, so I’m used to thinking differently about it.

The async feature is nice, but present in most of the modern languages (Perl included!), so I don’t think that should be considered the main reason to use Rust.

Bottom line: I like the language. It’s very different to what I was used to, but I can see many advantages. The downside is that you can’t write all those “quick and dirty” scripts which are the daily bread of the sysadmin. It lacks that practical, informal approach I’m used to.

Once I got acquainted with the language, I went shopping for “crates” (which is what modules are called in Rust) here: https://www.arewewebyet.org/.

Lately I have a bit of a dislike for object–relational mappings (ORM), so I didn’t go with diesel nor sqlx, but I went straight for tokio_postgres.

This saved me quite a bit of documentation reading and gave me direct access to the database. Nothing weird to report here. It feels like using any other DB driver in any other language, with a statement, the placeholders and the arguments. The difference, of course, is that you need to care about the data types which are coming out of the DB (again the Option Enum is your friend and the error messages are helpful).

To get data from the Internet, reqwest did the trick just fine without any surprise.

For XML deserialization, serde was paired with quick-xml. This is one of the interesting bits.

You start defining your data structures like this:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct OaiPmhResponse {
    #[serde(rename = "responseDate")]
    response_date: String,
    request: String,
    error: Option<ResponseError>,
    #[serde(rename = "ListRecords")]
    list_records: Option<ListRecords>,
}
// more definitions follow, to match the structure we expect

Then you feed the XML string to the from_str function like this:

use quick_xml::de::from_str;

fn parse_response (xml: &str) -> OaiPmhResponse {
    match from_str(xml) {
        Ok(res) => res,
        // return a dummy one with no records in it in case of errors
        Err(e) => OaiPmhResponse {
            response_date: String::from("NOW"),
            request: String::from("Invalid"),
            error: Some(ResponseError {
                code: String::from("Invalid XML"),
                message: e.to_string(),
            }),
            list_records: None,
        },
    }
}

which takes care of the parsing and gives you back either an Ok with the data structure you defined inside and the tags properly mapped, or an error. The structs can have methods attached so they provide a nice OOP-like encapsulation.

Once the data collection was successful, I moved to the web application itself.

I chose the Axum framework, maintained by the Tokio project and glued all the pieces together.

The core of the application is something like this:

#[derive(Serialize, Debug)]
struct Entry {
    entry_id: i32,
    rank: f32,
    title: String,
}

async fn search(
    State(pool): State<ConnectionPool>,
    Query(params): Query<HashMap<String, String>>,
) -> (StatusCode, Json<Vec::<Entry>>) {
    let conn = pool.get().await.expect("Failed to get a connection from the pool");
    let sql = r#"
SELECT entry_id, title, ts_rank_cd(search_vector, query) AS rank
FROM entry, websearch_to_tsquery($1) query
WHERE search_vector @@ query
ORDER BY rank DESC
LIMIT 10;
"#;
    let query = match params.get("query") {
        Some(value) => value,
        None => "",
    };
    let out = conn.query(sql, &[&query]).await.expect("Query should be valid")
        .iter().map(|row|
                    Entry {
                        entry_id: row.get(0),
                        title: row.get(1),
                        rank: row.get(2),
                    }).collect();
    tracing::debug!("{:?}", &out);
    (StatusCode::OK, Json(out))
}

Which simply runs the query using the input provided by the user, runs the full text search, and returns the serialized data as JSON.

During development it felt fast. The disappointment came when I populated the database with about 30,000 documents of various sizes. The Django application, despite returning more data and the facets, was still way faster. With the two applications running on the same (slow) machine I got a response in 925 milliseconds from the Rust application, and in 123 milliseconds for the Django one!

Now, most of the time is spent in the SQL query, so the race here is not Python vs. Rust, but Xapian vs. PostgreSQL’s full text search, with Xapian (Python is just providing an interface to the fast C++ code) winning by a large measure. Even if the Axum application is as fast as it can get, because it’s stripped to the bare minimum (it has no sessions, no authorization, no templates), the time saved is not enough to compensate for the lack of a dedicated and optimized full text search engine like Xapian. Of course I shouldn’t be too surprised.

To actually compete with Django + Xapian, I should probably use Tantivy, instead of relying on the PostgreSQL full text search. But that would be another adventure…

The initial plan turned out to be a failure, but this was really a nice and constructive excursion, as I could learn a new language, using its libraries to do common and useful tasks like downloading data, building small web applications, and interfacing with the database. Rust appears to have plenty of quality crates.

Beside the fact that this was just an excuse to study a new language, it remains true that rewriting existing, working applications is extremely unrewarding and most likely ineffective. Reaching parity with the current features requires a lot of time (and budget), and at the end of the story the gain could be minimal and better achieved with optimization (here I think about all our clients running Interchange).

However, if there is a need for a microservice doing a small task where speed is critical and where the application overhead should be minimal, Rust would be a viable option.

Handling text encoding in Perl

2025-04-29T00:00:00+00:00

When we are dealing with legacy applications, it’s very possible that the code we are looking at does not deal with Unicode characters, instead assuming all text is ASCII. This will cause a myriad of glitches and visual errors.

In 2025, after more than 30 years since Unicode was born, how is that possible that old applications still survive while ignoring or working around the whole issue?

Well, if your audience is mainly English speaking, it’s possible that you just experience glitches sometimes, with some characters like typographical quotes, non breaking spaces, etc. which are not really mission-critical. If, on the contrary, you need to deal every day with diacritics or even different languages (say, Italian and Slovenian), your application simply won’t survive without a good understanding of encoding.

In this article we are going to focus on Perl, but other languages face the same problems.

Back to the bytes

As we know, machines work with numbers and bytes. A string of text is made of bytes, and each of them is 8 bits (each bit is a 0 or a 1). So one byte allows 256 possible combinations of bits.

Plain ASCII is made by 128 characters (7 bits), so it fits nicely in one byte, leaving room for more. One character is exactly one byte, and one byte carries a character.

However, ASCII is not enough for most of languages, even if they use the Latin alphabet, because they use diacritics like é, à, č, and ž.

To address this problem, the ISO 8859 encoding standards appeared (there are others, like the Windows code pages, using the same idea, but of course using different code points). These standards use 8th bit not used by ASCII, still using a single byte for each character but double the combinations from ASCII, allowing 256 possible characters. That’s better, but still not great. It suffices for handling text in a couple of languages if they share the same characters, but not more. For this reason, there are various ISO 8859 encoding standards (8859-1, 8859-2, etc.) — one for each group of related languages (e.g. 8859-1 is for Western Europe, 8859-2 for Central Europe and so on, and even revisions of the same encoding, like 8859-15 and 8859-16).

The problem is that if you have a random string, you have to guess which is the correct encoding. The same byte value could represent an “È” or a “Č”. You need to look at the context (which language is this?) or search for an encoding declaration. Most important, you are simply not able to type È and Č in the same plain text document. If your company works in Italy using the 8859-15 encoding, it means you can’t even accept the correct name of a customer from Slovenia, a neighbour country, because the encoding simply doesn’t have a place for characters with a caron (like “č”) and you have to work around this real problem.

So finally came the Unicode age. This standard allows for more than a million characters, which should be enough. You can finally type English, Italian, Russian, Arabic, and emojis all in the same plain text. This is truly great, but it creates a complication for the programmer: the assumption that one byte is one character is not true anymore. The common encoding for Unicode is UTF-8, which is also backward compatible with ASCII. This means that if you have ASCII text, it is also valid UTF-8. Any other character which is not ASCII will instead take from two to three bytes and the programming language needs to be aware of this.

Into the language and back to the world

Text manipulation is a very common task. If you need to process a string, say “ÈČ”, like in this document, you should be able to tell that it is a string with two characters representing two letters. You want to be able to use regular expression on it, and so on.

Now, if we read it as a string of bytes, we get 4 of them and the newline, which is not what we want.

Let’s see an example:

#!/usr/bin/env perl

use strict;
use warnings;
use Data::Dumper::Concise;

# sample.txt contains ÈČ and a new line

{
    open my $fh, '<', 'sample.txt';
    while (my $l = <$fh>) {
        print $l;
        if ($l =~ m/\w\w/) {
            print "Found two characters\n"
        }
        print Dumper($l);
    }
    close $fh;
}

{
    open my $fh, '<:encoding(UTF-8)', 'sample.txt';
    while (my $l = <$fh>) {
        print $l;
        if ($l =~ m/\w\w/) {
            print "Found two characters\n"
        }
        print Dumper($l);
    }
    close $fh;
}

This is the output:

ÈČ
"\303\210\304\214\n"
Wide character in print at test.pl line 24, <$fh> line 1.
ÈČ
Found two characters
"\x{c8}\x{10c}\n"

In the first block the file is read verbatim, without any decoding. The regular expression doesn’t work, we have basically 4 bytes which don’t seem to mean much.

In the second block we decoded the input, converting it in the Perl internal representation. Now we can use regular expressions and have a consistent approach to text manipulation.

In the first block, we got a warning:

Wide character in print at test.pl line 25, <$fh> line 1

That’s because we printed something to the screen, but given that the string is now made by characters (decoded for internal use), Perl warns us that we need to encode it back to bytes (for the outside world to consume). A wide character is basically a character which needs to be encoded.

This can either be done by calling the encode() method from the Encode API:

use strict;
use warnings;
use Encode;
print encode("UTF-8", "\x{c8}\x{10c}\n");

Or, better, by declaring the global encoding for the standard output:

use strict;
use warnings;
binmode STDOUT, ":encoding(UTF-8)";
print "\x{c8}\x{10c}\n";

So, the golden rule is:

decode the string on input and get characters out of bytes
work with it in your program as a string of characters
encode the string on output

Any other approach is going to lead to double encoded characters (seeing things like Ã and Ä in English text is a clear symptom of this), corrupted text, and confusion.

Encoding strategies

If you are dealing with standard input/output on the shell, you should have this in your script:

binmode STDIN,  ":encoding(UTF-8)";
binmode STDOUT, ":encoding(UTF-8)";
binmode STDERR, ":encoding(UTF-8)";

So you’re decoding on input and encoding on output automatically.

For files, you can add the layer in the second argument of open like in the sample script above, or use a handy module like Path::Tiny, which provides methods like slurp_utf8 and spew_utf8 to read and write files using the correct encoding.

Interactions with web frameworks should always happen with the internal Perl representation. When you receive the input from a form, it should be considered already decoded. It’s also the framework’s responsibility to handle the encoding on output. Here at End Point we have many Interchange applications. Interchange can support this, via the MV_UTF8 variable.

The same rules apply to databases. It’s responsibility of the driver to take your strings and encode/decode them when talking to the database. E.g. DBD::Pg has the pg_enable_utf8 option, while DBD::mysql has mysql_enable_utf8. These options should usually be turned on or off explicitly. Not specifying the option is usually source of confusion because of the heuristic approach it requires for understanding the code.

Debugging strategies

It may not be the most correct approach, but I’ve been using Dumper for more than a decade and it works. You simply use Data::Dumper or Data::Dumper::Concise and call Dumper on the string you want to examine.

If you see hexadecimal codepoints like \x{c8}\x{10c}, it means the string is decoded and you’re working with the characters. If you see the raw bytes or characters with diacritics (the latter would happing if the terminal is interpreting the bytes and showing you the characters), you’re dealing with an encoded string. If you see weird characters in an English context, it probably means the text has been encoded more than once.

Migrate a web application to Unicode

If you’re still using legacy encoding systems like ISO 8859 or the similar Windows code pages, or worse, you simply don’t know and you’re relying on the browsers’ heuristics (they’re quite good at guessing) you should change to handle the input and the output correctly along the whole application:

Convert the templates from the encoding you are using to UTF-8 (iconv should do the trick).
Inspect and possibly convert the existing DB data
Make sure the DB drivers handle the I/O correctly
Make sure the web framework is decoding the input and encoding the output
Make sure the files you read and write are correctly handled
Clean up any workarounds you may have had in place

This looks like a challenging task, and it can be, but it’s totally worth it because fancy and well-supported characters nowadays are the norm. Typographical quotes like “this” and ‘this’ are very common and inserted by word processors automatically. So are emojis. People and customers simply expect them to work.

Band-aids

If your client is on a budget or can’t deal with a large upgrade like this one, which has the potential to be disruptive and expose bugs which are lurking around, you can try to downgrade the Unicode characters to ASCII with tools like Text::Unidecode (which has been ported to other languages as well). So typographical quotes will become the plain ASCII ones, diacritics will be stripped, and various other characters will get their ASCII representation. Not great, but better than dealing with unexpected behavior!

Django and Mojolicious: a quick comparison of two popular web frameworks

2025-02-06T00:00:00+00:00

Recently I’ve been working on a project with a Vue front-end and two back-ends, one in Python using the Django framework and one in Perl using the Mojolicious framework. So, it’s a good time to spend some words to share the experience and do a quick comparison.

Previously I wrote a post about Perl web frameworks, and now I’m expanding the subject into another language.

Django was chosen for this project because it’s been around for almost 20 years now and provides the needed maturity and stability to be long-running and low-budget. In this regard, it has proved a good choice so far. Recently it saw a major version upgrade without any problems to speak of. It could be argued that I should have used the Django REST Framework instead of plain Django. However, at the time the decision was made, adding a framework on top of another seemed a bit excessive. I don’t have many regrets about this, though.

Mojolicious is an old acquaintance. It used to have fast-paced development but seems very mature now, and it’s even been ported to JavaScript.

Both frameworks have just a few dependencies (which is fairly normal in the Python world, but not in the Perl one) and excellent documentation. They both follow the model-view-controller pattern. Let’s examine the components.

Views

Both frameworks come with a built-in template system (which can be swapped out with something else), but in this project we can skip the topic altogether as both frameworks are used only as back-end for transmitting JSON, without any HTML rendering involved.

However, let’s see how the rendering looks for the API we’re writing.

use Mojo::Base 'Mojolicious::Controller', -signatures;
sub check ($self) {
    $self->render(json => { status => 'OK' });
}

from django.http import JsonResponse
def status(request):
    return JsonResponse({ "status":  "OK" })

Nothing complicated here, just provide the right call.

Models

Django

Usually a model in context of web development means a database and here we are going to keep this assumption.

Django comes with a comprehensive object-relational mapping (ORM) system and it feels like the natural thing to use. I don’t think it makes much sense to use another ORM, or even to use raw SQL queries (though it is possible).

You usually start a Django project by defining the model. The Django ORM gives you the tools to manage the migrations, providing abstraction from the SQL. You need to define the field types and the relationships (joins and foreign keys) using the appropriate class methods.

For example:

from django.db import models
class User(AbstractUser):
    email = models.EmailField(null=False, blank=False)
    site = models.ForeignKey(Site, on_delete=models.CASCADE, related_name="site_users")
    libraries = models.ManyToManyField(Library, related_name="affiliated_users")
    expiration = models.DateTimeField(null=True, blank=True)
    created = models.DateTimeField(auto_now_add=True)
    last_modified = models.DateTimeField(auto_now=True)

These calls provide not only the SQL type to use, but also the validation. For example, the blank parameter is a validation option specifying whether Django will accept an empty value. It is different from the null option, which directly correlates to SQL. You can see we’re quite far from working with SQL, at least two layers of abstraction away.

In the example above, we’re also defining a foreign key between a site and a user (many-to-one), so each user belongs to one site. We also define a many-to-many relationship with the libraries record. I like how these relationships are defined, it’s very concise.

Thanks to these definitions, you get a whole admin console almost for free, which your admin users are sure to like. However, I’m not sure this is a silver bullet for solving all problems. With large tables and relationships the admin pages load slowly and they could become unusable very quickly. Of course, you can tune that by filtering out what you need and what you don’t, but that means things are not as simple as “an admin dashboard for free” — at the very least, there’s some configuring to do.

As for the query syntax, you usually need to call Class.objects.filter(). As you would expect from an ORM, you can chain the calls and finally get objects out of that, representing a database row, which, in turn, you can update or delete.

The syntax for the filter() call is based on the double underscore separator, so you can query over the relationships like this:

for agent in (Agent.objects.filter(canonical_agent_id__isnull=False)
              .prefetch_related('canonical_agent')
              .order_by('canonical_agent__name', 'name')
              .all()):
    agent.name = "Dummy"
    agent.save()

In this case, provided that we defined the foreign keys and the attributes in the model, we can search/order across the relationship. The __isnull suffix, as you can imagine, results in a WHERE canonical_agent_id IS NOT NULL query, while in the order_by call we sort over the joined table using the name column. Looks nice and readable, with a touch of magic.

Of course things are never so simple, so you can build complex queries with the Q class combined with bytewise operators (&, |).

Here’s an example of a simple case-insensitive search for a name containing multiple words:

from django.db.models import Q

def api_list(request)
    term = request.GET.get('search')
    if term
        words = [ w for w in re.split(r'\W+', term) if w ]
        if words:
            query = Q(name__icontains=words.pop())
            while words:
                query = query & Q(name__icontains=words.pop())
            # logger.debug(query)
            agents = Agent.objects.filter(query).all()

To sum up, the ORM is providing everything you need to stay away from the SQL. In fact, it seems like Django doesn’t like you doing raw SQL queries.

Mojolicious and Perl

In the Perl world things are a bit different.

The Mojolicious tutorial doesn’t even mention the database. You can use any ORM or no ORM at all, if you prefer so. However, Mojolicious makes the DB handle available everywhere in the application.

You could use DBIx::Connector, DBIx::Class, Mojo::Pg (which was developed with Mojolicious), or whatever you prefer.

For example, to use Mojo::Pg in the main application class:

package MyApp;
use Mojo::Base 'Mojolicious', -signatures;
use Mojo::Pg;
use Data::Dumper::Concise;

sub startup ($self) {
    my $config = $self->plugin('NotYAMLConfig');
    $self->log->info("Starting up with " . Dumper($config));
    $self->helper(pg => sub {
                      state $pg = Mojo::Pg->new($config->{dbi_connection_string});
                  });

In the routes you can call $self->pg to get the database object.

The three approaches I’ve mentioned here are different.

DBIx::Connector is basically a way to get you a safe DBI handle across forks and DB connection failures.

Mojo::Pg gives you the ability to do abstract queries but also gives some convenient methods to get the results. I wouldn’t call it a ORM; from a query you usually gets hashes, not objects, you don’t need to define the database layout, and it won’t produce migrations for you, though there is some migration support.

Here’s an example of standard and abstract queries:

sub list_texts ($self) {
    if (my $sid = $self->param('sid')) {
        my $sql = 'SELECT * FROM texts WHERE sid = ? ORDER BY sorting_index';
        @all = $self->pg->db->query($sql, $sid)->hashes->each;
    }
    $self->render(json => { texts => \@all });

The query above can be rewritten with an abstract query, using the same module.

@all = $self->pg->db->select(texts => undef,
                             { sid => $sid },
                             { order_by => 'sorting_index' })->hashes->each;

If it’s a simple, static query, it’s basically a matter of taste; do you prefer to see the SQL or not? The second version is usually nicer if you want to build a different query depending on the parameters, so you add or remove keys to the hashes which maps to query and finally execute it.

Now, speaking of taste, for complex queries with a lot of joins I honestly prefer to see the SQL query instead of wondering if the abstract one is producing the correct SQL. This is true regardless of the framework. I have the impression that it is faster, safer, and cleaner to have the explicit SQL in the code rather than leaving future developers (including future me) to wonder if the magic is happening or not.

Finally, nothing stops you from using DBIx::Class, which is the best ORM for Perl, even if it’s not exactly light on dependencies.

It’s very versatile, it can build queries of arbitrary complexity, and you usually get objects out of the queries you make. It doesn’t come with an admin dashboard, it doesn’t enforce the data types and it doesn’t ship any validation by default (of course, you can implement that manually). The query syntax is very close to the Mojo::Pg one (which is basically SQL::Abstract).

The gain here is that, like in Django’s ORM, you can attach your methods to the classes representing the rows, so the data definitions live with the code operating on them.

However, the fact that it builds an object for each result means you’re paying a performance penalty which sometimes can be very high. I think this is a problem common to all ORMs, regardless of the language and framework you’re using.

The difference with Django is that once you have chosen it as your framework, you are basically already sold to the ORM. With Mojolicious and other Perl frameworks (Catalyst, Dancer), you can still make the decision and, at least in theory, change it down the road.

My recommendation would be to keep the model, both code and business logic, decoupled from the web-specific code. This is not really doable with Django, but is fully doable with the Perl frameworks. Just put the DB configuration in a dedicated file and the business code in appropriate classes. Then you should be able to, for example, run a script without loading the web and the whole framework configuration. In this ideal scenario, the web framework just provides the glue between the user and your model.

Controllers

Routes are defined similarly between Django and Mojolicious. Usually you put the code in a class and then point to it, attaching a name to it so you can reference it elsewhere. The language is different, the style is different, but they essentially do the same thing.

Django:

from django.urls import path
from . import views
urlpatterns = [
    path("api/agents/<int:agent_id>", views.api_agent_view, name="api_agent_view"),
]

The function views.api_agent_view will receive the request with the agent_id as a parameter.

Mojolicious:

sub startup ($self) {
    # ....
    my $r = $self->routes;
    $r->get('/list/:sid')->to('API#list_texts')->name('api_list_texts');
}

The ->to method is routing the request to the Myapp::Controller::API::list_texts, which will receive the request with the sid as parameter.

This is pretty much the core business of every web framework: routing a request to a given function.

Mojolicious has also the ability to chain the routes (pretty much taken from Catalyst). The typical use is authorization:

sub startup ($self) {
    ...
    my $r = $self->routes;
    my $api = $r->under('/api/v1', sub ($c) {
        if ($c->req->headers->header('X-API-Key') eq 'testkey') {
            return 1;
        }
        $c->render(text => 'Authentication required!', status => 401);
        return undef;
    }
    $api->get('/check')->to('API#check')->name('api_check');

So the request to /api/v1/check will first go in the first block and the chain will abort if the API key is not set in the header. Otherwise it will proceed to run the API module’s check function.

Conclusion

I’m Perl guy and so I’m a bit biased toward Mojolicious, but I also have a pragmatic approach to programming. Python is widely used — they teach it in schools — while Perl is seen as old-school, if not dead (like all the mature technologies). So, Python could potentially attract more developers to your project, and this is important to consider.

Learning a new language like Python is not a big leap; it and Perl are quite similar despite the different syntax. I’d throw Ruby in the same basket.

Of course both languages provide high quality modules you can use, and these two frameworks are an excellent example.

The Perl and Raku Conference 2024

2024-08-02T00:00:00+00:00

Next Generation of Perl

I attended The Perl and Raku Conference in Las Vegas, NV, which took place June 25–28, 2024. It was HOT outside (over 40 °C/110 °F) but we stayed cool inside at the Alexis Park Resort.

Curtis Poe (Ovid) got things started with the keynote encouraging us to Party Like It’s 19100+e^iπ, and reminded us that Vegas is lexically scoped (what happens in Vegas stays in Vegas)! More importantly he reminded us that Perl is about people, not just the technology. The Perl community has been meeting all over the world since 1999, with this being the 25th anniversary of the first The Perl Conference (aka YAPC::NA).

Ovid Keynote

Meeting in person with people who you interact with primarily through digital channels, code commits, and MetaCPAN documentation really highlighted the importance of the community. On the first day, I messed up timezones, showed up an hour before registration opened, and witnessed the conference organizers and core members arrive and greet each other with hugs. I also enjoyed visiting with one of the very welcoming board members of The Perl and Raku Foundation (TPRF).

Many of the speakers and attendees put a “Hallway++” sticker on their badge which simply meant “talk to me, I’m here to meet and get to know people”. At breakfast one morning, I had the privilege of sitting with Jason Crome, the core maintainer of Dancer, a framework that I have used extensively. It was amazing to be able to pick the brain of one of the people who has intimate knowledge of the software.

Dancing with Cromedome

The Perl community is large and diverse, which is reflected in the Science Perl Committee and the all-Perl Koha Library Software in use at over 4,000 libraries and with its own annual conference. It was cool to hear about the Glue Photo Project, making algorithmic music, and gaming with the TinyNES.

Every community will experience conflict and this one is no different. The impact of Sawyer’s resignation at TPRC 2023 could be felt at this conference, and in response the community is focused on making things better with a new standards of conduct.

It wouldn’t be a conference in 2024 without talk of AI. We had some Musings on Generative AI and an introduction to PerlGPT, A Code Llama LLM Fine-Tuned For Perl.

There were, of course, a lot of talks about actual Perl code!

One of the things I enjoy about attending conferences is discovering things that I wasn’t looking for. Chad Granum gave a lightning talk on goto::file and I noticed the use of line directives which can be extremely helpful in debugging evaled code. For example, let’s say you are evaling subs into a hashref and then calling them like so:

my $sub1 = "sub {\n  print 'foo';\n  print 'bar';\n print 'baz';\n}";
my $sub2 = "sub {\n  print 'foo';\n  print 'bar';\n warn 'baz';\n}";
my $Sub = {
  sub1 => eval $sub1,
  sub2 => eval $sub2,
};
$Sub->{sub1}->();
$Sub->{sub2}->();

Any warn or die output will give you a line number, but no context as to which sub it originated from:

baz at (eval 2) line 4.

Making use of a line directive when evaling the subs like this:

my $sub1 = "sub {\n  print 'foo';\n  print 'bar';\n print 'baz';\n}";
my $sub2 = "sub {\n  print 'foo';\n  print 'bar';\n warn 'baz';\n}";
my $Sub = {
  sub1 => eval qq(#line 1 "sub1"\n$sub1),
  sub2 => eval qq(#line 1 "sub2"\n$sub2),
};
$Sub->{sub1}->();
$Sub->{sub2}->();

Will now result in a much more friendly:

baz at sub2 line 4.

And in true open source fashion, this has been turned into a pull request for Interchange!

The Las Vegas Sphere

I always enjoy hearing from others in the community about CPAN modules that are in their toolbox. I learned about DBIx::QuickDB, which you can use to spin up a database server on the fly, which removes the need for a running database server for tests, and also enables running concurrent tests which require a database server that would otherwise conflict. Combine this with a DBIx::Class schema and DBIx::Class::Fixtures and you have a very nice way to run some tests against fixed data:

use DBIx::QuickDB PSQL_DB  => {driver => 'PostgreSQL'};
my $dbh = PSQL_DB->connect;
$schema = Some::Schema->connect( sub { $dbh }, { on_connect_do => ["..."] } );
$schema->deploy();
my $fixtures = DBIx::Class::Fixtures->new({ config_dir => '...' });
$fixtures->populate({ no_deploy => 1, schema => $schema, directory => '...' });

ok($schema->resultset('Foo')->count >= 1, 'database populated');

Damian Conway gave a keynote on The Once and Future Perl, showing how far Perl has come as a language and how its rich history can be leveraged into the future: “if you can envisage what you could have done better in the past, then you can probably think of ways to make the future brighter!”.

He showed off his new Multi::Dispatch module which you can use now to write incredibly extensible (and beautiful) code. Here it is in action with a simple Data::Dumper clone in 5 lines of code:

use v5.26;
use Multi::Dispatch;

multi dd :before :where(VOID) (@data)   { say &next::variant }
multi dd ($k, $v)                       { dd($k) . ' => ' . dd($v) }
multi dd ($data :where(ARRAY))          { '[' . join(', ', map {dd($_)}                 @$data) . ']' }
multi dd ($data :where(HASH))           { '{' . join(', ', map {dd($_, $data->{$_})} keys %$data) . '}' }
multi dd ($data)                        { '"' . quotemeta($data) . '"' }

say dd ['foo', { bar => "baz" }];

With Smartmatch (given/when) scheduled for deprecation in 5.42 he has written a drop-in replacement that uses Multi::Dispatch: Switch::Right, which addresses the issues with the original implementation that was in core.

Also on display was the new class syntax introduced into core in 5.38 with use feature 'class'. Here’s an example of how it looks:

use v5.40;
use feature 'class';

class Point {
  field $x :param = 0;
  field $y :param = 0;

  method describe () {
      say "A point at ($x, $y)\n";
   }
}

Point->new(x => 5, y => 10)->describe;

If you are waiting for Perl 7, Damian is here to tell you that the future is now; you don’t have to wait for Perl 7 (or 17), it’s Perl 5 with Multi::Dispatch and use feature 'class'!

The last day of the conference provided an opportunity to go deeper into learning the new class syntax with a workshop on building a roguelike adventure game from scratch.

So what’s next? The London Perl & Raku Workshop is taking place on October 26, 2024 and Perl 5.42 is just around the corner!

—JAPH (Just Another Perl Hacker)

Perl Web Frameworks

2022-04-19T00:00:00+00:00

CGI

When I started programming, back in the day, CGI (the Common Gateway Interface) was still widely used. Usually the Apache webserver would just execute a script or a binary with some environment variables set and serve whatever the executable sent to the standard output, while keeping the standard error in the logs.

This simple and straightforward mechanism can still be used for small programs, but larger applications usually want to save the start-up time and live longer than just a single request.

At that time Perl was used far more often than now, and it had (and still has) the CGI.pm module to help the programmer to get the job done.

#!/usr/bin/env perl

use utf8;
use strict;
use warnings;
use CGI;

my $q = CGI->new;
print $q->header;
my $name = 'Marco';
print $q->p("Hello $name");
print "\n";

And it will output:

./cgi.pl
Content-Type: text/html; charset=ISO-8859-1

<p>Hello Marco</p>

Here the script mixes logic and formatting and the encoding it produces by default tells us that this comes from another age. But if you want something which is seldom used and gets executed on demand without persisting in the machine’s memory, this is still an option.

Please note that there are frameworks which can work in CGI mode, so there is no reason to use CGI.pm, beside having to maintain legacy programs.

Mojolicious

Fast-forward to 2022.

Nowadays Perl is just another language among dozens of them. But it still gets the job done and lets you write nice, maintainable code like any other modern language.

Mojolicious is currently the top choice if you want to do web development in Perl. It is an amazing framework, with a large and active community, and appears to have collected the best concepts that other web frameworks from other languages have to offer.

Let’s hack an app in a couple of minutes in a single file, like during the CGI days:

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;

use Mojolicious::Lite;

get '/' => sub {
    my ($c) = @_;
    $c->stash(name => "Marco");
    $c->render(template => 'index');
};

app->start;

__DATA__
@@ index.html.ep
Hello <%= $name %>

Here the structure is a bit different.

First, there’s a Domain Specific Language (DSL) to give you some sugar. This is the “Lite” version, while in a well-structured Mojolicious app one prefers to write class methods. We declare that the root (/) URL path of the application is going to execute some code. It populates the “stash” with some variables, and finally renders a template which can access the stashed variables. If you execute the script, you get:

./mojo.pl cgi 2> /dev/null
Status: 200 OK
Content-Length: 12
Date: Fri, 08 Apr 2022 12:33:52 GMT
Content-Type: text/html;charset=UTF-8

Hello Marco

The logging to the standard error stream is:

[2022-04-08 14:33:52.92508] [163133] [debug] [82ae3iV2] GET "/"
[2022-04-08 14:33:52.92532] [163133] [debug] [82ae3iV2] Routing to a callback
[2022-04-08 14:33:52.92565] [163133] [debug] [82ae3iV2] Rendering template "index.html.ep" from DATA section
[2022-04-08 14:33:52.92610] [163133] [debug] [82ae3iV2] 200 OK (0.001021s, 979.432/s)

This is basically what a modern framework is supposed to do.

The nice thing in this example is that we created a single-file prototype and launched it as a CGI. But we can also launch it as daemon and visit the given address with a browser, which is how you should normally deploy it, usually behind a reverse proxy like nginx.

./mojo.pl daemon
[2022-04-08 14:48:42.01827] [163409] [info] Listening at "http://*:3000"
Web application available at http://127.0.0.1:3000
[2022-04-08 14:48:48.53687] [163409] [debug] [CwM6zoUQ] GET "/"
[2022-04-08 14:48:48.53715] [163409] [debug] [CwM6zoUQ] Routing to a callback
[2022-04-08 14:48:48.53752] [163409] [debug] [CwM6zoUQ] Rendering template "index.html.ep" from DATA section
[2022-04-08 14:48:48.53808] [163409] [debug] [CwM6zoUQ] 200 OK (0.001209s, 827.130/s)

If you want you can even launch it with HTTPS as well (please note the syntax to pass the certificates).

./mojo.pl daemon -l 'https://[::]:8080?cert=./ssl/fullchain.pem&key=./ssl/privkey.pem' -m production

For a small application listening on a high port this is already enough and the whole deployment problem goes away.

Speaking about deployment, Mojolicious has basically no dependencies other than the core modules and comes with a lot of goodies, for example a non blocking user-agent.

Recently a legacy application needed to make some API calls. To speed up the process, we wanted to make the requests in parallel. And here’s the gist of the code:

package MyApp::Async;

# ... more modules here

use Mojo::UserAgent;
use Mojo::Promise;

# .... other methods here

sub example {
    my $email = 'test@example.com';
    my $ua = Mojo::UserAgent->new;
    foreach my $list ($self->get_lists) {
        my $promise = $ua->post_p($self->_url("/api/v2/endpoint/$list->{code}"),
                                  json => { email => $email })
          ->then(sub {
                     my $tx = shift;
                     my $res = $tx->result;
                     if ($res->code =~ m/^2/) {
                         $self->_update_db($data);
                     }
                     else {
                         die $tx->req->url . ' ' . $res->code;
                     }
                 });
        push @promises, $promise;
    }
    my $return = 0;
    Mojo::Promise->all(@promises)->then(sub { $return = 1 }, sub { $return = 0})->wait;
    return $return;
}

So a bunch of requests are run in parallel and then synced before returning. Does it remind you of JavaScript? Of course. A lot of common paradigms taken from other languages and frameworks were implemented here, and you can find the best of them in this nice package.

But the point here is that it doesn’t need dozens of new modules installed or upgraded. It’s just a single module in pure Perl that you can even install in your application tree. This is a huge advantage if you’re dealing with a legacy application which uses an old Perl tree and you want to play safe.

So, if you’re starting from scratch, go with Mojolicious. It lets you prototype fast and doesn’t let you down later.

However, starting from scratch is not always an option. Actually, it’s a rare opportunity. There’s a whole world of legacy applications and they generate real money every day. It’s simply not possible or even desirable to throw away something that works for something that would do the same thing but in a “cooler” way. In ten years, the way we’re coding will look old anyway.

Interchange

Wait. Isn’t Interchange an old e-commerce framework? Yes, it’s not exactly a generic web framework, on the contrary, it’s a specialized one, but it’s still a framework and you can still do things in a maintainable fashion. The key is using the so-called action maps:

ActionMap jump <<EOR
sub {
    # get the path parameters
    my ($action, @args) = split(/\//, shift);

    # get the query/body parameters
    my $param = $CGI->{param};

    # redirect to another page
    $Tag->deliver({ location => $final });

    # or serve JSON
    $Tag->deliver({ type => 'application/json', body => $json_string });

    # or serve a file
    $Tag->deliver({ type => 'text/plain', body => $bigfile });

    # or populate the "stash" and serve a template page
    $Tag->tmp(stash_variable => "Marco");
    $CGI->{mv_nextpage} = "test.html";
}
EOR

In pages/test.html you would put this template:

<p>Hello [scratch stash_variable]</p>

Now, I can’t show you a simple script which demonstrates this and you’ll have to take my word for it since we can’t go through the installation process here for a demo.

Interchange is old, and it shows its years, but it is actively maintained. It lacks many of Mojo’s goodies, but you can still do things in a reasonable way.

In the example the code will execute when a path starting with /jump/ is requested. The whole path is passed to the routine, so you can split at /, apply your logic, and finally either set $CGI->{mv_nextpage} to a file in the pages directory or output the response body directly with deliver. This way you can easily build, as a classical example, an API.

It’s a bit of a poor man’s MVC but it works. That’s basically the core of what a framework like Dancer does.

Dancer (1 & 2)

Dancer is basically Ruby’s Sinatra ported to Perl. As already mentioned, ideas developed in other languages and frameworks are often ported to Perl, and this is no exception.

Let’s see it in an action:

#!/usr/bin/env perl
use strict;
use warnings;
use Dancer2;

get '/' => sub {
    my $name = "Marco";
    return "Hello $name\n";
};

start;

Start the script:

Dancer2 v0.400000 server 22969 listening on http://0.0.0.0:3000

Try it with curl:

$ curl -D - http://0.0.0.0:3000
HTTP/1.0 200 OK
Date: Mon, 11 Apr 2022 07:22:18 GMT
Server: Perl Dancer2 0.400000
Server: Perl Dancer2 0.400000
Content-Length: 12
Content-Type: text/html; charset=UTF-8

Hello Marco

If in the script you say use Dancer; instead of use Dancer2, you get:

$ curl -D - http://0.0.0.0:3000
HTTP/1.0 200 OK
Server: Perl Dancer 1.3513
Content-Length: 12
Content-Type: text/html
X-Powered-By: Perl Dancer 1.3513

Hello Marco

Dancer’s core doesn’t do much more than routing. And you’ll also notice that the syntax is very similar to Mojolicious::Lite. So to get something done you need to start installing plugins which will provide the needed glue to interact with a database, work with your template system of choice, and more.

Today you would wonder why you should use Dancer and not Mojolicious, but when Dancer was at the peak of its popularity the games were still open. There were plenty of plugins being written and published on CPAN.

Around 2013 Dancer’s development team decided to rewrite it to make it better. The problem was that plugins and templates needed to be adapted as well. I’m under the impression that the energy got divided and the momentum was lost. Now there are two codebases and two plugin namespaces which do basically the same thing, because for the end user there is not much difference.

Catalyst

So what was attracting people to Dancer? When Dancer came out, Perl had a great MVC framework, which is still around, Catalyst. (And note that the main Mojolicious developer was on the Catalyst team.)

Now, the problem is that to get started with Catalyst, even if it has plenty of documentation, you need to be already acquainted with a lot of concepts and technologies. For example, the tutorial starts to talk about Template Toolkit and the DBIx::Class ORM very early.

These two modules are great and powerful and they deserve to be studied, but for someone new to modern web development, or even to Perl, it feels (and actually is) overwhelming.

So, why would you choose Catalyst today? Catalyst has the stability which Mojo, at least at the beginning, lacked, while backward compatibility is a priority for Catalyst. The other way to look at this is that Catalyst doesn’t see much current development, but someone could see this as a feature.

Even if Catalyst predates all the hyper-modern features that Mojo has, it’s still a modern framework, and a good one. I can’t show you a self contained script (you need a tree of files), but I’d like to show you what makes it very nice and powerful:

package MyApp::Controller::Root;

use Moose;
use namespace::autoclean;

BEGIN { extends 'Catalyst::Controller'; }

# start the chain with /foo/XX
sub foo :Chained('/') CaptureArgs(1) {
    my ($self, $c, $arg) = @_;
    $c->stash(name => "$arg");
}

# /foo/XX/bar/YY
sub bar :Chained('foo') Args(1) {
    my ($self, $c, $arg) = @_;
    $c->detach($c->view('JSON'));
}

# /foo/XX/another/YY
sub another :Chained('foo') Args(1) {
    my ($self, $c, $arg) = @_;
    $c->detach($c->view('HTML'));
}

So, if you hit /foo/marco/bar/test the second path fragment will be processed by the first method (CaptureArgs(1)) and saved in the stash. Then the second bar method will be chained to it and the name will be available in the stash. The last method will be hit with /foo/marco/another/test2. (Incidentally, please note that Mojolicious has nested routes as well.)

Now, I think it’s clear that in this way you can build deep hierarchies of paths with reusable components. This works really great with the DBIx::Class ORM, where you can chain queries as well. As you can imagine, this is far from a simple setup. On the contrary, this is an advanced setup for people who already know their way around web frameworks.

Conclusion

So, to sum up this excursion in the amazing land of Perl web frameworks: If you build something from scratch, go with Mojolicious. It’s your best bet. If nothing else, it’s super-easy to install, with basically no dependencies.

However, there’s no need to make a religion out of it. Rewriting code without a clear gain is a waste of time and money. A good developer should still be able to write maintainable code with the existing tools.

Full-text search on a budget: Xapian

2021-08-19T00:00:00+00:00

Over the years I’ve seen and implemented different full-text search applications using various technologies: plain SQL, PostgreSQL, Elasticsearch, Solr, and most recently Xapian.

While Solr and Elasticsearch are very well known, Xapian, despite the fact that it’s available and packaged in all the major GNU/Linux distributions, doesn’t seem to be so popular, at least not among project managers.

But Xapian is fast, advanced, can be configured to do faceted searches (so the user can filter the search results), and my favorite, is fast to build and has virtually no maintenance overhead.

Its main feature is that it’s not a stand-alone application, like Solr or Elasticsearch, but instead it’s a library written in C++ which has bindings for all the major languages (as advertised on its homepage). It has also great documentation.

Now, being in the e-commerce business, my typical use-case is that the client’s shop needs something faster and better than a search using a SQL query against the products table. And beware, even implementing a non-trivial SQL-based search can burn more hours than setting up Xapian.

With Xapian you can prototype very quickly, without losing hours wading through obscure options, setting up services and configuring firewalls. And yet, the prototype will allow you to build more advanced features once you need them.

I’m a Perl guy, so I will show you some Perl code, but the procedure is the same for the other languages. Even the documentation can be built specifically for your language!

Typically, to add a search engine to your site you need two pieces: an indexer to which you feed the data (from static files or databases or even fetching remote pages or whatever you need) and the search itself in the site.

Both the indexer and the search code need to load the Xapian library and point to the same Xapian database, which is usually a directory (or a file pointing to a directory).

Indexing

Now, stripped down to the minimum, this is what a typical indexer’s code looks like:

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian (':all');
use JSON qw/encode_json/;

my $dblocation = "xapiandb";
my $xapian = Search::Xapian::WritableDatabase->new($dblocation, DB_CREATE_OR_OPEN);
my $indexer = Search::Xapian::TermGenerator->new;
$indexer->set_database($xapian);
$indexer->set_stemmer(Search::Xapian::Stem->new("english"));
my @entries = ({ uri => '/blog/1', title => 'T1', text => 'Marco loves pizza' },
               { uri => '/blog/2', title => 'T2', text => 'They love chapati' });
foreach my $data (@entries) {
    my $doc = Search::Xapian::Document->new;
    my $qterm = 'Q' . $data->{uri};
    $doc->add_term($qterm);
    $doc->set_data(encode_json({ uri => $data->{uri}, title => $data->{title} }));
    $indexer->set_document($doc);
    $indexer->index_text($data->{text});
    $xapian->replace_document_by_term($qterm, $doc);
}

This code will create a xapiandb directory with the Xapian database, indexing the blog posts in the @entries array. In a real script, they would come from the database.

Still, there are a couple of things worth noting in this minimal code.

We set the stemmer for the given language, so the text passed to the indexer via the index_text call is parsed.

Then we store the data structure we want to retrieve later with set_data. The best thing to do is probably to serialize it with JSON, in this case I’m excluding the full text, which we don’t need in the output (but it would be wise to add a teaser).

Also, we use a Q prefix to produce an unique term to update the entry when it already exists.

Of course the indexer will need to grow if you need more power and more structured data (like filtering or searching a specific field), but at this point we want just to show something to our hypothetical client.

The database can be inspected very easily. Xapian comes with a tool called delve (or xapian-delve):

$ xapian-delve xapiandb -a -v -1
All terms in database (termfreq):
Q/blog/1 1
Q/blog/2 1
Zchapati 1
Zlove 2
Zmarco 1
Zpizza 1
Zthey 1
chapati 1
love 1
loves 1
marco 1
pizza 1
they 1

And you can also try a search from the command line with quest:

$ quest -d xapiandb "loves NOT chapati"
Parsed Query: Query((Zlove@1 AND_NOT Zchapati@2))
Exactly 1 matches
MSet:
1: [0.0953102]
{"title":"T1","uri":"/blog/1"}

$ quest -d xapiandb "pizza OR chapati"
Parsed Query: Query((Zpizza@1 OR Zchapati@2))
Exactly 2 matches
MSet:
1: [0.405465]
{"title":"T1","uri":"/blog/1"}
2: [0.405465]
{"uri":"/blog/2","title":"T2"}

As the example above shows, it should be clear that:

the search works as you would expect (with logical operators) out of the box
the stemming works, searching for “loves” and “love” is the same.
the results give us back the JSON we stored in the index.

Searching

So let’s call it done and move to the next part, the searcher.

Now, while the indexer is a single script, the search needs to be plugged into the live code of your site. For the purposes of this article, I will provide a script instead, which does basically the same thing as quest. Plugging it into the web application is left as an exercise for the reader. I would also suggest to put both the indexing and searching code in a single shared module, keeping the logic in a single location.

#!/usr/bin/env perl
use utf8;
use strict;
use warnings;
use Search::Xapian ':all';
use JSON;

my ($cgi) = join(' ', @ARGV);
my $dblocation = "xapiandb";
my $database = Search::Xapian::Database->new($dblocation);
my $enquire = Search::Xapian::Enquire->new($database);
my $qp = Search::Xapian::QueryParser->new;
$qp->set_database($database);
$qp->set_stemmer(Search::Xapian::Stem->new("english"));
$qp->set_stemming_strategy(STEM_SOME);
$qp->set_default_op(OP_AND);
my $query = $qp->parse_query($cgi, FLAG_PHRASE|FLAG_BOOLEAN|FLAG_WILDCARD);
$enquire->set_query($query);

# fetch the first 50 results
my $mset = $enquire->get_mset(0, 50);
print "Total results: " . $mset->get_matches_estimated . "\n";

my $json_pretty = JSON->new->pretty(1)->utf8(1)->canonical(1);
foreach my $m ($mset->items) {
    my $data = decode_json($m->get_document->get_data);
    # decode and reencode the json in a human-readable fashion
    print $json_pretty->encode($data);
}

If you’re wondering what those constants are and where to look for more, they are in the module’s documentation, in plain sight (we asked for them when loading the module with the :all argument).

Most of the code shown here is boilerplate, but that could change once you build up. Notably we set the stemmer for the current language and the query parser options, so we can use wildcard (e.g. piz*), the AND/OR operators, and quoting.

Let’s see the script in action.

Wildcard:

$ ./search.pl 'piz'
Total results: 0
$ ./search.pl 'piz*'
Total results: 1
{
   "title" : "T1",
   "uri" : "/blog/1"
}

Operators:

$ ./search.pl 'pizza OR chapati'
Total results: 2
{
   "title" : "T1",
   "uri" : "/blog/1"
}
{
   "title" : "T2",
   "uri" : "/blog/2"
}
$ ./search.pl 'pizza AND chapati'
Total results: 0

Quoting (beware here the double quotes to escape the shell):

$ ./search.pl '"loves chapati"'
Total results: 0
$ ./search.pl '"love chapati"'
Total results: 1
{
   "title" : "T2",
   "uri" : "/blog/2"
}

The whole thing already looks pretty good. Way better (and way faster to code and to execute) than a home-baked SQL search.

As already noted, this is just scratching the surface. Xapian can do much more: filtering, range queries, facets, sorting, even spelling corrections!

I don’t doubt that Solr & co. have their use-cases, but for the common scenario of a small/mid-sized e-shop or site, I think that this solution is more affordable and maintainable than having a whole separate application (like a Solr server) to maintain, upgrade and secure. Don’t forget that here we haven’t done a single HTTP request. We didn’t have to manage daemons, opening/closing ports, and the like. We didn’t have to configure a schema and a tokenizer in a separate application (and keep that aligned with the handling code). It’s all there in our (Perl) code in two files (as already noted, the logic should live in a single module).

We just installed a library (there is a very good chance that it’s already installed) and a Perl module.

The Xapian database lives on the disk and your code has full control over it. Also it’s normally your GNU/Linux distribution taking care of the security upgrades.

If your client is on a budget, building a full-text search Xapian can be the right choice, and you can scale it up on the go, as more features are required.

Regular Expression Inconsistencies With Unicode

2018-01-23T00:00:00+00:00

A casual stroll through the world of Unicode and regular expressions—Photo by Presidio of Monterey

Character classes in regular expressions are an extremely useful and widespread feature, but there are some relatively recent changes that you might not know of.

The issue stems from how different programming languages, locales, and character encodings treat predefined character classes. Take, for example, the expression \w which was introduced in Perl around the year 1990 (along with \d and \s and their inverted sets \W, \D, and \S).

The \w shorthand is a character class that matches “word characters” as the C language understands them: [a-zA-Z0-9_]. At least when ASCII was the main player in the character encoding scene that simple fact was true. With the standardization of Unicode and UTF-8, the meaning of \w has become a more foggy.

Perl

Take this example in a recent Perl version:

use 5.012; # use 5.012 or higher includes Unicode support
use utf8;  # necessary for Unicode string literals

print "username" =~ /^\w+$/; # 1
print "userاسم"  =~ /^\w+$/; # 1

Perl is treating \w differently here because the characters “اسم” (“ism” meaning “name” in Arabic) definitely don’t fall within [a-zA-Z0-9_]!

Beginning with Perl 5.12 from the year 2010, character classes are handled differently. Documentation on the topic is found in perlrecharclass. The rules aren’t as simple as with some languages, but can be generalized as such:

\w will match Unicode characters with the “Word” property (equivalent to \p{Word}), unless the /a (ASCII) flag is enabled, in which case it will be equivalent to the original [a-zA-Z0-9_].

Let’s see the /a flag in action.

use 5.012;
use utf8;

print "username" =~ /^\w+$/a; # 1
print "userاسم"  =~ /^\w+$/a; # 0

However, you should know that for code points below 256, these rules can change depending on whether Unicode or locale rules are on, so if you’re unsure, consult the perlre and perlrecharclass.

Keep in mind that these same questions of what the character classes include can apply to every predefined character class in whatever language you’re using, so remember to check language-specific implementations for other character class shorthands, such as \s and \d, not just \w.

Every language seems to do regular expressions a little bit differently, so here’s a short, incomplete guide for several other languages we use frequently.

Python

Take this example in Python 3.6.2:

>>> re.match(r'^\w+$', 'username')
<_sre.SRE_Match object; span=(0, 8), match='username'>
>>> re.match(r'^\w+$', 'userاسم')
<_sre.SRE_Match object; span=(0, 7), match='userاسم'>

Python is also treating \w differently here. Let’s take a look at the Python docs:

\w

For Unicode (str) patterns:

Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched (but the flag affects the entire regular expression, so in such cases using an explicit [a-zA-Z0-9_] may be a better choice).

For 8-bit (bytes) patterns:

Matches characters considered alphanumeric in the ASCII character set; this is equivalent to [a-zA-Z0-9_]. If the LOCALE flag is used, matches characters considered alphanumeric in the current locale and the underscore.

So \w includes “most characters that can be part of a word in any language, as well as numbers and the underscore”. A list of the characters that includes is difficult to pin down, so it would be best to use the re.ASCII flag as suggested when you’re unsure if you want letters from other languages matched:

>>> re.match(r'^\w+$', 'userاسم',  flags=re.ASCII)
>>> re.match(r'^\w+$', 'username', flags=re.ASCII)
<_sre.SRE_Match object; span=(0, 8), match='username'>

Ruby

Ruby’s Regexp class documentation gives a simple and useful explanation: backslash character classes (e.g. \w, \s, \d) are ASCII-only, while POSIX-style bracket expressions (e.g. [[:alnum:]]) include other Unicode characters.

irb(main):001:0> /^\w+$/         =~ "userاسم"
=> nil
irb(main):002:0> /^[[:word:]]+$/ =~ "userاسم"
=> 0

JavaScript

JavaScript doesn’t support POSIX-style bracket expressions, and its backslash character classes are simple, straightforward lists of ASCII characters. The MDN has simple explanations for each one.

JavaScript regular expressions do accept a /u flag, but it does not affect shorthand character classes. Consider these examples in Node.js:

> /^\w+$/.test("username");
true
> /^\w+$/.test("userﺎﺴﻣ");
false
> /^\w+$/u.test("username");
true
> /^\w+$/u.test("userﺎﺴﻣ");
false

We can see that the /u flag has no effect on what \w matches. Now let’s look at Unicode character lengths in JavaScript:

> '❤'.length
1
> '👩'.length
2
> '🀄️'.length
3

Because of the way Unicode is implemented in JavaScript, strings with Unicode characters outside the BMP (Basic Multilingual Plane) will appear to be longer than they are.

This can be accounted for in regular expressions with the /u flag, which only corrects character parsing, and does not affect shorthand character classes:

> let mystr = "hi👩there";
undefined
> mystr.length
9
> /hi.there/.test(mystr);
false
> /hi..there/.test(mystr);
true
> /hi.there/u.test(mystr);  # note the /u from here on
true
> /hi..there/u.test(mystr);
false
> /hi..there/u.test("hi👩👩there");
true

The excellent article "💩".length === 2 by Jonathan New goes into detail about the why this is, and explores various solutions. It also addresses some legacy inconsistencies, like how the old HEAVY BLACK HEART character and other older Unicode symbols might be represented differently.

PHP

PHP’s documentation explains that \w matches letters, digits, and the underscore as defined by your locale. It’s not totally clear about how Unicode is treated, but it uses the PCRE (Perl Compatible Regular Expressions) library which supports a /u flag that can be used to enable Unicode matching in character classes:

<?php

echo preg_match("/^\\w+$/", "username"), "\n";  # 1
echo preg_match("/^\\w+$/", "userاسم"),  "\n";  # 0

echo preg_match("/^\\w+$/u", "username"), "\n"; # 1
echo preg_match("/^\\w+$/u", "userاسم"),  "\n"; # 1

.NET

The .NET Quick Reference has a comprehensive guide to character classes. For word characters, it defines a specific group of Unicode categories including letters, modifiers, and connectors from many languages, but also points out that setting the ECMAScript Matching Behavior option will limit \w to [a-zA-Z_0-9], among other things. Microsoft’s documentation is clear and comprehensive with great examples, so I recommend referring to it frequently.

Go

Go follows the regular expression syntax used by Google’s RE2 engine, which has easy syntax for specifying whether you want Unicode characters to be captured or not:

package main

import (
	"fmt"
	"regexp"
)

func main() {
	// Perl-style
	fmt.Println(regexp.MatchString(`^\w+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^\w+$`, "userاسم"))  // false

	// POSIX-style
	fmt.Println(regexp.MatchString(`^[[:word:]]+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^[[:word:]]+$`, "userاسم"))  // false

	// Unicode character class
	fmt.Println(regexp.MatchString(`^\pL+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^\pL+$`, "userاسم"))  // true
}

You can see this code in action here.

grep

Implementations of grep vary widely across platforms and versions. On my personal computer with GNU grep 3.1, \w doesn’t work at all with default settings, matches only ASCII characters with the -P (PCRE) option, and matches Unicode characters with -E:

[phin@caballero ~]$ grep    "^\w+$" <(echo "username")  # no match
[phin@caballero ~]$ grep -P "^\w+$" <(echo "username")
username
[phin@caballero ~]$ grep -P "^\w+$" <(echo "userاسم")   # no match
[phin@caballero ~]$ grep -E "^\w+$" <(echo "username")
username
[phin@caballero ~]$ grep -E "^\w+$" <(echo "userاسم")
userاسم

Again, implementations vary a lot, so double check on your system before doing anything important.

Postgres migrating SQL_ASCII to UTF-8 with fix_latin

2017-07-21T00:00:00+00:00

(photograph by NOAA National Ocean Service)

Upgrading Postgres is not quite as painful as it used to be, thanks primarily to the pg_upgrade program, but there are times when it simply cannot be used. We recently had an existing End Point client come to us requesting help upgrading from their current Postgres database (version 9.2) to the latest version (9.6—but soon to be 10). They also wanted to finally move away from their SQL_ASCII encoding to UTF-8. As this meant that pg_upgrade could not be used, we also took the opportunity to enable checksums as well (this change cannot be done via pg_upgrade). Finally, they were moving their database server to new hardware. There were many lessons learned and bumps along the way for this migration, but for this post I’d like to focus on one of the most vexing problems, the database encoding.

When a Postgres database is created, it is set to a specific encoding. The most common one (and the default) is “UTF8”. This covers 99% of all user’s needs. The second most common one is the poorly-named “SQL_ASCII” encoding, which should be named “DANGER_DO_NOT_USE_THIS_ENCODING”, because it causes nothing but trouble. The SQL_ASCII encoding basically means no encoding at all, and simply stores any bytes you throw at it. This usually means the database ends up containing a whole mess of different encodings, creating a “byte soup” that will be difficult to sanitize by moving to a real encoding (i.e. UTF-8).

Many tools exist which convert text from one encoding to another. One of the most popular ones on Unix boxes is “iconv”. Although this program works great if your source text is using one encoding, it fails when it encounters byte soup.

For this migration, we first did a pg_dump from the old database to a newly created UTF-8 test database, just to see which tables had encoding problems. Quite a few did—but not all of them!—so we wrote a script to import tables in parallel, with some filtering for the problem ones. As mentioned above, iconv was not particularly helpful: looking at the tables closely showed evidence of many different encodings in each one: Windows-1252, ISO-8859-1, Japanese, Greek, and many others. There were even large bits that were plainly binary data (e.g. images) that simply got shoved into a text field somehow. This is the big problem with SQL_ASCII: it accepts everything, and does no validation whatsoever. The iconv program simply could not handle these tables, even when adding the //IGNORE option.

To better explain the problem and the solution, let’s create a small text file with a jumble of encodings. Discussions of how UTF-8 represents characters, and its interactions with Unicode, are avoided here, as Unicode is a dense, complex subject, and this article is dry enough already. :)

First, we want to add some items using the encodings ‘Windows-1252’ and ‘Latin-1’. These encoding systems were attempts to extend the basic ASCII character set to include more characters. As these encodings pre-date the invention of UTF-8, they do it in a very inelegant (and incompatible) way. Use of the “echo” command is a great way to add arbitrary bytes to a file as it allows direct hex input:

$ echo -e "[Windows-1252]   Euro: \x80   Double dagger: \x87" > sample.data
$ echo -e "[Latin-1]   Yen: \xa5   Half: \xbd" >> sample.data
$ echo -e "[Japanese]   Ship: \xe8\x88\xb9" >> sample.data
$ echo -e "[Invalid UTF-8]  Blob: \xf4\xa5\xa3\xa5" >> sample.data

This file looks ugly. Notice all the “wrong” characters when we simply view the file directly:

$ cat sample.data
[Windows-1252]   Euro: �   Double dagger: �
[Latin-1]   Yen: �   Half: �
[Japanese]   Ship: 船
[Invalid UTF-8]  Blob: ����

Running iconv is of little help:

## With no source encoding given, it errors on the Euro:
$ iconv -t utf8 sample.data >/dev/null
iconv: illegal input sequence at position 23

## We can tell it to ignore those errors, but it still barfs on the blob:
$ iconv -t utf8//ignore sample.data >/dev/null
iconv: illegal input sequence at position 123

## Telling it the source is Window-1252 fixes some things, but still sinks the Ship:
$ iconv -f windows-1252 -t utf8//ignore sample.data
[Windows-1252]   Euro: €   Double dagger: ‡
[Latin-1]   Yen: ¥   Half: ½
[Japanese]   Ship: èˆ¹
[Invalid UTF-8]  Blob: ô¥£¥

After testing a few other tools, we discovered the nifty Encoding::FixLatin, a Perl module which provides a command-line program called “fix_latin”. Rather than being authoritative like iconv, it tries its best to fix things up with educated guesses. Its documentation gives a good summary:

The script acts as a filter, taking source data which may contain a mix of ASCII, UTF8, ISO8859-1 and CP1252 characters, and producing output will be all ASCII/UTF8.

Multi-byte UTF8 characters will be passed through unchanged (although over-long UTF8 byte sequences will be converted to the shortest normal form). Single byte characters will be converted as follows:

0x00 - 0x7F ASCII - passed through unchanged 0x80 - 0x9F Converted to UTF8 using CP1252 mappings 0xA0 - 0xFF Converted to UTF8 using Latin-1 mappings

While this works great for fixing the Windows-1252 and Latin-1 problems (and thus accounted for at least 95% of our table’s bad encodings), it still allows “invalid” UTF-8 to pass on through. Which means that Postgres will still refuse to accept it. Let’s check our test file:

$ fix_latin sample.data
[Windows-1252]   Euro: €   Double dagger: ‡
[Latin-1]   Yen: ¥   Half: ½
[Japanese]   Ship: 船
[Invalid UTF-8]  Blob: ����

## Postgres will refuse to import that last part:
$ echo "SELECT E'"  "$(fix_latin sample.data)"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5

## Even adding iconv is of no help:
$ echo "SELECT E'"  "$(fix_latin sample.data | iconv -t utf-8)"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5

The UTF-8 specification is rather dense and puts many requirements on encoders and decoders. How well programs implement these requirements (and optional bits) varies, of course. But at the end of the day, we needed that data to go into a UTF-8 encoded Postgres database without complaint. When in doubt, go to the source! The relevant file in the Postgres source code responsible for rejecting bad UTF-8 (as in the examples above) is src/backend/utils/mb/wchar.c Analyzing that file shows a small but elegant piece of code whose job is to ensure only “legal” UTF-8 is accepted:

bool
pg_utf8_islegal(const unsigned char *source, int length)
{
  unsigned char a;

  switch (length)
  {
    default:
      /* reject lengths 5 and 6 for now */
      return false;
    case 4:
      a = source[3];
      if (a < 0x80 || a > 0xBF)
        return false;
      /* FALL THRU */
    case 3:
      a = source[2];
      if (a < 0x80 || a > 0xBF)
        return false;
      /* FALL THRU */
    case 2:
      a = source[1];
      switch (*source)
      {
        case 0xE0:
          if (a < 0xA0 || a > 0xBF)
            return false;
          break;
        case 0xED:
          if (a < 0x80 || a > 0x9F)
            return false;
          break;
        case 0xF0:
          if (a < 0x90 || a > 0xBF)
            return false;
          break;
        case 0xF4:
          if (a < 0x80 || a > 0x8F)
            return false;
          break;
        default:
          if (a < 0x80 || a > 0xBF)
            return false;
          break;
      }
      /* FALL THRU */
    case 1:
      a = *source;
      if (a >= 0x80 && a < 0xC2)
        return false;
      if (a > 0xF4)
        return false;
      break;
  }
  return true;
}

Now that we know the UTF-8 rules for Postgres, how do we ensure our data follows it? While we could have made another standalone filter to run after fix_latin, that would increase the migration time. So I made a quick patch to the fix_latin program itself, rewriting that C logic in Perl. A new option “–strict-utf8” was added. Its job is to simply enforce the rules found in the Postgres source code. If a character is invalid, it is replaced with a question mark (there are other choices for a replacement character, but we decided simple question marks were quick and easy—and the surrounding data was unlikely to be read or even used anyway).

Voila! All of the data was now going into Postgres without a problem. Observe:

$ echo "SELECT E'"  "$(fix_latin  sample.data)"  "';" | psql
ERROR:  invalid byte sequence for encoding "UTF8": 0xf4 0xa5 0xa3 0xa5

$ echo "SELECT E'"  "$(fix_latin --strict-utf8 sample.data)"  "';" | psql
                   ?column?
----------------------------------------------
  [Windows-1252]   Euro: €   Double dagger: ‡+
 [Latin-1]   Yen: ¥   Half: ½                +
 [Japanese]   Ship: 船                       +
 [Invalid UTF-8]  Blob: ????
(1 row)

What are the lessons here? First and foremost, never use SQL_ASCII. It’s outdated, dangerous, and will cause much pain down the road. Second, there are an amazing number of client encodings in use, especially for old data, but the world has pretty much standardized on UTF-8 these days, so even if you are stuck with SQL_ASCII, the amount of Windows-1252 and other monstrosities will be small. Third, don’t be afraid to go to the source. If Postgres is rejecting your data, it’s probably for a very good reason, so find out exactly why. There were other challenges to overcome in this migration, but the encoding was certainly one of the most interesting ones. Everyone, the client and us, is very happy to finally have everything using UTF-8!

Postal code pain and fun

2017-05-10T00:00:00+00:00

We do a lot of ecommerce development at End Point. You know the usual flow as a customer: Select products, add to the shopping cart, then check out. Checkout asks questions about the buyer, payment, and delivery, at least. Some online sales are for “soft goods”, downloadable items that don’t require a delivery address. Much of online sales are still for physical goods to be delivered to an address. For that, a postal code or zip code is usually required.

No postal code?

I say usually because there are some countries that do not use postal codes at all. An ecommerce site that expects to ship products to buyers in one of those countries needs to allow for an empty postal code at checkout time. Otherwise, customers may leave thinking they aren’t welcome there. The more creative among them will make up something to put in there, such as “00000” or “99999” or “NONE”.

Someone has helpfully assembled and maintains a machine-readable (in Ruby, easily convertible to JSON or other formats) list of the countries that don’t require a postal code. You may be surprised to see on the list such countries as Hong Kong, Ireland, Panama, Saudi Arabia, and South Africa. Some countries on the list actually do have postal codes but do not require them or commonly use them.

Do you really need the customer’s address?

When selling both downloadable and shipped products, it would be nice to not bother asking the customer for an address at all. Unfortunately even when there is no shipping address because there’s nothing to ship, the billing address is still needed if payment is made by credit card through a normal credit card payment gateway—as opposed to PayPal, Amazon Pay, Venmo, Bitcoin, or other alternative payment methods.

The credit card Address Verification System (AVS) allows merchants to ask a credit card issuing bank whether the mailing address provided matches the address on file for that credit card. Normally only two parts are checked: (1) the street address numeric part, for example, “123” if “123 Main St.” was provided; (2) the zip or postal code, normally only the first 5 digits for US zip codes, and often non-US postal code AVS doesn’t work at all with non-US banks.

Before sending the address to AVS, validating the format of postal codes is simple for many countries: 5 digits in the US (allowing an optional -nnnn for ZIP+4), and 4 or 5 digits in most others countries—see the Wikipedia List of postal codes in various countries for a high-level view. Canada is slightly more complicated: 6 characters total, alternating a letter followed by a number, formally with a space in the middle, like K1A 0B1 as explained in Wikipedia’s components of a Canadian postal code.

So most countries’ postal codes can be validated in software with simple regular expressions, to catch typos such as transpositions and missing or extra characters.

UK postcodes

The most complicated postal codes I have worked with is the United Kingdom’s, because they can be from 5 to 7 characters, with an unpredictable mix of letters and numbers, normally formatted with a space in the middle. The benefit they bring is that they encode a lot of detail about the address, and it’s possible to catch transposed character errors that would be missed in a purely numeric postal code. The Wikipedia article Postcodes in the United Kingdom has the gory details.

It is common to use a regular expression to validate UK postcodes in software, and many of these regexes are to some degree wrong. Most let through many invalid postcodes, and some disallow valid codes.

We recently had a client get a customer report of a valid UK postcode being rejected during checkout on their ecommerce site. The validation code was using a regex that is widely copied in software in the wild:

[A-PR-UWYZ0-9][A-HK-Y0-9][AEHMNPRTVXY0-9]?[ABEHMNPRVWXY0-9]?[0-9][ABD-HJLN-UW-Z]{2}

(This example removes support for the odd exception GIR 0AA for simplicity’s sake.)

The customer’s valid postcode that doesn’t pass that test was W1F 0DP, in London, which the Royal Mail website confirms is valid. The problem is that the regex above doesn’t allow for F in the third position, as that was not valid at the time the regex was written.

This is one problem with being too strict in validations of this sort: The rules change over time, usually to allow things that once were not allowed. Reusable, maintained software libraries that specialize in UK postal codes can keep up, but there is always lag time between when updates are released and when they’re incorporated into production software. And copied or customized regexes will likely stay the way they are until someone runs into a problem.

The ecommerce site in question is running on the Interchange ecommerce platform, which is based on Perl, so the most natural place to look for an updated validation routine is on CPAN, the Perl network of open source library code. There we find the nice module Geo::UK::Postcode which has a more current validation routine and a nice interface. It also has a function to format a UK postcode in the canonical way, capitalized (easy) and with the space in the correct place (less easy).

It also presents us with a new decision: Should we use the basic “valid” test, or the “strict” one? This is where it gets a little trickier. The “valid” check uses a regex validation approach will still let through some invalid postcodes, because it doesn’t know what all the current valid delivery destinations are. This module has a “strict” check that uses a comprehensive list of all the “outcode” data—which as you can see if you look at that source code, is extensive.

The bulkiness of that list, and its short shelf life—the likelihood that it will become outdated and reject a future valid postcode—makes strict validation checks like this of questionable value for basic ecommerce needs. Often it is better to let a few invalid postcodes through now so that future valid ones will also be allowed.

The ecommerce site I mentioned also does in-browser validation via JavaScript before ever submitting the order to the server. Loading a huge list of valid outcodes would waste a lot of bandwidth and slow down checkout loading, especially on mobile devices. So a more lax regex check there is a good choice.

When Christmas comes

There’s no Christmas gift of a single UK postal code validation solution for all needs, but there are some fun trivia notes in the Wikipedia page covering Non-geographic postal codes:

A fictional address is used by UK Royal Mail for letters to Santa Claus:

Santa’s Grotto
Reindeerland XM4 5HQ

Previously, the postcode SAN TA1 was used.

In Finland the special postal code 99999 is for Korvatunturi, the place where Santa Claus (Joulupukki in Finnish) is said to live, although mail is delivered to the Santa Claus Village in Rovaniemi.

In Canada the amount of mail sent to Santa Claus increased every Christmas, up to the point that Canada Post decided to start an official Santa Claus letter-response program in 1983. Approximately one million letters come in to Santa Claus each Christmas, including from outside of Canada, and they are answered in the same languages in which they are written. Canada Post introduced a special address for mail to Santa Claus, complete with its own postal code:

SANTA CLAUS
NORTH POLE H0H 0H0

In Belgium bpost sends a small present to children who have written a letter to Sinterklaas. They can use the non-geographic postal code 0612, which refers to the date Sinterklaas is celebrated (6 December), although a fictional town, street and house number are also used. In Dutch, the address is:

Sinterklaas
Spanjestraat 1
0612 Hemel

This translates as “1 Spain Street, 0612 Heaven”. In French, the street is called “Paradise Street”:

Saint-Nicolas
Rue du Paradis 1
0612 Ciel

That UK postcode for Santa doesn’t validate in some of the regexes, but the simpler Finnish, Canadian, and Belgian ones do, so if you want to order something online for Santa, you may want to choose one of those countries for delivery. :)

Perl Dancer Conference 2016 Day 1

2016-11-30T00:00:00+00:00

Perl Dancer Conference Day 1

The Perl Dancer Conference is a great event, now in its third year. The event took place in the same location as last year in Vienna, Austria at the Hotel Schani Wien. For those of you who have never visited Vienna, it is a perfect place to bring the family. From visiting the beautiful parks to taking a scenic ride on the Danube River, the beautiful and historic city is known for its rich art and musical culture, and has much to offer.

I was very excited to not only attend but also give a talk this year. My talk titled “Dancing in the Clouds” also coincided with the release of 2 new Perl modules Etcd3 and Dancer2::Plugin::Etcd. This article will be the first of a 3 part series, with the final article a focus on my talk and usage examples with the new modules.

Sawyer X (Sawyer X) — A bus tour through Dancer core

The Captain of Dancer core, SawyerX, took us on a bus tour through the core functionality of Dancer2. Using practical examples of code blocks from core, he explained how different areas of the code base worked. I personally enjoyed his explanation of how hooks are implemented and created. Learning from the 1st iteration of Dancer, the second version shows maturity and stability.

Stefan Hornburg (Racke) — No Act on ACT

If you have ever taken the time to review a Perl conference’s website or even purchase tickets to attend you have no doubt been in contact with ACT. “Act (A Conference Toolkit) is a multilingual, template-driven, multi-conference website that can manage the users, talks, schedule, and payment for your conference.” While this package has been around for many years, it is somewhat dreaded because of its lack of features.

Stefan outlines his work with Interchange6::Schema and the perl.dance website painting a picture of the replacement for ACT. Utilizing Dancer2, DBIx::Class, Moo and other modern Perl tools the infrastructure outlined is very compelling. The package has a user admin, e-commerce, and even a module to print out the passes. Although he notes that this is not a plug and play replacement for ACT yet, with a bit of work and support, it could be the future of Perl conference management.

Andrew Beverly — Implementing i18n in a Dancer application using Plugin::LogReport

Andrew extended his talk last year about internationalization with the Dancer2::Plugin::LogReport module. Using great examples, he not only outlined the usage of the module, but also got into details of how the process works on a technical level. Explaining the different ways that internationalization can be done, he begins to outline how he achieved his task of deploying i18n in a Dancer app.

Theo van Hoesel — Quickstep

Theo was a great addition to the conference this year. He was able to make the event on very short notice after Dancer core Jason Crome was not able to attend due to injury. Theo outlined the Act::Voyager project briefly and the general requirements of adding user friendly features to the ACT toolkit. He also spent a good amount of time explaining the concept of web caching and how many of the existing modules failed in the task of complying with RFC7234. He then explained how all of this brought him to create HTTP::Caching and how it has “The RFC 7234 compliant brains to do caching right”. Along with this contribution part of the HTTP::Bundle, his Dancer focused Dancer2::Plugin::HTTP::Bundle was explained.

Job van Achterberg (jkva) — Dancing with Disabilities

Job’s talk was a very interesting look into how taking a considerate approach to development and small changes to your code can improve a disabled web user’s experience. Using the tools in macOS Job showed how simple things such as naming a list are reflected in a disabled users ability to get information. What I found very interesting in this presentation was how awkward the tools were to use even for an experienced pro like Job. It really made me think a lot about the challenges the disabled face in something many of us take for granted.

Jason Lewis — The Lazy Programmer’s Guide to building HTML tables in Dancer2

Jason has become a regular on the #dancer freenode IRC channel. This year he decided to travel all the way from Australia to give his presentation on his experiences replacing Crystal Reports with Dancer2 Template::Toolkit and DataTables. Although a great deal of the presentation was focused on the features of the jQuery plugin DataTables, he gave nice examples of code he used to populate reports and the hurdles he faced replacing Crystal Reports functionality. The reports looked beautiful and were very easy to convert to other data types such as PDF and CSV.

Stefan Seifert (nine) — Perl 5 and Perl 6 — a great team

Stefan is a great presence at the conference, and his fun and witty personality carried over to his presentation. After opening with a really funny skit as a reporter reading the the news, he continued to outline the current state of Perl6 and how important it is for all of us as a community to embrace the fact that we are all in this together. He reminded us of the perils of Python 3’s launch and the lack of support even today. He then began to focus on the capabilities of using Perl 5 with Perl 6 together with Inline::Perl5 and Inline::Perl6 modules. To be honest before his talk I had given Perl 6 very little time. Stefan’s talk opened my eyes to the possibilities of utilizing the two versions together and the advantages that ecosystem has.

Please stop back for links to day 2 of the conference and a breakdown of my talk outlining etcd integration with Perl and Dancer2.

Use Java along with Perl

2016-10-18T00:00:00+00:00

While working with one of our clients, I was tasked with integrating a Java project with a Perl project. The Perl project is a web application which has a specific URL for the Java application to use. To ensure that the URL is called only from the Java application, I wanted to send a special hash value calculated using the request parameters, a timestamp, and a secret value.

The Perl code calculating the hash value looks like this:

use strict;
use warnings;

use LWP::UserAgent;
use Digest::HMAC_SHA1;
use Data::Dumper;

my $uri = 'param1/param2/params3';

my $ua = LWP::UserAgent->new;
my $hmac = Digest::HMAC_SHA1->new('secret_something');

my $ts = time;

$hmac->add($_) for (split (m{/}, $uri));
$hmac->add($ts);

my $calculated_hash = $hmac->hexdigest;

My first try for calculating the same hash in the Java code looked something like this (without class/package overhead):

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import org.apache.commons.codec.binary.Hex;

public String calculateHash(String[] values) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {

    java.util.Date date= new java.util.Date();
    Integer timestamp = (int) date.getTime()/1000;

    Mac mac = Mac.getInstance("HmacSHA1");

    SecretKeySpec signingKey = new SecretKeySpec("secret_something".getBytes(), "HmacSHA1");
    mac.init(signingKey);

    for(String value: values) {
        mac.update(value.getBytes());
    }

    mac.update(timestamp.getBytes());
    byte[] rawHmac = mac.doFinal();
    byte[] hexBytes = new Hex().encode(rawHmac);
    return new String(hexBytes, "UTF-8");
}

The code looks good and successfully calculated a hash. However, using the same parameters for the Perl and Java code, they were returning different results. After some debugging, I found that the only parameter causing problems was the timestamp. My first guess was that the problem was caused by the use of Integer as the timestamp type instead of some other numeric type. I tried a few things to get around that, but none of them worked.

Another idea was to check why it works for the String params, but not for Integer. I found that Perl treats the timestamp as a string and passes a string to the hash calculating method, so I tried emulating this by converting the timestamp into a String before using the getBytes() method:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import org.apache.commons.codec.binary.Hex;

public String calculateHash(String[] values) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {

    java.util.Date date= new java.util.Date();
    Integer timestamp = (int) date.getTime()/1000;

    Mac mac = Mac.getInstance("HmacSHA1");

    SecretKeySpec signingKey = new SecretKeySpec("secret_something".getBytes(), "HmacSHA1");
    mac.init(signingKey);

    for(String value: values) {
        mac.update(value.getBytes());
    }

    mac.update(timestamp.toString().getBytes());
    byte[] rawHmac = mac.doFinal();
    byte[] hexBytes = new Hex().encode(rawHmac);
    return new String(hexBytes, "UTF-8");
}

This worked perfectly, and there were no other problems with calculating the hash in Perl and Java.

Reach customers and drive sales with MailChimp

2016-08-22T00:00:00+00:00

It’s a good idea for ecommerce stores to regularly contact their customers. This not only reminds customers that your business exists, but also allows the sharing of new products and resources that can enrich the lives of your customers and clients. One of the easiest ways to stay in touch is by using an email newsletter service, such as MailChimp.

<img alt="" border="0" height="200" src="/blog/2016/08/reach-customers-and-drive-sales-with/image-0.png" title="Freddie, the MailChimp mascot" width="191"/>

MailChimp offers the regular suite of email newsletter services: lists, campaigns, and reports—but in addition, they allow an ecommerce store to integrate sales data back into MailChimp. When you have detailed shopping statistics for each subscriber, it opens new possibilities for customized marketing campaigns.

Endless possibilities

For example, imagine you have an email mailing list with 1,000 recipients. Instead of mailing the same generic newsletter to each subscriber, what if you could segment the list to identify your 100 best customers, and email them a special campaign?

Additional ideas could include:

Reach out to inactive subscribers, offering a coupon
Invite your best customers to a secret sale
Re-engage customers who placed items in their cart, but left without purchasing
Offer complementary products to purchasers of Product X

Automatic marketing

Once your store has sales data for subscribers, and you’ve decided on the campaigns you want to run with this data, the next step is to automate the process. This is where MailChimp’s Automation feature comes in. Spend some time up-front to craft the automated campaigns, then sit back and let MailChimp run them for you; day in, day out.

Steps to implement

There are several off-the-shelf integrations for ecommerce stores, including Magento and BigCommerce.

Users of Perl and Interchange can use our newly-released toolsets of the Mail::Chimp3 CPAN module and the integration for Interchange5.

Most businesses already have an email newsletter. Hopefully, you are sending regular email campaigns with it. This is a great first step. Going beyond this to segment your email list and reach out to these segments with relevant information to each of them, is the next step. Not only can this increase your sales, but it also respects your clients’ and customers’ time and preferences. It’s a win-win for all.

Additional resource: MailChimp for Online Sellers

Report on The Perl Conference 2016

2016-07-27T00:00:00+00:00

In June, I traveled to Orlando, Florida to attend the event formerly known as Yet Another Perl Conference (or YAPC::NA), now known as The Perl Conference. This was my second time in a row to attend this conference (after my first attendance back in 2007).

Conferences are a great place to learn how others are using various tools, hear about new features, and interact with the community. If you are speaking, it’s a great opportunity to brush up on your subject, which was true for me in the extreme, as I was able to give a talk on the PostgreSQL database, which I hadn’t used in a long time (more on that later).

The conference name

The event organizers were able to license the name The Perl Conference from O’Reilly Media, as O’Reilly doesn’t hold conferences by this name anymore. This name is now preferred over “YAPC” as it is more friendly to newcomers and more accurately describes the conference. More on the name change.

Notes from the conference

Over the three days of the conference, I was able to take in many talks. Here are some of my more interesting notes from various sessions:

Yes, this is happening @YAPCNA

— Jason Hall (@JayceHall) June 22, 2016

MetaCPAN is the best way to browse and search for Perl modules. Anyone can help with development of this fine project, via their GitHub.
Ricardo Signes says “use subroutine signatures!” They are “experimental”, but are around to stay.
Perl6 is written in Perl6 (and something called “Not Quite Perl”). This allows one to read the source to figure out how something is done. (There were many talks on Perl6, which is viewed as a different programming language, not a replacement for Perl5.)
jq is a command-line utility that can pretty-print JSON (non Perl, but nice!)
Ricardo Signes gave a talk on encoding that was over my head, but very interesting.
The presenter of Emacs as Perl IDE couldn’t attend, so Damian Conway spoke on VIM as Perl IDE (photo above)

From John Anderson’s talk:

Just say “no” to system Perl. Use plenv or the like.
There’s a DuckDuckGo bang command for searching MetaCPAN: !cpan [module]
Use JSON::MaybeXS over the plain JSON module.
Use Moo for object-oriented programming in Perl, or Moose if you must.
Subscribe to Perl Weekly
Submit your module on PrePAN first, to receive feedback before posting to CPAN.

Lee Johnson gave a talk called Battling a legacy schema with DBIx::Class (video). Key takeaways:

When should I use DBIC?
Something that has grown organically could be considered “legacy,” as it accumulates technical debt
With DBIC, you can start to manage that debt by adding relationships to your model, even if they aren’t in your database
RapidApp’s rdbic can help you visualize an existing database

D. Ruth Bavousett spoke on Perl::Critic, which is a tool for encouraging consistency. Basically, Perl::Critic looks at your source code, and makes suggestions for improvement, etc. These suggestions are known as “policies” and can be configured to enable or disable any of them, or to even write new policies. One suggestion was to create a Git hook to run the perlcritic command at the time code is committed to the source code repository (possibly using App::GitHooks). End Point has its own perlcritic configuration, which I have started trying to use more.

Logan Bell shared Strategies for leading a remote team. Some of the tools and techniques he uses include:

tmate for terminal sharing
HipChat, with a chat room just for complaining called “head to desk”
Holds one-on-one meetings every Monday for those he works with and directs
Has new team members work on-site with another team member their first week or so, to help understand personalities that don’t often come across well over chat
Tries to have in-person meetings every quarter or at least twice a year, to bring the team together

My talk

Finally, my own talk was titled Stranger in a Strange Land: PostgreSQL for MySQL users (video). I hadn’t used Postgres in about seven years, and I wanted to get re-acquainted with it, so naturally, I submitted a talk on it to spur myself into action!

In my talk, I covered:

the history of MySQL and Postgres
how to pronounce “PostgreSQL”
why one might be preferred over the other
how to convert an existing database to Postgres
and some tools and tips.

I enjoyed giving this talk, and hope others found it helpful. All in all, The Perl Conference was a great experience, and I hope to continue attending in the future!

All videos from this year’s conference

2015 Perl Dancer Conference videos

2015-12-30T00:00:00+00:00

The 2015 Perl Dancer Conference has recently released the presentation videos. This year the conference was hosted in beautiful Vienna, Austria. Josh Lavin and I were both honored to attend the conference as well as give talks. Earlier, Josh wrote summaries of the conference:

Conference Recap

Conference Presentations

SpaceCamps “The Final Frontier”

I gave a talk exploring new technologies for End Point’s own DevCamps development tool. During the presentation I detailed my research into containers and what a cloud-based development environment might look like.

SpaceCamps Presentation Video

AngularJS & Dancer for Modern Web Development

Josh detailed his experience migrating legacy applications utilizing Dancer, AngularJS, and modern Perl techniques. Josh highlighted the challenges he faced during the process, as well as lessons he learned along the way.

AngularJS & Dancer for Modern Web Development Presentation Video

Lightning Talks

Josh and I both gave short “lightning talks.” Josh’s was on Writing Unit Tests for a Legacy App (Interchange 5), and mine was on Plack & Interchange 5.

To review the rest of the presentations please checkout the Perl Dancer Conference YouTube channel.

Summary

The Perl Dancer community continues to flourish and the conference this year hosted a record 5 core Dancer developers. Dancer is about to release the finalized version of its long awaited plugin infrastructure for Dancer2. A lot of work on this was completed during the conference. Being an organizer of the conference, it brings me great joy to see this success. This news along with the release of Perl 6, I am certain 2016 will be a wonderful year for not only Dancer but the entire Perl community.

Top 7 Funniest Perl Modules

2015-11-04T00:00:00+00:00

And now for something completely different …

Programmers in general, and Perl programmers in particular, seem to have excellent, if warped, senses of humor. As a result, the CPAN library is replete with modules that have oddball names, or strange and wonderful purposes, or in some delightful cases—both!

Let’s take a look.

1. Bone::Easy

I’m going to take the coward’s way out on this one right away. Go see for yourself, or don’t.

2. Acme::EyeDrops

Really, anything in the Acme::* (meaning “perfect”) namespace is just programmer-comedy gold, depending on what you find amusing and what is just plain forehead-smacking stupid to you. This one allows you to transform your Perl programs (small ones work better) from this:

print "hello world\n";

to this:

jeff@Wyvern:~$ perl bin/eyedrops.pl
1 shapes completed.
use re 'eval';
                                      ''=~('('.'?'
           .'{'.(                   '`'|'%').("\["^
        '-').('`'|                '!').('`'|',').'"'
 .('['^'+')  .('['^              ')').('`'|')').('`'|
'.').('['^'/').('{'^            '[').'\\'.'"'.('`'|'('
).('`'|'%').('`'|',')          .('`'|',').('`'|"\/").(
  '{'^'[').('['^',').        ('`'|'/').('['^')').("\`"|
       ',').('`'|'$')      .'\\'.'\\'.('`'|'.').'\\'.'"'
      .';'.('!'^'+')     .'"'.'}'.')');$:='.'^'~';$~='@'|
     '(';$^=')'^'[';   $/='`'|'.';$,='('^'}';$\='`'|'!';$:
    =')'^'}';$~='*'   |'`';$^='+'^'_';$/='&'|'@';$,='['&'~';
    $\=','^"\|";$:=  '.'^'~';$~='@'|'(';$^=')'^'[';$/='`'|'.'
    ;$,='('^'}';$\  ='`'|'!';$:=')'^'}';$~='*'|'`';$^='+'^'_'
    ;$/='&'|'@';$,  ='['&'~';$\=','^'|';$:='.'^'~';$~='@'|'(';
    $^=')'^'[';$/='`'|'.';$,='('^'}';$\='`'|'!';$:=')'^'}';$~=
    '*'|'`';$^='+'^'_';$/='&'|'@';$,='['&'~';$\=','^'|';$:='.'^
     '~';$~='@'|'(';$^=')'^'[';$/='`'|'.';$,='('^'}';$\='`'|'!'
     ;$:=')'^'}';$~='*'|'`';$^='+'^'_';$/='&'|'@';$,='['&'~';$\=
      ','^'|';$:='.'^'~';$~='@'|'(';$^=')'^'[';$/='`'|'.';$,='('^
       '}';$\='`'|'!';$:=')'^'}';$~='*'|'`';$^='+'^'_';$/='&'|'@'
        ;$,='['&'~';$\=','^'|';$:='.'^'~';$~='@'|'(';$^="\)"^ '['
         ;$/='`'|'.';$,='('^'}';$\='`'|'!';$:=')'^'}';$~='*'  |((
          '`'));$^='+'^'_';$/='&'|'@';$,='['&'~';$\ =','^'|'  ;$:
            ='.'^'~';$~='@'|'(';$^=')'^'[';$/='`'|  '.';$,=   '('
             ^'}'; $\='`'|'!';$:=')'^'}';$~="\*"|   '`';$^=   '+'
                   ^'_';$/='&'|'@';$,='['&'~';$\    =(',')^   '|'
                   ;$:='.'^    '~';$~='@' |"\(";     $^=')'   ^+
                   '[';$/=     '`'|'.';   $,='('     ^"\}";  $\
                   =('`')|     "\!";$:=   "\)"^       "\}";  (
                   ($~))=      '*'|'`';   ($^)         ='+'
                   ^"\_";     $/=('&')|   '@';         ($,)
                    ='['&     "\~";$\=    ','^         '|';
                    ($:)=     '.'^'~'     ;$~=         '@'|
                    '(';     $^=')'       ^'['         ;$/=
                    '`'|     '.'          ;$,=         '('^
                    '}';      $\=         '`'          |((
                    '!'        ));       $:=           ')'
                    ^((         '}'     ));            $~=
                    '*'          |((   '`'              ))
                    ;(             ($^))=               ((
                    ((              '+'))               ))
                    ^+             "\_";$/=             ((
                   '&'            ))|+ "\@";            $,
                  =((           '['))&  '~';           $\=
                 ','^         "\|";$:=   '.'          ^'~'
               ;($~)=                                ('@')|
             "\(";$^=                               ')'^'['

Oh, that’s not just a picture of a camel. That’s actual Perl code; you can run that, and it executes in the exact same way as the original one-liner. So much more stylish. Plus, you can impress your boss/cow-orker/heroic scientist boyfriend.

3. common::sense

This one makes the list because (a) it is just so satisfying to see

use common::sense;

atop a Perl program, and (b) a citation of this on our company IRC chat is what planted the seed for this article.

Another is sanity.pm, as in:

use sanity;

Seems like a good approach.

4. Silly::Werder

Not a terribly interesting name, but it produces some head-scratching output. For instance,

Broringers isess ailerwreakers paciouspiris dests bursonsinvading buggers companislandet despa ascen?

I suppose you might use this to generate some Lorem ipsum-type text, or maybe temporary passwords? Dialog for your science fiction novel?

5. Moose*

Any module with the word “Moose” in it. “Moose” is a funny word.

6. D::oh

The humor here is a bit obscure: you have to have been around for Perl4-style namespace addressing, when you would have had to load this via:

use D'oh;

7. your

As in:

use your qw($wits %head @tools);

Here the name is the funny bit; the module itself is all business.

Well, that seems like enough to get you started. If you find others, post them here in the comments!

AngularJS & Dancer for Modern Web Development

2015-10-30T00:00:00+00:00

At the Perl Dancer Conference 2015, I gave a talk on AngularJS & Dancer for Modern Web Development. This is a write-up of the talk in blog post form.

Legacy Apps

It’s a fact of life as a software developer that a lot of us have to work with legacy software. There are many older platforms out there, still being actively used today, and still supporting valid businesses. Thus, legacy apps are unavoidable for many developers. Eventually, older apps are migrated to new platforms. Or they die a slow death. Or else the last developer maintaining the app dies.

Oh, to migrate

It would be wonderful if I could migrate every legacy app I work on to something like Perl Dancer. This isn’t always practical, but a developer can dream, right?

Of course, every circumstance is different. At the very least, it is helpful to consider ways that old apps can be migrated. Using new technologies can speed development, give you new features, and breathe new life into a project, often attracting new developers.

As I considered how to prepare my app for migration, here are a few things I came up with:

Break out of the Legacy App Paradigm
- Consider that there are better ways to do things than the way they’ve always been done
Use Modern Perl
- Use object-oriented Perl
- Use tests
- Use CPAN modules and don’t reinvent the wheel
Organize business logic
- Try to avoid placing logic in front-end code

You are in a legacy codebase

I explored how to start using testing, but I soon realized that this requires methods or subroutines. This was the sad realization that up till now, my life as a Perl programmer had been spent doing scripting. My code wasn’t testable, and looked like a relic with business logic strewn about.

Change

I set out to change my ways. I started exploring object-oriented Perl using Moo, since Dancer2 uses Moo. I started trying to write unit tests, and started to use classes and methods in my code.

Essentially, I began breaking down problems into smaller problems. This, after all, is how the best methods are written: short and simple, that do just one thing. I found that writing code this way was fun.

Crash

I quickly realized that I wasn’t able to run tests in my Legacy App, as it couldn’t be called from the command line (at least not out of the box, and not without weird hacks). Thus, if my modules depended on Legacy App code, I wouldn’t be able to call them from tests, because I couldn’t run these tests from the shell.

This led me to a further refinement: abstract away all Legacy App-specific code from my modules. Or, at least all the modules I could (I would still need a few modules to rely on the Legacy App, or else I wouldn’t be using it it all). This was a good idea, it turned out, as it follows the principle of Separation of Concerns, and the idea of Web App + App, which was mentioned frequently at the conference.

Now I was able to run tests on my modules!

Move already

This whole process of “getting ready to migrate” soon began to look like yak shaving. I realized that I should have moved to Dancer earlier, instead of trying to do weird hacks to get the Legacy App doing things as Dancer would do them.

However, it was a start, a step in the right direction. Lesson for me, tip for you.

And, the result was that my back-end code was all the more ready for working with Dancer. I would just need to change a few things, and presto! (More on this below.)

Front-End

With the back-end looking tidier, I now turned to focus on the front-end. There was a lot of business logic in my front-end code that needed to be cleaned up.

Here is an example of my Legacy App front-end code:

@_TOP_@
<h1>[scratch page_title]</h1>
[perl]
   my $has_course;
   for (grep {$_->{mv_ib} eq 'course'} @$Items) {
      $has_course++;
   }
   return $has_course ? '<p>You have a course!</p>' : '';
[/perl]
<button>Buy [if cgi items]more[else]now[/else][/if]</button>
@_BOTTOM_@

As you can see, the Legacy App allowed the embedding of all sorts of code into the HTML page. I had Legacy App tags (in the brackets), plus something called “embedded perl”, plus regular HTML. Add all this together and you get Tag Soup.

This kind of structure won’t look nice if you attempt to view it on your own machine in a web browser, absent from the Legacy App interpreting it. But let’s face it, this code doesn’t look nice anywhere.

Separation of Concerns

I thought about how to apply the principle of Separation of Concerns to my front-end code. One thing I landed on, which isn’t a new idea by any means, is the use of “HTML + placeholders”, whereby I would use some placeholders in my HTML, to be later replaced and filled in with data. Here is my first attempt at that:

@_TOP_@
[my-tag-attr-list
    page_title="[scratch page_title]"
    has_course="[perl] ... [/perl]"
    buy_phrase="Buy [if cgi items]more[else]now[/else][/if]"
]

    <h1>{PAGE_TITLE}</h1>
    {HAS_COURSE?}<p>You have a course!</p>{/HAS_COURSE?}
    <button>{BUY_PHRASE}</button>

[/my-tag-attr-list]
@_BOTTOM_@

What I have here uses the Legacy App’s built-in placeholder system. It attempts to set up all the code in the initial “my-tag-attr-list”, then the HTML uses placeholders (in braces) which get replaced upon the page being rendered. (The question-mark in the one placeholder is a conditional.)

This worked OK. However, the logic was still baked into the HTML page. I wondered how I could be more ready for Dancer? (Again, I should have just gone ahead and migrated.) I considered using Template::Toolkit, since it is used in Dancer, but it would be hard to add to my Legacy App.

Enter AngularJS (or your favorite JavaScript framework)

AngularJS is a JavaScript framework for front-end code. It displays data on your page, which it receives from your back-end via JSON feeds. This effectively allows you to separate your front-end from your back-end. It’s almost as if your front-end is consuming an API. (Novel idea!)

After implementing AngularJS, my Legacy App page looked like this (not showing JavaScript):

@_TOP_@
<h1 ng-bind="page.title"></h1>
<p ng-if="items.course">You have a course!</p>
<button ng-show="items">Buy more</button>
<button ng-hide="items">Buy now</button>
@_BOTTOM_@

Now all my Legacy App is doing for the front-end is basically “includes” to get the header/footer (the TOP and BOTTOM tags). The rest is HTML code with ng- attributes. These are what AngularJS uses to “do” things.

This is much cleaner than before. I am still using the Legacy App back-end, but all it has to do is “routing” to call the right module and deliver JSON (and do authentication).

Here’s a quick example of how the JavaScript might look:

<html ng-app="MyApp">
...
<script src="angular.min.js"></script>
<script>
  angular.module / factory / controller
  $scope.items = ...;
</script>
</html>

This is very simplified, but via its modules/factories/controllers, the AngularJS code handles how the JSON feeds are displayed in the page. It pulls in the JSON and can massage it for use by the ng- attributes, etc.

I don’t have to use AngularJS to do this—I could use a Template::Toolkit template delivered by Dancer, or any number of other templating systems. However, I like this method, because it doesn’t require a Perl developer to use. Rather, any competent JavaScript developer can take this and run with it.

Migration

Now the migration of my entire app to Dancer is much easier. I gave it a whirl with a handful of routes and modules, to test the waters. It went great.

For my modules that were the “App” (not the “Web App” and dependent on the Legacy App), very few changes were necessary. Here is an example of my original module:

package MyApp::Feedback;
use MyApp;
my $app = MyApp->new( ... );
sub list {
    my $self = shift;
    my $code = shift
        or return $app->die('Need code');
    my $rows = $app->dbh($feedback_table)->...;
    return $rows;
}

You’ll see that I am using a class called MyApp. I did this to get a custom die and a database handle. This isn’t really the proper way to do this (I’m learning), but it worked at the time.

Now, after converting that module for use with Dancer:

package MyApp::Feedback;
use Moo;
with MyApp::HasDatabase;
sub list {
    my $self = shift;
    my $code = shift
        or die 'Need code';
    my $rows = $self->dbh->...;
    return $rows;
}

My custom die has been replaced with a Perl die. Also, I am now using a Moo::Role for my database handle. And that’s all I changed!

Before

The biggest improvements were in things that I “stole” from Dancer. (Naturally, Dancer would do things better than I.) This is my Legacy App’s route for displaying and accepting feedback entries. It does not show any authentication checks. It handles feeding back an array of entries for an item (“list”), a single entry (GET), and saving an entry (POST):

sub _route_feedback {
    my $self = shift;
    my (undef, $sub_action, $code) = split '/', $self->route;
    $code ||= $sub_action;
    $self->_set_status('400 Bad Request');   # start with 400
    my $feedback = MyApp::Feedback->new;
    for ($sub_action) {
        when ("list") {
            my $feedbacks = $feedback->list($code);
            $self->_set_tmp( to_json($feedbacks) );
            $self->_set_path('special/json');
            $self->_set_content_type('application/json; charset=UTF-8');
            $self->_set_status('200 OK') if $feedbacks;
        }
        default {
            for ($self->method) {
                when ('GET') {
                    my $row = $feedback->get($code)
                        or return $self->_route_error;
                    $self->_set_tmp( to_json($row) );
                    $self->_set_path('special/json');
                    $self->_set_content_type('application/json; charset=UTF-8');
                    $self->_set_status('200 OK') if $row;
                }
                when ('POST') {
                    my $params = $self->body_parameters
                        or return $self->_route_error;
                    $params = from_json($params);
                    my $result = $feedback->save($params);
                    $self->_set_status('200 OK') if $result;
                    $self->_set_path('special/json');
                    $self->_set_content_type('application/json; charset=UTF-8');
                }
            }
        }
    }
}

After

Here are those same routes in Dancer:

prefix '/feedback' => sub {
    my $feedback = MyApp::Feedback->new;
    get '/list/:id' => sub {
        return $feedback->list( param 'id' );
    };
    get '/:code' => sub {
        return $feedback->get( param 'code' );
    };
    post '' => sub {
        return $feedback->save( scalar params );
    };
};

Dancer gives me a lot for free. It is a lot simpler. There’s still no authentication shown here, but everything else is done. (And I can use an authentication plugin to make even that easy.)

TMTOWTDI

For the front-end, we have options on how to use Dancer. We could have Dancer deliver the HTML files that contain AngularJS. Or, we could have the web server deliver them, as there is nothing special about them that says Dancer must deliver them. In fact, this is especially easy if our AngularJS code is a Single Page App, which is a single static HTML file with AngularJS “routes”. If we did this, and needed to handle authentication, we could look at using JSON Web Tokens.

Now starring Dancer

In hindsight, I probably should have moved to Dancer right away. The Legacy App was a pain to work with, as I built my own Routing module for it, and I also built my own Auth checking module. Dancer makes all this simpler.

In the process, though, I learned something…

Dancer is better?

I learned you can use tools improperly. You can do Dancer “wrong”. You can write tag soup in anything, even the best modern tools.

You can stuff all your business logic into Template::Toolkit tags. You can stuff logic into Dancer routes. You can do AngularJS “wrong” (I probably do).

Dancer is better:

Dancer is better when (thanks to Matt S Trout for these):

Routes contain code specific to the Web.
Routes call non-Dancer modules (where business logic lives; again, Web App + App).
The route returns the data in the appropriate format.

These make it easy to test. You are effectively talking to your back-end code as if it’s an API. Because it is.

The point is: start improving somewhere. Maybe you cannot write tests in everything, but you can try to write smart code.

Lessons learned

Separate concerns
Keep it testable
Just start somewhere

The end. Or maybe the beginning…

Perl Dancer Conference 2015 Report—Conference Days

2015-10-30T00:00:00+00:00

In my last post, I shared about the Training Days from the Perl Dancer 2015 conference, in Vienna, Austria. This post will cover the two days of the conference itself.

While there were several wonderful talks, Gert van der Spoel did a great job of writing recaps of all of them (Day 1, Day 2), so here I’ll cover the ones that stood out most to me.

Day One

Dancer Conference, by Alexis Sukrieh (used with permission)

Sawyer X spoke on the State of Dancer. One thing mentioned, which came up again later in the conference, was: Make the effort, move to Dancer 2! Dancer 1 is frozen. There have been some recent changes to Dancer:

Middlewares for static files, so these are handled outside of Dancer
New Hash::MultiValue parameter keywords (route_parameters, query_parameters, body_parameters; covered in my earlier post)
Delayed responses (asynchronous) with delayed keyword:
- Runs on the server after the request has finished.
- Streaming is also asynchronous, feeding the user chunks of data at a time.

Items coming soon to Dancer may include: Web Sockets (supported in Plack), per-route serialization (currently enabling a serializer such as JSON affects the entire app—later on, Russell released a module for this, which may make it back into the core), Dancer2::XS, and critic/linter policies.

Thomas Klausner shared about OAuth & Microservices. Microservices are a good tool to manage complexity, but you might want to aim for “monolith first”, according to Martin Fowler, and only later break up your app into microservices. In the old days, we had “fat” back-ends, which did everything and delivered the results to a browser. Now, we have “fat” front-ends, which take info from a back-end and massage it for display. One advantage of the microservice way of thinking is that mobile devices (or even third parties) can access the same APIs as your front-end website.

OAuth allows a user to login at your site, using their credentials from another site (such as Facebook or Google), so they don’t need a password for your site itself. This typically happens via JavaScript and cookies. However, to make your back-end “stateless”, you could use JSON Web Tokens (JWT). Thomas showed some examples of all this in action, using the OX Perl module.

One thing I found interesting that Thomas mentioned: Plack middleware is the correct place to implement most of the generic part of a web app. The framework is the wrong part. I think this mindset goes along with Sawyer’s comments about Web App + App in the Training Days.

Mickey Nasriachi shared his development on PONAPI, which implements the JSON API specification in Perl. The JSON API spec is a standard for creating APIs. It essentially absolves you from having to make decisions about how you should structure your API.

Panorama from the south tower of St. Stephen’s cathedral, by this author

Gert presented on Social Logins & eCommerce. This built on the earlier OAuth talk by Thomas. Here are some of the pros/cons to social login which Gert presented:

Pros—customer:
- Alleviates “password fatigue”
- Convenience
- Brand familiarity (with the social login provider)
Pros—eCommerce website:
- Expected customer retention
- Expected increase in sales
- Better target customers
- “Plug & Play” (if you pay)—some services exist to make it simple to integrate social logins, where you just integrate with them, and then you are effectively integrated with whatever social login providers they support. These include Janrain and LoginRadius
Cons—customer:
- Privacy concerns (sharing their social identity with your site)
- Security concerns (if their social account is hacked, so are all their accounts where they have used their social login)
- Confusion (especially on how to leave a site)
- Usefulness (no address details are provided by the social provider in the standard scope, so the customer still has to enter extra details on your site)
- Social account hostages (if you’ve used your social account to login elsewhere, you are reluctant to shut down your social account)
Cons—eCommerce website:
- Legal implications
- Implementation hurdles
- Usefulness
- Provider problem is your problem (e.g., if the social login provider goes down, all your customers who use it to login are unable to login to your site)
- Brand association (maybe you don’t want your site associated with certain social sites)
Cons—social provider:
- ???

Šimun Kodžoman spoke on Dancer + Meteor = mobile app. Meteor is a JavaScript framework for both server-side and client-side. It seems one of the most interesting aspects is you can use Meteor with the Android or iOS SDK to auto-generate a true mobile app, which has many more advantages than a simple HTML “app” created with PhoneGap. Šimun is using Dancer as a back-end for Meteor, because the server-side Meteor aspect is still new and unstable, and is also dependent on MongoDB, which cannot be used for everything.

End Point’s own Sam Batschelet shared his work on Space Camp, a new container-based setup for development environments. This pulls together several pieces, including CoreOS, systemd-nspawn, and etcd to provide a futuristic version of DevCamps.

Day Two

Conference goers, by Sam
(used with permission)

Andrew Baerg spoke on Taming the 1000-lb Gorilla that is Interchange 5. He shared how they have endeavored to manage their Interchange development in more modern ways, such as using unit tests and DBIC. One item I found especially interesting was the use of DBIx::Class::Fixtures to allow saving bits of information from a database to keep with a test. This is helpful when you have a bug from some database entry which you want to fix and ensure stays fixed, as databases can change over time, and without a “fixture” your test would not be able to run.

Russell Jenkins shared HowTo Contribute to Dancer 2. He went over the use of Git, including such helpful commands and tips as:

git status –short –branch
Write good commit messages: one line summary, less than 50 characters; longer description, wrapped to 72 characters; refer to and/or close issues
Work in a branch (you shall not commit to master)
“But I committed to master” –> branch and reset
git log –oneline –since=2.weeks
git add –fixup
The use of branches named with “feature/whatever” or “bugfix/whatever” can be helpful (this is Russell’s convention)

There are several Dancer 2 issues tagged “beginner suitable”, so it is easy for nearly anyone to contribute. The Dancer website is also on GitHub. You can even make simple edits directly in GitHub!

It was great to have the author of Dancer, Alexis Sukrieh, in attendance. He shared his original vision for Dancer, which filled a gap in the Perl ecosystem back in 2009. The goal for Dancer was to create a DSL (Domain-specific language) to provide a very simple way to develop web applications. The DSL provides “keywords” for use in the Dancer app, which are specific to Dancer (basically extra functionality for Perl). One of the core aspects of keeping it simple was to avoid the use of $self (a standby of object-oriented Perl, one of the things that you just “have to do”, typically).

Alexis mentioned that Dancer 1 is frozen—Dancer 2 full-speed ahead! He also shared some of his learnings along the way:

Fill a gap (define clearly the problem, present your solution)
Stick to your vision
Code is not enough (opensource needs attention; marketing matters)
Meet in person (collaboration is hard; online collaboration is very hard)
Kill the ego—you are not your code

While at the conference, Alexis even wrote a Dancer2 plugin, Dancer2::Plugin::ProbabilityRoute, which allows you to do A/B Testing in your Dancer app. (Another similar plugin is Dancer2::Plugin::Sixpack.)

Also check out Alexis’ recap.

Finally, I was privileged to speak as well, on AngularJS & Dancer for Modern Web Development. Since this post is already pretty long, I’ll save the details for another post.

Summary

In summary, the Perl Dancer conference was a great time of learning and building community. If I had to wrap it all up in one insight, it would be: Web App + App—that is, your application should be a compilation of: Plack middleware, Web App (Dancer), and App (Perl classes and methods).

Perl Dancer Conference 2015 Report — Training Days

2015-10-28T00:00:00+00:00

I just returned from the Perl Dancer Conference, held in Vienna, Austria. It was a jam-packed schedule of two days of training and two conference days, with five of the nine Dancer core developers in attendance.

Kohlmarkt street, Wien, by this author

If you aren’t familiar with Perl Dancer, it is a modern framework for Perl for building web applications. Dancer1 originated as a port of Ruby’s Sinatra project, but has officially been replaced with a rewrite called Dancer2, based on Moo, with Dancer1 being frozen and only receiving security fixes. The Interchange 5 e-commerce package is gradually being replaced by Dancer plugins.

Day 1 began with a training on Dancer2 by Sawyer X and Mickey Nasriachi, two Dancer core devs. During the training, the attendees worked on adding functionality to a sample Dancer app. Some of my takeaways from the training:

Think of your app as a Dancer Web App plus an App. These should ideally be two separate things, where the Dancer web app provides the URL routes for interaction with your App.
The lib directory contains all of your application. The recommendation for large productions is to separate your app into separate namespaces and classes. Some folks use a routes directory just for routing code, with lib reserved for the App itself.
It is recommended to add an empty .dancer file to your app’s directory, which indicates that this is a Dancer app (other Perl frameworks do similarly).
When running your Dancer app in development, you can use plackup -R lib bin/app.psgi which will restart the app automatically whenever something changes in lib.
Dancer handles all the standard HTTP verbs, except note that we must use del, not delete, as delete conflicts with the Perl keyword.
There are new keywords for retrieving parameters in your routes. Whereas before we only had param or params, it is now recommended to use:
- route_parameters,
- query_parameters, or
- body_parameters
- all of which can be used with ->get(‘foo’) which is always a single scalar, or ->get_all(‘foo’) which is always a list.
- These allow you to specify which area you want to retrieve parameters from, instead of being unsure which param you are getting, if identical names are used in multiple areas.

Day 2 was DBIx::Class training, led by Stefan Hornburg and Peter Mottram, with assistance from Peter Rabbitson, the DBIx::Class maintainer.

DBIx::Class (a.k.a. DBIC) is an Object Relational Mapper for Perl. It exists to provide a standard, object-oriented way to deal with SQL queries. I am new to DBIC, and it was a lot to take in, but at least one advantage I could see was helping a project be able to change database back-ends, without having to rewrite code (cue PostgreSQL vs MySQL arguments).

I took copious notes, but it seems that the true learning takes place only as one begins to implement and experiment. Without going into too much detail, some of my notes included:

Existing projects can use dbicdump to quickly get a DBIC schema from an existing database, which can be modified afterwards. For a new project, it is recommended to write the schema first.
DBIC allows you to place business logic in your application (not your web application), so it is easier to test (once again, the recurring theme of Web App + App).
The ResultSet is a representation of a query before it happens. On any ResultSet you can call ->as_query to find the actual SQL that is to be executed.
DBIx::Class::Schema::Config provides credential management for DBIC, and allows you to move your DSN/username/password out of your code, which is especially helpful if you use Git or a public GitHub.
DBIC is all about relationships (belongs_to, has_many, might_have, and has_one). many_to_many is not a relationship per se but a convenience.
DBIx::Class::Candy provides prettier, more modern metadata, but cannot currently be generated by dbicdump.
For deployment or migration, two helpful tools are Sqitch and DBIx::Class::DeploymentHandler. Sqitch is better for raw SQL, while DeploymentHandler is for DBIC-managed databases. These provide easy ways to migrate, deploy, upgrade, or downgrade a database.
Finally, RapidApp can read a database file or DBIC schema and provide a nice web interface for interacting with a database. As long as you define your columns properly, RapidApp can generate image fields, rich-text editors, date-pickers, etc.

The training days were truly like drinking from a firehose, with so much good information. I am looking forward to putting this into practice!

Stay tuned for my next blog post on the Conference Days.

YAPC::NA 2015 Conference Report

2015-09-18T00:00:00+00:00

In June, I attended the Yet Another Perl Conference (North America), held in Salt Lake City, Utah. I was able to take in a training day on Moose, as well as the full 3-day conference.

The Moose Master Class (slides and exercises here) was taught by Dave Rolsky (a Moose core developer), and was a full day of hands-on training and exercises in the Moose object-oriented system for Perl 5. I’ve been experimenting a bit this year with the related project Moo (essentially the best two-thirds of Moose, with quicker startup), and most of the concepts carry over, with just slight differences.

Moose and Moo allow the modern Perl developer to quickly write OO Perl code, saving quite a bit of work from the older “classic” methods of writing OO Perl. Some of the highlights of the Moose class include:

Sub-classing is discouraged; this is better done using Roles
Moose eliminates more typing; more typing can often equal more bugs
Using namespace::autoclean at the top is a best practice, as it cleans up after Moose
Roles are what a class does, not what it is. Roles add functionality.
Use types with MooseX::Types or Type::Tiny (for Moo)
Attributes can be objects (see slide 231)

Additional helpful resources for OO Perl and Moo.

At the YAPC::NA conference days, I attended all joint sessions, and breakout sessions that piqued my interest. Here are some of the things I noted:

The author of Form::Diva gave a lightning talk (approx. 5 minutes) about this module, which allows easier HTML form creation. I was able to chat with the author during a conference mixer, and the next time I need a long HTML form, I will be giving this a try.
One lightning talk presenter suggested making comments stand out, by altering your editor’s code highlight colors. Comments are often muted, but making them more noticeable helps developers, as comments are often important guides to the code.
plenv (which allows one to install multiple versions of Perl) can remember which Perl version you want for a certain directory (plenv local)
pinto is useful for managing modules for a project
Sawyer did a talk on web scraping in which he demonstrated the use of Web::Query, which provides jQuery-like syntax for finding elements in the page you wish to scrape. There are many tools for web scraping, but this one seems easy to use, if you know jQuery.
DBIC’s “deploy” will create new tables in a database, based on your schema. DBIx::Class::Fixtures can grab certain data into files for tests to use, so you can keep data around to ensure a bug is still fixed.

The presenter of What is this “testing” people keep talking about? did a great job researching a topic which he knew nothing about until after his talk was accepted. If there is ever a good way to learn something, it’s teaching it! Slides are here.

The talk on Docker (slides) was interesting. Highlights I noted: use busybox, then install Perl on top of busybox (you can run Plack from this Perl); Gentoo is easy to dockerize, as about half the size of Ubuntu; Perl Dockerfiles; build Perl on a local system, then copy to Docker image, in Docker file.

I attended some talks on the long-awaited Perl 6, which is apparently to be released by the end of this year. While I’m not sure how practical Perl 6 will be for a while, one of the most interesting topics was that Perl 6 knows how to do math, such as: solve for "x": x = 7 / 2. Perl 6 gets this “right”, as far as humans are concerned. It was interesting that many in attendance did not feel the answer should be “3.5”, due to what I suspect is prolonged exposure to how computers do math.

One talk not related to Perl was Scrum for One (video), which discussed how to use the principles of Scrum in one’s daily life. Helpful hints included thinking of your tasks in the User Story format: “as a $Person, I would like $Thing, so that $Accomplishment”; leave murky stories on the backlog, as you must know what “done” looks like; the current tasks should include things doable in the next week—this prevents you from worrying about all tasks in your list. Personally, I’ve started using Trello boards to implement this, such as: Done, Doing, ToDo, Later.

Finally, while a great technical conference, YAPC’s biggest strength is bringing together the Perl community. I found this evident myself, as I had the opportunity to meet another attendee from my city. We were introduced at the conference, not knowing each other previously. When you have two Perl developers in the same city, it is time to resurrect your local Perl Mongers group, which is what we did!

Perl’s CPAN is 20 years old

2015-08-17T00:00:00+00:00

(Photo by emma.kate on Flickr)

This is just a short note to celebrate the fact that the Comprehensive Perl Archive Network (CPAN) turned 20 years old yesterday!

CPAN began as a way to collect, mirror, and distribute open-source Perl packages. Over time it led to development of better packaging and module naming conventions; formed a community for feedback, testing, and contributions; and became one of the great strengths of the Perl world.

It is rare to find some needed functionality that is not available on CPAN. These days a more common problem is finding too much choice there, and needing to choose between several modules based on which are better maintained and supported on current versions of Perl, or kept current against external dependencies.

Perl does not get as much press these days as it once did, but it continues to thrive and improve. On that topic, our former co-worker Steph Skardal sent me an article called We’re still catching up to Perl by Avdi Grimm of Ruby fame. It is not an in-depth language comparison, just a brief observation to his fellow Rubyists that there is plenty to be learned from Perl. (Of course Perl has plenty to learn from Ruby and other languages too.)

Converting from Pivotal Tracker to Trello for project management

2015-08-03T00:00:00+00:00

For larger client projects, I find it helpful to maintain a list of tasks, with the ability to re-order, categorize, and mark tasks as completed. Add in the ability to share this list with coworkers or project owners, and you have a recipe for a better record and task-list for development.

I had been using Pivotal Tracker for this purpose, but I found a lot of its features were too complicated for a small team. On the simpler side, Trello offers project “boards” that meet many needs for project management. Plus, you can accomplish a lot with the free level of Trello.

No import

After being convinced that switching from Pivotal to Trello was the right move for my current project, I was dismayed to find that Trello offers no Import functionality, at least that I could find for front-end users. (They do have an API, but I didn’t relish taking time to learn this.) I could easily export my Pivotal project, but how to get those tasks into Trello cards?

One idea

In my search of Trello for an import feature, I found a feature called Email-to-board. Trello provides a custom email address for each Board you create, allowing you to send emails to this address, containing information for a new Trello card. (This email address is unique for each board, containing random letters and numbers, so only the Board owner can use it.)

What if I wrote a quick script that processed a Pivotal CSV export file, and sent an email to Trello for each row (task) in the file? The script might send out quite a few emails, but would it work? Time to try it.

Perl to the rescue

I started cooking up a simple Perl script to test the idea. With the help of some CPAN modules to easily process the CSV file and send the emails, I landed on something that worked. After running the script, each row in the CSV export became an email to my Trello board, containing the item’s title, description, estimate (difficulty level), and list of tasks required to complete it.

The script should work for most exports from Pivotal Tracker, and I have published it to my GitHub account, in case it is helpful for others who decide to move to Trello.

Find the pivotal2trello script on GitHub.

If you happen to try it, let me know if you experience any problems, or have any suggestions!

Documenting web services with Perl POD and AJAX

2015-06-26T00:00:00+00:00

Perl POD is a handy, convenient, but low-tech approach to embedded documentation. Consider a web service in Dancer:

get time => sub {
  return scalar(localtime());
};

(Disclaimer: my actual use-case of this technique was even more legacy: I was documenting Interchange Actionmaps that returned images, JSON, etc.)

Your application might have several, or even dozens of these, with various parameters, returning data in JSON or TXT or CSV or who-knows-what. I chose to document these in Perl POD (Plain Old Documentation) format, e.g.,

=pod

=head1 time

Retrieves the current time

=over 3

=item Parameters

None.

=item Example

=begin html

<script src="/js/example-time.js" type="text/javascript"></script>

=end html

=back

=cut

This block gets inserted right in-line with the web service code, so it’s immediately obvious to anyone maintaining it (and thus has the best chance of being maintained if and when the code changes!). Now I can generate an HTML page directly from my Perl code:

$ pod2html MyPackage.pm

Your output looks something like this (excerpted for clarity):

time

Retrieves the current time Parameters

None.

Example

Where the magic comes in is the JavaScript code that allows an in-line example, live and accurate, within the documentation page. You’ll actually get something more like this:

time

Retrieves the current time Parameters

None.

Example

(results appear here)

Note that the code I have below is not factored by choice; I could move a lot of it out to a common routine, but for clarity I’m leaving it all in-line. I am breaking up the script into a few chunks for discussion, but you can and should construct it all into one file (in my example, “js/example-time.js”).

/* example-time.js */
$(document).ready(
  function(){
    $('script[src$="/example-time.js"]').after(
"<form action='\"/time\"'>" +
/* Note 1 */
"<input data\"="" type='\"submit\"' value='\"Get'/>" +
"<input name="hide" none\"="" result\"="" style='\"display:' type='\"button\"' value='\"Hide'/>" +
"</form>" +
"<div id='\"time-result\"'></div>"
    );

Note 1: This being a painfully simple example of a web service, there are no additional inputs. If you have some, you would add them to the HTML being assembled into the

tag, and then using jQuery, add them below to the url parameter, or into the data structure as required by your particular web service.

This step just inserts a simple into the document. I chose to embed the form into the JavaScript code, rather than the POD, because it reduces the clutter and separates the example from the web service.

    var $form = $('form[action="/time"]');
    $form.submit(function(){
      $.ajax(
        {
          'url': $form.attr('action') /* Note 1 also */,
          'data': {},
          'dataType': 'text',
          'async': false,
          'success':
             function(data){
                 $('#time-result').html($('<pre;//>').html(data))
                     .addClass('json');
             },

Here we have a submit handler that performs a very simple AJAX submit using the form’s information, and upon success, inserts the results into a result

as a pre-formatted block. I added a “json” class which just tweaks the font and other typographic presentation a bit; you can provide your own if you wish.

I’m aware that there are various jQuery plug-ins that will handle AJAX-ifying a form, but I couldn’t get the exact behavior I wanted on my first tries, so I bailed out and just constructed this approach.

          'error':
             function(){
                 $('#time-result').html('Error retrieving data!')
                     .removeClass('json');
             },
/* */

(That stray-looking comment above is just a work-around for the syntax highlighter.)

Error handling goes here. If you have something more comprehensive, such as examining the result for error codes or messages, this is where you’d put it.

          'complete':
             function(){
                 $form.find('input[name="hide"]').show();
             }
         }
      );
      return false;
    }).find('input[type="button"]').click(function(){
      $('#time-result').html('');
    });
  }
);

And just a bit of UI kindness: we have a “hide” button to make the example go away. Some of my actual examples ran to dozens of lines of JSON output, so I wanted a way to clean up after the example.

Postgres “unsupported frontend protocol” mystery

2015-05-28T00:00:00+00:00

"Koala Portrait" by James Niland

The wonderful tail_n_mail program continues to provide me with new mysteries from our Postgres clients. One of the main functions it provides is to send an immediate email to us when an unexpected FATAL (or ERROR or PANIC) message appears in the Postgres logs. While these are often simple application errors, or deeper problems such as running out of disk space, once in a blue moon you see something completely unexpected. Some time ago, I saw a bunch of these messages appear in an email from a tail_n_mail email:

[1] From files A to B Count: 2
First: [A] 2015-12-01T06:30:00 server1 postgres[1948]
Last:  [B] 2015-12-01T06:30:00 server2 postgres[29107]
FATAL: unsupported frontend protocol 65363.19778: server supports 1.0 to 3.0

I knew what caused this error in general, but decided to get to the bottom of the problem. Before we go into the specific error, let’s review what causes this particular message to appear. When a Postgres client (such as psql or DBD::Pg) connects to Postgres, the first thing it does is to issue a startup message. One of the things included in this request is the version of the Postgres protocol the client wishes to use. Since 2003, Postgres servers have been using version 3.1. It is very rare to see a client or server that uses anything else. Because this protocol number request occurs at the very start of the connection request, non-Postgres programs often trigger this error, because the server is expecting a number at the start of the request.

We can verify this by use of a small Perl script that connects to the server, and sends an invalid protocol request:

#!/usr/bin/env perl

use strict;
use warnings;
use IO::Socket;

my $server = IO::Socket::UNIX->new('/tmp/.s.PGSQL.5432')
  or die "Could not connect!: $@";

my $packet = pack('nn', 1234,56789) . "user\0pg\0\0";
$packet = pack('N', length($packet) + 4). $packet;
$server->send($packet, 0);

After running the above program, a new error pops up in the Postgres logs as expected:

$ tail -1 /var/lib/pgsql/data/pg_log/postgres-2015-05-20.log
2015-05-21 12:00:00 EDT [unknown]@[unknown] [10281] FATAL:  unsupported frontend protocol 1234.56789: server supports 1.0 to 3.0

There is our error, as expected. The “unknown”s are because my log_line_prefix looks like this: %t %u@%d [%p]. While the timestamp (%t) and the process ID (%p) are easily filled in, the login failed, so both the user (%u) and database (%d) are still unknown.

Now on to our specific error, which you will recall is “unsupported frontend protocol 65363.19778”. The above program shows that the protocol number is sent in a specific format. Let’s use Perl to display the numbers 65363.19778 and see if there are any clues buried within it:

$ perl -e 'print pack "nn", 65363,19778'
�SMB

Some sort of unprintable character in there; let’s take a deeper look just for completeness:

$ perl -e 'print pack "nn", 65363,19778' | hd
00000000  ff 53 4d 42                                       |.SMB|
00000004

Aha! SMB is not just a random placement of three letters, it is a big clue as to what is causing this message. SMB stands for Server Message Block, and is used by a variety of things. We can guess that this is either some program randomly hitting the Postgres port without realizing what it is, or some sort of purposeful port scanner. Why would something want to connect to the port but not log in? For one, you can determine the version of Postgres without logging in.

To cut to the chase, the culprit is the nmap program. In addition to simply scanning ports, it has the ability to do a deeper inspection to determine not only what is running on each port, but what version it is as well (with the “-sV” argument). Let’s see nmap in action. We will use a non-standard Postgres port so as not to give it any additional hints about what is on that port:

$ nmap -p 5930 localhost -sV
Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-20 12:00 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000088s latency).
PORT     STATE SERVICE    VERSION
5930/tcp open  postgresql PostgreSQL DB
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
SF-Port5930-TCP:V=6.40%I=7%D=5/21%Time=504C5445%P=i686-pc-linux-emu%r(Kerb
SF:eros,85,"E\0\0\0\x84SFATAL\0C0A000\0Munsupported\x20frontend\x20protoco
SF:l\x2027265\.28208:\x20server\x20supports\x201\.0\x20to\x203\.0\0Fpostma
SF:ster\.c\0L1834\0RProcessStartupPacket\0\0")%r(SMBProgNeg,85,"E\0\0\0\x8
SF:4SFATAL\0C0A000\0Munsupported\x20frontend\x20protocol\x2065363\.19778:\
SF:x20server\x20supports\x201\.0\x20to\x203\.0\0Fpostmaster\.c\0L1834\0RPr
SF:ocessStartupPacket\0\0");

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.73 seconds

It looks like it triggered the “unsupported protocol” message, based on what was returned. Taking a peek at the Postgres 9.3 logs shows our mystery message:

$ tail -1 /var/lib/pgsql/pg9.3/pg_log/postgres-2015-05-20.log
2015-05-21 12:00:00 EDT [unknown]@[unknown] [2318] FATAL:  unsupported frontend protocol 65363.19778: server supports 1.0 to 3.0

As a final check, let’s confirm that nmap is using SMB when it runs the version check:

$ nmap localhost -p 5930 -sV --version-trace 2>/dev/null | grep SMB
Service scan sending probe SMBProgNeg to 127.0.0.1:5930 (tcp)
Service scan match (Probe SMBProgNeg matched with SMBProgNeg line 10662): 127.0.0.1:5930 is postgresql.  Version: |PostgreSQL DB|||
SF:ster\.c\0L1834\0RProcessStartupPacket\0\0")%r(SMBProgNeg,85,"E\0\0\0\x8

Bingo. Mystery solved. If you see that error in your logs, it is most likely caused by someone running nmap in version detection mode.

Interchange Loop Optimization

2015-02-09T00:00:00+00:00

It’s important to understand both how loops work in Interchange and the (very) fundamental differences between interpolating Interchange tag language (ITL) and the special loop tags (typically referred to as [PREFIX-*] in the literature). Absent this sometimes arcane knowledge, it is very easy to get stuck with inefficient loops even with relatively small loop sets. I’ll discuss both the function of loops and interpolation differences between the tag types while working through a [query] example. While all loop tags–[item-list], [loop], [search-list], and [query]–process similarly, it is to [query] where most complex loops will gravitate over time (to optimize the initiation phase of entering the loop) and where we have the most flexibility for coming up with alternative strategies to mitigate sluggish loop-processing.

Loop Processing

All loop tags are container tags in Interchange, meaning they have an open and close tag, and in between is the body. Only inside this body is it valid to define [PREFIX-] tags (notable exception of [PREFIX-quote] for the sql arg of [query]). This is because the [PREFIX-] tags are not true ITL. They are tightly coupled with the structure of the underlying rows of data and they are processed by distinct, optimized regular expressions serially. Outside the context of the row data from a result set, they are meaningless.

Moreover, the body of a loop tag is slurped into a scalar variable (as all bodies of container tags are handled via the ITL parser) and for each row in the record set of the loop, the contents are acted upon according to the [PREFIX-] tags defined within the body. The first important distinction to recognize here is, the per-row action on this scalar is limited to only the [PREFIX-] tags. The action occurring at loop time ignores any embedded ITL.

At the end of each row’s processing, the copy of the body tied to that one row is then concatenated to the results of all previous rows thus processed. For a loop with N rows (assuming no suppression by [if-PREFIX-*] conditionals) that means every instance of ITL originally placed into the loop body is now present N times in the fully assembled body string. Simple example:

[loop list='1 2 3']
[tmp junk][loop-code][/tmp]
[if scratch junk == 2]
I declare [loop-code] to be special!
[else]
Meh. [loop-code] is ordinary.
[/else]
[/if]
[/loop]

Once this result set with N=3 is processed, but before Interchange returns the results, the assembled return looks like the following string:

[tmp junk]1[/tmp]
[if scratch junk == 2]
I declare 1 to be special!
[else]
Meh. 1 is oridinary
[/else]
[/if]

[tmp junk]2[/tmp]
[if scratch junk == 2]
I declare 2 to be special!
[else]
Meh. 2 is oridinary
[/else]
[/if]

[tmp junk]3[/tmp]
[if scratch junk == 2]
I declare 3 to be special!
[else]
Meh. 3 is oridinary
[/else]
[/if]

Some important observations:

It doesn’t take much ITL to turn a loop body into a monster interpolation process. One must consider the complexity of the ITL in the body by a factor of the number of rows (total, or the “ml” matchlimit value).
ITL does nothing to short-circuit action of the [PREFIX-*] tags. Having [PREFIX-param], [PREFIX-calc], etc. inside an ITL [if] means all those loop tags parse regardless of the truth of the if condition.

ITL vs. Loop Tags

ITL maps to routines, both core and user-defined, determined at compile time. They are processed in order of discovery within the string handed to the ::interpolate_html() routine and have varied and complex attributes that must be resolved for each individual tag. Further, for many (if not most) tags, the return value is itself passed through a new call to ::interpolate_html(), acting on all embedded tags, in an action referred to as reparse. There is, relatively speaking, a good deal of overhead in processing through ::interpolate_html(), particularly with reparse potentially spawning off a great many more ::interpolate_html() calls.

Loop tags, by contrast, map to a pre-compiled set of regular expressions. In contrast to crawling the string and acting upon the tags in the order of discovery, each regex in turn is applied globally to the string. The size of the string is limited to the exact size of the single loop body, and there is no analogue to ITL’s reparse. Further, given this processing pattern, a careful observer might have noted that the order of operations can impact the structure. Specifically, tags processed earlier cannot depend on tags processed later. E.g., [PREFIX-param] processes ahead of [PREFIX-pos], and so:

[if-PREFIX-pos 2 eq [PREFIX-param bar]]

will work, but:

[if-PREFIX-param bar eq [PREFIX-pos 2]]

will not. While the above is a somewhat contrived exampled, the impacts of loop tag processing can be more easily seen in an example using [PREFIX-next]:

[PREFIX-next][PREFIX-param baz][/PREFIX-next]
Code I only want to run when baz is false, like this [PREFIX-exec foo][/PREFIX-exec] call

Because [PREFIX-next] is the absolute last loop tag to run, every other loop tag in the block is run before the next condition is checked. All [PREFIX-next] does is suppress the resulting body from the return, unlike Perl’s next, which short-circuits the remaining code in the loop block.

An Optimization Example

As long as you’re familiar with the idiosyncracies of [PREFIX-*] tags, you should make every effort to use them instead of ITL because they are substantially lighter weight and faster to process. A classic case that can yield remarkable performance gains is to directly swap an embedded [perl] or [calc] block with an equivalent [PREFIX-calc] block.

Let’s take a typical query with little consideration given to whether we use loop tags or ITL, not unlike many I’ve seen where the resource has just become unusably slow. This code originally was developed processing 50 records per page view, but the team using it has requested over time to increase that count.

[query
    list=1
    ml=500 [comment]Ouch! That's a big N[/comment]
    sql="
        SELECT *
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if global DEVELOPMENT]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if]
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
[if cgi show_inventory]
Inv:
[if cgi show_inventory eq all]
* Shipped: [either][sql-param is_shipped][or]pending[/either]
* Count: [inventory type=shipped sku=[sql-param sku]]
[/if]
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
[/if]
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

Considering this block out of context, it doesn’t seem all that unreasonable. However, let’s look at some of the pieces individually and see what can be done.

We use [if] in 3 different circumstances in the block. However, those values they test are static. They don’t change on any iteration. (We are excluding the potential of any other ITL present in the block from changing their values behind the scenes.)
[convert-date] may be convenient, but it is only one of a number of ways to address date formatting. Our database itself almost certainly has date-formatting routines, but one of the benefits of [convert-date] is you could have a mixed format underlying the data and it can make sense out of the date to some degree. So perhaps that’s why the developer has used [convert-date] here.
Good chance that stats_crunch() is pretty complicated and that’s why the developer wrote a catalog or global subroutine to handle it. Since we only want to see it in the development environment, it’d be nice if it only ran when it was needed. Right now, because of ITL happening on reparse, stats_crunch() fires for every row even if we have no intention of using its output.
We need that link to view our order, but on reparse it means ::interpolate_html() has to parse 500 [area] tags along with [either] and [cgi] x 500. All of these tags are lightweight, but parsing numbers are really going to catch up to us here.

Our goal here is to replace any ITL we can with an equivalent use of a loop tag or, absent the ability to remove ITL logically, to wrap that ITL into a subroutine that can itself be called in loop context with [PREFIX-exec]. The first thing I want to address are those [if] and [either] tags, the lowest hanging fruit:

[query
    list=1
    ml=500
    sql="
        <b>SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped</b>
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
<b>[if-sql-param is_development]</b>
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
<b>[/if-sql-param]</b>
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
<b>[if-sql-param show_inventory]</b>
Inv:
<b>[if-sql-param show_inventory eq all]</b>
* Shipped: <b>[sql-param show_inventory_shipped]</b>
* Count: [inventory type=shipped sku=[sql-param sku]]
<b>[/if-sql-param]</b>
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
<b>[/if-sql-param]</b>
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

By moving those evaluations into the SELECT list of the query, we’ve reduced the number of interpolations to arrive at those static values to 1 or, in the case of the [either] tag, 0 as we’ve offloaded the calculation entirely to the database. If is_shipped could be something perly false but not null, we would have to adjust our field accordingly, but in either case could still be easily managed as a database calculation. Moreover, by swapping in [if-sql-param is_development] for [if global DEVELOPMENT], we have kept stats_crunch() from running at all when in the production environment.

Next, we’ll consider [convert-date]:

Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]

My first attempt would be to address this similarly to the [if] and [either] conditions, and try to render the formatted date from a database function as an aliased field. However, let’s assume the underlying structure of the data varies and that’s not easily accomplished, and we still want [convert-date]. Luckily, Interchange supports that same tag as a filter, and [PREFIX-filter] is a loop tag:

Date: <b>[sql-filter convert_date."%b %d, %Y at %T"]</b>[sql-param order_date]<b>[/sql-filter]</b>

[PREFIX-filter] is very handy to keep in mind as many transformation tags have a filter wrapper for them. E.g., [currency] -> [PREFIX-filter currency]. And if the one you’re looking at doesn’t, you can build your own, easily.

Now to look at that [inventory] tag. The most direct approach assumes that the code inside [inventory] can be run in Safe, which often it can even if [inventory] is global. However, if [inventory] does run-time un-Safe things (such as creating an object) then it may not be possible. In such a case, we would want to create a global sub, like our hypothetical stats_crunch(), and invoke it via [PREFIX-exec]. However, let us assume we can safely (as it were) invoke it via the $Tag object to demonstrate another potent loop option: [PREFIX-sub].

[if-sql-param show_inventory]
<b>[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]</b>
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: <b>[sql-exec show_inventory]shipped[/sql-exec]</b>
[/if-sql-param]
* On Hand: <b>[sql-exec show_inventory]on_hand[/sql-exec]</b>
* Sold: <b>[sql-exec show_inventory]sold[/sql-exec]</b>
* Shipping: <b>[sql-exec show_inventory]shipping[/sql-exec]</b>
[/if-sql-param]

Let’s go over what this gives us:

[PREFIX-sub] creates an in-line catalog sub that is compiled at the start of processing, before looping actually begins. As such, the [PREFIX-sub] definitions can occur anywhere within the loop body and are then removed from the body to be parsed.
The body of the [PREFIX-exec] is passed to the sub as the first argument. We use that here for our static values to the “type” arg. If we also wanted to access [sql-param sku] from the call, we would have to include that in the body and set up a parser to extract it out of the one (and only) arg we can pass in. Instead, we can reference the $Row hash within the sub body just as we can do when using a [PREFIX-calc], with one minor adjustment to our [query] tag–we have to indicate to [query] we are operating on a row-hash basis instead of the default row-array basis. We do that by adding the hashref arg to the list:

[query
    list=1
    ml=500
    <b>hashref=1</b>

We still have access to the full functionality of [inventory] but we’ve removed the impact of having to parse that tag 2000 times (in the worst-case scenario) if left as ITL in the query body. If we run into Safe issues, that same sub body can either be created as a pre-compiled global sub or, if available, we can set our catalog AllowGlobal in which case catalog subs will no longer run under Safe.

Finally, all we have left to address is [area] and its args which themselves have ITL. I will leverage [PREFIX-sub] again as an easy way to manage the issue:

<b>[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]</b>
Order details:
    <a href="<b>[sql-exec area_order_view][/sql-exec]</b>">View [sql-param order_number]</a>

By packaging all of [area]’s requirements into the sub body, I can address all of the ITL at once.

So now, let’s put together the entire [query] rewrite to see the final product:

[query
    list=1
    ml=500
    hashref=1
    sql="
        SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if-sql-param is_development]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if-sql-param]
Date: [sql-filter convert_date."%b %d, %Y at %T"][sql-param order_date][/sql-filter]
[if-sql-param show_inventory]
[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: [sql-exec show_inventory]shipped[/sql-exec]
[/if-sql-param]
* On Hand: [sql-exec show_inventory]on_hand[/sql-exec]
* Sold: [sql-exec show_inventory]sold[/sql-exec]
* Shipping: [sql-exec show_inventory]shipping[/sql-exec]
[/if-sql-param]
[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]
Order details:
    <a href="[sql-exec area_order_view][/sql-exec]">View [sql-param order_number]</a>
[/query]

Voila! Our new query body is functionally identical to the original body, though admittedly a little more complicated to set up. However, the trade-off in efficiency is likely to be substantial.

I recently worked on a refactor for a client that was overall very similar to the above example, with a desired N value of 250. The code prior to refactoring took ~70s to complete. Once we had completed the refactor using the same tools as I’ve identified here, we brought down processing time to just under 3s, losing no functionality.

Time taken optimizing Interchange loops will almost always pay dividends.

Filling in header elements with Dancer and Template::Flute

2015-02-05T00:00:00+00:00

Inserting content into Dancer output files involves using a templating system. One such is Template::Flute. In its simplest possible explanation, it takes a Perl hash, an XML file, and an HTML template, and produces a finished HTML page.

For a project involving Dancer and Template::Flute, I needed a way to prepare each web page with its own set of JavaScript and CSS files. One way is to construct separate layout files for each different combination of .js and .css, but I figured there had to be a better way.

Here’s what I came up with: I use one layout for all my typical pages, and within the header, I have:

 <link class="additional_style" href="/css/additional.css" rel="stylesheet" type="text/css"/>
 <script class="additional_script" src="/javascripts/additional.js" type="text/javascript">
 </script>

The trick here is, there’s no such files “additional.css” and “additional.js”. Instead, those are placeholders for the actual CSS and JS files I want to link into each HTML file.

My Perl object has these fields (in addition to the other content):

$context->{additional_styles}    = [
	{ url => '/css/checkout.css' },
	{ url => '/css/colorbox.css' },
];
$context->{additional_scripts}   = [
	{ url => '/javascripts/sprintf.js' },
	{ url => '/javascripts/toCurrency.js' },
	{ url => '/javascripts/jquery.colorbox-min.js' },
	{ url => '/javascripts/checkout.js' },
];

while my XML file looks like this:

<specification>
<list iterator="additional_styles" name="additional_style">
  <param field="url" name="additional_style" target="href"/>
</list>
<list iterator="additional_scripts" name="additional_script">
  <param field="url" name="additional_script" target="src"/>
</list>
</specification>

So we have all the elements, but unless you have used all this before, you may not realize how we get the output. (Skip to the punchline if that’s not true.)

The XML file is a connector that tells Template::Flute how to mix the Perl hash into the HTML template. Usually you connect things via class names, so in the case of:

<link class="additional_style" href="/css/additional.css" rel="stylesheet" type="text/css"/>

the class name in the HTML and the name field in the XML connect, while the iterator field in the XML and the hash key in the Perl hashref do as well. The case of a

<list>

means that the hash value must be an arrayref of hashrefs, i.e.,

{
	"additional_style" => [
		{ url => "..." },
		...,
	],
}

Important note: if the hash value is undefined, you’ll get a run-time error when you try to expand the HTML template, and if you have an empty arrayref, the result of the expansion is an empty string (which is just what you want).

And so, through the magic of Template::Flute, what the browser sees is:

<link class="additional_style" href="/css/checkout.css" rel="stylesheet" type="text/css"/>
<link class="additional_style" href="/css/colorbox.css" rel="stylesheet" type="text/css"/>
<link class="additional_style" href="/css/admin/admin.css" rel="stylesheet" type="text/css"/>...
...
<script class="additional_script" src="/javascripts/sprintf.js" type="text/javascript">
</script>
<script class="additional_script" src="/javascripts/toCurrency.js" type="text/javascript">
</script>
<script class="additional_script" src="/javascripts/jquery.colorbox-min.js" type="text/javascript">
</script>
<script class="additional_script" src="/javascripts/checkout.js" type="text/javascript">
</script>

A Rusty Web? An Excursion of a Perl Guy into Rust Land

Handling text encoding in Perl

Back to the bytes

Into the language and back to the world

Encoding strategies

Debugging strategies

Migrate a web application to Unicode

Band-aids

Django and Mojolicious: a quick comparison of two popular web frameworks

Views

Models

Django

Mojolicious and Perl

Controllers

Conclusion

The Perl and Raku Conference 2024

Perl Web Frameworks

CGI

Mojolicious

Interchange

Dancer (1 & 2)

Catalyst

Conclusion

Full-text search on a budget: Xapian

Indexing

Searching

Regular Expression Inconsistencies With Unicode

Perl

Python

Ruby

JavaScript

PHP

.NET

Go

grep

Other links

Postgres migrating SQL_ASCII to UTF-8 with fix_latin

Postal code pain and fun

No postal code?

Do you really need the customer’s address?

UK postcodes

When Christmas comes

Perl Dancer Conference 2016 Day 1

Perl Dancer Conference Day 1

Sawyer X (Sawyer X) — A bus tour through Dancer core

Stefan Hornburg (Racke) — No Act on ACT

Andrew Beverly — Implementing i18n in a Dancer application using Plugin::LogReport

Theo van Hoesel — Quickstep

Job van Achterberg (jkva) — Dancing with Disabilities

Jason Lewis — The Lazy Programmer’s Guide to building HTML tables in Dancer2

Stefan Seifert (nine) — Perl 5 and Perl 6 — a great team

Use Java along with Perl

Reach customers and drive sales with MailChimp

Endless possibilities

Automatic marketing

Steps to implement

Go beyond the simple newsletter

Report on The Perl Conference 2016

The conference name

Notes from the conference

My talk

2015 Perl Dancer Conference videos

Conference Recap

Conference Presentations

SpaceCamps “The Final Frontier”

AngularJS & Dancer for Modern Web Development

Lightning Talks

Summary

Top 7 Funniest Perl Modules

1. Bone::Easy

2. Acme::EyeDrops

3. common::sense

4. Silly::Werder

5. Moose*

6. D::oh

7. your

AngularJS & Dancer for Modern Web Development

Legacy Apps

Oh, to migrate

You are in a legacy codebase

Perl Dancer Conference 2015 Report—Conference Days