How to Analyze Application Logs and Extract Actionable Insights

2025-02-28T00:00:00+00:00

Logs often accumulate unnoticed—until something breaks. Then, they suddenly become vital tools for diagnosing issues. By combining just a few command-line techniques, you can quickly spot recurring problems, identify suspicious activity, and strengthen your application’s defenses.

Below is a sample error log we’ll reference in the examples:

[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:57] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:35:00] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:35:01] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:36:10] [client 192.168.1.1:12345] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
[2025-01-01 14:36:11] [client 172.16.0.5:54321] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php

A Note on Commands

Throughout this blog, you’ll notice repeated use of commands like awk, grep, sed, uniq, and sort. These tools are indispensable for log analysis, allowing you to filter, transform, and summarize data efficiently. Here are some tips for learning these tools:

Use man <command> (e.g., man awk) to access the manual and dive deeper into a specific command
Experiment with small examples to understand how commands like awk process patterns or how grep extracts data
For a broader look at how these commands enhance productivity, check out my blog post: Practical Linux Command Line Tips for Productivity and Efficiency

These commands are not just for log analysis—they’re powerful for any data manipulation task you encounter. With practice, they can become essential to your workflow.

1. Debugging Application Errors

1.1 Summarize Frequent Issues

sed -E 's/^\[.*\] //' error.log | sort | uniq -c | sort -nr

Purpose: Remove timestamps/client’s IP, then sort and count duplicates
Outcome: A quick snapshot of which errors appear most often

3 PHP Notice: Undefined variable $foo ...
2 PHP Warning: Division by zero ...
2 PHP Fatal error: Call to undefined function baz() ...

1.2 Categorize Errors by Type

grep -oE 'PHP [^:]+' error.log | sort | uniq -c | sort -nr

Purpose: Extract error types (e.g., Notice, Warning, Fatal error)
Outcome: Know whether notices, warnings, or fatal errors dominate

3 Notice
2 Warning
2 Fatal error

1.3 Find Problematic Scripts

awk -F'in ' '{print $2}' error.log | awk '{print $1}' | sort | uniq -c | sort -nr

Purpose: Identify which files generate the most errors
Outcome: Pinpoint error hotspots for focused debugging

3 /var/www/html/index.php
2 /var/www/html/script.php
2 /var/www/html/lib.php

2. Monitoring System Behavior

2.1 Track Problematic IPs

awk -F'\[client ' '{print $2}' error.log | awk -F':' '{print $1}' | sort | uniq -c | sort -nr

Purpose: Count how many errors each IP triggers
Outcome: Identify suspicious or high-traffic IPs

4 192.168.1.1
2 10.0.0.2
1 172.16.0.5

2.2 Spot High-Error Time Periods

awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' error.log | sort | uniq -c | sort -nr

Purpose: Group errors by minute, revealing spikes or trends
Outcome: Correlate error surges with deployments or traffic peaks

3 2025-01-01 12:34:00
1 2025-01-01 14:36:00
1 2025-01-01 13:36:00
...

2.3 Analyze IP–Referer Pairs

grep "referer:" error.log | awk -F'\\[client ' '{print $2}' | awk -F':| referer: ' '{print $1, $3}' | sort | uniq -c | sort -nr

Purpose: Match IP addresses with the URLs they refer from
Outcome: Detect repeat offenders or malicious traffic patterns

3 192.168.1.1 Undefined variable $foo ...
2 10.0.0.2 Division by zero ...
1 172.16.0.5 Call to undefined function baz() ...

3. Enhancing Security

3.1 Watch Sensitive URLs

grep -E "admin|login" script_error_frequency.txt | sort -nr

Purpose: Flag frequent hits on admin or login pages
Outcome: Gauge whether sensitive endpoints are under attack

2 /var/www/html/login.php
2 /var/www/html/admin.php

3.2 Pinpoint Critical Times

grep -E "admin|login" error.log | awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' | sort | uniq -c | sort -nr

Purpose: Identify specific time windows for sensitive access
Outcome: Cross-reference suspicious activity with your security logs

2 2025-01-01 12:34:00
1 2025-01-01 13:35:00
1 2025-01-01 12:35:00

Conclusion

These commands demonstrate how quickly you can glean insights from logs. With a little creativity, you can expand them to track response times, detect performance bottlenecks, and safeguard critical endpoints. Whether you’re tackling PHP errors or any other type of log data, the same principles apply: filter, sort, count, and investigate.

Curious to learn more? Combine these strategies with automation tools, integrate them into CI/CD pipelines, or hook them up to visual dashboards. Your logs will become a gold mine of actionable information.

Integrating Laravel With a React Frontend

2021-05-07T00:00:00+00:00

Photo by Scott Webb on Unsplash

Frontend frameworks can be useful, and provide a lot of advantages over server-side rendering of views. It’s not uncommon now for websites to be purely presentational frontend applications. Thankfully Laravel provides some helpers for including a dedicated frontend, including a fantastic npm package, laravel-mix, which heavily simplifies the use of webpack.

In this article I’ll go over how to set up a new Laravel application to work with React as its frontend. While this article may focus on React, the main issues are the same regardless of framework. You’ll need to:

Add your JavaScript application to the project’s file system and set up a build process for the frontend sources
Write some additional code to bootstrap your frontend application once the page has loaded
Carefully set up URL conventions to distinguish between frontend and backend routes.

Scaffolding The Frontend

In a standard Laravel 8 application (created using composer create-project laravel/laravel <NAME>), the frontend JS application is stored in the /resources/js folder. Laravel provides a helper package called laravel/ui, which can be used to scaffold the frontend with many popular frameworks, including React. To scaffold an empty React application, you can run the following:

composer require laravel/ui
php artisan ui react

This will add a new folder resources/js/components/ with a single file called Example.js in it, which contains a basic stateless functional component called Example. It’ll also add a new line to resources/js/app.js that requires the Example component. Finally, webpack.mix.js will be updated to include adding React in the build. I’ll go over what this file does in the next section.

Compiling Assets With Laravel Mix

Laravel Mix is an npm package that comes bundled with every Laravel application. It’s not Laravel specific though; you can add it to any application where you want a simple build process. It defines helpers for popular frameworks, React included. The mix.react() helper automatically handles adding in Babel to support using JSX syntax. For Laravel, the frontend build process is configured in webpack.mix.js. By default, it includes some scaffolding code that gives you a general idea of how it can be used:

const mix = require("laravel-mix");
mix
  .js("resources/js/app.js", "public/js")
  .react()
  .sass("resources/sass/app.scss", "public/css");

To run this build process, use the npm run dev command. This will use laravel-mix to compile everything specified in webpack.mix.js. The output directory for the build is also specified there. You can also start a basic development server by running php artisan serve.

This works just fine out of the box, but one thing worth noting is that by default, it’ll package all the code, including your dependencies, in the same file: public/js/app.js. This will cause the entire dependency tree to be reloaded if you make even a single line change to your code. You can use the mix.extract() helper to put the modules into a separate file, public/js/vendor.js. This allows the browser to cache your dependencies, which won’t change too much, separately from your application, which will change much more often. Here’s how this looks in webpack.mix.js:

mix
  .js("resources/js/app.js", "public/js")
  .react()
  .extract(["react"])
  .sass("resources/sass/app.scss", "public/css");

Then, to actually include your built JavaScript sources, go to views/welcome.blade.php and add them in the header, in this order:

<head>
  . . .
  <!-- Include Frontend Application (webpack mix) -->
  <script defer src="/js/manifest.js"></script>
  <script defer src="/js/vendor.js"></script>
  <script defer src="/js/app.js"></script>
</head>

The order is important because each successive script depends on the content of the previous one being defined.

Notice that all the script tags have the defer attribute added to them. This forces the browser to wait until the DOM has fully loaded in order to execute the scripts. If you don’t add the defer attribute, you’ll end up with a blank screen when you try to load the application. This happens because the browser will, by default, run your scripts as soon as they’re loaded. And, when they’re in the head section, they get loaded before the body. So, if the script loads before the body, the root element of the React application won’t be in the DOM yet, which in turn causes the application to fail to load.

Handling Frontend Routing

The next roadblock to tackle for setting up the frontend is routing. If you’re planning to have the frontend do its own routing, you’ll need to make sure that the backend routes don’t clash with the frontend ones. You’ll also need to make sure that, for all routes that the backend doesn’t recognize, it falls back to rendering the layout page that bootstraps the frontend, and not a 404 page. If you fail to do the latter, nested frontend routes won’t work if you navigate to them directly, or refresh the page after navigating from the root URL.

One way to ensure the routes don’t clash is to add a prefix like /app/ for web routes. API routes already have the /api/ prefix set up by default, and shouldn’t pose any issues. Then, since all frontend routes won’t be recognized by Laravel, we’ll want to add a fallback route. The fallback route ensures that welcome.blade.php, which contains our root React component Example, gets rendered instead of a 404 error page for all frontend routes. We can do this by using Laravel’s Route::fallback() function in /routes/web.php:

Route::fallback(function() {
    return view(welcome);
});

Make sure you add this at the very bottom of /routes/web.php, so that it’s the last route registered by your application. This is recommended by the Laravel docs and is also good practice since this route should be the last possible route to match any given URL.

CSRF Tokens

One other thing that’s important to mention is that by default Laravel has built-in features for generating and verifying CSRF tokens. This is set up in the VerifyCsrfToken middleware class that comes bundled with a fresh application. It provides nice and easy helpers for Blade pages like @csrf to ease adding this to your forms as a hidden input. However, if you’re making forms outside of Blade in React, you might receive an error page that says 419 Page Expired when you try to submit a form or send a request:

This error happens for both vanilla HTML forms, and when sending a POST request via JavaScript, depending on the library being used. For example, I’ve encountered this issue when using jQuery, but not axios.

You can handle this in a few different ways. The easiest way is to simply add an exception for this route in your VerifyCsrfToken class:

class VerifyCsrfToken extends Middleware
{
    /**
     * The URIs that should be excluded from CSRF verification.
     *
     * @var array
     */
    protected $except = [
        "/my-route"
    ];
}

However, this removes CSRF protection entirely and in most cases, you’ll want the CSRF protection in your forms. This can be done by setting either X-XSRF-TOKEN or X-CSRF-TOKEN request headers, and also by adding a _token property to the request parameters containing the CSRF token. It’s important to note that these similarly named values are not the same thing. The XSRF token is just an encrypted version of the actual CSRF token. Laravel 8 always sets the XSRF-TOKEN cookie in the response headers by default:

This means that XSRF-TOKEN is defined in document.cookie when the page loads. By default, axios (which is included with your new Laravel application) automatically looks for this value in the cookie, and adds it to the request headers.

Conclusion

And that’s it! I’ve found Laravel works pretty well with a dedicated frontend once you get the initial setup out of the way. Have any questions? Feel free to leave a comment!

Advanced WordPress customizations

2020-11-27T00:00:00+00:00

Photo by Chris Harrison on Flickr, CC BY 2.0, cropped

WordPress is the most popular CMS, allowing you to create a professional-looking website at a relatively low cost. It also has a really strong community behind it, creating great content and supporting developers across the world.

But being popular also means being the main target of hacker attacks, and that’s why it’s crucial to keep the CMS, the theme, and all the plugins updated. When the requirements go beyond what WordPress offers on the surface, we need to find an efficient way to add our custom logic into the CMS without interfering with version upgrades, keeping the focus on security.

Custom CSS and JavaScript

A pretty common scenario in WordPress consists of installing a theme that fits most of our requirements and writing an extra layer of functionality over it to get that custom look and user experience we are looking for. Updating the theme files means that we cannot easily upgrade or change the theme without backing up our changes, and then restoring them after the upgrade, so that’s definitely not a good approach.

To make our way around this issue, some themes offer a section to add custom JavaScript or CSS rules. But sometimes we need change themes in the middle of our developing process, so I usually rely on plugins to make my frontend changes. One simple, straightforward plugin I usually install to add custom styles and frontend scripts is Simple Custom CSS and JS.

It has several customization options I usually need for SEO purposes, including the possibility to create several independent code segments and load each one in a different section (header/footer) to improve loading speed. We can also include our custom content as embedded source/styles or as external references. It also includes an editor with syntax highlighting, and allows adding custom content to the WordPress admin section.

Custom PHP code

Another thing I usually need when customizing a WordPress website is the ability to run my own PHP code inside a WordPress page or post. That is not allowed by the CMS by default, but like most things in WordPress, there is a plugin that will make our lives easier: It’s called Insert PHP Code Snippet.

You can create your own PHP routines as snippets that can be inserted with shortcodes into a WordPress page, posts, template, or whenever you can add a shortcode. This will allow running any custom backend code anywhere on the website.

To run the code on a page or post, all that needs to be done is pasting the shortcode on the content with the “PHP” button that appears on the button bar on the editor:

Custom hooks

When our logic needs to be fired up from an event on the CMS, or if we need to make changes to the default WordPress behavior or data, we will need to use the add_action() function or the add_filter() function.

add_action() allows us to execute a PHP function on specific points of the WordPress execution, for example when a post is created or commented, or when a user is created. A full list of actions is available here.
add_filter() allows us to update information associated with WordPress, for example, to set a custom CSS class for the body or to replace the login button text. A full list of filters is available here.

The following example sends an email to the webmaster when a new comment is created, using the add_action() function:

function email_comment() {
  wp_mail('webmaster@website.com', 'New comment', 'New comment posted on the website');
}
add_action('comment_post','email_comment');

Here’s another example that allows to perform a string replacement across the website by using the add_filter() function:

function replace_text($text) {
  return str_replace('Text to search', 'Text to replace with', $text);
}
add_filter('gettext', 'replace_text');

We can add these code segments using the PHP Snippet plugin (recommended) or using the functions.php file included in our theme. This last option is not recommended since it has the difficulty we discussed above about missing our custom content after upgrading the theme.

Conclusion

WordPress is a great CMS with an elegant backend and a big list of themes and plugins we can use. But before we start adding custom code, we need to make sure it will persist after upgrading everything else to the latest version.

The techniques we saw in this post are meant for websites that only need some small custom changes or improvements. If we need to include a complete layer of logic into a website, it’s always recommended to write a custom plugin from scratch, if there is nothing out there that serves the purpose. But that’s material for another blog post!

At End Point, we build web solutions in WordPress and many other technologies, keeping security and high standards in mind. Contact us if you want to find out more.

Craft: A CMS for developers

2020-10-31T00:00:00+00:00

As a software engineer, I thrive and thoroughly enjoy working on fully custom software products, applications conceived to model and help in the execution of some business process and that are built from the ground up by a team of developers.

Such projects are often complex and expensive, though, and for some clients, they can be overkill. Some clients come up with requirements that are better served by off-the-shelf software solutions. One group of such solutions are content management systems (CMS). As a rule of thumb, if a client wants a website whose main purpose is to showcase some content, their brand or image, and custom business logic requirements are limited, then chances are that a CMS will fit the bill nicely.

Lately we’ve been using the Craft CMS for a client that meets the aforementioned criteria, and I gotta say, I’ve been pleasantly surprised by the developer experience it offers.

Unlike most of the technology and products we discuss in our blog, Craft CMS is not Open Source or Free Software. The source code is readily available in GitHub for anybody to use, study, and modify, but commercial use of it is restricted and certain features are exclusive to a so-called “Pro” edition. Learn more by reading their license and their pricing structure.

The features that we will discuss in this article are all part of the no-charge “Solo” edition of Craft CMS 3 that can be used for noncommercial websites.

In this article I’m going to talk through a few of the key aspects of Craft that make me think that it’s really a CMS made for developers. Let’s get started:

Craft is easy to get up and running

Craft is just a PHP application. And it is as typical as modern PHP applications go, capable of being initially set up with Composer and of running on top of a MySQL database (it also supports Postgres!) and the Apache web server. It can all be done via console too, if that’s how you roll.

If you already have a box with Apache, PHP, MySQL and Composer, it all amounts to little more than creating a MySQL database for Craft, composer installing the Craft package, sorting out some permissions, running php craft setup, following the prompts, and finally, configuring a virtual host in Apache to serve the web directory from inside where Craft was installed.

All of this is explained in Craft’s official documentation.

Craft is easy to put in containers

For ease of development and project bootstrapping, I’ve created a containerized setup with Docker and Docker Compose that encapsulates some infrastructure tailored to my development needs. You can get the relevant files here.

If you want to follow along, clone that repo, and you’ll end up with this file structure (as shown by the tree command):

.
├── apache_config
│   └── 000-default.conf
├── docker-compose.yml
├── Dockerfile
└── README.md

1 directory, 4 files

This setup includes two containers: one for running Apache and Craft, and another for running MySQL. The apache_config/000-default.conf contains some Apache VirtualHost configuration for serving the site. docker-compose.yml defines the whole infrastructure: both containers, a network that they use to talk to each other, and a volume to persist MySQL database files. The Dockerfile is the definition of the image for the container that runs Apache and Craft.

Feel free to explore the files; I’ve made sure to comment them so that they are easy to understand and modify as you see fit.

Note: If you want to run this setup, be sure to change the ServerAdmin value in apache_config/000-default.conf, and the USER, UID, and GID values in docker-compose.yml under services > web > build > args according to your environment and user account information.

If you have Docker and Docker Compose installed in your machine, you can go to the directory just created by the git clone and:

Run docker-compose up to set up the infrastructure. You will see Docker and Docker Compose creating the image defined in Dockerfile and the containers defined in docker-compose.yml. Then the logs of the various containers will start showing. It you want to run this in the background, use docker-compose up -d instead and it will give you control of the terminal immediately after it’s done.
Run docker-compose exec web bash to connect to the web container. This is the container that has Craft’s code and is running Apache. You’ll be “logged into” the container and be placed in /var/www. This is the directory where we will install Craft.

Once in there, run composer create-project craftcms/craft ./install to install Craft with Composer. In other words, it will download all of the files that Craft needs to run. You should see something like this at the end:

> @php craft setup/welcome

   ______ .______          ___       _______ .___________.
  /      ||   _  \        /   \     |   ____||           |
 |  ,----'|  |_)  |      /  ^  \    |  |__   `---|  |----`
 |  |     |      /      /  /_\  \   |   __|      |  |
 |  `----.|  |\  \----./  _____  \  |  |         |  |
  \______|| _| `._____/__/     \__\ |__|         |__|

     A       N   E   W       I   N   S   T   A   L   L
               ______ .___  ___.      _______.
              /      ||   \/   |     /       |
             |  ,----'|  \  /  |    |   (----`
             |  |     |  |\/|  |     \   \
             |  `----.|  |  |  | .----)   |
              \______||__|  |__| |_______/


Generating an application ID ... done (CraftCMS--xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
Generating a security key ... done (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)

Welcome to Craft CMS! Run the following command if you want to setup Craft from your terminal:

    /var/www/install/craft setup

After that’s done, use this bit of bash black magic:
```
(shopt -s dotglob; mv -v ./install/* .)
```
This command moves all the files that composer just downloaded into /var/www/install out from there and into /var/www. Then rmdir install to remove the install directory because we no longer need it.
Now do php craft setup to use Craft’s CLI to set up the site, its configuration, and its database structure. Just follow the prompts. When it asks you for database configuration, choose mysql as the database driver, set mysql as the database server name (because that’s the name we’ve given it in our docker-compose.yml), and use the environment variables defined in docker-compose.yml around lines 14 to 17 for the rest of the values. You can change these to whatever you want as long as you make sure that it coincides with how you defined your MySQL database container in the docker-compose.yml file. The prompts should look something like this:
```
Which database driver are you using? [mysql,pgsql,?]: mysql
Database server name or IP address: [127.0.0.1] mysql
Database port: [3306]
Database username: [root] craft
Database password:
Database name: craft_demo
Database table prefix:
Testing database credentials ... success!
Saving database credentials to your .env file ... done
```
Next, sort out some Craft file permission requirements with this command:
```
chmod -R o+w config storage web/cpresources
```
These are directories that Craft needs write access to.
Now start up Apache with sudo service apache2 start.

And that’s it! Open a browser to localhost or 127.0.0.1 and you should see your Craft 3 homepage:

You can start playing with the control panel or the templates/index.twig right away.

Craft’s design makes sense

When it comes to content modeling, Craft offers a set of abstractions that make sense. The main concepts to understand are sections and entries. Entries are the main pieces of content. An “article” in a news site or a “post” in a blog. Sections are the way Craft groups entries together. They are useful when your site has multiple streams of content. You can, for example, have a site where you publish news, opinion pieces, and random thoughts. With Craft, that would translate neatly into three separate sections, each one with its own type of entries.

Craft also allows you to set up custom fields for every type of entry. For example, the entries on your news section may need to include a link to the original source of the news, while your opinion pieces need a short description instead. You can configure your entries using custom fields so that they include the data that makes sense for your use case.

Let’s see what that looks like in concrete terms. Now that we have a Craft instance running in localhost, go to localhost/admin in your browser. You should see Craft’s control panel. Click on the “Settings” option in the navigation bar to the left of the screen, then select the “Sections” item under “Content”:

Next, click on the “+ New Section” button by the top of the screen and you’ll be shown the section creation form. We will create a “News” section so let’s fill in the form like this:

The “Name” and “Handle” fields are pretty self explanatory. The “Section Type” is a concept we haven’t discussed yet. “Channel” is the most appropriate for a news section, which is a stream of multiple entries with the same structure.

There are other types: “Single” is a type which you would use for entries that are unique, like a home or contact page. For sections of type “Single”, there’s generally one single entry that fits in them. This is unlike “Channels” which fit multiple entries. The other section type is “Structure”, which also accommodates multiple entries, but rather than a stream of ever-growing content, it’s more appropriate for similar entries that share a certain theme. A “Structure” section type is appropriate for things like services offered or projects in a portfolio.

Learn all about entries, sections, section types, and more in Craft’s official docs.

Now that we’ve filled the form, click the “Save and edit entry types” button. This has created our new “News” section and defined the “entry type” that this section will be able to contain. The control panel now shows this:

In Craft, a section can contain multiple types of entries. For our purposes with the news section, though, just the default one is enough. Click on it, and you’ll see an editor where you can select fields that make up that entry type:

The editor I mentioned before is below the “Field Layout” title. Here’s where we can pick and choose which fields make up the entries for the “News” section. We have a fresh installation of Craft though, so we don’t have any fields. Let’s create a few by going to Settings > Fields.

This is where we can define new fields to be used for our entries throughout the site. Click the “+ New Field” button near the top of the screen and you’ll be presented with the field creation form where you can specify all manner of details. For now, we just care about “Name”, “Handle” and “Field Type”. Let’s create three fields:

One named “Heading” with a type of “Plain Text”.
One named “Body” with a type of “Plain Text”.
One named “Source” with a type of “URL”.

You should end up with something like this in the control panel’s “Fields” page (localhost/admin/settings/fields):

Now, if we go back to our “News” section’s default entry type at http://localhost/admin/settings/sections/1/entrytypes/1 or Settings > Sections > News > Entry Types > News…

You can see how the new fields that we just created are present in the “Field Layout” panel. In order to make these fields available for our “News” entries, we just need to drag them into the box named “Content” inside the greyish area.

Click the “Save” button at the top, and that’s all it takes to set up a “Channel” section, an entry type for it, and a few fields.

Now that we’ve set up the blueprints for them, let’s actually create a few entries in the “News” section. To do so, click on the “Entries” link in the navigation bar to the left which should’ve revealed itself by now, and you’ll see this screen:

If you’re used to CMS back ends, this is pretty familiar. In this screen you can create new entries and browse existing ones.

Click the big red “+ New Entry” button and select “News” in the resulting pop-up menu. You should see a form with the fields that we defined in the “Field Layout” panel during previous steps. Feel free to create a few news entries. I’ve created these two:

Craft gives you complete freedom over your front end

Most CMSs can be thought of as having two components: a front end and a back end. The back end is where content is authored and the front end is where the style and structure in which the content is presented. In Craft, most of the effort has gone into creating a solid, highly customizable back end.

As we’ve just seen, Craft comes out of the box with a back end control panel where site administrators and content creators can author new content. As far as front end goes though, Craft has nothing. For a developer well versed in front end web technologies, this is freeing and transformative.

Craft makes no assumption and makes no decision for you when it comes to developing your site’s look and feel. It gets out of your way and lets you do your job. There’s no concept of “theme”. There’s no obscure framework to learn and integrate into. There’s no proprietary templating language to struggle with. In Craft, you are completely free to write HTML, CSS and JS as you see fit to obtain your desired effect for your site.

You can develop templates using the tried and true Twig templating engine, which, if you have some experience with PHP, you’ve most likely already encountered and worked with. All the content created in the back end is exposed to the Twig templates via objects. Let’s see how.

First we need to specify a template for our sections. Continuing with our example, let’s assign a template to our “News” section. Go to Settings > Sections > News and scroll down to find the “Site Settings” area. In the table there, type news into the “Template” column. Now go to the template directory where craft was installed and create a new news.twig file. The contents can be simple, like this:

<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{{ entry.title }} - My Craft Demo</title>
</head>
<body>
    <h1>{{ entry.heading }}</h1>
    <p>{{ entry.body }}</p>
    <p><a href="{{ entry.source }}">Source</a></p>
</body>
</html>

The only noteworthy aspect of this template is how we are injecting the data that we defined in the back end into this template. We use double curly brackets to reference the entry variable. This is provided to Twig by Craft and contains all the fields that we defined for our entries in the “News” section.

With that done, save and visit any of the entries you created and you’ll see something like this:

As you can see, this entry’s URL is localhost/news/i-just-learned-that-craft-uses-twig. Yours will obviously differ depending on the title (and slug) that you gave them.

What this example lacks in complexity, it more than makes up for in potential. This is a plain old HTML document that we’ve created, with a Twig template, of course. This is the complete freedom that I like about Craft. From this point on, you can do whatever you want in terms of front end development: use whatever CSS or JavaScript framework or library you want, organize your template files in a way that makes sense to you, your team, and your website, etc. The sky is the limit.

That’s good for individual news pages. But now let’s try to link to them from the homepage. To do so, we need to edit the templates/index.twig file. Around line 174, remove the <ul> that’s there along with all its <li>s and put this instead:

{% set entries = craft.entries().section('news').all() %}

<ul>
    {% for entry in entries %}
        <li><a href="{{ entry.url }}">{{ entry.title }}</a></li>
    {% endfor %}
</ul>

Here, we leverage Twig’s templating engine capabilities, sprinkled with some of Craft’s features to obtain a list of all the entries in our “News” section. Then, we iterate over them to render links.

Effectively, Craft enhances what you can do with Twig by exposing an API for accessing the data that exists in the CMS back end.

If you’re familiar with any sort of templating language like those included in most web application frameworks like Ruby on Rails, Symfony, ASP.NET Core MVC, etc., you’ll probably feel right at home with this.

Here’s what the homepage looks like now:

You can click on any of the links and they will take you to the specific entry page.

You can learn more about querying entries in Craft’s official documentation.

Craft is cool 🕶️

So, in conclusion, I’ve found that Craft is a cool tool to have in the toolbox. It is a full-fledged CMS with tons of customization opportunities for how to model and organize the content and data of your site. When it comes to developing the front end, though, it gets out of your way and lets you do your job. That, to me, is a big win.

An introduction to automated testing for web applications with Symfony

2020-09-22T00:00:00+00:00

Testing is an immense topic in software engineering. A lot has been written and a lot of experience has been collected on it by the greater software development community. There are many different tests, techniques, approaches, philosophies, strategies.

With such a big topic, it would be futile to try touching on every aspect of it in this article. Instead, I’ll try to take a pragmatic approach and discuss a testing strategy I’ve found success with in the past as well how much testing is necessary before I feel comfortable putting code into production. This article could also serve as a sort of introduction to automated testing using the Symfony framework as a vehicle to explore various types of testing without diving too deep into edge cases or framework specifics, and instead leaning more into the concepts and design decisions that go into writing them. Still, we’ll make sure to have a running and competent test suite by the end.

So we’re going to talk about automated testing, which in its own right is a very important part of the larger discipline of software testing. It’s also a topic that, as a developer (and as such, responsible for implementing this type of tests), I’m passionate about.

Let’s get started.

The types of tests we’re going to write

For web applications, as far as automated tests go, there are three categories I think are essential to have and which complement each other very well:

Unit tests: These are the most numerous, low-level, and, in my opinion, the most important type of developer tests. Unit tests don’t only make sure that the system does what it is supposed to do, but also that it’s correctly factored where individual components are decoupled. Unit tests focus on exercising specific classes and methods running in complete isolation, which becomes harder if the class you want to test is tightly coupled with its dependencies/collaborators. These tests validate the behavior of basic programming constructs like classes and the algorithms within them. They need to be small and fast, since their intent is to be run frequently by developers as they implement features and change existing ones.
Integration tests: These tests go one level of abstraction higher when compared to unit tests. They make sure that independently developed components work properly together. In web apps, I like my integration tests to focus primarily on how the system interacts with external components. In this article, we’re going to use integration-level tests to validate functionality that has to do with interaction with a database and an external Web API.
Functional tests: These are the tests at the highest level of abstraction. These try to closely mimic the user’s experience with the system by interacting with the app as a user would. In terms of a web app, this means making HTTP requests, clicking buttons, filling out and submitting forms, inspecting HTML results, etc. As these tests exercise the complete system end to end, they are allowed to be a bit slower. This is not an issue because they are not run as frequently as unit or integration tests.

If we can build an automated testing suite that provides good coverage and exercises the system at these three levels, I will feel confident that the system under test can work properly in production. An added bonus is that with tests like these, we would have a live documentation of the system, the features it provides and, to an extent, how it works.

Virtually all serious software development ecosystems have at least one automated testing framework or library which we can leverage to write our tests. For our purposes in this article, we’re going to be using the Symfony PHP framework which integrates beautifully with PHPUnit to provide developers with an effective and even fun way to write tests.

Getting to know the system under test

To help illustrate the topic I’ve prepared a very straightforward weather app. It only offers one feature: it will allow the user to see the current weather of a given city in the US. It does this by presenting a form where people can type in a city and a state, submit it, and get their information back.

The app obtains this information by contacting the OpenWeatherMap Web API. It also stores all requests in a database for posterity.

The site is a typical Symfony web app. It uses the MVC pattern and Domain Driven Design concepts like entities, repositories and services. Here’s a diagram explaining the static structure of the app:

For a simple app like this, our entities are little more than containers for our data. The repositories take care of encapsulating the database access logic, and the services contain the logic that leverages the other objects to fulfill our business requirements.

The front end, as you’ll see, is super simple. Not really any client-side JavaScript logic to speak of, so this is more of an old-school, back-end-only app. Good enough for what we’re trying to do here though.

So our only use case is the current weather request. We do, however, have a couple of alternate scenarios within that use case. First, if the user types in an invalid US state code, the app will show a validation error. Second, if the user inputs a city that the OpenWeatherMap Web API can’t find, the app will show another error message.

To get a better idea of how the classes interact with each other, here’s a sequence diagram detailing how the app serves the main weather query scenario:

As you can see, the controller receives the request and calls upon the service class to validate the input and retrieve the information for the city specified by the user. Then, the service takes care of orchestrating the other objects like entities, repositories, and other services to fulfill the request and return back the weather information that eventually gets rendered with the template.

You can explore the source code for our demo app here. The interesting files are under the src and template directories. Their contents should be self-explanatory: src/Controller contains our controller, src/Service contains the service classes, and so on.

If you want to run it, you need to have installed PHP, Composer, the Symfony CLI, and these extensions: php-sqlite3, php-xml, php-curl. If you like Docker, there’s a Dockerfile in the repo that you can use to fire up a container ready for running the app.

Once you’ve got all that set up, do the following:

Clone the git repo with git clone git@github.com:megakevin/end-point-blog-automated-testing-with-symfony.git.
cd into the directory that was just created and install the project’s dependencies with composer install.
Initialize the database with bin/console doctrine:schema:create.
Fire up the application with composer serve.
Go to localhost:3000.

You should now be able to see something like this:

Ok, now that we understand the system under test, what it does, and how it works, we’re ready to test it.

Unit tests

Surveying a class to write unit tests for it

I’d like to start by looking into the simplest kind of tests that we need to write for this app. Those would be unit tests for our entity classes. These tests are simple because the classes that they exercise are simple as well. If you look at our Weather and WeatherQuery classes inside src/Entity you’ll see that they contain little more than some fields with their corresponding accessors and some convenience factory methods. They also don’t have any dependencies, which is convenient because that’s one less thing for our test fixtures to worry about.

So, the first step I always take is inspecting the class that I’m about to test, to try to determine what’s interesting from a testing perspective. I try to think about what the main responsibility of the class is, what sort of logic is actually adding value, what things could be broken inadvertently by other developers, what potential changes in the code I would like the test suite to alert developers of (by failing tests!), what features would benefit from having their API captured/documented in the form of an automated test. I ask myself these questions because oftentimes it’s not feasible to achieve 100% code coverage with unit tests (or with any kind of tests, for that matter). So, in those cases when we need to be strategic about what tests we write, I try to write those that will add the most value. When faced with the reality of limited resources, I try to approach these things from a “best bang for your buck” angle.

With that in mind, if we look at our WeatherQuery class, here are the things that come to my attention:

First, there are some fields annotated with validation logic. That’s the @Assert comments on top of the field definitions. For example:

/**
* @ORM\Column(type="string", length=255)
* @Assert\NotBlank(message="The city should not be blank.")
* @Assert\Type("string")
*/
private $city;

There are a few more. It’d be interesting to test those validation rules.

Second, for every field, there are accessor methods defined as well. That is, getters and setters like these:

public function getState(): ?string
{
    return $this->state;
}

public function setState(string $state): self
{
    $this->state = $state;

    return $this;
}

These methods are very simple. If we were going for 100% coverage, we may want to exercise those accessors. But being the strategic developers we are, I think we can ignore those for now. The logic is very simple, not really much opportunity for things to go wrong here.

These types of methods become even less relevant in other languages like C# or Ruby which support accessors as language level constructs. Testing those would be a moot point in those languages, since it would mean testing language/framework features, and not code that we own. There’s no good reason to do that.

Third, there’s a convenience factory method defined at the bottom of the class. It looks like this:

public static function build(string $city, string $state): WeatherQuery
{
    $city = ucwords($city);
    $state = strtoupper($state);

    $weatherQuery = (new WeatherQuery())
        ->setCity($city)
        ->setState($state)
        ->setCreated(new DateTime())
    ;

    return $weatherQuery;
}

This build method has some interesting logic to it. There’s some pre-processing that happens to the input parameters. It’d also be interesting for a test to exercise that the resulting object is correctly constructed.

Deciding what tests to write

So by analyzing our class we’ve identified parts of it that are interesting to test. We want to test the validation logic and the build method.

Let’s start with the build method as that one’s easier.

Now that we’ve identified the unit that we will test, we need to come up with test cases that exercise it in various ways. Looking at build’s code line by line, I can come up with a few interesting test cases. Here’s my thought process:

The first thing that I notice is that the method takes two parameters, city and state, and does this with them:

$city = ucwords($city);
$state = strtoupper($state);

So the test suite should validate that these values get processed like this.

There’s also this part where the object to return gets constructed:

$weatherQuery = (new WeatherQuery())
    ->setCity($city)
    ->setState($state)
    ->setCreated(new DateTime())
;

This code is taking the parameters and assigning them to fields in the object. That’d be something interesting to validate. There’s also a “created” field that gets initialized with the current date and time. Also interesting.

Having seen that, I can come up with this set of test cases for the build method:

Test that it assigns the parameters to the correct fields in the resulting object.
Test that it capitalizes each word in the city.
Test that it capitalizes the state.
Test that it sets the current moment as the “created” field in the resulting object.

Tactics for writing the unit tests

I think we’ve addressed the main concerns of this method. Now let’s write it in PHP with PHPUnit.

Where to put them and what to name them

In the demo application that you’ve hopefully downloaded and explored, we write our tests under the tests directory. This is the default location Symfony gives us for our tests and I think it’s a good one. For our unit tests, we will put them in tests/unit. Tests for this class in particular go in the tests/unit/Entity/WeatherQueryTest.php file.

Notice our naming and file location convention. The name of the file that contains a given class’s test cases is the same as the class itself, only with the word Test as a suffix. We’ve also made sure to mimic the project’s src directory structure in our tests/unit directory. So, in this example, the WeatherQuery class is located in src/Entity/WeatherQuery.php. That means that its tests should live in tests/unit/Entity/WeatherQueryTest.php. Following this convention keeps things simple and easy to navigate and maintain.

What a PHPUnit test class looks like

PHPUnit makes writing tests easy. A collection of tests is defined as a series of methods inside a class that extends the PHPUnit\Framework\TestCase class. Here’s what our WeatherQueryTest.php file looks like:

namespace App\Tests\Unit\Entity;

use PHPUnit\Framework\TestCase;

// ...

class WeatherQueryTest extends TestCase
{
    // build
    public function testBuildAssignsTheParametersToTheCorrectFields()
    {
        // ...
    }

    // ...
}

Notice how the test class name is, again, the same name of the class under test, suffixed with the word Test. Notice how it extends from the TestCase class and notice how our test methods (that is, test cases) all begin with the word test. This is necessary for PHPUnit. This is how we signal to it that this is, in fact, a test case that it needs to run.

How to name the individual test methods

Another important thing to note is the naming convention that we’re using here for our test methods. Remember the test cases that we came up with above:

Test that it assigns the parameters to the correct fields in the resulting object.
Test that it capitalizes each word in the city.
Test that it capitalizes the state.
Test that it sets the current moment as the “created” field in the resulting object.

These are wordy descriptions of exactly what the test cases are about. Just by reading these, we can get a good idea of what the purpose of the test case is, and its expected outcome. For the same reasons of readability and self documentation, the names of our test methods in the actual test suite implementation need to be as close to that as possible. I’ve come up with these:

testBuildAssignsTheParametersToTheCorrectFields
testBuildCapitalizesTheGivenCityParameter
testBuildCapitalizesTheGivenStateParameter
testBuildSetsTheCurrentMomentAsTheCreatedField

They begin with the word test in order to fulfill PHPUnit’s requirements. Then, they include as many details as possible on what they are about.

The anatomy of a test case

When it comes to actually writing the tests, I put great focus on making them as easy to understand as possible. Test cases do not only serve the purpose of validating system behavior. They also can serve as a live documentation of the system’s features, and, in the case of unit tests, its inner workings. To be able to fulfill that purpose, tests need to be easy to read, navigate, and understand.

That’s why I always like to call attention to the general three steps that almost every test follows: “Setup”, “Exercise” and “Verify”. Or, as I like to call them: “Arrange”, “Act” and “Assert”. So the first thing I do is add those three as comments in the test method body:

public function testBuildAssignsTheParametersToTheCorrectFields()
{
    // Arrange
    // Act
    // Assert
}

“Arrange” is the step where we set up our test fixture. That is, we configure the world around the unit under test — the input, dependencies, etc. — so that we can control all the variables that go into the execution of the unit.

“Act” is quite simply where we have our unit under test be executed.

“Assert” is where we verify that the unit under test behaved correctly, that it met the test case’s expectations. We normally do this by checking the output of the method or some other side effect that its execution causes.

In the Behavior Driven Development world, “Given”, “When” and “Then” are parallels for these. You may see that terminology used as well. The spirit is the same.

Usually the next most obvious step is Act. This example is like that: We just need to call our build method and capture its output. Adding that, our test method would look like this now:

public function testBuildAssignsTheParametersToTheCorrectFields()
{
    // Arrange
    // Act
    $result = WeatherQuery::build($testCity, $testState);
    // Assert
}

By writing our Act portion, we discovered that we need parameters, AKA input values. That’s what the Arrange step is for:

public function testBuildAssignsTheParametersToTheCorrectFields()
{
    // Arrange
    $testCity = 'MyCity';
    $testState = 'ST';

    // Act
    $result = WeatherQuery::build($testCity, $testState);

    // Assert
}

Finally, we need to actually validate that the build method did what we expected. We do that by writing some assertions. What did we expect it to do, though? Well, the name of the test method has the answer: we expect it to “assign the given parameters into the correct fields of the resulting object”. In PHPUnit, we do it like so:

public function testBuildAssignsTheParametersToTheCorrectFields()
{
    // Arrange
    $testCity = 'MyCity';
    $testState = 'ST';

    // Act
    $result = WeatherQuery::build($testCity, $testState);

    // Assert
    $this->assertEquals($testCity, $result->getCity());
    $this->assertEquals($testState, $result->getState());
}

PHPUnit has an extensive number of different types of assertion methods. In this case, we use one of the simpler ones: assertEquals. This method will compare two values and report the test as a failure if they are different. It will report it as a success if they are equal.

In this test case, we want PHPUnit to validate for us that the resulting object’s city field is the same that we passed in. Same for the object’s state field.

And that does it for our first unit test! We’ve prepared some input, called the unit that we wanted to test, and inspected its state to validate that it did what it was supposed to do.

Running PHPUnit tests

Symfony also makes it easy to run unit tests. By default, Symfony projects created with either symfony new my_project_name --full or composer create-project symfony/website-skeleton my_project_name already include the PHPUnit library and the configuration necessary to execute unit tests like this one. You can run it, along with all the others in this test class with this command:

bin/phpunit tests/unit/Entity/WeatherQueryTest.php

Which will result in something like this:

$ bin/phpunit tests/unit/Entity/WeatherQueryTest.php
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing App\Tests\Unit\Entity\WeatherQueryTest
........                                                            8 / 8 (100%)

Time: 64 ms, Memory: 8.00 MB

OK (8 tests, 9 assertions)

Notice how we just call the PHPUnit executable and pass it the file name that contains the tests that we want to run.

I prefer the TestDox style output though, so I like to use this instead:

bin/phpunit --testdox tests/unit/Entity/WeatherQueryTest.php

Which gives us:

$ bin/phpunit --testdox tests/unit/Entity/WeatherQueryTest.php
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing App\Tests\Unit\Entity\WeatherQueryTest
App\Tests\Unit\Entity\WeatherQuery
 ✔ Build assigns the parameters to the correct fields
 ✔ Build capitalizes the given city parameter
 ✔ Build capitalizes the given state parameter
 ✔ Build sets the current moment as the created field
...

Time: 68 ms, Memory: 8.00 MB

OK (8 tests, 9 assertions)

And this is where those really long and detailed names for the test methods pays off. Now running a test suite results in output that reads in plain English and serves as a specification of sorts on how a given class works. This fulfills the automated testing suite’s secondary objective of serving as live documentation for the system.

Other interesting details

Now that we’ve discussed my thought process for writing this first unit test, the rest of the build unit tests should be pretty self-explanatory. They all follow the same pattern; the only difference is the Arrange and Assert parts, which change according to what the particular tests’s purpose is.

For instance, notice how the testBuildCapitalizesTheGivenCityParameter test passes the build function 'my city' as the value for the city parameter. Then, it asserts that the return object’s property was set to My City. Thus, assuring that the $city = ucwords($city); line in the build method’s implementation is working properly. If somebody changes this by mistake, the test will break and let them know that they have to fix it.

Another interesting test case is this one: testBuildSetsTheCurrentMomentAsTheCreatedField. Here, we need to assert that the WeatherQuery object returned by the build method has its “created” field set to the current moment in time. However, timestamps are so precise that one obtained before calling the method and one obtained inside it are different. We still need to assert equality though. So, to deal with that, we use this variation of the assertEquals assertion:

$this->assertEquals(
    (new DateTime())->getTimestamp(),
    $result->getCreated()->getTimestamp(),
    '', 1
);

This one tests for equality, but with a certain allowed difference. If the difference is within this margin of error, then the assertion deems that the two values are equal — just what we need in this case. This basically says: “Assert that the current timestamp and the resulting object’s timestamp are ‘pretty much’ the same.” That’s as good a job as PHPUnit can do to for that assertion, and that’s good enough for us.

Testing behavior with varying input

Ok, we’ve now tested one piece of functionality defined within the WeatherQuery class. There is something else to test though: the validation rules. The class has several validation rules defined as annotations. We are able to define validation rules like this thanks to Symfony’s Validator component. To validate an object against those rules, we need to use the Symfony\Component\Validator\Validation class. Here’s what a test that exercises those validation values looks like:

public function testValidation($city, $state, $expected)
{
    // Arrange
    $subject = WeatherQuery::build($city, $state);

    $validator = Validation::createValidatorBuilder()
        ->enableAnnotationMapping()
        ->getValidator();

    // Act
    $result = $validator->validate($subject);

    // Assert
    $this->assertEquals($expected, count($result) == 0);
}

If you are familiar at all with Symfony, then you may know what all this is about. If not, then it’s not hard at all. We just need this recipe: Validation::createValidatorBuilder()->enableAnnotationMapping()->getValidator(); to obtain a Validator instance. We can pass a WeatherQuery object to it and it will validate it for us using the annotations defined within the class.

That’s how we validate an object in Symfony. However, validation is a common technique that’s done regardless of framework. In general, to test validation rules, we always use the same approach:

Create the validation subject with some input data.
Call whatever component to validate said input data.
Assert that the validation result is what we expect given the input.

This means that testing a set of validation rules boils down to running the same process over and over again with different input. The input is the only thing that varies. PHPUnit has the data provider concept which lends itself beautifully for these scenarios.

To illustrate this, let’s continue with our validation example. Consider the annotation and signature of the test method:

/**
* @dataProvider getValidationTestCases
*/
public function testValidation($city, $state, $expected)
{
    // ...
}

The @dataProvider getValidationTestCases comment tells PHPUnit to look for a method named getValidationTestCases to obtain the list of sets of arguments to pass to the annotated test method.

Our testValidation method expects three parameters: a city, a state, and a boolean representing whether the validation should succeed or not. So, the getValidationTestCases method needs to provide them. Let’s look at it now:

public function getValidationTestCases()
{
  return [
    'Succeeds when data is correct' => [ 'New York', 'NY', true ],
    'Fails when city is missing' => [ '', 'NY', false ],
    'Fails when state is missing' => [ 'New York', '', false ],
    'Fails when state is not a valid US state' => [ 'New York', 'AAA', false ],
  ];
}

getValidationTestCases returns an associative array that contains four items. Each item’s key is a description of the test case and the value is the data that makes up the test case. That data is what will get passed as parameters to testValidation. The keys are used in the TestDox output to make it more descriptive, like so:

$ bin/phpunit --testdox tests/unit/Entity/WeatherQueryTest.php
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing App\Tests\Unit\Entity\WeatherQueryTest
App\Tests\Unit\Entity\WeatherQuery
...
 ✔ Validation with data set "Succeeds when data is correct"
 ✔ Validation with data set "Fails when city is missing"
 ✔ Validation with data set "Fails when state is missing"
 ✔ Validation with data set "Fails when state is not a valid US state"

Time: 68 ms, Memory: 8.00 MB

OK (8 tests, 9 assertions)

As you can see, this results in PHPUnit running the testValidation method four times, once per each test case defined in getValidationTestCases, using their corresponding data as parameters.

I find PHPUnit’s data provider feature very useful for testing input validation logic. However, it can be used for other types of tests as well. I always consider trying it out whenever I see a series of test cases that look very similar to each other. This usually means that they can be written in a generic way to eliminate repetition. The data provider feature makes it easy to push variability out of the test cases code and into the input arguments.

Using mocks to test classes with dependencies

Now let’s move on to a set of classes that are generally a bit less straightforward to test: services. Service classes, as opposed to entities, are seldom so independent and self-contained. Because of their very nature as integrators and orchestrators of other classes that fulfill core business logic, services often have dependencies and collaborators.

Take a look at our WeatherService class in src/Service/WeatherService.php. Just by looking at the constructor you can see that it depends on others to function:

public function __construct(
    ValidatorInterface $validator,
    WeatherQueryRepository $repository,
    WeatherApiClient $apiClient
) {
    $this->validator = $validator;
    $this->repository = $repository;
    $this->apiClient = $apiClient;
}

In order to work properly, the WeatherService needs a few objects: a ValidatorInterface, a WeatherQueryRepository, and a WeatherApiClient. Instead of directly creating these objects though, this class uses a technique called dependency injection, where the objects it needs are passed to it via its constructor. In other words, all of its “dependencies” are “injected” into it. Dependency injection is key for unit testing. Here’s why: For unit tests, our objective is to test a unit in complete isolation. This means that, within a given test case, we want to exercise the code of one class and one class alone. If an object leverages other objects to do some work, we don’t care about those other objects. If we did, then the unit test loses focus. It becomes something else. The unit under test is no longer a unit.

Dependency injection allows client code to specify which concrete dependencies a given object will use. That’s a feature that test cases can take advantage of to pass in fake objects that it sets up and controls. We call these mocks.

Mocks are essentially fake objects that a test fixture uses to pass as dependencies to its test subject. They are objects that, to the eyes of the unit under test, are the real deal objects that they normally use and work with. However, in reality they are made up objects that look like the subject’s dependencies that the test case fully controls and can inspect.

Why do unit tests need to test objects in isolation? Because unit tests need to be simple, fast, and easy to understand. Having a test case focused only on validating a single, small piece of functionality is a great way to achieve those three goals.

Ok, let’s see an example of some mock objects in action. Consider the testGetCurrentWeatherDoesNotReturnSuccessWhenTheApiCallIsUnsuccessful test case in tests/unit/Service/WeatherServiceTest.php:

public function testGetCurrentWeatherDoesNotReturnSuccessWhenTheApiCallIsUnsuccessful()
{
    // Arrange
    $validator = Validation::createValidatorBuilder()
        ->enableAnnotationMapping()
        ->getValidator();

    $mockRepository = $this->createMock(WeatherQueryRepository::class);

    $mockApiClient = $this->createMock(WeatherApiClient::class);
    $mockApiClient->method('getCurrentWeather')->willReturn([
        'success' => false
    ]);

    $service = new WeatherService(
        $validator,
        $mockRepository,
        $mockApiClient
    );

    // Act
    $result = $service->getCurrentWeather('New York', 'NY');

    // Assert
    $this->assertFalse($result['success']);
}

One feature of the GetCurrentWeather method in the WeatherService is that it uses a dependency, WeatherApiClient, to make an HTTP request to the OpenWeatherMap Web API, by calling its getCurrentWeather method. If the request is not successful, WeatherApiClient returns an array which contains a success field that’s set to false. The good thing about a decoupled design made possible by dependency injection is that the WeatherService doesn’t care about HTTP requests or responses or any of that. It only cares and knows about WeatherApiClient’s contract. In this case, the contract dictates that it returns an array with a success field. Knowing this, the test case can create a mock object that looks just like a WeatherApiClient instance and make it return something that will make the WeatherService think that the request failed. It’s done like this:

$mockApiClient = $this->createMock(WeatherApiClient::class);
$mockApiClient->method('getCurrentWeather')->willReturn([
    'success' => false
]);

The first statement uses PHPUnit’s createMock method to obtain a fake object. Then, the second statement configures it by specifying that, whenever the getCurrentWeather gets called on that mock, it will return the value that will make WeatherService think that the request failed.

After that’s set up, the rest of the test should be straightforward: In the Arrange section we set up other mocks for each of WeatherService’s constructor parameters (i.e. injected dependencies) and create a real instance of WeatherService which is our unit under test; in the Act section, we exercise our UUT by calling the method we want to test; and finally, in the Assert section, we validate that the UUT has returned a result that signals that the operation was unsuccessful by looking at the resulting array’s success field.

Verifying behavior instead of state: Expectations with PHPUnit

In this last example, we used the mock to control the UUT’s behavior and asserted on the result by verifying data or state. Mocks can do much more than that, though. We can configure them to return whatever value we want, like we just did, but we can also inspect them to know if they have been called, what parameters were given to them, etc. This allows us to write a slightly different style of test — one that, instead of verifying state, verifies behavior. Here’s an example of such a test case:

public function testGetCurrentWeatherCallsOnTheApiClientToGetWeatherData()
{
    // Arrange
    $validator = Validation::createValidatorBuilder()
        ->enableAnnotationMapping()
        ->getValidator();

    $mockRepository = $this->createMock(WeatherQueryRepository::class);

    $mockApiClient = $this->createMock(WeatherApiClient::class);
    $mockApiClient->method('getCurrentWeather')->willReturn([
        'success' => true,
        'response' => $this->testApiResponse
    ]);

    $service = new WeatherService(
        $validator,
        $mockRepository,
        $mockApiClient
    );

    // Expect
    $mockApiClient
        ->expects($this->once())
        ->method('getCurrentWeather')
        ->with('New York', 'NY')
    ;

    // Act
    $service->getCurrentWeather('New York', 'NY');
}

See how this test case is not verifying a result from a method call. Instead, it verifies whether our mock was called in a specific way by the unit under test. Also, our typical Arrange, Act, Assert formula has changed a bit. The test now reads Arrange, Expect, Act. This is how we write expectation-style test cases in PHPUnit. Let’s look at the statement:

$mockApiClient
    ->expects($this->once())
    ->method('getCurrentWeather')
    ->with('New York', 'NY')
;

This basically says: “In this test case, expect that the mock WeatherApiClient’s getCurrentWeather method gets called with 'New York' and 'NY' as parameters.” If, within the test case, this doesn’t happen, then it will be reported as a failure.

The thought process of writing tests with mocks

How do we write these types of tests, though? Well, we read the code that we want to test line by line, try to identify the spots where dependencies are used, and how their results affect the behavior of the unit under test. This way we can mock them properly, configuring those mocks so that we can trigger the execution paths within the unit under test that we want to exercise. These types of tests are as white-box as can be, and are really coupled with the implementation. Most changes to the implementation will break them. This is good because subtle bugs can be caught. This could also be bad because tests breaking often means more work fixing them. I personally like my tests to have this fine-grained focus, so I’m a big fan of mocking aggressively, but the most important thing is to find the balance that works best for you, your team and your project.

To continue our example, if we look at the getCurrentWeather method in WeatherService:

public function getCurrentWeather(string $city, string $state)
{
    $weatherQuery = WeatherQuery::build($city, $state);

    $this->repository->add($weatherQuery);

    $result = $this->apiClient->getCurrentWeather($city, $state);

    if (!$result['success']) {
        return [
            'success' => false,
            'weatherQuery' => $weatherQuery
        ];
    }

    $apiResponse = $result['response'];

    $weather = Weather::build($weatherQuery, $apiResponse);

    return [
        'success' => true,
        'weather' => $weather,
        'weatherQuery' => $weatherQuery
    ];
}

We can work through it line by line and we see that:

First, it takes its parameters and creates a new instance of WeatherQuery:

$weatherQuery = WeatherQuery::build($city, $state);

Nothing to do with this for now, from a testing perspective. We can’t mock anything here because it’s a static dependency. WeatherQuery is being referenced directly and its static build method is being called. Being a static dependency means that, as far as the test is concerned, WeatherQuery::build may as well be defined inside WeatherService. That’s why we normally should try to avoid static dependencies like this and prefer using interfaces and passing dependencies into objects via dependency injection. If we wanted to test WeatherService without WeatherQuery, we wouldn’t be able to do so. Because they are statically bound to one another. No big deal in this case, since this dependency is a simple factory method that saves WeatherService from having to construct an object that’s pretty simple. If the logic to construct WeatherQuery instances was too complex to warrant it being isolated from WeatherService tests, then we’d be in trouble. We’d need to refactor to maybe add a new WeatherQuery factory class and pass that into WeatherService as an injected dependency that can be then mocked. We’re good with what we have now though, as it is not that complex.

Then getCurrentWeather calls on the repository to store the WeatherQuery in the database:

$this->repository->add($weatherQuery);

This one is interesting. It’s a textbook example of a side effect that we can easily write a test for using mocks. The mock in this case would be of the WeatherQueryRepository dependency and we would verify that its add method gets called with the expected parameter. This unit test would be impossible to write without mocks, because the result of this call to WeatherQueryRepository’s add is not captured nor returned as part of getCurrentWeather’s result. The only insight that a test case could have into this aspect of getCurrentWeather’s execution is via a mock.

The test case that covers this line is testGetCurrentWeatherCallsOnTheRepositoryToSaveANewWeatherQuery.

Next up, we’ve got:

$result = $this->apiClient->getCurrentWeather($city, $state);

This calls the WeatherApiClient’s getCurrentWeather method. This is also one that we can test by mocking the dependency, and doing some behavior verification on whether and how the method was called. testGetCurrentWeatherCallsOnTheApiClientToGetWeatherData does this.

We can also do state verification because the result gets captured in a variable and used for the unit under test’s return value. As we can see in the conditional statement that follows:

if (!$result['success']) {
    return [
        'success' => false,
        'weatherQuery' => $weatherQuery
    ];
}

The tests that cover this are:

testGetCurrentWeatherDoesNotReturnSuccessWhenTheApiCallIsUnsuccessful
testGetCurrentWeatherReturnsAWeatherQueryObjectWithCorrectFieldsWhenTheApiCallIsUnsuccessful

Finally, the method captures the API response, builds a Weather object based on it (again, a static dependency that we can’t intercept with a mock), and returns a result.

$apiResponse = $result['response'];

$weather = Weather::build($weatherQuery, $apiResponse);

return [
    'success' => true,
    'weather' => $weather,
    'weatherQuery' => $weatherQuery
];

Here, we can test that the resulting associative array’s fields are being generated correctly. We do that with this trifecta of test cases:

testGetCurrentWeatherReturnsSuccessWhenTheApiCallIsSuccessful
testGetCurrentWeatherReturnsAWeatherObjectWithCorrectFieldsWhenTheApiCallIsSuccessful
testGetCurrentWeatherReturnsAWeatherQueryObjectWithCorrectFieldsWhenTheApiCallIsSuccessful

You can run all of the unit tests in the demo with bin/phpunit --testdox tests/unit.

Integration tests

Like I said, I firmly believe that unit tests are the bread and butter of any serious automated test suite. However, I also like to do some higher-level tests to supplement the unit tests. Integration tests are such tests. These focus on exercising system behavior that interacts with external components like databases and web APIs. In fact, in our demo app’s test suite we have both. Let’s look into them.

Testing database interaction

Many frameworks offer ways to help testing database interaction code by creating databases that are exclusive to the test environment. Symfony is no exception. The idea is simple: We create a new database that is exclusive for testing, configure the test runner to use that, and write some tests that exercise some code that talks to the database. Luckily for us, we decided to encapsulate all database interaction logic inside a repository class, so for our database integration tests, it’s obvious where we should focus on: WeatherQueryRepository.

Consider the sole test case at tests/functional/Repository/WeatherQueryRepositoryTest.php:

public function testAddSavesANewRecordIntoTheDatabase()
{
    // Arrange
    $testWeatherQuery1 = WeatherQuery::build('My City 1', 'MY STATE 1');
    $testWeatherQuery2 = WeatherQuery::build('My City 2', 'MY STATE 2');

    self::bootKernel();
    $repository = self::$container->get(WeatherQueryRepository::class);

    // Act
    $repository->add($testWeatherQuery1);
    $repository->add($testWeatherQuery2);

    // Assert
    $records = $repository->findAll();

    $this->assertEquals(2, count($records));
    $this->assertEquals('My City 1', $records[0]->getCity());
    $this->assertEquals('MY STATE 1', $records[0]->getState());
    $this->assertEquals('My City 2', $records[1]->getCity());
    $this->assertEquals('MY STATE 2', $records[1]->getState());
}

This one is pretty straightforward. We create two objects, call the add method to insert them as records, then query the database with findAll and assert on the data to make sure the objects we get out from the database are the same that we put in.

One interesting aspect about this test class is that we are calling two methods on WeatherQueryRepository: add and findAll. However, we only have a test case for add. Why is that? If we look at WeatherQueryRepository’s implementation, we find out quickly: the add method is the only method in our class. We get findAll from Doctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepository, a framework base class that our repository inherits from. The test suite for our app should only be concerned in testing the code that we wrote and own. Our test suite has no business testing code that belongs to Symfony, Doctrine, or any other framework or library. Test cases for those belong in their test suites, not ours.

Another interesting point is that we can’t create a “real” WeatherQueryRepository to play with on our own, outside of the context of Symfony. That’s why we:

have this test class extend Symfony\Bundle\FrameworkBundle\Test\KernelTestCase instead of plain old PHPUnit\Framework\TestCase
have to go through some hoops to obtain a WeatherQueryRepository instance:

self::bootKernel();
$repository = self::$container->get(WeatherQueryRepository::class);

This is just how Symfony allows us to obtain fully featured objects in the context of our tests, just as they would be if they were being executed in the application’s normal day-to-day runtime. We ask Symfony for an instance instead of us directly newing it up. And this is good news, because with a framework like Symfony that uses dependency injection so heavily, and many of our custom classes end up depending on complex framework objects, instantiating them on our own can get complicated.

In Symfony, there are a few configurations that need to happen so that we are able to run database integration tests. Symfony’s own documentation is excellent for learning how to, but in a nutshell, here’s what needs to happen:

Install this bundle which allows each test case to run with the same, unmodified database: composer require --dev dama/doctrine-test-bundle.

Enable it by adding the following to the phpunit.xml.dist file:

<phpunit> 
    <!-- ... -->
    <extensions>
        <extension class="DAMA\DoctrineTestBundle\PHPUnit\PHPUnitExtension"/>
    </extensions>
    <!-- ... -->
</phpunit>

Create a test database with bin/console doctrine:schema:create --env=test.

Once that setup is done, tests can be run with something like tests/functional/Repository/WeatherQueryRepositoryTest.php.

$ bin/phpunit --testdox tests/functional/Repository/WeatherQueryRepositoryTest.php
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

Testing App\Tests\Functional\Repository\WeatherQueryRepositoryTest
App\Tests\Functional\Repository\WeatherQueryRepository
 ✔ Add saves a new record into the database

Time: 160 ms, Memory: 20.00 MB

OK (1 test, 5 assertions)

Testing calls to an external Web API

Another interesting aspect of our application that our integration test should cover is the logic that calls on the OpenWeatherMap Web API. In tests/functional/Service/WeatherApiClientTest.php, we’ve done just that. Look at the file and you’ll see that it is pretty similar to our other integration test that we wrote for WeatherQueryRepository:

<?php

namespace App\Tests\Functional\Service;

use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
use App\Service\WeatherApiClient;

class WeatherApiClientTest extends KernelTestCase
{
    public function testGetCurrentWeatherInvokesTheApiAndReturnsTheExpectedResponse()
    {
        // Arrange
        self::bootKernel();
        $apiClient = self::$container->get(WeatherApiClient::class);

        // Act
        $result = $apiClient->getCurrentWeather('New York', 'NY');

        // Assert
        $this->assertArrayHasKey('weather', $result['response']);
        $this->assertGreaterThanOrEqual(1, $result['response']['weather']);
        $this->assertArrayHasKey('main', $result['response']['weather'][0]);
        $this->assertArrayHasKey('description', $result['response']['weather'][0]);
        
        $this->assertArrayHasKey('main', $result['response']);
        $this->assertArrayHasKey('temp', $result['response']['main']);
        $this->assertArrayHasKey('feels_like', $result['response']['main']);
        $this->assertArrayHasKey('temp_min', $result['response']['main']);
        $this->assertArrayHasKey('temp_max', $result['response']['main']);
        $this->assertArrayHasKey('pressure', $result['response']['main']);
        $this->assertArrayHasKey('humidity', $result['response']['main']);

        $this->assertArrayHasKey('visibility', $result['response']);

        $this->assertArrayHasKey('wind', $result['response']);
        $this->assertArrayHasKey('speed', $result['response']['wind']);
        $this->assertArrayHasKey('deg', $result['response']['wind']);
    }
}

The same key elements are here: A test class that inherits from Symfony\Bundle\FrameworkBundle\Test\KernelTestCase, a test method that starts up the Symfony kernel and obtains a fully configured instance of WeatherApiClient, and the usual Arrange, Act, and Assert structure that should be familiar by now. The Act portion just calls the method that we want to test and captures its return value. The assertions are done using state verification to validate that the result from the call to the API contains the data that we expect. Simple.

Running this test with a command like bin/phpunit tests/functional/Service/WeatherApiClientTest.php will actually make a real HTTP request to the OpenWeatherMap Web API and return back its response.

Perhaps the most interesting aspect to discuss about this integration test is how different it is from a unit test written against the same class. If we look inside src/Service/WeatherApiClient.php, we see that the getCurrentWeather method is where the magic happens:

public function getCurrentWeather(string $city, string $state)
{
    $response = $this->httpClient->request(
        'GET',
        'http://api.openweathermap.org/data/2.5/weather?q=' . $city . ',' . $state . ',us&appid=' . $this->openWeatherMapAppId
    );

    if ($response->getStatusCode() != 200) {
        return [
            'success' => false,
        ];
    }

    $response = json_decode($response->getContent(), true);

    return [
        'success' => true,
        'response' => $response,
    ];
}

The method leverages the $httpClient (which is an object of type Symfony\Contracts\HttpClient\HttpClientInterface, injected into our WeatherApiClient class by Symfony), to make an HTTP request to the OpenWeatherMap Web API endpoint. Depending on its response, it constructs and returns a result value. There’s some logic in this method, calls to dependencies being made, results being inspected, conditionals, associative arrays being constructed, etc. Our integration test doesn’t care about any of that. It only cares that the Web API is getting called and that it returns what it expects.

Unit tests, on the other hand, do care about these details. As a result, tests/unit/Service/WeatherApiClientTest.php, which contains the unit tests for that same class, goes more in depth into it and exercises all the possible code paths, while providing a mock for the HttpClientInterface dependency. This makes sure that the unit test does not execute code from anything other than the unit under test. So, we end up with unit test cases like:

testGetCurrentWeatherCallsOnTheHttpClientToMakeAGetRequestToTheWeatherApi
testGetCurrentWeatherDoesNotReturnSuccessIfTheResponseStatusCodeIsNot200
testGetCurrentWeatherReturnsSuccessAndResponseDataAsArrayIfTheResponseStatusCodeIs200

Which should be self-explanatory at this point.

Essentially, the integration test would break if the API was down at some point, or if its contract has changed and we call it incorrectly. The unit test would break if somebody modifies the way the algorithm functions. They complement each other well.

Functional tests

Now it’s time to move on to the final type of test that we’re going to discuss in this article, and the one that sits at the highest level of abstraction: functional tests. The focus of this kind of test is to exercise the complete system, end to end, while trying to mimic the experience that a user would have with it.

To achieve this, Symfony gives us some tools that we can use to interact with the application at the protocol level, just like a browser would. That is, sending HTTP requests to and inspecting the responses from a complete application running in a sandbox, fully configured and integrated with all of its external components like databases, etc.

Let’s look at tests/functional/Controller/WeatherControllerTest.php to see that in action. The first thing to note is that functional test classes need to inherit from Symfony\Bundle\FrameworkBundle\Test\WebTestCase. This is how we tell Symfony that we need the tools necessary to talk to our application as if we were doing so via HTTP.

use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;

class WeatherControllerTest extends WebTestCase
{
    // ...
}

Then, in the test cases themselves, we find our usual three-step structure. Take the first one, for example:

public function testIndexWorks()
{
    // Arrange
    $client = static::createClient();

    // Act
    $client->request('GET', '/');

    // Assert
    $this->assertResponseStatusCodeSame(200);
}

This is a simple test which makes a GET request to the root route of the site and expects it to return a 200 HTTP status code. Notice how we use the static::createClient() method to obtain an HTTP client object that we can use to talk to our application. Next we use the client to do just that with $client->request('GET', '/'); and finally use the assertResponseStatusCodeSame assertion to check that the request was successful. All of these methods are available for our test because we defined our test class by extending WebTestCase.

We can go much deeper into our assertions on the responses though. Look at the testShowDisplaysAllWeatherInfoIfGivenValidInput test case, for example:

public function testShowDisplaysAllWeatherInfoIfGivenValidInput()
{
    // Arrange
    $client = static::createClient();

    // Act
    $client->request('GET', '/show/New%20York/NY');

    // Assert
    $this->assertSelectorTextContains('div.wrapper', 'The current weather');
    $this->assertSelectorTextContains('div.wrapper', 'Temp');
    $this->assertSelectorTextContains('div.wrapper', 'Feels like');
    // ...
}

This is very similar to the previous one. The main differences are that, first, we’re requesting a different route (the one that results in a page that presents the actual weather data that was requested), and second, that instead of only asserting HTTP status code of the response, we assert on the actual contents of the page that would be rendered in a browser. We can use assertSelectorTextContains for that.

Symfony includes many more goodies that help with functional testing. You can learn more in the official docs. There are a few more examples in the demo source code so you may want to look them over now that you understand what their intent is and how they are written.

Closing thoughts

In this article, we’ve gone through some of the basics of automated testing for web apps. We’ve used Symfony as a vehicle for conveying that information and, as such, learned a little bit about what it takes to build a multifaceted test suite with that framework. And we did all that by offering many examples from a real (albeit simple) test suite that works.

We’ve discussed the three types of tests that I like to use in terms of their level of abstraction: unit tests, integration tests, and functional tests; and what they are, what aspect of the system they focus on, and how to write them. I believe that a test suite that offers good coverage while including these three kinds of tests offers the best bang for the buck and gives us a cozy safety net to fall back to while we develop our applications. These three levels of testing complement each other very well.

We’ve also touched on the general approach and thought process behind writing automated tests. We discussed a three step approach: Arrange, Act, and Assert. We also talked about test cases that do classic state verification, where input is provided to a unit under test, the unit is exercised, and finally its output is verified.

We also saw a more advanced testing technique where, by leveraging mock objects, we do behavior verification. Here, we assert on whether the unit under test has called its collaborators/dependencies in a certain way, a style which is great for testing side effects and classes that need other dependencies in order to work. These two styles of tests work best when used together to compose a larger test suite.

And that’s it for now! This has been quite a trip into some aspects of software testing. Like I said at the beginning, this discipline is huge, and one never stops learning. But I think what we’ve discussed here is a good starting point to continue deepening our understanding while also serving as a competent enough strategy to use in a real-world application.

Job opening: PHP / JavaScript developer

2020-06-29T00:00:00+00:00

We are looking for a PHP software engineer to work with us during business hours somewhere in the UTC-7 to UTC-4 time zones (U.S. Pacific to Eastern Time). This role can be full-time or part-time.

We are an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. The company turns 25 years old this year!

Even before COVID-19 most of us worked remotely from home offices. We collaborate using SSH, GitHub, GitLab, chat, video conferencing, and of course email and phones.

What you will be doing:

Develop new web applications and support existing ones for our clients
Work together with End Point co-workers and our clients’ in-house staff
Use your desktop OS of choice: Linux, macOS, Windows
Use open source tools and contribute back as opportunity arises

What you bring:

Professional experience developing and supporting web applications in these technical areas:

5+ years of development with PHP and front-end JavaScript
Frameworks such as Symfony, Laravel, Magento and Vue.js, React, Angular
Databases such as PostgreSQL, MySQL, MongoDB, Redis, Solr, Elasticsearch, etc.
Security consciousness
Git version control
Automated testing
HTTP, REST APIs
Bonus for familiarity with another ecosystem such as Ruby on Rails, Python/Django, Java, .NET/C#, Node.js, Go, Rust, Scala, Kotlin, Swift …

These work traits are just as important:

Strong verbal and written communication skills
An eye for detail
Tenacity in solving problems
A feeling of ownership of your projects
Work both independently and as part of a team
Focus on customer needs
A good remote work environment

What work here offers:

Collaborate with knowledgeable, friendly, helpful, and diligent co-workers around the world
Flexible, sane work hours
Annual bonus opportunity
Freedom from being tied to an office location
For full-time staff: paid holidays and vacation
For U.S. employees: health insurance subsidy and 401(k) retirement savings plan

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.) Include your location, a resume/CV, your GitHub or LinkedIn URLs, and whatever else helps us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

Magento 2: Creating a custom theme

2020-06-24T00:00:00+00:00

Photo by Maria Eklind, CC BY-SA 2.0

In my previous post, we went through the steps needed to create a custom module in Magento 2. While modules consist of a set of classes to add new features to Magento, a theme controls how these features, and the entire website in general, will be displayed to the user. As stated in the Magento guide, a theme uses a combination of custom templates, layouts, styles, and images to provide a consistent look and feel across a Magento store.

Creating a new Magento 2 theme

We can create a theme based on a default “parent” theme or create a standalone theme from scratch. In most cases, I would recommend the first option. For this example, we will use Luma as our parent theme. The other option would be inheriting from the default “blank” theme.

Here’s an initial task list to get our new theme ready:

Create a new directory for the theme
Create the registration.php script
Create the theme.xml information file
Activate the new theme

Creating a new directory for the theme

While all our backend code should go in app/code, the frontend content is expected to go in app/design. And as our theme will only apply design changes to the frontend content, we should create the new directory for it under the path app/design/frontend. If we want to create a theme for the admin area instead, we need to create the directory inside app/design/adminhtml.

Let’s create a directory named EndPoint (our vendor name, continuing with the example from our previous article) and a subdirectory inside it, MyTheme:

cd {website_root}
mkdir -p app/design/frontend/EndPoint/MyTheme

Creating registration.php

Similar to the file we created for our module, registration.php tells Magento to register the new theme with the name and location we specify. Our file will be located at app/design/frontend/EndPoint/MyTheme/registration.php and should have the following content:

<?php
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::THEME,
    'frontend/EndPoint/MyTheme',
    __DIR__
);

This way, Magento will know what path our theme will have.

Creating theme.xml

The next step is to create our theme information file, where we will specify the theme name and parent theme. So our app/design/frontend/EndPoint/MyTheme/theme.xml file should have the following content:

<?xml version="1.0"?>
<theme xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Config/etc/theme.xsd">
    <title>MyTheme</title>
    <parent>Magento/luma</parent>
</theme>

Optional: If we want our theme to be easily distributed as a package, we can also create a composer.json file in the theme’s root directory. The content for the file should be as follows, specifying the theme’s description, dependencies, version, and license types:

{
    "name": "endpoint/mytheme",
    "description": "My Theme by End Point",
    "require": {
        "magento/theme-frontend-luma": "100.0.*",
        "magento/framework": "100.0.*"
    },
    "type": "magento2-theme",
    "version": "100.0.1",
    "license": [
        "OSL-3.0",
        "AFL-3.0"
    ],
    "autoload": {
        "files": [
            "registration.php"
        ]
    }
}

Activating our new theme

That was easy! We have everything we need to activate our new theme. To do that we log in to our admin area and enable our theme. Once in the dashboard, we need to go to Content > Design > Configuration, edit our store view, and select our new theme from the dropdown list:

Magento will search for new themes every time we log in to the admin area, so our new theme will appear on the list automatically.

Adding custom content

We have the basic structure for our theme, but when enabled, it will look the same as its parent theme (Luma, in this case), since we didn’t add any design rules or static files yet. Let’s do some more things with our theme to change how it’s displayed in the frontend:

Create a custom etc/view.xml file
Add a custom logo
Add static files (JavaScript, CSS, images, fonts)
Add a custom layout

Creating a custom view file

etc/view.xml controls many frontend configurations like the product thumbnail width, how the product image gallery is displayed, and the image magnifier tool, among other things. To add our custom view file to our theme, we need to copy the existing file from our parent theme. For Luma, it will be located at vendor/magento/theme-frontend-luma/etc/view.xml. To copy the file, we need to run the following in our website’s root:

mkdir -p app/design/frontend/EndPoint/MyTheme/etc
cp vendor/magento/theme-frontend-blank/etc/view.xml app/design/frontend/EndPoint/MyTheme/etc/view.xml

And then we can use our preferred text editor to change the values we want, like setting a custom size for the images in the category page grid:

<image id="category_page_grid" type="small_image">
    <width>300</width>
    <height>300</height>
</image>

Adding a custom logo

Adding a logo to our theme is really simple. We just need to save our picture in SVG format as web/images/logo.svg. If we want to use a different filename or format for our logo, we will have to create a default layout file for our theme in the path /Magento_Theme/layout/default.xml inside our theme root with content similar to this:

<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
  <body>
    <referenceBlock name="logo">
      <arguments>
        <argument name="logo_file" xsi:type="string">images/custom_logo.png</argument>
        <argument name="logo_width" xsi:type="number">300</argument>
        <argument name="logo_height" xsi:type="number">200</argument>
        <argument name="logo_alt" xsi:type="string">Custom logo name</argument>
      </arguments>
    </referenceBlock>
  </body>
</page>

We can use different image formats such as SVG, PNG, or JPG. We can also use a custom width and height for the logo, and set a custom alternate text.

Adding static files (JavaScript/CSS/images/fonts)

All the static files should be located inside the web directory. Common static files include JavaScript files, stylesheets, images, and fonts. The JavaScript files should be located at web/js, stylesheets at web/css, images at web/images, and our custom fonts should be located at web/fonts.

All the static files will be published as direct links, without any processing from Magento, at the pub/static/frontend/EndPoint/MyTheme/en_US path. The default locale/language is en_US; we can change it for our theme if needed.

Adding a custom layout

Finally, if we want to use the new assets we added and have custom content on different sections of our website, we need to extend or override the existing layout files from our parent theme.

For example, if we want to add a reference to new stylesheet and JavaScript files we added, we need to extend the existing header layout from our parent theme. To do this, we will create a new layout file located at Magento_Theme/layout/default_head_blocks.xml:

<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
  <head>
    <!-- Custom stylesheet -->
    <css src="css/mytheme.css"/>

    <!-- Custom JavaScript -->
    <script src="js/mytheme.js"/>
  </head>
</page>

This way we will be adding a reference to a new stylesheet named mytheme.css that we have inside the web/css directory of our theme, and a new script named mytheme.js inside the web/js directory.

After we make all the desired changes to our theme, we will need to tell Magento to update the frontend. We need to deploy the new changes and clear the cache. We can achieve that by running this from our website root:

php bin/magento setup:static-content:deploy
php bin/magento cache:clean

This process can take some minutes to complete. After it’s done, we can go to our website to see if the frontend changes are applied:

Looks awesome! Of course, there’s a lot more we can do from there, from extending or overriding layouts from modules (Magento or third-party) to bundling scripts or using Less files for our custom styles. But, that is material for later posts, so that’s all for now! Please add any questions you might have below.

Linux Development in Windows 10 with Docker and WSL 2

2020-06-18T00:00:00+00:00

I’m first and foremost a Windows guy. But for a few years now, moving away from working mostly with .NET and into a plethora of open source technologies has given me the opportunity to change platforms and run a Linux-based system as my daily driver. Ubuntu, which I honestly love for work, has been serving me well by supporting my development workflow with languages like PHP, JavaScript and Ruby. And with the help of the excellent Visual Studio Code editor, I’ve never looked back. There’s always been an inclination in the back of my mind though, to take some time and give Windows another shot.

With the latest improvements coming to the Windows Subsystem for Linux with its second version, the new and exciting Windows Terminal, and Docker support for running containers inside WSL2, I think the time is now.

In this post, we’ll walk through the steps I took to set up a PHP development environment in Windows, running in a Ubuntu Docker container running on WSL 2, and VS Code. Let’s go.

Note: You have to be on the latest version of Windows 10 Pro (Version 2004) in order to install WSL 2 by the usual methods. If not, you’d need to be part of the Windows Insider Program to have access to the software.

What’s new with WSL 2

This is best explained by the official documentation. However, being a WSL 1 veteran, I’ll mention a few improvements made which have sparked my interest in trying it again.

1. It’s faster and more compatible

WSL 2 introduces a complete architectural overhaul. Now, Windows ships with a full Linux Kernel which WSL 2 distributions run on. This results in greatly improved file system performance and much better compatibility with Linux programs. It’s no longer running a Linux look-alike, but actual Linux.

2. It’s better integrated with Windows

This is a small one: we can now use the Windows explorer to browse files within a WSL distribution. This is not a WSL 2 exclusive feature, it has been there for a while now. I think it’s worth mentioning though, because it truly is a great convenience and a far cry from WSL’s first release, where Microsoft specifically advised against manipulating WSL distribution file systems from Windows. If nothing else, this makes WSL feel like a first class citizen in the Windows ecosystem and shows that Microsoft actually cares about making it a good experience.

3. It can run Docker

I’ve recently been learning more and more about Docker and it’s quickly becoming my preferred way of setting up development environments. Due to its lightweightness, ease of use, repeatability, and VM-like compartmentalization, I find it really convenient to develop against a purpose-built Docker container, rather than directly in my local machine. And with VS Code’s Remote development extension, the whole thing is very easy to set up. Docker for Windows now supports running containers within WSL, so I’m eager to try that out and see how it all works.

4. A newer version means several bugfixes

Performance notwithstanding, WSL’s first release was pretty stable. I did, however, encounter some weird bugs and gotchas when working with the likes of SSH and Ruby during certain tasks. It was nothing major, as workarounds were readily available. I’ve already discussed some of them here, so I won’t bother mentioning them here again. But thanks to the fact that the technology has matured since last time I saw it, and considering the architectural direction it is going in, I’m excited to not have to deal with any number of quirks.

The development environment

Ok, now with some of the motivation out of the way, let’s try and build a quick PHP Hello World app running in a Docker container inside WSL 2, make sure we can edit and debug it with VS Code, and access it in a browser from Windows.

Step 1: Install WSL 2 and Ubuntu

Step 1 is obviously to install WSL and a Linux distribution that we like. Microsoft’s own documentation offers an excellent guide on how to do just that. But in summary, we need to:

Enable the “Windows Subsystem for Linux” and “Virtual Machine Platform” features by running these on an elevated PowerShell:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Restart your machine.
Set WSL 2 as the default version with: wsl --set-default-version 2, also from PowerShell.
Install your desired distribution from the Microsoft Store. I chose Ubuntu 20.04 LTS.
After installing, open the “Ubuntu 20.04 LTS” app from the Start menu and it should come up with a command line console. Wait for it to finish installing. It should prompt for a username and password along the way. Choose something you won’t forget.

Optionally, you can install the Windows Terminal app to get a better command line experience. Windows Terminal can be used to interact with PowerShell and the classic CMD, as well as with our WSL distributions.

Step 2: Install Docker

Installing Docker is very straightforward. Just download the installer for Docker Desktop for Windows, execute it, and follow the wizard’s steps. Make sure that during setup the “Use the WSL 2 based engine” option is selected. In most cases, the installer will detect WSL 2 and automatically have the option selected.

Follow the official instructions for more details on the process, but it really is that simple.

Step 3: Install some useful VS Code extensions

Our objective is to create a new development environment inside a Docker container and connect to it directly with VS Code. To do that, we use a few useful extensions:

The Docker extension which allows us to browse and manage images and containers and other types of Docker assets.
The Remote - WSL extension which allows VS Code to connect to a WSL distribution.
The Remote - Containers extension which allows VS Code to connect to a container.

Step 4: Create the development container

The extensions that we installed will allow us to use VS Code to work on code from within our WSL Ubuntu as well as from the container. Specifically, we want to connect VS Code to a container. There are a few ways to do this, but I will describe the one I think is the easiest, most convenient and “automagic” by fully leveraging the tools.

Let’s begin by opening a WSL Ubuntu terminal session, which will show something like this:

Welcome to Ubuntu 20.04 LTS (GNU/Linux 4.19.104-microsoft-standard x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

...

kevin@kevin-thinkpad:/mnt/c/Users/kevin$

The project directory

Let’s change to our home, create a new directory for our new project, and change into it.

$ cd
$ mkdir php-in-docker-demo
$ cd php-in-docker-demo

Because we installed the Remote - WSL extension, we can open up this directory in VS Code with code .. Opening a terminal (Ctrl + `) in this VS Code instance opens WSL console, not Windows.

The Dockerfile

Now let’s create a new file called Dockerfile which will define what our development environment image will look like. For a no-frills PHP environment, mine looks like this:

# The FROM statement says that our image will be based on the official Ubuntu Docker image from Docker Hub: https://hub.docker.com/_/ubuntu
FROM ubuntu

# Install packages, not allowing apt to ask any questions since we can't answer.
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y software-properties-common php php-xdebug composer

# Configure Xdebug so that the VS Code debugger can use it.
RUN echo "xdebug.remote_enable=on" >> /etc/php/7.4/mods-available/xdebug.ini
RUN echo "xdebug.remote_autostart=on" >> /etc/php/7.4/mods-available/xdebug.ini

# The CMD statement tells Docker which command to run when it starts up the container.
# Here, we just call bash
CMD ["bash"]

This script will later be used to create our development container. It will have PHP, Xdebug and Composer. This is all we need for our simple Hello World app. For more complex scenarios, other software like database clients or PHP extensions can be easily installed with additional RUN statements that call upon the apt package manager.

Consider reading through Docker’s official documentation on Dockerfiles to learn more.

The configuration file

Now, to leverage VS Code’s capabilities, let’s add a development container configuration file. In our current location, we need to create a new directory called .devcontainer and, inside that, a new file called devcontainer.json. I put these contents in mine:

{
    // The name used by VS Code to identify this development environment
    "name": "PHP in Docker Demo",

    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",

    // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
    "dockerFile": "../Dockerfile",

    // Add the IDs of extensions you want installed when the container is created.
    // This is the VS Code PHP Debug extension.
    // It needs to be installed in the container for us to have access to it.
    "extensions": [
        "felixfbecker.php-debug"
    ],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // When we run our PHP app, we will use this port.
    "forwardPorts": [5000],
}

A default version of this file can be automatically generated by running the “Remote-Containers: Add Development Container Configuration Files…” command in VS Code’s Command Palette (Ctrl + Shift + P).

The development container

Now that we have all that in place, we can create our image, run our container, and start coding our app. Bring up the VS Code Command Palette with Ctrl + Shift + P and run the “Remote-Containers: Reopen in Container” command. The command will:

Read the Dockerfile and create an image based on that. This is like running docker build -t AUTOGENERATED_IMAGE_ID .
Run a container based on that image with the settings specified in .devcontainer/devcontainer.json. In our case, all it will do is enable the container’s port 5000 to be accessible by the host. This is more or less like running: docker run -d -p 5000:5000 -v ${PWD}:/workspaces/php-in-docker-demo AUTOGENERATED_IMAGE_ID
Open a new VS Code instance connected to the container with the /workspaces/php-in-docker-demo directory open.

It will take a while, but after it’s done, we will have a VS Code instance running directly in the container. Open the VS Code terminal with Ctrl + ` and see for yourself. It will show a prompt looking like this:

root@ec5be7dd0b9b:/workspaces/php-in-docker-demo#

You can for example, run php -v in this terminal, and expect something along these lines:

PHP 7.4.3 (cli) (built: May 26 2020 12:24:22) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

This is PHP running, not in Windows, not in our WSL Ubuntu, but in the Docker container.

Hello Windows + WSL 2 + Ubuntu + Docker + PHP + VS Code

Let’s now create our app. Add a new index.php file containing something silly like:

<?php

echo "Hello Windows + WSL 2 + Ubuntu + Docker + PHP + Visual Studio Code!";

Then, in the VS Code console (remember, Ctrl + `), start up an instance of the built in PHP development server wth php -S 0.0.0.0:5000. It’s important that we use port 5000 because that’s the one that we configured our container to use.

Navigate to http://localhost:5000/ in your browser and feel good about a job well done.

Interactive debugging

When configuring our development container, we added Xdebug and the PHP Debug VS Code extension. This means that VS Code can leverage Xdebug to provide an interactive debugging experience for PHP code.

Almost everyting is set up at this point, we just need to do the usual VS Code configuration and add a launch.json file. To do so, in VS Code, press Ctrl + Shift + D to bring up the “Run” panel, click on the “create a launch.json file” link, and in the resulting “Select Environment” menu, select “PHP”.

After that, the “Run” panel will show a green triangular “Start Debugging” button next to a “Listen to XDebug” text. If you haven’t already, start up a dev web server with php -S 0.0.0.0:5000, click on the “Start Debugging” button, put a breakpoint somewhere in your index.php file, and finally open up http://localhost:5000/ in a browser.

We’re interactively debugging PHP code running on a Docker container in WSL from our Windows IDE/editor. Pretty cool, huh?

And that’s all for now. In this article we’ve learned how to set up a Linux development environment using Docker containers and WSL 2, with Windows 10 Pro. This is a nice approach for anybody who’s confortable on Windows and needs access to a Linux environment for development; and have that environment be easy to reproduce.

Resources:

Magento 2: Creating a custom module

2020-04-01T00:00:00+00:00

Photo by Babatunde Olajide, cropped from original

A Magento module is a set of classes and routines that will depend on and interact with other Magento classes in order to add a specific feature to a Magento application. While a theme is orientated towards the front-end and user experience, a module is orientated towards backend logic and application flow.

We will need to create a custom module if we want to add or change the existing logic at a level where Magento doesn’t provide a setting or option for it. For example, if our business has a specific feature or set of features or requirements that are not common to the market, a module can fill that gap for us.

Creating a basic Magento 2 module

Creating a simple module in Magento 2 is not that hard. We will need to accomplish the following tasks:

Create a new directory for the module
Create a registration.php script
Create a etc/module.xml information file
Install the new module

Creating a new directory for the module

Where should the new directory for our module be placed? We have two options to choose from:

app/code/{vendor}/
vendor/{vendor}/

If your module is intended for a specific website you’re working on, you can use the first option. If you’re creating a module with the intention of it being used on several websites, it’s best to choose the second option. We’ll use the first for this example.

Let’s create a directory named EndPoint (our vendor name) with a subdirectory inside it, MyModule:

cd {website_root}
mkdir -p app/code/EndPoint/MyModule

Creating the registration.php script

The registration.php file tells Magento to register the new module under a specific name and location. Let’s create a file named app/code/EndPoint/MyModule/registration.php with the folllowing content:

<?php
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'EndPoint_MyModule',
    __DIR__
);

We’re telling Magento that our module will be named EndPoint_MyModule.

Creating the etc/module.xml information file

Now, let’s create our module information file, where we’ll specify the module version number. First, we need to create the etc directory inside app/code/EndPoint/MyModule:

mkdir app/code/EndPoint/MyModule/etc

then create module.xml with the following content:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Module/etc/module.xsd">
    <module name="EndPoint_MyModule" setup_version="1.0.0">
    </module>
</config>

Installing the new module

That’s it! We have everything we need to install our new module. Now we need to tell Magento we want to install and enable our new module. So from our website root we need to run:

php bin/magento setup:upgrade

Magento will output a list of module names and configuration updates, and our new module EndPoint_MyModule should be listed in that output.

Adding a custom route to our module

Now we have a working, enabled module, but it’s not doing anything yet! What’s a simple way to check that our module is enabled? Let’s set up a custom route, so if we hit a URL like https://{our_website}/mymodule/test/helloworld we can return a custom response from a controller.

Creating a custom route will need some steps on its own:

Create a new directory for the controller
Create a etc/routes.xml file
Create the controller
Upgrade the new module

Creating a new directory for the controller

First we need to create a new directory where the new PHP controller for our custom route will live. The new directory path should be:

app/code/EndPoint/MyModule/Controller

We can create as many directory levels we want, depending on our desired path. For example, if we create a class named Index in app/code/EndPoint/MyModule/Controller, the URL that will be routed to this controller will be https://{our_website}/mymodule/index (the “Controller” directory is ignored). If we create a class named HelloWorld in app/code/EndPoint/MyModule/Controller/Test, the resulting URL will be https://{our_website}/mymodule/test/helloworld.

Creating the etc/routes.xml file

routes.xml will tell Magento what base URL will be used for our module. First, we need to create the “frontend” directory where the routes.xml file needs to be placed:

mkdir app/code/EndPoint/MyModule/etc/frontend

In this example, we want the base URL to be MyModule, so we need to create an XML file inside the new directory that will route all requests made to the given URL to our module controllers:

<?xml version="1.0" ?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:App/etc/routes.xsd">
    <router id="standard">
        <route frontName="mymodule" id="mymodule">
            <module name="EndPoint_MyModule"/>
        </route>
    </router>
</config>

Creating the controller

If we want to respond to requests for https://{our_website}/mymodule/test/helloworld we first need to create the base Controller directory and the Test subdirectory:

mkdir -p app/code/EndPoint/MyModule/Controller/Test

Under this directory, we’ll create our custom Magento controller. All route controllers should extend \Magento\Framework\App\Action\Action. We also need to have a public construct() method to pass the context to our ancestor and an execute() function that will be called when the URL is hit:

<?php

namespace EndPoint\MyModule\Controller\Test;

class HelloWorld extends \Magento\Framework\App\Action\Action
{

    public function __construct(
        \Magento\Framework\App\Action\Context $context
    ) {
        parent::__construct(
            $context
        );
    }

    public function execute()
    {
        echo "Hello world!";
    }

}

Upgrading the new module

We have everything in place to tell Magento we have new changes to be deployed. How we do that? First, we need to upgrade our Magento setup. But since we added a new controller that gets parameters from the dependency injector in the construct, we also need to compile the dependency injection engine (including factories, proxies, and interceptors). And finally, we need to clear the cache so new content will be served from our custom URL:

php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento cache:flush

This process can take a few minutes to complete, but after it’s done we can try to reach our new custom URL. If we get a response like the one below:

That means our module is working!

That’s all for now. In upcoming posts, we’ll start complicating things a bit by overriding Magento classes with our custom ones and creating custom controllers that will return information from the Magento core classes. We will also explore how to customize the front-end by creating a theme. Don’t forget to add any questions, suggestions, or issues in the comments below!

Cooking with CAS

2020-03-10T00:00:00+00:00

Photo by Flickr user reidrac, licensed under CC BY-SA 2.0

One of our customers asked us to host a new suite of web-based applications for them and to protect them with a single sign-on (SSO) solution. Ok, easy enough; these applications were in fact designed with a particular SSO system in mind already, but our situation required a different one, and we eventually chose Apereo’s open source Central Authentication Server project, or CAS. I’d like to describe the conversion process we went through.

The ingredients

Our customer’s application suite included:

The principal Java application using JAAS authentication
Another Java application based on Spring Security
A pair of PHP applications
A few automated tasks that needed to authenticate.

The original SSO system sets a header on each request, identifying an authenticated user. This requires a gateway system to sanitize request headers to ensure malicious users cannot forge a header themselves. It also requires each application inspect request headers and respond appropriately.

CAS is a bit more complex: applications redirect unauthenticated requests to a CAS server, which authenticates the user through any of various configurable methods. The CAS server then redirects the user back to the original application with a parameter called a “Service Ticket”, a seemingly random number identifying an individual authentication request. The original application contacts the CAS server directly to validate the service ticket and to collect information to identify the user. It can then establish a session for that user, and proceed normally.

To CAS-enable an application, we incorporate one of the CAS client libraries, which exist for various languages. In fact we won’t use the Java client directly, but rather we’ll incorporate components that extend it. When evaluating CAS, I was a bit concerned by what appeared to be a surprisingly limited selection of actively supported client libraries, and of course your results may vary, but we found software to meet our own needs without too much difficulty.

Configuring Wildfly Authentication

The most important application in the suite depends on the JAAS-based security subsystem of the Wildfly application server it’s deployed to. Originally it used a custom LoginModule that inspected request headers to find the ID of the authenticated user. Our first task was to configure our proxy server to remove this header from every request. We planned to disable the old authentication system entirely, of course, but this change ensured that even if we mistakenly left it enabled somewhere, malicious users couldn’t exploit it for access.

This application uses a declarative security policy: the deployment descriptor identifies a set of user roles, and another set of “security constraints”. Each security constraint describes one or more URL patterns used in the application, and the set of user roles allowed to access URLs matching those patterns. Here are two examples:

<!-- clients -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/clients/*</url-pattern>
        <url-pattern>/client/edit/*</url-pattern>
        <url-pattern>/client/edit_tree/*</url-pattern>
        <url-pattern>/client/view/*</url-pattern>
        <url-pattern>/client/view/id/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>client_role</role-name>
    </auth-constraint>
</security-constraint>

<!-- places -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/places/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>manage_places</role-name>
    </auth-constraint>
</security-constraint>

We installed the cas-extension library in our Wildfly server to handle the CAS protocol. When an unauthenticated user attempts to access a URL matching a pattern in one of the application’s security constraints, the CAS extension automatically intercepts control and redirects the user to the CAS server. Assuming the user authenticates successfully, our application will receive another request with a service ticket parameter. The cas-extension intercepts this request as well, validates the service ticket, and creates an “identity assertion”, which it sends to the Wildfly security system. Wildfly’s role mapper queries a database to find the user’s roles, after which the authentication process is complete.

Configuration of the cas-extension begins with a CAS profile, a combination of the URL of the CAS server and the URL of the service the extension should protect.

<subsystem xmlns="urn:soulwing.org:cas:1.0">
    <cas-profile name="default" service-url="https://our.application.server/application" server-url="https://our.cas.server"/>
</subsystem>

This tells the extension where to send authentication requests, where to listen for requests returning from the CAS server, and where to validate service tickets. Next we need to validate that identity assertion, and figure out what roles belong to the user. This happens in a Wildfly security domain.

<security-domain name="MySecurityDomain">
    <authentication>
        <login-module name="IdentityAssertion" code="org.soulwing.cas.jaas.IdentityAssertionLoginModule" flag="required" module="org.soulwing.cas"/>
    </authentication>
    <mapping>
        <mapping-module code="DatabaseRoles" type="role">
            <module-option name="dsJndiName" value="java:/comp/env/jdbc/databaseConnection"/>
            <module-option name="rolesQuery" value="select role from user_roles where user_id = ?"/>
        </mapping-module>
    </mapping>
</security-domain>

The authentication portion of the security domain refers to a JAAS LoginModule shipped with the CAS extension, which simply verifies that the identity assertion comes from the CAS extension and not somewhere else. Then the mapping portion (documented here) looks up the given user in a database to find what roles it should be assigned.

The last piece of the puzzle is a CAS deployment descriptor for our application, which activates cas-extension for that application. In its simplest form, this is an empty file in the right place, but ours ended up a little more complex. It identifies both the CAS profile we want to use (unnecessary in this case, as there’s only one CAS profile on the system, but it helped keep things more clear in our minds), and instructs the extension to load some other libraries into our application.

<?xml version="1.0" encoding="UTF-8"?>
<cas xmlns="urn:soulwing.org:cas:1.0">
  <profile>default</profile>
  <add-api-dependencies/>
</cas>

This configuration proved sufficient to let users log in and use the application, but as is sometimes the case with single sign-on, we needed a little more work to let them log out properly. Each application in the suite sets a cookie in the user’s browser to identify its session. The CAS server likewise sets a cookie. When a user logs out of the application, that application’s session cookie is destroyed, but we also need to destroy the CAS server’s session cookie as well as the other applications’ cookies. Single log-out can be complicated, and I won’t go into the full setup here. One reason the CAS deployment descriptor loads cas-extension libraries was so we could use cas-extension to generate the proper logout URL, and redirect our users to that URL once the application has destroyed its own session.

Configuring Spring Authentication

Another Java-based application in our suite uses Spring Security. This document describes the bulk of the configuration, which seemed less straightforward than for cas-extension, but follows essentially the same mechanism. Here we configured a Spring UserDetailsService to execute the same database query we used above to find the user’s roles. I’m not fully conversant in the large stack of beans Spring uses to manage the process, and it took some time to get this configuration sorted out.

Enter PHP

Two of these applications use PHP, which meant yet another configuration. Apereo maintains a PHP client, which includes several helpful examples. I tracked down the part of the application that authenticates users, and replaced the existing code with calls to phpCAS:

phpCAS::client(CAS_VERSION_2_0, $phpCASHost, $phpCASPort, $phpCASContext);
phpCAS::forceAuthentication();
$_SESSION[EXPORT_SERVERNAME]['umdid'] = phpCAS::getUser();

The forceAuthentication call determines the current phase of the authentication process this request is in, whether it’s unauthenticated, fully authenticated, or requires service ticket validation, and responds appropriately. We then set a session variable to the ID of the authenticated user, which replicates what the original authentication code would have done.

One of these two applications requires authenticated access to a REST API exposed by the Wildfly application. CAS calls this “proxy” authentication, when one application requests access to another. Here, CAS issues not only its usual service ticket, but also a “proxy granting ticket”. When the application wants to use the API, it asks the CAS server for a service ticket, using the proxy granting ticket. The CAS server itself requires some new configuration in this case, but for the PHP code, the only difference in the login phase from the simpler, non-proxy case is that we call phpCAS::proxy instead of phpCAS::client to configure CAS. Later, when calling the service itself, we used more phpCAS services in place of the cURL library the original used.

$service = \phpCAS::getProxiedService(PHPCAS_PROXIED_SERVICE_HTTP_GET);
$service->setUrl($serviceURL);
$service->send();

The return of header authentication

Finally, we have a few automated tasks which use various APIs, and need to authenticate. We can’t redirect requests to a CAS server and expect a user to provide credentials, so we’ve taken our cue from the applications’ original form, and configured CAS to recognize a “trusted header”. We add this header to any requests issued by these automated jobs. Of course, we’ve also configured the proxy to disallow this header from any systems outside our internal network.

A few loose ends

Of course, there were other considerations in this project that I’ve not covered here. Configuring CAS itself wasn’t necessarily straightforward, and included a custom authentication module I hope to describe in a later article. Selecting among CAS server’s available deployment options and fitting the winner into our existing infrastructure in a way that makes it easy to manage and monitor was another task entirely. We needed to customize the server’s default user interface to prevent it from offering users an imaginary method to recover forgotten passwords. The series of redirected browser requests involved in the CAS protocol presents a notable performance impact under some circumstances. And it has taken me no small effort to learn to appreciate the CAS server’s sometimes distressingly circular documentation. But several months into the project, CAS seems to be working well enough for our purposes that other users of the same application suite have begun to express interest.

Symfony Quickstart

2020-03-02T00:00:00+00:00

Photo by Arindam Mahanta

This article is written for anyone who has experience working with PHP and is starting to work with Symfony. We won’t assume any prior knowledge you might have with Symfony, so if getting started with this framework is a high priority, then this article is for you. I know that it can be difficult and you may be unsure where to look; I was in the same situation when I first worked with Symfony.

You might be pointed to consult the documentation, but even though the documentation is very detailed and nicely written, you might have a very urgent issue to solve, and thus not have time to read multiple articles about the framework before you start working. You might just need to quickly start, solve a few issues and worry about the details later.

How do I run this stuff?

If your project does not exist yet, you will need to set up Symfony, using the steps nicely outlined in Symfony’s setup guide.

Assuming that the project already exists and you need to quickly start working on it, you will need to run composer install in the root folder of the project to make sure that the dependencies are properly set. This could result in errors; for example, PHP might be not installed in your development environment. In this case, of course, you need to install PHP, which should not be a big problem if we continue to assume that you have some experience with PHP.

Another problem might be that Composer is not yet installed on your machine. If this is the case, install Composer by following the steps here.

You might be missing some PHP extensions at this point or have other problems. If so, read the error messages you get and solve them. If you don’t understand some error messages, don’t worry, you aren’t the first one struggling to make Symfony work. Search for the error message you got paired with the Symfony keyword and find others’ solutions.

Composer

Composer is a server-side package manager which is frequently used by modern PHP applications. In the previous section we executed composer install. Let’s understand this command now. This command reads composer.json, resolves dependencies and installs them into the vendor folder.

If you intend to update to the latest versions of the dependencies outlined in composer.lock, run composer update or its alias composer upgrade.

There are two very important files to know about: composer.json and composer.lock. In composer.json the dependencies are specified as JSON. The actual value of a dependency is an expression to denote what kinds of versions are allowed. composer.lock specifies which dependencies should not be updated to the latest version and it contains the exact version number for the installed PHP packages, so if you have a composer.lock file and you want to fetch the latest versions allowed by composer.json, then you need to run composer update.

Since our article is just a quick start, we’ll avoid delving too deep into the realms of Composer, but if you want to learn more, it’s very well documented. You’ll need an understanding of Composer to work on Symfony projects, but for the very short term you can get by with a minimal understanding.

Whenever you are pulling work done by others, you will need to make sure that the dependencies are updated. If there is any change in the downloaded composer.json, composer install should be executed. If you check out a different branch you should run composer install unless you are absolutely sure it’s not needed.

Yarn

Yarn is a package manager which we can use to manage client-side packages if we choose to (Symfony does not enforce its use). If you already have a Node.js server installed, installing yarn is as simple as sudo npm install -g yarn with -g specifying a global installation. If you don’t have Node.js installed, it’s worth downloading.

yarn install installs the dependencies of a project. You will need to run this (if Yarn is used in the project) when you first work with the project at least once, as well as whenever there’s a change in package.json (which serves the same purpose for Yarn as composer.json for Composer).

Consult Yarn’s documentation for more information.

webpack

webpack bundles client-side files. You can configure webpack to pack file content from a source location and send the packed content to a target location. Read more here.

Encore can be restarted with yarn run encore <env>, which packs files into the target. However, this is unfriendly to the developer, because now each change in client-side code requires us running yarn run encore <env> to make sure that the newest client-side code will be used. This quickly becomes frustrating, since JavaScript and CSS, which never required deployment before, now need to be built. However, you can add the --watch switch like yarn run encore <env> --watch and from there on changes will be watched and detected, so you can be confident that client-side code will end up in the target path and you won’t have to worry about it.

The drawback, of course, is that this will eat up resources and will still be time consuming if the client-side code is very large. As a result, it is much smarter to just use the source path while in development mode. Sometimes browser caching will still cause us trouble while working this way, but a seasoned frontend developer is already used to that.

webpack is a nice tool and can be useful when used well, but it’s a good idea to limit its use. Avoiding webpack in development mode can be very helpful, allowing us to nicely debug and not worry about building client-side code. In production mode it adds a lot of value in obfuscating, minifying, and packing the code.

MVC

Model-View-Controller as a pattern is a standard for Symfony. Model means domain model, usually represented by entity and repository classes along with the business logic that uses them. View is the templating that eventually generates the HTML which will be served to the browser and displayed to the user. Controllers are the engines, the driving force, since the actions allowed to be performed can be found in them, putting everything together.

Doctrine

Doctrine is an ORM, which is very helpful in general for working with databases. Symfony does not enforce its use. You can use Flourishlib instead, or write your own queries. You can create an entity class (which represents a given table in the database) manually, but there is an automatic way to achieve that as well:

php bin/console make:entity

This will start a Q&A, which will be a discussion between the Symfony console and the programmer. After the programmer answers all the questions, the entity class will be generated. Typically these questions will be asked:

Entity name
Property names (until empty name is given)
- Their field type
- Their field length
- Whether they are nullable

The documentation gives us a very good example, resulting in:

// src/Entity/Product.php
namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity(repositoryClass="App\Repository\ProductRepository")
 */
class Product
{
    /**
     * @ORM\Id
     * @ORM\GeneratedValue
     * @ORM\Column(type="integer")
     */
    private $id;

    /**
     * @ORM\Column(type="string", length=255)
     */
    private $name;

    /**
     * @ORM\Column(type="integer")
     */
    private $price;

    public function getId()
    {
        return $this->id;
    }

    // ... getter and setter methods
}

We notice a few things here. First of all, there is an annotation.

/**
 * @ORM\Entity(repositoryClass="App\Repository\ProductRepository")
 */

This means that this entity has a repository class called ProductRepository in the namespace mentioned above.

/**
 * @ORM\Id
 * @ORM\GeneratedValue
 * @ORM\Column(type="integer")
 */

This means that the column is of type integer, a primary key, and it’s a generated value. We notice that the length can be given as well. The data members are private and setters and getters are automatically generated.

The entity class is paired with a repository of its own, like:

namespace App\Repository;

use App\Entity\Product;
use Doctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepository;
use Doctrine\Persistence\ManagerRegistry;

class ProductRepository extends ServiceEntityRepository
{
    public function __construct(ManagerRegistry $registry)
    {
        parent::__construct($registry, Product::class);
    }
}

Note that it extends ServiceEntityRepository. This offers quite a few features if we use it as a Doctrine repository. In the controller we can load a repository, like:

$this->getDoctrine()->getRepository(Product::class)

You can search for objects via find, findBy or findOneBy, as described in the API documentation. This way, we can load data from the database and use them as objects. Example:

$repository = $this->getDoctrine()->getRepository(Product::class);

// look for a single Product by its primary key (usually "id")
$product = $repository->find($id);

// look for a single Product by name
$product = $repository->findOneBy(['name' => 'Keyboard']);
// or find by name and price
$product = $repository->findOneBy([
    'name' => 'Keyboard',
    'price' => 1999,
]);

// look for multiple Product objects matching the name, ordered by price
$products = $repository->findBy(
    ['name' => 'Keyboard'],
    ['price' => 'ASC']
);

// look for *all* Product objects
$products = $repository->findAll();

In general, the criteria to search by is an array of key-value pairs. The same can be said about sorting.

You can load a repository from within another by calling

$this->getEntityManager()->getRepository("name")

However, in general I would suggest that repositories should not have interdependency. It’s far better in my opinion to have a business layer, a Service between the controller and the repository classes, which would combine different repositories’ operations if needed.

Getting or setting values of data members can be done via getters and setters very simply. As the documentation says, persist and remove are methods that notify the Unit Of Work that some write operations should occur, but the write operations are not executed yet at this point. The reason is simple: we may have multiple write operations to do and we might want to avoid doing them all as separate requests to the database, which might be on an entirely different machine on the other side of the globe. Requests to the database are costly operations and they accumulate. To avoid this, the application server first acknowledges what should be done, or, in other words, prepares some write operations. The flush method does the actual request sending for the write operations.

There is no ORM which solves everything, so sometimes we want to write our own scripts. We can of course do it, but Doctrine offers DQL (Doctrine Query Language) as a compromise.

<?php
$query = $em->createQuery('SELECT u FROM ForumUser u WHERE (u.username = :name OR u.username = :name2) AND u.id = :id');
$query->setParameters(array(
    'name' => 'Bob',
    'name2' => 'Alice',
    'id' => 321,
));
$users = $query->getResult(); // array of ForumUser objects

The example above is a parameterized query, where we search by username and id, without knowing what the actual filter values are at the time of development. These values are determined at runtime.

You can apply entity changes upon the database via

php bin/console doctrine:migrations:migrate

Honestly I’m not very fond of changing the database schema based on entity classes; I consider this to be an anti-pattern, because we use a tool to generate a schema based on entities. Such a tool can have bugs or cause errors in the entities. It is much better to do it the other way around—that is, make a proper database schema, write SQL commands to change the schema whenever we need it and generate entity classes that way. Luckily Doctrine offers that feature as well:

php bin/console doctrine:mapping:import "App\Entity" annotation --path=src/Entity

Unfortunately it’s a trend to write entity classes and generate schema changes based on those, but that way we add a layer of complexity around schema planning, which should be as simple as possible. If something goes south with these migrations for any reason, that could result in very serious problems, while if we are to write schema-changing scripts in SQL rather than generating them from entity classes that we write (or generate), we at least have the means to acquire full understanding of what happens with the schema. Not everyone is a database specialist and most programmers are much more comfortable writing or even generating entity classes. But even if we generate them, it’s not much simpler in most cases than writing a small script which creates a table or alters it or drops it, so the objective gain from generating schema-changing scripts from entity classes is negligible at best.

On the other hand, the complex tool that we add is difficult to understand unless one delves into the actual code which generates the schema and the way the schema is generated is out of our hands, unless we are content with the current version of doctrine or accept the price of hacking into the code each time a new version is downloaded.

In short, my advice is to try to avoid generating schema changes based on entity classes whenever possible. We can always find exceptional cases, but in general, SQL is clearer than a tool that generates SQL.

Controller

You can write or generate a controller. This is how you would generate one:

php bin/console make:controller <NamedController>

The command above will actually generate a controller class and ensure that it can be used in the project. Read more here.

You can generate a whole CRUD (support for Create, Read, Update, Delete features) for an entity via

php bin/console make:crud <YourEntity>

The documentation provides a simple example:

// src/Controller/LuckyController.php
namespace App\Controller;

use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;

class LuckyController
{
    /**
     * @Route("/lucky/number/{max}", name="app_lucky_number")
     */
    public function number($max)
    {
        $number = random_int(0, $max);

        return new Response(
            '<html><body>Lucky number: '.$number.'</body></html>'
        );
    }
}

Note that there is annotation for this controller:

/**
 * @Route("/lucky/number/{max}", name="app_lucky_number")
 /*

The first parameter describes the path. Of course we can elaborate our routes more; let’s consider the example below:

namespace App\Controller;

use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;

class MyController extends AbstractController
{
    /**
     * @Route("/home")
     */
    public function home()
    {
        return new Response("home",  Response::HTTP_OK,
            ['content-type' => 'text/plain']);
    }

    /**
     * @Route("/about", methods={"GET", "POST"})
     */
    public function about(Request $request)
    {
        $method = $request->getRealMethod();
        $msg = "about: " . $method;

        return new Response($msg,  Response::HTTP_OK,
            ['content-type' => 'text/plain']);
    }

    /**
     * @Route("/news/{id}", requirements={"page"="\d+"})
     */
    public function news($id)
    {
        $msg = 'News ' . $id;

        return new Response($msg,  Response::HTTP_OK,
            ['content-type' => 'text/plain']);
    }
}

Code taken from zetcode.com.

As we can see, we can specify which HTTP methods we support or even specify regular expressions for the URL parameters. Generating a URL is not difficult at all, one just needs to call $this->generateURL.

Symfony and HTTP

As we could see earlier, Symfony actions are returning a Response. You can send JSON as Response, or even a JSONResponse, as the documentation describes. You can even specify a JSONP callback via

$response->setCallback('handleResponse')

Don’t be afraid of this, it’s not very difficult and it’s well-documented.

Requests are also handled nicely:

$request->query->get('id')

gets a GET parameter, while

$request->request->get('category', 'default value')

is getting a post value.

$request->query->all()

retrieves all GET parameters, while

$request->request->all()

retrieves all POST parameters.

Caching

Symfony supports caching through the Cache module, which, when switched on, will cache configuration and routes. If controller actions or routes are changed, then it is advisable to clear the cache via:

bin/console cache:clear && bin/console cache:warmup

Whether or not to switch on caching for your application at dev mode depends on the frequency of a need to cache cleanup. Read more here.

Configuration

Symfony configuration can be done via YAML, XML or PHP files. In the projects I have been working with, configuration was done in YAML files stored in the config folder inside the root folder of the project.

Inside the src folder we can find Kernel.php which glues together the project.

Twig

Symfony is well-paired with Twig, a popular template engine. You might ask why we don’t use PHP for this purpose. We can use PHP or Smarty or other options for view templating, Twig is not required.

Using template engines instead of allowing the developers to use PHP as a template engine has the drawback of limiting the options. However, this could also be seen as a positive thing. Yes, it limits the options of the developers, but this way developers will not find it so easy and convenient to temporarily (?) implement their business logic as part of the view.

I’m okay with using PHP as a template engine as long as we do not mix up the layers of the application and by convention the view remains “only” the view. I’m also okay with using template engines, even though it is less convenient to use when we need to do some experimenting and code-writing for a very short time, after which it is reverted. In that kind of experimental work enforcing this separation is unnecessary bureaucracy, but the gain is that we will not have to deal with views with unnecessary implementations. So, this is a harmless form of bureaucracy, if such a thing exists.

Summary

Symfony is a Framework which provides us with a way to work in conformity with the MVC pattern. Models are Entity classes along with their Repository pair, representing database tables, while Doctrine watches over the whole process, if used. Views are templates that generate HTML, for which Twig is a convenient tool, but you can use Symfony without relying on the usage of Twig. Configuration can be written in YAML, XML or PHP. PHP dependencies are handled by Composer, client-side dependencies can be handled by Yarn. Symfony has its own console that can be used to run different commands.

All in all, there are lots of features which Symfony offers, so using Symfony a good option to consider when planning a project. Getting started with Symfony can be difficult, but I hope this article will help you get started using Symfony in your projects.

Seeking a PHP developer

2019-10-17T00:00:00+00:00

We are looking for another PHP software engineer, to work with us during business hours somewhere in the UTC-7 to UTC-4 time zones (U.S. Pacific to Eastern Time). This is a contract-to-hire role, and can be full time or part time.

End Point is a 23-year-old Internet technology consulting company based in New York City, with about 50 employees, most working remotely from home offices. We collaborate using SSH, GitLab, GitHub, chat, video conferencing, and good old email and phones.

We serve many clients ranging from small family businesses to large corporations.

What you will be doing:

Develop new web applications and support existing ones for our clients
Work together with End Point co-workers and our clients’ in-house staff
Use your desktop OS of choice: Linux, macOS, Windows
Use open source tools and contribute back as opportunity arises

What you bring:

Professional experience with web application development and support:

5–7 years of development with PHP (older 5.x and newer 7.x versions) and JavaScript
Frameworks such as Symfony or Laravel
Databases such as PostgreSQL, MySQL, MongoDB, Redis, Solr, Elasticsearch, etc.
Security consciousness
Git version control
Automated testing
HTTP, REST APIs, and/or SOAP
Magento experience is a plus!

And just as important:

Strong verbal and written communication skills
A good remote work environment
An eye for detail
Tenacity in solving problems
Ownership of projects to get things done well
Work both independently and as part of a team
Focus on customer needs

What work here offers:

Flexible work hours
Annual bonus opportunity
Freedom from being tied to an office location
Collaborate with knowledgeable, friendly, and diligent co-workers around the world
For full-time staff: paid holidays and vacation
For U.S. employees: health insurance subsidy and 401(k) retirement savings plan

Getting in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.) Include your location, a resume/CV, your GitHub or LinkedIn URLs, or whatever else helps us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

How to set up a local development environment for WordPress from scratch

2019-08-07T00:00:00+00:00

I recently got pulled into a project for a client who wanted to have a new WordPress website developed for them. I started by setting up a development environment with the niceties that I’m used to from my other application development work. That is, a development server, interactive debugging, linting, and a good editor.

Another thing that I wanted was not to have to deal with LAMP or WAMP or XAMPP or any of that. I wanted a clean, from scratch installation where I knew and controlled everything that was there. I’ve got nothing against those packages, but I think that, by setting up everything manually, I’d be able to better learn the technology as I would know exactly how everything is set up under the hood. The shortcuts could come later.

Luckily for me, there aren’t many pieces when it comes to setting up a basic, running development environment for WordPress. You only need three things: 1. MySQL, 2. PHP, and 3. WordPress itself. I also wanted a few other goodies and we’ll get there.

Let’s go through the steps that I took to set all of this up:

1. Set up PHP

In Ubuntu, installing PHP is easy enough. Just run the following command:

sudo apt-get install php

After that’s done, run php -v to validate that it was successfully installed. It should result in something like this:

PHP 7.2.19-0ubuntu0.18.04.1 (cli) (built: Jun  4 2019 14:48:12) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.19-0ubuntu0.18.04.1, Copyright (c) 1999-2018, by Zend Technologies

There’s one particular PHP extension that we’re going to need. Let’s install it with:

sudo apt-get install php-mysql

The php-mysql extension is necessary for our PHP installation to interact with MySQL. That’s all that’s needed to run WordPress as far as PHP is concerned.

2. Set up MySQL

WordPress uses MySQL for all of its data storage concerns. So, let’s install it. Again, in Ubuntu, installing and setting up MySQL is super easy. First, we need to run this command:

sudo apt-get install mysql-server

This will install both the MySQL database engine and a command-line client for us to connect to it and do some initial configuration. We now need to log into our freshly installed MySQL instance. But first, make sure that it’s running with:

sudo service mysql start

Now that our instance is running, log into it as root with:

sudo mysql -u root

This will open up the MySQL command-line client where we can do some initial configuration to support WordPress.

We now need to create a new MySQL user that will be used by WordPress to log into the database. You can do so with a command like this from within the MySQL CLI client:

CREATE USER 'wordpress_user'@'localhost' IDENTIFIED BY 'password';

Obviously, choose a username and password that work for you. I like to keep things simple and obvious so that’s what I use. Also obviously, use a strong, unique password in a production environment.

Now, create a new database that will be used by WordPress with:

CREATE DATABASE wordpress_dev;

Again, feel free to choose a name that suits your needs.

Now, we need to allow the user that we created a few steps ago to access and control that new database. Since this is only a dev environment, let’s just give our WordPress user access to everything. This can be done with:

GRANT ALL ON *.* TO 'wordpress_user'@'localhost'

With that, we’re done with MySQL, we can close the CLI client with the exit command.

3. Set up WordPress

Now that we have our prerequisites ready, we can proceed to setting up the actual WordPress site. It turns out, this is pretty easy as well. Get a new directory ready and let’s get started.

First, we need to download the package of WordPress files from the official site. In Ubuntu, this can be done with this command:

wget https://wordpress.org/latest.tar.gz

This will result in a new latest.tar.gz file being created in your directory. Now, extract it with:

tar xvzf latest.tar.gz

Then, enter the new wordpress directory that gets created as a result of the last operation. Explore the wordpress directory and you should be able to see a bunch of wp-* files and directories. These are all the files that WordPress needs to run.

Before running WordPress though, we need to configure it so that it uses the MySQL database that we just created. We do this by specifying that configuration in a wp-config.php file. This file does not exist yet, but we have a wp-config-sample.php file that we can use as a template. Create the new wp-config.php file based on wp-config-sample.php with the following command:

cp wp-config-sample.php wp-config.php

Now, in the new wp-config.php file, put in your MySQL database information. Starting around line 23, it should look like this:

/** The name of the database for WordPress */
define( 'DB_NAME', 'wordpress_dev' );

/** MySQL database username */
define( 'DB_USER', 'wordpress_user' );

/** MySQL database password */
define( 'DB_PASSWORD', 'password' );

/** MySQL hostname */
define( 'DB_HOST', 'localhost' );

/** Database Charset to use in creating database tables. */
define( 'DB_CHARSET', 'utf8' );

/** The Database Collate type. Don't change this if in doubt. */
define( 'DB_COLLATE', '' );

4. Run WordPress

Now we’re finally ready to actually run WordPress. WordPress, as a web application, needs a web server like Apache to run. We don’t have Apache though; what we have is PHP’s built-in development web server. From within our wordpress directory, we can fire up the built-in web server with:

php -S localhost:3000

Now it’s just a matter of navigating to the localhost:3000/wp-admin/install.php page in your browser of choice. This page should show up on screen:

Just follow the steps within the wizard at install.php and your new development WordPress site will be ready to go in no time.

Bonus 1: Set up a linter: PHP Code Sniffer

Prerequisites: Composer and the php-xml extension

You can install Composer by following these instructions. In Ubuntu, installing the php-xml extension can be done with:

sudo apt-get install php-xml

Now, to set up the PHP Code Sniffer linter, just follow these steps, from your wordpress directory:

Install PHP Code Sniffer with composer require --dev squizlabs/php_codesniffer.
Download the WordPress PHP Code Sniffer standards with composer require --dev wp-coding-standards/wpcs.
Install it into phpcs with: vendor/bin/phpcs --config-set installed_paths vendor/wp-coding-standards/wpcs.
Sniff something with vendor/bin/phpcs --standard=WordPress index.php.

If all goes well, you should be able to see reports like this (which is exaggerated for demonstration purposes; your default index.php file will not show as many warnings):

FILE: /home/kevin/projects/random/wordpress/index.php
-----------------------------------------------------------------------------------------
FOUND 5 ERRORS AND 1 WARNING AFFECTING 2 LINES
-----------------------------------------------------------------------------------------
 14 | ERROR   | [x] Expected 1 spaces after opening bracket; 0 found
 14 | ERROR   | [x] Expected 1 spaces before closing bracket; 0 found
 17 | WARNING | [x] "require" is a statement not a function; no parentheses are required
 17 | ERROR   | [x] Expected 1 spaces after opening bracket; 0 found
 17 | ERROR   | [x] Expected 1 spaces before closing bracket; 0 found
 17 | ERROR   | [x] Concat operator must be surrounded by a single space
-----------------------------------------------------------------------------------------
PHPCBF CAN FIX THE 6 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-----------------------------------------------------------------------------------------

Time: 103ms; Memory: 10MB

Bonus 2: Setup interactive debugging with VS Code

Prerequisites: VS Code, XDebug and the PHP Debug VS Code extension

For writing PHP, my editor of choice is VS Code. You can get the editor from the official download site. VS Code has a huge extensions ecosystem where you can find almost anything. Naturally, there’s a debugger for PHP. It’s aptly called PHP Debug. See the instructions on how to set it up. The PHP Debug extension works on top of XDebug, a debugger for PHP. Luckily for us, PHP Debug’s instructions page includes all the details on how to install and set up both itself and XDebug.

If you followed the instructions, you should now have a new .vscode/launch.json file within your wordpress directory with the following contents:

{
   "version": "0.2.0",
   "configurations": [
       {
           "name": "Listen for XDebug",
           "type": "php",
           "request": "launch",
           "port": 9000
       }
   ]
}

This is what’s called a launch configuration in VS Code speak. This tells VS Code’s debugger all the info it needs to attach to a running PHP process. To see it in action, fire up the built-in web development server with:

php -S localhost:3000

Then, click the Start Debugging button. That’s the green triangle icon near the top of the screen, when you select the Debug sidebar in VS Code. It should have the Listen for XDebug option selected.

Now it’s just a matter of putting a breakpoint anywhere within the source code and request the page from your browser. You can set a breakpoint by clicking right next to the line number indicator, within any file. Here, I’ve put a breakpoint on line 14 in index.php:

When the code execution hits the breakpoint, it should stop right there and allow you to inspect variables and the like, just like any other debugger.

And that’s all for now! Hopefully this little write-up can help you jumpstart your next WordPress development project.

Running Magento 2 in Windows with XAMPP

2019-03-22T00:00:00+00:00

Photo by Nicole De Khors · Burst, Some Rights Reserved

Magento is an open source ecommerce platform, written in PHP and relying on MySQL/MariaDB for persistence. According to BuiltWith, Magento is the third most used platform in ecommerce websites. It began its life in 2008 with its first general release, and a major update (Magento 2) was released in 2015.

And now, more than three years after, Magento 1 is slowly dying: There won’t be any more quality fixes or security updates from June 2020, and there won’t be extended support for fixes or new payment methods. So the obvious choice will be Magento 2 from now on.

But is it fully tested yet? Is it stable enough? If we already have a website running with Magento 1, what should we do? Migrating to Magento 2 is not just hitting an “Update” button: Themes are incompatible, most extensions won’t work, and of course, there’s a big set of changes to get familiar with.

So a good approach might be to get a clean Magento 2 version deployed locally, to look what we need to do to get our website updated and running, test the backend, find where the configuration sections are located, and so on. And many business users, and even some developers like myself, have Microsoft Windows installed on our computers.

Environment setup

The environment I used for this testing installation was Windows 10 Professional. As a first step, we’ll need to make sure that localhost is published in our local hosts file:

Navigate to the folder %SystemRoot%\system32\drivers\etc
Backup the existing hosts file
Open a text editor with administrator rights
Open the hosts file
Make sure the first line after the commented (#) lines is 127.0.0.1 localhost and the second is ::1 localhost
Open a cmd window with administrator rights and run the command ipconfig /flushdns

Now we’re ready to install the environment needed to run Magento. I recommend using XAMPP, a free Apache distribution for Windows that includes MariaDB, PHP, and Perl in a single package. Magento 2 currently runs with PHP 7.2 and it will not work with newer versions.

Install XAMPP 7.2 with default settings: apachefriends.org/xampp-files/7.2.15/xampp-win32-7.2.15-0-VC15-installer.exe
Download Magento 2. You will need to register on the website first: magento.com/tech-resources/download
Create a new “magento” subfolder (or whatever name you prefer) inside the htdocs folder in the XAMPP installation (usually C:\xampp) and uncompress the Magento 2 archive there.
Start the XAMPP Control Panel from the Windows start menu. In the “Apache” section, click the “Config” button and, on the menu that appears, select “PHP (php.ini)”. Remove the semicolon before the extension=intl, extension=soap, and extension=xsl texts to enable the intl, soap and xsl extensions.

Screenshot of the XAMPP Control Panel with the PHP config menu displayed.

MySQL and Magento setup

We have all the files in place and the environment ready to start configuring the database and install Magento 2.

Start the Apache and MySQL services from the XAMPP Control Panel. Wait for the green status texts to appear.
Create a “magento” (or whatever name you prefer) database in MySQL from phpMyAdmin, installed already on XAMPP: localhost/phpmyadmin
Run the Magento 2 setup from localhost/magento (replace the “magento” part of the URL with whatever name you have chosen to host Magento). If the setup program requires to do any additional configuration change, do it as instructed. Do a screenshot or save the final page contents for later use.

Example of the page that will be displayed when the Magento 2 installation finishes.

Fixing known issues

When we finish the installation process, we will have a Magento 2 instance running on our host. But we’re not ready yet! There are a couple known bugs with Magento 2 and XAMPP at the moment I’m writing this post:

The Magento admin page may not work (it shows a blank page): Fix this issue by updating the “isPathInDirectories” function inside the Validator.php file as instructed in this article.
We might not be able to upload images (like a custom logo or product pictures): Fix this issue by updating the design_config_form.xml file as shown here.

After we finish making these changes, we will need to restart the Apache service from the XAMPP control panel. And that’s it! We should be ready to open the Magento front page, and login into the backend.

An empty home page with the default theme enabled in Magento 2.

This is how the Magento 2 back-end dashboard looks like once we login.

Setting up the store: A roadmap

The purpose of this article is to get Magento 2 up and running on Windows computer, and that’s what I hope we’ve achieved so far. But just to point in a direction on what to do next, below is my usual roadmap to set up the store, add a few products and publish the catalog on the home page.

Change the default store name and logo from the section Content > Design > Configuration and choose “Edit” in the “Default store view”.
Set up the default store configuration from the section Stores > Configuration.
Create the product categories from the section Catalog > Categories. Set up a category name and its dependency from another category, or from the default master category.
Create/edit the product attributes from the section Stores > Attributes (check the option “visible in storefront” to show the products in the product page). For example, custom attributes for book products would be Author, Publisher…
Add the attributes to the default attribute set from the section Stores > Attribute. This is required for the attributes to show up when adding a new product by default.
Create a set of products from the section Catalog > Products. Set the custom variations for each product from the “Customizable options” section. Set up a short and long description, a custom SKU for each customizable option, and a tax mode.
Add a widget to the homepage to show the product catalog from the section Content > Homepage > Edit > Show/Hide Editor > Add widget > Products list. Here we can also add any content we like, including HTML tags, images, and third-party content.

Conclusion

It’s possible—and pretty straightforward—to run a Magento 2 instance in Windows with XAMPP. But this is just a kick-off, a simple overview: there’s a lot more to do if we want to get a full ecommerce website that is ready to scale up and support enterprise-level traffic, like caching optimization, load balancing, and advanced server-side monitoring.

At End Point, we have professionals with a strong background in deploying powerful, reliable, fully responsive, fast, and SEO-optimized ecommerce websites. We have experience in migrating Magento to newer versions, and even creating full Magento 2 websites from scratch. Drop us a line if you want to hear more.

How to use Zend Framework components in a console app

2018-11-21T00:00:00+00:00

When Rasmus Lerdorf created PHP in the ’90s, I bet he never thought that his language would become the engine that powers much of the web, even today, 23 years later.

PHP is indeed super popular. Part of that popularity stems for the fact that it can run pretty much anywhere. You’d be hard pressed to find a hosting solution that doesn’t support PHP out of the box, even the cheapest ones. It is also fully functional in pretty much any version of the most used operating systems so development and deployment have a very low barrier of entry.

Both because and as a result of this ubiquity and popularity, PHP also boasts an expansive, active community and a rich ecosystem of documentation, tools, libraries and frameworks. Zend Framework is a great example of the latter.

I started using Zend Framework a good while ago now, when it became clear to me that writing vanilla PHP wasn’t going to cut it anymore for moderately sized projects. I needed help and Zend Framework extended a very welcomed helping hand. Zend Framework is basically a big collection of libraries and frameworks that cover most of the needs that we would have as web developers building PHP applications.

While most parts of Zend Framework are standalone components that we can plug into an existing code base, picking and choosing what’s needed, there’s also the likes of Zend MVC. Zend MVC is, in its own right, a framework for developing web applications using the MVC design pattern (shocker, I know). A Zend MVC application is itself comprised of several other components from Zend Framework’s library like Zend Form (which helps with developing HTML forms), Zend DB (which helps with database access), Zend Validator (which helps with input validation), and many more.

Obviously, these are very useful within the context of web applications but they can also be valuable on their own. One could put together a PHP console app and make one’s life much easier by leveraging the features provided by these libraries. Coming back to the point of PHP’s ubiquity, it is very easy to write such a script that runs in a server where a PHP web app is already deployed. No installation of an additional language interpreter needed, which means no need for special server permissions. No fuss, just good old PHP, which is already running on the server, being used to perform some backend task, some database maintenance, some file downloading, etc.

So, the question becomes, how do we leverage the power of Zend Framework components outside the context of a Zend MVC application? Composer is the key.

Composer is a widely used package manager and module loader for PHP, and it is what powers Zend MVC applications’ module resolution engine.

In vanilla PHP, the include and require statements are the main forms of code reuse. When we want to bring functions or classes defined in other files to our script, these are what we use. PHP also supports namespaces. In their most abstract definition, namespaces are things used for organizing other things. In a programming language, they are little more than names (series of characters) that group together program constructs like classes, functions, variables, etc.

Some PHP file in our source code could look like this:

// calc.php
<?php

namespace Application\MathTools;

class ArithmeticCalculator
{
    public function add($x, $y)
    {
        return $x + $y;
    }
}

The namespace statement near the top says that our ArithmeticCalculator is in the Application\MathTools namespace.

Elsewhere in our project, we could have something like this:

// index.php
<?php

use Application\MathTools\ArithmeticCalculator;

$calc = new ArithmeticCalculator();

$result = $calc->add(5, 6);

echo $result;

In this file, we’re getting a reference to the ArithmeticCalculator via its namespace. That is, we’re calling it with its fully qualified name.

The funny thing, though, is that namespaces do not serve to resolve dependencies. That is, just because we write use Application\MathTools\ArithmeticCalculator; does not mean that PHP knows where the file that actually contains the class is located. In other words, the use statement doesn’t know how to go out of the current file and look for the corresponding namespace statements in other files to figure out what to include. Given that, the savvy reader would be able to deduce that the previous code doesn’t actually work. In fact, if we run this as it is, it will most surely throw an exception with PHP complaining about just what we discussed.

Not fun. We’re asking PHP to use a namespace that’s not in the current executing context. To fix the code above we just need to add our trusty include (or require) statement. Like this:

// index.php
<?php

include 'calc.php';

// ... rest of the code ...

Now it does work. Well, hurray, I guess. It’s still kind of a bummer because we now have to use BOTH include/require AND use if we want to use namespaces. Imagine how quickly will that become a mess in a project with a good number of files. Thankfully, Composer can help us with that.

Alright, after that long tangent about requiring and including scripts and using namespaces, finally Composer comes back into the picture. Composer includes what’s called an autoloader. An autoloader is precisely what we need here because we can use it to make it so our scripts don’t need to use require/include statements anymore. The autoloader takes notice of the use statements and then includes the actual files where the source code lives as it sees fit. Alright, let’s see how we can create a Composer managed PHP project and use the files that we already have.

First of all, make sure to install Composer from their official website.

Let’s create a new directory to work in. I’m going to call mine my-console-app. Go into it and run composer init in your console.

This will result in Composer’s config generator showing up. It is an interactive prompt that helps you through the setup of your project’s composer.json file. That file’s purpose is to store references to all of your dependencies as well as some general project information like name, description, author, license and the like. For now, just hit enter a few times until asked whether you’d like to interactively define your dependencies for both dev and prod. Say no to those and then finally say yes when asked to confirm generation. When all is said and done, the whole thing should look like this:

$ composer init


  Welcome to the Composer config generator



This command will guide you through creating your composer.json config.

Package name (<vendor>/<name>) [kevin/my-console-app]:
Description []:
Author [Kevin Campusano <kcampusano@endpointdev.com>, n to skip]:
Minimum Stability []:
Package Type (e.g. library, project, metapackage, composer-plugin) []:
License []:

Define your dependencies.

Would you like to define your dependencies (require) interactively [yes]? n
Would you like to define your dev dependencies (require-dev) interactively [yes]? n

{
    "name": "kevin/my-console-app",
    "authors": [
        {
            "name": "Kevin Campusano",
            "email": "kcampusano@endpointdev.com"
        }
    ],
    "require": {}
}

Do you confirm generation [yes]? y

You should have a brand new composer.json file created with that final JSON content that composer init showed us in the console.

Alright, now let’s, in that same directory, create two PHP files with the contents that we saw above. Here they are again for your convenience:

// calc.php
<?php

namespace Application\MathTools;

class ArithmeticCalculator
{
    public function add($x, $y)
    {
        return $x + $y;
    }
}

// index.php
<?php

use Application\MathTools\ArithmeticCalculator;

$calc = new ArithmeticCalculator();

$result = $calc->add(5, 6);

echo $result;

The directory now looks like this:

$ ls
calc.php  composer.json  index.php

This, right now, still doesn’t work. We need to wire up the autoloading. First, let’s tell Composer which files we want to autoload by adding this to our composer.json file.

// composer.json
{
    // ...
    "autoload": {
        "classmap": [
            "calc.php"
        ]
    },
    // ...
}

Here, we just use a file name directly but it also supports directories.

Next, we need to generate the actual runtime component that will do the autoloading. We do this with this command:

composer dump-autoload -o

This creates a series of files that contain the actual autoloading logic to fulfill what we described in our composer.json file. They live inside a new vendor directory created at the root of our project. A quick ls reveals the current state of our directory:

$ ls
calc.php  composer.json  index.php  vendor

This new vendor directory is also where Composer stores all of the dependencies that we download and add to our project. More on that later though.

Ok, so back to our code. The only thing remaining before we can enjoy Composer’s autoloading feature is to include the file generated in the previous step in our script. Let’s add the following line at the top of our index.php file:

// index.php
<?php
include 'vendor/autoload.php';

// ...

Now, run index.php in your favorite web server or in the console and bask in the glory of hardcoded simple arithmetics.

$ php index.php
11

Pretty sweet, huh? So, what have we done so far? We’ve created a Composer project by setting up a composer.json file. We’ve developed a simple application that adds two hardcoded numbers using a separate calculator class. That class is defined in a separate file which is annotated with a namespace. It gets included in our main application entry point (i.e. index.php) using Composer’s autoloading feature. That is, instead of including the file directly with something like include 'calc.php';, we’ve included it using namespaces and use statements like use Application\MathTools\ArithmeticCalculator;. The only gotcha is that, in order to enable autoloading in our main file, we need to include the autoloader using include 'vendor/autoload.php';. We generated this file by configuring the composer.json file with the list of files that contain classes that we want to autoload and then running Composer’s own composer dump-autoload -o command.

Ok, now that we’ve learned all that, we’re ready to get back to our initial question: “How do we leverage the power of Zend Framework components outside the context of a Zend MVC application?”

As we’ve seen already, we can run our application in multiple ways. We can serve our directory in a web server and access index.php from a browser. We can also just run it from the console with something like php index.php. The result will be the same: the addition of 5 plus 6 printed to either the console or the response buffer. This is a console application so let’s stick with running it via the console.

Now, as we’ve discussed before, Zend Framework offers many libraries that we can use. For the purposes of our demonstration, let’s use the Zend Validator library. We can install it in our project by running:

composer require zendframework/zend-validator

After a few seconds of Composer printing out stuff to the console, the package will be installed. The composer.json file will be updated and will include the new package in the require section. Like this:

// composer.json
{
    // ...
    "require": {
        "zendframework/zend-validator": "^2.10"
    }
}

The required files were downloaded into the vendor folder but we don’t really care too much about that right now. It’s important to note though, that we should never be changing anything in there, just let Composer do its job. Now, back in our index.php file we can “use” our new validator package and start validating some stuff. Our file could look like this:

<?php
include 'vendor/autoload.php';

use Zend\Validator\EmailAddress;
use Zend\Validator\CreditCard;

// Imagine these come from a file, database or http request or something
$emailsToValidate = [
    "kcampusano@endpointdev.com",
    "what is this?",
    312321,
    null
];

$creditCardsToValidate = [
    "4111111111111111",
    "what is this?",
    9128173,
    null
];

echo "Validating emails... \n\n";

$emailValidator = new EmailAddress();
foreach ($emailsToValidate as $email) {
    if ($emailValidator->isValid($email)) {
        echo "✔ the {$email} email is valid\n";
    } else {
        echo "✕ the {$email} email is not valid\n";
    }
}

echo "\n";

echo "Validating credit cards... \n\n";

$creditCardValidator = new CreditCard();
foreach ($creditCardsToValidate as $card) {
    if ($creditCardValidator->isValid($card)) {
        echo "✔ the {$card} credit card is valid\n";
    } else {
        echo "✕ the {$card} credit card is not valid\n";
    }
}

echo "\n";

Demonstrating the breadth of functionality that we can take advantage of by using the Zend Validator component is obviously well beyond the scope of this post. However, in the example above, we can see what including such a component in our scripts could look like. We put together a simple credit card and email validator that works on incoming arrays of data. All we had to do was run composer require zendframework/zend-validator to install our package and put this line at the beginning of our script include 'vendor/autoload.php';. We didn’t even have to generate the autoloader like we did with our custom ArithmeticCalculator class. This is because Composer knows how to deal with dependencies installed with composer require without us needing to tell it so.

And that concludes today’s post about how to use Zend Framework components in a console app in PHP. As a bonus, we also learned about code reuse with require and include, what namespaces are, how to use them, what they can and cannot do, what is Composer, how to set up a project using it and how to use it for autoloading modules. That’s a pretty good bang for your buck.

You can find all the source code that we wrote for this post in this GitHub repo.

Start basic application with Vue.js 2 and Drupal 8

2018-04-05T00:00:00+00:00

Introduction

The purpose of creating this post is to show how fast can you build web applications with Vue.js on the front-end and Drupal on the back-end side.

Let’s call our project “Awesome Nerds”.

What do we need?

Debian/Ubuntu system
Internet
30 minutes
Vagrant & VirtualBox
Git
Vim
Yarn

Step by step

Here’s what we’re going to do:

Install Vagrant, VirtualBox, and Git
Setup a new Drupal 8 project that will be our back-end project
Setup a new Vue.js project that will be our front-end application
Let’s code

Install Vagrant, VirtualBox, and Git

Open your console and run:

$ sudo apt-get install software-properties-common - getting some common libraries

$ sudo apt-add-repository ppa:ansible/ansible

$ sudo apt-get update

$ sudo apt-get install ansible - installing Ansible

$ wget ‘https://releases.hashicorp.com/vagrant/2.0.2/vagrant_2.0.2_x86_64.deb’ && dpkg -i vagrant_2.0.2_x86_64.deb - installing Vagrant

$ sudo apt-get install dkms

$ deb https://download.virtualbox.org/virtualbox/debian <mydist> contrib

$ wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -

$ wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -

$ sudo apt-get update

$ sudo apt-get install virtualbox-5.2 git vim nfs-kernel-server - installing VirtualBox

$ vagrant plugin install vagrant-vbguest

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

$ sudo apt-get update && sudo apt-get install yarn - installing Yarn

Now we can continue to the back-end project installation.

Install and setup back-end project

Create a new folder for the project:

$ mkdir awesome_nerds && cd awesome_nerds

$ mkdir frontend backend

We need to clone the DrupalVM repository. DrupalVM is a Drupal setup that helps to encapsulate services with Vagrant. Run:

$ cd backend

$ git clone git@github.com:geerlingguy/drupal-vm.git .

Let’s name our project:

$ vim default.config.yml

Look and set these two settings so they look like this:

vagrant_hostname: awesomenerds.backend                                   
vagrant_machine_name: awesomenerds_backend

Quit Vim and run:

$ vagrant up

It will take a while to set up everything, you can get a coffee or browse some memes or go to the next chapter and start creating our front-end project.

When it’s ready we will have a running Drupal 8 setup with MySQL, PHP 7, and Apache (you can configure this stack in default.config.yml if you prefer nginx for example).

Drupal project files are in the drupal directory and that’s the only folder that you would want to add to a project Git repository.

Setup new Vue.js project

We will use a minimal project skeleton from https://github.com/vuejs-templates/webpack.

Run:

$ cd awesome_nerds/frontend

$ git clone https://github.com/vuejs-templates/webpack .

$ yarn install -g vue-cli

$ vue init webpack awesome_nerds

Name the project “awesome_nerds” (yes!) and just hit enter to install with defaults.

When you run:

$ yarn run dev

you will get a fresh Vue.js application running on http://localhost:8080.

Let’s code!

Now we are ready for development. It can be really rapid, both Vue.js 2 and Drupal 8 are impressively good and it’s just a matter of finding a good idea for your new start-up.

In my next post I will continue and code a simple social application using the REST API of Drupal and our Vue.js front-end.

Thank you and good luck!

Regular Expression Inconsistencies With Unicode

2018-01-23T00:00:00+00:00

A casual stroll through the world of Unicode and regular expressions—Photo by Presidio of Monterey

Character classes in regular expressions are an extremely useful and widespread feature, but there are some relatively recent changes that you might not know of.

The issue stems from how different programming languages, locales, and character encodings treat predefined character classes. Take, for example, the expression \w which was introduced in Perl around the year 1990 (along with \d and \s and their inverted sets \W, \D, and \S).

The \w shorthand is a character class that matches “word characters” as the C language understands them: [a-zA-Z0-9_]. At least when ASCII was the main player in the character encoding scene that simple fact was true. With the standardization of Unicode and UTF-8, the meaning of \w has become a more foggy.

Perl

Take this example in a recent Perl version:

use 5.012; # use 5.012 or higher includes Unicode support
use utf8;  # necessary for Unicode string literals

print "username" =~ /^\w+$/; # 1
print "userاسم"  =~ /^\w+$/; # 1

Perl is treating \w differently here because the characters “اسم” (“ism” meaning “name” in Arabic) definitely don’t fall within [a-zA-Z0-9_]!

Beginning with Perl 5.12 from the year 2010, character classes are handled differently. Documentation on the topic is found in perlrecharclass. The rules aren’t as simple as with some languages, but can be generalized as such:

\w will match Unicode characters with the “Word” property (equivalent to \p{Word}), unless the /a (ASCII) flag is enabled, in which case it will be equivalent to the original [a-zA-Z0-9_].

Let’s see the /a flag in action.

use 5.012;
use utf8;

print "username" =~ /^\w+$/a; # 1
print "userاسم"  =~ /^\w+$/a; # 0

However, you should know that for code points below 256, these rules can change depending on whether Unicode or locale rules are on, so if you’re unsure, consult the perlre and perlrecharclass.

Keep in mind that these same questions of what the character classes include can apply to every predefined character class in whatever language you’re using, so remember to check language-specific implementations for other character class shorthands, such as \s and \d, not just \w.

Every language seems to do regular expressions a little bit differently, so here’s a short, incomplete guide for several other languages we use frequently.

Python

Take this example in Python 3.6.2:

>>> re.match(r'^\w+$', 'username')
<_sre.SRE_Match object; span=(0, 8), match='username'>
>>> re.match(r'^\w+$', 'userاسم')
<_sre.SRE_Match object; span=(0, 7), match='userاسم'>

Python is also treating \w differently here. Let’s take a look at the Python docs:

\w

For Unicode (str) patterns:

Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched (but the flag affects the entire regular expression, so in such cases using an explicit [a-zA-Z0-9_] may be a better choice).

For 8-bit (bytes) patterns:

Matches characters considered alphanumeric in the ASCII character set; this is equivalent to [a-zA-Z0-9_]. If the LOCALE flag is used, matches characters considered alphanumeric in the current locale and the underscore.

So \w includes “most characters that can be part of a word in any language, as well as numbers and the underscore”. A list of the characters that includes is difficult to pin down, so it would be best to use the re.ASCII flag as suggested when you’re unsure if you want letters from other languages matched:

>>> re.match(r'^\w+$', 'userاسم',  flags=re.ASCII)
>>> re.match(r'^\w+$', 'username', flags=re.ASCII)
<_sre.SRE_Match object; span=(0, 8), match='username'>

Ruby

Ruby’s Regexp class documentation gives a simple and useful explanation: backslash character classes (e.g. \w, \s, \d) are ASCII-only, while POSIX-style bracket expressions (e.g. [[:alnum:]]) include other Unicode characters.

irb(main):001:0> /^\w+$/         =~ "userاسم"
=> nil
irb(main):002:0> /^[[:word:]]+$/ =~ "userاسم"
=> 0

JavaScript

JavaScript doesn’t support POSIX-style bracket expressions, and its backslash character classes are simple, straightforward lists of ASCII characters. The MDN has simple explanations for each one.

JavaScript regular expressions do accept a /u flag, but it does not affect shorthand character classes. Consider these examples in Node.js:

> /^\w+$/.test("username");
true
> /^\w+$/.test("userﺎﺴﻣ");
false
> /^\w+$/u.test("username");
true
> /^\w+$/u.test("userﺎﺴﻣ");
false

We can see that the /u flag has no effect on what \w matches. Now let’s look at Unicode character lengths in JavaScript:

> '❤'.length
1
> '👩'.length
2
> '🀄️'.length
3

Because of the way Unicode is implemented in JavaScript, strings with Unicode characters outside the BMP (Basic Multilingual Plane) will appear to be longer than they are.

This can be accounted for in regular expressions with the /u flag, which only corrects character parsing, and does not affect shorthand character classes:

> let mystr = "hi👩there";
undefined
> mystr.length
9
> /hi.there/.test(mystr);
false
> /hi..there/.test(mystr);
true
> /hi.there/u.test(mystr);  # note the /u from here on
true
> /hi..there/u.test(mystr);
false
> /hi..there/u.test("hi👩👩there");
true

The excellent article "💩".length === 2 by Jonathan New goes into detail about the why this is, and explores various solutions. It also addresses some legacy inconsistencies, like how the old HEAVY BLACK HEART character and other older Unicode symbols might be represented differently.

PHP

PHP’s documentation explains that \w matches letters, digits, and the underscore as defined by your locale. It’s not totally clear about how Unicode is treated, but it uses the PCRE (Perl Compatible Regular Expressions) library which supports a /u flag that can be used to enable Unicode matching in character classes:

<?php

echo preg_match("/^\\w+$/", "username"), "\n";  # 1
echo preg_match("/^\\w+$/", "userاسم"),  "\n";  # 0

echo preg_match("/^\\w+$/u", "username"), "\n"; # 1
echo preg_match("/^\\w+$/u", "userاسم"),  "\n"; # 1

.NET

The .NET Quick Reference has a comprehensive guide to character classes. For word characters, it defines a specific group of Unicode categories including letters, modifiers, and connectors from many languages, but also points out that setting the ECMAScript Matching Behavior option will limit \w to [a-zA-Z_0-9], among other things. Microsoft’s documentation is clear and comprehensive with great examples, so I recommend referring to it frequently.

Go

Go follows the regular expression syntax used by Google’s RE2 engine, which has easy syntax for specifying whether you want Unicode characters to be captured or not:

package main

import (
	"fmt"
	"regexp"
)

func main() {
	// Perl-style
	fmt.Println(regexp.MatchString(`^\w+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^\w+$`, "userاسم"))  // false

	// POSIX-style
	fmt.Println(regexp.MatchString(`^[[:word:]]+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^[[:word:]]+$`, "userاسم"))  // false

	// Unicode character class
	fmt.Println(regexp.MatchString(`^\pL+$`, "username")) // true
	fmt.Println(regexp.MatchString(`^\pL+$`, "userاسم"))  // true
}

You can see this code in action here.

grep

Implementations of grep vary widely across platforms and versions. On my personal computer with GNU grep 3.1, \w doesn’t work at all with default settings, matches only ASCII characters with the -P (PCRE) option, and matches Unicode characters with -E:

[phin@caballero ~]$ grep    "^\w+$" <(echo "username")  # no match
[phin@caballero ~]$ grep -P "^\w+$" <(echo "username")
username
[phin@caballero ~]$ grep -P "^\w+$" <(echo "userاسم")   # no match
[phin@caballero ~]$ grep -E "^\w+$" <(echo "username")
username
[phin@caballero ~]$ grep -E "^\w+$" <(echo "userاسم")
userاسم

Again, implementations vary a lot, so double check on your system before doing anything important.

Drupal — rapid development

2017-05-26T00:00:00+00:00

Here at End Point, we had the pleasure to be a part of multiple Drupal 6, 7 and 8 projects. Most of our clients wanted to use the latest Drupal version, to have a long term support, stable platform.

A few years ago, I already had big experience with PHP itself and other, various PHP frameworks like WordPress, Joomla! or TYPO3. I was happy to use all of them, but then one of our clients asked us for a simple Drupal 6 task. That’s how I started my Drupal journey which continues until now.

To be honest, I had a difficult start, it was different, new and pretty inscrutable for me. After a few days of reading documentation and playing with the system I was ready to do some simple work. Here, I wanted to share my thoughts about Drupal and tell you why I LOVE! it.

Low learning curve

It took, of course, a few months until I was ready to build something more complex, but it really takes a few days only to be ready for simple development. It’s not only about Drupal, but also PHP, it’s much cheaper to maintain and extend a project. Maybe it’s not so important with smaller projects, but definitely important for massive code bases. Programmers can jump in and start being productive really quick.

Great documentation

Drupal documentation is well structured and constantly developed, usually you can find what you need within a few minutes. It’s critical and must have for any other framework and not so common unfortunately.

Big community

The Drupal community is one of the biggest IT communities I have ever encountered. They extend, fix and document the Drupal core regularly. Most of them have their other jobs and work on this project just for fun and with passion.

It’s free

It’s an open source project, that’s one of the biggest pros here. You can get it for free, you can get support for free, you can join the community for free too (:)).

Modules

On the official Drupal website you can find tons of free plugins/modules. It’s a time and money saver, you don’t need to reinvent the wheel for every new widget on your website and focus on fireworks.

Usually you can just go there and find a proper component. E-commerce shop? Slideshow? Online classifieds website? No problem! It’s all there.

PHP7 support

I can often hear from other developers that PHP is slow, well, it’s not the Road Runner, but come on, unless you are Facebook (and I think that they, correct me if I’m wrong, still use PHP :)) it’s just OK to use PHP.

Drupal fully supports PHP7.

With PHP7 it’s much faster, better and safer. To learn more: https://pages.zend.com/rs/zendtechnologies/images/PHP7-Performance%20Infographic.pdf.

In the infographic you can see that PHP7 is much faster than Ruby, Perl and Python when you try to render a Mandelbrot fractal. In general, you definitely can’t say that PHP is slow, same as Drupal.

REST API support

Drupal has the built in, ready to use API system. In a few moments you can spawn a new API endpoint for you application. You don’t need to implement a whole API by yourself, I did it a few times in multiple languages, believe me, it’s problematic.

Perfect for a backend system

Drupal is a perfect candidate for a backend system. Let’s imagine that you want to build a beautiful, mobile application. You want to let editors, other people to edit content. You want to grab this content through the API. It’s easy as pie with Drupal.

Drupal’s web interface is stable and easy to use.

Power of taxonomies

Taxonomies are, really basically, just dictionaries. The best thing about taxonomies is that you don’t need to touch code to play with them.

Let’s say that on your website you want to create a list of states in the USA. Using most of the frameworks you need to ask your developer/technical person to do so. With taxonomies you just need a few clicks and that’s it, you can put in on your website. That’s sweet, not only for non technical person, but for us, developers as well. Again, you can focus on actually making the website attractive, rather than spending time on things that can be automated.

Summary

Of course, Drupal is not perfect, but it’s undeniably a great tool. Mobile application, single page application, corporate website—there are no limits for this content management system. And actually, it is, in my opinion, the best tool to manage your content and it does not mean that you need to use Drupal to present it. You can create a mobile, ReactJS, AngularJS, VueJS application and combine it with Drupal easily.

I hope that you’ve had a good reading and wish to hear back from you! Thanks.

Making cross-blogs queries in multi-site WordPress performant

2016-11-02T00:00:00+00:00

Some time ago I was working on customizing a WordPress system for a client. The system was running in a multi-site mode, being a host of a large number of blogs.

Because some blogs had not been updated in a long while, we wanted to pull information about recent posts from all of the blogs. This in turn was going to be used for pruning any blogs that weren’t considered “active”.

While the above description may sound simple, the scale of the system made the task a bit more involving that it would be usually.

How WordPress handles the “multi-site” scenario

The goal of computing the summary of posts for many blogs residing in the hypotethical blogging platform, in the same database doesn’t seem so complicated. Tasks like that are being performed all the time using relational databases.

The problem in WordPress arises though because of the very unusual way that it organises blogs data. Let’s see how the database tables look like in the “normal” mode first:

It has a number of tables that start with user configurable prefix. In the case of the screenshot above, the prefix was wp_.

We can see there’s a wp_posts table which contains rows related to blog posts. Thinking about the multi-blog setup, one would expect some kind of a blog_id column in the wp_posts column. Selecting data for the given blog would still be a cinch. It would also be very performant after adding an index on that column.

Instead, this is what we’re getting when setting up WordPress in a multi-site mode:

WordPress just creates a new set of tables we seen before appending the index of the blog to the tables prefix! Instead of having a nice, clean and easy to use wp_posts with the blog_id column, we get a number of tables—one for each blog: wp_1_posts, wp_2_posts etc. Why does it matter that much? Just try to get the counts of posts in each blog in one query—it’s impossible with such a tables setup. Getting such info involves querying each table separately. This means that with each new blog within the system, the cost of running such sequence of queries adds up dramatically. This is also known as a N+1 problem—bad WordPress! bad!

The approach around it

The counts of posts for each blog was needed to be computed very quickly in my case. The system consisted of hundreds of different mini-blogs and the stats were to be shown in the admin dashboard. Obviously making admins wait for the page to load for a long time wasn’t an option.

I went the route of creating an additional database table, holding the info about the number of posts for each blog. This table was then being updated upon each post creation, update and deletion. It was also updated upon the blog removal.

WordPress has a helper function for ensuring the table is created in the database. You feed it the DDL containing the definition of the table and it makes sure it is present in the database.

I created a function that was being fired on each plugin class instantiation, while making that class a singleton. Here’s the code that makes the plugin class a singleton:

class BlogsPruner
{
  private static $instance;

  private function __construct()
  {
    $this->ensureDatabaseSetup();
    $this->ensureBlogStatsGetUpdated();
    // ... other initialization here
  }

  public static function singleton()
  {
    if(!isset(self::$instance))
    {
      $_class = __CLASS__;
      self::$instance = new $_class;
    }
    return self::$instance;
  }

  public function __clone()
  {
    trigger_error('Cannot clone the pruner plugin', E_USER_ERROR);
  }

  // ... rest of the class
}

BlogsPruner::singleton();

The next step was to implement the function ensuring there’s a stats table in the database:

function ensureDatabaseSetup()
  {
    global $wpdb;
    $tableName = $this->blogStatsTableName();
    $charsetCollate = $wpdb->get_charset_collate();

    $sql = "CREATE TABLE $tableName (
            blog_id bigint(20),
            count_posts int(2),
            UNIQUE KEY blog_id (blog_id)
    ) $charsetCollate;";

    require_once( ABSPATH . 'wp-admin/includes/upgrade.php' );
    dbDelta( $sql );
    $this->ensureBlogStatsUpdated();
  }

This code uses a helper function to correctly construct the name for the table:

function blogStatsTableName()
{
  global $wpdb;
  return $wpdb->base_prefix . 'blog_stats';
}

This made sure the table was using the correct prefix, just like all the other tables in the database.

Now, I needed to ensure the stats were updated upon each post change:

function ensureBlogStatsGetUpdated()
{
  add_action('save_post', array($this, 'onPostUpdated'));
  add_action('delete_post', array($this, 'onPostDeleted'));
}

function onPostUpdated($postId)
{
  global $blog_id;
  $post = get_post($postId);
  if(wp_is_post_revision($postId) || $post->post_status == 'auto-draft')
  {
    return;
  }
  $this->updateBlogStats($blog_id);
}

function onPostDeleted()
{
  global $blog_id;
  $this->updateBlogStats($blog_id);
}

function updateBlogStats($blogId)
{
  $count = $this->getBlogUserCreatedPostsCount($blogId);
  $this->updateBlogPostsCount($blogId, $count);
}

// Here we're specifically not including the post that is auto-created
// upon the blog creation:
function getBlogUserCreatedPostsCount($blogId)
{
  global $wpdb;
  $sql = "SELECT
            COUNT(DISTINCT `wp_" . $blogId . "_posts`.`id`) AS count_user_posts
          FROM `wp_blogs`
          INNER JOIN `wp_" . $blogId . "_posts`
                  ON `wp_blogs`.`blog_id` = $blogId
          WHERE
            `wp_" . $blogId . "_posts`.`post_type` = 'post' AND
            `wp_" . $blogId . "_posts`.`post_status` = 'publish' AND
            TIMESTAMPDIFF(SECOND, `wp_" . $blogId . "_posts`.`post_date`, `wp_blogs`.`last_updated`) > 60";
  $row = $wpdb->get_row($sql);
  return intval($row->count_user_posts);
}

function updateBlogPostsCount($blogId, $count)
{
  global $wpdb;
  $data = array('count_posts' => $count);
  $where = array('blog_id' => $blogId);
  $wpdb->update($this->blogStatsTableName(), $data, $where);
}

The actual production plugin implemented many more features than this sample code demonstrates. It was listing the blogs that could be considered stale, automatically pruning them after specified in the admin screen time and allowing admins to configure it via the WordPress interface. The full set of features is beyond the scope of this post.

The overall result made getting the statistics about very large set of blogs very fast. The cost of querying for the number of posts of each blog was moved to incremental, small updates upon each post being created, modified or removed. For the end user, this cost was imperceptable.

Final thoughts

WordPress is loved by many users. If you’re not just a user but also working with the code, there’s a number of traps you may fall into though. If I were to employ techniques that get advised as “WordPress usual/default/preferred”—I’d end up with a very unhappy client who’s be owning a very broken WordPress system. Fortunately, the set of WordPress tables isn’t casted in stone and you can freely extend it—as long as you’re cautious and know what you’re doing. Provided that these two prerequisites are met—WordPress is just a database backed platform—like any other.

Create categories in Virtuemart 3 with Joomla 2.5 and 3.5 programmatically

2016-08-18T00:00:00+00:00

Introduction

Code that I’m going to show you is ready to download here: https://github.com/peter-hank/com_morevirtuemart

Virtuemart is an open-source e-commerce application written in PHP. It’s pretty popular with a 4% share of the whole e-commerce market (https://blog.aheadworks.com/2015/05/ecommerce-platforms-popularity-may-2015-two-platforms-take-half/). Today I will show you how to extend it and use its functionality from an external Joomla (free and open-source content management system) component.

Creating a component

We are going to create a new component to show the code in a nice and clear form. I’m using a component generator from here: https://www.component-creator.com. We don’t need any tables, models and views for the purpose of this blog post. Plain and simple component is what we need. After creating the component download it and install in a Joomla administration interface.

Component overview

The component structure looks like this:

# tree components/com_morevirtuemart
components/com_morevirtuemart
├── controller.php
├── controllers
│   └── index.html
├── helpers
│   ├── index.html
│   └── morevirtuemart.php
├── index.html
├── models
│   ├── fields
│   │   ├── createdby.php
│   │   ├── filemultiple.php
│   │   ├── foreignkey.php
│   │   ├── index.html
│   │   ├── modifiedby.php
│   │   ├── submit.php
│   │   ├── timecreated.php
│   │   └── timeupdated.php
│   ├── forms
│   │   └── index.html
│   └── index.html
├── morevirtuemart.php
├── router.php
└── views
    └── index.html

We don’t need to use more than a generic controller in a main directory. Its content is not very exciting for now:

<?php

// No direct access
defined('_JEXEC') or die;

jimport('joomla.application.component.controller');

/**
 * Class MorevirtuemartController
 *
 * @since  1.6
 */
class MorevirtuemartController extends JControllerLegacy
{
 /**
  * Method to display a view.
  *
  * @param   boolean $cachable  If true, the view output will be cached
  * @param   mixed   $urlparams An array of safe url parameters and their variable types, for valid values see {@link JFilterInput::clean()}.
  *
  * @return  JController   This object to support chaining.
  *
  * @since    1.5
  */
 public function display($cachable = false, $urlparams = false)
 {
  $view = JFactory::getApplication()->input->getCmd('view', '');
  JFactory::getApplication()->input->set('view', $view);

  parent::display($cachable, $urlparams);

  return $this;
 }
}

We can remove a display function and replace it with a function called createCategory so a MorevirtuemartController class will look like this:

class MorevirtuemartController extends JControllerLegacy
{
 public function createCategory ()
 {
  echo 'I\'m alive!';

  die();
 }
}

To test that our component is fully working right now try to open Joomla with this url: index.php?option=com_morevirtuemart&task=createCategory. Of course you need to prepend it with your domain name to make it work. The result should be just an empty page with our text: “I’m alive!”

Now you know how to call a controller task within a browser.

Creating a category

To use Virtuemart classes we need to initialize all of its logic and configuration. Look here:

<?php

// No direct access
defined('_JEXEC') or die;

jimport('joomla.application.component.controller');

// loading Virtuemart classes and configuration
if (!class_exists( 'VmConfig' )) require(JPATH_ADMINISTRATOR . DS . 'components' . DS . 'com_virtuemart'.DS.'helpers'.DS.'config.php');
$config= VmConfig::loadConfig();
if (!class_exists( 'VirtueMartModelVendor' )) require(JPATH_VM_ADMINISTRATOR.DS.'models'.DS.'vendor.php');
if(!class_exists('TableMedias')) require(JPATH_VM_ADMINISTRATOR.DS.'tables'.DS.'medias.php');
if(!class_exists('TableCategories')) require(JPATH_VM_ADMINISTRATOR.DS.'tables'.DS.'categories.php');
if (!class_exists( 'VirtueMartModelCategory' )) require(JPATH_VM_ADMINISTRATOR.DS.'models'.DS.'category.php');

/**
 * Class MorevirtuemartController
 *
 * @since  1.6
 */
class MorevirtuemartController extends JControllerLegacy
{
 public function createCategory ()
 {
  echo 'I\'m alive!';
  die();
 }
}

What we are doing here is importing needed Virtuemart classes and initializing the configuration of each. We are all set now and ready for an implementation.

The most difficult thing here is how to work with a Virtuemart internal authorization system. Before an every CRUD action there is a check for an authorization access. We need to find a workaround for this, there are multiple way of doing this, I will show you how I did that. Create a new file in components/com_morevirtuemart/model and call it category.php. Inside this file put this piece of code:

<?php

// No direct access
defined('_JEXEC') or die;

class VirtueMartModelCategoryLocal extends VirtueMartModelCategory {
 public function __construct() {
  parent::__construct();
 }
}

I hope it’s pretty clear what we are doing here: extending a Virtuemart category model. What I’m going to do now is to copy a store function from a VirtueMartModelCategory class and put it in our local class of a category model and remove an authentication check from it (you need to secure it by yourself but using a different method). The result of this action is:

<?php

// No direct access
defined('_JEXEC') or die;

class VirtueMartModelCategoryLocal extends VirtueMartModelCategory {
  public function __construct()
  {
    parent::__construct();
  }

  public function store(&$data)
  {
    $table = $this->getTable('categories');

    if ( !array_key_exists ('category_template' , $data ) ){
      $data['category_template'] = $data['category_layout'] = $data['category_product_layout'] = 0 ;
    }
    if(VmConfig::get('categorytemplate') == $data['category_template'] ){
      $data['category_template'] = 0;
    }

    if(VmConfig::get('categorylayout') == $data['category_layout']){
      $data['category_layout'] = 0;
    }

    if(VmConfig::get('productlayout') == $data['category_product_layout']){
      $data['category_product_layout'] = 0;
    }

    $table->bindChecknStore($data);

    if(!empty($data['virtuemart_category_id'])){
      $xdata['category_child_id'] = (int)$data['virtuemart_category_id'];
      $xdata['category_parent_id'] = empty($data['category_parent_id'])? 0:(int)$data['category_parent_id'];
      $xdata['ordering'] = empty($data['ordering'])? 0: (int)$data['ordering'];

        $table = $this->getTable('category_categories');

      $table->bindChecknStore($xdata);

    }

    $this->clearCategoryRelatedCaches();

    return $data['virtuemart_category_id'] ;
  }
}

In addition, I removed call to media creation. If you need to create media along with the categories you need to extend it as well, the same as we did with a category model. We are all set now to create a new category from a component controller, we just need to include an extended model there and call a store function. Our controller should look like this now:

<!--?php

// No direct access
defined('_JEXEC') or die;

jimport('joomla.application.component.controller');

// loading Virtuemart classes and configuration
defined('VMPATH_ROOT') or define('VMPATH_ROOT', JPATH_ROOT);
defined('VMPATH_ADMIN') or define('VMPATH_ADMIN', VMPATH_ROOT.DS.'administrator' . DS . 'components' . DS . 'com_virtuemart');
if (!class_exists( 'VmConfig' )) require(JPATH_ROOT.DS.'administrator' . DS . 'components' . DS . 'com_virtuemart' . DS . 'helpers' . DS . 'config.php');
VmConfig::loadConfig();
if (!class_exists( 'VmController' )) require(VMPATH_ADMIN.DS.'helpers' . DS . 'vmcontroller.php');
if (!class_exists( 'VmModel' )) require(VMPATH_ADMIN.DS.'helpers' . DS . 'vmmodel.php');
require(JPATH_ROOT . DS . 'administrator' . DS . 'components' . DS . 'com_virtuemart' . DS . 'models' . DS . 'category.php');
require(JPATH_ROOT . DS . 'administrator' . DS . 'components' . DS . 'com_virtuemart' . DS . 'models' . DS . 'product.php');
require(JPATH_ROOT . DS . 'administrator' . DS . 'components' . DS . 'com_virtuemart' . DS . 'models' . DS . 'media.php');

// loading an exteneded category model
require(__DIR__ . DS . 'models' . DS . 'category.php');

/**
 * Class MorevirtuemartController
 *
 * @since  1.6
 */
class MorevirtuemartController extends JControllerLegacy
{
 public function __construct()
 {
  parent::__construct();

  $this--->categoryModel = new VirtueMartModelCategoryLocal();
 }

 public function createCategory ()
 {
  $catData = [
   'category_name' => 'Brand new Virtuemart category',
  'category_parent_id' => 0,
   'published' => 0
   ];

  $catId = $this->categoryModel->store($catData);

  echo 'Created a new category';

  die();
 }
}

Wow! The new category should be there. There are many more available attributes to be set, we’ve used a needed minimum: category name, its parent and status.

The end

There are a few things I should mention:

ensure to add an authentication to your code, you can use a Joomla authentication system,
be aware that Virtuemart code can change in time so be careful with updates,
I recommend to use a CLI script for this kind of operations.

Thanks for reading.

Scrape web content with PHP (no API? no problem)

2016-07-07T00:00:00+00:00

Introduction

There is a lot of data flowing everywhere. Not structured, not useful pieces of data moving here and there. Getting this data and structuring, processing can make it really expensive. There are companies making billions of dollars just (huh?) for scraping web content and showing in a nice form.

Another reason for doing such things can be for example, lack of an API from a source website. In this case, it’s the only way to get data that you need to process.

Today I will show you how to get web data using PHP and that it can be as easy as pie.

Just do it

There are multiple scraping scripts ready to use. I can recommend one of them: PHP Simple HTML DOM Parser. It’s extremely easy to start with and initial cost is almost nothing, it’s open sourced also.

First, download a library from an official site: https://sourceforge.net/project/showfiles.php?group_id=218559. You can use a composer version too, it’s here: https://github.com/sunra/php-simple-html-dom-parser.

Let’s say that you have downloaded this file already. It’s just a one PHP file called simple_html_dom.php. Create a new PHP file called scraper.php and include mentioned library like this:

<?php

require('simple_html_dom.php');

In our example, we will scrape top 10 trending YouTube videos and create a nice array of links and names out of it. We will use this link: https://www.youtube.com/feed/trending?gl=GB.

We need to grab this page first. Using PHP it’s just a one additional line in our script:

<?php

require('simple_html_dom.php');

// Create DOM from URL or file
$html = file_get_html('https://www.youtube.com/feed/trending?gl=GB');

A PHP object was just created with the YouTube page structure.

Look at the YouTube page structure to find a repeating structure for a list of videos. It’s best to use Chrome developer tools and its HTML browser. At the time of writing this post (it can change in the future of course) it’s:

<ul class="expanded-shelf-content-list has-multiple-items">
 <li class="expanded-shelf-content-item-wrapper">...</li>
 <li class="expanded-shelf-content-item-wrapper">...</li>
 <li class="expanded-shelf-content-item-wrapper">...</li>
 ...
</ul>

Thanks Google! This time it will be easy. Sometimes a structure of the page lacks of classes and ids and it’s more difficult to select exactly what we need.

Now, for each item of expanded-shelf-content-item-wrapper we need to find its title and url. Using developer tools again, it’s easy to achieve:

<a
 class="yt-uix-sessionlink yt-uix-tile-link yt-ui-ellipsis yt-ui-ellipsis-2 spf-link "
 dir="ltr"
 aria-describedby="description-id-284683"
 title="KeemStar Swatted My Friend."
 href="/watch?v=oChvoP8zEBw">
 KeemStar Swatted My Friend
</a>

Jackpot! We have both things that we need in the same HTML tag. Now, let’s grab this data:

<?php

require('simple_html_dom.php');

// Create DOM from URL or file
$html = file_get_html('https://www.youtube.com/feed/trending');

// creating an array of elements
$videos = [];

// Find top ten videos
$i = 1;
foreach ($html->find('li.expanded-shelf-content-item-wrapper') as $video) {
        if ($i > 10) {
                break;
        }

        // Find item link element
        $videoDetails = $video->find('a.yt-uix-tile-link', 0);

        // get title attribute
        $videoTitle = $videoDetails->title;

        // get href attribute
        $videoUrl = 'https://youtube.com' . $videoDetails->href;

        // push to a list of videos
        $videos[] = [
                'title' => $videoTitle,
                'url' => $videoUrl
        ];

        $i++;
}

var_dump($videos);

Look, it’s simple as using CSS. What we just did? First, we extracted all videos and started looping through them here:

foreach ($html->find('li.expanded-shelf-content-item-wrapper') as $video) {

Then, just extracted a title and url per each video item here:

// Find item link element
$videoDetails = $video->find('a.yt-uix-tile-link', 0);

// get title attribute
$videoTitle = $videoDetails->title;

At the end, we just push an array object with scraped data to the array and dump it. The result looks like this:

array(10) {
  [0]=>
  array(2) {
    ["title"]=>
    string(90) "Enzo Amore & Big Cass help John Cena even the odds against The Club: Raw, July 4, 2016"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=940-maoRY3c"
  }
  [1]=>
  array(2) {
    ["title"]=>
    string(77) "Loose Women Reveal Sex Toys Confessions In Hilarious Discussion | Loose Women"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=Xxzy_bZwNcI"
  }
  [2]=>
  array(2) {
    ["title"]=>
    string(51) "Tinie Tempah - Mamacita ft. Wizkid (Official Video)"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=J4GQxzUdZNo"
  }
  [3]=>
  array(2) {
    ["title"]=>
    string(54) "Michael Gove's Shows you What's Under his Kilt"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=GIpVBLDky30"
  }
  [4]=>
  array(2) {
    ["title"]=>
    string(25) "Deception, Lies, and CSGO"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=_8fU2QG-lV0"
  }
  [5]=>
  array(2) {
    ["title"]=>
    string(68) "Last Week Tonight with John Oliver: Independence Day (Web Exclusive)"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=IQwMCQFgQgo"
  }
  [6]=>
  array(2) {
    ["title"]=>
    string(21) "Last Week I Ate A Pug"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=TTk5uQL2oO8"
  }
  [7]=>
  array(2) {
    ["title"]=>
    string(59) "PEP GUARDIOLA VS NOEL GALLAGHER | Exclusive First Interview"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=ZWE8qkmhGmc"
  }
  [8]=>
  array(2) {
    ["title"]=>
    string(78) "Skins, lies and videotape - Enough of these dishonest hacks. [strong language]"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=8z_VY8KZpMU"
  }
  [9]=>
  array(2) {
    ["title"]=>
    string(62) "We Are America ft. John Cena | Love Has No Labels | Ad Council"
    ["url"]=>
    string(39) "https://youtube.com/watch?v=0MdK8hBkR3s"
  }
}

Isn’t it easy?

The end

I have some advice if you want to make this kind of script be processing the same page all the time:

set the user agent header to simulate a real web browser request
make calls with a random delay to avoid blacklisting from a web server
use PHP 7
try to optimize the script as much as possible

You can use this script for production code but, to be honest, it’s not the most optimal approach. If you are not satisfied, code it by yourself :-).

Nice documentation is located here: http://simplehtmldom.sourceforge.net/

Medium-inspired Parallax Blur Effect For WordPress

2016-01-22T00:00:00+00:00

Are you are running a WordPress blog, but secretly dying to have that Medium parallax blur effect? I recently implemented this, and would like to share it with you. By the way, while I was working on the article, the effect was removed from Medium, which only makes having one on the website more precious.

Let’s assume that we have our custom theme class MyTheme. In functions.php:

class MyThemeBaseFunctions {
  public function __construct() {
    add_image_size('blurred', 1600, 1280, true);
    add_filter('wp_generate_attachment_metadata', array($this,'wp_blur_attachment_filter'));
  }
}

We added a custom image size blurred, and a callback wp_blur_attachment_filter to wp_generate_attachment_metadata. Here’s where the magic happens.

Before that let’s talk a little about ImageMagick, a powerful library for image processing that we will use to create the blurred effect. After some experimenting I figured that the image needed to be darkened, and then a regular blur should be applied with sigma=20. You can read more about these settings at ImageMagick Blur Usage. I used Gaussian Blur at first, but found the processing was extremely slow, and there wasn’t much difference in the end result compared to other blur methods.

Now we are ready to write a blurring function:

public function wp_blur_attachment_filter($image_data) {
    if ( !isset($image_data['sizes']['blurred']) )
       return $image_data;
       $upload_dir = wp_upload_dir();
       $src = $upload_dir['path'] . '/' . $image_data['sizes']['large']['file'];
       $destination = $upload_dir['path'] . '/' . $image_data['sizes']['blurred']['file'];
       $imagick = new \Imagick($src);
       $imagick->blurImage(0, 20, Imagick::CHANNEL_ALL);
       $imagick->modulateImage(75, 105, 100);
       $imagick->writeImage($destination);
       return $image_data;
 }

I darken the image:

$imagick->modulateImage(75, 105, 100);

And I blur the image:

$imagick->blurImage(0, 20, Imagick::CHANNEL_ALL);

Now we are able to use the custom image size in the template. Place the helper function in functions.php:

public static function non_blurred($src) {
  $url = get_site_url() . substr($src, strrpos($src, '/wp-content'));
  $post_ID = attachment_url_to_postid($url);
  list($url, $width, $height) = wp_get_attachment_image_src($post_ID, 'large');
  return $url;
}

public static function blurred($src) {
  $url = get_site_url() . substr($src, strrpos($src, '/wp-content'));
  $post_ID = attachment_url_to_postid($src);
  list($url, $width, $height) = wp_get_attachment_image_src($post_ID, 'blurred');
  return $url;
}

And now use it in the template like this:

<div class="blurImg">
  <div style="background-image: url('<?php echo MyTheme::non_blurred(get_theme_mod( 'header' )); ?>')"></div>
  <div style="background-image: url('<?php echo MyTheme::blurred(get_theme_mod('header')); ?>'); opacity: 0;" class="blur"></div>
</div>
<header></header>

Add CSS:

.blurImg {
  height: 440px;
  left: 0;
  position: relative;
  top: 0;
  width: 100%;
  z-index: -1;
}

.blurImg > div {
  background-position: center center;
  background-repeat: no-repeat;
  background-size: cover;
  height: 440px;
  position: fixed;
  width: 100%;
}

header {
  padding: 0 20px;
  position: absolute;
  top: 0;
  width: 100%;
  z-index: 1;
}

Add JavaScript magic sauce to gradually replace the non-blurred image with the blurred version as the user scrolls:

(function() {
    jQuery(window).scroll(function() {
      var H = 240;
      var oVal = jQuery(window).scrollTop() / H;
      jQuery(".blurImg .blur").css("opacity", oVal);
    });
  }).call(this);

Generating the blurred version can be very strenuous on the server. I would oftentimes receive the error PHP Fatal error: Maximum execution time of 30 seconds exceeded. There are ways to work around that. One way is to use a quicker method of blurring the image by shrinking it, blurring, and resizing it back. Another way is to use a background job, something like this:

add_action( 'add_attachment', array($this, 'wp_blur_attachment_filter') );
add_action( 'wp_blur_attachment_filter_hook', 'wp_blur_attachment_filter_callback');
function wp_blur_attachment_filter_callback($path) {
  $path_parts = pathinfo($path);
  $imagick = new \Imagick($path);
  $imagick->blurImage(0, 20, Imagick::CHANNEL_ALL);
  $imagick->modulateImage(75, 105, 100);
  $destination = dirname($path) . "/" . $path_parts['filename'] . "_darken_blur." . $path_parts['extension'];
  $imagick->writeImage($destination);
}

public function wp_blur_attachment_filter($post_ID) {
  $path = get_attached_file( $post_ID );
  wp_schedule_single_event(time(), 'wp_blur_attachment_filter_hook', array($path));
}

Or better yet, use cloud image processing—I wrote about that here.

I hope you found this writeup useful!

Install WordPress on Heroku in OS X Yosemite

2016-01-12T00:00:00+00:00

I wanted to install WordPress locally for my blog (about programming!), but using MAMP, XAMP or even Vagrant for this seemed overkill. I wanted a light setup. PHP and Apache are already integrated into Mac OS X, so why not use them? I wanted to deploy the app to Heroku, so that was another thing, since Heroku only provides PostgreSQL, not MySQL, out of the box. I’d like to share my research on how I did it.

WordPress with Heroku support

I found this handy WordPress template with built-in Heroku support. It has everything one needs to run WordPress on Heroku: PostgreSQL for WordPress (because MySQL on Heroku is a paid service), Amazon S3 and Cloudfront for your uploads since Heroku has an ephemeral file system, WP Sendgrid to send emails and WordPress HTTPS. Check out a copy with this command:

git clone https://github.com/mhoofman/wordpress-heroku.git

Let’s run the project locally first because a file cannot be written to Heroku’s file system, and updating and installing plugins or themes should be done locally anyways and then pushed to Heroku. I’m using PhpStorm for my PHP development.

Configuring Apache

mkdir -p ~/Sites
echo "<html><body><h1>my site works</h1></body></html>" > ~/sites/index.html.en

Enable PHP support:

sudo vi /etc/apache2/httpd.conf

Uncomment the following lines to look like this:

LoadModule php5_module libexec/apache2/libphp5.so
LoadModule userdir_module libexec/apache2/mod_userdir.so
Include /private/etc/apache2/extra/httpd-userdir.conf

Save and exit. Open the following file:

sudo vi /etc/apache2/extra/httpd-userdir.conf

Uncomment the following line to look like this:

Include /private/etc/apache2/users/*.conf

Save and exit. Open or create:

sudo vi /etc/apache2/users/~YOURUSERNAME.conf

Type the following in there:

<Directory "/Users/~YOURUSERNAME/Sites/">
    AddLanguage en .en
    LanguagePriority en fr de
    ForceLanguagePriority Fallback
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from localhost
    Require all granted
</Directory>

Restart Apache with:

sudo apachectl restart

Go to http://localhost/~YOURUSER/wordpress-heroku/ and enjoy the results of your work! OK, not so fast! There are more steps to make it happen ;)

Enabling PostgreSQL for PHP

Your PHP installation appears to be missing the PostgreSQL extension which is required by WordPress with PG4WP.

Here is a handy script to fix this problem Install PHP PGSQL extensions on Mac OS X Yosemite (change PHP_VER with your PHP version).

Creating the database

Hit http://localhost/~YOURUSER/blog-heroku/wp-admin/install.php

Error establishing a database connection

The template we are using is tailored for the deployment to Heroku, which means wp-config.php takes its values from the DATABASE_URL environment variable that Heroku config creates in local environment pointing to the database source on Heroku servers.

pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
> createdb wordpress
> psql wordpress
CREATE USER wordpress WITH PASSWORD 'wordpress';
GRANT ALL PRIVILEGES ON DATABASE wordpress to wordpress;

In wp-config.php, edit as follows. Make sure it matches the database and user that you just created.

$db = parse_url($_ENV["DATABASE_URL"] ? $_ENV["DATABASE_URL"] : "postgres://wordpress:wordpress@localhost:5432/wordpress");

Now 5 hours later, you are completely ready for the famous 5-min install ;D. Go to http://localhost/~YOURUSER/blog-heroku/wp-admin/install.php

Uploading the custom theme/plugin

What to do next? Of course, upload a custom theme or plugin.

Unable to create directory wp-content/uploads/2015/08. Is its parent directory writable by the server?
$ cd ~/Sites/THESITE
$ sudo chown -R _www wordpress
$ sudo chmod -R g+w wordpress

If you encounter an error asking you for FTP credentials in order to do this:

To perform the requested action, WordPress needs to access your web server.
Please enter your FTP credentials to proceed.

If you do not remember your credentials, you should contact your web host.

The problem is that Apache HTTP Server in Mac OS X runs under the user account _www which belongs to the group _www. To allow WordPress to perform operations with Apache, one way to do this is to change the owner of the wordpress directory and its contents to _www. Keep the group as staff, a group to which your user account belongs and give write permissions to the group.

$ cd ~/Sites/THESITE
$ sudo chown -R _www wordpress
$ sudo chmod -R g+w wordpress

This way, no file nor directory is world-writable.

Remember to commit your plugins/themes because due to the nature of Heroku all of the files will be overwritten there if uncommitted or not in the database, effectively wiping out all of your changes at each server restart if you do them on the server.

I installed this pretty theme for myself called Literatum—just bragging.

Deployment to Heroku

One of the most exciting last steps. This will make your blog visible to the world! Commit the changes:

rm -rf .git
git init
git add .
git commit -m "Initial commit"

Create Heroku app:

$ cd wordpress-heroku
$ heroku create
$ heroku addons:create heroku-postgresql
$ heroku pg:promote HEROKU_POSTGRESQL_INSTANCE
$ heroku addons:create sendgrid:starter

Your first deployment!

git push heroku master

Go to http://YOURAPP.herokuapp.com/wp-admin/install.php and run the famous 5-minute setup again, activate all the plugins and the chosen custom theme aaand… You are done!

Hope you will find this write-up useful and it will help you create your blog on the web!

Image Processing In The Cloud With Blitline and Wordpress

2015-12-01T00:00:00+00:00

Working with ImageMagick can be difficult. First, you have to get it installed on your OS (do you have Dev libs in place?), then you have to enable it in the language of your choice, then get it working in your application. After all that, do it all over again on the staging server where debugging may be complicated, and you may not have Admin rights. Meet Image Processing in the Cloud. Meet Blitline.

I’m doing a lot of things with Wordpress now, so we’ll set it up with Wordpress and PHP.

Step 1

Get a free developer account with Blitline, and note your application id.

Step 2

Get the Blitline PHP wrapper library Blitline_php. It’s clean and awesome, but unfortunately at the time of writing it was missing a few things, like being able to run your own Image Magick script and set a postback URL for when the job is finished. Yes, those are all useful features of Blitline cloud image processing! I’m still waiting on my pull request to be incorporated into the official version, so you can use mine that has these two useful features for now in my Blitline_php

Step 3

Now it’s time to integrate it in our application. Since it’s Wordpress, I’m doing it in the ‘wp_generate_attachment_metadata’ callback in functions.php

require_once dirname(__FILE__) . '/blitline_php/lib/blitline_php.php';
...
add_filter( 'wp_generate_attachment_metadata', array($this,'wp_blur_attachment_filter'), 10, 2 );
...
public function wp_blur_attachment_filter($image_data, $attachment_id) {
  $url = wp_get_attachment_url($attachment_id);
  list($src, $width, $length) = wp_get_attachment_image_src($attachment_id);
  $data = pathinfo($src);
  $dest = $data['filename'] . '_darken_75_105_100_blur_0_20.' . $data['extension'];
  $Blit = new Blitline_php();
  $Blit->load($url, $dest);
  $Blit->do_script("convert input.png -blur 0x20 -modulate 75,105,100 output.png");
  $Blit->set_postback_url( get_site_url() . '/wp-json/myapp/v1/blitline_callback');
  $results = $Blit->process();
  if ($results->success()) {
    foreach($results->get_images() as $name => $url) {
      error_log("Processed: {$name} at {$url}\n");
    }
  } else {
    error_log($results->get_errors());
  }
}

We are sending a JSON POST request to Blitline to make the blurred and saturated version of the uploaded image. You can track the progress of your jobs here. The request will return a URL to the image on the Blitline server, but the image may not be there right away, because the processing is asynchronous.

I tried to set up S3 bucket integration (yes, Blitline can upload to S3 for you!), but the setup procedure is quite tedious. You have to manually enter your AWS Canonical ID (and obtain it first from Amazon) on the Blitline page. Then you have to create a special policy in your bucket for Blitline. This is a lot of hassle, and giving permissions to someone else might not be the way to go for you. For me personally it didn’t work, because my policy was being automatically overwritten all the time. I don’t even know why. So here’s where the postback URL comes in play.

Step 4

I’m using this plugin WP-API V2 that soon will become part of Wordpress to make REST endpoints. In wp-content/mu-plugins/my-app-custom-endpoints/lib/endpoints.php

add_action('rest_api_init', function () {
  register_rest_route('portfolio/v1', '/blitline_callback', array(
    'methods' => 'POST',
    'callback' => 'process_blitline_callback',
  ));
});

In wp-content/mu-plugins/loader.php

require_once dirname(__FILE__) . '/blitline_php/lib/blitline_php.php';
require_once dirname(__FILE__) . '/my-app-custom-endpoints/api.php';

In wp-content/mu-plugins/my-app-custom-endpoints/api.php

if( ! defined( 'ABSPATH' ) ) exit;

require_once dirname(__FILE__) . '/lib/endpoints.php';

Here’s the fun part. Add to wp-content/mu-plugins/my-app-custom-endpoints/lib/endpoints.php

use Aws\S3\S3Client;

function process_blitline_callback($request) {
  if( !class_exists( 'WP_Http' ) )
    include_once( ABSPATH . WPINC. '/class-http.php' );

  $s3Client = S3Client::factory(array(
    'credentials' => array(
      'key'    => 'YOUR S3 KEY',
      'secret' => 'YOUR S3 SECRET'
    )
  ));
  $photo = new WP_Http();

  $body = $request->get_body_params();

  $var = (array) json_decode(stripslashes($body['results']), true);

  if (isset($var['images'][0]['error'])) {
    error_log('Error ' . $var['images'][0]['error']);
  return;
  }

  $photo = $photo->request( $var['images'][0]['s3_url'] );

  $photo_name = $var['images'][0]['image_identifier'];

  $attachment = wp_upload_bits(
    $photo_name, null, $photo['body'],
    date("Y-m", strtotime($photo['headers']['last-modified']))
  );

  $upload_dir = wp_upload_dir();

  $s3Client->putObject(array(
    'Bucket' => "yourbucket",
    'Key'    => 'wp-content/uploads' . $upload_dir['subdir'] . '/' . $photo_name,
    'SourceFile'   => $attachment['file'],
    'ACL'    => 'public-read'
  ));
}

In the callback we download the processed image from the temporary Blitline URL. One little bonus in here is the upload to Amazon S3 bucket. I use Amazon PHP SDK to achieve that. Note the permissions. This was one last thing when I actually almost gave up trying to make Blitline postback URL work. When the image finally appeared in my bucket, it wasn’t accessible from the outside, because I didn’t set permissions.

Step 5. If it doesn’t work. Debugging.

I used Firefox add-on HttpRequester to post the mock response from Blitline to my application. If you don’t want to deploy each time you change the code, here’s another useful thing LocalTunnel, so you can expose your localhost to the internet and set the postback to your local app.

And that’s how you do image processing in the cloud!

Broken wikis due to PHP and MediaWiki “namespace” conflicts

2015-11-09T00:00:00+00:00

Photo by Elliot Gilfix

I was recently tasked with resurrecting an ancient wiki. In this case, a wiki last updated in 2005, running MediaWiki version 1.5.2, and that needed to get transformed to something more modern (in this case, version 1.25.3). The old settings and extensions were not important, but we did want to preserve any content that was made.

The items available to me were a tarball of the mediawiki directory (including the LocalSettings.php file), and a MySQL dump of the wiki database. To import the items to the new wiki (which already had been created and was gathering content), an XML dump needed to be generated. MediaWiki has two simple command-line scripts to export and import your wiki, named dumpBackup.php and importDump.php. So it was simply a matter of getting the wiki up and running enough to run dumpBackup.php.

My first thought was to simply bring the wiki up as it was—all the files were in place, after all, and specifically designed to read the old version of the schema. (Because the database scheme changes over time, newer MediaWikis cannot run against older database dumps.) So I unpacked the MediaWiki directory, and prepared to resurrect the database.

Rather than MySQL, the distro I was using defaulted to using the freer and arguably better MariaDB, which installed painlessly.

## Create a quick dummy database:
$ echo 'create database footest' | sudo mysql

## Install the 1.5.2 MediaWiki database into it:
$ cat mysql-acme-wiki.sql | sudo mysql footest

## Sanity test as the output of the above commands is very minimal:
echo 'select count(*) from revision' | sudo mysql footest
count(*)
727977

Success! The MariaDB instance was easily able to parse and load the old MySQL file. The next step was to unpack the old 1.5.2 mediawiki directory into Apache’s docroot, adjust the LocalSettings.php file to point to the newly created database, and try and access the wiki. Once all that was done, however, both the browser and the command-line scripts spat out the same error:

Parse error: syntax error, unexpected 'Namespace' (T_NAMESPACE), 
  expecting identifier (T_STRING) in 
  /var/www/html/wiki/includes/Namespace.php on line 52

What is this about? Turns out that some years ago, someone added a class to MediaWiki with the terrible name of “Namespace”. Years later, PHP finally caved to user demands and added some non-optimal support for namespaces, which means that (surprise), “namespace” is now a reserved word. In short, older versions of MediaWiki cannot run with modern (5.3.0 or greater) versions of PHP. Amusingly, a web search for this error on DuckDuckGo revealed not only many people asking about this error and/or offering solutions, but many results were actual wikis that are currently not working! Thus, their wiki was working fine one moment, and then PHP was (probably automatically) upgraded, and now the wiki is dead. But DuckDuckGo is happy to show you the wiki and its now-single page of output, the error above. :)

There are three groups to blame for this sad situation, as well as three obvious solutions to the problem. The first group to share the blame, and the most culpable, is the MediaWiki developers who chose the word “Namespace” as a class name. As PHP has always had very non-existent/poor support for packages, namespaces, and scoping, it is vital that all your PHP variables, class names, etc. are as unique as possible. To that end, the name of the class was changed at some point
to “MWNamespace”—but the damage has been done. The second group to share the blame is the PHP developers, both for not having namespace support for so long, and for making it into a reserved word full knowing that one of the poster children for “mature” PHP apps, MediaWiki, was using “namespace”. Still, we cannot blame them too much for picking what is a pretty obvious word choice. The third group to blame is the owners of all those wikis out there that are suffering that syntax error. They ought to be repairing their wikis. The fixes are pretty simple, which leads us to the three solutions to the problem.

MediaWiki’s cool install image

The quickest (and arguably worst) solution is to downgrade PHP to something older than 5.3. At that point, the wiki will probably work again. Unless it’s a museum (static) wiki, and you do not intend to upgrade anything on the server ever again, this solution will not work long term. The second solution is to upgrade your MediaWiki! The upgrade process is actually very robust and works well even for very old versions of MediaWiki (as we shall see below). The third solution is to make some quick edits to the code to replace all uses of “Namespace” with “MWNamespace”. Not a good solution, but ideal when you just need to get the wiki up and running. Thus, it’s the solution I tried for the original problem.

However, once I solved the Namespace problem by renaming to MWNamespace, some other problems popped up. I will not run through them here—although they were small and quickly solved, it began to feel like a neverending whack-a-mole game, and I decided to cut the Gordian knot with a completely different approach.

As mentioned, MediaWiki has an upgrade process, which means that you can install the software and it will, in theory, transform your database schema and data to the new version. However, version 1.5 of MediaWiki was released in October 2005, almost exactly 10 years ago from the current release (1.25.3 as of this writing). Ten years is a really, really long time on the Internet. Could MediaWiki really convert something that old? (spoilers: yes!). Only one way to find out. First, I prepared the old database for the upgrade. Note that all of this was done on a private local machine where security was not an issue.

## As before, install mariadb and import into the 'footest' database
$ echo 'create database footest' | sudo mysql test
$ cat mysql-acme-wiki.sql | sudo mysql footest
$ echo "set password for 'root'@'localhost' = password('foobar')" | sudo mysql test

Next, I grabbed the latest version of MediaWiki, verified it, put it in place, and started up the webserver:

$ wget http://releases.wikimedia.org/mediawiki/1.25/mediawiki-1.25.3.tar.gz
$ wget http://releases.wikimedia.org/mediawiki/1.25/mediawiki-1.25.3.tar.gz.sig

$ gpg --verify mediawiki-1.25.3.tar.gz.sig 
gpg: assuming signed data in `mediawiki-1.25.3.tar.gz'
gpg: Signature made Fri 16 Oct 2015 01:09:35 PM EDT using RSA key ID 23107F8A
gpg: Good signature from "Chad Horohoe <chad@wikimedia.org>"
gpg:                 aka "keybase.io/demon <demon@keybase.io>"
gpg:                 aka "Chad Horohoe (Personal e-mail) <innocentkiller@gmail.com>"
gpg:                 aka "Chad Horohoe (Alias for existing email) <chadh@wikimedia.org>"
## Chad's cool. Ignore the below.
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 41B2 ABE8 17AD D3E5 2BDA  946F 72BC 1C5D 2310 7F8A</chadh@wikimedia.org></innocentkiller@gmail.com></demon@keybase.io></chad@wikimedia.org>

$ tar xvfz mediawiki-1.25.3.tar.gz
$ mv mediawiki-1.25.3 /var/www/html/
$ cd /var/www/html/mediawiki-1.25.3
## Because "composer" is a really terrible idea:
$ git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git 
$ sudo service httpd start

Now, we can call up the web page to install MediaWiki.

Visit http://localhost/mediawiki-1.25.3, see the familiar yellow flower
Click “set up the wiki”
Click next until you find “Database name”, and set to “footest”
Set the “Database password:” to “foobar”
Aha! Looks what shows up: “Upgrade existing installation” and “There are MediaWiki tables in this database. To upgrade them to MediaWiki 1.25.3, click Continue”

It worked! Next messages are: “Upgrade complete. You can now start using your wiki. If you want to regenerate your LocalSettings.php file, click the button below. This is not recommended unless you are having problems with your wiki.” That message is a little misleading. You almost certainly do want to generate a new LocalSettings.php file when doing an upgrade like this. So say yes, leave the database choices as they are, and name your wiki something easily greppable like “ABCD”. Create an admin account, save the generated LocalSettings.php file, and move it to your mediawiki directory.

At this point, we can do what we came here for: generate a XML dump of the wiki content in the database, so we can import it somewhere else. We only wanted the actual content, and did not want to worry about the history of the pages, so the command was:

$ php maintenance/dumpBackup.php --current > acme.wiki.2005.xml

It ran without a hitch. However, close examination showed that it had an amazing amount of unwanted stuff from the “MediaWiki:” namespace. While there are probably some clever solutions that could be devised to cut them out of the XML file (either on export, import, or in between), sometimes quick beats clever, and I simply opened the file in an editor and removed all the “page” sections with a title beginning with “MediaWiki:”. Finally, the file was shipped to the production wiki running 1.25.3, and the old content was added in a snap:

$ php maintenance/importDump.php acme.wiki.2005.xml

The script will recommend rebuilding the “Recent changes” page by running rebuildrecentchanges.php (can we get consistentCaps please MW devs?). However, this data is at least 10 years old, and Recent changes only goes back 90 days by default in version 1.25.3 (and even shorter in previous versions). So, one final step:

## 20 years should be sufficient
$ echo '$wgRCMAxAge = 20 * 365 * 24 * 3600;' >> LocalSettings.php
$ php maintenance/rebuildrecentchanges.php

Voila! All of the data from this ancient wiki is now in place on a modern wiki!

How to Analyze Application Logs and Extract Actionable Insights

A Note on Commands

1. Debugging Application Errors

1.1 Summarize Frequent Issues

1.2 Categorize Errors by Type

1.3 Find Problematic Scripts

2. Monitoring System Behavior

2.1 Track Problematic IPs

2.2 Spot High-Error Time Periods

2.3 Analyze IP–Referer Pairs

3. Enhancing Security

3.1 Watch Sensitive URLs

3.2 Pinpoint Critical Times

Conclusion

Integrating Laravel With a React Frontend

Scaffolding The Frontend

Compiling Assets With Laravel Mix

Handling Frontend Routing

CSRF Tokens

Conclusion

Advanced WordPress customizations

Custom CSS and JavaScript

Custom PHP code

Custom hooks

Conclusion

Craft: A CMS for developers

Craft is easy to get up and running

Craft is easy to put in containers

Craft’s design makes sense

Craft gives you complete freedom over your front end

Craft is cool 🕶️

An introduction to automated testing for web applications with Symfony

The types of tests we’re going to write

Getting to know the system under test

Unit tests

Surveying a class to write unit tests for it

Deciding what tests to write

Tactics for writing the unit tests

Where to put them and what to name them

What a PHPUnit test class looks like

How to name the individual test methods

The anatomy of a test case

Running PHPUnit tests

Other interesting details

Testing behavior with varying input

Using mocks to test classes with dependencies

Verifying behavior instead of state: Expectations with PHPUnit

The thought process of writing tests with mocks

Integration tests

Testing database interaction

Testing calls to an external Web API

Functional tests

Closing thoughts

Job opening: PHP / JavaScript developer

What you will be doing:

What you bring:

What work here offers:

Get in touch with us:

Magento 2: Creating a custom theme

Creating a new Magento 2 theme

Creating a new directory for the theme

Creating registration.php

Creating theme.xml

Activating our new theme

Adding custom content

Creating a custom view file

Adding a custom logo

Adding static files (JavaScript/​CSS/​images/​fonts)

Adding a custom layout

Linux Development in Windows 10 with Docker and WSL 2

What’s new with WSL 2

1. It’s faster and more compatible

2. It’s better integrated with Windows

3. It can run Docker

4. A newer version means several bugfixes

The development environment

Step 1: Install WSL 2 and Ubuntu

Step 2: Install Docker

Step 3: Install some useful VS Code extensions

Step 4: Create the development container

Adding static files (JavaScript/CSS/images/fonts)