https://www.endpointdev.com/blog/tags/apache/2016-02-09T00:00:00+00:00End Point DevBonked By Basic_auth Because Bcrypthttps://www.endpointdev.com/blog/2016/02/bonked-by-basicauth-because-bcrypt/2016-02-09T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float: right; padding: 0 1em 1em 2em; text-align: center;"><a href="/blog/2016/02/bonked-by-basicauth-because-bcrypt/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2016/02/bonked-by-basicauth-because-bcrypt/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/xUswo">Alligator photo</a> by <a href="https://www.flickr.com/people/johnjack/">Random McRandomhead</a></small></div>
<p><strong>TL;DR: Don’t use a high bcrypt cost with HTTP basic auth!</strong></p>
<p>Recently we had a client approach us with reports of a slow wiki experience. This was
for a <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> we recently installed for them;
there were no
<a href="https://www.mediawiki.org/wiki/Manual:Extensions">fancy extensions</a>, and the hardware, the OS, and the
<a href="https://en.wikipedia.org/wiki/Apache_HTTP_Server">Apache web server</a> were solid, perfectly normal choices.
I was tasked to dive in and solve this issue.</p>
<p>The first step in any troubleshooting is to verify and duplicate the problem. While the
wiki did feel a bit sluggish, it was not as bad as the reports we were getting of taking
over 15 seconds to view a page. A side-by-side comparison with a similar wiki seemed a
good place to start. I called up the main wiki page on both the client wiki and End Point’s
internal wiki. Both were running the latest version of MediaWiki, had the same type of
servers (located a similar distance from me), were using the same version of Apache, and
had roughly the same
<a href="https://en.wikipedia.org/wiki/Load_%28computing%29">server load</a>. While both
wiki’s pages had roughly the same amount of content, the client one loaded noticeably slower.
It took less than a second for the End Point wiki, and around ten seconds for the client one!</p>
<p>The first possible culprit was MediaWiki itself. Perhaps something was misconfigured there, or
some extension was slowing everything down? MediaWiki has
<a href="https://www.mediawiki.org/wiki/Manual:How_to_debug">good debugging tools</a>. Inside the both wiki’s <strong>LocalSettings.php</strong> file I
turned on debugging temporarily with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$wgDebugLogFile</span> = <span style="color:#d20;background-color:#fff0f0">'/tmp/mediawiki.debug'</span>;
<span style="color:#369">$wgDebugDBTransactions</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#369">$wgDebugDumpSql</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#369">$wgDebugTimestamps</span> = <span style="color:#080;font-weight:bold">true</span>;
</code></pre></div><p>I reloaded the page, then commented out the $wgDebugLogFile line to stop it from
growing large (the debug output can be quite verbose!). Here’s some snippets from
the generated log file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">0.9151 4.2M Start request GET /wiki/Main_Page
...
[caches] main: SqlBagOStuff, message: SqlBagOStuff, parser: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
0.9266 9.2M Implicit transaction open enabled.
0.9279 9.2M Query wikidb (1) (slave): SET /* DatabasePostgres::open */ client_encoding='UTF8'
0.9282 9.2M Resource id #127: Transaction state changed from IDLE -> ACTIVE
0.9268 9.2M Query wikidb (2) (slave): SET /* DatabasePostgres::open */ datestyle = 'ISO, YMD'
...
0.9587 9.2M Query wikidb (11) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
0.9573 9.5M Query wikidb (12) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'list' LIMIT 1
0.9567 10.8M Query wikidb (13) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'preload' LIMIT 1
0.9572 10.8M Query wikidb (14) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'preload' LIMIT 1
...
0.9875 21.2M Query wikidb (195) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-mycontris' LIMIT 1
0.9873 21.2M Query wikidb (196) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:tooltip-pt-logout' LIMIT 1
0.9868 21.2M Query wikidb (197) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-logout' LIMIT 1
0.9883 21.2M Query wikidb (198) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:vector-more-actions' LIMIT 1
</code></pre></div><p>Just to load a simple page, there were 194 SELECT statements! And 137 of those were trying to look in the l10n_cache table, one
row at a time. Clearly, there is lots of room for improvement there. Someday, I may even jump in and tackle that. But for now,
despite being very inefficient, it is also very fast. Because of the $wgDebugTimestamps, it was easy to compute how much
time both wikis spent actually creating the page and sending it back to Apache. In this case, the difference was minimal,
which meant MediaWiki was not the culprit.</p>
<p>I then turned my attention to Apache. Perhaps it was compiled differently? Perhaps there was some
obscure
<a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">SSL</a> bug slowing things down for everyone? These were unlikely, but it was worth checking the Apache logs
(which were in /var/log/httpd). There are
two main logs Apache uses: access and error. The latter revealed nothing at all when I loaded the
main wiki page. The access logs looked fairly normal:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">85.236.207.120 - greg [19/Jan/2016:12:23:21 -0500] "GET /wiki/Main_Page HTTP/1.1" 200 23558 "-" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/extensions/balloons/js/balloon.config.js HTTP/1.1" 200 4128 "https://wiki.endpoint.com/wiki/Main_Page
" "Mozilla/5.0 Firefox/43.0"
...
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector HTTP/1.1" 200 58697 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/resources/assets/poweredby_mediawiki_88x31.png HTTP/1.1" 200 3525 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"
</code></pre></div><p>Still nothing out of the ordinary. What to do next? When all else fails, go to the system calls. It’s about as close to bare metal as you can easily get on a Linux system. In this case, I decided to run
<a href="https://en.wikipedia.org/wiki/Strace">strace</a> on the Apache
<a href="https://en.wikipedia.org/wiki/Daemon_%28computing%29">daemon</a> to see exactly where the time was being spent. As expected, there were a large handful of httpd processes already
spawned and waiting for a connection. While there was no way to know which one would field my requests, some shell-fu allowed me
to strace them all at once:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">## The -u prevents us from picking the parent httpd process, because it is owned by root!
$ strace -o greg.httpd.trace -tt -ff `pgrep -u apache httpd | xargs -n 1 echo -p | xargs`
Process 5148 attached
Process 4848 attached
Process 5656 attached
Process 4948 attached
Process 5149 attached
Process 5148 attached
Process 4858 attached
Process 5657 attached
Process 4852 attached
Process 4853 attached
^CProcess 5148 detached
Process 4848 detached
Process 5656 detached
Process 4948 detached
Process 5149 detached
Process 5148 detached
Process 4858 detached
Process 5657 detached
Process 4852 detached
Process 4853 detached
</code></pre></div><p>Looking at one of the output of one of these revealed some important clues:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ head greg.httpd.trace.4948
13:00:28.799807 read(14, "\27\3\3\2\221\0\0\0\0\0\0\0\1\35-\332\3123(\200\302\"\251'g\256\363b5"..., 8000) = 666
13:00:28.799995 stat("/wiki/htdocs/mediawiki/load.php", {st_mode=S_IFREG|0644, st_size=1755, ...}) = 0
13:00:28.800126 open("/wiki/htpasswd.users", O_RDONLY|O_CLOEXEC) = 15
13:00:28.800176 fstat(15, {st_mode=S_IFREG|0640, st_size=947, ...}) = 0
13:00:28.800204 read(15, "alice:{SHA}jA0EAgMCMEpo4Wa3n/9gy"..., 4096) = 2802
13:00:28.800230 close(15) = 0
13:00:29.496369 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
13:00:29.496863 rt_sigaction(SIGPROF, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670}, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670
</code></pre></div><p>Aha! If you look close at those timestamps, you will notice that the time gap from the call to close() and the subsequent setitimer() is quite large at .69 seconds. That’s a long time for Apache to be waiting around for something. The second clue is the file it just opened: “htpasswd.users”. Seeing the top of the file, with the {SHA} in quotes, made me realize the problem—htpasswd files now support bcrypt as an authentication method, and bcrypt is designed to be secure—and slow. Sure enough, the htpasswd file had bcrypt entries with a high cost for the people that were having the most issues with the speed. This is what the file looked like (names and values changed):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">alice:{SHA}jA0EAgMCMEpo4Wa3n/9gybBBsDPa
greg:$2y$13$+lE6+EwgtzP0m8K8VQDnYMRDRMf6rNMRZsCzko07QQpskKI9xbb/y9
mallory:$2y$15$ww8Q4HMI1Md51kul2Hiz4ctetPqJ95cmspH8T81JHfqRvmg===rVgn
carol:7RnEKJWc38uEO
bob:$apr1$uKX9Z63CqPOGX4lD1R4yVZsloJyZGf+
jon:$2y$08$SUe3Z8sgEpyDWbWhUUUU5wtVTwlpEdc7QyXOg3e5WBwM4Hu35/OSo1
eve:$apr1$I/hv09PcpU0VfXhyG7ZGaMz7Vhxi1Tm
</code></pre></div><p>I recognized the bcrypt format right away ($2y$13$). The people who were complaining the most (e.g. mallory in the example above) about the speed of the wiki had the highest costs, while those with low costs (e.g. jon), and those using something other than bcrypt (everyone else above), were not complaining at all!</p>
<p>The ‘cost’ is the number after the second dollar sign: as you can see, some of them had a cost of <strong>15</strong>, which is much more expensive than a cost of <strong>13</strong>, which is what my user (“greg”) was using. This was a smoking gun, but one more step was needed for proof. I adjusted the cost of my password to something low using the <a href="https://httpd.apache.org/docs/current/programs/htpasswd.html">htpasswd program</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ htpasswd -B -C 6 /wiki/htpasswd.users greg
New password:
Re-type new password:
Updating password for user greg
</code></pre></div><p>Voila! The page loaded in a flash. I then changed the cost to 15 and suddenly the wiki was even slower than before—taking
upwards of 15 seconds to load the main page of the wiki. Mystery solved. All those high cost bcrypt requests are also not good
for the server: not only does it use a lot of CPU, but ends up keeping the Apache daemon tied up waiting for the bcrypt to
finish, rather than simply finishing up quickly and going back to the main pool.</p>
<p>You may be asking a few questions at this point, however. Why would htpasswd offer a footgun like this? Why such a radical
difference in effect for slightly different costs? Is bcrypt a good practice for a htpasswd file? Let’s attempt to answer
those. Before we do, we have to learn a little bit about bcrypt and passwords in general. Some of this is purposefully
oversimplified, so be gently in the comments. :)</p>
<p>Passwords themselves are never stored on a server (aka the machine doing the authentication). Instead, the server stores
a hash of the password. This is created by what is known as a “one-way” function, that creates a unique fingerprint of your
password. If this fingerprint (aka hash) is discovered, there is no direct way to see the password that created it. When
you login to a site, it creates a hash of the password you give it, then compares that hash to the one it has stored. Thus,
it can verify that you have given it the correct password without actually having to store the password.</p>
<p>For a long time, very simple algorithms were used to create these hashes. However, as computers became more powerful,
and as the field of cryptography advanced, it became easier to “crack” these hashes and determine the password that
was used to create them. This was an important problem, and one of the solutions that people came up with was the
<a href="https://en.wikipedia.org/wiki/Bcrypt">bcrypt algorithm</a>, which makes the computation of the hash very expensive, in terms of computer speed. Furthermore, that
speed is adjustable, and determined by the “cost” given at creation time. You may have noticed the <strong>-C</strong> option I
used in the htpasswd example above. That number indicates the number of rounds the algorithm must go through. However, the cost
given leads to 2^code rounds, which means that the cost is exponential. In other words, a cost of 13 means that bcrypt runs
2 to the 13th power rounds, or 8,192 rounds. A cost of 14 is 2 to the 14th power, or 16,384 rounds—twice as slow as
a cost of 13! A cost of 15 is 32,768 rounds, etc. Thus, one can see why even a cost of 15 would be much slower than a cost of 13.</p>
<p>A web page usually returns more than just the requested HTML. There are commonly images, CSS, and JavaScript that must also be
loaded from the webserver to fully render the page. Each of these requests must go through basic auth, and thus get slowed down
by bcrypt. This is why even though each basic authentication via bcrypt of 15 only takes a couple of seconds, the entire web
page can take much longer.</p>
<p>What encryption options are available for htpasswd program? The bcrypt option was introduced without much fanfare in
<a href="https://httpd.apache.org/docs/2.4/new_features_2_4.html#programs">version 2.4.4 of Apache</a>, which was released on February 25, 2013. So, it’s been around a while. The output of –help shows
us that bcrypt is the only secure one, but allows for other legacy ones to be used. Also note that the range of costs for
bcrypt range from 4 to 31:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> -m Force MD5 encryption of the password (default).
-B Force bcrypt encryption of the password (very secure).
-C Set the computing time used for the bcrypt algorithm
(higher is more secure but slower, default: 5, valid: 4 to 31).
-d Force CRYPT encryption of the password (8 chars max, insecure).
-s Force SHA encryption of the password (insecure).
-p Do not encrypt the password (plaintext, insecure).
</code></pre></div><p>So should you use bcrypt for your htpasswd? Absolutely yes. Even a lower cost bcrypt is incredibly
more secure than using MD5, CRYPT, or SHA. A cost of 10 is roughly the same speed as those,
but a much, much better choice. You can measure the time it takes to create or update your password via
the command-line htpasswd command to get a rough idea of how much impact it will have on your
website. You can use the time it takes to run the htpasswd command as rough proxy for the total
page load time. Here are some numbers I generated on my local box. Numbers represent the average
of a few runs:</p>
<table border="1">
<tbody><tr>
<th>Bcrypt cost</th>
<th>htpasswd creation time</th>
<th>Web page load time</th>
</tr>
<tr><td>10</td><td>0.079</td><td>5.68 seconds</td></tr>
<tr><td>12</td><td>0.268</td><td>6.77 seconds</td></tr>
<tr><td>14</td><td>0.979</td><td>10.78 seconds</td></tr>
<tr><td>16</td><td>3.684</td><td>25.72 seconds</td></tr>
<tr><td>18</td><td>14.683</td><td>88.85 seconds</td></tr>
<tr><td>20</td><td>58.680</td><td>358.80 seconds</td></tr>
<tr><td>22</td><td>236.369</td><td>1357.82 seconds</td></tr>
<tr><td>31</td><td>186,173 seconds<br/>(51 hours and 42 minutes!!)</td><td>Ah...no</td></tr>
</tbody></table>
<p>There are times where you really do want a higher bcrypt cost. The basic auth usage in this
scenario is really the exception, and not the norm. In most cases, a password will be used to log in
to something, and you will either create a persistent connection (e.g. SSH), or a cookie with a
temporary token will be issued (e.g. almost every website in the world). In those cases, a few
seconds delay are quite acceptable, as it is a rare event.</p>
<p>So why do we even care about passwords so much, especially for something like basic auth and
a htpasswd file? After all, if someone can view the contents of the htpasswd file, they can
also more than likely view whatever material on the web server it was designed to protect.
These days, however, it’s important to view strong hashes such as bcrypt as not just
protecting data, but protecting the password as well. Why? Password reuse.
It’s very common for people to use the same (or very similar) password on all the sites
they visit. The danger is thus not that an attacker can view the file contents protected
by the htpasswd file, but that an attacker can use that password on the user’s email accounts, or
on other sites the user may have visited and used the same password.</p>
<p>What bcrypt cost should you use? The general answer is to use the highest possible cost
you can get away with. Take something with such a high cost that is causes discomfort
to the users, then dial it back a tiny bit. Measure it out and see what your server
can handle. For general bcrypt use, start with 13, but don’t be afraid to keep going up until it takes a
wall clock second or two to run. For basic auth, use something very fast: perhaps 9 or less. Anything
that takes over a second to create via htpasswd will slow a site down noticeably!</p>
Install WordPress on Heroku in OS X Yosemitehttps://www.endpointdev.com/blog/2016/01/install-wordpress-on-heroku-in-os-x/2016-01-12T00:00:00+00:00Marina Lohova
<p>I wanted to install WordPress locally for my blog (about programming!), but using MAMP, XAMP or even Vagrant for this seemed overkill. I wanted a light setup. PHP and Apache are already integrated into Mac OS X, so why not use them? I wanted to deploy the app to Heroku, so that was another thing, since Heroku only provides PostgreSQL, not MySQL, out of the box. I’d like to share my research on how I did it.</p>
<h3 id="wordpress-with-heroku-support">WordPress with Heroku support</h3>
<p>I found <a href="https://github.com/mhoofman/wordpress-heroku">this handy WordPress template with built-in Heroku support</a>. It has everything one needs to run WordPress on Heroku: PostgreSQL for WordPress (because MySQL on Heroku is a paid service), Amazon S3 and Cloudfront for your uploads since Heroku has an ephemeral file system, WP Sendgrid to send emails and WordPress HTTPS. Check out a copy with this command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">git clone https://github.com/mhoofman/wordpress-heroku.git
</code></pre></div><p>Let’s run the project locally first because a file cannot be written to Heroku’s file system, and updating and installing plugins or themes should be done locally anyways and then pushed to Heroku. I’m using <a href="https://www.jetbrains.com/phpstorm/">PhpStorm</a> for my PHP development.</p>
<h3 id="configuring-apache">Configuring Apache</h3>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mkdir -p ~/Sites
<span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"<html><body><h1>my site works</h1></body></html>"</span> > ~/sites/index.html.en
</code></pre></div><p>Enable PHP support:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">sudo vi /etc/apache2/httpd.conf
</code></pre></div><p>Uncomment the following lines to look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">LoadModule php5_module libexec/apache2/libphp5.so
LoadModule userdir_module libexec/apache2/mod_userdir.so
Include /private/etc/apache2/extra/httpd-userdir.conf
</code></pre></div><p>Save and exit. Open the following file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">sudo vi /etc/apache2/extra/httpd-userdir.conf
</code></pre></div><p>Uncomment the following line to look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Include /private/etc/apache2/users/*.conf
</code></pre></div><p>Save and exit. Open or create:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">sudo vi /etc/apache2/users/~YOURUSERNAME.conf
</code></pre></div><p>Type the following in there:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><Directory "/Users/~YOURUSERNAME/Sites/">
AddLanguage en .en
LanguagePriority en fr de
ForceLanguagePriority Fallback
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from localhost
Require all granted
</Directory>
</code></pre></div><p>Restart Apache with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">sudo apachectl restart
</code></pre></div><p>Go to http://localhost/~YOURUSER/wordpress-heroku/ and enjoy the results of your work! OK, not so fast! There are more steps to make it happen ;)</p>
<h3 id="enabling-postgresql-for-php">Enabling PostgreSQL for PHP</h3>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Your PHP installation appears to be missing the PostgreSQL extension which is required by WordPress with PG4WP.
</code></pre></div><p>Here is a handy script to fix this problem <a href="https://gist.github.com/marinalohova/ec5d77ffd9d8e8acce2c">Install PHP PGSQL extensions on Mac OS X Yosemite (change PHP_VER with your PHP version)</a>.</p>
<h3 id="creating-the-database">Creating the database</h3>
<p>Hit http://localhost/~YOURUSER/blog-heroku/wp-admin/install.php</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html">Error establishing a database connection
</code></pre></div><p>The template we are using is tailored for the deployment to Heroku, which means wp-config.php takes its values from the DATABASE_URL environment variable that Heroku config creates in local environment pointing to the database source on Heroku servers.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
> createdb wordpress
> psql wordpress
CREATE USER wordpress WITH PASSWORD <span style="color:#d20;background-color:#fff0f0">'wordpress'</span>;
GRANT ALL PRIVILEGES ON DATABASE wordpress to wordpress;
</code></pre></div><p>In wp-config.php, edit as follows. Make sure it matches the database and user that you just created.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$db</span> = parse_url(<span style="color:#369">$_ENV</span>[<span style="color:#d20;background-color:#fff0f0">"DATABASE_URL"</span>] ? <span style="color:#369">$_ENV</span>[<span style="color:#d20;background-color:#fff0f0">"DATABASE_URL"</span>] : <span style="color:#d20;background-color:#fff0f0">"postgres://wordpress:wordpress@localhost:5432/wordpress"</span>);
</code></pre></div><p>Now 5 hours later, you are completely ready for the famous 5-min install ;D. Go to http://localhost/~YOURUSER/blog-heroku/wp-admin/install.php</p>
<h3 id="uploading-the-custom-themeplugin">Uploading the custom theme/plugin</h3>
<p>What to do next? Of course, upload a custom theme or plugin.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Unable to create directory wp-content/uploads/2015/08. Is its parent directory writable by the server?
$ cd ~/Sites/THESITE
$ sudo chown -R _www wordpress
$ sudo chmod -R g+w wordpress
</code></pre></div><p>If you encounter an error asking you for FTP credentials in order to do this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">To perform the requested action, WordPress needs to access your web server.
Please enter your FTP credentials to proceed.
If you do not remember your credentials, you should contact your web host.
</code></pre></div><p>The problem is that Apache HTTP Server in Mac OS X runs under the user account _www which belongs to the group _www. To allow WordPress to perform operations with Apache, one way to do this is to change the owner of the wordpress directory and its contents to _www. Keep the group as staff, a group to which your user account belongs and give write permissions to the group.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#038">cd</span> ~/Sites/THESITE
$ sudo chown -R _www wordpress
$ sudo chmod -R g+w wordpress
</code></pre></div><p>This way, no file nor directory is world-writable.</p>
<p>Remember to commit your plugins/themes because due to the nature of Heroku all of the files will be overwritten there if uncommitted or not in the database, effectively wiping out all of your changes at each server restart if you do them on the server.</p>
<p>I installed this pretty theme for myself called Literatum—just bragging.</p>
<h3 id="deployment-to-heroku">Deployment to Heroku</h3>
<p>One of the most exciting last steps. This will make your blog visible to the world! Commit the changes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">rm -rf .git
git init
git add .
git commit -m <span style="color:#d20;background-color:#fff0f0">"Initial commit"</span>
</code></pre></div><p>Create Heroku app:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#038">cd</span> wordpress-heroku
$ heroku create
$ heroku addons:create heroku-postgresql
$ heroku pg:promote HEROKU_POSTGRESQL_INSTANCE
$ heroku addons:create sendgrid:starter
</code></pre></div><p>Your first deployment!</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">git push heroku master
</code></pre></div><p>Go to <a href="http://YOURAPP.herokuapp.com/wp-admin/install.php">http://YOURAPP.herokuapp.com/wp-admin/install.php</a> and run the famous 5-minute setup again, activate all the plugins and the chosen custom theme aaand… You are done!</p>
<p>Hope you will find this write-up useful and it will help you create your blog on the web!</p>
Runaway Rewrite Rulehttps://www.endpointdev.com/blog/2014/07/runaway-rewrite-rule/2014-07-16T00:00:00+00:00Jeff Boes
<p>I am <em>not</em> an expert in Apache configuration. When I have to delve into a *.conf file for more than five minutes, I come out needing an aspirin, or at least a nerve-soothing cupcake. But necessity is the mother of contention, or something like that.</p>
<p>My application recently had added some new URLs, which were being parsed by your typical <a href="https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller">MVC</a> route handler (although in Perl, because that’s how I roll, and not in <a href="http://perldancer.org/">Dancer</a>, because … well, I don’t think it had been invented yet when this application first drew breath). 99.9% of the URLs worked just fine:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">/browse/:brand/:category (the pattern)
/browse/acme/widget
/browse/ben-n-jerry/ice-cream
</code></pre></div><p>and so on. Suddenly a report reached me that one particular brand was failing:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">/browse/unseen-images/stuff
</code></pre></div><p>(“unseen-images” has been changed to protect the innocent. The key here is the word “images”; put a pin in that and hang on.)</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">/browse/unseen-images
</code></pre></div><p>worked just fine. What’s worse, instrumenting the route handler code proved that it wasn’t even being called for /browse/unseen-images/foo or any of its siblings, whether :category was valid or not.</p>
<p>Making sure my bottle of aspirin was at hand, I dove into the Apache configuration. I added –</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">RewriteLog /path-to-logs/logs/rewrite_log
RewriteLogLevel 9
</code></pre></div><p>and while its output was fascinating, it wasn’t very enlightening. However, I did stumble upon this gem:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">RewriteRule ^/.*images/.* - [NE,PT,L]
</code></pre></div><p>Aha! Oho! A runaway regular expression is our culprit. I’m pretty sure this was added innocently, hoping to catch things like</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">/css/images/foo.jpg
/images/foo.png
</code></pre></div><p>and so on, but it misfired and gathered up my application URL. I replaced this temporarily with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">RewriteRule ^/(.+/)*images/.* - [NE,PT,L]
</code></pre></div><p>“Temporarily” because I’m still trying to find someone who knows why that particular kind of rewrite was deemed necessary, so I don’t know whether my replacement rule will have the same effect in the cases where it is supposed to be doing a job.</p>
<p>Is there a moral to this story? I don’t know just yet, but it’s probably something like “Regular expressions are powerful, use them with care”, or maybe “When rewrite rules are good, they are very, very good, but when they are bad they are horrid.”</p>
SELinux, PHP and FTP issueshttps://www.endpointdev.com/blog/2014/06/selinux-php-and-ftp-issues/2014-06-20T00:00:00+00:00Emanuele “Lele” Calò
<p>Sometimes it feels like working with SELinux is much like playing Wack-A-Mole. You manage to squash a bug/issue and another one appears elsewhere.</p>
<p>A similar situation happened to one of our customers when he tried connecting via FTP from his PHP code (through Apache).</p>
<p>After much debugging and a lot more Google-ing it turned out it was just a matter of enabling the right SELinux boolean setting.</p>
<p>In order to verify that it really was SELinux fault, we usually keep an eye on the “/var/log/audit/audit.log” log file and then temporarily set SELinux to “Permissive” with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">setenforce <span style="color:#00d;font-weight:bold">0</span>
</code></pre></div><p>In our case things started working as expected so we knew that it was SELinux fault, though we had no “AVC (denial)” error in the audit.log file, neither in Enforce nor in Permissive.</p>
<p>When this kind of situations happens it’s usually a matter of finding which SELinux booleans needs to be toggled.</p>
<p>To discover which SELinux booleans is blocking the wanted behavior we need to temporarily disable the “dontaudit” setting by using:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">semodule -DB
</code></pre></div><p>and then continue looking at the audit.log file. In our case we found that the interested setting was “httpd_can_network_connect”.</p>
<p>First we verified that it really was set to off:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">getsebool httpd_can_network_connect
</code></pre></div><p>If it is actually set to “off” then go on with the next steps, otherwise you’ll probably need to investigate somewhere else.</p>
<p>Next set the SELinux boolean to “on” and put SELinux back to “Enforce” by running:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">setsebool <span style="color:#369">httpd_can_network_connect</span>=<span style="color:#00d;font-weight:bold">1</span>
setenforce <span style="color:#00d;font-weight:bold">1</span>
</code></pre></div><p>Now check again the the code is still running as expected and if so set the SELinux boolean to stick between reboots:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">setsebool -P <span style="color:#369">httpd_can_network_connect</span>=<span style="color:#00d;font-weight:bold">1</span>
</code></pre></div><p>If you toggled the “dontaudit” setting, remember to re-enable it or you’ll end up with a very noisy log file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">semodule -B
</code></pre></div><p>If everything went well your PHP code trying to connect via FTP should now be working. If that’s not the case, keep searching for errors and let us know in the comments what was your problem.</p>
<p>Feel free to skim through our other articles for some ideas and hints:</p>
<ul>
<li><a href="/blog/2013/11/selinux-fix-for-sudo-pam/">SELinux fix for sudo PAM audit_log_acct_message() failed</a></li>
<li><a href="/blog/2013/11/selinux-and-need-of-talking-about/">SELinux and the need of talking about problems</a></li>
<li><a href="/blog/2012/05/selinux-local-policy-modules/">SELinux Local Policy Modules</a></li>
<li><a href="/blog/2009/03/passenger-and-selinux/">Passenger and SELinux</a></li>
</ul>
DAD Troublehttps://www.endpointdev.com/blog/2014/06/dad-trouble/2014-06-15T00:00:00+00:00Josh Williams
<p>I never thought I’d say it, but these days technology is simply moving too fast for DAD. It’s just the way it is. Of course it’s not DAD’s fault, it’s just the world doesn’t want to wait.</p>
<p>Before I get to that, I want to mention some trouble we’d recently started seeing with nginx failing to start on boot. It’s just been on our most recently obtained servers, both Debian-based (including Ubuntu) and RHEL-based installations. Some were Linode VM’s, others were bare metal hardware systems. After boot and once we got in to try and see what was happening, nginx would happily start manually. The only clue was one line that had been left in the error log:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">2014/06/14 23:33:20 [emerg] 2221#0: bind() to [2607:f0d0:2001:103::8]:80 failed (99: Cannot assign requested address)
</code></pre></div><p>And it wasn’t just nginx; Apache httpd in one instance gave us similar trouble:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address [2600:3c00::f03c:91ff:fe73:687f]:80
no listening sockets available, shutting down
</code></pre></div><p>As an interim fix, since at the moment these systems only had one IPv6 each, we told nginx or httpd to listen on all addresses. But not liking to leave a mystery unsolved, once we were able to schedule a long enough maintenance window on a system to reboot it a few times and see what’s going on, we found that the interface was in a “tentative” state for a short interval.</p>
<p>That was the clue we needed. For some reason, the boot process was allowed to continue before DAD (Duplicate Address Detection) has a chance to decide that if the interface is allowed to use the provided IPv6 address. It’s probably been doing this all along, but the servers that were affected just didn’t boot fast enough to try binding before the interface was ready. Now, things are faster, and service start-up was winning the race.</p>
<p>For us, the addresses are either static or autoconfigured, and we’re confident that a duplicate address situation won’t be a problem. So we turned off dad_transmits by setting this in sysctl.conf:</p>
<pre tabindex="0"><code>net.ipv6.conf.all.dad_transmits = 0
</code></pre><p>Success! No more bind problems on boot preventing a service from starting.</p>
<p>Incidentally, I believe this <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705996">has been solved in Debian</a> by making the interface wait until it’s out of the “tentative” state, but it doesn’t look like it’s been backported to current stable. It should be in Ubuntu as of the current LTS (14.04) however.</p>
<p>Happy Father’s Day!</p>
phpMemcachedAdmin: Graphical/Web Administration for memcachedhttps://www.endpointdev.com/blog/2014/02/phpmemcachedadmin-graphicalweb/2014-02-03T00:00:00+00:00Emanuele “Lele” Calò
<p>Memcached is an awesome tool, though it doesn’t offer the best interactive administration experience out there, with its command manually run via a telnet/nc connection.</p>
<p>Well luckily enough there’s an alternative to all that pain and its name is <em><a href="http://code.google.com/p/phpmemcacheadmin/">phpMemcachedAdmin</a></em></p>
<p>While the development stopped in the end of 2012 (if we don’t consider minor forks) the features offered are quite interesting.</p>
<p>Quoting directly from the main site:</p>
<p><em>[…]</em></p>
<p><em>This program allows to see in real-time (top-like) or from the start of the server, stats for get, set, delete, increment, decrement, evictions, reclaimed, cas command, as well as server stats (network, items, server version) with googlecharts and server internal configuration</em></p>
<p><em>You can go further to see each server slabs, occupation, memory wasted and items (key & value).</em></p>
<p><em>Another part can execute commands to any memcached server : get, set, delete, flush_all, as well as execute any commands (like stats) with telnet</em></p>
<p><em>[…]</em></p>
<p>Which is incredible news, even more considering the complete lack of similar features from the native Memcached code.</p>
<p>Since all the code basically revolves around a PHP file, the setup phase is just a matter of creating a new VirtualHost section inside a working web server which can serve PHP files and placing the uncompressed code tarball inside the docroot. For more details and configuration info, please consult the code homepage.</p>
<p>It’s important to highlight that phpMemcachedAdmin has absolutely no layer of security. Unless you’re setting up phpMemcachedAdmin just for teaching or testing purposes, it’s strongly advisable to protect it with at least a simple Basic Auth mechanism and possibly HTTPS in order to protect your Memcached cluster from potential malicious attackers.</p>
<p>Once you have a working instance you’ll see something similar to the following screenshots:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-0-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-0.png"/></a></div>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-1-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-1.png"/></a></div>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-2-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/02/phpmemcachedadmin-graphicalweb/image-2.png"/></a></div>
<p>Every Memcached administrator or heavy user have, at least once, wished to have such a powerful tool and now you can finally put your favourite terminal at rest… at least for the next minute.</p>
Redirect from HTTP to HTTPS before basic authhttps://www.endpointdev.com/blog/2012/12/while-reviewing-pci-scan-results-for/2012-12-22T00:00:00+00:00Ron Phipps
<p>While reviewing PCI scan results for a client I found an issue where the scanner had an issue with a private admin URL requesting basic http auth over HTTP. The admin portion of the site has its own authentication method and it is served completely over HTTPS. We have a second layer of protection with basic auth, but the issue is the username and password could be snooped on since it can be accessed via HTTP.</p>
<p>The initial research and attempts at fixing the problem did not work out as intended. Until I found <a href="http://misof.blog.matfyz.sk/p10659-redirection-to-https-done-the-right-way">this blog post on the subject</a>. The blog laid out all of the ways that I had already tried and then a new solution was presented.</p>
<p>I followed the recommended hack which is to use SSLRequireSSL in a location matching the admin and a custom 403 ErrorDocument. This 403 ErrorDocument does a bit of munging of the URL and redirects from HTTP to HTTPS. The instructions in the blog did have one issue, in our environment I could not serve the 403 document from the admin, I had to have it in an area that could be accessed by HTTP and by the public. I’m not sure how it could work being served from a URL that requires ssl and is protected by basic auth. The reason that this hack does work is because SSLRequireSSL is processed before any auth requirements and ErrorDocument 403 is presented when SSL is not being used.</p>
<p>Now hopefully the scanner will be happy (as happy as a scanner can be) by always requiring HTTPS when /admin appears in the URL and presenting an error when that is not the case, before the basic auth is requested.</p>
Slash URLhttps://www.endpointdev.com/blog/2012/12/slash-url/2012-12-04T00:00:00+00:00Jeff Boes
<p>There’s always more to learn in this job. Today I learned that <a href="http://www.jampmark.com/web-scripting/5-solutions-to-url-encoded-slashes-problem-in-apache.html">Apache web server</a> is smarter than me.</p>
<p>A typical SEO-friendly solution to Interchange pre-defined searches (item categories, manufacturer lists, etc.) is to put together a URL that includes the search parameter, but looks like a hierarchical URL:</p>
<p>/accessories/Mens-Briefs.html</p>
<p>/manufacturer/Hanes.html</p>
<p>Through the magic of <a href="http://interchange.rtfm.info/icdocs/config/ActionMap.html">actionmaps</a>, we can serve up a search results page that looks for products which match on the “accessories” or “manufacturer” field. The problem comes when a less-savvy person adds a field value that includes a slash:</p>
<p>accessories: “Socks/Hosiery”</p>
<p>or</p>
<p>manufacturer: “Disney/Pixar”</p>
<p>Within my actionmap Perl code, I wanted to redirect some URLs to the canonical actionmap page (because we were trying to short-circuit a crazy Web spider, but that’s beside the point). So I ended up (after several wild goose chases) with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$new_path</span> = <span style="color:#d20;background-color:#fff0f0">'/accessories/'</span> .
<span style="color:#b06;font-weight:bold">Vend::Tags</span>->filter({body => (<span style="color:#038">join</span> <span style="color:#d20;background-color:#fff0f0">'%2f'</span> => (<span style="color:#038">grep</span> { <span style="color:#080;background-color:#fff0ff">/\D/</span> } <span style="color:#369">@path</span>)),
op => <span style="color:#d20;background-color:#fff0f0">'urlencode'</span>, }) .
<span style="color:#d20;background-color:#fff0f0">'.html'</span>;
</code></pre></div><p>By this I mean: I put together my path out of my selected elements, joined them with a URL-encoded slash character (%2f), and then further URL-encoded the result. This was counter-intuitive, but as you can see at the first link in this article, it’s necessary because Apache is smarter than you. Well, than me anyway.</p>
Custom Apache log to only show HTML requestshttps://www.endpointdev.com/blog/2012/04/custom-apache-log-to-only-show-html/2012-04-06T00:00:00+00:00Ron Phipps
<p>Today while working on an AJAX issue for <a href="https://www.collegedistrict.com">CollegeDistrict.com</a> I came across a need to only see HTML requests to Apache while leaving out all of the many requests for images, CSS, and JavaScript files. This would make it quite easy to see when AJAX requests were making it through properly.</p>
<p>I found <a href="https://web.archive.org/web/20120428121948/https://www.serverwatch.com/tutorials/article.php/10825_3376671_2/Advanced-Logging-Techniques-With-Apache.htm">a solution which worked well</a> and used these settings in our development <code>httpd.conf</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">SetEnvIf Request_URI "(\.html|\.shtml)$" html
CustomLog logs/html.log common env=html
</code></pre></div>