https://www.endpointdev.com/blog/tags/mediawiki/2016-03-11T00:00:00+00:00End Point DevMediaWiki extension EmailDiff: notification emails improvedhttps://www.endpointdev.com/blog/2016/03/mediawiki-extension-emaildiff/2016-03-11T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; padding: 0 0 .5em 1.5em; text-align: center;"><a href="/blog/2016/03/mediawiki-extension-emaildiff/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2016/03/mediawiki-extension-emaildiff/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/bdWyHP">Photo</a> by <a href="https://www.flickr.com/photos/karenandbrademerson/">Karen and Brad Emerson</a></small></div>
<p>One of the nice things about MediaWiki is the ability to use
<a href="https://www.mediawiki.org/wiki/Manual:Extensions">extensions</a> to extend the core functionality in many ways. I’ve just released a
new version of an extension I wrote called EmailDiff that helps provide a much needed
function. When one is using a MediaWiki site, and a page is on your
watchlist—or your username is inside<br>
<a href="https://www.mediawiki.org/wiki/Manual:$wgUsersNotifiedOnAllChanges">the ‘UsersNotifiedOnAllChanges’ array</a>—you will receive an email whenever a page
is changed. However, this email simply gives you the editor’s summary and states
“the page has been changed, here’s some links if you want to see exactly what”.
With the EmailDiff extension enabled, a full diff of what exactly has changed is sent
in the email itself. This is extremely valuable because you can quickly see exactly what has
changed, without leaving your email client to open a browser (and potentially have to login),
and without breaking <a href="https://en.wikipedia.org/wiki/Flow_%28psychology%29">your flow</a>.</p>
<p>Normally, a MediaWiki notification email for a page change will look something like this:</p>
<pre tabindex="0"><code>Subject: MediaWiki page Project:Sandbox requirements has been changed by Zimmerman
Dear Turnstep,
The MediaWiki page Project:Sandbox requirements has been changed on
16 November 2015 by Zimmerman, see
https://www.mediawiki.org/wiki/Project:Sandbox for the
current revision.
See
https://www.mediawiki.org/w/index.php?title=Project:Sandbox&diff=next&oldid=7076877
to view this change.
See
https://www.mediawiki.org/w/index.php?title=Project:Sandbox&diff=0&oldid=8657769
for all changes since your last visit.
Editor's summary: important thoughts
Contact the editor:
mail: https://www.mediawiki.org/wiki/Special:EmailUser/Zimmerman
wiki: https://www.mediawiki.org/wiki/User:Zimmerman
There will be no other notifications in case of further activity unless
you visit this page while logged in. You could also reset the
notification flags for all your watched pages on your watchlist.
Your friendly MediaWiki notification system
--
To change your email notification settings, visit
https://www.mediawiki.org/wiki/Special:Preferences
To change your watchlist settings, visit
https://www.mediawiki.org/wiki/Special:EditWatchlist
To delete the page from your watchlist, visit
https://www.mediawiki.org/w/index.php?title=Project:Sandbox&action=unwatch
Feedback and further assistance:
https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents
</code></pre><p>The above is the default email message for page changes on mediawiki.org. As you can
see, it is very wordy, but conveys little actual information. In contrast,
here is the EmailDiff extension, along with the suggested changes in the
notification email template mentioned below:</p>
<pre tabindex="0"><code>Subject: MediaWiki page Project:Sandbox requirements has been changed by Zimmerman (diff)
Page: Project:Sandbox
Summary: important thoughts
User: Zimmerman Time: 11 November 2015
Version differences:
@@ -846,5 +887,3 @@
In cattle, temperament can affect production traits such as carcass and meat
quality or milk yield as well as affecting the animal's overall health and
-reproduction. Cattle temperament is defined as "the consistent behavioral and physiological
-difference observed between individuals in response to a stressor or environmental
+reproduction. If you succeed in tipping a cow only partway, such that only one
+of its feet is still on the ground, you have created lean beef. Such a feat is
+well done. Naturally, being outside, the cow is unstable. When it falls over,
+it becomes ground beef. Cattle temperament is defined as "the consistent behavioral
+and physiological difference observed between individuals in response to a stressor or environmental
challenge and is used to describe the relatively stable difference in the behavioral
predisposition of an animal, which can be related to psychobiological mechanisms.
</code></pre><p>That is so much better: short, sweet, and showing exactly the information you need. The
lack of a diff has long been a pet peeve of mine, so much so that I wrote this
functionality a long time ago as some hacks to the core code. Now, however,
everything is bottled up into one neat extension.</p>
<p>This extension works by use of the great
<a href="https://www.mediawiki.org/wiki/Manual:Hooks">hook system of MediaWiki</a>. In particular, it uses the <strong>SendPersonaliedNotificationEmail</strong>
hook. It is not yet included in MediaWiki, but I am hoping it will get included for version 1.27.
The hook fires just before the normal notification email is about to be sent. The extension generates
the diff, and sticks it inside the email body. It will also append the string ‘(diff)’ to the subject
line, but that is configurable (see below).</p>
<p>The extension has changed a lot over the years, moving forward along
with MediaWiki, whose support of extensions gets better all the time.
The current version of the EmailDiff extension, 1.7, requires a
MediaWiki version of 1.25 or better, as it uses the
<a href="/blog/2015/10/mediawiki-extensionjson-change-in-125/">new extension.json format</a>.</p>
<p>Installation is pretty straightforward with four steps. First, visit the
<a href="https://www.mediawiki.org/wiki/Extension:EmailDiff">official extension page</a> at mediawiki.org, download the tarball, and untar
it into your extensions directory. Second, add this line to your
LocalSettings.php file:</p>
<pre tabindex="0"><code>wfLoadExtension( 'EmailDiff' );
</code></pre><p>If you need to change any of the configuration settings, add them to
LocalSettings.php right below the wfLoadExtension
line. Currently, the only two configuration items are:</p>
<ul>
<li><strong>$wgEmailDiffSubjectSuffix</strong> This is a string that gets added to the
end of any notification emails that contain a diff. Defaults to <em><strong>(diff)</strong></em>.</li>
<li><strong>$wgEmailDiffCommand</strong> This is the command used to execute the diff.
It should not need to be changed for most systems. Defaults to
<em><strong>"/usr/bin/diff -u OLDFILE NEWFILE | /usr/bin/tail –lines=+3 > DIFFFILE"</strong></em></li>
</ul>
<p>As mentioned above, this extension requires the <strong>SendPersonaliedNotificationEmail</strong> hook to exist.
For the third step, you need to add the hook in if it does not exist by editing the
includes/mail/EmailNotification.php file. Insert this line at the
bottom of the <strong>sendPersonalized</strong> function, just before the final return:</p>
<pre tabindex="0"><code>Hooks::run( 'SendPersonalizedNotificationEmail',
[ $watchingUser, $this->oldid, $this->title, &$headers, &$this->subject, &$body ] );
</code></pre><p>The final step is to modify the template used to send the notification emails. You can find it
by editing this page on your wiki: <strong>MediaWiki::Enotif_body</strong>, and adding the string <strong>$PAGEDIFF</strong>
where you want the diff to appear. I recommend cleaning up the template while you are in there. Here is
my preferred template:</p>
<pre tabindex="0"><code>Page: $PAGETITLE
Summary: $PAGESUMMARY $PAGEMINOREDIT
User: $PAGEEDITOR Time: $PAGEEDITDATE
$PAGEDIFF
$NEWPAGE
</code></pre><p>Once installed, you will need to activate the email diffs for one or more users. A new
<a href="https://www.mediawiki.org/wiki/Help:Preferences">user preference</a> that allows emailing of diffs has been added. It is off by default; to turn
it on, a user should visit their “Preferences” link, go to the “User profile” section, and look inside
the “Email options” for a new checkbox that says “Send a diff of changes” (or the same but in a
different language, if the
<a href="https://www.mediawiki.org/wiki/Localisation">localization</a> has been set up). The checkbox will look like this:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2016/03/mediawiki-extension-emaildiff/image-1-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2016/03/mediawiki-extension-emaildiff/image-1.png"/></a></div>
<p>Just check the box, click the “Save” button, and your notification emails become
much more awesome. To enable email diffs for everyone on your wiki, add this line to your
LocalSettings.php file:</p>
<pre tabindex="0"><code>$wgDefaultUserOptions['enotifshowdiff'] = true;
</code></pre><p>There are some limitations to this extension that should be mentioned. As each
page edit will potentially cause three files to be created on the operating system
as well as invoking an external diff command, large and extremely busy wikis may see a
performance impact. However, file creations are cheap and the diff command is
fast, so unless you are Wikipedia, it’s probably worth at least testing out to
see if the impact is meaningful.</p>
<p>I also like these emails as a kind of audit trail for the wiki. On that note,
email notifications do NOT get sent to changes you have made yourself! Well, they do
for me, but that has required some hacking of the core MediaWiki code. Maybe someday
I will attempt to make that into a user preference and/or extension as well. :)</p>
Bonked By Basic_auth Because Bcrypthttps://www.endpointdev.com/blog/2016/02/bonked-by-basicauth-because-bcrypt/2016-02-09T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float: right; padding: 0 1em 1em 2em; text-align: center;"><a href="/blog/2016/02/bonked-by-basicauth-because-bcrypt/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2016/02/bonked-by-basicauth-because-bcrypt/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/xUswo">Alligator photo</a> by <a href="https://www.flickr.com/people/johnjack/">Random McRandomhead</a></small></div>
<p><strong>TL;DR: Don’t use a high bcrypt cost with HTTP basic auth!</strong></p>
<p>Recently we had a client approach us with reports of a slow wiki experience. This was
for a <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> we recently installed for them;
there were no
<a href="https://www.mediawiki.org/wiki/Manual:Extensions">fancy extensions</a>, and the hardware, the OS, and the
<a href="https://en.wikipedia.org/wiki/Apache_HTTP_Server">Apache web server</a> were solid, perfectly normal choices.
I was tasked to dive in and solve this issue.</p>
<p>The first step in any troubleshooting is to verify and duplicate the problem. While the
wiki did feel a bit sluggish, it was not as bad as the reports we were getting of taking
over 15 seconds to view a page. A side-by-side comparison with a similar wiki seemed a
good place to start. I called up the main wiki page on both the client wiki and End Point’s
internal wiki. Both were running the latest version of MediaWiki, had the same type of
servers (located a similar distance from me), were using the same version of Apache, and
had roughly the same
<a href="https://en.wikipedia.org/wiki/Load_%28computing%29">server load</a>. While both
wiki’s pages had roughly the same amount of content, the client one loaded noticeably slower.
It took less than a second for the End Point wiki, and around ten seconds for the client one!</p>
<p>The first possible culprit was MediaWiki itself. Perhaps something was misconfigured there, or
some extension was slowing everything down? MediaWiki has
<a href="https://www.mediawiki.org/wiki/Manual:How_to_debug">good debugging tools</a>. Inside the both wiki’s <strong>LocalSettings.php</strong> file I
turned on debugging temporarily with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$wgDebugLogFile</span> = <span style="color:#d20;background-color:#fff0f0">'/tmp/mediawiki.debug'</span>;
<span style="color:#369">$wgDebugDBTransactions</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#369">$wgDebugDumpSql</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#369">$wgDebugTimestamps</span> = <span style="color:#080;font-weight:bold">true</span>;
</code></pre></div><p>I reloaded the page, then commented out the $wgDebugLogFile line to stop it from
growing large (the debug output can be quite verbose!). Here’s some snippets from
the generated log file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">0.9151 4.2M Start request GET /wiki/Main_Page
...
[caches] main: SqlBagOStuff, message: SqlBagOStuff, parser: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
0.9266 9.2M Implicit transaction open enabled.
0.9279 9.2M Query wikidb (1) (slave): SET /* DatabasePostgres::open */ client_encoding='UTF8'
0.9282 9.2M Resource id #127: Transaction state changed from IDLE -> ACTIVE
0.9268 9.2M Query wikidb (2) (slave): SET /* DatabasePostgres::open */ datestyle = 'ISO, YMD'
...
0.9587 9.2M Query wikidb (11) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
0.9573 9.5M Query wikidb (12) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'list' LIMIT 1
0.9567 10.8M Query wikidb (13) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'preload' LIMIT 1
0.9572 10.8M Query wikidb (14) (slave): SELECT /* LCStoreDB::get */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'preload' LIMIT 1
...
0.9875 21.2M Query wikidb (195) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-mycontris' LIMIT 1
0.9873 21.2M Query wikidb (196) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:tooltip-pt-logout' LIMIT 1
0.9868 21.2M Query wikidb (197) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-logout' LIMIT 1
0.9883 21.2M Query wikidb (198) (slave): SELECT /* LCStoreDB::get Greg */ lc_value FROM "l10n_cache" WHERE lc_lang = 'en' AND lc_key = 'messages:vector-more-actions' LIMIT 1
</code></pre></div><p>Just to load a simple page, there were 194 SELECT statements! And 137 of those were trying to look in the l10n_cache table, one
row at a time. Clearly, there is lots of room for improvement there. Someday, I may even jump in and tackle that. But for now,
despite being very inefficient, it is also very fast. Because of the $wgDebugTimestamps, it was easy to compute how much
time both wikis spent actually creating the page and sending it back to Apache. In this case, the difference was minimal,
which meant MediaWiki was not the culprit.</p>
<p>I then turned my attention to Apache. Perhaps it was compiled differently? Perhaps there was some
obscure
<a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">SSL</a> bug slowing things down for everyone? These were unlikely, but it was worth checking the Apache logs
(which were in /var/log/httpd). There are
two main logs Apache uses: access and error. The latter revealed nothing at all when I loaded the
main wiki page. The access logs looked fairly normal:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">85.236.207.120 - greg [19/Jan/2016:12:23:21 -0500] "GET /wiki/Main_Page HTTP/1.1" 200 23558 "-" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/extensions/balloons/js/balloon.config.js HTTP/1.1" 200 4128 "https://wiki.endpoint.com/wiki/Main_Page
" "Mozilla/5.0 Firefox/43.0"
...
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector HTTP/1.1" 200 58697 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/resources/assets/poweredby_mediawiki_88x31.png HTTP/1.1" 200 3525 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"
</code></pre></div><p>Still nothing out of the ordinary. What to do next? When all else fails, go to the system calls. It’s about as close to bare metal as you can easily get on a Linux system. In this case, I decided to run
<a href="https://en.wikipedia.org/wiki/Strace">strace</a> on the Apache
<a href="https://en.wikipedia.org/wiki/Daemon_%28computing%29">daemon</a> to see exactly where the time was being spent. As expected, there were a large handful of httpd processes already
spawned and waiting for a connection. While there was no way to know which one would field my requests, some shell-fu allowed me
to strace them all at once:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">## The -u prevents us from picking the parent httpd process, because it is owned by root!
$ strace -o greg.httpd.trace -tt -ff `pgrep -u apache httpd | xargs -n 1 echo -p | xargs`
Process 5148 attached
Process 4848 attached
Process 5656 attached
Process 4948 attached
Process 5149 attached
Process 5148 attached
Process 4858 attached
Process 5657 attached
Process 4852 attached
Process 4853 attached
^CProcess 5148 detached
Process 4848 detached
Process 5656 detached
Process 4948 detached
Process 5149 detached
Process 5148 detached
Process 4858 detached
Process 5657 detached
Process 4852 detached
Process 4853 detached
</code></pre></div><p>Looking at one of the output of one of these revealed some important clues:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ head greg.httpd.trace.4948
13:00:28.799807 read(14, "\27\3\3\2\221\0\0\0\0\0\0\0\1\35-\332\3123(\200\302\"\251'g\256\363b5"..., 8000) = 666
13:00:28.799995 stat("/wiki/htdocs/mediawiki/load.php", {st_mode=S_IFREG|0644, st_size=1755, ...}) = 0
13:00:28.800126 open("/wiki/htpasswd.users", O_RDONLY|O_CLOEXEC) = 15
13:00:28.800176 fstat(15, {st_mode=S_IFREG|0640, st_size=947, ...}) = 0
13:00:28.800204 read(15, "alice:{SHA}jA0EAgMCMEpo4Wa3n/9gy"..., 4096) = 2802
13:00:28.800230 close(15) = 0
13:00:29.496369 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
13:00:29.496863 rt_sigaction(SIGPROF, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670}, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670
</code></pre></div><p>Aha! If you look close at those timestamps, you will notice that the time gap from the call to close() and the subsequent setitimer() is quite large at .69 seconds. That’s a long time for Apache to be waiting around for something. The second clue is the file it just opened: “htpasswd.users”. Seeing the top of the file, with the {SHA} in quotes, made me realize the problem—htpasswd files now support bcrypt as an authentication method, and bcrypt is designed to be secure—and slow. Sure enough, the htpasswd file had bcrypt entries with a high cost for the people that were having the most issues with the speed. This is what the file looked like (names and values changed):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">alice:{SHA}jA0EAgMCMEpo4Wa3n/9gybBBsDPa
greg:$2y$13$+lE6+EwgtzP0m8K8VQDnYMRDRMf6rNMRZsCzko07QQpskKI9xbb/y9
mallory:$2y$15$ww8Q4HMI1Md51kul2Hiz4ctetPqJ95cmspH8T81JHfqRvmg===rVgn
carol:7RnEKJWc38uEO
bob:$apr1$uKX9Z63CqPOGX4lD1R4yVZsloJyZGf+
jon:$2y$08$SUe3Z8sgEpyDWbWhUUUU5wtVTwlpEdc7QyXOg3e5WBwM4Hu35/OSo1
eve:$apr1$I/hv09PcpU0VfXhyG7ZGaMz7Vhxi1Tm
</code></pre></div><p>I recognized the bcrypt format right away ($2y$13$). The people who were complaining the most (e.g. mallory in the example above) about the speed of the wiki had the highest costs, while those with low costs (e.g. jon), and those using something other than bcrypt (everyone else above), were not complaining at all!</p>
<p>The ‘cost’ is the number after the second dollar sign: as you can see, some of them had a cost of <strong>15</strong>, which is much more expensive than a cost of <strong>13</strong>, which is what my user (“greg”) was using. This was a smoking gun, but one more step was needed for proof. I adjusted the cost of my password to something low using the <a href="https://httpd.apache.org/docs/current/programs/htpasswd.html">htpasswd program</a>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ htpasswd -B -C 6 /wiki/htpasswd.users greg
New password:
Re-type new password:
Updating password for user greg
</code></pre></div><p>Voila! The page loaded in a flash. I then changed the cost to 15 and suddenly the wiki was even slower than before—taking
upwards of 15 seconds to load the main page of the wiki. Mystery solved. All those high cost bcrypt requests are also not good
for the server: not only does it use a lot of CPU, but ends up keeping the Apache daemon tied up waiting for the bcrypt to
finish, rather than simply finishing up quickly and going back to the main pool.</p>
<p>You may be asking a few questions at this point, however. Why would htpasswd offer a footgun like this? Why such a radical
difference in effect for slightly different costs? Is bcrypt a good practice for a htpasswd file? Let’s attempt to answer
those. Before we do, we have to learn a little bit about bcrypt and passwords in general. Some of this is purposefully
oversimplified, so be gently in the comments. :)</p>
<p>Passwords themselves are never stored on a server (aka the machine doing the authentication). Instead, the server stores
a hash of the password. This is created by what is known as a “one-way” function, that creates a unique fingerprint of your
password. If this fingerprint (aka hash) is discovered, there is no direct way to see the password that created it. When
you login to a site, it creates a hash of the password you give it, then compares that hash to the one it has stored. Thus,
it can verify that you have given it the correct password without actually having to store the password.</p>
<p>For a long time, very simple algorithms were used to create these hashes. However, as computers became more powerful,
and as the field of cryptography advanced, it became easier to “crack” these hashes and determine the password that
was used to create them. This was an important problem, and one of the solutions that people came up with was the
<a href="https://en.wikipedia.org/wiki/Bcrypt">bcrypt algorithm</a>, which makes the computation of the hash very expensive, in terms of computer speed. Furthermore, that
speed is adjustable, and determined by the “cost” given at creation time. You may have noticed the <strong>-C</strong> option I
used in the htpasswd example above. That number indicates the number of rounds the algorithm must go through. However, the cost
given leads to 2^code rounds, which means that the cost is exponential. In other words, a cost of 13 means that bcrypt runs
2 to the 13th power rounds, or 8,192 rounds. A cost of 14 is 2 to the 14th power, or 16,384 rounds—twice as slow as
a cost of 13! A cost of 15 is 32,768 rounds, etc. Thus, one can see why even a cost of 15 would be much slower than a cost of 13.</p>
<p>A web page usually returns more than just the requested HTML. There are commonly images, CSS, and JavaScript that must also be
loaded from the webserver to fully render the page. Each of these requests must go through basic auth, and thus get slowed down
by bcrypt. This is why even though each basic authentication via bcrypt of 15 only takes a couple of seconds, the entire web
page can take much longer.</p>
<p>What encryption options are available for htpasswd program? The bcrypt option was introduced without much fanfare in
<a href="https://httpd.apache.org/docs/2.4/new_features_2_4.html#programs">version 2.4.4 of Apache</a>, which was released on February 25, 2013. So, it’s been around a while. The output of –help shows
us that bcrypt is the only secure one, but allows for other legacy ones to be used. Also note that the range of costs for
bcrypt range from 4 to 31:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> -m Force MD5 encryption of the password (default).
-B Force bcrypt encryption of the password (very secure).
-C Set the computing time used for the bcrypt algorithm
(higher is more secure but slower, default: 5, valid: 4 to 31).
-d Force CRYPT encryption of the password (8 chars max, insecure).
-s Force SHA encryption of the password (insecure).
-p Do not encrypt the password (plaintext, insecure).
</code></pre></div><p>So should you use bcrypt for your htpasswd? Absolutely yes. Even a lower cost bcrypt is incredibly
more secure than using MD5, CRYPT, or SHA. A cost of 10 is roughly the same speed as those,
but a much, much better choice. You can measure the time it takes to create or update your password via
the command-line htpasswd command to get a rough idea of how much impact it will have on your
website. You can use the time it takes to run the htpasswd command as rough proxy for the total
page load time. Here are some numbers I generated on my local box. Numbers represent the average
of a few runs:</p>
<table border="1">
<tbody><tr>
<th>Bcrypt cost</th>
<th>htpasswd creation time</th>
<th>Web page load time</th>
</tr>
<tr><td>10</td><td>0.079</td><td>5.68 seconds</td></tr>
<tr><td>12</td><td>0.268</td><td>6.77 seconds</td></tr>
<tr><td>14</td><td>0.979</td><td>10.78 seconds</td></tr>
<tr><td>16</td><td>3.684</td><td>25.72 seconds</td></tr>
<tr><td>18</td><td>14.683</td><td>88.85 seconds</td></tr>
<tr><td>20</td><td>58.680</td><td>358.80 seconds</td></tr>
<tr><td>22</td><td>236.369</td><td>1357.82 seconds</td></tr>
<tr><td>31</td><td>186,173 seconds<br/>(51 hours and 42 minutes!!)</td><td>Ah...no</td></tr>
</tbody></table>
<p>There are times where you really do want a higher bcrypt cost. The basic auth usage in this
scenario is really the exception, and not the norm. In most cases, a password will be used to log in
to something, and you will either create a persistent connection (e.g. SSH), or a cookie with a
temporary token will be issued (e.g. almost every website in the world). In those cases, a few
seconds delay are quite acceptable, as it is a rare event.</p>
<p>So why do we even care about passwords so much, especially for something like basic auth and
a htpasswd file? After all, if someone can view the contents of the htpasswd file, they can
also more than likely view whatever material on the web server it was designed to protect.
These days, however, it’s important to view strong hashes such as bcrypt as not just
protecting data, but protecting the password as well. Why? Password reuse.
It’s very common for people to use the same (or very similar) password on all the sites
they visit. The danger is thus not that an attacker can view the file contents protected
by the htpasswd file, but that an attacker can use that password on the user’s email accounts, or
on other sites the user may have visited and used the same password.</p>
<p>What bcrypt cost should you use? The general answer is to use the highest possible cost
you can get away with. Take something with such a high cost that is causes discomfort
to the users, then dial it back a tiny bit. Measure it out and see what your server
can handle. For general bcrypt use, start with 13, but don’t be afraid to keep going up until it takes a
wall clock second or two to run. For basic auth, use something very fast: perhaps 9 or less. Anything
that takes over a second to create via htpasswd will slow a site down noticeably!</p>
MediaWiki major upgrade processhttps://www.endpointdev.com/blog/2016/01/mediawiki-major-upgrade-process/2016-01-13T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; text-align: center;"><a href="/blog/2016/01/mediawiki-major-upgrade-process/image-0-big.jpeg" id="jA0ECgMCAgq+j/dF5fVg0p8B+8AMgSoFokDOSe5KCFaxWZJhpt3VCeVufE4QxYsVo4JdJsN/neWFX+7wvaOGwJ4PWGvCJkIhTWbhp40tAbTL4uxmK3Qn7vru5vFtQvwJab4uSmKNp+lG4rTqWrHU2gi0wLlpFCKQ7WabVQ5cAdwwUInGP/13zrBj9J0Cg9FAZVjb5M9qFDNl6kip6Gg8dl5IXp+PKyhRd09Nng4yBKM==zRAb" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2016/01/mediawiki-major-upgrade-process/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/26krAr">Flower photo</a> by <a href="https://www.flickr.com/photos/mayrace/">May Race</a></small></div>
<p>Keeping your MediaWiki site up to date with the latest version is, like many sysadmin tasks, a never-ending chore. In a previous article I covered how to <a href="/blog/2014/10/mediawiki-minor-upgrade-with-patches/">upgrade minor revisions of MediaWiki with patches.</a> In this one, I’ll cover my solution to doing a “major” upgrade to MediaWiki. While the <a href="https://www.mediawiki.org/wiki/Manual:Upgrading">official upgrade instructions</a> are good, they don’t cover everything.</p>
<p>MediaWiki, like Postgres, uses a three-section version number in which the first two
numbers combined give the major version, and the number on the end the revision of
that branch. Thus, version 1.26.2 is the third revision (0, then 1, then 2) of the
1.26 version of MediaWiki. Moving from one major version to another (for example 1.25
to 1.26) is a larger undertaking than updating the revision, as it involves significant
software changes, whereas a minor update (in which only the revision changes) simply
provides bug fixes.</p>
<p>The first step to a major MediaWiki upgrade is to try it on a cloned, test version of your wiki.
See <a href="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/">this article</a> on how to make such a clone. Then run through the steps below to find any problems
that may crop up. When done, run through again, but this time on the actual live site.
For this article, we will use MediaWiki installed in <strong>~intranet/htdocs/mediawiki</strong>, and
going from version 1.25.3 to 1.26.2</p>
<h3 id="preparation">Preparation</h3>
<p>Before making any changes, make sure everything is up to date in
<a href="https://rogerdudler.github.io/git-guide/">Git</a>. You do have your MediaWiki
site controlled by Git, right? If not, go do so right now. Then check you are on the main branch
and have no outstanding changes. It should look like this:</p>
<pre tabindex="0"><code>$ cd ~/htdocs/mediawiki
$ git status
# On branch master
nothing to commit, working directory clean
</code></pre><h3 id="download">Download</h3>
<p>Time to grab the new major version. Always get the latest revision in the current
branch. For this example, we want the highest in the 1.26 branch, which as of this
writing is 1.26.2. You can always find a prominent link on <a href="https://mediawiki.org/">mediawiki.org</a>. Make sure you
grab both the tarball (tar.gz) and the signature (.tar.gz.sig) file, then use gnupg to verify it:</p>
<pre tabindex="0"><code>$ wget https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz
$ wget https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz.sig
$ gpg mediawiki-1.26.2.tar.gz.sig
gpg: assuming signed data in `mediawiki-1.26.2.tar.gz'
gpg: Signature made Sun 20 Dec 2015 08:13:14 PM EST using RSA key ID 23107F8A
gpg: please do a --check-trustdb
gpg: Good signature from "Chad Horohoe <chad@wikimedia.org>"
gpg: aka "keybase.io/demon <demon@keybase.io>"
gpg: aka "Chad Horohoe (Personal e-mail) <innocentkiller@gmail.com>"
gpg: aka "Chad Horohoe (Alias for existing email) <chadh@wikimedia.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A
</code></pre><p>Copy the tarball to your server, and untar it in the same base directory as your
mediawiki installation:</p>
<pre tabindex="0"><code>$ cd ~/htdocs
$ tar xvfz ~/mediawiki-1.26.2.tar.gz
</code></pre><h3 id="copy-files">Copy files</h3>
<p>Copy the LocalSettings.php file over, as well as any custom images (e.g. the logo, which I like
to keep nice and visible at the top level):</p>
<pre tabindex="0"><code>$ cp mediawiki/LocalSettings.php mediawiki-1.26.2/
$ cp mediawiki/Wiki_logo.png mediawiki-1.26.2/
</code></pre><p>Setup the images directory. The tarball comes with a dummy directory containing a few unimportant files. We want to replace
that with our existing one. I keep the images directory a level up from the actual mediawiki
directory, and symlink it in. This allows for easy testing and upgrades:</p>
<pre tabindex="0"><code>$ cd ~/htdocs/mediawiki-1.26.2
$ rm -fr images/ ## Careful, make sure you are in the right directory! :)
$ ln -s ../images/ .
</code></pre><h3 id="copy-extensions">Copy extensions</h3>
<p>Now it is time to copy over the extensions. MediaWiki bundles a number of extensions in
the tarball, as they are considered “core” extensions. We do not want to overwrite these
with our old versions. We do want to copy any extensions that exist in our old
mediawiki directory, yet not in our newly created one. To help keep things straight and
reduce typing, let’s make some symlinks for the existing (old) MediaWiki and for the
current (new) MediaWiki, naming them “aa” and “bb” respectively. Then we use “diff” to help
us copy the right extensions over:</p>
<pre tabindex="0"><code>$ cd ~/htdocs
$ ln -s mediawiki aa
$ ln -s mediawiki-1.26.2 bb
## Visually check things over with:
$ diff aa/extensions bb/extensions | grep 'Only in aa' | awk '{print $4}' | more
## Do the copying:
$ diff aa/extensions bb/extensions | grep 'Only in aa' | awk '{print $4}' | xargs -iZ cp -r aa/extensions/Z bb/extensions/Z
</code></pre><p>Extensions may not be the only way you have modified your installation. There could
be skins, custom scripts, etc. Copy these over now, being sure to only copy what is
truly still needed. Here’s one way to check on the differences:</p>
<pre tabindex="0"><code>$ cd ~/htdocs
$ diff -r aa bb | grep 'Only in aa' | more
</code></pre><h3 id="check-into-git">Check into git</h3>
<p>Now that everything is copied over, we can check the 1.26.2 changes into git. To do
so, we will move the git directory from the old directory to the new one. Remember to let anyone who might
be developing in that directory know what you are doing first!</p>
<pre tabindex="0"><code>$ mv aa/.git bb/
## Don’t forget this important file:
$ mv aa/.gitignore bb/
$ cd mediawiki-1.26.2
$ git add .
$ git commit -a -m "Upgrade to version 1.26.2"
$ git status
# On branch master
nothing to commit, working directory clean
</code></pre><h3 id="extension-modifications">Extension modifications</h3>
<p>This is a good time to make any extension changes that are needed for the new version.
These should have been revealed in the first round, using the cloned test wiki. In our case,
we needed an updated and locally hacked version of the <a href="https://www.mediawiki.org/wiki/Extension:Auth_remoteuser">Auth_remoteuser extension</a>:</p>
<pre tabindex="0"><code>$ cd ~/htdocs/mediawiki-1.26.2/extensions
$ rm -fr Auth_remoteuser/
$ tar xvfz ~/Auth_remoteuser.tgz
$ git add Auth_remoteuser
$ git commit -a -m "New version of Auth_remoteuser extension, with custom fix for wpPassword problem"
</code></pre><h3 id="core-modifications">Core modifications</h3>
<p>One of the trickiest part of major upgrades is the fact that all the files are simply replaced.
Normally not a problem, but what if you are in the habit of modifying the core files because sometimes
extensions cannot do what you want? My solution is to tag the changes prominently—using a PHP comment
that contains the string “END POINT”. This makes it easy to generate a list of files that may
need the local changes applied again. After using “git log” to find the commit ID of the 1.26.2
changes (message was “Upgrade to version 1.26.2”), we can grep for the unique string and
figure out which files to examine:</p>
<pre tabindex="0"><code>$ git log 1a83a996b9d00444302683fb6de6e86c4f4006e7 -1 -p | grep -E 'diff|END POINT' | grep -B1 END
diff --git a/includes/mail/EmailNotification.php b/includes/mail/EmailNotification.php
- // END POINT CHANGE: ignore the watchlist timestamp when sending notifications
- // END POINT CHANGE: send diffs in the emails
diff --git a/includes/search/SearchEngine.php b/includes/search/SearchEngine.php
- // END POINT CHANGE: Remove common domain suffixes
</code></pre><p>At that point, manually edit both the new and old version of the files and make the
needed changes. After that, remember to commit all your changes into git.</p>
<h3 id="final-changes">Final changes</h3>
<p>Time to make the final change, and move the live site over. The goal is to minimize the downtime,
so we will move the directories around and run the update.php script on one line. This is an excellent
time to notify anyone who may be using the wiki that there may be a few bumps.</p>
<pre tabindex="0"><code>## Inform people the upgrade is coming, then:
$ mv mediawiki old_mediawiki; mv mediawiki-1.26.2 mediawiki; cd mediawiki; php maintenance/update.php --quick
$ rm ~/htdocs/aa ~/htdocs/bb
</code></pre><h3 id="testing">Testing</h3>
<p>Hopefully everything works! Time to do some testing. First, visit your wiki’s Special:Version page and
make sure it says 1.26.2 (or whatever version you just installed). Next, test that most things are still
working by:</p>
<ul>
<li>Logging in, and…</li>
<li>Editing a page, then…</li>
<li>Upload an image, plus…</li>
<li>Test all your extensions.</li>
</ul>
<p>For that last bullet, having an extension testing page is very handy. This is simply an unused page on the
wiki that tries to utilize as many active extensions as possible, so that reloading the page should quickly
allow a tally of working and non-working extensions. I like to give each extension a header with its name,
a text description of what should be seen, and then the actual extension in action.</p>
<p>That’s the end of the major upgrade for MediaWiki! Hopefully in the future the upgrade process will
be better designed (I have ideas on that—but that’s the topic of another article). One final check you can do is to
open a screen and tail -f the httpd error log for your site. After the upgrade, this is a helpful
way to spot any issues as they come up.</p>
Broken wikis due to PHP and MediaWiki “namespace” conflictshttps://www.endpointdev.com/blog/2015/11/broken-wikis-due-to-php-and-mediawiki/2015-11-09T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; text-align: center;"><a href="/blog/2015/11/broken-wikis-due-to-php-and-mediawiki/image-0-big.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2015/11/broken-wikis-due-to-php-and-mediawiki/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/79LtQP">Photo</a> by <a href="https://www.flickr.com/people/mothernaturephotos/">Elliot Gilfix</a></small></div>
<p>I was recently tasked with resurrecting an ancient wiki. In this case, a wiki last
updated in 2005, running <a href="https://mediawiki.org/">MediaWiki</a> version 1.5.2, and that needed to get transformed
to something more modern (in this case, version 1.25.3). The old settings and extensions were not important,
but we did want to preserve any content that was made.</p>
<p>The items available to me were a tarball of the mediawiki directory (including the
LocalSettings.php file), and a MySQL dump of the wiki database. To import the items
to the new wiki (which already had been created and was gathering content), an
XML dump needed to be generated. MediaWiki has two simple command-line scripts
to export and import your wiki, named dumpBackup.php and
importDump.php. So it
was simply a matter of getting the wiki up and running enough to run dumpBackup.php.</p>
<p>My first thought was to simply bring the wiki up as it was—all the files were in
place, after all, and specifically designed to read the old version of the schema.
(Because the database scheme changes over time, newer MediaWikis cannot run against
older database dumps.) So I unpacked the MediaWiki directory, and prepared to
resurrect the database.</p>
<p>Rather than MySQL, the distro I was using defaulted to using the freer and
arguably better <a href="https://mariadb.org/">MariaDB</a>, which installed painlessly.</p>
<pre tabindex="0"><code>## Create a quick dummy database:
$ echo 'create database footest' | sudo mysql
## Install the 1.5.2 MediaWiki database into it:
$ cat mysql-acme-wiki.sql | sudo mysql footest
## Sanity test as the output of the above commands is very minimal:
echo 'select count(*) from revision' | sudo mysql footest
count(*)
727977
</code></pre><p>Success! The MariaDB instance was easily able to parse and load the
old MySQL file. The next step was to unpack the old 1.5.2 mediawiki directory
into Apache’s docroot, adjust the LocalSettings.php file to point to the
newly created database, and try and access the wiki. Once all that was done, however, both the
browser and the command-line scripts spat out the same error:</p>
<pre tabindex="0"><code>Parse error: syntax error, unexpected 'Namespace' (T_NAMESPACE),
expecting identifier (T_STRING) in
/var/www/html/wiki/includes/Namespace.php on line 52
</code></pre><p>What is this about? Turns out that some years ago, someone added a class to
MediaWiki with the terrible name of “Namespace”. Years later, PHP finally
caved to user demands and added some non-optimal support for namespaces, which
means that (surprise), “namespace” is now a reserved word. In short, older
versions of MediaWiki cannot run with modern (5.3.0 or greater) versions
of PHP. Amusingly, a web search for this error on DuckDuckGo revealed not
only many people asking about this error and/or offering solutions, but
many results were actual wikis that are currently not working!
Thus, their wiki was working fine one moment, and then PHP was (probably automatically)
upgraded, and now the wiki is dead. But DuckDuckGo is happy to show you
the wiki and its now-single page of output, the error above. :)</p>
<p>There are three groups to blame for this sad situation, as well as
three obvious solutions to the problem. The first group to share the
blame, and the most culpable, is the MediaWiki developers who chose
the word “Namespace” as a class name. As PHP has always had very
non-existent/poor support for packages, namespaces, and scoping, it is
vital that all your PHP variables, class names, etc. are as unique as possible.
To that end, the name of the class was changed at some point<br>
to “MWNamespace”—but the damage has been done. The second group to share the
blame is the PHP developers, both for not having namespace support for
so long, and for making it into a reserved word full knowing that one of
the poster children for “mature” PHP apps, MediaWiki, was using “namespace”.
Still, we cannot blame them too much for picking what is a pretty obvious
word choice. The third group to blame is the owners of all those wikis
out there that are suffering that syntax error. They ought to be repairing their
wikis. The fixes are pretty simple, which leads us to the three solutions to the problem.</p>
<div class="separator" style="clear: both; padding: 0em 0em 2em 2em; float:right; text-align: center;"><a href="/blog/2015/11/broken-wikis-due-to-php-and-mediawiki/image-1-big.png" id="gtsm.com/mediawiki_flower.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2015/11/broken-wikis-due-to-php-and-mediawiki/image-1.png"/></a><br/><small>MediaWiki’s cool install image</small></div>
<p>The quickest (and arguably worst) solution is to downgrade PHP to
something older than 5.3. At that point, the wiki will probably work
again. Unless it’s a museum (static) wiki, and you do not intend to
upgrade anything on the server ever again, this solution will not
work long term. The second solution is to upgrade your MediaWiki! The upgrade process is actually very
robust and works well even for very old versions of MediaWiki (as
we shall see below). The third solution is to make some quick edits
to the code to replace all uses of “Namespace” with “MWNamespace”.
Not a good solution, but ideal when you just need to get the wiki up
and running. Thus, it’s the solution I tried for the original problem.</p>
<p>However, once I solved the Namespace problem by renaming to MWNamespace,
some other problems popped up. I will not run through them here—although they were
small and quickly solved, it began to feel like a neverending whack-a-mole
game, and I decided to cut the Gordian knot with a completely different
approach.</p>
<p>As mentioned, MediaWiki has an upgrade process, which means that
you can install the software and it will, in theory, transform your
database schema and data to the new version. However, version
1.5 of MediaWiki was released in October 2005, almost exactly
10 years ago from the current release (1.25.3 as of this writing).
Ten years is a really, really long time on the Internet.
Could MediaWiki really convert something that old? (spoilers: yes!).
Only one way to find out. First, I prepared the old database for the upgrade.
Note that all of this was done on a private local machine where security was not
an issue.</p>
<pre tabindex="0"><code>## As before, install mariadb and import into the 'footest' database
$ echo 'create database footest' | sudo mysql test
$ cat mysql-acme-wiki.sql | sudo mysql footest
$ echo "set password for 'root'@'localhost' = password('foobar')" | sudo mysql test
</code></pre><p>Next, I grabbed the latest version of MediaWiki, verified it, put it in place, and
started up the webserver:</p>
<pre tabindex="0"><code>$ wget http://releases.wikimedia.org/mediawiki/1.25/mediawiki-1.25.3.tar.gz
$ wget http://releases.wikimedia.org/mediawiki/1.25/mediawiki-1.25.3.tar.gz.sig
$ gpg --verify mediawiki-1.25.3.tar.gz.sig
gpg: assuming signed data in `mediawiki-1.25.3.tar.gz'
gpg: Signature made Fri 16 Oct 2015 01:09:35 PM EDT using RSA key ID 23107F8A
gpg: Good signature from "Chad Horohoe <chad@wikimedia.org>"
gpg: aka "keybase.io/demon <demon@keybase.io>"
gpg: aka "Chad Horohoe (Personal e-mail) <innocentkiller@gmail.com>"
gpg: aka "Chad Horohoe (Alias for existing email) <chadh@wikimedia.org>"
## Chad's cool. Ignore the below.
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A</chadh@wikimedia.org></innocentkiller@gmail.com></demon@keybase.io></chad@wikimedia.org>
$ tar xvfz mediawiki-1.25.3.tar.gz
$ mv mediawiki-1.25.3 /var/www/html/
$ cd /var/www/html/mediawiki-1.25.3
## Because "composer" is a really terrible idea:
$ git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git
$ sudo service httpd start
</code></pre><p>Now, we can call up the web page to install MediaWiki.</p>
<ul>
<li>Visit http://localhost/mediawiki-1.25.3, see the familiar yellow flower</li>
<li>Click “set up the wiki”</li>
<li>Click next until you find “Database name”, and set to “footest”</li>
<li>Set the “Database password:” to “foobar”</li>
<li>Aha! Looks what shows up: “Upgrade existing installation” and “There are MediaWiki tables in this database. To upgrade them to MediaWiki 1.25.3, click Continue”</li>
</ul>
<p>It worked! Next messages are: “Upgrade complete. You can now start using your wiki. If you want to regenerate your LocalSettings.php file, click the button below. This is not recommended unless you are having problems with your wiki.” That message is a little misleading. You almost certainly <em>do</em> want to generate a new LocalSettings.php file when doing an upgrade like this. So say yes, leave the database choices as they are, and name your wiki something easily greppable like “ABCD”. Create an admin account, save the generated LocalSettings.php file, and move it to your mediawiki directory.</p>
<p>At this point, we can do what we came here for: generate a XML dump of the wiki content in the database, so we can import it somewhere else.
We only wanted the actual content, and did not want to worry about the history of the pages, so the command was:</p>
<pre tabindex="0"><code>$ php maintenance/dumpBackup.php --current > acme.wiki.2005.xml
</code></pre><p>It ran without a hitch. However, close examination showed that it had an amazing amount of unwanted stuff from the
“MediaWiki:” namespace. While there are probably some clever solutions that could be devised to cut them out of the
XML file (either on export, import, or in between), sometimes quick beats clever, and I simply opened the file in an
editor and removed all the “page” sections with a title beginning with “MediaWiki:”. Finally, the file was shipped
to the production wiki running 1.25.3, and the old content was added in a snap:</p>
<pre tabindex="0"><code>$ php maintenance/importDump.php acme.wiki.2005.xml
</code></pre><p>The script will recommend rebuilding the “Recent changes” page by running rebuildrecentchanges.php (can we
get consistentCaps please MW devs?). However, this data is at least 10 years old, and Recent changes only goes back
90 days by default in version 1.25.3 (and even shorter in previous versions). So, one final step:</p>
<pre tabindex="0"><code>## 20 years should be sufficient
$ echo '$wgRCMAxAge = 20 * 365 * 24 * 3600;' >> LocalSettings.php
$ php maintenance/rebuildrecentchanges.php
</code></pre><p>Voila! All of the data from this ancient wiki is now in place on a modern wiki!</p>
MediaWiki extension.json change in 1.25https://www.endpointdev.com/blog/2015/10/mediawiki-extensionjson-change-in-125/2015-10-17T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2015/10/mediawiki-extensionjson-change-in-125/image-0-big.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2015/10/mediawiki-extensionjson-change-in-125/image-0.png"/></a></div>
<p>I recently released a new version of the
<a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a>
<a href="https://www.mediawiki.org/wiki/Extension:RequestTracker">“Request Tracker” extension</a>, which provides
a nice interface to your
<a href="https://bestpractical.com/rt/">RequestTracker</a> instance, allowing you to view the tickets right
inside of your wiki. There are two major changes I want to point out. First, the name has
changed from <strong>“RT”</strong> to <strong>“RequestTracker”</strong>. Second, it is using the brand-new way of writing
MediaWiki extensions, featuring the extension.json file.</p>
<p>The name change rationale is easy to understand: I wanted it to be more intuitive and easier to find. A search for
“RT” on mediawiki.org ends up finding references to the WikiMedia RequestTracker system,
while a
<a href="https://www.mediawiki.org/w/index.php?search=RequestTracker">search for “RequestTracker”</a>
finds the new extension right away. Also, the name was
too short and failed to indicate to people what it was. The “rt” tag used by the extension stays
the same. However, to produce a table showing all open tickets for user “alois”, you still write:</p>
<pre tabindex="0"><code><rt u='alois'></rt>
</code></pre><p>The other major change was to modernize it. As of version 1.25 of MediaWiki,
extensions are encouraged to use a new system to register themselves with MediaWiki. Previously,
an extension would have a PHP file named after the extension that was responsible for doing
the registration and setup—usually by mucking with global variables! There was no
way for MediaWiki to figure out what the extension was going to do without parsing the entire file, and
thereby activating the extension. The new method relies on a standard JSON file called
extension.json. Thus, in the RequestTracker extension, the file RequestTracker.php has
been replaced with the much smaller and simpler extension.json file.</p>
<p>Before going further, it should be pointed out that this is a big change for extensions,
and was not without controversy. However, as of MediaWiki 1.25 it is the new standard for extensions, and I think
the project is better for it. The old way will continue to be supported, but extension
authors should be using extension.json for new extensions, and converting existing
ones over. As an aside, this is another indication that <a href="http://json.org/">JSON</a> has won the data format war.
Sorry, <a href="http://www.json.org/xml.html">XML</a>, you were too big and bloated. Nice try <a href="http://yaml.org/">YAML</a>, but you were a little <em>too</em> free-form. JSON isn’t perfect,
but it is the best solution of its kind. For further evidence, see Postgres,
which now has <a href="https://www.postgresql.org/docs/current/static/datatype-json.html">outstanding support for JSON and JSONB</a>. I added support for YAML output to EXPLAIN
in Postgres some years back, but nobody (including me!) was excited enough about YAML to do
more than that with it. :)</p>
<p>The extension.json file asks you to fill in some standard metadata fields about the extension, which are then used by MediaWiki to register and set up the extension. Another advantage
of doing it this way is that you no longer need to add a bunch of ugly include_once()
function calls to your LocalSettings.php file. Now, you simply call the name of the extension as an argument to the wfLoadExtension() function. You can even load multiple extensions at once with wfLoadExtensions():</p>
<pre tabindex="0"><code>## Old way:
require_once("$IP/extensions/RequestTracker/RequestTracker.php");
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
## New way:
wfLoadExtension( 'RequestTracker' );
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
## Or even load three extensions at once:
wfLoadExtensions( array( 'RequestTracker', 'Balloons', 'WikiEditor' ) );
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
</code></pre><p>Note that configuration changes specific to the extension still must be defined in
the LocalSettings.php file.</p>
<p>So what should go into the extension.json file? The
<a href="https://www.mediawiki.org/wiki/Manual:Developing_extensions">extension development documentation</a> has some suggested
fields, and you can also view the <a href="https://phabricator.wikimedia.org/diffusion/MW/browse/master/docs/extension.schema.v2.json">canonical extension.json schema</a>. Let’s take a quick look at the RequestTracker/extension.json file. Don’t worry, it’s not
too long.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl">{
<span style="color:#d20;background-color:#fff0f0">"manifest_version"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>,
<span style="color:#d20;background-color:#fff0f0">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"parserhook"</span>,
<span style="color:#d20;background-color:#fff0f0">"author"</span>: [
<span style="color:#d20;background-color:#fff0f0">"Greg Sabino Mullane"</span>
],
<span style="color:#d20;background-color:#fff0f0">"version"</span>: <span style="color:#d20;background-color:#fff0f0">"2.0"</span>,
<span style="color:#d20;background-color:#fff0f0">"url"</span>: <span style="color:#d20;background-color:#fff0f0">"https://www.mediawiki.org/wiki/Extension:RequestTracker"</span>,
<span style="color:#d20;background-color:#fff0f0">"descriptionmsg"</span>: <span style="color:#d20;background-color:#fff0f0">"rt-desc"</span>,
<span style="color:#d20;background-color:#fff0f0">"license-name"</span>: <span style="color:#d20;background-color:#fff0f0">"PostgreSQL"</span>,
<span style="color:#d20;background-color:#fff0f0">"requires"</span> : {
<span style="color:#d20;background-color:#fff0f0">"MediaWiki"</span>: <span style="color:#d20;background-color:#fff0f0">">= 1.25.0"</span>
},
<span style="color:#d20;background-color:#fff0f0">"AutoloadClasses"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>: <span style="color:#d20;background-color:#fff0f0">"RequestTracker_body.php"</span>
},
<span style="color:#d20;background-color:#fff0f0">"Hooks"</span>: {
<span style="color:#d20;background-color:#fff0f0">"ParserFirstCallInit"</span> : [
<span style="color:#d20;background-color:#fff0f0">"RequestTracker::wfRequestTrackerParserInit"</span>
]
},
<span style="color:#d20;background-color:#fff0f0">"MessagesDirs"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>: [
<span style="color:#d20;background-color:#fff0f0">"i18n"</span>
]
},
<span style="color:#d20;background-color:#fff0f0">"config"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_URL"</span>: <span style="color:#d20;background-color:#fff0f0">"http://rt.example.com/Ticket/Display.html?id"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_DBconn"</span>: <span style="color:#d20;background-color:#fff0f0">"user=rt dbname=rt"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Formats"</span>: [],
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Cachepage"</span>: <span style="color:#00d;font-weight:bold">0</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Useballoons"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Active"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Sortable"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_LASTUPDATED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_LASTUPDATED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_CREATED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_CREATED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_RESOLVED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_RESOLVED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_NOW"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>
}
}
</code></pre></div><p>The first field in the file is manifest_version, and simply indicates the extension.json schema version. Right now it is marked as required, and I figure it does no harm to throw it in there. The name field should be self-explanatory, and should match your CamelCase extension name, which will also be the subdirectory where your extension will live under the extensions/ directory.
The type field simply tells what kind of extension this is, and is mostly used to determine which section of the Special:Version page an extension will appear under. The author is also self-explanatory, but note that this is a JSON array, allowing for multiple items if needed. The version and url are highly recommended. For the license, I chose the dirt-simple <a href="https://opensource.org/licenses/postgresql">PostgreSQL license</a>, whose only fault is its name. The descriptionmsg is what will appear as the description of the extension on the Special:Version page. As it is a user-facing text, it is subject to
internationalization, and thus <strong>rt-desc</strong> is converted to your current language by looking up the language file inside of the extension’s i18n directory.</p>
<p>The requires field only supports a “MediaWiki” subkey at the moment. In this case, I have it
set to require at least version 1.25 of MediaWiki—as anything lower will not even be able to read
this file! The AutoloadClasses key is the new way of loading code needed by the extension. As before, this should be stored in a php file with the name of the extension, an underscore, and the word “body” (e.g. RequestTracker_body.php). This file contains all of the functions that perform the actual work of the extension.</p>
<p>The Hooks field is one of the big advantages of the new extension.json format. Rather than
worrying about modifying global variables, you can simply let MediaWiki know what functions
are associated with which hooks. In the case of RequestTracker, we need to do some magic whenever
a <strong><rt></strong> tag is encountered. To that end, we need to instruct the parser that we will be handling
any <strong><rt></strong> tags it encounters, and also tell it what to do when it finds them. Those details
are inside the wfRequestTrackerParserInit function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">function</span> <span style="color:#06b;font-weight:bold">wfRequestTrackerParserInit</span>( Parser <span style="color:#369">$parser</span> ) {
<span style="color:#369">$parser</span>-><span style="color:#369">setHook</span>( <span style="color:#d20;background-color:#fff0f0">'rt'</span>, <span style="color:#d20;background-color:#fff0f0">'RequestTracker::wfRequestTrackerRender'</span> );
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">true</span>;
}
</code></pre></div><p>The config field provides a list of all user-configurable variables used by the extension, along with their default values.</p>
<p>The MessagesDirs field tells MediaWiki where to find your localization files. This
should always be in the standard place, the i18n directory.
Inside that directory are localization files, one for each language, as well as a special
file named qqq.json, which gives information about each message
string as a guide to translators. The language files are of the format “xxx.json”, where
“xxx” is the language code. For example, RequestTracker/i18n/en.json
contains English versions of all the messages used by the extension. The i18n files look like this:</p>
<pre tabindex="0"><code>$ cat en.json
{
"rt-desc" : "Fancy interface to RequestTracker using <code>&lt;rt&gt;</code> tag",
"rt-inactive" : "The RequestTracker extension is not active",
"rt-badcontent" : "Invalid content args: must be a simple word. You tried: <b>$1</b>",
"rt-badquery" : "The RequestTracker extension encountered an error when talking to the RequestTracker database",
"rt-badlimit" : "Invalid LIMIT (l) arg: must be a number. You tried: <b>$1</b>",
"rt-badorderby" : "Invalid ORDER BY (ob) arg: must be a standard field (see documentation). You tried: <b>$1</b>",
"rt-badstatus" : "Invalid status (s) arg: must be a standard field (see documentation). You tried: <b>$1</b>",
"rt-badcfield" : "Invalid custom field arg: must be a simple word. You tried: <b>$1</b>",
"rt-badqueue" : "Invalid queue (q) arg: must be a simple word. You tried: <b>$1</b>",
"rt-badowner" : "Invalid owner (o) arg: must be a valud username. You tried: <b>$1</b>",
"rt-nomatches" : "No matching RequestTracker tickets were found"
}
$ cat fr.json
{
"@metadata": {
"authors": [
"Josh Tolley"
]
},
"rt-desc" : "Interface sophistiquée de RequestTracker avec l'élement <code>&lt;rt&gt;</code>.",
"rt-inactive" : "Le module RequestTracker n'est pas actif.",
"rt-badcontent" : "Paramètre de contenu « $1 » est invalide: cela doit être un mot simple.",
"rt-badquery" : "Le module RequestTracker ne peut pas contacter sa base de données.",
"rt-badlimit" : "Paramètre à LIMIT (l) « $1 » est invalide: cela doit être un nombre entier.",
"rt-badorderby : "Paramètre à ORDER BY (ob) « $1 » est invalide: cela doit être un champs standard. Voir le manuel utilisateur.",
"rt-badstatus" : "Paramètre de status (s) « $1 » est invalide: cela doit être un champs standard. Voir le manuel utilisateur.",
"rt-badcfield" : "Paramètre de champs personalisé « $1 » est invalide: cela doit être un mot simple.",
"rt-badqueue" : "Paramètre de queue (q) « $1 » est invalide: cela doit être un mot simple.",
"rt-badowner" : "Paramètre de propriétaire (o) « $1 » est invalide: cela doit être un mot simple.",
"rt-nomatches" : "Aucun ticket trouvé"
}
</code></pre><p>One other small change I made to the extension was to allow both ticket numbers and queue names
to be used inside of the tag. To view a specific ticket, one was always able to do this:</p>
<pre tabindex="0"><code><rt>6567</rt>
</code></pre><p>This would produce the text “RT #6567”, with information on the ticket available on mouseover,
and hyperlinked to the ticket inside of RT. However, I often found myself using this extension
to view all the open tickets in a certain queue like this:</p>
<pre tabindex="0"><code><rt q="dyson"></rt>
</code></pre><p>It seems easier to simply add the queue name inside the tags, so in this new version
one can simply do this:</p>
<pre tabindex="0"><code><rt>dyson</rt>
</code></pre><p>If you are running MediaWiki 1.25 or better, try out the new RequestTracker extension! If
you are stuck on an older version, use the RT extension and upgrade as soon as you can. :)</p>
MediaWiki complete test wiki via cloninghttps://www.endpointdev.com/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/2015-06-06T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; padding-left: 2em; padding-bottom: 1em; text-align: center;"><a href="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-0-big.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" id="jA0EAgMCshiJ2sWFNg5gyVkDpsZvh+IoZdIClG04/TSA8gqRN8ct" src="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/eavCXg">Liska</a> by <a href="https://www.flickr.com/photos/tambako/">Tambako The Jaguar</a></small></div>
<p>Being able to create a quick copy of your <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> site is an important skill that has many benefits. Any time you are testing an upgrade, whether major or <a href="/blog/2014/10/mediawiki-minor-upgrade-with-patches/">minor</a>, it is great to be able to perform the upgrade on a test site first. Tracking down bugs becomes a lot easier when you can add all the debugging statements you need and not worry about affecting any of the users of your wiki. Creating and modifying extensions also goes a lot smoother when you can work with an identical copy of your production wiki. I will outline the steps I use to create such a copy, also known as a “test wiki”.</p>
<p>Before creating a copy, there are two things that should be done to an existing MediaWiki installation: <a href="http://sixrevisions.com/resources/git-tutorials-beginners/">use git</a>, and move the <strong>images</strong> directory. By “use git”, I mean to put your existing mediawiki directory (e.g. where your LocalSettings.php file lives) into version control. Because the MediaWiki software is not that large, it is simplest to just add nearly everything into git, with the exception of the images and the cache information. Here is a recipe to do just that:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ cd /var/www/mediawiki
$ git init .
Initialized empty Git repository in /var/www/mediawiki/.git/</span>
$ echo /cache/ >> .gitignore
$ echo /images/ >> .gitignore
$ git add --force .
$ git commit -a -m "Initial MediaWiki commit, version 1.24"
[master (root-commit) bd7db2b] Initial MediaWiki commit, version 1.24
10024 files changed, 1910576 insertions(+)
create mode 100644 .gitignore
...
</code></pre></div><p>Replace that commit message with your specific version, of course, or whatever you like, although I highly recommend your git commits always mention the version on major changes.</p>
<p>The second thing that should be done is to move the images directory and use a symlink to the new location. The “images” directory in MediaWiki is special in many ways. It is the only directory (except “cache”) directly written by MediaWiki (all other changes are stored in the database, not on disk). It is the only directory that comes pre-populated in the MediaWiki tarballs and is always a pain to upgrade. Finally, it invariably contains a large collection of static files that are not well suited for version control, and are usually better backed up and stored in ways different than the rest of MediaWiki. For all these reasons, I recommend making images into a symlink. The simplest recipe is to just move the images directory “up a level”. This will also help us below when cloning the wiki.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#038">cd</span> /var/www/mediawiki
$ mv images ..
$ ln -s ../images .
</code></pre></div><p>Now that those two important prerequisites are out of the way, let’s get a quick overview of the steps to create a clone of your wiki:</p>
<ul>
<li>Make a backup of your existing wiki (files and database)</li>
<li>Make a copy of your database</li>
<li>Create a new directory, and copy the mediawiki files into it</li>
<li>Create a new git branch</li>
<li>Adjust the LocalSettings.php file</li>
<li>Mark it clearly as a test wiki</li>
<li>Do a git commit</li>
<li>Adjust your web server</li>
</ul>
<p>The first step is to make a backup of your existing wiki. You can never have too many backups, and right before you go copying a lot of files is a great time to create one.
Before backing up, make sure everything is up to date in git with “git status”. Make a backup of the mediawiki directory, for example with tar, making sure the resulting backup file is well labeled:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ tar cfz /backups/mediawiki.backup.20150601.tar.gz --exclude=mediawiki/cache --anchored mediawiki/
</code></pre></div><p>If your images directory is somewhere else, make sure you back that up as well.
Backing up your database is dead simple if you are using Postgres:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ pg_dump wikidb | gzip --fast > /backups/mediawiki.database.backup.wikidb.20150601.pg.gz
</code></pre></div><p>The next step is to create a new copy of the database for your cloned wiki to access:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dropdb test_wikidb
$ createdb -T wikidb test_wikidb
$ psql test_wikidb -c <span style="color:#d20;background-color:#fff0f0">'alter database test_wikidb owner to wikiuser'</span>
</code></pre></div><p>Now we want to create a new directory for the cloned wiki, and populate it with files from the production wiki. For this example, the existing production wiki lives in /var/www/mediawiki, and the new cloned test wiki will live in /var/www/test_mediawiki.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#038">cd</span> /var/www
$ mkdir test_mediawiki
$ rsync -a -W --exclude=/images/ mediawiki/ test_mediawiki/
<span style="color:#888">## rsync will copy symlinks as well - such as the images directory!</span>
</code></pre></div><p>I like to create a new git branch right away, to avoid any confusion with the “actual” git
repository in the production mediawiki directory. If you do end up making any changes in the test directory, it’s easy enough to roll them into the other git repo.
Branch names should be short and clearly indicate why you have created this copy of the wiki. Doing this means the name shows up as the first line whenever you do a “git status”, which is nice.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ cd /var/www/test_mediawiki
$ git checkout -b testing_version_1.25.2
Switched to a new branch 'testing_version_1.25.2'
</code></pre></div><p>The next step is critical: editing the LocalSettings.php file!
As this was copied from the production wiki, we need to make sure it points back to itself via paths, and that it connects to our newly created database. Add all these to the bottom of your test_mediawiki/LocalSettings.php file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#888">## Change important paths:
</span><span style="color:#888"></span><span style="color:#369">$wgArticlePath</span> = <span style="color:#d20;background-color:#fff0f0">'/testwiki/$1'</span>;
<span style="color:#369">$wgScriptPath</span> = <span style="color:#d20;background-color:#fff0f0">'/test_mediawiki'</span>;
<span style="color:#888">## Point to the correct database:
</span><span style="color:#888"></span><span style="color:#369">$wgDBname</span> = <span style="color:#d20;background-color:#fff0f0">'test_wikidb'</span>;
<span style="color:#888">## The logo may be hardcoded, so:
</span><span style="color:#888"></span><span style="color:#369">$wgLogo</span> = <span style="color:#d20;background-color:#fff0f0">'/test_mediawiki/End_Point_logo.png'</span>;
<span style="color:#888">## Disable all email notifications:
</span><span style="color:#888"></span><span style="color:#369">$wgUsersNotifiedOnAllChanges</span> = <span style="color:#080;font-weight:bold">array</span>();
</code></pre></div><p>It’s also a good idea to make this wiki read-only until needed. Also important if you symlinked the images directory is to disallow any uploads. If you need to enable uploads, and thus writes to the images directory, make sure you remove the symlink and create a new images directory! You can either copy all the files from the production wiki, or simply leave it empty and expect to see a lot of “missing file” errors, which can safely be ignored.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$wgReadOnly</span> = <span style="color:#d20;background-color:#fff0f0">'Test wiki: upgrading to MediaWiki 1.25.2'</span>;
<span style="color:#369">$wgEnableUploads</span> = <span style="color:#080;font-weight:bold">false</span>;
</code></pre></div><p>The $wgReadOnly message will appear when people try to edit a page, but we want to make it very visible to all users so as soon as they see the wiki that “here be Danger” (and edits will be lost). To that end, there are four additional steps you can take. First, you can set a sitewide message. This will appear near the top of every page. You can add HTML to this, and it is set in your LocalSettings.php file as $wgSiteNotice. You can also change the $wgSiteName parameter, which will appear in the title of every page.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$wgSiteNotice</span> = <span style="color:#d20;background-color:#fff0f0">'<strong>TEST WIKI ONLY!</strong>'</span>;
<span style="color:#369">$wgSitename</span> = <span style="color:#d20;background-color:#fff0f0">'TEST WIKI'</span>;
</code></pre></div><p>The third additional step is to change the CSS of every page. I use this to slightly change the background color of every page. This requires that the $wgUseSiteCss setting is enabled. It is on by default, but there is no harm setting it to true explicitly.
Getting it to work on all pages, including the login page, requires enabling $wgAllowSiteCSSOnRestrictedPages as well.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#369">$wgUseSiteCss</span> = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#369">$wgAllowSiteCSSOnRestrictedPages</span> = <span style="color:#080;font-weight:bold">true</span>;
</code></pre></div><p>Once the above is done, navigate to <code>MediaWiki:Common.css</code> and add the text below. Note that you may need to wait until “making the wiki active” step below—and comment out the $wgReadOnly variable.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css">* { <span style="color:#080;font-weight:bold">background-color</span>: <span style="color:#00d;font-weight:bold">#ddeeff</span> <span style="color:#c00;font-weight:bold">!important</span> }
</code></pre></div><p>The last method to mark the wiki as test only is to change the wiki logo. You can replace it with a custom image, or you can modify the existing logo. I like the latter approach.
Annotating text is easy from the command line by using ImageMagick. Use the “polaroid” feature to give it a nice effect (use “-polaroid 0” to avoid the neat little rotation). The command and the
result:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ convert End_Point.logo.png -caption <span style="color:#d20;background-color:#fff0f0">"TEST WIKI ONLY\!"</span> -gravity center -polaroid <span style="color:#00d;font-weight:bold">20</span> End_Point.tilted.testonly.png
</code></pre></div><table border="1">
<tbody><tr><td><a href="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-1.png" imageanchor="1"><img border="0" id="DpGxzTyk3O8za1Aig6jaLjob/evsLLg9TQpoLlnHUm2xImQk" src="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-1.png"/></a></td>
<td><a href="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-2.png" imageanchor="1"><img border="0" id="v/+S6fL1Nz6M3/jqUNUM8l4Pub/Ewg44E61hvYvcxQ===D/fO" src="/blog/2015/06/mediawiki-complete-test-wiki-via-cloning/image-2.png"/></a></td></tr>
<tr><th>Original</th><th>Captioned</th></tr>
</tbody></table>
<p>At this point, all of the changes to the test wiki are complete, so we/you should commit all your changes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git commit -a -m "Changes for the test wiki"
</code></pre></div><p>The final step is to make your test wiki active by adjusting your web server. Generally this is easy and basically means copying the existing wiki parameters. For Apache, it can be as simple as adding a new Alias directive to your <code>httpd.conf</code> file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Alias /testwiki /var/www/test_mediawiki/index.php
</code></pre></div><p>Reload the web server, and Bob’s your uncle. You now have a fully functional, safely sandboxed, magnificently marked-up copy of your production wiki. The above may seem like a lot of work, but this was an overly-detailed post—the actual work only takes around 10 minutes, or much less if you script it!</p>
Prevent MediaWiki showing PHP version with new extension: ControlSpecialVersionhttps://www.endpointdev.com/blog/2014/10/prevent-mediawiki-showing-php-version/2014-10-29T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; text-align: center;"><a href="/blog/2014/10/prevent-mediawiki-showing-php-version/image-0-big.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2014/10/prevent-mediawiki-showing-php-version/image-0.png"/></a><br/>Sok Kwu Wan</div>
<p>I recently created a new <a href="https://www.mediawiki.org/wiki/Manual:Extensions">MediaWiki extension</a> named <strong>ControlSpecialVersion</strong> whose purpose is to allow some control over what is shown on MediaWiki’s “special” page Special:Version. The latest version of this extension can be <a href="https://www.mediawiki.org/wiki/Extension:ControlSpecialVersion">downloaded from Mediawiki.org</a>.
The primary purpose of the module is to prevent showing the PHP and database versions to the public.</p>
<p>As with most MediaWiki extensions, installation is easy: download the tarball, unzip it into your <strong>extensions</strong> directory, and add this line to your LocalSettings.php file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">require_once( "$IP/extensions/ControlSpecialVersion/ControlSpecialVersion.php" );
</code></pre></div><p>By default, the extension removes the PHP version information from the page. It also changes the PostgreSQL reported version from its revision to simply the major version, and changes the name from the terrible-but-official “PostgreSQL” to the widely-accepted “Postgres”. Here is what the <strong>Software</strong> section of bucardo.org looks like before and after the extension is used:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/10/prevent-mediawiki-showing-php-version/image-1-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/10/prevent-mediawiki-showing-php-version/image-1.png"/></a></div>
<div class="separator" style="clear: both; text-align: center; padding-bottom: 1em"><a href="/blog/2014/10/prevent-mediawiki-showing-php-version/image-2-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/10/prevent-mediawiki-showing-php-version/image-2.png"/></a></div>
<p>Note that we are also eliding the git revision information (sha and date). You can also do things such as hide the revision information from the extension list, remove the versions entirely, or even remove an extension from showing up at all. All the configuration parameters can be found on the <a href="https://www.mediawiki.org/wiki/Extension:ControlSpecialVersion">extension’s page on mediawiki.org</a>.</p>
<p>It should be noted that there are typically two other places in which your PHP version may be exposed, both in the HTTP headers. If you are running Apache, it may show the version as part of the <strong>Server</strong> heading. To turn this off, edit you httpd.conf file and change the <strong>ServerTokens</strong> directive to <strong>ProductOnly</strong>. The other header is known as <strong>X-Powered-By</strong> and is added by PHP to any pages it serves (e.g. MediaWiki pages). To disable this header, edit your php.ini file and make sure <strong>expose_php</strong> is set to <strong>Off</strong>.</p>
<p>While these methods may or may not make your server safer, there really is no reason to expose certain information to the world. With this extension, you at least have the choice now.</p>
MediaWiki minor upgrade with patcheshttps://www.endpointdev.com/blog/2014/10/mediawiki-minor-upgrade-with-patches/2014-10-02T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; text-align: center;"><a href="/blog/2014/10/mediawiki-minor-upgrade-with-patches/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2014/10/mediawiki-minor-upgrade-with-patches/image-0.jpeg"/></a><br/>
<small><a href="https://flic.kr/p/bTzeNx">Photo</a> by <a href="https://www.flickr.com/photos/31246066@N04/">Ian Sane</a></small></div>
<p>One of the more mundane (but important!) tasks for those running <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> is keeping it updated with the latest version of the software. This is usually a fairly easy process. While the <a href="http://www.mediawiki.org/wiki/Manual:Upgrading">offical upgrade instructions</a> for MediaWiki are good, they are missing some important items. I will lay out in detail what we do to upgrade MediaWiki installations.</p>
<p>Note that this is for “minor” upgrades to MediaWiki, where minor is defined as not moving more than a couple of actual versions, and not requiring anything other than patching some files. I will cover major upgrades in a future post. For this article, I assume you have full shell access, and not simply FTP, to the server that MediaWiki is running on.</p>
<p>The first step in upgrading is knowing when to upgrade - in other words, making sure you know about new releases. The best way to do this is to subscribe to the low-volume <a href="https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce">mediawiki-announce mailing list</a>. The MediaWiki maintainers have a wonderful new policy of sending out “pre-announcement” emails stating the exact time that the new version will be released. Once we see that announcement, or when the version is actually released, we open a support ticket, which serves the dual purpose of making sure the upgrade does not get forgotten about, and of keeping an official record of the upgrade.</p>
<p>The official announcement should mention the location of a patch tarball, for example
<strong><a href="http://releases.wikimedia.org/mediawiki/1.23/mediawiki-1.23.5.patch.gz">http://releases.wikimedia.org/mediawiki/1.23/mediawiki-1.23.5.patch.gz</a></strong>. If not, you can find the patches in the directory at <strong><a href="http://releases.wikimedia.org/mediawiki/">http://releases.wikimedia.org/mediawiki/</a></strong>: look for your version, and the relevant patch. Download the patch, and grab the signature file as well, which will be the same file with “dot sig” appended to it. In the example above, the sig file would be <strong><a href="http://releases.wikimedia.org/mediawiki/1.23/mediawiki-1.23.5.patch.gz.sig">http://releases.wikimedia.org/mediawiki/1.23/mediawiki-1.23.5.patch.gz.sig</a></strong>.</p>
<p>It is important to know that these patch files <em>only</em> cover patching from the previous version. If you are running version 1.23.2, for example, you would need to download and apply the patches for versions 1.23.3 and 1.23.4, before tackling version 1.23.5. You can also create your own patch file by checking out the MediaWiki git repository and using the version tags. In the previous example, you could run “git diff 1.23.2 1.23.5”.</p>
<p>Once the patch is downloaded, I like to give it three sanity checks before installing it. First, is the <a href="http://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a> signature valid? Second, does this patch look sane? Third, does the patch match what is in the official git repository for MediaWiki?</p>
<p>To check the PGP signature, you use the sig file, which is a small external signature that one of the MediaWiki maintainers has generated for the patch itself. Since you may not have the public PGP key already, you should both verify the file and ask <a href="http://en.wikipedia.org/wiki/GNU_Privacy_Guard">gpg</a> to download the needed public key in one step. Here’s what it looks like when you do:</p>
<pre tabindex="0"><code>$ gpg --keyserver pgp.mit.edu --keyserver-options auto-key-retrieve --verify mediawiki‑1.23.5.patch.gz.sig
gpg: Signature made Wed 01 Oct 2014 06:21:47 PM EDT using RSA key ID 5DC00AA7
gpg: requesting key 5DC00AA7 from hkp server pgp.mit.edu
gpg: key 5DC00AA7: public key "Markus Glaser <glaser@hallowelt.biz>" imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 5 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 5u
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
gpg: Good signature from "Markus Glaser <glaser@hallowelt.biz>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 280D B784 5A1D CAC9 2BB5 A00A 946B 0256 5DC0 0AA7
</glaser@hallowelt.biz></glaser@hallowelt.biz>
</code></pre><p>The important line here is the one saying “Good signature”. The usage of gpg and PGP is beyond the scope of this article, but feel free to ask questions in the comments. Once verified, the next step is to make sure the patch looks sane. In other words, read through it and see exactly what it does! It helps to read <a href="https://git.wikimedia.org/blob/mediawiki%2Fcore.git/REL1_23/RELEASE-NOTES-1.23">the release notes</a> right before you do this. Then:</p>
<pre tabindex="0"><code>$ gunzip -c mediawiki-1.23.5.patch.gz | more
</code></pre><p>While reading through, make note of any files that have been locally patched - you will need to check on them later. If you are not used to reading diff outputs, this may be a little confusing, but give it a shot anyway, so you know what you are patching. Most MediaWiki version upgrades are very small patches, and only alter a few items across a few files. Once that is done, the final sanity check is to make sure this patch matches what it in the canonical MediaWiki git repository.</p>
<p>This is actually a fairly tricky task, as it turns out the patch files are generated from a custom script, and are not just the output of “git diff old_version new_version”. Feel free to skip ahead, this is one method I found for making sure the patch file and the git diff match up. By “git diff”, I mean the output of “git diff 1.23.4 1.23.5”, for example. The biggest problem is that the files are ordered differently. Thus, even if you remove all but the actual diff portions, you cannot easily compare them. Here, “patchfile” is the downloaded and gunzipped patch file, e.g. mediawiki-1.23.5.patch, and “gitfile” is the output of git diff across two different versions, e.g. the output of “git diff 1.23.4 1.23.5”. First, we want to ensure that they both have the same group of files being diffed. Then we walk through each file in the order given by the patchfile, and generate a cross-tag diff. This is saved to a file, and then compared to the original patchfile. They will not be identical, but should match up for the actual diff portions of the file.</p>
<pre tabindex="0"><code>## The -f42 may change from version to version
$ diff -s <(grep diff patchfile | cut -d' ' -f42 | cut -d/ -f2- | sort) <( grep diff gitfile | cut -d' ' -f4 | cut -d/ -f2- | sort)
Files /dev/fd/63 and /dev/fd/62 are identical
$ grep diff patchfile | cut -d' ' -f24 | cut -d/ -f2- | grep -v RELEASE | xargs -L1 git diff 1.23.4 1.23.5 > gitfile2
$ diff -b patchfile gitfile2
</code></pre><p>Okay, we have verified that the patch looks sane. The next step is to make sure your MediaWiki has a clean git status. If you don’t have your MediaWiki in <a href="http://git-scm.com/">git</a>, now is the time to do so. It’s as simple as:</p>
<pre tabindex="0"><code>$ cd /your/wiki/directory
$ echo -ne "images/\ncache/\n" > .gitignore
$ git init
$ git add .
$ git commit -a -q -m "Initial import of our MediaWiki directory"
</code></pre><p>Run “git status” and make sure you don’t have any changed but uncommitted files. Once that is done, you are ready to apply the patch. Gunzip the patch file first, run the actual patch command in dryrun mode first, then do the final patch:</p>
<pre tabindex="0"><code>$ gunzip ~/mediawiki-1.23.5.patch.gz
$ patch -p1 --dry-run -i ~/mediawiki-1.23.5.patch
$ patch -p1 -i ~/mediawiki-1.23.5.patch
</code></pre><p>You may not have the “tests” directory installed, in which case it is safe to skip any missing file errors related to that directory. Just answer “Y” when asked if it is okay to skip that file. Here is an example of an actual patch from MediaWiki 1.23.3 to version 1.23.4:</p>
<pre tabindex="0"><code>$ patch -p1 -i ~/mediawiki-1.23.4.patch
patching file includes/config/GlobalVarConfig.php
patching file includes/db/DatabaseMysqli.php
patching file includes/DefaultSettings.php
patching file includes/libs/XmlTypeCheck.php
patching file includes/Sanitizer.php
patching file includes/upload/UploadBase.php
patching file RELEASE-NOTES-1.23
can't find file to patch at input line 387
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -Nruw -x messages -x '*.png' -x '*.jpg' -x '*.xcf' -x '*.gif' -x '*.svg' -x '*.tiff' -x '*.zip' -x '*.xmp' -x '.git*' mediawiki-1.23.3/tests/phpunit/includes/upload/UploadBaseTest.php mediawiki-1.23.4/tests/phpunit/includes/upload/UploadBaseTest.php
|--- mediawiki-1.23.3/tests/phpunit/includes/upload/UploadBaseTest.php 2014-09-24 19:58:10.961599096 +0000
|+++ mediawiki-1.23.4/tests/phpunit/includes/upload/UploadBaseTest.php 2014-09-24 19:55:15.538575503 +0000
--------------------------
File to patch:
Skip this patch? [y] y
Skipping patch.
2 out of 2 hunks ignored
</code></pre><p>The jump from 1.23.4 to 1.23.5 was much cleaner:</p>
<pre tabindex="0"><code>$ patch -p1 -i ~/mediawiki-1.23.5.patch
patching file includes/DefaultSettings.php
patching file includes/OutputPage.php
patching file RELEASE-NOTES-1.23
</code></pre><p>Once the patch is applied, immediately check everything into git. This keeps the patch separate from other changes in your git history, and allows us to roll back the patch easily if needed. State the version in your commit message:</p>
<pre tabindex="0"><code>$ git commit -a -m "Applied mediawiki-1.23.5.patch to move from version 1.23.4 to 1.23.5"
</code></pre><p>The next step is to run the update script. This almost always does nothing for minor releases, but it’s a good practice to get into. Running it is simple:</p>
<pre tabindex="0"><code>$ php maintenance/update.php --quiet --quick
</code></pre><p>The “quick” option prevents the usual five-second warning. The “quiet” option is supposed to turn off any non-error output, but if you are using <a href="https://semantic-mediawiki.org/">Semantic MediaWiki</a>, you will still receive a screen-full of unwanted output. I need to submit a patch to fix that someday. :)</p>
<p>Now that the new version is installed, make sure the wiki is still working! First, visit the Special:Version page and confirm that the new version number appears. Then make sure you can view a random page, that you can edit a page, and that you can upload an image. Finally, load your extension testing page.</p>
<p>You don’t have an extension testing page? To make one, create a new page named “Extension_testing”. On this page, include as many working examples of your extensions as possible, especially non-standard or heavily-used ones. For each extension, put the name of the extension in a header, describe what the output should be, and then have the extension do something interesting in such a way that a non-working extension will be noticed very quickly when viewing the page!</p>
<p>If you have any locally patched files (we almost always do, especially UserMailer.php!), now is the time to check that the patch did not mess up your local changes. If they did, make adjustments as needed, then make sure to git commit everything.</p>
<p>At this point, your wiki should be up and running the latest version of MediaWiki. Notify the users of the wiki as needed, then close out the support ticket, noting any problems you encountered. Upgrading via patch is a very straightforward procedure, but major upgrades are not! Watch for a future post on that.</p>
Version differences via GitHub from the command linehttps://www.endpointdev.com/blog/2014/07/version-differences-via-github-from/2014-07-09T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float:right; text-align: center;"><a href="/blog/2014/07/version-differences-via-github-from/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2014/07/version-differences-via-github-from/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/7pSEW1">Photo</a> by <a href="https://www.flickr.com/photos/bensonkua/">Benson Kua</a></small></div>
<p>I work with a lot of open source projects, and I use the command-line for almost everything. It often happens that I need to examine a file from a project, and thanks to bash, <a href="https://github.com/">Github</a>, and curl, I can do so easily, without even needing to have the repo handy. One of the things I do sometimes is compare a file across versions to see what has changed. For example, I needed to see what changes were made between versions 1.22 and 1.23 to the file includes/UserMailer.php which is part of the <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> project. For this trick to work, the project must be on Github, and must label their versions in a consistent manner, either via git branches or git tags.</p>
<p>MediaWiki exists on Github as <a href="https://github.com/wikimedia/mediawiki-core">wikimedia/mediawiki-core</a>. The MediaWiki project tags all of their releases in the format X.Y.Z, so in this example we can use the git tags <strong>1.22.0</strong> and <strong>1.23.0</strong>. Github is very nice because you can view a specific file at a certain commit (aka a tag), and even grab it over the web as a plain text file. The format is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">https://raw.githubusercontent.com/PROJECTNAME/BRANCH-OR-TAG/FILE
</code></pre></div><p>Note that you can use a tag <em><strong>OR</strong></em> a branch! So to compare these two files, we can use one of these pairs:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">https://raw.githubusercontent.com/wikimedia/mediawiki-core/REL1_21/includes/UserMailer.php
https://raw.githubusercontent.com/wikimedia/mediawiki-core/REL1_22/includes/UserMailer.php
https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.21.0/includes/UserMailer.php
https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.22.0/includes/UserMailer.php
</code></pre></div><p>All that is left is to treat git as a web service and compare the two files at the command line ourselves. The program <strong>curl</strong> is a great tool for downloading the files, as it dumps to stdout by default. We will add a <strong>-s</strong> flag (for “silent”) to prevent it from showing the progress meter as it usually does. The last bit of the puzzle is to use <(), bash’s process substitution feature, to trick diff into comparing the curl outputs as if they were files. So our final command is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">diff <(curl -s https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.21.0/includes/UserMailer.php) \
<(curl -s https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.22.0/includes/UserMailer.php) \
| more
</code></pre></div><p>Voila! A quick and simple glance at what changed between those two tags. This should work for any project on Github. You can also replace the branch or tag with the word “master” to see the current version. For example, the PostgreSQL project lives on github as postgres/postgres. They use the format RELX_Y_Z in their tags. To see what has changed since release 9.3.4 in the psql help file (as a context diff), run:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">diff<span style="color:#bbb"> </span>-<span style="color:#080;font-weight:bold">c</span><span style="color:#bbb"> </span><(curl<span style="color:#bbb"> </span>-s<span style="color:#bbb"> </span>https://raw.githubusercontent.com/postgres/postgres/REL9_3_4/src/bin/psql/help.<span style="color:#080;font-weight:bold">c</span>)<span style="color:#bbb"> </span><span style="color:#a61717;background-color:#e3d2d2">\</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><(curl<span style="color:#bbb"> </span>-s<span style="color:#bbb"> </span>https://raw.githubusercontent.com/postgres/postgres/master/src/bin/psql/help.<span style="color:#080;font-weight:bold">c</span>)<span style="color:#bbb">
</span></code></pre></div><p>You are not limited to diff, of course. For a final example, let’s see how many times Tom Lane is mentioned in the version 9 release notes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">for i in {0,1,2,3,4}
do grep -Fc 'Tom Lane' \
<(curl -s https://raw.githubusercontent.com/postgres/postgres/master/doc/src/sgml/release-9.$i.sgml)
done
272
206
174
115
16
</code></pre></div><p>The last number is so low relative to the rest because 9.4 is still under development. Rest assured Tom’s contributions have not slowed down! :) Thanks to Github for providing such a useful service for so many open source projects, and for providing the raw text to allow useful hacks like this.</p>
MediaWiki extensions and wfLoadExtensionMessageshttps://www.endpointdev.com/blog/2014/05/mediawiki-extensions-and/2014-05-09T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float: right; text-align: center;"><a href="/blog/2014/05/mediawiki-extensions-and/image-0.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2014/05/mediawiki-extensions-and/image-0.jpeg"/></a><br/><small><a href="https://flic.kr/p/dykcFR">Image</a> by <a href="https://www.flickr.com/photos/yukonlife/">Susan Drury</a></small>
</div>
<p>Upgrading MediaWiki can be a challenging task, especially if you use a lot of extensions.
While the core upgrade process usually goes smoothly, it’s rare you can upgrade a major
version or two without having to muddle with your collection of extensions. Extensions are bits of code that extend what MediaWiki can do. Only a few are packaged with and maintained alongside MediaWiki itself—the great majority are written by third-party developers. When the MediaWiki API changes, it is up to those developers to update their extension so it works with the new version of MediaWiki. This does not always happen. Take for example one of the more common errors seen on a MediaWiki upgrade since 1.21 was released:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">[Tue May 06 11:21:52 2014] [error] [client 12.34.56.78] PHP Fatal error: Call to undefined function wfLoadExtensionMessages() in /home/beckett/mediawiki/extensions/PdfExport/PdfExport.php on line 83, referer: http://test.ziggy.com/wiki/Main_Page
</code></pre></div><p>This is because the <strong>wfLoadExtensionMessages</strong> function, which many extensions use, has
been deprecated since MediaWiki version 1.16 and was finally removed in 1.21, resulting in the
error seen above. Luckily, this function has been a no-op since 1.16, so it is safe
to comment it out and/or make a dummy function in your LocalSettings.php file (see below).</p>
<p>Sadly, the release notes for 1.21 make no mention of
<a href="https://lists.gt.net/wiki/wikitech/214619">this fairly major change</a>. Let’s
walk through as if we didn’t know anything about it and see how we could solve the
given error with the help of git. For this example, we’ll use the
<a href="https://www.mediawiki.org/wiki/Extension:Pdf_Export">Pdf Export extension</a>,
which allows you to export your wiki pages into PDF form. A pretty handy extension, and
one which completely fails to work in MediaWiki version 1.21 or better.</p>
<p>First, let’s verify that wfLoadExtensionMessages does not exist at all in version 1.21 of MediaWiki. For
these examples, I’ve checked out the MediaWiki code via git, and am relying on
the fact that lightweight git tags were made for all the versions we are interested in.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git clone https://github.com/SemanticMediaWiki/SemanticMediaWiki.git mediawiki
$ cd mediawiki
$ git grep wfLoadExtensionMessages 1.21.0
1.21.0:HISTORY:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php
</code></pre></div><p>A nice feature of <code>git grep</code> is the ability to simply use a tag after the search string. In this
case, we see that the only mention of wfLoadExtensionMessages in the entire codebase is an
old mention of it in the history file. Let’s see what version that bug is from:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git grep -n wfLoadExtensionMessages 1.21.0
1.21.0:HISTORY:5280:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php
$ git show 1.21.0:HISTORY | head -5280 | tac | grep '===' -m1
=== Bug fixes in 1.12 ===
</code></pre></div><p>That message is from way back in version 1.12, and doesn’t concern us. Let’s take a look at
what tags exist in the 1.20 branch so we can scan the latest one:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git tag | grep '^1\.20'
1.20.0
1.20.0rc1
1.20.0rc2
1.20.1
1.20.2
1.20.3
1.20.4
1.20.5
1.20.6
1.20.7
1.20.8
</code></pre></div><p>Now we can peek inside version 1.20.8 and see what that function did before it was removed.
By using the -A and -B (after and before) arguments to grep, we can see the entire function in
context:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git grep wfLoadExtensionMessages 1.20.0
1.20.0:HISTORY:* (bug 12880) wfLoadExtensionMessages does not use $fallback from MessagesXx.php
1.20.0:includes/GlobalFunctions.php:function wfLoadExtensionMessages() {
$ git show 1.20.8:includes/GlobalFunctions.php | grep -B6 -A2 LoadExtensionMessages
/**
* Load an extension messages file
*
* @deprecated since 1.16, warnings in 1.18, remove in 1.20
* @codeCoverageIgnore
*/
function wfLoadExtensionMessages() {
wfDeprecated( __FUNCTION__, '1.16' );
}
</code></pre></div><p>Thus wfLoadExtensionMessages was basically a no-op in MediaWiki version 1.20, with the caveat that it will write
a deprecation warning to your error log (or, in modern versions, the debug log unless $wgDevelopmentWarnings is set).
Next we want to find the last time this function did something useful—which should be version 1.15 according to
the comment above. Thus:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git show 1.15.0:includes/GlobalFunctions.php | grep -A4 LoadExtensionMessages
function wfLoadExtensionMessages( $extensionName, $langcode = false ) {
global $wgExtensionMessagesFiles, $wgMessageCache, $wgLang, $wgContLang;
#For recording whether extension message files have been loaded in a given language.
static $loaded = array();
</code></pre></div><p>So, it’s a pretty safe bet that unless you are upgrading from 1.15.0 or earlier, it should
be completely safe to remove it. When was 1.16.0 released? There are no dates in the HISTORY
file (shame), but the date it was tagged should be a good guess:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git show 1.16.0 | grep -m1 Date
Date: Wed Jul 28 07:11:03 2010 +0000
</code></pre></div><p>So what should you do with extensions that are still using this deprecated function? There are
two quick solutions: comment it out inside the extension, or add a dummy function to your version
of MediaWiki.</p>
<p>Changing the extension itself is certainly quick and easy. To get the PdfExport extension to work,
we only have to comments out two calls to wfLoadExtensionMessages inside of the file
PdfExport.php, and one inside of PdfExport_body.php. The diff:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ git difftool -y -x "diff -u1"
--- /tmp/7YqvXv_PdfExport.php 2014-05-08 12:45:03 -0400
+++ PdfExport.php 2014-05-08 12:34:39 -0400
@@ -82,3 +82,3 @@
if ($img_page > 0 || $img_page === false) {
- wfLoadExtensionMessages('PdfPrint');
+ //wfLoadExtensionMessages('PdfPrint');
$nav_urls['pdfprint'] = array(
@@ -92,3 +92,3 @@
function wfSpecialPdfToolbox (&$monobook) {
- wfLoadExtensionMessages('PdfPrint');
+ //wfLoadExtensionMessages('PdfPrint');
if (isset($monobook->data['nav_urls']['pdfprint']))
--- /tmp/7gO8Hz_PdfExport_body.php 2014-05-08 12:45:03 -0400
+++ PdfExport_body.php 2014-05-08 12:34:44 -0400
@@ -44,3 +44,3 @@
// For backwards compatibility
- wfLoadExtensionMessages('PdfPrint');
+ //wfLoadExtensionMessages('PdfPrint');
</code></pre></div><p>A better way is to add a dummy function to LocalSettings.php. This ensures that any extension
we add in the future will continue to work unmodified. Just throw this at the bottom
on your LocalSettings.php:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">function</span> <span style="color:#06b;font-weight:bold">wfLoadExtensionMessages</span>() { }
</code></pre></div><p>Probably the best overall solution is to not only add that to your LocalSettings.php,
but to try to get the extension changed as well. You can notify the author, or try to
fix it yourself and release a new version if the extension has been abandoned. You might
also look to see if the extension has been superseded by a different extension, as sometime
happens.</p>
<p>While there may be other compatibility issues when upgrading MediaWiki, for some extensions
(such as PdfExport), this is the only change needed to make it work again on newer versions of MediaWiki!</p>
Chrome, onmousemove, and MediaWiki JavaScripthttps://www.endpointdev.com/blog/2014/04/chrome-onmousemove-and-mediawiki/2014-04-18T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float: right; text-align: center;"><a href="/blog/2014/04/chrome-onmousemove-and-mediawiki/image-0-big.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2014/04/chrome-onmousemove-and-mediawiki/image-0.jpeg"/></a>
<br/><small><a href="https://flic.kr/p/aM4L46">Image</a> by Flickr user <a href="https://www.flickr.com/photos/archer10/">Dennis Jarvis</a></small></div>
<p><em>tl;dr: avoid using onmousemove events with Google Chrome.</em></p>
<p>I recently fielded a complaint about not being able to select text with the mouse on a wiki running the <a href="https://www.mediawiki.org/wiki/MediaWiki"> MediaWiki software</a>. After some troubleshooting and research, I narrowed the problem down to a bug in the Chrome browser regarding the onmousemove event. The solution in this case was to tweak JavaScript to use onmouseover instead of onmousemove.</p>
<p>The first step in troubleshooting is to duplicate the problem. In this case, the page worked fine for me in Firefox, so I tried using the same browser as the reporter: Chrome. Sure enough, I could no longer hold down the mouse button and select text on the page. Now that the browser was implicated, it was time to see what it was about this page that caused the problem.</p>
<p>It seemed fairly unlikely that something like this would go unfixed if it was happening on the flagship MediaWiki site, Wikipedia. Sure enough, that site worked fine, I could select the text with no problem. Testing some other random sites showed no problems either. Some googling indicated others had similar problems with Chrome, and gave a bunch of workarounds for selecting the text. However, I wanted a fix, not a workaround.</p>
<p>There were hints that JavaScript was involved, so I disabled JavaScript in Chrome, reloaded the page, and suddenly everything started working again. Call that big clue number two. The next step was to see what was different between the local MediaWiki installation and Wikipedia. The local site was a few versions behind, but I was fortuitously testing an upgrade on a test server. This showed the problem still existed on the newer version, which meant that the problem was something specific to the wiki itself.</p>
<p>The most likely culprit was one of the many installed <a href="https://www.mediawiki.org/wiki/Manual:Extensions">MediaWiki extensions</a>, which are small pieces of code that perform certain actions on a wiki. These often have their own JavaScript that they run, which was still the most likely problem.</p>
<p>Then it was some basic troubleshooting. After turning JavaScript back on, I edited the LocalSettings.php file and commented out all the user-supplied extensions. This made the problem disappear again. Then I commented out half the extensions, then half again, etc., until I was able to narrow the problem down to a single extension.</p>
<p>The extension in question, known simply as “<a href="https://www.mediawiki.org/wiki/Extension:Balloons">balloons</a>”, has actually been removed from the MediaWiki extensions site, for “prolonged security issues with the code.” The extension allows creation of very nice looking pop up CSS “balloons” full of text. I’m guessing the concern is because the arguments for the balloons were not sanitized properly. In a public wiki, this would be a problem, but this was for a private intranet, so we were not worried about continuing to use the extension. As a side note, such changes would be caught anyway as this wiki sends an email to a few people on any change, including a full text diff of all the changes.</p>
<p>Looking inside the JavaScript used by the extension, I was able to narrow the problem down to a single line inside balloons/js/balloons.js:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"> <span style="color:#888">// Track the cursor every time the mouse moves
</span><span style="color:#888"></span> <span style="color:#038">document</span>.onmousemove = <span style="color:#080;font-weight:bold">this</span>.setActiveCoordinates;
</code></pre></div><p>Sure enough, duck-duck-going through the Internet quickly found <a href="https://code.google.com/p/chromium/issues/detail?id=170631">a fairly incriminating Chromium bug</a>, indicating that onmousemove did not work very well at all. Looking over the balloon extension code, it appeared that onmouseover would probably be good enough to gather the same information and allow the extension to work while not blocking the ability for people to select text. One small replacement of “move” to “over”, and everything was back to working as it should!</p>
<p>So in summary, if you cannot select text with the mouse in Chrome (or you see any other odd mouse-related behaviors), suspect an onmousemove issue.</p>