https://www.endpointdev.com/blog/tags/json/2021-12-16T00:00:00+00:00End Point DevUsing Postman to Test APIshttps://www.endpointdev.com/blog/2021/12/using-postman-to-test-apis/2021-12-16T00:00:00+00:00Couragyn Chretien
<p><img src="/blog/2021/12/using-postman-to-test-apis/20071221_144412000-sm.jpg" alt="Photo of large rusty iron chain with blurry sea in background"></p>
<!-- photo by Josh Ausborne -->
<p>Postman is an easy-to-use tool that facilitates testing APIs. The GUI avoids the pain of old-school command-line tools and other time-consuming practices. Postman is helpful during development, for collaborating, and integral in automated testing.</p>
<p>It’s one of the most popular and longest-running API tools out there, so there’s a lot of documentation and community forums to provide whatever assistance is needed.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/postman.png" alt="Postman logo"></p>
<h3 id="use-in-development">Use in development</h3>
<p>The days of only manually testing APIs with curl are gone.</p>
<p>Instead we can use Postman to create reusable API calls. Postman will remember information such as the header configuration and data contained in the body. The API call can be sent from the GUI client, and it will display the response in a human-readable format.</p>
<p>Saved requests can be grouped in Collections. These can be imported and exported for collaborative use in a team.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/gui.png" alt="Postman GUI screenshot"></p>
<h3 id="automated-testing">Automated Testing</h3>
<p>Full API testing can be done with the click of a button. Investing time now into building up the Postman infastructure will save development time down the line.</p>
<p>Custom Test Suites can be written in JavaScript and run manually or by a script. Periodically running these will ensure your application’s API experiences limited regression. Below are a few simple tests that can be run using Postman’s JavaScript <code>pm</code> object.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// Post request
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">const</span> postRequest = {
url: <span style="color:#d20;background-color:#fff0f0">'https://endpointdev.com/post'</span>,
method: <span style="color:#d20;background-color:#fff0f0">'POST'</span>,
header: {
<span style="color:#d20;background-color:#fff0f0">'Content-Type'</span>: <span style="color:#d20;background-color:#fff0f0">'application/json'</span>
},
body: {
mode: <span style="color:#d20;background-color:#fff0f0">'raw'</span>,
raw: JSON.stringify({ key: <span style="color:#d20;background-color:#fff0f0">'posting this text'</span> })
}
};
pm.sendRequest(postRequest, (error, response) => {
console.log(error ? error : response.json());
});
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// Get request
</span><span style="color:#888"></span>pm.sendRequest(<span style="color:#d20;background-color:#fff0f0">'https://endpointdev.com/get'</span>, (error, response) => {
<span style="color:#080;font-weight:bold">if</span> (error) {
console.log(error);
}
pm.test(<span style="color:#d20;background-color:#fff0f0">'response should be okay to process'</span>, () => {
pm.expect(response).to.have.property(<span style="color:#d20;background-color:#fff0f0">'status'</span>, <span style="color:#d20;background-color:#fff0f0">'OK'</span>);
pm.expect(response).to.have.property(<span style="color:#d20;background-color:#fff0f0">'code'</span>, <span style="color:#00d;font-weight:bold">200</span>);
pm.expect(error).to.equal(<span style="color:#080;font-weight:bold">null</span>);
});
});
</code></pre></div><p>These tests can be hooked into your CI/CD pipeline easily for truly automated testing. Postman integrates with Jenkins with help from Newman, Postman’s command-line Collection Runner. For a step-by-step guide getting this up and running see their <a href="https://learning.postman.com/docs/running-collections/using-newman-cli/integration-with-jenkins/">Integrating with Jenkins</a> documentation.</p>
<h3 id="alternatives">Alternatives</h3>
<p>Insomnia and Paw are two of the top alternatives to Postman. They have mostly the same capabilities with slight differences.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/insomnia.png" alt="Insomnia logo"></p>
<p>Insomnia, like Postman, has a free version and an upgraded paid version. The UI tends to be a bit easier on the eyes. Its lightweight nature allows it to run faster and be a bit more responsive than Postman typically is. The main downside is the inability to write testing for standard requests—it needs to be in OpenAPI format.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/paw.png" alt="Paw logo"></p>
<p>Paw is Apple to Postman’s Android. It was originally a Mac-only application and has only recently become available on other platforms. Like most Mac apps its UI is streamlined and will be very easy to pick up for any Mac users. The original Mac app performs better than the cross-platform version, and it has many extensions to expand its capabilities. The main downside is that there is no free version.</p>
<h3 id="reference">Reference</h3>
<ul>
<li><a href="https://www.postman.com/">Postman</a></li>
<li><a href="https://insomnia.rest/">Insomnia</a></li>
<li><a href="https://paw.cloud/">Paw</a></li>
</ul>
What is serialization?https://www.endpointdev.com/blog/2021/05/what-is-serialization/2021-05-06T00:00:00+00:00Zed Jensen
<p><img src="/blog/2021/05/what-is-serialization/banner.jpg" alt="Mailbox">
Photo by <a href="https://unsplash.com/@briantagalog">Brian Patrick Tagalog</a> on <a href="https://unsplash.com/photos/axwRgfER-sA">Unsplash</a></p>
<p>Serialization is a process used constantly by most applications today. However, there are some common misconceptions and misunderstandings about what it is and how it works; I hope to clear up a few of these in this post. I’ll be talking specifically about serialization and not marshalling, a related process.</p>
<h3 id="what-is-serialization">What is serialization?</h3>
<p>Most developers know that complex objects need to be transformed into another format before they can be sent to a server, but many might not be aware that every time they print an object in the Python or JavaScript console, the same type of thing is happening. Variables and objects as they’re stored in memory—either in a headless program or one with developer tools attached—are not really usable to us humans.</p>
<p>Data serialization is the process of taking an object in memory and translating it to another format. This may entail encoding the information as a chunk of binary to store in a database, creating a string representation that a human can understand, or saving a config file from the options a user selected in an application. The reverse—deserialization—takes an object in one of these formats and converts it to an in-memory object the program can work with. This two-way process of translation is a very important part of the ability of various programs and computers to communicate with one another.</p>
<p>An example of serialization that we deal with every day can be found in the way we view numbers on a calculator. Computers use binary numbers, not decimal, so how do we ask one to add 230 and 4 and get back 234? Because the 230 and the 4 are deserialized to their machine representations, added in that format, and then serialized again in a form we understand: 234. To get 230 in a form the computer understands, it has to read each digit one at a time, figure out what that digit’s value is (i.e. the 2 is 200 and the 3 is 30), and then add them together. It’s easy to overlook how often this concept appears in everything we do with computers!</p>
<h3 id="why-its-important-to-understand-how-it-works">Why it’s important to understand how it works</h3>
<p>As a developer, there are many reasons you should be familiar with how serialization works as well as the various formats available, including:</p>
<ul>
<li>Different formats are best suited for different use cases.</li>
<li>Standardization varies between formats. For example, INI files have no single specification, but TOML does. YAML 1.2 came out in 2009 but most YAML parsers still implement only parts of the earlier YAML 1.1 spec.</li>
<li>Each application typically supports only one or a few formats.</li>
<li>Formats have different goals, such as readability & simplicity for humans, speed for computers, and conciseness for storage space and transfer efficiency.</li>
<li>Applications use the various formats very differently from each other.</li>
</ul>
<p>Before you start working on a project, it will certainly pay off to make sure you’re familiar with the options for serialization formats so you can pick the one most suited to your particular use case.</p>
<h3 id="binary-vs-human-readable-serialization">Binary vs. human-readable serialization</h3>
<p>There’s one more important distinction to be made before I show any examples, and that is human-readable vs. binary serialization. The advantage of human-readability is obvious: debugging in particular is much simpler, but other things like scanning data for keywords is much easier as well. Binary serialization, however, can be much faster to process for both the sender and recipient, it can sometimes include information that’s hard to represent in plain text, and it can be much more efficient with space without needing separate compression. I’ll stick to reviewing human-readable formats in this post.</p>
<h3 id="common-serialization-formats-with-examples">Common serialization formats with examples</h3>
<h4 id="csv">CSV</h4>
<p>For my examples, I’ll have a simple JavaScript object representing myself, with properties including my name, recent books I’ve read, and my favorite food. I’ll start with <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> (comma-separated values) because it’s intended for simpler data records than most of the other formats I’ll be showing; you’ll notice that there isn’t an easy way to do object hierarchies or lists. CSV files begin with a list of the column names followed by the rows of data:</p>
<pre tabindex="0"><code class="language-csv" data-lang="csv">name,favorite_food_name,favorite_food_prep_time,recent_book
Zed,Pizza,30,Leviathan Wakes
</code></pre><p>CSV files are most often used for storing or transferring tabular data, but there’s no single specification, so the implementation can be fairly different in different programs. The most common differences involve data with commas or line breaks, requiring quoting of some or all elements, and escaping some characters.</p>
<h4 id="tsv">TSV</h4>
<p>Files in tab-separated values (TSV) format are also fairly common, using tabs instead of commas to separate columns of data.</p>
<p>Because the tab character is rarely used in text put into table format, it is less of a problem as a separator than the very frequently-occurring comma. Typically no quoting or escaping of any kind is needed or possible in a TSV file.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">name favorite_food_name favorite_food_prep_time recent_book
Zed Pizza 30 Leviathan Wakes
</code></pre></div><p>For the rest of my examples of each format, I’ll show the command (and library, if needed) that I used to get the serialized form of my object.</p>
<h4 id="json">JSON</h4>
<p><a href="https://www.json.org/json-en.html">JSON</a> stands for JavaScript Object Notation, and thus you might be fooled into thinking that it’s just an extension of JavaScript itself. However, this isn’t the case; it was originally derived from JavaScript syntax, but it has significant differences. For example, JSON has a stricter syntax for declaring objects. For my example, using the Google Chrome developer console I declared my object like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">const</span> me = {
name: <span style="color:#d20;background-color:#fff0f0">'Zed'</span>,
recent_books: [
<span style="color:#d20;background-color:#fff0f0">'Leviathan Wakes'</span>,
<span style="color:#d20;background-color:#fff0f0">'Pride and Prejudice and Zombies'</span>
],
favorite_food: {
name: <span style="color:#d20;background-color:#fff0f0">'Pizza'</span>,
prep_time: <span style="color:#00d;font-weight:bold">30</span>
}
};
</code></pre></div><p>You’ll notice that the property names aren’t quoted and the strings are single-quoted with <code>'</code>. This is perfectly valid JavaScript, but invalid JSON. Let’s see what an equivalent JSON file could look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Zed"</span>,
<span style="color:#b06;font-weight:bold">"recent_books"</span>: [
<span style="color:#d20;background-color:#fff0f0">"Leviathan Wakes"</span>,
<span style="color:#d20;background-color:#fff0f0">"Pride and Prejudice and Zombies"</span>
],
<span style="color:#b06;font-weight:bold">"favorite_food"</span>: {
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Pizza"</span>,
<span style="color:#b06;font-weight:bold">"prep_time"</span>: <span style="color:#00d;font-weight:bold">30</span>
}
}
</code></pre></div><p>JSON requires property names to be quoted, and only double quotes <code>"</code> are allowed. It’s true that they look very similar, but the difference is important. Also notice that this JSON is formatted in an easy-to-read way, on multiple lines with indentation. This is called pretty-printing and is possible because JSON doesn’t care about whitespace.</p>
<p>Imagine my JavaScript application wants to send this object to some server that’s expecting JSON, using any other platform such as Java or .NET and not necessarily JavaScript. It would need to serialize the object from memory into a JSON string first, which can be done by JavaScript itself:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">> <span style="color:#080;font-weight:bold">let</span> meJSON = JSON.stringify(me);
> console.log(meJSON);
{<span style="color:#d20;background-color:#fff0f0">"name"</span>:<span style="color:#d20;background-color:#fff0f0">"Zed"</span>,<span style="color:#d20;background-color:#fff0f0">"recent_books"</span>:[<span style="color:#d20;background-color:#fff0f0">"Leviathan Wakes"</span>,<span style="color:#d20;background-color:#fff0f0">"Pride and Prejudice and Zombies"</span>],<span style="color:#d20;background-color:#fff0f0">"favorite_food"</span>:{<span style="color:#d20;background-color:#fff0f0">"name"</span>:<span style="color:#d20;background-color:#fff0f0">"Pizza"</span>,<span style="color:#d20;background-color:#fff0f0">"prep_time"</span>:<span style="color:#00d;font-weight:bold">30</span>}}
</code></pre></div><p>Note that the result here has no extra line breaks or spaces. This is called minifying, and is the reverse of pretty-printing. The flexibility allowed by these two processes is one reason people like JSON.</p>
<p>Parsing our example back into a JavaScript object is also very easy:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">> console.log(JSON.parse(meJSON));
{
name: <span style="color:#d20;background-color:#fff0f0">'Zed'</span>,
recent_books: [ <span style="color:#d20;background-color:#fff0f0">'Leviathan Wakes'</span>, <span style="color:#d20;background-color:#fff0f0">'Pride and Prejudice and Zombies'</span> ],
favorite_food: { name: <span style="color:#d20;background-color:#fff0f0">'Pizza'</span>, prep_time: <span style="color:#00d;font-weight:bold">30</span> }
}
</code></pre></div><p>The easy integration with JavaScript is a big reason JSON is so popular. I showed these examples to highlight how easy it is to use, but also to point out that sometimes we might use serialization without being aware of what’s going on under the hood; it’s important to remember that JSON texts aren’t JavaScript objects, and there may be instances where it makes more sense to use another format.</p>
<p>For instance, if you need a config file format that’s easy for humans to read, it is very helpful to allow comments that are not part of the data structure once it is read into memory. But CSV, TSV, and JSON do not allow for comments. The most obvious or popular choice isn’t always the only one, or the best one, so let’s keep looking at other formats.</p>
<h4 id="xml">XML</h4>
<p><a href="https://www.w3.org/XML/">XML</a> is well known as the markup language of which HTML is a subset, or at least a close sibling. It can also be used for serialization of data, and allows us to add comments such as the one at the beginning:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#888"><!-- My favorites as of May 2021 --></span>
<span style="color:#b06;font-weight:bold"><name></span>Zed<span style="color:#b06;font-weight:bold"></name></span>
<span style="color:#b06;font-weight:bold"><recent_books></span>Leviathan Wakes<span style="color:#b06;font-weight:bold"></recent_books></span>
<span style="color:#b06;font-weight:bold"><recent_books></span>Pride and Prejudice and Zombies<span style="color:#b06;font-weight:bold"></recent_books></span>
<span style="color:#b06;font-weight:bold"><favorite_food></span>
<span style="color:#b06;font-weight:bold"><name></span>Pizza<span style="color:#b06;font-weight:bold"></name></span>
<span style="color:#b06;font-weight:bold"><prep_time></span>30<span style="color:#b06;font-weight:bold"></prep_time></span>
<span style="color:#b06;font-weight:bold"></favorite_food></span>
</code></pre></div><p>XML has the benefit of being widely used, and it can represent more complex data structures since each element can also optionally have various attributes, and ordering of its child elements is significant.</p>
<p>But XML is unpleasant to type and for many use cases feels rather complex and bloated, so it suffers when compared to other formats we are looking at in this post.</p>
<h4 id="yaml">YAML</h4>
<p><a href="https://yaml.org/">YAML</a> is a serialization format for all kinds of data that’s designed to be human-readable. Simple files look fine, like our example:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml"><span style="color:#b06;font-weight:bold">---</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888"># My favorites as of May 2021</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#b06;font-weight:bold">name</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"Zed"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#b06;font-weight:bold">recent_books</span>:<span style="color:#bbb">
</span><span style="color:#bbb"> </span>- <span style="color:#d20;background-color:#fff0f0">"Leviathan Wakes"</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>- <span style="color:#d20;background-color:#fff0f0">"Pride and Prejudice and Zombies"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#b06;font-weight:bold">favorite_food</span>:<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#b06;font-weight:bold">name</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"pizza"</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#b06;font-weight:bold">prep_time</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">30</span><span style="color:#bbb">
</span></code></pre></div><p>However, the YAML specification is far from simple, and quite a bit has been written on why it’s better to use other formats where possible:</p>
<ul>
<li><a href="https://github.com/cblp/yaml-sucks">YAML sucks.</a></li>
<li><a href="https://www.arp242.net/yaml-config.html">YAML: probably not so great after all</a></li>
<li><a href="https://allan.reyes.sh/programming/2018/06/20/The-YAML-NOrway-Law.html">The YAML-NOrway Law</a></li>
</ul>
<h4 id="ini">INI</h4>
<p><a href="https://en.wikipedia.org/wiki/INI_file">INI</a>, short for <em>initialization</em>, is well-known and has been around since the ’90s or earlier. It was most notably used for configuration files in Windows, especially in the era before Windows 95. INI files are still used in many places, including Windows and Linux programs’ system configuration files such as for the Git version control system.</p>
<p>Our example in INI format looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#888">; My favorites as of May 2021</span>
<span style="color:#369">name</span>=<span style="color:#d20;background-color:#fff0f0">Zed</span>
<span style="color:#369">recent_books[]</span>=<span style="color:#d20;background-color:#fff0f0">Leviathan Wakes</span>
<span style="color:#369">recent_books[]</span>=<span style="color:#d20;background-color:#fff0f0">Pride and Prejudice and Zombies</span>
<span style="color:#080;font-weight:bold">[favorite_food]</span>
<span style="color:#369">name</span>=<span style="color:#d20;background-color:#fff0f0">Pizza</span>
<span style="color:#369">prep_time</span>=<span style="color:#d20;background-color:#fff0f0">30</span>
</code></pre></div><p>INI has no single specification, so one project’s config files might use different syntax from another. This makes it hard to recommend over newer formats like TOML.</p>
<h4 id="toml">TOML</h4>
<p><a href="https://toml.io/en/">TOML</a>, which stands for Tom’s Obvious Minimal Language, is a more recent addition to serialization formats; its first version was released in 2013. TOML maps directly to dictionary objects and is intended especially for configuration files as an alternative to INI. It has similar syntax to INI as well:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-toml" data-lang="toml"><span style="color:#888"># My favorites as of May 2021</span>
name = <span style="color:#d20;background-color:#fff0f0">"Zed"</span>
recent_books = [
<span style="color:#d20;background-color:#fff0f0">"Leviathan Wakes"</span>,
<span style="color:#d20;background-color:#fff0f0">"Pride and Prejudice and Zombies"</span>
]
[favorite_food]
name = <span style="color:#d20;background-color:#fff0f0">"Pizza"</span>
prep_time = <span style="color:#00d;font-weight:bold">30</span>
</code></pre></div><p>Unlike INI and YAML, TOML has a very clear and well-defined specification, and seems like a great option for new projects in the future. It is currently used most prominently by the Rust programming language tools. There is a list of TOML libraries per language and version on the <a href="https://github.com/toml-lang/toml/wiki">TOML wiki at GitHub</a>.</p>
<h4 id="phps-serialize">PHP’s serialize()</h4>
<p><a href="https://www.php.net/manual/en/function.serialize.php">PHP’s serialization output</a> isn’t quite as readable, but the data is still recognizable for someone scanning visually for keywords or doing a more rigorous search. Converting from JSON is fairly simple:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#888">#!/usr/bin/env php
</span><span style="color:#888"></span>
<?php
<span style="color:#369">$json</span> = <span style="color:#d20;background-color:#fff0f0">'
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "name": "Zed",
</span><span style="color:#d20;background-color:#fff0f0"> "recent_books": [
</span><span style="color:#d20;background-color:#fff0f0"> "Leviathan Wakes",
</span><span style="color:#d20;background-color:#fff0f0"> "Pride and Prejudice and Zombies"
</span><span style="color:#d20;background-color:#fff0f0"> ],
</span><span style="color:#d20;background-color:#fff0f0"> "favorite_food": {
</span><span style="color:#d20;background-color:#fff0f0"> "name": "Pizza",
</span><span style="color:#d20;background-color:#fff0f0"> "prep_time": 30
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0">'</span>;
<span style="color:#369">$obj</span> = json_decode(<span style="color:#369">$json</span>, <span style="color:#080;font-weight:bold">true</span>);
<span style="color:#080;font-weight:bold">echo</span> serialize(<span style="color:#369">$obj</span>);
</code></pre></div><p>And the result:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">a:3:{s:4:"name";s:3:"Zed";s:12:"recent_books";a:2:{i:0;s:15:"Leviathan Wakes";i:1;s:27:"Pride and Prejudice and Zombies";}s:13:"favorite_food";a:2:{s:4:"name";s:5:"Pizza";s:9:"prep_time";i:30;}}
</code></pre></div><p>PHP serialize() does not allow for comments, but it does support full object marshalling, which it is more commonly used for.</p>
<h4 id="perls-datadumper">Perl’s Data::Dumper</h4>
<p>Perl’s <a href="https://metacpan.org/pod/Data::Dumper">Data::Dumper</a> module serializes data in a format specifically for Perl to load back into memory:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#888">#!/usr/bin/env perl</span>
<span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">strict</span>;
<span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">warnings</span>;
<span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">JSON</span>;
<span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">Data::Dumper</span> <span style="color:#d20;background-color:#fff0f0">'Dumper'</span>;
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$json</span> = <span style="color:#d20;background-color:#fff0f0"><<'</span><span style="color:#d20;background-color:#fff0f0">END</span><span style="color:#d20;background-color:#fff0f0">';
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "name": "Zed",
</span><span style="color:#d20;background-color:#fff0f0"> "recent_books": [
</span><span style="color:#d20;background-color:#fff0f0"> "Leviathan Wakes",
</span><span style="color:#d20;background-color:#fff0f0"> "Pride and Prejudice and Zombies"
</span><span style="color:#d20;background-color:#fff0f0"> ],
</span><span style="color:#d20;background-color:#fff0f0"> "favorite_food": {
</span><span style="color:#d20;background-color:#fff0f0"> "name": "Pizza",
</span><span style="color:#d20;background-color:#fff0f0"> "prep_time": 30
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0"></span><span style="color:#d20;background-color:#fff0f0">END</span>
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$hash</span> = decode_json <span style="color:#369">$json</span>;
<span style="color:#080;font-weight:bold">print</span> Dumper(<span style="color:#369">$hash</span>);
</code></pre></div><p>And the result, which is a valid Perl statement:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#369">$VAR1</span> = {
<span style="color:#d20;background-color:#fff0f0">'recent_books'</span> => [
<span style="color:#d20;background-color:#fff0f0">'Leviathan Wakes'</span>,
<span style="color:#d20;background-color:#fff0f0">'Pride and Prejudice and Zombies'</span>
],
<span style="color:#d20;background-color:#fff0f0">'name'</span> => <span style="color:#d20;background-color:#fff0f0">'Zed'</span>,
<span style="color:#d20;background-color:#fff0f0">'favorite_food'</span> => {
<span style="color:#d20;background-color:#fff0f0">'name'</span> => <span style="color:#d20;background-color:#fff0f0">'Pizza'</span>,
<span style="color:#d20;background-color:#fff0f0">'prep_time'</span> => <span style="color:#00d;font-weight:bold">30</span>
}
}
</code></pre></div><h3 id="conclusion">Conclusion</h3>
<p>Serialization is an extremely common function that we as programmers should be familiar with. Knowing which is a good option for a new project can save time and money, as well as making things easier for developers and API users.</p>
<p>Please leave a comment if I have missed your favorite format!</p>
<h4 id="further-reading">Further reading</h4>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Serialization">Serialization on Wikipedia</a></li>
<li><a href="https://en.wikipedia.org/wiki/Marshalling_(computer_science)">Marshaling on Wikipedia</a></li>
</ul>
Serialization and Deserialization Issues in Spring RESThttps://www.endpointdev.com/blog/2020/03/serialization-issues-spring-rest/2020-03-17T00:00:00+00:00Kürşat Kutlu Aydemir
<p><img src="/blog/2020/03/serialization-issues-spring-rest/image-0.jpg" alt="Mosaic pattern"></p>
<p><a href="https://unsplash.com/photos/hCzHhu1v0fA">Photo</a> by <a href="https://unsplash.com/@anniespratt">Annie Spratt</a></p>
<p><a href="https://spring.io/projects/spring-boot">Spring Boot</a> projects primarily use the JSON library <a href="https://github.com/FasterXML/jackson">Jackson</a> to serialize and deserialize objects. It is especially useful that Jackson automatically serializes objects returned from REST APIs and deserializes complex type parameters like <code>@RequestBody</code>.</p>
<p>In a Spring Boot project the automatically registered <code>MappingJackson2HttpMessageConverter</code> is usually enough and makes JSON conversions simple, but this may have some issues which need custom configuration. Let’s go over a few good practices for them.</p>
<h3 id="configuring-a-custom-jackson-objectmapper">Configuring a Custom Jackson ObjectMapper</h3>
<p>In Spring REST projects a custom implementation of <code>MappingJackson2HttpMessageConverter</code> helps to create the custom <code>ObjectMapper</code>, as seen below. Whatever custom implementation you need to add to the custom <code>ObjectMapper</code> can be handled by this custom converter:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">CustomHttpMessageConverter</span> <span style="color:#080;font-weight:bold">extends</span> MappingJackson2HttpMessageConverter {
<span style="color:#080;font-weight:bold">private</span> ObjectMapper <span style="color:#06b;font-weight:bold">initCustomObjectMapper</span>() {
ObjectMapper customObjectMapper = <span style="color:#080;font-weight:bold">new</span> ObjectMapper();
<span style="color:#080;font-weight:bold">return</span> customObjectMapper;
}
<span style="color:#888">// ...
</span><span style="color:#888"></span>}
</code></pre></div><p>Additionally, some <code>MappingJackson2HttpMessageConverter</code> methods, such as <code>writeInternal</code>, can be useful to override in certain cases. I’ll give a few examples in this article.</p>
<p>In Spring Boot you also need to register a custom <code>MappingJackson2HttpMessageConverter</code> like below:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#555">@Bean</span>
MappingJackson2HttpMessageConverter <span style="color:#06b;font-weight:bold">mappingJackson2HttpMessageConverter</span>() {
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> CustomHttpMessageConverter();
}
</code></pre></div><h3 id="serialization">Serialization</h3>
<h4 id="pretty-printing">Pretty-printing</h4>
<p>Pretty-printing in Jackson is disabled by default. By enabling <code>SerializationFeature.INDENT_OUTPUT</code> in the <code>ObjectMapper</code> configuration pretty-print output is enabled (as in the example below). Normally a custom <code>ObjectMapper</code> is not necessary for setting the pretty-print configuration. In some cases, however, like one case of mine in a recent customer project, this configuration might be necessary.</p>
<p>For example, passing a URL parameter can enable pretty-printing. In this case having a custom <code>ObjectMapper</code> with pretty-print enabled and keeping the default <code>ObjectMapper</code> of <code>MappingJackson2HttpMessageConverter</code> as is could be a better option.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">CustomHttpMessageConverter</span> <span style="color:#080;font-weight:bold">extends</span> MappingJackson2HttpMessageConverter {
<span style="color:#080;font-weight:bold">private</span> ObjectMapper <span style="color:#06b;font-weight:bold">initiatePrettyObjectMapper</span>() {
ObjectMapper customObjectMapper = <span style="color:#080;font-weight:bold">new</span> ObjectMapper();
customObjectMapper.<span style="color:#369">configure</span>(SerializationFeature.<span style="color:#369">INDENT_OUTPUT</span>, <span style="color:#080;font-weight:bold">true</span>);
<span style="color:#888">// additional indentation for arrays
</span><span style="color:#888"></span> DefaultPrettyPrinter pp = <span style="color:#080;font-weight:bold">new</span> DefaultPrettyPrinter();
pp.<span style="color:#369">indentArraysWith</span>(<span style="color:#080;font-weight:bold">new</span> DefaultIndenter());
customObjectMapper.<span style="color:#369">setDefaultPrettyPrinter</span>(pp);
<span style="color:#080;font-weight:bold">return</span> customObjectMapper;
}
}
</code></pre></div><h4 id="conditionally-filtering-the-fields">Conditionally Filtering the Fields</h4>
<p>When serializing a response object you may need to include or ignore one or more fields depending on their values. Let’s assume a model class <code>UserResponse</code> like below.</p>
<p>Notice that we used <code>@JsonIgnore</code> which is completely discarding the annotated field from serialization. Conditional filtering is different and it can be done using <code>SimpleBeanPropertyFilter</code> objects set to the filter provider of the <code>ObjectMapper</code> objects. Also notice that <code>@JsonFilter</code> annotation is used for <code>UserResponse</code> which points to which filter will be used by <code>ObjectMapper</code> during the serialization.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#555">@JsonFilter</span>(<span style="color:#d20;background-color:#fff0f0">"userCodeFilter"</span>)
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">UserResponse</span> {
<span style="color:#080;font-weight:bold">public</span> Integer userId;
<span style="color:#080;font-weight:bold">public</span> String username;
<span style="color:#080;font-weight:bold">public</span> Integer code;
<span style="color:#555">@JsonIgnore</span>
<span style="color:#080;font-weight:bold">public</span> String status;
}
</code></pre></div><p>Here we add a filter called <code>userCodeFilter</code>—like the one we added to the custom <code>ObjectMapper</code> of <code>CustomHttpMessageConverter</code>—which will include the <code>UserResponse</code> class’s code field in the serialization if its value is greater than 0. You can add multiple filters to <code>ObjectMapper</code> for different models.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">CustomHttpMessageConverter</span> <span style="color:#080;font-weight:bold">extends</span> MappingJackson2HttpMessageConverter {
<span style="color:#080;font-weight:bold">private</span> ObjectMapper <span style="color:#06b;font-weight:bold">initiatePrettyObjectMapper</span>() {
ObjectMapper customObjectMapper = <span style="color:#080;font-weight:bold">new</span> ObjectMapper();
customObjectMapper.<span style="color:#369">configure</span>(SerializationFeature.<span style="color:#369">INDENT_OUTPUT</span>, <span style="color:#080;font-weight:bold">true</span>);
<span style="color:#888">// additional indentation for arrays
</span><span style="color:#888"></span> DefaultPrettyPrinter pp = <span style="color:#080;font-weight:bold">new</span> DefaultPrettyPrinter();
pp.<span style="color:#369">indentArraysWith</span>(<span style="color:#080;font-weight:bold">new</span> DefaultIndenter());
customObjectMapper.<span style="color:#369">setDefaultPrettyPrinter</span>(pp);
PropertyFilter userCodeFilter = <span style="color:#080;font-weight:bold">new</span> SimpleBeanPropertyFilter() {
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">serializeAsField</span>(Object pojo, JsonGenerator jgen, SerializerProvider provider, PropertyWriter writer)
<span style="color:#080;font-weight:bold">throws</span> Exception {
<span style="color:#080;font-weight:bold">if</span> (include(writer)) {
<span style="color:#080;font-weight:bold">if</span> (!writer.<span style="color:#369">getName</span>().<span style="color:#369">equals</span>(<span style="color:#d20;background-color:#fff0f0">"code"</span>)) {
writer.<span style="color:#369">serializeAsField</span>(pojo, jgen, provider);
<span style="color:#080;font-weight:bold">return</span>;
}
<span style="color:#888;font-weight:bold">int</span> intValue = ((UserResponse) pojo).<span style="color:#369">code</span>;
<span style="color:#080;font-weight:bold">if</span> (intValue > 0) {
writer.<span style="color:#369">serializeAsField</span>(pojo, jgen, provider);
}
} <span style="color:#080;font-weight:bold">else</span> <span style="color:#080;font-weight:bold">if</span> (!jgen.<span style="color:#369">canOmitFields</span>()) {
writer.<span style="color:#369">serializeAsOmittedField</span>(pojo, jgen, provider);
}
}
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#888;font-weight:bold">boolean</span> <span style="color:#06b;font-weight:bold">include</span>(BeanPropertyWriter writer) {
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">true</span>;
}
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#888;font-weight:bold">boolean</span> <span style="color:#06b;font-weight:bold">include</span>(PropertyWriter writer) {
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">true</span>;
}
};
FilterProvider filters = <span style="color:#080;font-weight:bold">new</span> SimpleFilterProvider().<span style="color:#369">addFilter</span>(<span style="color:#d20;background-color:#fff0f0">"userCodeFilter"</span>, userCodeFilter);
customObjectMapper.<span style="color:#369">setFilterProvider</span>(filters);
<span style="color:#080;font-weight:bold">return</span> customObjectMapper;
}
}
</code></pre></div><h3 id="deserialization">Deserialization</h3>
<h4 id="json-string-parse-error-handling-in-spring-boot">JSON String Parse Error Handling in Spring Boot</h4>
<p>This one is a little tricky. Deserialization of a JSON <code>@RequestParam</code> object can cause parsing errors if the JSON object is not well-formed. The errors thrown in Jackson’s deserialization level just before it’s pushed to Spring Boot occur at that level, so Spring Boot doesn’t catch these errors.</p>
<p>Deserialization of Jackson maps JSON to POJOs and finally returns the expected Java class object. If the JSON is not well-formed, parsing cannot be done and <code>MappingJackson2HttpMessageConverter</code> internally throws a parsing error. Since this exception is not caught by Spring Boot and no object is returned, the REST controller would be unresponsive, having a badly-formed JSON payload.</p>
<p>Here we can override the internal <code>read</code> method of <code>MappingJackson2HttpMessageConverter</code>, hack the <code>ReadJavaType</code> with a <code>customReadJavaType</code> method, and make it return an internal error when the deserialization fails to parse the JSON input, rather than throwing an exception which is not seen or handled by Spring Boot.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">public</span> Object <span style="color:#06b;font-weight:bold">read</span>(Type type, <span style="color:#555">@Nullable</span> Class<?> contextClass, HttpInputMessage inputMessage)
<span style="color:#080;font-weight:bold">throws</span> IOException, HttpMessageNotReadableException {
objectMapper.<span style="color:#369">enable</span>(DeserializationFeature.<span style="color:#369">FAIL_ON_UNKNOWN_PROPERTIES</span>);
JavaType javaType = getJavaType(type, contextClass);
<span style="color:#080;font-weight:bold">return</span> customReadJavaType(javaType, inputMessage);
}
<span style="color:#080;font-weight:bold">private</span> Object <span style="color:#06b;font-weight:bold">customReadJavaType</span>(JavaType javaType, HttpInputMessage inputMessage) <span style="color:#080;font-weight:bold">throws</span> IOException {
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#080;font-weight:bold">if</span> (inputMessage <span style="color:#080;font-weight:bold">instanceof</span> MappingJacksonInputMessage) {
Class<?> deserializationView = ((MappingJacksonInputMessage) inputMessage).<span style="color:#369">getDeserializationView</span>();
<span style="color:#080;font-weight:bold">if</span> (deserializationView != <span style="color:#080;font-weight:bold">null</span>) {
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">this</span>.<span style="color:#369">objectMapper</span>.<span style="color:#369">readerWithView</span>(deserializationView).<span style="color:#369">forType</span>(javaType).
readValue(inputMessage.<span style="color:#369">getBody</span>());
}
}
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">this</span>.<span style="color:#369">objectMapper</span>.<span style="color:#369">readValue</span>(inputMessage.<span style="color:#369">getBody</span>(), javaType);
}
<span style="color:#080;font-weight:bold">catch</span> (InvalidDefinitionException ex) {
<span style="color:#888">//throw new HttpMessageConversionException("Type definition error: " + ex.getType(), ex);
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">"Type definition error"</span>;
}
<span style="color:#080;font-weight:bold">catch</span> (JsonProcessingException ex) {
<span style="color:#888">//throw new HttpMessageNotReadableException("JSON parse error: " + ex.getOriginalMessage(), ex, inputMessage);
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">"JSON parse error"</span>;
}
}
</code></pre></div><p>This way you can return errors occurring at the deserialization level to Spring Boot, which expects a deserialized object but gets a <code>String</code> value which can be caught and translated into a <code>ControllerAdvice</code> handled exception. This also makes it easier to catch JSON parsing errors without using any third party JSON libraries like Gson.</p>
Making sense of XML/JSON items in the shellhttps://www.endpointdev.com/blog/2019/12/making-sense-of-xml-json-in-shell/2019-12-31T00:00:00+00:00Muhammad Najmi bin Ahmad Zabidi
<p><img src="/blog/2019/12/making-sense-of-xml-json-in-shell/image-0.jpg" alt="a shell"></p>
<p>Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.</p>
<p>Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.</p>
<p>These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are <code>xmlstarlet</code> for XML parsing and <code>jq</code> for JSON.</p>
<h3 id="xml">XML</h3>
<p>I have the following XML elements, and I’ll save them to a file called data.xml:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#b06;font-weight:bold"><rss</span> <span style="color:#369">version=</span><span style="color:#d20;background-color:#fff0f0">"2.0"</span>
<span style="color:#369">xmlns:dc=</span><span style="color:#d20;background-color:#fff0f0">"http://purl.org/dc/elements/1.1/"</span>
<span style="color:#369">xmlns:sy=</span><span style="color:#d20;background-color:#fff0f0">"http://purl.org/rss/1.0/modules/syndication/"</span>
<span style="color:#369">xmlns:admin=</span><span style="color:#d20;background-color:#fff0f0">"http://webns.net/mvcb/"</span>
<span style="color:#369">xmlns:rdf=</span><span style="color:#d20;background-color:#fff0f0">"http://www.w3.org/1999/02/22-rdf-syntax-ns#"</span>
<span style="color:#369">xmlns:content=</span><span style="color:#d20;background-color:#fff0f0">"http://purl.org/rss/1.0/modules/content/"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><channel></span>
<span style="color:#b06;font-weight:bold"><title></span>eSolat JAKIM : Waktu Solat Hari Ini<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><link></span>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam<span style="color:#b06;font-weight:bold"></link></span>
<span style="color:#b06;font-weight:bold"><description></span>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"><dc:language></span>ms<span style="color:#b06;font-weight:bold"></dc:language></span>
<span style="color:#b06;font-weight:bold"><dc:creator></span>www.e-solat.gov.my<span style="color:#b06;font-weight:bold"></dc:creator></span>
<span style="color:#b06;font-weight:bold"><dc:rights></span>Copyright JAKIM<span style="color:#b06;font-weight:bold"></dc:rights></span>
<span style="color:#b06;font-weight:bold"><dc:date></span>26-12-2019 00:37:31<span style="color:#b06;font-weight:bold"></dc:date></span>
<span style="color:#b06;font-weight:bold"><admin:generatorAgent</span> <span style="color:#369">rdf:resource=</span><span style="color:#d20;background-color:#fff0f0">"expressionengine"</span> <span style="color:#b06;font-weight:bold">/></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Imsak<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>05:53:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Subuh<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>06:03:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Syuruk<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>07:14:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Zohor<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>13:16:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Asar<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>16:39:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Maghrib<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>19:13:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"><item></span>
<span style="color:#b06;font-weight:bold"><title></span>Isyak<span style="color:#b06;font-weight:bold"></title></span>
<span style="color:#b06;font-weight:bold"><description></span>20:28:00<span style="color:#b06;font-weight:bold"></description></span>
<span style="color:#b06;font-weight:bold"></item></span>
<span style="color:#b06;font-weight:bold"></channel></span>
<span style="color:#b06;font-weight:bold"></rss></span>
</code></pre></div><p>I’ll use xmlstarlet, together with a bunch of the related flags to parse these elements into something which is more eye-friendly.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">xmlstarlet sel -T -t -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"------------------------------"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"Area: "</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m <span style="color:#d20;background-color:#fff0f0">"//channel"</span> -v <span style="color:#d20;background-color:#fff0f0">"link"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"Date: "</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m <span style="color:#d20;background-color:#fff0f0">"//channel"</span> -v <span style="color:#d20;background-color:#fff0f0">"dc:date"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"------------------------------"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -t -m <span style="color:#d20;background-color:#fff0f0">"//item"</span> -o <span style="color:#d20;background-color:#fff0f0">"Time: "</span> -v <span style="color:#d20;background-color:#fff0f0">"title"</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">" -- "</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"Time: "</span> -v <span style="color:#d20;background-color:#fff0f0">"description"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> /path/to/data.xml
</code></pre></div><p>The output looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">------------------------------
Area: Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam
Date: 26-12-2019 00:37:31
------------------------------
Time: Imsak -- Time: 05:53:00
Time: Subuh -- Time: 06:03:00
Time: Syuruk -- Time: 07:14:00
Time: Zohor -- Time: 13:16:00
Time: Asar -- Time: 16:39:00
Time: Maghrib -- Time: 19:13:00
Time: Isyak -- Time: 20:28:00
</code></pre></div><p>I’ll put this in a Bash script, and call it xmlstarlet-time.sh.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#c00;font-weight:bold">#!/bin/bash
</span><span style="color:#c00;font-weight:bold"></span>
<span style="color:#369">XMLPATH</span>=/home/seth/data.xml
<span style="color:#080;font-weight:bold">if</span> [[ -z <span style="color:#369">$1</span> ]]; <span style="color:#080;font-weight:bold">then</span>
<span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"Put the location code"</span>
<span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#369">$0</span><span style="color:#d20;background-color:#fff0f0"> <location code>"</span>
<span style="color:#038">echo</span> -n
<span style="color:#038">exit</span>
<span style="color:#080;font-weight:bold">fi</span>
lynx -source <span style="color:#d20;background-color:#fff0f0">"https://www.e-solat.gov.my/index.php?r=esolatApi/xmlfeed&zon=</span><span style="color:#369">$1</span><span style="color:#d20;background-color:#fff0f0">"</span> > <span style="color:#369">$XMLPATH</span>
xmlstarlet sel -T -t -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"------------------------------"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"Area: "</span> -m <span style="color:#d20;background-color:#fff0f0">"//channel"</span> -v <span style="color:#d20;background-color:#fff0f0">"link"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"Date: "</span> -m <span style="color:#d20;background-color:#fff0f0">"//channel"</span> -v <span style="color:#d20;background-color:#fff0f0">"dc:date"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -o <span style="color:#d20;background-color:#fff0f0">"------------------------------"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -t -m <span style="color:#d20;background-color:#fff0f0">"//item"</span> -o <span style="color:#d20;background-color:#fff0f0">"Time: "</span> -v <span style="color:#d20;background-color:#fff0f0">"title"</span> -o <span style="color:#d20;background-color:#fff0f0">" -- "</span> -o <span style="color:#d20;background-color:#fff0f0">"Time: "</span> -v <span style="color:#d20;background-color:#fff0f0">"description"</span> -n <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> <span style="color:#369">$XMLPATH</span>
</code></pre></div><p>Now, after making it executable with <code>chmod +x xmlstarlet-time.sh</code>, I can just run the script whenever I need the info. In my case, I would type <code>./xmlstarlet-time.sh SGR01</code> in order to get the above information. I got the code (in my case) from the XML URL above. Your use case will likely be different.</p>
<h3 id="json">JSON</h3>
<p>Let’s say I want to grab the latest currency exchange, taking the base of USD from exchangeratesapi.io. I can use curl to get the data.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -s <span style="color:#d20;background-color:#fff0f0">'https://api.exchangeratesapi.io/api/latest?base=USD'</span>
</code></pre></div><p>Which will return:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{<span style="color:#b06;font-weight:bold">"rates"</span>:{<span style="color:#b06;font-weight:bold">"CAD"</span>:<span style="color:#00d;font-weight:bold">1.3160649819</span>,<span style="color:#b06;font-weight:bold">"HKD"</span>:<span style="color:#00d;font-weight:bold">7.7879061372</span>,<span style="color:#b06;font-weight:bold">"ISK"</span>:<span style="color:#00d;font-weight:bold">122.3826714801</span>,<span style="color:#b06;font-weight:bold">"PHP"</span>:<span style="color:#00d;font-weight:bold">50.8402527076</span>,<span style="color:#b06;font-weight:bold">"DKK"</span>:<span style="color:#00d;font-weight:bold">6.7429602888</span>,<span style="color:#b06;font-weight:bold">"HUF"</span>:<span style="color:#00d;font-weight:bold">299.4223826715</span>,<span style="color:#b06;font-weight:bold">"CZK"</span>:<span style="color:#00d;font-weight:bold">23.0009025271</span>,<span style="color:#b06;font-weight:bold">"GBP"</span>:<span style="color:#00d;font-weight:bold">0.7719584838</span>,<span style="color:#b06;font-weight:bold">"RON"</span>:<span style="color:#00d;font-weight:bold">4.3131768953</span>,<span style="color:#b06;font-weight:bold">"SEK"</span>:<span style="color:#00d;font-weight:bold">9.4361913357</span>,<span style="color:#b06;font-weight:bold">"IDR"</span>:<span style="color:#00d;font-weight:bold">13985.0180505415</span>,<span style="color:#b06;font-weight:bold">"INR"</span>:<span style="color:#00d;font-weight:bold">71.2567689531</span>,<span style="color:#b06;font-weight:bold">"BRL"</span>:<span style="color:#00d;font-weight:bold">4.0835740072</span>,<span style="color:#b06;font-weight:bold">"RUB"</span>:<span style="color:#00d;font-weight:bold">62.0877256318</span>,<span style="color:#b06;font-weight:bold">"HRK"</span>:<span style="color:#00d;font-weight:bold">6.719765343</span>,<span style="color:#b06;font-weight:bold">"JPY"</span>:<span style="color:#00d;font-weight:bold">109.3772563177</span>,<span style="color:#b06;font-weight:bold">"THB"</span>:<span style="color:#00d;font-weight:bold">30.155234657</span>,<span style="color:#b06;font-weight:bold">"CHF"</span>:<span style="color:#00d;font-weight:bold">0.9817689531</span>,<span style="color:#b06;font-weight:bold">"EUR"</span>:<span style="color:#00d;font-weight:bold">0.9025270758</span>,<span style="color:#b06;font-weight:bold">"MYR"</span>:<span style="color:#00d;font-weight:bold">4.1364620939</span>,<span style="color:#b06;font-weight:bold">"BGN"</span>:<span style="color:#00d;font-weight:bold">1.7651624549</span>,<span style="color:#b06;font-weight:bold">"TRY"</span>:<span style="color:#00d;font-weight:bold">5.9561371841</span>,<span style="color:#b06;font-weight:bold">"CNY"</span>:<span style="color:#00d;font-weight:bold">7.0074909747</span>,<span style="color:#b06;font-weight:bold">"NOK"</span>:<span style="color:#00d;font-weight:bold">8.94566787</span>,<span style="color:#b06;font-weight:bold">"NZD"</span>:<span style="color:#00d;font-weight:bold">1.5086642599</span>,<span style="color:#b06;font-weight:bold">"ZAR"</span>:<span style="color:#00d;font-weight:bold">14.1935018051</span>,<span style="color:#b06;font-weight:bold">"USD"</span>:<span style="color:#00d;font-weight:bold">1.0</span>,<span style="color:#b06;font-weight:bold">"MXN"</span>:<span style="color:#00d;font-weight:bold">18.9626353791</span>,<span style="color:#b06;font-weight:bold">"SGD"</span>:<span style="color:#00d;font-weight:bold">1.3553249097</span>,<span style="color:#b06;font-weight:bold">"AUD"</span>:<span style="color:#00d;font-weight:bold">1.4457581227</span>,<span style="color:#b06;font-weight:bold">"ILS"</span>:<span style="color:#00d;font-weight:bold">3.4714801444</span>,<span style="color:#b06;font-weight:bold">"KRW"</span>:<span style="color:#00d;font-weight:bold">1162.1931407942</span>,<span style="color:#b06;font-weight:bold">"PLN"</span>:<span style="color:#00d;font-weight:bold">3.8445848375</span>},<span style="color:#b06;font-weight:bold">"base"</span>:<span style="color:#d20;background-color:#fff0f0">"USD"</span>,<span style="color:#b06;font-weight:bold">"date"</span>:<span style="color:#d20;background-color:#fff0f0">"2019-12-24"</span>}
</code></pre></div><p>Using jq, we can format the information more readably:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -s <span style="color:#d20;background-color:#fff0f0">'https://api.exchangeratesapi.io/api/latest?base=USD'</span> | jq
{
<span style="color:#d20;background-color:#fff0f0">"rates"</span>: {
<span style="color:#d20;background-color:#fff0f0">"CAD"</span>: 1.3160649819,
<span style="color:#d20;background-color:#fff0f0">"HKD"</span>: 7.7879061372,
<span style="color:#d20;background-color:#fff0f0">"ISK"</span>: 122.3826714801,
<span style="color:#d20;background-color:#fff0f0">"PHP"</span>: 50.8402527076,
<span style="color:#d20;background-color:#fff0f0">"DKK"</span>: 6.7429602888,
<span style="color:#d20;background-color:#fff0f0">"HUF"</span>: 299.4223826715,
<span style="color:#d20;background-color:#fff0f0">"CZK"</span>: 23.0009025271,
<span style="color:#d20;background-color:#fff0f0">"GBP"</span>: 0.7719584838,
<span style="color:#d20;background-color:#fff0f0">"RON"</span>: 4.3131768953,
<span style="color:#d20;background-color:#fff0f0">"SEK"</span>: 9.4361913357,
<span style="color:#d20;background-color:#fff0f0">"IDR"</span>: 13985.0180505415,
<span style="color:#d20;background-color:#fff0f0">"INR"</span>: 71.2567689531,
<span style="color:#d20;background-color:#fff0f0">"BRL"</span>: 4.0835740072,
<span style="color:#d20;background-color:#fff0f0">"RUB"</span>: 62.0877256318,
<span style="color:#d20;background-color:#fff0f0">"HRK"</span>: 6.719765343,
<span style="color:#d20;background-color:#fff0f0">"JPY"</span>: 109.3772563177,
<span style="color:#d20;background-color:#fff0f0">"THB"</span>: 30.155234657,
<span style="color:#d20;background-color:#fff0f0">"CHF"</span>: 0.9817689531,
<span style="color:#d20;background-color:#fff0f0">"EUR"</span>: 0.9025270758,
<span style="color:#d20;background-color:#fff0f0">"MYR"</span>: 4.1364620939,
<span style="color:#d20;background-color:#fff0f0">"BGN"</span>: 1.7651624549,
<span style="color:#d20;background-color:#fff0f0">"TRY"</span>: 5.9561371841,
<span style="color:#d20;background-color:#fff0f0">"CNY"</span>: 7.0074909747,
<span style="color:#d20;background-color:#fff0f0">"NOK"</span>: 8.94566787,
<span style="color:#d20;background-color:#fff0f0">"NZD"</span>: 1.5086642599,
<span style="color:#d20;background-color:#fff0f0">"ZAR"</span>: 14.1935018051,
<span style="color:#d20;background-color:#fff0f0">"USD"</span>: 1,
<span style="color:#d20;background-color:#fff0f0">"MXN"</span>: 18.9626353791,
<span style="color:#d20;background-color:#fff0f0">"SGD"</span>: 1.3553249097,
<span style="color:#d20;background-color:#fff0f0">"AUD"</span>: 1.4457581227,
<span style="color:#d20;background-color:#fff0f0">"ILS"</span>: 3.4714801444,
<span style="color:#d20;background-color:#fff0f0">"KRW"</span>: 1162.1931407942,
<span style="color:#d20;background-color:#fff0f0">"PLN"</span>: 3.8445848375
},
<span style="color:#d20;background-color:#fff0f0">"base"</span>: <span style="color:#d20;background-color:#fff0f0">"USD"</span>,
<span style="color:#d20;background-color:#fff0f0">"date"</span>: <span style="color:#d20;background-color:#fff0f0">"2019-12-24"</span>
}
</code></pre></div><p>Next, I can make use of the tool in my shell script.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#c00;font-weight:bold">#!/bin/bash -l
</span><span style="color:#c00;font-weight:bold"></span>
<span style="color:#369">RED</span>=<span style="color:#d20;background-color:#fff0f0">'\033[0;31m'</span>
<span style="color:#369">YELLOW</span>=<span style="color:#d20;background-color:#fff0f0">'\033[1;33m'</span>
<span style="color:#369">GREEN</span>=<span style="color:#d20;background-color:#fff0f0">'\033[0;32m'</span>
<span style="color:#369">NC</span>=<span style="color:#d20;background-color:#fff0f0">'\033[0m'</span>
<span style="color:#080;font-weight:bold">if</span> [ -z <span style="color:#d20;background-color:#fff0f0">`</span>which jq<span style="color:#d20;background-color:#fff0f0">`</span> ]; <span style="color:#080;font-weight:bold">then</span>
<span style="color:#038">printf</span> <span style="color:#d20;background-color:#fff0f0">"You need to install jq, a JSON parsing tool \n"</span>
<span style="color:#038">exit</span>
<span style="color:#080;font-weight:bold">fi</span>
<span style="color:#369">sourcemoney</span>=<span style="color:#080;font-weight:bold">$(</span><span style="color:#038">echo</span> <span style="color:#369">$2</span> | tr <span style="color:#d20;background-color:#fff0f0">'[:lower:]'</span> <span style="color:#d20;background-color:#fff0f0">'[:upper:]'</span><span style="color:#080;font-weight:bold">)</span>
<span style="color:#369">target</span>=<span style="color:#080;font-weight:bold">$(</span><span style="color:#038">echo</span> <span style="color:#369">$3</span> | tr <span style="color:#d20;background-color:#fff0f0">'[:lower:]'</span> <span style="color:#d20;background-color:#fff0f0">'[:upper:]'</span><span style="color:#080;font-weight:bold">)</span>
<span style="color:#369">sumber</span>=<span style="color:#080;font-weight:bold">$(</span>curl -s <span style="color:#d20;background-color:#fff0f0">"https://api.exchangeratesapi.io/api/latest?base=</span><span style="color:#369">$sourcemoney</span><span style="color:#d20;background-color:#fff0f0">"</span> | jq . | grep -i <span style="color:#369">$target</span> | awk -F <span style="color:#d20;background-color:#fff0f0">':|,'</span> <span style="color:#d20;background-color:#fff0f0">'{ print $2 }'</span><span style="color:#080;font-weight:bold">)</span>
<span style="color:#369">jumlah</span>=<span style="color:#080;font-weight:bold">$(</span><span style="color:#038">printf</span> <span style="color:#d20;background-color:#fff0f0">"%f*%f\n"</span> <span style="color:#369">$1</span> <span style="color:#369">$sumber</span> | bc<span style="color:#080;font-weight:bold">)</span>
<span style="color:#038">printf</span> <span style="color:#d20;background-color:#fff0f0">"Price per unit: </span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">GREEN</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">1 </span><span style="color:#369">$sourcemoney</span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">NC</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0"> = </span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">YELLOW</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#369">$target</span><span style="color:#d20;background-color:#fff0f0"> %.2f</span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">NC</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">\n"</span> <span style="color:#369">$sumber</span>
<span style="color:#038">echo</span> -e <span style="color:#d20;background-color:#fff0f0">"Source money: </span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">YELLOW</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#369">$sourcemoney</span><span style="color:#d20;background-color:#fff0f0"> </span><span style="color:#369">$1</span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">NC</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">"</span>
<span style="color:#038">echo</span> -n
<span style="color:#038">printf</span> <span style="color:#d20;background-color:#fff0f0">"Total money after the conversion: </span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">YELLOW</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#369">$target</span><span style="color:#d20;background-color:#fff0f0"> %.2f </span><span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">NC</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">\n"</span> <span style="color:#369">$jumlah</span>
</code></pre></div><p>Then I can save the script into a file called moneychanger-with-api.sh and make it executable with <code>chmod +x moneychanger-with-api.sh</code>.</p>
<p>And now the script will do the parsing for me, without the need for a browser.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ ./moneychanger-with-api.sh 100 usd eur
Price per unit: 1 USD = EUR 0.90
Source money: USD 100
Total money after the conversion: EUR 90.25
$ ./moneychanger-with-api.sh 100 eur sgd
Price per unit: 1 EUR = SGD 1.50
Source money: EUR 100
Total money after the conversion: SGD 150.17
</code></pre></div>
MediaWiki extension.json change in 1.25https://www.endpointdev.com/blog/2015/10/mediawiki-extensionjson-change-in-125/2015-10-17T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2015/10/mediawiki-extensionjson-change-in-125/image-0-big.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="/blog/2015/10/mediawiki-extensionjson-change-in-125/image-0.png"/></a></div>
<p>I recently released a new version of the
<a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a>
<a href="https://www.mediawiki.org/wiki/Extension:RequestTracker">“Request Tracker” extension</a>, which provides
a nice interface to your
<a href="https://bestpractical.com/rt/">RequestTracker</a> instance, allowing you to view the tickets right
inside of your wiki. There are two major changes I want to point out. First, the name has
changed from <strong>“RT”</strong> to <strong>“RequestTracker”</strong>. Second, it is using the brand-new way of writing
MediaWiki extensions, featuring the extension.json file.</p>
<p>The name change rationale is easy to understand: I wanted it to be more intuitive and easier to find. A search for
“RT” on mediawiki.org ends up finding references to the WikiMedia RequestTracker system,
while a
<a href="https://www.mediawiki.org/w/index.php?search=RequestTracker">search for “RequestTracker”</a>
finds the new extension right away. Also, the name was
too short and failed to indicate to people what it was. The “rt” tag used by the extension stays
the same. However, to produce a table showing all open tickets for user “alois”, you still write:</p>
<pre tabindex="0"><code><rt u='alois'></rt>
</code></pre><p>The other major change was to modernize it. As of version 1.25 of MediaWiki,
extensions are encouraged to use a new system to register themselves with MediaWiki. Previously,
an extension would have a PHP file named after the extension that was responsible for doing
the registration and setup—usually by mucking with global variables! There was no
way for MediaWiki to figure out what the extension was going to do without parsing the entire file, and
thereby activating the extension. The new method relies on a standard JSON file called
extension.json. Thus, in the RequestTracker extension, the file RequestTracker.php has
been replaced with the much smaller and simpler extension.json file.</p>
<p>Before going further, it should be pointed out that this is a big change for extensions,
and was not without controversy. However, as of MediaWiki 1.25 it is the new standard for extensions, and I think
the project is better for it. The old way will continue to be supported, but extension
authors should be using extension.json for new extensions, and converting existing
ones over. As an aside, this is another indication that <a href="http://json.org/">JSON</a> has won the data format war.
Sorry, <a href="http://www.json.org/xml.html">XML</a>, you were too big and bloated. Nice try <a href="http://yaml.org/">YAML</a>, but you were a little <em>too</em> free-form. JSON isn’t perfect,
but it is the best solution of its kind. For further evidence, see Postgres,
which now has <a href="https://www.postgresql.org/docs/current/static/datatype-json.html">outstanding support for JSON and JSONB</a>. I added support for YAML output to EXPLAIN
in Postgres some years back, but nobody (including me!) was excited enough about YAML to do
more than that with it. :)</p>
<p>The extension.json file asks you to fill in some standard metadata fields about the extension, which are then used by MediaWiki to register and set up the extension. Another advantage
of doing it this way is that you no longer need to add a bunch of ugly include_once()
function calls to your LocalSettings.php file. Now, you simply call the name of the extension as an argument to the wfLoadExtension() function. You can even load multiple extensions at once with wfLoadExtensions():</p>
<pre tabindex="0"><code>## Old way:
require_once("$IP/extensions/RequestTracker/RequestTracker.php");
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
## New way:
wfLoadExtension( 'RequestTracker' );
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
## Or even load three extensions at once:
wfLoadExtensions( array( 'RequestTracker', 'Balloons', 'WikiEditor' ) );
$wgRequestTrackerURL = 'https://rt.endpoint.com/Ticket/Display.html?id';
</code></pre><p>Note that configuration changes specific to the extension still must be defined in
the LocalSettings.php file.</p>
<p>So what should go into the extension.json file? The
<a href="https://www.mediawiki.org/wiki/Manual:Developing_extensions">extension development documentation</a> has some suggested
fields, and you can also view the <a href="https://phabricator.wikimedia.org/diffusion/MW/browse/master/docs/extension.schema.v2.json">canonical extension.json schema</a>. Let’s take a quick look at the RequestTracker/extension.json file. Don’t worry, it’s not
too long.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl">{
<span style="color:#d20;background-color:#fff0f0">"manifest_version"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>,
<span style="color:#d20;background-color:#fff0f0">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"parserhook"</span>,
<span style="color:#d20;background-color:#fff0f0">"author"</span>: [
<span style="color:#d20;background-color:#fff0f0">"Greg Sabino Mullane"</span>
],
<span style="color:#d20;background-color:#fff0f0">"version"</span>: <span style="color:#d20;background-color:#fff0f0">"2.0"</span>,
<span style="color:#d20;background-color:#fff0f0">"url"</span>: <span style="color:#d20;background-color:#fff0f0">"https://www.mediawiki.org/wiki/Extension:RequestTracker"</span>,
<span style="color:#d20;background-color:#fff0f0">"descriptionmsg"</span>: <span style="color:#d20;background-color:#fff0f0">"rt-desc"</span>,
<span style="color:#d20;background-color:#fff0f0">"license-name"</span>: <span style="color:#d20;background-color:#fff0f0">"PostgreSQL"</span>,
<span style="color:#d20;background-color:#fff0f0">"requires"</span> : {
<span style="color:#d20;background-color:#fff0f0">"MediaWiki"</span>: <span style="color:#d20;background-color:#fff0f0">">= 1.25.0"</span>
},
<span style="color:#d20;background-color:#fff0f0">"AutoloadClasses"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>: <span style="color:#d20;background-color:#fff0f0">"RequestTracker_body.php"</span>
},
<span style="color:#d20;background-color:#fff0f0">"Hooks"</span>: {
<span style="color:#d20;background-color:#fff0f0">"ParserFirstCallInit"</span> : [
<span style="color:#d20;background-color:#fff0f0">"RequestTracker::wfRequestTrackerParserInit"</span>
]
},
<span style="color:#d20;background-color:#fff0f0">"MessagesDirs"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker"</span>: [
<span style="color:#d20;background-color:#fff0f0">"i18n"</span>
]
},
<span style="color:#d20;background-color:#fff0f0">"config"</span>: {
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_URL"</span>: <span style="color:#d20;background-color:#fff0f0">"http://rt.example.com/Ticket/Display.html?id"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_DBconn"</span>: <span style="color:#d20;background-color:#fff0f0">"user=rt dbname=rt"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Formats"</span>: [],
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Cachepage"</span>: <span style="color:#00d;font-weight:bold">0</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Useballoons"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Active"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_Sortable"</span>: <span style="color:#00d;font-weight:bold">1</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_LASTUPDATED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_LASTUPDATED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_CREATED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_CREATED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_RESOLVED"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_RESOLVED2"</span>: <span style="color:#d20;background-color:#fff0f0">"FMMonth DD, YYYY"</span>,
<span style="color:#d20;background-color:#fff0f0">"RequestTracker_TIMEFORMAT_NOW"</span>: <span style="color:#d20;background-color:#fff0f0">"FMHH:MI AM FMMonth DD, YYYY"</span>
}
}
</code></pre></div><p>The first field in the file is manifest_version, and simply indicates the extension.json schema version. Right now it is marked as required, and I figure it does no harm to throw it in there. The name field should be self-explanatory, and should match your CamelCase extension name, which will also be the subdirectory where your extension will live under the extensions/ directory.
The type field simply tells what kind of extension this is, and is mostly used to determine which section of the Special:Version page an extension will appear under. The author is also self-explanatory, but note that this is a JSON array, allowing for multiple items if needed. The version and url are highly recommended. For the license, I chose the dirt-simple <a href="https://opensource.org/licenses/postgresql">PostgreSQL license</a>, whose only fault is its name. The descriptionmsg is what will appear as the description of the extension on the Special:Version page. As it is a user-facing text, it is subject to
internationalization, and thus <strong>rt-desc</strong> is converted to your current language by looking up the language file inside of the extension’s i18n directory.</p>
<p>The requires field only supports a “MediaWiki” subkey at the moment. In this case, I have it
set to require at least version 1.25 of MediaWiki—as anything lower will not even be able to read
this file! The AutoloadClasses key is the new way of loading code needed by the extension. As before, this should be stored in a php file with the name of the extension, an underscore, and the word “body” (e.g. RequestTracker_body.php). This file contains all of the functions that perform the actual work of the extension.</p>
<p>The Hooks field is one of the big advantages of the new extension.json format. Rather than
worrying about modifying global variables, you can simply let MediaWiki know what functions
are associated with which hooks. In the case of RequestTracker, we need to do some magic whenever
a <strong><rt></strong> tag is encountered. To that end, we need to instruct the parser that we will be handling
any <strong><rt></strong> tags it encounters, and also tell it what to do when it finds them. Those details
are inside the wfRequestTrackerParserInit function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">function</span> <span style="color:#06b;font-weight:bold">wfRequestTrackerParserInit</span>( Parser <span style="color:#369">$parser</span> ) {
<span style="color:#369">$parser</span>-><span style="color:#369">setHook</span>( <span style="color:#d20;background-color:#fff0f0">'rt'</span>, <span style="color:#d20;background-color:#fff0f0">'RequestTracker::wfRequestTrackerRender'</span> );
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">true</span>;
}
</code></pre></div><p>The config field provides a list of all user-configurable variables used by the extension, along with their default values.</p>
<p>The MessagesDirs field tells MediaWiki where to find your localization files. This
should always be in the standard place, the i18n directory.
Inside that directory are localization files, one for each language, as well as a special
file named qqq.json, which gives information about each message
string as a guide to translators. The language files are of the format “xxx.json”, where
“xxx” is the language code. For example, RequestTracker/i18n/en.json
contains English versions of all the messages used by the extension. The i18n files look like this:</p>
<pre tabindex="0"><code>$ cat en.json
{
"rt-desc" : "Fancy interface to RequestTracker using <code>&lt;rt&gt;</code> tag",
"rt-inactive" : "The RequestTracker extension is not active",
"rt-badcontent" : "Invalid content args: must be a simple word. You tried: <b>$1</b>",
"rt-badquery" : "The RequestTracker extension encountered an error when talking to the RequestTracker database",
"rt-badlimit" : "Invalid LIMIT (l) arg: must be a number. You tried: <b>$1</b>",
"rt-badorderby" : "Invalid ORDER BY (ob) arg: must be a standard field (see documentation). You tried: <b>$1</b>",
"rt-badstatus" : "Invalid status (s) arg: must be a standard field (see documentation). You tried: <b>$1</b>",
"rt-badcfield" : "Invalid custom field arg: must be a simple word. You tried: <b>$1</b>",
"rt-badqueue" : "Invalid queue (q) arg: must be a simple word. You tried: <b>$1</b>",
"rt-badowner" : "Invalid owner (o) arg: must be a valud username. You tried: <b>$1</b>",
"rt-nomatches" : "No matching RequestTracker tickets were found"
}
$ cat fr.json
{
"@metadata": {
"authors": [
"Josh Tolley"
]
},
"rt-desc" : "Interface sophistiquée de RequestTracker avec l'élement <code>&lt;rt&gt;</code>.",
"rt-inactive" : "Le module RequestTracker n'est pas actif.",
"rt-badcontent" : "Paramètre de contenu « $1 » est invalide: cela doit être un mot simple.",
"rt-badquery" : "Le module RequestTracker ne peut pas contacter sa base de données.",
"rt-badlimit" : "Paramètre à LIMIT (l) « $1 » est invalide: cela doit être un nombre entier.",
"rt-badorderby : "Paramètre à ORDER BY (ob) « $1 » est invalide: cela doit être un champs standard. Voir le manuel utilisateur.",
"rt-badstatus" : "Paramètre de status (s) « $1 » est invalide: cela doit être un champs standard. Voir le manuel utilisateur.",
"rt-badcfield" : "Paramètre de champs personalisé « $1 » est invalide: cela doit être un mot simple.",
"rt-badqueue" : "Paramètre de queue (q) « $1 » est invalide: cela doit être un mot simple.",
"rt-badowner" : "Paramètre de propriétaire (o) « $1 » est invalide: cela doit être un mot simple.",
"rt-nomatches" : "Aucun ticket trouvé"
}
</code></pre><p>One other small change I made to the extension was to allow both ticket numbers and queue names
to be used inside of the tag. To view a specific ticket, one was always able to do this:</p>
<pre tabindex="0"><code><rt>6567</rt>
</code></pre><p>This would produce the text “RT #6567”, with information on the ticket available on mouseover,
and hyperlinked to the ticket inside of RT. However, I often found myself using this extension
to view all the open tickets in a certain queue like this:</p>
<pre tabindex="0"><code><rt q="dyson"></rt>
</code></pre><p>It seems easier to simply add the queue name inside the tags, so in this new version
one can simply do this:</p>
<pre tabindex="0"><code><rt>dyson</rt>
</code></pre><p>If you are running MediaWiki 1.25 or better, try out the new RequestTracker extension! If
you are stuck on an older version, use the RT extension and upgrade as soon as you can. :)</p>
DBD::Pg escaping placeholders with backslasheshttps://www.endpointdev.com/blog/2015/01/dbdpg-escaping-placeholders-with/2015-01-12T00:00:00+00:00Greg Sabino Mullane
<div class="separator" style="clear: both; float: right; margin-bottom: 1em; text-align: center;"><a href="/blog/2015/01/dbdpg-escaping-placeholders-with/image-0-big.jpeg" imageanchor="1" style="clear: right; margin-left: 1em;"><img border="0" src="/blog/2015/01/dbdpg-escaping-placeholders-with/image-0.jpeg"/></a>
<br/><small><a href="https://flic.kr/p/4vUdLJ">Image</a> by <a href="https://www.flickr.com/photos/spine/">Rick Audet</a></small></div>
<p>The popularity of using JSON and <a href="https://www.depesz.com/2014/03/25/waiting-for-9-4-introduce-jsonb-a-structured-format-for-storing-json/">JSONB</a> within Postgres has forced a solution to the problem of question mark overload. JSON (as well as hstore) uses the question mark as an operator in its queries, and Perl DBI (esp. DBD::Pg) uses the question mark to indicate a placeholder. <a href="http://search.cpan.org/dist/DBD-Pg/">Version 3.5.0 of DBD::Pg</a> has solved this by allowing the use of a backslash character before the question mark, to indicate it is NOT a placeholder. We will see some code samples after establishing a little background.</p>
<p>First, what are placeholders? They are special characters within a SQL statement that allow you to defer adding actual values until a later time. This has a number of advantages. First, it completely removes the need to worry about quoting your values. Second, it allows efficient re-use of queries. Third, it reduces network traffic as you do not need to send the entire query each time it is re-run. Fourth, it can allow for seamless translation of data types from Postgres to your client language and back again (for example, DBD::Pg translates easily between Perl arrays and Postgres arrays). There are three styles of placeholders supported by DBD::Pg—question marks, dollar-signs, and colon-names.</p>
<p>Next, what are Postgres operators? They are special symbols withing a SQL statement that perform some action using as inputs the strings to the left and right side of them. It sounds more complicated than it is. Take this query:</p>
<pre tabindex="0"><code>SELECT count(*) FROM pg_class WHERE relpages > 24;
</code></pre><p>In this case, the operator is “>”—the greater than sign. It compares the things on its left (in this case, the value of the relpages column) with the things on its right (in this case, the number 24). The operator will return true or false—in this case, it will return true only if the value on its left is larger than the value on its right. Postgres is extremely extensible, which means it is easy to add all types of new things to it. Adding your own operator is fairly easy. Here’s an example that duplicates the greater-than operator, but with a ? symbol:</p>
<pre tabindex="0"><code>CREATE OPERATOR ? (procedure=int4gt, leftarg=integer, rightarg=integer);
</code></pre><p>Now the operator is ready to go. You should be able to run queries like this:</p>
<pre tabindex="0"><code>SELECT count(*) FROM pg_class WHERE relpages ? 24;
</code></pre><p>The list of characters that can make up an operator is fairly small. The <a href="https://www.postgresql.org/docs/9.4/static/sql-createoperator.html">documentation</a> has the detailed rules, but the basic list is <strong>+ - * / < > = ~ ! @ # % ^ & | ` ?</strong>. Note that an operator can consist of more than one character, for example, <strong>>=</strong></p>
<p>A question mark inside a SQL query can be both a placeholder and an operator, and the driver has no real way to figure out which is which. The first real use of a question mark as an operator was with the <a href="https://www.postgresql.org/docs/current/static/functions-geometry.html">geometric operators</a> and then with the <a href="https://www.postgresql.org/docs/current/static/hstore.html">hstore module</a>, which allows storing and querying of key/value pairs. It uses a lone question mark to determine if a given value appears as a key in a hstore column. For example, if the goal is to find all rows in which an hstore column contains the value foobar, the SQL would be:</p>
<pre tabindex="0"><code>SELECT * FROM mytable WHERE myhstorecol ? 'foobar';
</code></pre><p>However, if you were to try this via a Perl script using the question-mark placeholder style, DBD::Pg would get confused (and rightly so):</p>
<pre tabindex="0"><code>$sth = $dbh->prepare('SELECT * FROM mytable WHERE myhstorecol ? ?');
$sth->execute('foobar');
DBD::Pg::st execute failed: called with 1 bind variables when 2 are needed
</code></pre><p>Trying to use another placeholder style still does not work, as DBD::Pg still picks it up as a possible placeholder</p>
<pre tabindex="0"><code>$sth = $dbh->prepare('SELECT * FROM mytable WHERE myhstorecol ? $1');
$sth->execute('foobar');
Cannot mix placeholder styles "?" and "$1"
</code></pre><p>A few years ago, a solution was developed: by setting the database handle attribute “pg_placeholder_dollaronly” to true, DBD::Pg will ignore the question mark and only treat dollar-sign numbers as placeholders:</p>
<pre tabindex="0"><code>$dbh->{pg_placeholder_dollaronly} = 1;
$sth = $dbh->prepare('SELECT * FROM mytable WHERE myhstorecol ? $1');
$sth->execute('foobar');
## No error!
</code></pre><p>Then came JSON and JSONB. Just like hstore, they have three operators with question marks in them: ?, ?& and ?|—all of which will prevent the use of question-mark placeholders. However, some frameworks and supporting modules (e.g. SQL::Abstract and DBIx::Class) only support the question mark style of placeholder! Hence, another solution was needed. After some <a href="http://codeverge.com/perl.dbi.users/escaping-placeholders-take-2/2026098">discussion</a> on the dbi-users list, it was agreed that a backslash before a placeholder character would allow that character to be “escaped” and sent as-is to the database (minus the backslash). Thus, as of version 3.5.0 of DBD::Pg, the above query can be written as:</p>
<pre tabindex="0"><code>use DBD::Pg 3.5.0;
$SQL = "SELECT * FROM mytable WHERE hstorecol \\? ?");
$sth = $dbh->prepare($SQL);
$sth->execute('foobar');
# No error!
$SQL = "SELECT * FROM mytable2 WHERE jsoncol \\? ?");
$sth = $dbh->prepare($SQL);
$sth->execute('foobar');
# Still no error!
</code></pre><p>So, a fairly elegant solution. The only caveat is to beware of single and double quotes. The latter require two backslashes, of course. I recommend you always use double quotes and get in the habit of consistently using double backslashes. Not only will you thus never have to worry about single-vs-double, but it adds a nice little visual garnish to help that important backslash trick stand out a little more.</p>
<p>Much thanks to Tim Bunce for reporting this issue, herding it through dbi-users, and helping write the final DBD::Pg solution and code!</p>
PostgreSQL as NoSQL with Data Validationhttps://www.endpointdev.com/blog/2013/06/postgresql-as-nosql-with-data-validation/2013-06-03T00:00:00+00:00Szymon Lipiński
<p>PostgreSQL is a relational database with many great features. There are also many so called NoSQL databases, some of them, like CouchDB, are document databases. However the document in CouchDB is automatically enhanced with a “_id” field, if it is not present. When you want to get this one document, you can use this “_id” field—it behaves exactly like the primary key from relational databases. PostgreSQL stores data in tables’ rows while CouchDB stores data as JSON documents. On one hand CouchDB seems like a great solution, as you can have all the different data from different PostgreSQL tables in just one JSON document. This flexibility comes with a cost of no constraints on the data structure, which can be really appealing at the first moment and really frustrating when you have a huge database and some of the documents contain bad values or there are missing some fields.</p>
<p>PostgreSQL 9.3 comes with great features which can turn it into a NoSQL database, with full transaction support, storing JSON documents with constraints on the fields data.</p>
<h3 id="simple-example">Simple Example</h3>
<p>I will show how to do it using a very simple example of a table with products. Each product has a name, description, some id number, price, currency and number of products we have in stock.</p>
<h3 id="postgresql-version">PostgreSQL Version</h3>
<p>The simple table in PostgreSQL can look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb"> </span><span style="color:#038">SERIAL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>name<span style="color:#bbb"> </span><span style="color:#038">TEXT</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>description<span style="color:#bbb"> </span><span style="color:#038">TEXT</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>price<span style="color:#bbb"> </span><span style="color:#038">DECIMAL</span>(<span style="color:#00d;font-weight:bold">10</span>,<span style="color:#00d;font-weight:bold">2</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span>currency<span style="color:#bbb"> </span><span style="color:#038">TEXT</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>in_stock<span style="color:#bbb"> </span><span style="color:#038">INTEGER</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>This table allows us to insert products like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(name,<span style="color:#bbb"> </span>description,<span style="color:#bbb"> </span>price,<span style="color:#bbb"> </span>currency,<span style="color:#bbb"> </span>in_stock)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'shoes'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'blue shoes'</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'dollars'</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">5</span>);<span style="color:#bbb">
</span></code></pre></div><p>Unfortunately the above table also allows for adding rows missing some important information:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(name,<span style="color:#bbb"> </span>description,<span style="color:#bbb"> </span>price,<span style="color:#bbb"> </span>currency,<span style="color:#bbb"> </span>in_stock)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">''</span>,<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">null</span>,<span style="color:#bbb"> </span>-<span style="color:#00d;font-weight:bold">20</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'handa'</span>,<span style="color:#bbb"> </span>-<span style="color:#00d;font-weight:bold">42</span>);<span style="color:#bbb">
</span></code></pre></div><p>This should be fixed by adding constraints in the database. Assume that we want to always have unique not empty name, not empty description, non negative price and in_stock, and the currency should always be dollars. The table with such constraints is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb"> </span><span style="color:#038">SERIAL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>name<span style="color:#bbb"> </span><span style="color:#038">TEXT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>description<span style="color:#bbb"> </span><span style="color:#038">TEXT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>price<span style="color:#bbb"> </span><span style="color:#038">DECIMAL</span>(<span style="color:#00d;font-weight:bold">10</span>,<span style="color:#00d;font-weight:bold">2</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>currency<span style="color:#bbb"> </span><span style="color:#038">TEXT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>in_stock<span style="color:#bbb"> </span><span style="color:#038">INTEGER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">length</span>(name)<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(description<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">length</span>(description)<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(price<span style="color:#bbb"> </span>>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">0</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(currency<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'dollars'</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(in_stock<span style="color:#bbb"> </span>>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb"> </span>><span style="color:#bbb">
</span></code></pre></div><p>Now all the operations, like adding or modifying a row, which violate any of those constraints, just fail. Let’s check:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(name,<span style="color:#bbb"> </span>description,<span style="color:#bbb"> </span>price,<span style="color:#bbb"> </span>currency,<span style="color:#bbb"> </span>in_stock)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'shoes'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'blue shoes'</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'dollars'</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">5</span>);<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(name,<span style="color:#bbb"> </span>description,<span style="color:#bbb"> </span>price,<span style="color:#bbb"> </span>currency,<span style="color:#bbb"> </span>in_stock)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'shoes'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'blue shoes'</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'dollars'</span>,<span style="color:#bbb"> </span>-<span style="color:#00d;font-weight:bold">1</span>);<span style="color:#bbb">
</span><span style="color:#bbb"></span>ERROR:<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">new</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">row</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">for</span><span style="color:#bbb"> </span>relation<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"products"</span><span style="color:#bbb"> </span>violates<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">check</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">constraint</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"products_in_stock_check"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>DETAIL:<span style="color:#bbb"> </span>Failing<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">row</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">contains</span><span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">2</span>,<span style="color:#bbb"> </span>shoes,<span style="color:#bbb"> </span>blue<span style="color:#bbb"> </span>shoes,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb"> </span>dollars,<span style="color:#bbb"> </span>-<span style="color:#00d;font-weight:bold">1</span>).<span style="color:#bbb">
</span></code></pre></div><h3 id="nosql-version">NoSQL Version</h3>
<p>In CouchDB the inserted row in the above table, would be just a JSON looking like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#a61717;background-color:#e3d2d2">{</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"id"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"name"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"shoes"</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"description"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"blue_shoes"</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"price"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"currency"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"dollars"</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"in_stock"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">5</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#a61717;background-color:#e3d2d2">}</span><span style="color:#bbb">
</span></code></pre></div><h4 id="the-trivial-solution">The Trivial Solution</h4>
<p>In PostgreSQL we can store this JSON as a row in the products table:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span><span style="color:#038">TEXT</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>This works like most of the NoSQL datatabases, no checks, no errors with bad fields. As a result, you can modify the data the way you want, the problem begins when your application expects that the price is a number, and you get a string there, or there is no price at all.</p>
<h4 id="validate-json">Validate JSON</h4>
<p>CouchDB validates JSON before saving the document into database. In PostgreSQL 9.2 there is the nice type for that, it is named JSON. The JSON type can store only a proper JSON, there is validation performed before converting into this type.</p>
<p>Let’s change the definition of the table to:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>JSON<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>We can insert correct JSON into this table:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products(<span style="color:#080;font-weight:bold">data</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span>(<span style="color:#d20;background-color:#fff0f0">'{
</span><span style="color:#d20;background-color:#fff0f0"> "id": 1,
</span><span style="color:#d20;background-color:#fff0f0"> "name": "shoes",
</span><span style="color:#d20;background-color:#fff0f0"> "description": "blue_shoes",
</span><span style="color:#d20;background-color:#fff0f0"> "price": 12.34,
</span><span style="color:#d20;background-color:#fff0f0"> "currency": "dollars",
</span><span style="color:#d20;background-color:#fff0f0"> "in_stock": 5
</span><span style="color:#d20;background-color:#fff0f0">}'</span>);<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>products;<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">data</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">----------------------------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#a61717;background-color:#e3d2d2">{</span><span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"id"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>,<span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"name"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"shoes"</span>,<span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"description"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"blue_shoes"</span>,+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"price"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">12</span>.<span style="color:#00d;font-weight:bold">34</span>,<span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"currency"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"dollars"</span>,<span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"in_stock"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">5</span><span style="color:#bbb"> </span>+<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#a61717;background-color:#e3d2d2">}</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>(<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">row</span>)<span style="color:#bbb">
</span></code></pre></div><p>This works, but inserting not a valid JSON ends with an error:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products(<span style="color:#080;font-weight:bold">data</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span>(<span style="color:#d20;background-color:#fff0f0">'{
</span><span style="color:#d20;background-color:#fff0f0"> "id": 1,
</span><span style="color:#d20;background-color:#fff0f0"> "name": "shoes",
</span><span style="color:#d20;background-color:#fff0f0"> "description": "blue_shoes",
</span><span style="color:#d20;background-color:#fff0f0"> "price": 12.34,
</span><span style="color:#d20;background-color:#fff0f0"> "currency": "dollars",
</span><span style="color:#d20;background-color:#fff0f0"> "in_stock": 5,
</span><span style="color:#d20;background-color:#fff0f0">}'</span>);<span style="color:#bbb">
</span><span style="color:#bbb"></span>ERROR:<span style="color:#bbb"> </span>invalid<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">input</span><span style="color:#bbb"> </span>syntax<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">for</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">type</span><span style="color:#bbb"> </span>json<span style="color:#bbb">
</span><span style="color:#bbb"></span>LINE<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>:<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products(<span style="color:#080;font-weight:bold">data</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span>(<span style="color:#d20;background-color:#fff0f0">'{
</span><span style="color:#d20;background-color:#fff0f0"> ^
</span><span style="color:#d20;background-color:#fff0f0">DETAIL: Expected string, but found "}".
</span><span style="color:#d20;background-color:#fff0f0">CONTEXT: JSON data, line 5: ...,
</span><span style="color:#d20;background-color:#fff0f0"> "currency": "dollars",
</span><span style="color:#d20;background-color:#fff0f0"> "in_stock": 5,
</span><span style="color:#d20;background-color:#fff0f0">}
</span></code></pre></div><p>The problem with formatting can be hard to notice (I’ve added comma after the last field, JSON doesn’t like it).</p>
<h4 id="validating-fields">Validating Fields</h4>
<p>OK, so we have a solution which looks almost like the first native PostgreSQL solution: we have data which validates. It doesn’t mean the data is sensible.</p>
<p>Let’s add checks for validating the data.</p>
<p>In PostgreSQL 9.3, which has not been released yet, there are some new great features for manipulating JSON values. There are defined operators for the JSON type, which give you easy access to the fields and values.</p>
<p>I will use only one operator “-»”, but you can find more information in <a href="https://www.postgresql.org/docs/9.3/static/functions-json.html">PostgreSQL documentation</a>.</p>
<p>I also need to validate the types of the fields, including id field. This is something Postgres just checks because of the types definitions. I am going to use some other syntax for the checks, as I want to name it. It will be easier to look at problem with specific field instead of searching through the whole huge JSON.</p>
<p>The table with the constraints looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>JSON,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_id<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>((<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'id'</span>)::<span style="color:#038">integer</span><span style="color:#bbb"> </span>>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'id'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_name<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">length</span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'name'</span>)<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'name'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_description<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">length</span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'description'</span>)<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'description'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_price<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>((<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'price'</span>)::<span style="color:#038">decimal</span><span style="color:#bbb"> </span>>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'price'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_currency<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'currency'</span><span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'dollars'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'currency'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>validate_in_stock<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>((<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>)::<span style="color:#038">integer</span><span style="color:#bbb"> </span>>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#a61717;background-color:#e3d2d2">}</span><span style="color:#bbb">
</span></code></pre></div><p>The “-»” operator allows me to get the value of a specific field from JSON, check if it exists and validate it.</p>
<p>Let’s add a JSON without a description:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">postgres=#<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>products(<span style="color:#080;font-weight:bold">data</span>)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">VALUES</span>(<span style="color:#d20;background-color:#fff0f0">'{
</span><span style="color:#d20;background-color:#fff0f0"> "id": 1,
</span><span style="color:#d20;background-color:#fff0f0"> "name": "d",
</span><span style="color:#d20;background-color:#fff0f0"> "price": 1.0,
</span><span style="color:#d20;background-color:#fff0f0"> "currency": "dollars",
</span><span style="color:#d20;background-color:#fff0f0"> "in_stock": 5
</span><span style="color:#d20;background-color:#fff0f0">}'</span>);<span style="color:#bbb">
</span></code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">ERROR:<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">new</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">row</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">for</span><span style="color:#bbb"> </span>relation<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"products"</span><span style="color:#bbb"> </span>violates<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">check</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">constraint</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"validate_description"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>DETAIL:<span style="color:#bbb"> </span>Failing<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">row</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">contains</span><span style="color:#bbb"> </span>(<span style="color:#a61717;background-color:#e3d2d2">{</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"id"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"name"</span>:<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"d"</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"price"</span>:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>.<span style="color:#00d;font-weight:bold">0</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"currency...). >
</span></code></pre></div><p>There is one more validation left. The id and name fields should be unique. This can be easily done with two indexes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INDEX</span><span style="color:#bbb"> </span>ui_products_id<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>products((<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'id'</span>));<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INDEX</span><span style="color:#bbb"> </span>ui_products_name<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>products((<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'name'</span>));<span style="color:#bbb">
</span></code></pre></div><p>Now when you try to add a JSON document which id which already exists in database, then you will have an error like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">ERROR:<span style="color:#bbb"> </span>duplicate<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">key</span><span style="color:#bbb"> </span>value<span style="color:#bbb"> </span>violates<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">unique</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">constraint</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"ui_products_id"</span><span style="color:#bbb">
</span><span style="color:#bbb"></span>DETAIL:<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Key</span><span style="color:#bbb"> </span>((<span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>->><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'id'</span>::<span style="color:#038">text</span>))=(<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb"> </span>already<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">exists</span>.<span style="color:#bbb">
</span><span style="color:#bbb"></span>ERROR:<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">current</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">transaction</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">is</span><span style="color:#bbb"> </span>aborted,<span style="color:#bbb"> </span>commands<span style="color:#bbb"> </span>ignored<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">until</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">end</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">of</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">transaction</span><span style="color:#bbb"> </span>block<span style="color:#bbb">
</span></code></pre></div><h4 id="id-generation">Id Generation</h4>
<p>In NoSQL databases the id field is usually some UUID. This is an identifier generated with algorithms with a very small chance of generating the same value, even when you generate them on different machines. So I’m not going to touch it here.</p>
<h3 id="searching">Searching</h3>
<p>You can search the JSON data normally like you were searching columns in a table. Let’s search for the most expensive product we have in stock:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>in_stock<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ORDER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">BY</span><span style="color:#bbb"> </span>price<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">DESC</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIMIT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>;<span style="color:#bbb">
</span></code></pre></div><p>The JSON version is very similar:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>)::<span style="color:#038">integer</span><span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ORDER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">BY</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'price'</span>)::<span style="color:#038">decimal</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">DESC</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIMIT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>;<span style="color:#bbb">
</span></code></pre></div><p>This query can be very inefficient. It needs to read all the rows, parse JSON fields and check the in_stock and price fields, convert into proper types and then sort. The plan of such a query, after filling the table with 100k rows, looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#bbb"> </span>QUERY<span style="color:#bbb"> </span>PLAN<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">-----------------------------------------------------------------------------------------------------------------------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Limit</span><span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">9256</span>.<span style="color:#00d;font-weight:bold">48</span>..<span style="color:#00d;font-weight:bold">9256</span>.<span style="color:#00d;font-weight:bold">48</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">32</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">412</span>.<span style="color:#00d;font-weight:bold">911</span>..<span style="color:#00d;font-weight:bold">412</span>.<span style="color:#00d;font-weight:bold">912</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>-><span style="color:#bbb"> </span>Sort<span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">9256</span>.<span style="color:#00d;font-weight:bold">48</span>..<span style="color:#00d;font-weight:bold">9499</span>.<span style="color:#00d;font-weight:bold">05</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">97027</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">32</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">412</span>.<span style="color:#00d;font-weight:bold">910</span>..<span style="color:#00d;font-weight:bold">412</span>.<span style="color:#00d;font-weight:bold">910</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Sort<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Key</span>:<span style="color:#bbb"> </span>(((<span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>->><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'price'</span>::<span style="color:#038">text</span>))::<span style="color:#038">numeric</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Sort<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Method</span>:<span style="color:#bbb"> </span>top-N<span style="color:#bbb"> </span>heapsort<span style="color:#bbb"> </span>Memory:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">25</span>kB<span style="color:#bbb">
</span><span style="color:#bbb"> </span>-><span style="color:#bbb"> </span>Seq<span style="color:#bbb"> </span>Scan<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">on</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">00</span>..<span style="color:#00d;font-weight:bold">8771</span>.<span style="color:#00d;font-weight:bold">34</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">97027</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">32</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">022</span>..<span style="color:#00d;font-weight:bold">375</span>.<span style="color:#00d;font-weight:bold">624</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">100000</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Filter:<span style="color:#bbb"> </span>(((<span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>->><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>::<span style="color:#038">text</span>))::<span style="color:#038">integer</span><span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Total<span style="color:#bbb"> </span>runtime:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">412</span>.<span style="color:#00d;font-weight:bold">939</span><span style="color:#bbb"> </span>ms<span style="color:#bbb">
</span><span style="color:#bbb"></span>(<span style="color:#00d;font-weight:bold">7</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>)<span style="color:#bbb">
</span></code></pre></div><p>The “Seq Scan” line means that PostgreSQL needs to read the whole table. The time of 412 ms is not that bad, but can we make it better?</p>
<p>Fortunately PostgreSQL has a great feature: indexes on expressions, also named as functional indexes. It can store in the index sorted values of some expressions, and if the same expressions occur in a query, then the index can be used.</p>
<p>The indexes I need are:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INDEX</span><span style="color:#bbb"> </span>i_products_in_stock<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>products((<span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>)::<span style="color:#038">integer</span><span style="color:#bbb"> </span>));<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INDEX</span><span style="color:#bbb"> </span>i_products_price<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>products((<span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">data</span>->><span style="color:#d20;background-color:#fff0f0">'price'</span>)::<span style="color:#038">decimal</span><span style="color:#bbb"> </span>));<span style="color:#bbb">
</span></code></pre></div><p>Notice the double parenthesis, they are required because of the non trivial expression.</p>
<p>The plan now looks a little bit different, after creating indexes and running analyze on the products table:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#bbb"> </span>QUERY<span style="color:#bbb"> </span>PLAN<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">----------------------------------------------------------------------------------------------------------------------------------------------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Limit</span><span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">42</span>..<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">55</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">32</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">041</span>..<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">041</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>-><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Index</span><span style="color:#bbb"> </span>Scan<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">Backward</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">using</span><span style="color:#bbb"> </span>i_products_price<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">on</span><span style="color:#bbb"> </span>products<span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">42</span>..<span style="color:#00d;font-weight:bold">13690</span>.<span style="color:#00d;font-weight:bold">06</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">100000</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">32</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">041</span>..<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">041</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">1</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Filter:<span style="color:#bbb"> </span>(((<span style="color:#080;font-weight:bold">data</span><span style="color:#bbb"> </span>->><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'in_stock'</span>::<span style="color:#038">text</span>))::<span style="color:#038">integer</span><span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Total<span style="color:#bbb"> </span>runtime:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">062</span><span style="color:#bbb"> </span>ms<span style="color:#bbb">
</span><span style="color:#bbb"></span>(<span style="color:#00d;font-weight:bold">4</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>)<span style="color:#bbb">
</span></code></pre></div><p>So it is 664k percent faster.</p>
<h3 id="the-json-advantage">The JSON Advantage</h3>
<p>The JSON solution has got one nice feature which the native PostgreSQL hasn’t. The application can add its own fields on the fly without altering any table. JSON field is just a text, however with some validation. The new field won’t be checked by the indexes and constraints I’ve shown you above.</p>
<p>What’s more, you can add a constraint for this field later. This way you can have the best from both worlds: easy data model changing and consistent JSON structure across the database.</p>
<p>On the other hand you could of course add a trigger checking the JSON, before saving it to database, to check the list of available fields. This way you could prevent adding new fields by the application.</p>
<h3 id="summary">Summary</h3>
<p>So, I’ve shown you how you can use PostgreSQL as a simple NoSQL database storing JSON blobs of text. The great advantage over the simple NoSQL databases storing blobs is that you can constrain the blobs, so they are always correct and you shouldn’t have any problems with parsing and getting them from the database.</p>
<p>You can also query the database very easily, with huge speed. The ad-hoc queries are really simple, much simpler than the map-reduce queries which are needed in many NoSQL databases.</p>
Elasticsearch: Give me object!https://www.endpointdev.com/blog/2013/04/elasticsearch-object-mapping-eof-400/2013-04-30T00:00:00+00:00Miguel Alatorre
<p>I’m currently working on a project where Elasticsearch is used to index copious amounts of data with sometimes deeply nested JSON. A recurring error I’ve experienced is caused by a field not conforming to the type listed in the mapping. Let’s reproduce it on a small scale.</p>
<p>Assuming you have Elasticsearch installed, let’s create an index and mapping:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -XPUT <span style="color:#d20;background-color:#fff0f0">'http://localhost:9200/test'</span> -d <span style="color:#d20;background-color:#fff0f0">'
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "mappings": {
</span><span style="color:#d20;background-color:#fff0f0"> "item": {
</span><span style="color:#d20;background-color:#fff0f0"> "properties": {
</span><span style="color:#d20;background-color:#fff0f0"> "state": {
</span><span style="color:#d20;background-color:#fff0f0"> "properties": {
</span><span style="color:#d20;background-color:#fff0f0"> "name": {"type": "string"}
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0">'</span>
{<span style="color:#d20;background-color:#fff0f0">"ok"</span>:true,<span style="color:#d20;background-color:#fff0f0">"acknowledged"</span>:true}
</code></pre></div><p>Since we’ve defined properties for the “state” field, Elasticsearch will automatically treat it as an object.* Let’s now add some documents:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -XPUT <span style="color:#d20;background-color:#fff0f0">'http://localhost:9200/test/item/1'</span> -d <span style="color:#d20;background-color:#fff0f0">'
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "state": {
</span><span style="color:#d20;background-color:#fff0f0"> "name": "North Carolina"
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0">'</span>
{<span style="color:#d20;background-color:#fff0f0">"ok"</span>:true,<span style="color:#d20;background-color:#fff0f0">"_index"</span>:<span style="color:#d20;background-color:#fff0f0">"test"</span>,<span style="color:#d20;background-color:#fff0f0">"_type"</span>:<span style="color:#d20;background-color:#fff0f0">"item"</span>,<span style="color:#d20;background-color:#fff0f0">"_id"</span>:<span style="color:#d20;background-color:#fff0f0">"1"</span>,<span style="color:#d20;background-color:#fff0f0">"_version"</span>:1}
</code></pre></div><p>Success! Let’s now get into trouble:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -XPUT <span style="color:#d20;background-color:#fff0f0">'http://localhost:9200/test/item/2'</span> -d <span style="color:#d20;background-color:#fff0f0">'
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "state": "California"
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0">'</span>
{<span style="color:#d20;background-color:#fff0f0">"error"</span>:<span style="color:#d20;background-color:#fff0f0">"MapperParsingException[object mapping for [state] tried to parse as object, but got EOF, has a concrete value been provided to it?]"</span>,<span style="color:#d20;background-color:#fff0f0">"status"</span>:400}
</code></pre></div><p>The solution: check any non-objects in your data against your mapping schema and you’ll be sure to find a mismatch.</p>
<p>One thing to note is that the explicit creation of the mapping is unnecessary since Elasticsearch creates it using the first added document. Try this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ curl -XPUT <span style="color:#d20;background-color:#fff0f0">'http://localhost:9200/test2/item/1'</span> -d <span style="color:#d20;background-color:#fff0f0">'
</span><span style="color:#d20;background-color:#fff0f0">{
</span><span style="color:#d20;background-color:#fff0f0"> "state": {
</span><span style="color:#d20;background-color:#fff0f0"> "name": "North Carolina"
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">}
</span><span style="color:#d20;background-color:#fff0f0">'</span>
{<span style="color:#d20;background-color:#fff0f0">"ok"</span>:true,<span style="color:#d20;background-color:#fff0f0">"_index"</span>:<span style="color:#d20;background-color:#fff0f0">"test2"</span>,<span style="color:#d20;background-color:#fff0f0">"_type"</span>:<span style="color:#d20;background-color:#fff0f0">"item"</span>,<span style="color:#d20;background-color:#fff0f0">"_id"</span>:<span style="color:#d20;background-color:#fff0f0">"1"</span>,<span style="color:#d20;background-color:#fff0f0">"_version"</span>:1}
$ curl -XGET <span style="color:#d20;background-color:#fff0f0">'http://localhost:9200/test2/_mapping'</span>
{
<span style="color:#d20;background-color:#fff0f0">"test2"</span>: {
<span style="color:#d20;background-color:#fff0f0">"item"</span>: {
<span style="color:#d20;background-color:#fff0f0">"properties"</span>: {
<span style="color:#d20;background-color:#fff0f0">"state"</span>: {
<span style="color:#d20;background-color:#fff0f0">"dynamic"</span>:<span style="color:#d20;background-color:#fff0f0">"true"</span>,
<span style="color:#d20;background-color:#fff0f0">"properties"</span>: {
<span style="color:#d20;background-color:#fff0f0">"name"</span>: {<span style="color:#d20;background-color:#fff0f0">"type"</span>:<span style="color:#d20;background-color:#fff0f0">"string"</span>}
}
}
}
}
}
}
</code></pre></div><p>So, this stays true to the statement: <a href="https://elasticsearch.com/products/elasticsearch/">“Elasticsearch is schema-less, just toss it a typed JSON document and it will automatically index it.”</a> You can throw your car keys at Elasticsearch and it will index, however, as noted above, just be sure to keep throwing nothing but car keys.</p>
<p>*Anything with one or more nested key-value pairs is considered an object in Elasticsearch. For more on the object type, see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html">here</a>.</p>
Interchange “on-the-fly” itemshttps://www.endpointdev.com/blog/2012/07/interchange-on-fly-items/2012-07-26T00:00:00+00:00Jeff Boes
<p>Interchange has a handy feature (which, in my almost-seven-years of involvement, I’d not seen or suspected) allowing you to create an item “on-the-fly”, without requiring any updates to your products table. Here’s a recipe for making this work.</p>
<p>First, you need to tell Interchange that you’re going to make use of this feature (in catalog.cfg).</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">OnFly onfly
</code></pre></div><p>Simple, no? The “OnFly” directive names a subroutine that is called to pre-process the custom item before it’s added to the cart. The default “onfly” routine can be found in the system tag “onfly”: code/SystemTag/onfly.coretag in the standard Interchange installation. (If you need more that what it provides, that’s beyond the scope of my post, so good luck, bon voyage, and please write back to let us know what you learned!)</p>
<p>Then, you need to submit some special form parameters to set up the cart:</p>
<ul>
<li>mv_order_item: the item number identifying this line</li>
<li>mv_order_fly: a structured string with | (vertical bar) delimiters. Each sub-field specifies something about the custom item, thus:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">description=My custom item|price=12.34
</code></pre></div><p>Now, in my particular case, I was encapsulating an XML feed of products from another site (a parts supplier) so that the client (a retail seller) could offer replacement parts, but not have to incorporate thousands of additional lines in the “products” table. So after drilling down to the appropriate model and showing the available parts, each item got the following bit of JavaScript (AJAX) code associated with its add-to-cart button:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> $row = $(<span style="color:#080;font-weight:bold">this</span>).parents(<span style="color:#d20;background-color:#fff0f0">'tr'</span>);
$.ajax({
url: <span style="color:#d20;background-color:#fff0f0">'/cgi-bin/mycat/process'</span>,
data: {
mv_todo: <span style="color:#d20;background-color:#fff0f0">'refresh'</span>,
mv_order_quantity: <span style="color:#00d;font-weight:bold">1</span>,
mv_order_item: $row.find(<span style="color:#d20;background-color:#fff0f0">'td.item_number'</span>).html(),
mv_order_fly: <span style="color:#d20;background-color:#fff0f0">'description='</span>
+ $row.find(<span style="color:#d20;background-color:#fff0f0">'td.description'</span>).html().replace(<span style="color:#d20;background-color:#fff0f0">'|'</span>,<span style="color:#d20;background-color:#fff0f0">''</span>)
+ <span style="color:#d20;background-color:#fff0f0">'|'</span>
+ <span style="color:#d20;background-color:#fff0f0">'price='</span>
+ $row.find(<span style="color:#d20;background-color:#fff0f0">'td.price'</span>).html().replace(<span style="color:#d20;background-color:#fff0f0">'$'</span>,<span style="color:#d20;background-color:#fff0f0">''</span>).replace(<span style="color:#d20;background-color:#fff0f0">','</span>,<span style="color:#d20;background-color:#fff0f0">''</span>)
},
method: <span style="color:#d20;background-color:#fff0f0">'POST'</span>,
success: <span style="color:#080;font-weight:bold">function</span>(data, status) {
$(<span style="color:#d20;background-color:#fff0f0">'#msg_div'</span>).html(<span style="color:#d20;background-color:#fff0f0">'Added to cart.'</span>)
}
});
</code></pre></div><p>And that’s all it took. With Interchange, you don’t even need a special “landing page” for your AJAX submission; Interchange handles all the cart-updating out of sight.</p>
<p>I still need to add some post-processing to handle errors, and update the current page so I can see the new cart line count, but the basics are done.</p>
Web service integration in PHP, jQuery, Perl and Interchangehttps://www.endpointdev.com/blog/2012/06/web-service-integration-in-php-jquery/2012-06-13T00:00:00+00:00Ron Phipps
<p><img src="/blog/2012/06/web-service-integration-in-php-jquery/image-0.jpeg" /></p>
<p>Jeff Boes presented on one of his latest projects.</p>
<p>CityPass decided on a project to convert their checkout from being served by Interchange to have the interface served by PHP, but continue to interact with Interchange for the checkout process through a web service.</p>
<p>The original site was entirely served by Interchange, the client then took on a project to convert the frontend to PHP while leveraging Interchange for frontend logic such as pricing and shipping as well as for backend administration for order fulfillment.</p>
<p>Technologies used in the frontend rewrite:</p>
<ul>
<li>PHP</li>
<li>jQuery for jStorage, back-button support and checkout business logic</li>
<li>AJAX web services for prices, discounts, click-tracking</li>
</ul>
<p>The Interchange handler is conduit.am that handles the processing of the URL. From this ActionMap the URLs are decoded and passed to a Perl module, Data.pm, which handles processing the input and returning the results.</p>
<p>An order is just a JSON object so testing of the web service is easy. We have a known hash, we post to the proper URL and compare the results and verify they are the same. New test cases are also easy, we can capture any order (JSON) to a log file and add it as a test case.</p>
Simple Pagination with AJAXhttps://www.endpointdev.com/blog/2012/05/simple-pagination-with-ajax-heres/2012-05-24T00:00:00+00:00Jeff Boes
<p>Here’s a common problem: you have a set of results you want to display (search results, or products in a category) and you want to paginate them in a way that doesn’t submit and re-display your results page every time. AJAX is a clear winner in this; I’ll outline a very simple, introductory approach for carrying this off.</p>
<p>(I’m assuming that the reader has some modest familiarity with JavaScript and jQuery, but no great expertise. My solutions below will tend toward the “Cargo Cult” programming model, so that you can cut and paste, tweak, and go, but with enough “how and why” sprinkled in so you will come away knowing enough to extend the solution as needed.)</p>
<p>Firstly, you have to have the server-side processing in place to serve up paginated results in a way you can use. We’ll assume that you can write or adapt your current results source to produce this for a given URL and parameters:</p>
<pre tabindex="0"><code>/search?param1=123&param2=ABC&sort=colA,colB&offset=0&size=24
</code></pre><p>That URL offers a state-less way to retrieve a slice of results: in this case, it corresponds to a query something like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span><span style="color:#a61717;background-color:#e3d2d2">…</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span><span style="color:#a61717;background-color:#e3d2d2">…</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>param1=<span style="color:#d20;background-color:#fff0f0">'123'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb"> </span>param2=<span style="color:#d20;background-color:#fff0f0">'ABC'</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ORDER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">BY</span><span style="color:#bbb"> </span>colA,colB<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">OFFSET</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIMIT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">24</span><span style="color:#bbb">
</span></code></pre></div><p>You can see that this will generate a slice of 0-24 results; changing “offset” will get other slices, which is the foundation of our ability to “page” the results.</p>
<p>The code behind “/search” should return a JSON structure suitable for your needs. My usual approach is to assemble what I want in Perl, then pass it through JSON::to_json:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$results</span> = perform_search(...);
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$json</span> = <span style="color:#b06;font-weight:bold">JSON::</span>to_json(<span style="color:#369">$results</span>);
</code></pre></div><p>Don’t forget to include an appropriate document header:</p>
<pre tabindex="0"><code>Content-type: application/json
</code></pre><p>Now we need a JavaScript function to retrieve a slice; I’ll use jQuery here as it’s my preferred solution (and because I’m not at all fluent in non-jQuery approaches!).</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">function</span>(){
$.ajax({ url: <span style="color:#d20;background-color:#fff0f0">'/search'</span>, data: { <span style="color:#a61717;background-color:#e3d2d2">…</span>, offset: $offset, limit: <span style="color:#00d;font-weight:bold">24</span> },
<span style="color:#a61717;background-color:#e3d2d2">…</span>}
</code></pre></div><p>You’ll need to keep track of (or calculate) the offset within your page. My approach is to drop each result into a div or similar HTML construction, then I can count them on the fly:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> $offset = $(<span style="color:#d20;background-color:#fff0f0">'div.search_result'</span>).length;
</code></pre></div><p>For the “data” passed in the AJAX call above, you need to assemble your query parameters, most likely from form elements on the page. (Newbie note: you can put <input> and <select> elements in the page without a surrounding <form>, as jQuery doesn’t care—you aren’t going to submit a form, but construct a URL that looks like it came from a form.) Here’s one useful model to follow:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> $data = { offset: $offset, limit: <span style="color:#00d;font-weight:bold">24</span> };
$.each([<span style="color:#d20;background-color:#fff0f0">'param1'</span>, <span style="color:#d20;background-color:#fff0f0">'param2'</span>], <span style="color:#080;font-weight:bold">function</span>(ix, val) {
$data[val] = $(<span style="color:#d20;background-color:#fff0f0">'input[name='</span> + val + <span style="color:#d20;background-color:#fff0f0">'], select[name='</span> + val + <span style="color:#d20;background-color:#fff0f0">']'</span>).val();
};
</code></pre></div><p>and then:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">$.ajax({ url: <span style="color:#d20;background-color:#fff0f0">'/search'</span>, data: $data, <span style="color:#a61717;background-color:#e3d2d2">…</span> });
</code></pre></div><p>Now we need something to handle the returned data. Within the ajax() call, we reference (or construct) a function that takes at least one argument:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">function</span>(results) { <span style="color:#a61717;background-color:#e3d2d2">…</span> }
</code></pre></div><p>“results” is of course the JSON structure you built up in the “/search” response. Here, we’ll assume that you are just sending back an array of objects:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">[ { col1: <span style="color:#d20;background-color:#fff0f0">'val1'</span>, col2: <span style="color:#d20;background-color:#fff0f0">'val2'</span>, col3: <span style="color:#d20;background-color:#fff0f0">'val3'</span> }, { col1: <span style="color:#d20;background-color:#fff0f0">'val4'</span>, col2: <span style="color:#d20;background-color:#fff0f0">'val5'</span>, col3: <span style="color:#d20;background-color:#fff0f0">'val6'</span> } ]
</code></pre></div><p>would represent two rows of three columns each. We can now process these:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">$.each(results, <span style="color:#080;font-weight:bold">function</span>(ix, val){
<span style="color:#080;font-weight:bold">var</span> new_result = $(<span style="color:#d20;background-color:#fff0f0">'div.search_result'</span>).first().clone();
$(new_result).find(<span style="color:#d20;background-color:#fff0f0">'span.col1'</span>).html(val.col1);
$(new_result).find(<span style="color:#d20;background-color:#fff0f0">'span.col2'</span>).html(val.col2);
$(new_result).find(<span style="color:#d20;background-color:#fff0f0">'span.col3'</span>).html(val.col3);
$(<span style="color:#d20;background-color:#fff0f0">'div.search_result'</span>).last().append(new_result);
};
</code></pre></div><p>In place of an actual database query or search engine, I have a simple PHP program that sends back a chunk of simulated rows. A few other notes and finesses:</p>
<ul>
<li>
<p>In the HTML document, I have a template for the “search_result” DIV that is hidden. I can style this any way I like, then I clone it for each returned result row. Note that it’s initially hidden, so after appending a new clone to the page, I have to “show()” it.</p>
</li>
<li>
<p>I do some very simple arrangement of the results by inserting a hard break after every four results. You could do much fancier arrangements: assigning CSS classes based on whether the new result is at an edge of the grid, for instance.</p>
</li>
<li>
<p>Error handling in this example is very rudimentary.</p>
</li>
</ul>
<p>Here’s a screenshot of what you might expect, with Firebug showing the returned JSON object.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2012/05/simple-pagination-with-ajax-heres/image-0.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="233" src="/blog/2012/05/simple-pagination-with-ajax-heres/image-0.png" width="320"/></a></div>
A Little Less of the Middlehttps://www.endpointdev.com/blog/2012/03/little-less-of-middle/2012-03-14T00:00:00+00:00Josh Williams
<div class="separator" style="clear: both; text-align: center; float:right">
<img alt="elephant" border="0" src="https://joshwilliams.name/monitoring/5258954374_52b77c8090_m.jpg" style="clear: right"/>
<br/><a href="https://www.flickr.com/photos/34968534@N07/5258954374/">elephant</a> by <a href="https://www.flickr.com/photos/34968534@N07/">esclarabunda</a>
</div>
<p>I’ve been meaning to exercise a bit more. You know, just to keep the mid section nice and trim. But getting into that habit doesn’t seem to be so easy. Trimming middleware from an app, that’s something that can catch my attention.</p>
<p>Something that caught my eye recently is a couple <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5384a73f98d9829725186a7b65bf4f8adb3cfaf1">recent commits</a> to Postgres 9.2 that adds a JSON data type. Or more specifically, the <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=39909d1d39ae57c3a655fc7010e394e26b90fec9">second commit</a> that adds a couple handy output functions: array_to_json() and row_to_json(). If you want to try it out on 9.1, those have been made available <a href="https://web.archive.org/web/20120208114210/http://people.planetpostgresql.org/andrew/index.php?/archives/255-JSON-for-PG-9.2-...-and-now-for-9.1!.html">as a backported extension</a>.</p>
<p>Lately I’ve been doing a bit of work with jQuery, using it for AJAX-y stuff but passing <a href="https://www.json.org/">JSON</a> around instead. (AJAJ?) And traditionally that involves something in between the database and the client rewriting rows from one format to another. Not that it’s all that difficult; for example, in Python it’s a simple module call:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">jsonresult = json.dumps(cursor.fetchall())
</code></pre></div><p>… assuming I don’t have any columns needing processing: <em>TypeError: datetime.datetime(2012, 3, 09, 18, 34, 20, 730250, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)) is not JSON serializable</em> Similarly in PHP I can stitch together a JSON array to pass back to the front end:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">while</span> (<span style="color:#369">$row</span> = pg_fetch_assoc(<span style="color:#369">$rs</span>))
{
<span style="color:#369">$rows</span>[] = <span style="color:#369">$row</span>;
}
<span style="color:#369">$jsonresult</span> = json_encode(<span style="color:#369">$rows</span>)
</code></pre></div><p>Now I can trim that out, and embed the encoding right into the database query:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>row_to_json(pages)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>pages<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>page_id<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">5</span>;<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">-- or, to return an array of rows
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>array_to_json(array_agg(pages))<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>pages<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>page_title<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIKE</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'A Little Less%'</span>;<span style="color:#bbb">
</span></code></pre></div><p>Notice the use of the row-type reference to the table itself after the SELECT, rather than just a single column. This outputs:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">[{<span style="color:#d20;background-color:#fff0f0">"page_id"</span>:<span style="color:#00d;font-weight:bold">105</span>,<span style="color:#d20;background-color:#fff0f0">"today"</span>:<span style="color:#d20;background-color:#fff0f0">"\u03c0 day"</span>,<span style="color:#d20;background-color:#fff0f0">"page_title"</span>:<span style="color:#d20;background-color:#fff0f0">"A Little Less of the Middle"</span>,<span style="color:#d20;background-color:#fff0f0">"contents"</span>:<span style="color:#d20;background-color:#fff0f0">"I've been meaning to exercise a bit more. You..."</span>,<span style="color:#d20;background-color:#fff0f0">"published_on"</span>:<span style="color:#d20;background-color:#fff0f0">"2012-03-15 03:30:00+00"</span>}]
</code></pre></div><p>Compare that to the output from json_encode() above, where the database driver treated everything as a string, even the page_id integer. The other difference is the Postgres code doesn’t do any quoting on Unicode characters:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">[{<span style="color:#d20;background-color:#fff0f0">"page_id"</span>:<span style="color:#d20;background-color:#fff0f0">"105"</span>,<span style="color:#d20;background-color:#fff0f0">"today"</span>:<span style="color:#d20;background-color:#fff0f0">"π day"</span>,<span style="color:#d20;background-color:#fff0f0">"page_title"</span>:<span style="color:#d20;background-color:#fff0f0">"A Little Less of the Middle"</span>,<span style="color:#d20;background-color:#fff0f0">"contents"</span>:<span style="color:#d20;background-color:#fff0f0">"I've been meaning to exercise a bit more. You..."</span>,<span style="color:#d20;background-color:#fff0f0">"published_on"</span>:<span style="color:#d20;background-color:#fff0f0">"2012-03-15 03:30:00+00"</span>}]
</code></pre></div><p>I’m a bit on the fence about whether it’s a real replacement for doing it in middleware, especially in some web use cases where you typically want to do things like anti-XSS type processing on some fields before sending them off to a browser somewhere. Besides, at the moment at least, there’s no built-in way to break JSON back apart in the database. But I’m sure there’s some places getting direct JSON is helpful, and it’s certainly an interesting start.</p>
Lock up your keyshttps://www.endpointdev.com/blog/2012/02/lock-up-your-keys/2012-02-09T00:00:00+00:00Jeff Boes
<h3 id="locking-hash-keys-with-hashutil">Locking hash keys with Hash::Util</h3>
<p>It’s a given that you shouldn’t write Perl without “use strict”; it prevents all kinds of silent bugs involving misspelled and uninitialized variables. A similar aid for misspelled and uninitialized hash keys exists in the module “<a href="http://search.cpan.org/perldoc?Hash::Util">Hash::Util</a>”.</p>
<p>By way of background: I was working on a long chunk of code that prepares an e-commerce order for storage in a database. Many of the incoming fields map directly to the table, but others do not. The interface between this code and the page which submits a large JSON structure was in flux for a while, so from time to time I had to chase bugs involving “missing” or “extra” fields. I settled on a restricted hash to help me squash these and future bugs.</p>
<p>The idea of a restricted hash is to clamp down on Perl’s rather loose “record” structure (by which I mean the common practice of using a hash to represent a record with named fields), which is great in some circumstances. While in most programming languages you must pre-declare a structure and live with it, in Perl hashes you can add new keys on the fly, misspellings and all. A restricted hash can only have a particular set of keys, but is still a hash for all other purposes.</p>
<p>An example:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">my</span> <span style="color:#369">%hash</span> = (aaa => <span style="color:#00d;font-weight:bold">1</span>, bbb => <span style="color:#00d;font-weight:bold">2</span>);
</code></pre></div><p>Attempts to reference $hash{ccc} will not return an error, but only an undefined value. We can now lock the hash so that its current roster of keys will be constant:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">Hash::Util</span> <span style="color:#2b2;background-color:#f0fff0">qw(lock_keys)</span>;
lock_keys(<span style="color:#369">%hash</span>);
</code></pre></div><p>and now $hash{ccc} is not only undefined, it’s a run-time error:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#369">$hash</span>{ccc};
Attempt to access disallowed key <span style="color:#d20;background-color:#fff0f0">'ccc'</span> in a restricted hash
</code></pre></div><p>If we know the list of keys before the hash is initialized, we can set it up like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#080;font-weight:bold">my</span> <span style="color:#369">%hash</span>;
lock_keys(<span style="color:#369">%hash</span>, <span style="color:#2b2;background-color:#f0fff0">qw(aaa bbb ccc)</span>);
</code></pre></div><p>Keep in mind the values of $hash{aaa}, etc. are mutable (can be undefined, not exist, scalars, references, etc.), just like a normal hash.</p>
<p>What if our key roster needs to change over the course of the program? In my example, there were several kinds of transactions being sent via JSON, and I needed to validate and restrict fields based on the presence and values of other fields. E.g.,</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#080;font-weight:bold">if</span> (<span style="color:#369">$hash</span>{record_type} <span style="color:#080">eq</span> <span style="color:#d20;background-color:#fff0f0">'A'</span>) {
<span style="color:#888"># validate %hash for aaa, bbb, ccc</span>
}
<span style="color:#080;font-weight:bold">else</span> {
<span style="color:#888"># validate %hash for aaa, bbb, ddd; ccc should not appear</span>
}
</code></pre></div><p>You can add to or modify the accepted keys as you go, but it’s a two-step process: not even Hash::Util can modify the keys of a locked hash, so you have to unlock and re-lock:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#080;font-weight:bold">my</span> <span style="color:#369">%hash</span>;
lock_keys(<span style="color:#369">%hash</span>, <span style="color:#2b2;background-color:#f0fff0">qw(record_type aaa bbb ccc)</span>);
<span style="color:#888"># …</span>
unlock_keys(<span style="color:#369">%hash</span>);
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#369">$hash</span>{record_type} <span style="color:#080">eq</span> <span style="color:#d20;background-color:#fff0f0">'A'</span>) {
lock_keys(<span style="color:#369">%hash</span>, <span style="color:#2b2;background-color:#f0fff0">qw(record_type aaa bbb ccc)</span>);
}
<span style="color:#080;font-weight:bold">else</span> {
lock_keys(<span style="color:#369">%hash</span>, <span style="color:#2b2;background-color:#f0fff0">qw(record_type aaa bbb ddd)</span>);
}
</code></pre></div><p>Of course, that’s kind of wordy: we’d really rather just splice in a key here and there. Hash::Util has you covered, because you can retrieve the list of legal keys for a hash (even if it’s not currently locked):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> lock_keys_plus(<span style="color:#369">%hash</span>, <span style="color:#2b2;background-color:#f0fff0">qw(ddd)</span>);
</code></pre></div><p>adds ‘ddd’ to the list, keeping the previous keys as well. However, if any of the <em>legal</em> keys are not <em>current keys</em>, they won’t make it into the key roster. Instead, use:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> lock_keys_plus(<span style="color:#369">%hash</span>, (legal_keys(<span style="color:#369">%hash</span>), <span style="color:#2b2;background-color:#f0fff0">qw(more keys here)</span>));
</code></pre></div><p>Everything shown here for hashes is also available for hashrefs: for instance, to lock up a hashref $hr:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> lock_ref_keys(<span style="color:#369">$hr</span>);
unlock_ref_keys(<span style="color:#369">$hr</span>);
lock_ref_keys_plus(<span style="color:#369">$hr</span>, (legal_ref_keys(<span style="color:#369">$hr</span>), <span style="color:#2b2;background-color:#f0fff0">qw(other keys)</span>));
</code></pre></div><p>Of course, adding all this locking and unlocking adds complexity to your code, so you should consider carefully whether it’s justified. In my case I had 60+ keys, in a nested structure, spanning 1500 lines of code — I just could not keep all the correct spellings in my head any more, so now when I write</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"> <span style="color:#080;font-weight:bold">if</span> (<span style="color:#369">$opt</span>->{order_status})
</code></pre></div><p>when I mean “transaction_status”, I’ll get a helpful run-time error instead of a silent skip of that block of code.</p>
<p>Are there other approaches? Yes, depending on your needs: JSON::Schema, for instance, will let you validate a JSON structure against a “golden master”. However, it does not prevent subsequent assignments to the structure, creating new keys on the fly (possibly in error). Moose would support a restricted object like this, but may add more complexity than you need, so Hash::Util may be the appropriate, lighter-weight approach.</p>
CRUD, RESTful, and JSON for Address Management in Interchangehttps://www.endpointdev.com/blog/2011/10/crud-restful-and-json-for-address/2011-10-04T00:00:00+00:00Steph Skardal
<p>Recently, I worked on a large project for <a href="https://www.papersource.com/">Paper Source</a> that introduced the functionality to allow users to split orders into multiple addresses, which is especially valuable during the holiday season for gift purchase. Paper Source runs on <a href="http://www.icdevgroup.org/i/dev">Interchange</a>, one of the ecommerce frameworks End Point is intimately familiar with.</p>
<div class="separator" style="clear: both; text-align: left;">
<a href="/blog/2011/10/crud-restful-and-json-for-address/image-0.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" src="/blog/2011/10/crud-restful-and-json-for-address/image-0.png" width="740"/></a></div>
<p>This project requires <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> functionality for address editing during the checkout process; users must be able to add, edit, list, and remove addresses to be assigned to the various items in the cart. Another challenge here is that both logged in and logged out users are required to have this functionality, where addresses modified by logged in users would persist between sessions, and addresses of logged out users are available during their session and destroyed after they end the session (or in Paper Source’s case, when they close the browser window).</p>
<p>With these requirements, I set off to develop an architecture that followed <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">RESTful practices</a>, described below:</p>
<h3 id="listing-addresses">Listing Addresses</h3>
<p>A Perl module contains a method for listing user addresses, which is shown below in simplified form. The user_addresses scratch variable is created and contains an array of hashed addresses. <em>For those unfamiliar with Interchange, <a href="http://www.icdevgroup.org/docs/glossary/scratch.html">scratch space</a> is created as part of a user session which contains variables accessible throughout the user session or limited to a single page request. The Tag->tmpn call below sets the scratch variable accepting arguments key, value that is accessible during the current page request. This is not unlike setting a variable in a controller to be used in the view in MVC architecture.</em></p>
<p>Also of note is the use of to_json, which transforms the address into JSON-ified form, which is used rather than manually looping through the addresses.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">sub</span> <span style="color:#06b;font-weight:bold">list</span> {
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$addresses</span>;
<span style="color:#080;font-weight:bold">if</span>(<span style="color:#369">$::Session</span>->{username}) { <span style="color:#888"># user is logged in</span>
<span style="color:#888"># $results = SELECT * FROM addresses WHERE username = ?</span>
<span style="color:#080;font-weight:bold">foreach</span> <span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$key</span> (<span style="color:#038">keys</span> <span style="color:#369">%$results</span>) {
<span style="color:#038">push</span> <span style="color:#369">@$addresses</span>, <span style="color:#369">$results</span>->{<span style="color:#369">$key</span>};
}
}
<span style="color:#080;font-weight:bold">else</span> {
<span style="color:#369">$addresses</span> = <span style="color:#369">$::Session</span>->{stored_addresses};
}
<span style="color:#080;font-weight:bold">foreach</span> <span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$address</span> (<span style="color:#369">@$addresses</span>) {
<span style="color:#369">$address</span>->{json} = to_json(<span style="color:#369">$address</span>);
}
<span style="color:#369">$:Tag</span>->tmpn(<span style="color:#d20;background-color:#fff0f0">"user_addresses"</span>, <span style="color:#369">$addresses</span>);
<span style="color:#080;font-weight:bold">return</span>;
}
</code></pre></div><p>In the HTML template, Interchange’s loop tag is used to loop through the addresses. Note: There may be a better way to avoid trailing commas in Interchange’s loop tag – please share it if you know the secret!</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><script type="text/javascript">
var addresses = {
[loop object.mv_results=`$Scratch->{user_addresses}`]
'address_[loop-param id]': [loop-param json],[/loop]
'dummy' : ''
};
</script>
</code></pre></div><p>For example, the above code might yield the JSON object shown below. These addresses are also used in the dropdowns shown in the screenshot at the beginning of this article.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><script type=<span style="color:#d20;background-color:#fff0f0">"text/javascript"</span>>
<span style="color:#080;font-weight:bold">var</span> addresses = {
<span style="color:#d20;background-color:#fff0f0">'address_116971'</span>: {<span style="color:#d20;background-color:#fff0f0">"country"</span>:<span style="color:#d20;background-color:#fff0f0">"US"</span>,<span style="color:#d20;background-color:#fff0f0">"nickname"</span>:<span style="color:#d20;background-color:#fff0f0">"Sister"</span>,<span style="color:#d20;background-color:#fff0f0">"fname"</span>:<span style="color:#d20;background-color:#fff0f0">"Jackie"</span>, ... },
<span style="color:#d20;background-color:#fff0f0">'address_116969'</span>: {<span style="color:#d20;background-color:#fff0f0">"country"</span>:<span style="color:#d20;background-color:#fff0f0">"US"</span>,<span style="color:#d20;background-color:#fff0f0">"nickname"</span>:<span style="color:#d20;background-color:#fff0f0">"Personal"</span>,<span style="color:#d20;background-color:#fff0f0">"fname"</span>:<span style="color:#d20;background-color:#fff0f0">"Stephanie"</span>, ... },
<span style="color:#d20;background-color:#fff0f0">'dummy'</span> : <span style="color:#d20;background-color:#fff0f0">''</span>
};
<<span style="color:#a61717;background-color:#e3d2d2">/script></span>
</code></pre></div><h3 id="creating-an-address">Creating an Address</h3>
<p>Next, I added functionality for creating an address. Similar to the method above, the <strong>add</strong> subroutine is called with the use of a custom <a href="http://interchange.rtfm.info/icdocs/config/ActionMap.html">Interchange Actionmap</a>. The add method handles logged in and logged out use cases:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">sub</span> <span style="color:#06b;font-weight:bold">add</span> {
<span style="color:#888"># do server-side error checking</span>
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$result</span>;
<span style="color:#038">eval</span> {
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$address</span>;
<span style="color:#080;font-weight:bold">if</span>(<span style="color:#369">$::Session</span>->{username}) {
<span style="color:#888"># store address in database with INSERT</span>
<span style="color:#888"># $address is new address, with id of last_insert_id</span>
}
<span style="color:#080;font-weight:bold">else</span> {
<span style="color:#888"># determine key to store address in session</span>
<span style="color:#888"># store address in Session->{stored_addresses}</span>
<span style="color:#888"># $address is new address, with key as id</span>
}
<span style="color:#369">$result</span> = { address => to_json(<span style="color:#369">$address</span>), success => <span style="color:#00d;font-weight:bold">1</span> };
};
<span style="color:#080;font-weight:bold">if</span>(<span style="color:#d70">$@</span>) {
<span style="color:#369">$result</span> = { error_msg => <span style="color:#d20;background-color:#fff0f0">"Error: ..."</span>, success => <span style="color:#00d;font-weight:bold">0</span> };
}
<span style="color:#369">$::Tag</span>->tmpn(<span style="color:#d20;background-color:#fff0f0">"result"</span>, to_json(<span style="color:#369">$result</span>));
<span style="color:#369">$::CGI</span>->{mv_nextpage} = <span style="color:#d20;background-color:#fff0f0">"ajax/standard.html"</span>; <span style="color:#888"># sets HTML template used</span>
<span style="color:#080;font-weight:bold">return</span>;
}
</code></pre></div><p>This method was called via AJAX, which looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">$.ajax({
url: <span style="color:#d20;background-color:#fff0f0">"/address_management/add"</span>,
data: address_data, <span style="color:#888">//parameterized address data
</span><span style="color:#888"></span> dataType: <span style="color:#d20;background-color:#fff0f0">"JSON"</span>,
success: <span style="color:#080;font-weight:bold">function</span>(data) {
<span style="color:#080;font-weight:bold">if</span>(data.success) {
<span style="color:#888">//updated addresses JavaScript variable
</span><span style="color:#888"></span> <span style="color:#888">//updated on page HTML (adds to dropdown)
</span><span style="color:#888"></span> } <span style="color:#080;font-weight:bold">else</span> {
<span style="color:#888">//notifies user of errors
</span><span style="color:#888"></span> }
},
failure: <span style="color:#080;font-weight:bold">function</span>(data) {
<span style="color:#888">//alert that could not process
</span><span style="color:#888"></span> }
});
</code></pre></div><h3 id="edit-address">Edit Address</h3>
<p>The edit address functionality is similar to the code for creating an address, the only difference being that the database and Session variable was updated, and the URL called for editing is “/address_management/edit/:id”. <a href="https://mustache.github.io/">Mustache</a> is used to render the form prepopulated with the current address values.</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="/blog/2011/10/crud-restful-and-json-for-address/image-1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" src="/blog/2011/10/crud-restful-and-json-for-address/image-1.png" width="750"/></a></div>
<h3 id="remove-address">Remove Address</h3>
<p>Finally, an address management page was created to include the ability to remove addresses. This method uses an AJAX method similar to the one shown above, and the Perl module contains the following:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#080;font-weight:bold">sub</span> <span style="color:#06b;font-weight:bold">remove</span> {
<span style="color:#080;font-weight:bold">my</span> (<span style="color:#369">$self</span>, <span style="color:#369">$dbh</span>) = <span style="color:#369">@_</span>;
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$result</span>;
<span style="color:#038">eval</span> {
<span style="color:#888"># database DELETE FROM addresses</span>
<span style="color:#369">$result</span>->{success} = <span style="color:#00d;font-weight:bold">1</span>;
};
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#d70">$@</span>) {
<span style="color:#369">$result</span>->{success} = <span style="color:#00d;font-weight:bold">0</span>;
}
<span style="color:#369">$::Tag</span>->tmpn(<span style="color:#d20;background-color:#fff0f0">"result"</span>, to_json(<span style="color:#369">$result</span>));
<span style="color:#369">$::CGI</span>->{mv_nextpage} = <span style="color:#d20;background-color:#fff0f0">'ajax/standard.html'</span>;
<span style="color:#080;font-weight:bold">return</span>;
}
</code></pre></div><h3 id="conclusion">Conclusion</h3>
<p>The advantages to the RESTful CRUD methods described here is that the client side JavaScript and HTML code is reused for both logged in and logged out users. The server-side responds to all requests in a similar manner for both logged in and logged out use cases. Reusing client-side code eases maintenance because there is less code to support and less code to update when changes are necessary. Additionally, these CRUD methods are reused in the user address management page (shown below), which takes advantage of reusable server-side and client-side modular elements.</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="/blog/2011/10/crud-restful-and-json-for-address/image-2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" src="/blog/2011/10/crud-restful-and-json-for-address/image-2.png" width="615"/></a></div>
<p>In addition to the work described here, quite a bit of work was done on the backend for order processing to create a new structure for storing addresses per order. Out of the box Interchange functionality stores the billing and shipping addresses in the transactions (orders) table, and individual item information in the orderlines table (item sku, quantity, price, etc.). With this new functionality, shipping addresses were pulled out of the transactions table into their own table (shipped_addresses), and orderline items mapped to these shipping addresses. To preserve historical order data, shipped_addresses is copied from addresses for logged in users and remains untouched by the user.</p>
<table cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td align="center" valign="top">Data Model Before<br/><img border="0" src="/blog/2011/10/crud-restful-and-json-for-address/image-3.png" width="250"/></td>
<td align="center" valign="top">Data Model After<br/><img border="0" src="/blog/2011/10/crud-restful-and-json-for-address/image-4.png" width="400"/></td></tr>
</tbody></table>
<p>Paper Source has several special products, like gift cards and a large set of personalized products. Additional changes were required as well to accomodate the new data model. For example, a script that reported on gift card shipping addresses required updating to adhere to the new data structure.</p>
JSON pretty-printerhttps://www.endpointdev.com/blog/2011/02/json-pretty-printer/2011-02-01T00:00:00+00:00Jon Jensen
<p>The other day Sonny Cook and I were troubleshooting some YUI JavaScript code and looking at some fairly complex JSON. It would obviously be a lot easier to read if each nested data structure were indented, and spacing standardized.</p>
<p>I threw together a little Perl program based on the JSON man page:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-perl" data-lang="perl"><span style="color:#888">#!/usr/bin/env perl</span>
<span style="color:#080;font-weight:bold">use</span> <span style="color:#b06;font-weight:bold">JSON</span>;
<span style="color:#080;font-weight:bold">my</span> <span style="color:#369">$json</span> = <span style="color:#b06;font-weight:bold">JSON</span>-><span style="color:#080;font-weight:bold">new</span>;
<span style="color:#038">undef</span> <span style="color:#d70">$/</span>;
<span style="color:#080;font-weight:bold">while</span> (<>) {
<span style="color:#080;font-weight:bold">print</span> <span style="color:#369">$json</span>-><span style="color:#b06;font-weight:bold">pretty</span>->encode(<span style="color:#369">$json</span>->decode(<span style="color:#369">$_</span>));
}
</code></pre></div><p>It took all of 2 or 3 minutes and I even left out strictures and warnings. Living on the edge!</p>
<p>It turns a mess like this (sample from json.org):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">{<span style="color:#d20;background-color:#fff0f0">"glossary"</span>:{<span style="color:#d20;background-color:#fff0f0">"title"</span>:<span style="color:#d20;background-color:#fff0f0">"example glossary"</span>,<span style="color:#d20;background-color:#fff0f0">"GlossDiv"</span>:{<span style="color:#d20;background-color:#fff0f0">"title"</span>:<span style="color:#d20;background-color:#fff0f0">"S"</span>,<span style="color:#d20;background-color:#fff0f0">"GlossList"</span>:
{<span style="color:#d20;background-color:#fff0f0">"GlossEntry"</span>:{<span style="color:#d20;background-color:#fff0f0">"ID"</span>:<span style="color:#d20;background-color:#fff0f0">"SGML"</span>,<span style="color:#d20;background-color:#fff0f0">"SortAs"</span>:<span style="color:#d20;background-color:#fff0f0">"SGML"</span>,<span style="color:#d20;background-color:#fff0f0">"GlossTerm"</span>:<span style="color:#d20;background-color:#fff0f0">"Standard Generalized Markup Language"</span>,
<span style="color:#d20;background-color:#fff0f0">"Acronym"</span>:<span style="color:#d20;background-color:#fff0f0">"SGML"</span>,<span style="color:#d20;background-color:#fff0f0">"Abbrev"</span>:<span style="color:#d20;background-color:#fff0f0">"ISO 8879:1986"</span>,<span style="color:#d20;background-color:#fff0f0">"GlossDef"</span>:{<span style="color:#d20;background-color:#fff0f0">"para"</span>:
<span style="color:#d20;background-color:#fff0f0">"A meta-markup language,used to create markup languages such as DocBook."</span>,
<span style="color:#d20;background-color:#fff0f0">"GlossSeeAlso"</span>:[<span style="color:#d20;background-color:#fff0f0">"GML"</span>,<span style="color:#d20;background-color:#fff0f0">"XML"</span>]},<span style="color:#d20;background-color:#fff0f0">"GlossSee"</span>:<span style="color:#d20;background-color:#fff0f0">"markup"</span>}}}}}
</code></pre></div><p>into this much more readable specimen:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">{
<span style="color:#d20;background-color:#fff0f0">"glossary"</span> : {
<span style="color:#d20;background-color:#fff0f0">"GlossDiv"</span> : {
<span style="color:#d20;background-color:#fff0f0">"GlossList"</span> : {
<span style="color:#d20;background-color:#fff0f0">"GlossEntry"</span> : {
<span style="color:#d20;background-color:#fff0f0">"GlossDef"</span> : {
<span style="color:#d20;background-color:#fff0f0">"para"</span> : <span style="color:#d20;background-color:#fff0f0">"A meta-markup language,used to create markup languages such as DocBook."</span>,
<span style="color:#d20;background-color:#fff0f0">"GlossSeeAlso"</span> : [
<span style="color:#d20;background-color:#fff0f0">"GML"</span>,
<span style="color:#d20;background-color:#fff0f0">"XML"</span>
]
},
<span style="color:#d20;background-color:#fff0f0">"GlossTerm"</span> : <span style="color:#d20;background-color:#fff0f0">"Standard Generalized Markup Language"</span>,
<span style="color:#d20;background-color:#fff0f0">"ID"</span> : <span style="color:#d20;background-color:#fff0f0">"SGML"</span>,
<span style="color:#d20;background-color:#fff0f0">"SortAs"</span> : <span style="color:#d20;background-color:#fff0f0">"SGML"</span>,
<span style="color:#d20;background-color:#fff0f0">"Acronym"</span> : <span style="color:#d20;background-color:#fff0f0">"SGML"</span>,
<span style="color:#d20;background-color:#fff0f0">"Abbrev"</span> : <span style="color:#d20;background-color:#fff0f0">"ISO 8879:1986"</span>,
<span style="color:#d20;background-color:#fff0f0">"GlossSee"</span> : <span style="color:#d20;background-color:#fff0f0">"markup"</span>
}
},
<span style="color:#d20;background-color:#fff0f0">"title"</span> : <span style="color:#d20;background-color:#fff0f0">"S"</span>
},
<span style="color:#d20;background-color:#fff0f0">"title"</span> : <span style="color:#d20;background-color:#fff0f0">"example glossary"</span>
}
}
</code></pre></div><p>But today I thought back to that and figured surely something like that must already be at hand if I’d just looked for it. Sure enough, there are many easy options that work conveniently from the shell, similarly to that script:</p>
<ul>
<li>json_xs (Perl JSON::XS)</li>
<li>python -mjson.tool (Python 2.6+)</li>
<li>prettify_json.rb (Ruby json gem)</li>
</ul>
<p>And those were just the ones that were likely already on the machine I was using! Hooray for convenience.</p>