https://www.endpointdev.com/blog/tags/development/2023-12-27T00:00:00+00:00End Point DevClient Profile: J.G. Title Companyhttps://www.endpointdev.com/blog/2023/12/client-profile-j-g-title-company/2023-12-27T00:00:00+00:00Juan Pablo Ventoso
<p><img src="/blog/2023/12/client-profile-j-g-title-company/j-g-title-company.webp" alt="The J.G. Title Company logo sits in the center of an image on a blue background with futuristic designs reminiscent of circuit boards. Text on either side of the logo reads “Nationwide titling”, with text under reading “info@jgtitleco.com” and “https://jgtitleco.com”"></p>
<p><a href="https://jgtitleco.com/">J.G. Title</a> is a prominent service company specializing in automotive dealership titling and registration processes across the country. They simplify operations for their dealers, businesses, and individuals by giving tax/fee quotes, document validation, and checklists for all jurisdictions.</p>
<p>Jordan Kivett, the owner of J.G. Title, envisioned an innovative web application to simplify dealership operations. He contacted us to request a quote for developing this solution and making his vision a reality. This project became a great opportunity for our <a href="/team/">.NET team</a> to build a robust system with state-of-the-art technologies functioning seamlessly together.</p>
<h3 id="the-solution">The solution</h3>
<p>J.G. Title submitted a detailed document containing a description of the requirements for the J.G. Title Suite app along with workflow diagrams explaining their current business processes. After analyzing all the documents and having some initial meetings, we decided to divide the project into different stages, and estimate each phase of work separately. We ended up with three phases:</p>
<ul>
<li>Phase I: Create a product that allows users to enter deals into the system and retrieve the quote estimation, including taxes, fees, and charges for services across 50 states.</li>
<li>Phase II: Implement several UI improvements such as a detailed dealership dashboard, financial management, and integration with <a href="https://quickbooks.intuit.com/">QuickBooks</a>.</li>
<li>Phase III: Add an ambitious PDF document management section featuring document recognition, analysis, and manipulation, to perform automatic validations on the document’s contents and automatically generate PDF outputs from the deal’s details.</li>
</ul>
<h3 id="tech-stack">Tech stack</h3>
<ul>
<li><a href="https://dotnet.microsoft.com/en-us/learn/dotnet/what-is-dotnet">.NET</a> 6 with <a href="https://learn.microsoft.com/en-us/dotnet/csharp/tour-of-csharp/">C#</a></li>
<li><a href="https://www.postgresql.org/">PostgreSQL</a> 14</li>
<li><a href="https://rockylinux.org/">Rocky Linux</a> 9 with <a href="https://www.nginx.com/">Nginx</a></li>
</ul>
<p>We had already worked with this combination of technologies for several clients, and we are confident in the combined power in a robust database such as Postgres along with .NET 6 running in a Linux environment. It has proven to be a stable, fast, and reliable setup for our web solutions.</p>
<p>We also used <a href="https://trello.com/">Trello</a> for tracking the project’s progress and tasks, <a href="https://github.com/">GitHub</a> for version control, and <a href="https://visualstudio.microsoft.com/vs/">Visual Studio 2022</a> or <a href="https://code.visualstudio.com/">VS Code</a> with <a href="https://learn.microsoft.com/en-us/dotnet/core/tools/">.NET CLI</a> as a coding environment. Some of us also used <a href="https://code.visualstudio.com/docs/devcontainers/tutorial">VS Code dev containers</a>. <a href="https://www.pgadmin.org/">pgAdmin</a> is our usual tool to manage the local Postgres database, as well as <a href="https://www.postman.com/">Postman</a> to test our API endpoints.</p>
<h3 id="the-team">The team</h3>
<p>Most of our .NET team is involved on this project. We also receive valuable help from our hosting team and our Postgres developers.</p>
<ul>
<li><a href="/team/bimal-gharti-magar/">Bimal Gharti Magar</a></li>
<li><a href="/team/dan-briones/">Dan Briones</a></li>
<li><a href="/team/dylan-wooters/">Dylan Wooters</a></li>
<li><a href="/team/juan-pablo-ventoso/">Juan Pablo Ventoso</a></li>
<li><a href="/team/kevin-campusano/">Kevin Campusano</a></li>
<li><a href="/team/mike-delange/">Mike DeLange</a></li>
</ul>
<p>The client is also highly involved in all aspects of the development process, from requirements gathering to documenting, testing, and providing feedback. We have bi-weekly standup calls, and the entire team (End Point + J.G. Title) interacts actively through Trello, working together and updating each task as the work progresses.</p>
<h3 id="results">Results</h3>
<p>We began working on the project on October 25, 2022, and completed Phase I within the expected timeline, on June 7, 2023. The next phases were launched iteratively, with new deployments usually scheduled twice a week.</p>
<p>The application is now widely used by dealerships and the J.G. Title team. They have over 200 users registered in the system, with more than 8,000 deals quoted, and new deals being added at an increasing rate.</p>
<p><img src="/blog/2023/12/client-profile-j-g-title-company/j-g-title-suite-featured.webp" alt="On the right side is an image of a laptop and phone, each running the J.G. Title Suite application. Text on the left reads “From chaos to clarity. Simplify, streamline, succeed: Embrace clarity in your dealership’s tax and title operations. Our dedicated team of specialists conducted extensive research and established partnerships with local DMVs and state Treasurers to become the trusted partner for automotive dealerships and businesses nationwide, offering efficient and reliable title and registration services. We continue transforming automotive title management on a national scale and welcome you to experience the future of automotive titling.""><br>
The J.G. Title Suite application, featured on the company’s website.</p>
<p>We are continuing work on several tasks related to Phase III of the project. As the application keeps growing and users send feedback, new phases of work will surely be added to the project in the future.</p>
<p>This partnership is a testament to our shared vision for efficiency and innovation, and we’re excited to continue reshaping the industry with J.G. Title Company!</p>
Developers: Time and Teamworkhttps://www.endpointdev.com/blog/2023/12/developers-time-teamwork/2023-12-04T00:00:00+00:00Greg Hanson
<p><img src="/blog/2023/12/developers-time-teamwork/utah-exposed-rock-mountain.webp" alt="A steep golden hillside interspersed with green shrubs, behind power lines. Above the hill is a half moon on a deep blue sky."></p>
<!-- Photo by Seth Jensen, 2023. -->
<p>Time and teamwork. How do these two concepts fit together? The focus of this post is how blending them can improve not only your programming skills, but also your timeline of projects.</p>
<p>For a recent project I am the project manager of, we had an urgent request from a client for a developer who could be available in the next 2 weeks to work on a project that had about a 6 week timeline.</p>
<h3 id="the-project">The project</h3>
<p>Initially, it seemed like a pretty easy request to fill. We had a developer who had worked previously for this client, in the very same environment, albeit on a different project. So the roadmap was there: Get the developer into the systems, get the requirements, and do the work.</p>
<p>This client, like many, had some specific barriers to entry, like needing a company email address, access through a VPN, Duo 2FA, familiarity with Monday.com tracking, Office 365, Microsoft Teams for meetings, and so forth. But, as mentioned, the dev we would be using had worked previously in this environment, and so we already had most of the above in place.</p>
<p>So we got started with the first few initiation meetings, getting the basics of the project laid out, tasks assigned, and timeline established.</p>
<p>Then <a href="https://en.wikipedia.org/wiki/Murphy%27s_law">Mr. Murphy</a> showed up as he often does, and the dev we had planned to use got pulled out of the project for an urgent need elsewhere. Okay, project manager, what do we do now?</p>
<h3 id="time">Time</h3>
<p>Well, we dig into the pool of developers we have, and see what we can do. Here is the first recognition of <em>time</em>. We are already about a week into this thing, and now we are starting over. So time is not currently our friend.</p>
<h3 id="teamwork">Teamwork</h3>
<p>I did, however, have two developers who had worked together on many other projects. Luckily, these two developers had also worked with the first developer who was now leaving the project. So we immediately had a few <em>transfer of information</em> meetings.</p>
<h3 id="time-and-teamwork">Time and teamwork</h3>
<p>We had a lot of specific questions about what had been requested, and where it all lived. Because these two had worked together for so long, the time it took to get the two new developers up to speed was greatly reduced.</p>
<p>They were aware of each other’s strengths and weaknesses. As I thought about how much these developers’ rapport sped things up, I realized that this was a direct effect of <em>time</em> and <em>teamwork</em>. Also, because of the <em>time</em> spent together, they trusted each other. Respect was present instead of uncertainty, and exchanges were filled with value.</p>
<p>As most of you who are reading this know, programming can be performed in a number of different ways. You can be a lone wolf, you can be a tag team, or you can be a cog in a very large machine. There are places where each of those approaches is the most efficient.</p>
<p>In most cases when you have a pair or a small group, <em>two minds are better than one</em>. If you add <em>time</em> in as the multiplier so to speak, you end up with another dimension to that. These two developers have worked together for quite some <em>time</em>, and that is what enabled them to move so quickly in an unfamiliar environment. As the project progressed, these two sounded off to each other, and as a result many issues were solved internally, rather than ending up on the client’s board. <em>Time</em> spent waiting for client responses was minimized.</p>
<h3 id="client-satisfaction">Client satisfaction</h3>
<p>That doesn’t seem like a big deal, and the client may not have noticed that there were fewer clarification issues than on other projects. However, this <em>did</em> make a difference, and we were able to recover all of the time lost due to the unexpected loss of the original developer, even shaving off a little time on the overall project.</p>
<p>The result was a very happy client. Remember, this client found out a week into their project that the developer we had assigned to the project was now unable to continue. So there was quite a swing from losing a developer to finishing the project slightly ahead of schedule. Our client will remember that.</p>
<p>So I guess the moral of the story is something we probably all know inherently: Time contributes to the effectiveness of teamwork. The longer the relationship lasts, the higher the respect and trust level is between the team members. Knowing this, we can try to keep compatible developers together as much as is practically possible. Mix and match, and try to grow this trust over time. Build <em>teams</em> and come back to them, <em>time</em> after <em>time</em>.</p>
How to create a Hugo website without a themehttps://www.endpointdev.com/blog/2023/02/how-to-create-a-hugo-website-without-a-theme/2023-02-14T00:00:00+00:00Seth Jensen
<p><img src="/blog/2023/02/how-to-create-a-hugo-website-without-a-theme/light-through-tree.webp" alt="A light shines through the branches of a tree, creating rays through the thick fog, and softly illuminating the building behind the tree. The smallest holes in the branches create lines of light showing the movement of the camera."></p>
<!-- Photo by Seth Jensen, 2023 -->
<p>Since <a href="/blog/2021/08/converting-to-hugo/">converting this website</a> to the Hugo static site generator a couple of years ago, I’ve used Hugo for lots of other projects. It’s blazing fast, simple, and makes small website projects much easier.</p>
<p>One of the sites I’ve built with Hugo is a simple site to keep notes for my university classes. Hugo’s documentation tends to assume you’re using a theme, but for such a basic site using a theme would add unnecessary complexity I didn’t want to deal with. So, in this article I’ll show you how to create a site without a theme.</p>
<h3 id="creating-a-site">Creating a site</h3>
<p>First, <a href="https://gohugo.io/installation/">install Hugo</a>.</p>
<p>If you want to use <a href="https://sass-lang.com/documentation/syntax">SCSS</a>, as I do in the example below, make sure to install the “extended” version of Hugo.</p>
<p>Then, run the following command to create a Hugo site:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ hugo new site notes
</code></pre></div><p>Get into the new <code>notes</code> directory, and let’s edit the config file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-toml" data-lang="toml">baseURL = <span style="color:#d20;background-color:#fff0f0">"http://example.org/"</span>
languageCode = <span style="color:#d20;background-color:#fff0f0">"en-us"</span>
title = <span style="color:#d20;background-color:#fff0f0">"Notes"</span>
pluralizelisttitles = <span style="color:#080;font-weight:bold">false</span>
</code></pre></div><p>Other than the title, the only thing I changed here is disabling <code>pluralizelisttitles</code>. Hugo expects you to name your sections something singular (e.g., put your blog posts in a <code>post</code> directory), then automatically pluralizes the title when listing content in that section (<code>Posts</code>). I prefer the singular for my university class names (Math 112 instead of Math 112s and so on).</p>
<h3 id="layout">Layout</h3>
<p>For a site as simple as ours, we only need a few layout files. Create a directory called <code>_default</code> in the <code>layout</code> directory, to put three files in: <code>baseof.html</code>, <code>list.html</code>, and <code>single.html</code>. We’ll go through each of these files and what it does.</p>
<blockquote>
<p>If you need to add different templates for different sections, or if you’re curious about why we use <code>_default</code> here, see Hugo’s <a href="https://gohugo.io/templates/lookup-order/">lookup order docs</a>.</p>
</blockquote>
<h4 id="baseofhtml">baseof.html</h4>
<p>Every page on the site looks for a base template, which we provide in <code>baseof.html</code>. We’ll use it for the HTML boilerplate — <code><!doctype html></code>, the <code><html></code> tag, the <code><head></code> tag with some logic for what to load, and the <code><body></code> tag with a <a href="https://www.mikedane.com/static-site-generators/hugo/block-templates/">block template</a> for the page content.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><span style="color:#c00;font-weight:bold"><!doctype html></span>
<<span style="color:#b06;font-weight:bold">html</span>>
<<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">charset</span>=<span style="color:#d20;background-color:#fff0f0">"utf-8"</span>>
<<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">name</span>=<span style="color:#d20;background-color:#fff0f0">"viewport"</span> <span style="color:#369">content</span>=<span style="color:#d20;background-color:#fff0f0">"width=device-width, initial-scale=1"</span>>
<<span style="color:#b06;font-weight:bold">title</span>>{{ or .Title .Site.Title }}{{ if ne .Kind "home" }} | {{ .CurrentSection.Title }}{{ end }}</<span style="color:#b06;font-weight:bold">title</span>>
{{ $style := resources.Get "scss/style.scss" | toCSS | minify }}
<<span style="color:#b06;font-weight:bold">link</span> <span style="color:#369">rel</span>=<span style="color:#d20;background-color:#fff0f0">"stylesheet"</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ $style.RelPermalink }}"</span>>
</<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">body</span>>
{{ block "page" . }}{{ end }}
</<span style="color:#b06;font-weight:bold">body</span>>
</<span style="color:#b06;font-weight:bold">html</span>>
</code></pre></div><h4 id="listhtml">list.html</h4>
<p>The list template defines the layout for section list pages. Sections are defined by directories in the <code>content</code> folder.</p>
<blockquote>
<p>Read more about Sections in the <a href="https://gohugo.io/content-management/sections/">Hugo docs</a>.</p>
</blockquote>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html">{{ define "page" }}
<<span style="color:#b06;font-weight:bold">article</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"container"</span>>
{{ if ne .Kind "home" }}
<<span style="color:#b06;font-weight:bold">a</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ .Site.Home.RelPermalink }}"</span>>Home</<span style="color:#b06;font-weight:bold">a</span>>
{{ end }}
<<span style="color:#b06;font-weight:bold">h1</span>>{{ .Title }}</<span style="color:#b06;font-weight:bold">h1</span>>
{{ range .Sections }}
<<span style="color:#b06;font-weight:bold">h2</span>><<span style="color:#b06;font-weight:bold">a</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ .RelPermalink }}"</span>>{{ .Title }}</<span style="color:#b06;font-weight:bold">a</span>></<span style="color:#b06;font-weight:bold">h2</span>>
<<span style="color:#b06;font-weight:bold">ul</span>>
{{ range .Pages }}
<<span style="color:#b06;font-weight:bold">li</span>>
<<span style="color:#b06;font-weight:bold">a</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ .RelPermalink }}"</span>>{{ or .Title .RelPermalink }}</<span style="color:#b06;font-weight:bold">a</span>>
</<span style="color:#b06;font-weight:bold">li</span>>
{{ end }}
</<span style="color:#b06;font-weight:bold">ul</span>>
{{ end }}
<<span style="color:#b06;font-weight:bold">h2</span>>Other</<span style="color:#b06;font-weight:bold">h2</span>>
<<span style="color:#b06;font-weight:bold">ul</span>>
{{ range .RegularPages }}
<<span style="color:#b06;font-weight:bold">li</span>>
<<span style="color:#b06;font-weight:bold">a</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ .RelPermalink }}"</span>>{{ or .Title .RelPermalink }}</<span style="color:#b06;font-weight:bold">a</span>>
</<span style="color:#b06;font-weight:bold">li</span>>
{{ end }}
</<span style="color:#b06;font-weight:bold">ul</span>>
</<span style="color:#b06;font-weight:bold">article</span>>
{{ end }}
</code></pre></div><p>Here’s a breakdown of what’s going on:</p>
<ul>
<li>
<p>We define the <code>page</code> block we saw in the base template.</p>
</li>
<li>
<p>If we’re not on a <code>home</code> page, we show a link to home.</p>
</li>
<li>
<p>We range through every subsection of the current Section, listing all the pages for each.</p>
<p>Note that in Hugo, everything is a Page, of varying types (like Section). A Section can have multiple subsections as well as Regular Pages, which are just Markdown files with some front matter. This allows us to list direct subsections alongside the Regular Pages for each section, while listing non-section Pages in our “Other” section. You can read about all the available Pages variables in <a href="https://gohugo.io/variables/page/#pages">Hugo’s docs</a>.</p>
</li>
<li>
<p>Then, we list the Regular Pages of the current section.</p>
</li>
</ul>
<h4 id="singlehtml">single.html</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html">{{ define "page" }}
<<span style="color:#b06;font-weight:bold">article</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"container"</span>>
<<span style="color:#b06;font-weight:bold">a</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"{{ .CurrentSection.RelPermalink }}"</span>>Back to {{ .CurrentSection.Title }}</<span style="color:#b06;font-weight:bold">a</span>>
<<span style="color:#b06;font-weight:bold">h1</span>>{{ .Title }}</<span style="color:#b06;font-weight:bold">h1</span>>
{{ .Content }}
</<span style="color:#b06;font-weight:bold">article</span>>
{{ end }}
</code></pre></div><p>This one’s simple:</p>
<ul>
<li>Define the same <code>page</code> block — remember, the <code>list</code> and <code>single</code> templates both extend from the same <code>baseof</code> template.</li>
<li>Provide a link back to the current Section’s list page.</li>
<li>Get the title from the page’s front matter.</li>
<li>Show the content (here that means everything except the front matter) run through a Markdown parser.</li>
</ul>
<h3 id="styling">Styling</h3>
<p>I prefer to style with SCSS, which Hugo supports with <a href="https://gohugo.io/hugo-pipes/scss-sass/">Pipes</a>.</p>
<p>Because I’m using SCSS, I’ll put my styling in the <code>assets/scss</code> directory (you’ll need to create it). If you want to use plain CSS, you could put it in a <code>static/css</code> folder — everything in <code>static</code> is copied to the build folder, preserving the existing structure. Then, you can link to it normally, without the Hugo Pipe I use for SCSS parsing.</p>
<p>To keep it simple, my <code>style.scss</code> is less than 100 lines long. Here’s part of it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scss" data-lang="scss"><span style="color:#369">$paradise-pink</span>: <span style="color:#00d;font-weight:bold">#EA1F4B</span>;
<span style="color:#369">$paradise-pink-light</span>: <span style="color:#00d;font-weight:bold">#EF5778</span>;
<span style="color:#b06;font-weight:bold">html</span> {
<span style="color:#369">font-size</span>: <span style="color:#00d;font-weight:bold">62</span><span style="color:#00d;font-weight:bold">.5</span><span style="color:#888;font-weight:bold">%</span>;
}
<span style="color:#b06;font-weight:bold">body</span> {
<span style="color:#369">font-size</span>: <span style="color:#00d;font-weight:bold">1</span><span style="color:#00d;font-weight:bold">.6</span><span style="color:#888;font-weight:bold">rem</span>;
<span style="color:#369">line-height</span>: <span style="color:#00d;font-weight:bold">1</span><span style="color:#00d;font-weight:bold">.3</span>;
}
<span style="color:#b06;font-weight:bold">h2</span>, <span style="color:#b06;font-weight:bold">h3</span>, <span style="color:#b06;font-weight:bold">h4</span>, <span style="color:#b06;font-weight:bold">h5</span>, <span style="color:#b06;font-weight:bold">h6</span> {
<span style="color:#369">margin</span>: <span style="color:#00d;font-weight:bold">3</span><span style="color:#00d;font-weight:bold">.8</span><span style="color:#888;font-weight:bold">rem</span> <span style="color:#00d;font-weight:bold">0</span> <span style="color:#00d;font-weight:bold">.6</span><span style="color:#888;font-weight:bold">rem</span>;
}
<span style="color:#b06;font-weight:bold">h1</span> {
<span style="color:#369">font-size</span>: <span style="color:#00d;font-weight:bold">3</span><span style="color:#00d;font-weight:bold">.2</span><span style="color:#888;font-weight:bold">rem</span>;
}
<span style="color:#b06;font-weight:bold">h2</span> {
<span style="color:#369">font-size</span>: <span style="color:#00d;font-weight:bold">2</span><span style="color:#00d;font-weight:bold">.5</span><span style="color:#888;font-weight:bold">rem</span>;
}
<span style="color:#b06;font-weight:bold">p</span>, <span style="color:#b06;font-weight:bold">ul</span>, <span style="color:#b06;font-weight:bold">ol</span> {
<span style="color:#369">font-size</span>: <span style="color:#00d;font-weight:bold">1</span><span style="color:#00d;font-weight:bold">.8</span><span style="color:#888;font-weight:bold">rem</span>;
<span style="color:#369">margin</span>: <span style="color:#00d;font-weight:bold">0</span> <span style="color:#00d;font-weight:bold">0</span> <span style="color:#00d;font-weight:bold">.8</span><span style="color:#888;font-weight:bold">em</span>;
}
<span style="color:#b06;font-weight:bold">a</span> {
<span style="color:#369">text-decoration</span>: none;
<span style="color:#369">color</span>: <span style="color:#369">$paradise-pink</span>;
<span style="color:#080;font-weight:bold">&</span><span style="color:#555">:hover</span> {
<span style="color:#369">color</span>: <span style="color:#369">$paradise-pink-light</span>;
}
}
<span style="color:#b06;font-weight:bold">article</span> {
<span style="color:#369">padding</span>: <span style="color:#00d;font-weight:bold">2</span><span style="color:#888;font-weight:bold">rem</span> <span style="color:#00d;font-weight:bold">1</span><span style="color:#888;font-weight:bold">rem</span>;
<span style="color:#369">max-width</span>: <span style="color:#06b;font-weight:bold">calc</span>(<span style="color:#00d;font-weight:bold">44</span><span style="color:#888;font-weight:bold">em</span> + <span style="color:#00d;font-weight:bold">2</span><span style="color:#888;font-weight:bold">rem</span>);
<span style="color:#369">margin</span>: <span style="color:#00d;font-weight:bold">0</span> auto;
}
</code></pre></div><blockquote>
<p>See <a href="https://www.aleksandrhovhannisyan.com/blog/62-5-percent-font-size-trick/">Aleksandr Hovhannisyan’s article</a> for a good explanation of setting the <code>html</code> font size to 62.5%.</p>
</blockquote>
<h3 id="content">Content</h3>
<p>As I mentioned earlier, Hugo’s sections are defined by directories in the <code>content</code> directory. Here’s what my notes look like after I organize them by university semester:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">└── content
├── _index.md
├── winter-2022
│ ├── _index.md
│ ├── cs-235
│ │ ├── _index.md
│ │ ├── lecture-9.md
│ │ ├── lecture-10.md
│ │ └── quicksort-getting-started.md
└── winter-2023
├── _index.md
└── ec-en-240
├── _index.md
└── lecture-01-09.md
</code></pre></div><p>The <a href="https://gohugo.io/variables/page/#pages"><code>_index.md</code></a> files are used to add frontmatter and content to list pages. In this site, I just use them to name my sections. If you don’t specify anything, it’ll use the directory name.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">---
title: Electrical and Computer Engineering 240
---
</code></pre></div><p>And here’s one of those Markdown files; I’ll use <code>lecture-9.md</code> as an example:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-markdown" data-lang="markdown">---
title: "Lecture 9"
date: 2022-02-01T09:39:28-07:00
katex: true
---
<span style="color:#d20;background-color:#fff0f0">`sumto()`</span> function: \\(\sum_1^i i\\)
Can be done recursively: <span style="color:#d20;background-color:#fff0f0">`sumto(5) = 5 + sumto(4)`</span>
Recursive function rules:
<span style="color:#080;font-weight:bold">1.</span> have a base case (a case to stop the recursion)
<span style="color:#080;font-weight:bold">2.</span> the recursive function must progress toward the base case (otherwise it will recurse infinitely)
<span style="color:#080;font-weight:bold">3.</span> trust the induction
<span style="color:#080;font-weight:bold">4.</span> Make sure that it won't make too many recursive calls
<span style="color:#080;font-weight:bold">5.</span> Never run the same arguments of the recursive function twice. Caching may help
</code></pre></div><p>You may notice a <code>katex: true</code> key & value in the front matter. That’s an optional extra goody I’ve added, which we’ll go over soon!</p>
<h3 id="building-and-using-the-development-server">Building and using the development server</h3>
<p>To deploy the site to the <code>build</code> directory, run:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ hugo
</code></pre></div><p>Or to use Hugo’s development server (one of its strongest features, in my opinion), go to the base site directory and start it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ cd notes
$ hugo serve
</code></pre></div><p>It will give you a localhost address with a port (<code>1313</code> unless it’s taken by another service) that you can open in your browser of choice.</p>
<p>Now you should have a shiny new Hugo site to work with, no theme required!</p>
<p>Here’s the list page template in action:</p>
<p><img src="/blog/2023/02/how-to-create-a-hugo-website-without-a-theme/home.webp" alt="A web browser open to localhost:39333, showing the home page of the notes site: A header reads “Notes”, followed by smaller headers. The first is pink, reading “Winter-2022”, with its own bulleted list containing the item “Lecture 9”, also in pink. The second is black and reads “Other”, with no list beneath it."></p>
<p>And the single page template, plus our content file:</p>
<p><img src="/blog/2023/02/how-to-create-a-hugo-website-without-a-theme/lecture-9.webp" alt="A web browser open to localhost:39333/winter-2022/cs-235/lecture-9/. At the top of the page is pink text (a link) reading “Back to Winter-2022”. Below is a header reading “Lecture 9”. Followed by the text of the lecture-9.md file. sumto() appears in an inline code block, as well as the later sumto()... text."></p>
<h3 id="adding-katex-support">Adding KaTeX support</h3>
<p><a href="https://katex.org/">KaTeX</a> is a lightweight JavaScript TeX implementation. I use enough math in my notes as an engineering student that I wanted an easy way to show math notation.</p>
<p>Although I use math notation pretty often, I wanted to only load the JavaScript when necessary. For this, I set a front matter variable (<code>katex</code>), and render a <code>katex.html</code> partial in the site’s <code><head></code> if it’s there.</p>
<p>Add the following to <code>baseof.html</code>, in the <code><head></code> tag:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">{{ if .Params.katex }}{{ partial "partials/katex.html" . }}{{ end }}
</code></pre></div><p>The <code>partial</code> function takes a path relative to <code>layouts</code>, so <code>partials/katex.html</code> finds the file in <code>layouts/partials/katex.html</code>.</p>
<p>Put the following in <code>layouts/partials/katex.html</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><link rel="stylesheet" href="/katex/katex.min.css">
<script defer src="/katex/katex.min.js"></script>
<script defer src="/katex/contrib/auto-render.min.js" onload="renderMathInElement(document.body);"></script>
</code></pre></div><p>I <a href="https://katex.org/docs/browser.html#download--host-things-yourself">downloaded</a> KaTeX myself, which is nice for serving your own content while offline. But you could easily modify this partial to use CDN-hosted files.</p>
<p>If you host the files yourself, just download and move the necessary files to <code>static/katex</code>. You only need to keep a few files out of the ones KaTeX ships with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">└── static
└── katex
├── contrib
│ └── auto-render.min.js
├── fonts
│ ├── KaTeX_AMS-Regular.ttf
│ ├── KaTeX_AMS-Regular.woff
│ ├── KaTeX_AMS-Regular.woff2
│ ├── KaTeX_Caligraphic-Bold.ttf
│ ├── KaTeX_Caligraphic-Bold.woff
│ ├── KaTeX_Caligraphic-Bold.woff2
│ ├── KaTeX_Caligraphic-Regular.ttf
│ └── … (more fonts here, not shown to save space)
├── katex.min.css
└── katex.min.js
</code></pre></div><p>Remember that the contents of <code>static</code> are directly copied to the build directory, so you don’t need to include <code>static/</code> when referencing them in your templates.</p>
<p>Now we can turn on the <code>katex</code> switch and use math notation! The <a href="https://katex.org/docs/autorender.html#api">auto-render extension</a> will, by default, look for blocks denoted by <code>\\(</code> <code>\\)</code>, or <code>$$</code> <code>$$</code> for bigger symbols and centered.</p>
<p>Let’s make a new file in <code>content</code> called <code>katex-test.md</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-markdown" data-lang="markdown">---
title: "Some math notes"
katex: true
---
\\(
\text{A system of } m \text{ equations in } n \text{ variables:}\\newline
a_{1,1}x_1 + a_{1,2}x_2 + ... + a_{1,n}x_n = b_1\\newline
a_{2,1}x_1 + a_{2,2}x_2 + ... + a_{2,n}x_n = b_2\\newline
\vdots\\newline
a_{m,1}x_1 + a_{m,2}x_2 + ... + a_{m,n}x_n = b_m
\\)
This one's in "display mode" — centered, with larger symbols:
$$
\begin{bmatrix}
a_{1,1} <span style="color:#a61717;background-color:#e3d2d2">&</span> a_{1,2} <span style="color:#a61717;background-color:#e3d2d2">&</span> ... <span style="color:#a61717;background-color:#e3d2d2">&</span> a_{1,n} <span style="color:#a61717;background-color:#e3d2d2">&</span> \mid b_1\\\\
a_{2,1} <span style="color:#a61717;background-color:#e3d2d2">&</span> a_{2,2} <span style="color:#a61717;background-color:#e3d2d2">&</span> ... <span style="color:#a61717;background-color:#e3d2d2">&</span> a_{2,n} <span style="color:#a61717;background-color:#e3d2d2">&</span> \mid b_2\\\\
\vdots\\\\
a_{m,1} <span style="color:#a61717;background-color:#e3d2d2">&</span> 1_{m,2} <span style="color:#a61717;background-color:#e3d2d2">&</span> ... <span style="color:#a61717;background-color:#e3d2d2">&</span> a_{m,n} <span style="color:#a61717;background-color:#e3d2d2">&</span> \mid b_m
\end{bmatrix}
$$
</code></pre></div><blockquote>
<p>The highlighting here works fine, but it doesn’t work exactly how I’d expect. Markdown should always render first, then KaTeX should search for delimiters in the generated HTML, but the backslash behavior doesn’t seem consistent — I have to escape the backslash in <code>\newline</code> or Markdown catches the <code>\n</code>, but <code>\\(</code> collapses to <code>\(</code> and works fine, even with KaTeX looking for <code>\\(</code> (which looks like <code>\\\\(</code> when properly escaped in Markdown). Exploring that may be a topic for another blog post…</p>
</blockquote>
<p>Now you should be able to include math notation in your notes to your heart’s content!</p>
<p><img src="/blog/2023/02/how-to-create-a-hugo-website-without-a-theme/math-notes.webp" alt="A web browser open to localhost:39333/katex-test/. At the top of the page is pink text (a link) reading “Back to Notes”. Below is a header reading “Some math notes”. Followed by the text of the katex-test.md file. Most of the text is rendered in a math font by KaTeX. The first block is left-justified, while the second is center-justified."></p>
Middleware: Is that still a thing?https://www.endpointdev.com/blog/2022/05/middleware-is-that-still-a-thing/2022-05-26T00:00:00+00:00Richard Templet
<p><img src="/blog/2022/05/middleware-is-that-still-a-thing/20220401-013133.webp" alt="Photo looking from ground level up at concrete and crushed stone building against a blue sky with some white clouds"></p>
<!-- photo by Jon Jensen -->
<p>The simple answer to the question in the title is simply, yes! Despite the term being many decades old and well past its hype peak, middleware is still very much a thing and has become a key part of the technical landscape that is critical in day-to-day functioning of systems.</p>
<p>So there are still some questions to be answered: What is middleware? What does it do? And maybe most importantly: Why do we care?</p>
<h3 id="what-is-it">What is it?</h3>
<p>In its simplest meaning, middleware is an application that sits between other applications and shuffles data between them. There is normally one system requesting the information and the middleware figures out where to get that requested data and makes the request to another system.</p>
<p>An easy example of this is buying something at a retail store using your credit or debit card. When you swipe your card, the business makes a request to a service (some middleware) to ask if there’s enough room on the card for that purchase. Then that system makes a request to the appropriate bank or card holding company to ask the same question. The bank or card holding company replies with either a yes or no and that answer is then relayed back to the terminal where you swiped your card.</p>
<p>You might be thinking, “Why do we need the middleware? Just have the terminal ask the right bank!” Well, that’s a benefit of middleware: The terminal software doesn’t need to have all the same logic coded into it to know whether or how to talk to Visa or Mastercard as well as Citibank or Capital One or Bank of America to get the answer. It just talks to the middleware and lets it figure all of that out. This way you can have many different types of terminals or credit card taking applications available but only one middleware application needs to have the smarts and authorization to know where and how to get the answer.</p>
<p>The term middleware is most often used to describe business systems internal to a company. It is a subset of the broader concept “API” (application programming interface), which includes services provided for external users, either the general public or specific authorized customers or business partners. If a service is used directly by humans of the general public, it is typically called SaaS (software as a service).</p>
<h3 id="what-does-it-do">What does it do?</h3>
<p>As I explained in the example above, the simplistic definition of middleware is an application that shuffles data between two systems. In reality, middleware is sometimes way more complicated than that!</p>
<p>In that example of a credit card charge terminal, using middleware also allows for the retailer to set centralized company-wide rules for fraud screening, add or remove supported payment methods, or change which payment gateway is used, without having to update each terminal individually.</p>
<p>Middleware also can do data manipulation and mapping between two systems. Let’s consider a more complicated case like booking a cross-country flight.</p>
<p>You’ve probably used sites like Google Flights or Expedia to search for ticket costs. You go to their site, put in your starting airport(s) and your destination airport(s), select the dates when you’d like to travel, and maybe check the box saying you’ve got some wiggle room on when you leave and come back. After that you click the search button and within a few seconds, you’ve got tons of flight options to sort through! The search results can be sorted by price, number of stops, and airlines, to name a few.</p>
<p>So taking a step back to marvel at what just happened, how do you think they got that much data from that many sources so quickly? Well, the answer is middleware.</p>
<p>At some point, a middleware system made a request to each of the airlines to ask for their available flights with their dates, costs, amenities, etc. and stored that information on their own system to be used for your search. There are many different ways that data could be captured. We won’t delve into all that excitement, but what we do know is some system (middleware) had to fetch this data from each of the airlines and store it to be used for your search. It has to organize and store that data in a way that makes sense to the search engine to use so you can get back your search results in a speedy fashion.</p>
<p>There’s a good chance that the data from American Airlines will differ in its structure and data points compared to Delta. It’s the middleware’s job to take each of those data feeds and organize and store them somewhere for the search engine to use. In this part of the example, the middleware is taking a request to fetch the Delta flight data, turning that request into a request to another system for the data then doing whatever data manipulation and storage of that data to satisfy the search engine.</p>
<h3 id="why-do-we-care">Why do we care?</h3>
<p>The main reason that we care is we’ve become used to having a single location (website, application etc.) that allows us access to data from multiple places.</p>
<p>In our credit card example, there are many different types of credit cards we can use to check out at our local gas station. Having them require us to only use a Shell gas card would be bad for their business! We’ve grown accustomed to being able to use our Visa, Mastercard, American Express, or Shell gas card at that same gas station. This is all made possible due to middleware behind the scenes doing the hard work.</p>
<p>In many cases, the function of gathering data from many sources in one place, keeping it current, and allowing it to be searched, sorted, etc. is not just one means to an end, but it actually <em>is</em> the whole business! It is why services like Google Flights and Expedia exist.</p>
<p>Middleware can serve many other purposes even when only used internally by a single company:</p>
<ul>
<li><strong>authentication</strong> — making sure who- or whatever made a request is supposed to be able to</li>
<li><strong>authorization</strong> — making sure the kind of request being made is one this user is allowed to</li>
<li><strong>auditing</strong> — logging or sampling traffic in middleware is often more comprehensive and convenient than at each service individually</li>
<li><strong>rate-limiting</strong> — ensuring the user is not using too many resources too fast, or that one service isn’t overused and interfering with access to other services</li>
<li><strong>caching</strong> — storing answers to frequent requests to reuse for a limited time, reducing the number of live requests to busy internal applications and databases</li>
<li><strong>billing</strong> — tracking usage so it can be paid for or at least each user’s usage is fairly recorded</li>
<li><strong>fault-tolerance</strong> — retrying a request if an internal system is temporarily unavailable, or rerouting requests to a different system, to shield users from outages</li>
<li><strong>load balancing</strong> — spreading requests across multiple services to allow greater total throughput</li>
<li><strong>translation</strong> — modern APIs typically use JSON to structure their data, and middleware can translate between a nice JSON interface and older systems using a variety of more complex and archaic data formats</li>
<li><strong>migrations</strong> — transparently moving from an old system to a newer one, possibly in phases or temporarily for testing, without users having to change anything or even be aware</li>
</ul>
<p>In closing, while middleware isn’t commonly mentioned or even thought about, it is a very important part of what makes the technological world function.</p>
EditorConfig: Ending the Spaces vs. Tabs Confusionhttps://www.endpointdev.com/blog/2022/04/editorconfig-ending-spaces-vs-tabs-confusion/2022-04-30T00:00:00+00:00Jon Jensen
<p><img src="/blog/2022/04/editorconfig-ending-spaces-vs-tabs-confusion/20220316_184133.webp" alt="">
Photo by Garrett Skinner</p>
<h3 id="varieties-of-text-formatting">Varieties of text formatting</h3>
<p>Most everyone who has worked on a software development project with a group of other people has encountered the problem of source code being formatted in different ways by different text editors, IDEs, and operating systems.</p>
<p>The main variations go back to the 1970s or earlier, and include the questions:</p>
<ul>
<li>Will indentation be done by tabs (an ASCII control character) or spaces?
<ul>
<li>If indentation is done by spaces, how many spaces are used for each indentation level?</li>
</ul>
</li>
<li>What will indicate the end of each line (EOL)? The choices are:
<ul>
<li>a line feed (LF), used by the Unix family including Linux and modern macOS</li>
<li>a carriage return (CR), used by old pre-Unix Macintosh and some now-obscure operating systems</li>
<li>both together (CRLF) used by Windows and most Internet protocols</li>
</ul>
</li>
<li>Which character set encoding will be used? Common choices are:
<ul>
<li>Unicode UTF-8 encoding, used by Linux, macOS, and most other Unixes, and standard on the Internet</li>
<li>Unicode UTF-16 encoding (with either little-endian or big-endian encoding), used by modern Windows</li>
<li>legacy ISO-8859 and Windows “code page” encodings in older documents and codebases</li>
</ul>
</li>
</ul>
<h3 id="editor-configurations-in-conflict">Editor configurations in conflict</h3>
<p>Causing widespread frustration, by default, text editors and IDEs generally are each configured differently, and once set, the choices apply broadly from then on. But each developer can simply configure their editor to follow their team’s standards, right? Well, maybe.</p>
<p>First, getting that to happen for every developer and every different editor being used isn’t straightforward. It typically requires a document showing instructions and/or screenshots of how to configure each editor. It may have to be redone after a major upgrade or move to a new computer.</p>
<p>Second, and often a more persistent problem, standards may vary across different projects and even for different types of files within a given project. Ruby code is typically indented with 2 spaces, while perhaps in your project JavaScript uses 4 spaces and HTML uses tabs.</p>
<p>If you start a new project from scratch you can probably settle on a single standard, but in existing large codebases, it can make a lot of version control change “noise” to mess with that.</p>
<p>Computers are good at keeping track of lots of little details, so isn’t there some way to have the computer deal with this?</p>
<h3 id="storing-configuration-in-the-project">Storing configuration in the project</h3>
<p>What if we store the text editor’s or IDE’s configuration in the project instead of per user, so it can go with the project to each new developer and tell their editor how to behave?</p>
<p>For many years that has been possible with some editors, but the configuration had to be set up separately for each editor, and often the feature is disabled by default.</p>
<p>Let’s consider the two most popular terminal-based editors on Unix, partisans in a long-running editor war:</p>
<h4 id="vim">Vim</h4>
<p>Vim has a feature called a “modeline” that allows for configuration settings to appear within the top or bottom 5 lines of the file.</p>
<p>For example, to instruct Vim to use spaces instead of tabs and 4-space tab stops, we can add to the top or bottom of our C source code file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#888">/* vim: tabstop=4 shiftwidth=4 expandtab
</span><span style="color:#888"> */</span>
</code></pre></div><p>Since it gets tedious putting those special configuration comments in each file, Vim has an option to read a <code>.vimrc</code> file from the current directory, which applies to all files there and can be committed to version control.</p>
<p>This feature is disabled by default because Vim has in the past been vulnerable to files with malicious settings running arbitrary code.</p>
<p>You can <code>:set exrc secure</code> to enable the modeline feature in a code base you trust, and also to restrict what it can do.</p>
<h4 id="emacs">Emacs</h4>
<p>In Emacs the same thing can be done on the first or second line of the file. (Of course its setting names differ from Vim’s.) For example consider this configuration in C source code:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#888">/* -*- mode: c; indent-tabs-mode: nil; c-basic-offset: 4; tab-width: 4 -*- */</span>
</code></pre></div><p>Alternately you can use “Local Variables” set at the end of the file in as many lines as needed:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-c" data-lang="c"><span style="color:#888">/* Local Variables: */</span>
<span style="color:#888">/* mode: c */</span>
<span style="color:#888">/* indent-tabs-mode: nil */</span>
<span style="color:#888">/* c-basic-offset: 4 */</span>
<span style="color:#888">/* tab-width: 4 */</span>
<span style="color:#888">/* End: */</span>
</code></pre></div><p>Emacs also has “Directory Variables” that can be set in the file <code>.dir-locals.el</code> for a directory and its subdirectories.</p>
<h4 id="others">Others</h4>
<p>Even if someone has gone to the trouble to set up such editor configuration files and add them to the project code repository, how often has that been done for your editor or IDE?</p>
<p>And how often is one out of sync with the others?</p>
<p>This is not the way to success.</p>
<h3 id="editorconfig-to-the-rescue">EditorConfig to the rescue</h3>
<p>About 10 years ago Trey Hunner and Hong Xu shared with the world EditorConfig, their creation to solve this problem across ideally all editors.</p>
<p>They intentionally kept EditorConfig fairly limited in scope. It covers a limited number of the most important editor options so that the standard would be simple enough to be implemented for every editor either internally or as a plugin, and there would be no arbitrary code execution possible to cause security problems.</p>
<p>In EditorConfig the configuration for our examples and hypotheticals above lives in a <code>.editorconfig</code> file in the root of the project that looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#888"># top-most EditorConfig file</span>
<span style="color:#369">root</span> = <span style="color:#d20;background-color:#fff0f0">true</span>
<span style="color:#888"># basics for all files in our project</span>
<span style="color:#080;font-weight:bold">[*]</span>
<span style="color:#369">charset</span> = <span style="color:#d20;background-color:#fff0f0">utf-8</span>
<span style="color:#369">end_of_line</span> = <span style="color:#d20;background-color:#fff0f0">lf</span>
<span style="color:#888"># C and JavaScript source get 4-space indents</span>
<span style="color:#080;font-weight:bold">[*.{c,js}]</span>
<span style="color:#369">indent_style</span> = <span style="color:#d20;background-color:#fff0f0">space</span>
<span style="color:#369">indent_size</span> = <span style="color:#d20;background-color:#fff0f0">4</span>
<span style="color:#888"># Ruby gets 2-space indents</span>
<span style="color:#080;font-weight:bold">[*.rb]</span>
<span style="color:#369">indent_style</span> = <span style="color:#d20;background-color:#fff0f0">space</span>
<span style="color:#369">indent_size</span> = <span style="color:#d20;background-color:#fff0f0">2</span>
<span style="color:#888"># HTML gets tab indents</span>
<span style="color:#080;font-weight:bold">[*.html]</span>
<span style="color:#369">indent_style</span> = <span style="color:#d20;background-color:#fff0f0">tab</span>
</code></pre></div><p>In a big project you may want to have separate, smaller <code>.editorconfig</code> files in different directories. You can omit the <code>root = true</code> setting in subdirectories to inherit settings from the top-level <code>.editorconfig</code> file.</p>
<p>There are a couple of other options that are nice to specify.</p>
<p>This one removes any tabs or spaces from the end of lines:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#369">trim_trailing_whitespace</span> = <span style="color:#d20;background-color:#fff0f0">true</span>
</code></pre></div><p>Those are rarely needed or semantically meaningful, so it’s nice to remove them. But there are a few cases where they can matter such as in Markdown.</p>
<p>This one determines whether the last line in the file will end with a newline:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#369">insert_final_newline</span> = <span style="color:#d20;background-color:#fff0f0">true</span>
</code></pre></div><p>By default some editors add to the last line a newline (such as Vim) and some don’t (such as Emacs), leading to needless changes as various developers change files.</p>
<p>Typically every line should end with a newline, so that’s a good editor feature to enable. But you could have some text template that should <em>not</em> end with a newline, so might need to specify <code>false</code> for that type of file.</p>
<p>And those are most of the features of EditorConfig! <a href="https://editorconfig.org/#file-format-details">The file format details</a> are easy to digest.</p>
<h3 id="editor-amp-ide-support">Editor & IDE support</h3>
<p>EditorConfig is now widely supported. These popular editors & IDEs recognize <code>.editorconfig</code> files with no extra work:</p>
<ul>
<li>IntelliJ IDEA and most of its language-specific variants</li>
<li>GitHub</li>
<li>GitLab</li>
<li>Visual Studio</li>
<li>BBEdit</li>
<li>and others</li>
</ul>
<p>And these support it with a plugin:</p>
<ul>
<li>VS Code</li>
<li>Vim</li>
<li>Emacs</li>
<li>Sublime Text</li>
<li>TextMate</li>
<li>Eclipse</li>
<li>Atom</li>
<li>Notepad++</li>
<li>Geany</li>
<li>and others</li>
</ul>
<p>The plugins are typically easy to install system-wide from your operating system’s package manager, or else locally for your user only.</p>
<h3 id="do-you-need-it">Do you need it?</h3>
<p>Yes, I think you do.</p>
<p>I know of no reason for any developer not to use EditorConfig, in every editor, for every project. It’s simple and at long last solves this small set of problems well.</p>
<p>One possible counterargument: If, before every version control commit, you run an automatic code formatter such as Prettier (in Node.js, for many languages) or a language-specific one such as <code>gofmt</code>, <code>rustfmt</code>, etc., you could perhaps live without your editor knowing how your files should be saved.</p>
<p>But isn’t it better if your editor knows what kind of line endings and indents to use, rather than waiting for a code formatter to correct such fundamental things after you save? It is easy to start with a single <code>.editorconfig</code> file long before you have a continuous integration set up for the project.</p>
<p>And many projects don’t format code automatically, and instead just “lint” it to report on deviations from the project standards. But that requires work to correct, and can be ignored if not enforced.</p>
<p>Many <a href="https://github.com/editorconfig/editorconfig/wiki/Projects-Using-EditorConfig">open source projects large and small use EditorConfig</a>, including this blog itself. But in recent months I have found several developers who had not yet heard of EditorConfig, so I want to spread awareness of it. I hope you’ll <a href="https://editorconfig.org/">use EditorConfig</a> too!</p>
Quartz Scheduler as a Servicehttps://www.endpointdev.com/blog/2022/04/quartz-scheduler-as-a-service/2022-04-18T00:00:00+00:00Kürşat Kutlu Aydemir
<p><img src="/blog/2022/04/quartz-scheduler-as-a-service/pexels-mat-brown-552598.webp" alt="Close-up view of mechanical watch with roman numerals and day of month and month pointers"></p>
<p>Photo by Mat Brown from Pexels</p>
<h3 id="quartz-job-scheduler">Quartz Job Scheduler</h3>
<p>“Quartz is a richly featured, open source job scheduling library that can be integrated within virtually any Java application — from the smallest stand-alone application to the largest e-commerce system.” (<a href="http://www.quartz-scheduler.org/overview/">Quartz Scheduler overview</a>)</p>
<p>Besides its advanced features, most basic and frequently used feature is job scheduling and job execution. Some frameworks like Spring Scheduler have their integration practice using Quartz Scheduler which allows using its default scheduling method.</p>
<p>In this post I am going to tell you a different approach to show how we can use Quartz Scheduler to schedule our jobs. We actually still will be using the existing scheduling mechanism of Quartz but we’re going to show how we can manage the scheduled and unscheduled jobs online. This way you can manage all the available jobs or create new ones on the fly.</p>
<h3 id="quartz-scheduler-as-a-service">Quartz Scheduler as a Service</h3>
<p>Previously I led development of an enterprise “Business Service Management” software to replace IBM’s TBSM product at a major telco company in Turkey. This was a challenging project and found a solid place in the customer environment after a successful release.</p>
<p>Scheduled key performance indicators (KPI) retrieval and background reporting jobs were a significant part of this project. KPIs were either internal business service availability and health metrics or measured metrics calculated and stored in external data sources. Reports are also another type of schedulable jobs as many organizations need the data to be reported at certain intervals.</p>
<p>In an enterprise web application with such needs you would need to allow your customer to create their own customized scheduled jobs (KPIs, reports, etc.) in an easily manageable way. For this I came up with a simple solution by blending the existing Quartz Scheduler scheduling mechanism with some spice.</p>
<p>So here is the model we used:</p>
<ul>
<li>A database table for creating/updating scheduler job definitions</li>
<li>Observer Schduler Job for observing the scheduler job table to watch for any updates in the scheduled jobs: new job, updated job, or disabled job, etc.</li>
<li>Business Job: You might define several schedulable business job types. KPI is one of those and I am going to give an example of it.</li>
</ul>
<p>Simplicity should be a design goal, however the details can have their complexities.</p>
<p>This design doesn’t replace or provide an alternative to how Quartz Scheduler schedules its jobs. That is subject to job persistence and is out of this article’s scope. I am assuming we are scheduling the jobs all in Quartz Scheduler’s RAM-store or Job-store.</p>
<h4 id="read-and-manage-job-data">Read and Manage Job Data</h4>
<p>Ideally you should store and manage the jobs as services in a database and you can then connect to this job storage either via DB connection or API. For security reasons even if you think that your application or services are internal and totally authenticated and authorised you should still perform DB operations via APIs. But for capability perspective yes you can use many ways to read and manage a data storage.</p>
<p>For this simple project I am not going to use a database but instead a JSON file as job service definitions repository. But you can simply convert this method to a database or API method.</p>
<p>I am going to use a JSON file named <code>kpi.json</code> in my project and define a simple set of attributes for each KPI item. Any service or scheduled job can have more or fewer attributes according to the requirements of the business use case.</p>
<h4 id="spring-application">Spring Application</h4>
<p>You can use any framework or even without using any framework you can create your application from scratch and build a JAR. Here in this project I chose to go with Spring framework. You can also simply initialize a Spring application <a href="https://start.spring.io/">here</a>.</p>
<h4 id="design">Design</h4>
<p>As I suggested a model above as a scheduling service solution, here is a high-level design of the model.</p>
<p><img src="/blog/2022/04/quartz-scheduler-as-a-service/qs-service-design.png" alt="Quartz Scheduler service model diagram"></p>
<p>The overall solution would have a data storage for holding scheduled job service definitions and a UI for managing their attributes like enabling/disabling or changing scheduling dates etc.</p>
<p>In this solution we have two different Quartz Scheduler job types: observer job and business job. Observer job is a single job triggered frequently, say, every 5 seconds or every 1 minute, and checks the existing job definitions in the job storage. If it sees any update on the job definitions or new jobs it behaves accordingly. Business jobs are the job definitions found in job storage and designed to perform certain business actions. The business jobs can be notification jobs, KPI measuring jobs, and any other scheduled business jobs which should have their own scheduling interval.</p>
<p>In this example project I specifically used KPI term as the business case just to make it more relevant.</p>
<h4 id="scheduler">Scheduler</h4>
<p><code>KPIJobWatcher</code> class is responsible to schedule the observer job. In Spring application startup this is going to be our starting point to the scheduling service management.</p>
<h4 id="spring-application-startup">Spring Application Startup</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#555">@SpringBootApplication</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">QSchedulerApplication</span> {
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">main</span>(String[] args) {
SpringApplication.<span style="color:#369">run</span>(QSchedulerApplication.<span style="color:#369">class</span>, args);
}
Scheduler kpiScheduler;
<span style="color:#555">@EventListener</span>(ApplicationReadyEvent.<span style="color:#369">class</span>)
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">onAppStartUp</span>() {
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#888">// initializing KPI Trigger
</span><span style="color:#888"></span> SchedulerFactory sf = <span style="color:#080;font-weight:bold">new</span> StdSchedulerFactory();
kpiScheduler = sf.<span style="color:#369">getScheduler</span>();
<span style="color:#888">// watcher runs an observer job which monitors and manages KPI jobs
</span><span style="color:#888"></span> KPIJobWatcher watcher = <span style="color:#080;font-weight:bold">new</span> KPIJobWatcher(kpiScheduler);
watcher.<span style="color:#369">run</span>();
} <span style="color:#080;font-weight:bold">catch</span> (Exception e) {
e.<span style="color:#369">printStackTrace</span>();
}
}
}
</code></pre></div><p><code>KPIJobWatcher</code> schedules the Observer job:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">KPIJobWatcher</span> {
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Logger logger = LoggerFactory.<span style="color:#369">getLogger</span>(KPIJobWatcher.<span style="color:#369">class</span>);
Scheduler kpiScheduler;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#06b;font-weight:bold">KPIJobWatcher</span>(Scheduler s) {
kpiScheduler = s;
}
<span style="color:#888">/**
</span><span style="color:#888"> * run KPIJobWatcher
</span><span style="color:#888"> * @throws Exception
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">run</span>() <span style="color:#080;font-weight:bold">throws</span> Exception {
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#888">// Setting the KPI Job factory of observerScheduler
</span><span style="color:#888"></span> KPIJobFactory jf = <span style="color:#080;font-weight:bold">new</span> KPIJobFactory((StdScheduler)kpiScheduler);
kpiScheduler.<span style="color:#369">setJobFactory</span>(jf);
<span style="color:#888">// Scheduling KPI Observer Job
</span><span style="color:#888"></span> JobDetail observerJob = newJob(KPIObserverJob.<span style="color:#369">class</span>)
.<span style="color:#369">withIdentity</span>(<span style="color:#d20;background-color:#fff0f0">"observerJob"</span>, <span style="color:#d20;background-color:#fff0f0">"observergroup"</span>)
.<span style="color:#369">build</span>();
SimpleTrigger trigger = newTrigger()
.<span style="color:#369">withIdentity</span>(observerJob.<span style="color:#369">getKey</span>() + <span style="color:#d20;background-color:#fff0f0">"_trigger"</span>, <span style="color:#d20;background-color:#fff0f0">"observergroup"</span>)
.<span style="color:#369">withSchedule</span>(org.<span style="color:#369">quartz</span>.<span style="color:#369">SimpleScheduleBuilder</span>.<span style="color:#369">simpleSchedule</span>()
.<span style="color:#369">withIntervalInSeconds</span>(10)
.<span style="color:#369">repeatForever</span>())
.<span style="color:#369">build</span>();
Date ft = kpiScheduler.<span style="color:#369">scheduleJob</span>(observerJob, trigger);
logger.<span style="color:#369">info</span>(observerJob.<span style="color:#369">getKey</span>() + <span style="color:#d20;background-color:#fff0f0">" has been scheduled to run at: "</span> + ft);
<span style="color:#888">// Starting KPI Observer Scheduler
</span><span style="color:#888"></span> kpiScheduler.<span style="color:#369">start</span>();
} <span style="color:#080;font-weight:bold">catch</span> (Exception e) {
e.<span style="color:#369">printStackTrace</span>();
}
}
}
</code></pre></div><p>Before moving on to observer job here I want to notice that you can use a custom <code>JobFactory</code> and attach it to the current scheduler object so that you can use custom jobs with custom constructors created within this custom JobFactory as part of the factory design pattern.</p>
<h4 id="jobfactory">JobFactory</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">KPIJobFactory</span> <span style="color:#080;font-weight:bold">implements</span> JobFactory {
Scheduler kpiScheduler;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#06b;font-weight:bold">KPIJobFactory</span>(Scheduler s) {
kpiScheduler = s;
}
<span style="color:#080;font-weight:bold">public</span> KPIObserverJob <span style="color:#06b;font-weight:bold">newJob</span>(TriggerFiredBundle bundle, Scheduler Scheduler) <span style="color:#080;font-weight:bold">throws</span> SchedulerException {
JobDetail jobDetail = bundle.<span style="color:#369">getJobDetail</span>();
Class<KPIObserverJob> jobClass = (Class<KPIObserverJob>) jobDetail.<span style="color:#369">getJobClass</span>();
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#888">// this is how we construct our custom job with custom factory
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> jobClass.<span style="color:#369">getConstructor</span>(Scheduler.<span style="color:#369">getClass</span>()).<span style="color:#369">newInstance</span>(kpiScheduler);
} <span style="color:#080;font-weight:bold">catch</span> (Exception e) {
e.<span style="color:#369">printStackTrace</span>();
}
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">null</span>;
}
}
</code></pre></div><h4 id="observer-job">Observer Job</h4>
<p>As suggested in the model above the observer job is triggered frequently and manages the overall scheduling job service in the background. I created the <code>KPIObserverJob</code> class as the observer job in this project and as you can see in the previous section <code>KPIJobFactory</code> creates instances of this observer job.</p>
<h5 id="kpiobserverjob">KPIObserverJob</h5>
<p>Observer Job has some specific methods like <code>ScheduleJob</code> and <code>UnscheduleJob</code> to manage scheduling jobs.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-Java" data-lang="Java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">KPIObserverJob</span> <span style="color:#080;font-weight:bold">implements</span> Job {
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Logger logger = LoggerFactory.<span style="color:#369">getLogger</span>(KPIObserverJob.<span style="color:#369">class</span>);
List<JobDetail> jobList;
Scheduler kpiScheduler;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#06b;font-weight:bold">KPIObserverJob</span>(StdScheduler s) {
kpiScheduler = s;
}
List<String> scheduledJobList;
HashMap<String, JobDetail> alreadyScheduledJobList;
String cronFormat = <span style="color:#d20;background-color:#fff0f0">"SECOND MINUTE HOUR DAY_OF_MON MONTH DAY_OF_WEEK"</span>;
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">execute</span>(JobExecutionContext context) <span style="color:#080;font-weight:bold">throws</span> JobExecutionException {
scheduledJobList = <span style="color:#080;font-weight:bold">new</span> ArrayList<String>();
alreadyScheduledJobList = <span style="color:#080;font-weight:bold">new</span> HashMap<String, JobDetail>();
<span style="color:#888">//JobKey jobKey = context.getJobDetail().getKey();
</span><span style="color:#888"></span>
jobList = <span style="color:#080;font-weight:bold">new</span> ArrayList<JobDetail>();
<span style="color:#888">// Get all KPIs and create their jobs
</span><span style="color:#888"></span> CreateJobs();
<span style="color:#888">// Get the list of currently scheduled KPI jobs
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">try</span> {
<span style="color:#080;font-weight:bold">for</span> (String groupName : kpiScheduler.<span style="color:#369">getJobGroupNames</span>()) {
<span style="color:#080;font-weight:bold">for</span> (JobKey jk : kpiScheduler.<span style="color:#369">getJobKeys</span>(GroupMatcher.<span style="color:#369">jobGroupEquals</span>(groupName))) {
String jobName = jk.<span style="color:#369">getName</span>();
String jobGroup = jk.<span style="color:#369">getGroup</span>();
scheduledJobList.<span style="color:#369">add</span>(jobName);
JobDetail jd = kpiScheduler.<span style="color:#369">getJobDetail</span>(jk);
alreadyScheduledJobList.<span style="color:#369">put</span>(jobName, jd);
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"already scheduled jobName {}"</span>, jobName);
}
}
} <span style="color:#080;font-weight:bold">catch</span> (SchedulerException e) {
e.<span style="color:#369">printStackTrace</span>();
}
<span style="color:#888">// Schedule or unschedule KPI jobs if not done yet
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">for</span> (JobDetail job : jobList) {
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#080;font-weight:bold">if</span> (!scheduledJobList.<span style="color:#369">contains</span>(job.<span style="color:#369">getKey</span>().<span style="color:#369">getName</span>()))
{
<span style="color:#080;font-weight:bold">if</span> (job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getInt</span>(<span style="color:#d20;background-color:#fff0f0">"isRunning"</span>) == 1) {
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"scheduling job: kpiJobName_{}"</span>, job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
ScheduleJob(job);
}
} <span style="color:#080;font-weight:bold">else</span> {
<span style="color:#888">// Check any changes in the KPI job definition
</span><span style="color:#888"></span> JobDetail sJD = alreadyScheduledJobList.<span style="color:#369">get</span>(<span style="color:#d20;background-color:#fff0f0">"kpiJobName_"</span> + job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
<span style="color:#080;font-weight:bold">if</span> (!job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>).<span style="color:#369">equals</span>(sJD.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>))) {
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"rescheduling job: kpiJobName {} , new cron: {}"</span>,
job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>), job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>));
UnscheduleJob(job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
ScheduleJob(job);
}
<span style="color:#080;font-weight:bold">if</span> (job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getInt</span>(<span style="color:#d20;background-color:#fff0f0">"isRunning"</span>) == 0) {
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"Unscheduling: kpiJobName {}"</span> + job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
UnscheduleJob(job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
}
}
} <span style="color:#080;font-weight:bold">catch</span> (SchedulerException e) {
e.<span style="color:#369">printStackTrace</span>();
}
}
<span style="color:#888">// Finally unschedule deleted jobs if they are not listed anymore
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">for</span> (String kpiName : scheduledJobList) {
<span style="color:#888;font-weight:bold">boolean</span> unschedule = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#080;font-weight:bold">if</span> (!kpiName.<span style="color:#369">equals</span>(<span style="color:#d20;background-color:#fff0f0">"observerJob"</span>)) {
JobDetail toBeRemovedJob = <span style="color:#080;font-weight:bold">null</span>;
<span style="color:#080;font-weight:bold">for</span> (JobDetail jdetail : jobList) {
<span style="color:#080;font-weight:bold">if</span> (jdetail.<span style="color:#369">getKey</span>().<span style="color:#369">getName</span>().<span style="color:#369">equals</span>(kpiName)) {
unschedule = <span style="color:#080;font-weight:bold">false</span>;
}
}
<span style="color:#080;font-weight:bold">if</span> (unschedule) {
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"Unscheduling: "</span> + <span style="color:#d20;background-color:#fff0f0">"kpiJobId"</span> + kpiName.<span style="color:#369">split</span>(<span style="color:#d20;background-color:#fff0f0">"_"</span>)[1]);
UnscheduleJob(kpiName.<span style="color:#369">split</span>(<span style="color:#d20;background-color:#fff0f0">"_"</span>)[1]);
}
}
}
}
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Type KPI_JSON_TYPE = <span style="color:#080;font-weight:bold">new</span> TypeToken<List<KPI_JSON>>() {}.<span style="color:#369">getType</span>();
<span style="color:#888">/**
</span><span style="color:#888"> * Create Quartz Scheduler jobs from the job records read from a data source
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">CreateJobs</span>() {
Gson gson = <span style="color:#080;font-weight:bold">new</span> Gson();
<span style="color:#080;font-weight:bold">try</span> {
<span style="color:#888">// kpi.json as a service data storage where we get KPI job data to be scheduled
</span><span style="color:#888"></span> JsonReader reader = <span style="color:#080;font-weight:bold">new</span> JsonReader(<span style="color:#080;font-weight:bold">new</span> FileReader(<span style="color:#d20;background-color:#fff0f0">"kpi.json"</span>));
List<KPI_JSON> kpiList = gson.<span style="color:#369">fromJson</span>(reader, KPI_JSON_TYPE);
<span style="color:#080;font-weight:bold">for</span> (KPI_JSON kpiItem : kpiList) {
logger.<span style="color:#369">info</span>(<span style="color:#d20;background-color:#fff0f0">"Found KPI in kpi.json: {} , enabled: {}"</span>, kpiItem.<span style="color:#369">getName</span>(), kpiItem.<span style="color:#369">getIsRunning</span>());
JobDetail job = newJob(KPIJSONJob.<span style="color:#369">class</span>)
.<span style="color:#369">withIdentity</span>(<span style="color:#d20;background-color:#fff0f0">"kpiJobName_"</span> + kpiItem.<span style="color:#369">getName</span>(), <span style="color:#d20;background-color:#fff0f0">"kpigroup"</span>)
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>, kpiItem.<span style="color:#369">getName</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>, kpiItem.<span style="color:#369">getCron</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"lastRan"</span>, kpiItem.<span style="color:#369">getLastRan</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"kpiDescription"</span>, kpiItem.<span style="color:#369">getKpiDescription</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"lastMeasuredValue"</span>, kpiItem.<span style="color:#369">getLastMeasuredValue</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"filename"</span>, kpiItem.<span style="color:#369">getFilename</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"type"</span>, kpiItem.<span style="color:#369">getType</span>())
.<span style="color:#369">usingJobData</span>(<span style="color:#d20;background-color:#fff0f0">"isRunning"</span>, kpiItem.<span style="color:#369">getIsRunning</span>())
.<span style="color:#369">build</span>();
jobList.<span style="color:#369">add</span>(job);
}
} <span style="color:#080;font-weight:bold">catch</span> (Exception e) {
e.<span style="color:#369">printStackTrace</span>();
}
}
<span style="color:#888">/**
</span><span style="color:#888"> * Schedule a job
</span><span style="color:#888"> * @param job
</span><span style="color:#888"> * @throws SchedulerException
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">ScheduleJob</span>(JobDetail job) <span style="color:#080;font-weight:bold">throws</span> SchedulerException {
String cron = job.<span style="color:#369">getJobDataMap</span>().<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>);
CronTrigger trigger = newTrigger()
.<span style="color:#369">withIdentity</span>(job.<span style="color:#369">getKey</span>().<span style="color:#369">getName</span>() + <span style="color:#d20;background-color:#fff0f0">"_trigger"</span>, <span style="color:#d20;background-color:#fff0f0">"kpigroup"</span>)
.<span style="color:#369">withSchedule</span>(cronSchedule(cron))
.<span style="color:#369">startNow</span>()
.<span style="color:#369">build</span>();
Date ft = kpiScheduler.<span style="color:#369">scheduleJob</span>(job, trigger);
}
<span style="color:#888">/**
</span><span style="color:#888"> * Unschedule a job
</span><span style="color:#888"> * @param kpiName
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">UnscheduleJob</span>(String kpiName) {
TriggerKey tk = <span style="color:#080;font-weight:bold">new</span> TriggerKey(<span style="color:#d20;background-color:#fff0f0">"kpiJobName_"</span> + kpiName + <span style="color:#d20;background-color:#fff0f0">"_trigger"</span>, <span style="color:#d20;background-color:#fff0f0">"kpigroup"</span>);
<span style="color:#080;font-weight:bold">try</span> {
kpiScheduler.<span style="color:#369">unscheduleJob</span>(tk);
kpiScheduler.<span style="color:#369">deleteJob</span>(<span style="color:#080;font-weight:bold">new</span> JobKey(<span style="color:#d20;background-color:#fff0f0">"kpiJobName_"</span> + kpiName, <span style="color:#d20;background-color:#fff0f0">"kpigroup"</span>));
} <span style="color:#080;font-weight:bold">catch</span> (SchedulerException e) {
e.<span style="color:#369">printStackTrace</span>();
}
}
}
</code></pre></div><h4 id="business-jobs">Business Jobs</h4>
<p>Business jobs, as suggested in the model, can be any schedulable jobs. Managing/updating the business jobs frequently is a key point here. As the enterprise demands grow and change continuously, KPIs are generated at intervals (daily, weekly, monthly, etc.) and for frequent notification needs this kind of scheduling job management can be an important part of a solution.</p>
<p>Here I created <code>KPIJSONJob</code> as my business job:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">KPIJSONJob</span> <span style="color:#080;font-weight:bold">implements</span> Job {
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Logger logger = LoggerFactory.<span style="color:#369">getLogger</span>(KPIJSONJob.<span style="color:#369">class</span>);
<span style="color:#080;font-weight:bold">private</span> KPI_JSON kpi;
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">execute</span>(JobExecutionContext context) <span style="color:#080;font-weight:bold">throws</span> JobExecutionException {
JobDataMap dataMap = context.<span style="color:#369">getJobDetail</span>().<span style="color:#369">getJobDataMap</span>();
kpi.<span style="color:#369">setName</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiName"</span>));
kpi.<span style="color:#369">setKpiDescription</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"kpiDescription"</span>));
kpi.<span style="color:#369">setIsRunning</span>(dataMap.<span style="color:#369">getInt</span>(<span style="color:#d20;background-color:#fff0f0">"isRunning"</span>));
kpi.<span style="color:#369">setFilename</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"filename"</span>));
kpi.<span style="color:#369">setCron</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"cron"</span>));
kpi.<span style="color:#369">setLastRan</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"lastRan"</span>));
kpi.<span style="color:#369">setType</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"type"</span>));
kpi.<span style="color:#369">setLastMeasuredValue</span>(dataMap.<span style="color:#369">getString</span>(<span style="color:#d20;background-color:#fff0f0">"lastMeasuredValue"</span>));
<span style="color:#080;font-weight:bold">this</span>.<span style="color:#369">processKPI</span>();
}
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">KPIMeasured</span> {
<span style="color:#080;font-weight:bold">public</span> String name;
<span style="color:#080;font-weight:bold">public</span> String value;
}
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Type KPIMEASURED_TYPE = <span style="color:#080;font-weight:bold">new</span> TypeToken<List<KPIMeasured>>() {}.<span style="color:#369">getType</span>();
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">processKPI</span>() {
<span style="color:#888">// processKPI is supposed to get the KPI measured value from an external datasource and updates kpi.json
</span><span style="color:#888"></span> <span style="color:#888">// ...
</span><span style="color:#888"></span> }
}
</code></pre></div><h4 id="running-this-solution">Running this Solution</h4>
<p>Let’s give it a try and see it in action. Say we have the <code>kpi.json</code> as our job storage, with the following KPI jobs defined:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">[
{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"critical_ticket_count"</span>,
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"JSON_FILE"</span>,
<span style="color:#b06;font-weight:bold">"cron"</span>: <span style="color:#d20;background-color:#fff0f0">"0 0 1 ? * * *"</span>,
<span style="color:#b06;font-weight:bold">"isRunning"</span>: <span style="color:#00d;font-weight:bold">0</span>,
<span style="color:#b06;font-weight:bold">"lastRan"</span>: <span style="color:#d20;background-color:#fff0f0">"2022-04-01 01:00:00"</span>,
<span style="color:#b06;font-weight:bold">"kpiDescription"</span>: <span style="color:#d20;background-color:#fff0f0">"Open critical ticket count"</span>,
<span style="color:#b06;font-weight:bold">"lastMeasuredValue"</span>: <span style="color:#d20;background-color:#fff0f0">"7"</span>,
<span style="color:#b06;font-weight:bold">"filename"</span>: <span style="color:#d20;background-color:#fff0f0">"kpi_measured.json"</span>
},
{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"failed_customer_api_call"</span>,
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"JSON_FILE"</span>,
<span style="color:#b06;font-weight:bold">"cron"</span>: <span style="color:#d20;background-color:#fff0f0">"0 0 2 ? * * *"</span>,
<span style="color:#b06;font-weight:bold">"isRunning"</span>: <span style="color:#00d;font-weight:bold">0</span>,
<span style="color:#b06;font-weight:bold">"lastRan"</span>: <span style="color:#d20;background-color:#fff0f0">"2022-04-01 02:00:00"</span>,
<span style="color:#b06;font-weight:bold">"kpiDescription"</span>: <span style="color:#d20;background-color:#fff0f0">"Last 24-Hour failed API call count"</span>,
<span style="color:#b06;font-weight:bold">"lastMeasuredValue"</span>: <span style="color:#d20;background-color:#fff0f0">"23"</span>,
<span style="color:#b06;font-weight:bold">"filename"</span>: <span style="color:#d20;background-color:#fff0f0">"kpi_measured.json"</span>
}
]
</code></pre></div><p>When we run the Spring application it starts logging like below:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">2022-04-11 13:48:12.354 INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: critical_ticket_count , enabled: 0
2022-04-11 13:48:12.355 INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:48:12.355 INFO 13033 --- [eduler_Worker-1] com.example.qscheduler.KPIObserverJob : already scheduled jobName observerJob
</code></pre></div><p>Initially I set the <code>isRunning</code> attribute of those KPI jobs to 0 and my scheduler service is not scheduling them. My KPIObserverJob triggers every 10 seconds because I set it to trigger that way in <code>KPIJobWatcher</code>.</p>
<p>Now let’s see if I update <code>critical_ticket_count</code> KPI’s <code>isRunning</code> value to 1:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">2022-04-11 13:52:32.347 INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: critical_ticket_count , enabled: 1
2022-04-11 13:52:32.348 INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:52:32.348 INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob : already scheduled jobName observerJob
2022-04-11 13:52:32.349 INFO 13033 --- [eduler_Worker-7] com.example.qscheduler.KPIObserverJob : scheduling job: kpiJobName_critical_ticket_count
</code></pre></div><p>As you can see from the logs <code>ObserverJob</code> noticed the enabled job and scheduled it.</p>
<p>Let’s change the <code>cron</code> scheduling rule of <code>critical_ticket_count</code> job to <code>0 0 3 ? * * *</code> and see the logs again:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">2022-04-11 13:55:12.354 INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: critical_ticket_count , enabled: 1
2022-04-11 13:55:12.355 INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob : Found KPI in kpi.json: failed_customer_api_call , enabled: 0
2022-04-11 13:55:12.356 INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob : already scheduled jobName observerJob
2022-04-11 13:55:12.356 INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob : already scheduled jobName kpiJobName_critical_ticket_count
2022-04-11 13:55:12.356 INFO 13033 --- [eduler_Worker-3] com.example.qscheduler.KPIObserverJob : rescheduling job: kpiJobName critical_ticket_count , new cron: 0 0 3 ? * * *
</code></pre></div><p>The Observer job now rescheduled the job since we changed its cron rule. These are all how we make observer job manage the KPI business jobs. If you have more attributes and if you want your observer job to reschedule or perform different operations on business job definition updates you should enrich your <code>ObserverJob</code>.</p>
<h3 id="extend-by-creating-a-management-ui">Extend by Creating a Management UI</h3>
<p>Managing the scheduler jobs using a UI is not in the scope of this post. But that is not much different than managing any data on a web application. I encourage you to do your own implementations if this solution sounds useful to you.</p>
<h3 id="conclusion">Conclusion</h3>
<p>This solution helps you create your own scheduling job management solution on the fly and lets you create, update, or delete the Quartz Scheduler jobs dynamically.</p>
<p>The complete implementation can be found in the <a href="https://github.com/ashemez/QScheduler">GitHub project</a>.</p>
Extending Your Jetty Distribution’s Capabilitieshttps://www.endpointdev.com/blog/2022/03/extending-jetty-distribution-capabilities/2022-03-31T00:00:00+00:00Kürşat Kutlu Aydemir
<p><img src="/blog/2022/03/extending-jetty-distribution-capabilities/jetty-logo.svg" alt="Jetty Logo"></p>
<h3 id="what-is-jetty">What is Jetty?</h3>
<p>“Jetty is a lightweight highly scalable Java-based web server and servlet engine.” (<a href="https://github.com/eclipse/jetty.project">Jetty Project</a>)</p>
<p>Jetty can run standalone or embedded in a Java application and the details about running a Jetty webserver can be found in the Jetty Project Git repository and <a href="https://www.eclipse.org/jetty/documentation">documentation</a> as well. The Jetty project has been hosted at the Eclipse Foundation since 2009 (<a href="https://www.eclipse.org/jetty/">Jetty, Eclipse</a>).</p>
<h3 id="know-your-jetty">Know Your Jetty</h3>
<p>In many legacy environments using the Jetty web server there may be an older version of Jetty. If you know the version of the Jetty distribution in your environment then you can find its source code in the Jetty project GitHub repo. Some of the distributions are in project releases but most of the distributions can be found in the tags as well.</p>
<p>For instance <code>jetty-9.4.15.v20190215</code> distribution can be found in the Jetty project tags at this URL: <code>https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.15.v20190215</code></p>
<p>When you clone the <code>jetty.project</code> Git repo, you can then easily switch to any specific release tag:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ git clone git@github.com:eclipse/jetty.project.git
$ git checkout jetty-9.4.15.v20190215
</code></pre></div><p>Then you can build or add your custom code in that version.</p>
<h3 id="extending-your-jetty-capabilities">Extending Your Jetty Capabilities</h3>
<p>The reason you might want to build Jetty yourself is that you have a specific Jetty version in your environment and want to add some custom handlers or wrappers so that you can add additional capabilities in your environment.</p>
<p>Jetty is written in Java and you can add new features or patch your own fork like other open-source Java projects.</p>
<h3 id="build">Build</h3>
<p>Once you have your target version code base you can just work on that individually. This is one way to add new features to your Jetty distribution.</p>
<p>After you add your custom code you’ll need to build. You can find the building instructions on Jetty Project GitHub home, which is simply:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ mvn clean install
</code></pre></div><p>If you want to skip the tests the option below is your friend:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ mvn clean install -DskipTests
</code></pre></div><h3 id="compile-classes-individually">Compile Classes Individually</h3>
<p>This is a tricky way to inject your newly created custom classes into your Jetty distribution. In this way, instead of building the whole Jetty project, you can just create individual custom Java classes consuming Jetty libraries and compile them manually. You don’t need the whole project this way.</p>
<p>If we come back to the question: what new features would I want to add to my new or ancient local Jetty distribution? Well, that really depends on the issues you face or improvements you need to add.</p>
<p>For one of our customers, once we needed to log request and response headers in Jetty. We couldn’t find an existing way to do that. So I decided to create a custom RequestLog handler class and inject this into the Jetty deployment we already have rather than building the whole project.</p>
<p>Even if you don’t build the whole project it is still useful and handy to get the whole project code to refer the existing code and prepare your code by learning the existing way things are done in the project.</p>
<p>I found <code>RequestLog</code> interface in <code>jetty-server</code> sub-project and it is created under <code>org.eclipse.jetty.server</code> package. There is also a class <code>RequestLogCollection</code> in the same level implementing <code>RequestLog</code> which may give you some idea about the implementations.</p>
<p>So I followed the structure and created my custom handler in the same level and implemented <code>RequestLog</code>. Below is a part of my <code>CustomRequestLog</code> class:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">package</span> <span style="color:#b06;font-weight:bold">org.eclipse.jetty.server</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">com.google.gson.Gson</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">com.google.gson.GsonBuilder</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">org.eclipse.jetty.http.pathmap.PathMappings</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">org.eclipse.jetty.util.component.ContainerLifeCycle</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">org.eclipse.jetty.util.log.Log</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">org.eclipse.jetty.util.log.Logger</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.io.IOException</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.text.DateFormat</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.text.SimpleDateFormat</span>;
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.util.*</span>;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">CustomRequestLog</span> <span style="color:#080;font-weight:bold">extends</span> ContainerLifeCycle <span style="color:#080;font-weight:bold">implements</span> RequestLog
{
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#080;font-weight:bold">final</span> Logger LOG = Log.<span style="color:#369">getLogger</span>(CustomRequestLog.<span style="color:#369">class</span>);
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">static</span> ThreadLocal<StringBuilder> _buffers = ThreadLocal.<span style="color:#369">withInitial</span>(() -> <span style="color:#080;font-weight:bold">new</span> StringBuilder(256));
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">final</span> Writer _requestLogWriter;
<span style="color:#080;font-weight:bold">private</span> String[] _ignorePaths;
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">transient</span> PathMappings<String> _ignorePathMap;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#06b;font-weight:bold">CustomRequestLog</span>(Writer requestLogWriter)
{
<span style="color:#080;font-weight:bold">this</span>.<span style="color:#369">_requestLogWriter</span> = requestLogWriter;
addBean(_requestLogWriter);
}
<span style="color:#888">/**
</span><span style="color:#888"> * Is logging enabled
</span><span style="color:#888"> *
</span><span style="color:#888"> * @return true if logging is enabled
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#888;font-weight:bold">boolean</span> <span style="color:#06b;font-weight:bold">isEnabled</span>()
{
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">true</span>;
}
<span style="color:#888">/**
</span><span style="color:#888"> * Write requestEntry out. (to disk or slf4j log)
</span><span style="color:#888"> *
</span><span style="color:#888"> * @param requestEntry the request entry
</span><span style="color:#888"> * @throws IOException if unable to write the entry
</span><span style="color:#888"> */</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">write</span>(String requestEntry) <span style="color:#080;font-weight:bold">throws</span> IOException
{
_requestLogWriter.<span style="color:#369">write</span>(requestEntry);
}
<span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">append</span>(StringBuilder buf, String s)
{
<span style="color:#080;font-weight:bold">if</span> (s == <span style="color:#080;font-weight:bold">null</span> || s.<span style="color:#369">length</span>() == 0)
buf.<span style="color:#369">append</span>(<span style="color:#d20;background-color:#fff0f0">'-'</span>);
<span style="color:#080;font-weight:bold">else</span>
buf.<span style="color:#369">append</span>(s);
}
<span style="color:#888">/**
</span><span style="color:#888"> * Writes the request and response information to the output stream.
</span><span style="color:#888"> *
</span><span style="color:#888"> * @see RequestLog#log(Request, Response)
</span><span style="color:#888"> */</span>
<span style="color:#555">@Override</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">log</span>(Request request, Response response)
{
<span style="color:#080;font-weight:bold">try</span>
{
<span style="color:#080;font-weight:bold">if</span> (_ignorePathMap != <span style="color:#080;font-weight:bold">null</span> && _ignorePathMap.<span style="color:#369">getMatch</span>(request.<span style="color:#369">getRequestURI</span>()) != <span style="color:#080;font-weight:bold">null</span>)
<span style="color:#080;font-weight:bold">return</span>;
<span style="color:#080;font-weight:bold">if</span> (!isEnabled())
<span style="color:#080;font-weight:bold">return</span>;
StringBuilder buf = _buffers.<span style="color:#369">get</span>();
buf.<span style="color:#369">setLength</span>(0);
Gson gsonObj = <span style="color:#080;font-weight:bold">new</span> GsonBuilder().<span style="color:#369">disableHtmlEscaping</span>().<span style="color:#369">create</span>();
Map<String, Object> reqLogMap = <span style="color:#080;font-weight:bold">new</span> HashMap<String, Object>();
Map<String, String> reqHeaderMap = <span style="color:#080;font-weight:bold">new</span> HashMap<String, String>();
<span style="color:#888">// epoch timestamp
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"timestamp_epoch"</span>, System.<span style="color:#369">currentTimeMillis</span>());
<span style="color:#888">// timestamp
</span><span style="color:#888"></span> DateFormat df = <span style="color:#080;font-weight:bold">new</span> SimpleDateFormat(<span style="color:#d20;background-color:#fff0f0">"yyyy-MM-dd'T'HH:mm:ss.SSSXXX"</span>);
String nowAsString = df.<span style="color:#369">format</span>(<span style="color:#080;font-weight:bold">new</span> Date());
reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"timestamp"</span>, nowAsString);
<span style="color:#888">// request headers
</span><span style="color:#888"></span> List<String> reqHeaderList = Collections.<span style="color:#369">list</span>(request.<span style="color:#369">getHeaderNames</span>());
<span style="color:#080;font-weight:bold">for</span>(String headerName : reqHeaderList) {
reqHeaderMap.<span style="color:#369">put</span>(headerName.<span style="color:#369">toLowerCase</span>(), request.<span style="color:#369">getHeader</span>(headerName));
}
reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"request_headers"</span>, reqHeaderMap);
<span style="color:#888">// response headers
</span><span style="color:#888"></span> Map<String, String> resHeaderMap = <span style="color:#080;font-weight:bold">new</span> HashMap<String, String>();
<span style="color:#080;font-weight:bold">for</span>(String headerName : response.<span style="color:#369">getHeaderNames</span>()) {
resHeaderMap.<span style="color:#369">put</span>(headerName.<span style="color:#369">toLowerCase</span>(), response.<span style="color:#369">getHeader</span>(headerName));
}
reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"response_headers"</span>, resHeaderMap);
<span style="color:#888">// http method
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"http_method"</span>, request.<span style="color:#369">getMethod</span>());
<span style="color:#888">// original URI
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"original_uri"</span>, request.<span style="color:#369">getOriginalURI</span>());
<span style="color:#888">// protocol
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"protocol"</span>, request.<span style="color:#369">getProtocol</span>());
<span style="color:#888">// http status
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"http_status"</span>, response.<span style="color:#369">getStatus</span>());
<span style="color:#888">// query string
</span><span style="color:#888"></span> reqLogMap.<span style="color:#369">put</span>(<span style="color:#d20;background-color:#fff0f0">"query_string"</span>, request.<span style="color:#369">getQueryString</span>());
String reqJSONStr = gsonObj.<span style="color:#369">toJson</span>(reqLogMap);
buf.<span style="color:#369">append</span>(reqJSONStr);
String log = buf.<span style="color:#369">toString</span>();
write(log);
}
<span style="color:#080;font-weight:bold">catch</span> (IOException e)
{
LOG.<span style="color:#369">warn</span>(e);
}
}
}
</code></pre></div><p>In this custom RequestLog class the most interesting part is <code>public void log(Request request, Response response)</code> method where the logging operation is actually done. You can simply override the existing logging behaviour and put anything you want. Here I added the raw request and response headers coming and going through Jetty server.</p>
<p>Now it is time to compile this class. You can find many tutorials about compiling a single Java class using classpath. Here’s how I did it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ javac -cp <span style="color:#d20;background-color:#fff0f0">".:</span><span style="color:#369">$JETTY_HOME</span><span style="color:#d20;background-color:#fff0f0">/lib/jetty-server-9.4.15.v20190215.jar:</span><span style="color:#369">$JETTY_HOME</span><span style="color:#d20;background-color:#fff0f0">/lib/jetty-http-9.4.15.v20190215.jar:</span><span style="color:#369">$JETTY_HOME</span><span style="color:#d20;background-color:#fff0f0">/lib/jetty-util-9.4.15.v20190215.jar:</span><span style="color:#369">$JETTY_HOME</span><span style="color:#d20;background-color:#fff0f0">/lib/servlet-api-3.1.jar:</span><span style="color:#369">$JETTY_HOME</span><span style="color:#d20;background-color:#fff0f0">/lib/gson-2.8.2.jar"</span> CustomRequestLog.java
</code></pre></div><p>If you look at my classpath I even added a third party library <code>gson-2.8.2.jar</code> since I also used this in my custom code. Remember to put this in your <code>$JETTY_HOME</code> directory as well.</p>
<p>The command above generates the <code>CustomRequestLog.class</code> file which is now available to be injected. So where do you need to inject this?</p>
<p>Since I followed where the <code>RequestLog</code> interface is located and packaged we better inject this into the same project JAR file, which is <code>jetty-server.jar</code>. In my environment it is <code>jetty-server-9.4.15.v20190215.jar</code>. I also added other required dependencies in the classpath to compile this code.</p>
<p>Now, I want to inject <code>CustomRequestLog.class</code> into <code>jetty-server-9.4.15.v20190215.jar</code>. I copied this jar into a temporary directory and I extracted the content of <code>jetty-server-9.4.15.v20190215.jar</code> into the temp directory using this command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ jar xf jetty-server-9.4.15.v20190215.jar
</code></pre></div><p>This command extracts all the content of the jar file including resource files and the classes in their corresponding directory structure <code>org/eclipse/jetty/server</code>. You would see <code>RequestLog.class</code> also extracted in this directory.</p>
<p>So what we need to do is now simply copy our <code>CustomRequestLog.class</code> into this extracted <code>org/eclipse/jetty/server</code> directory and pack up the JAR file again by running this command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">$ jar cvf jetty-server-9.4.15.v20190215.jar org/ META-INF/
</code></pre></div><p>This command re-bundles compiled code along with the other extracted resources (in this case the <code>META-INF/</code> directory only) and creates our injected JAR file. You’d better create this injected Jetty JAR file in the temp directory so that you can control the backup of existing original JAR files.</p>
<p>For this specific case I added this custom <code>RequestLog</code> handler in my Jetty config file <code>jetty.xml</code>. It may not be the case for all the custom changes or extensions you’d add to your Jetty instance.</p>
<p>Here is an example <code>RequestLog</code> config entry for this custom handler:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#b06;font-weight:bold"><Set</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"RequestLog"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><New</span> <span style="color:#369">id=</span><span style="color:#d20;background-color:#fff0f0">"RequestLog"</span> <span style="color:#369">class=</span><span style="color:#d20;background-color:#fff0f0">"org.eclipse.jetty.server.CustomRequestLog"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#888"><!-- Writer --></span>
<span style="color:#b06;font-weight:bold"><Arg></span>
<span style="color:#b06;font-weight:bold"><New</span> <span style="color:#369">class=</span><span style="color:#d20;background-color:#fff0f0">"org.eclipse.jetty.server.AsyncRequestLogWriter"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><Arg></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.base"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"."</span> <span style="color:#b06;font-weight:bold">/></span>/
<span style="color:#b06;font-weight:bold"><Property></span>
<span style="color:#b06;font-weight:bold"><Name></span>jetty.requestlog.filePath<span style="color:#b06;font-weight:bold"></Name></span>
<span style="color:#b06;font-weight:bold"><Default></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.requestlog.dir"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"logs"</span><span style="color:#b06;font-weight:bold">/></span>/yyyy_mm_dd.request.log
<span style="color:#b06;font-weight:bold"></Default></span>
<span style="color:#b06;font-weight:bold"></Property></span>
<span style="color:#b06;font-weight:bold"></Arg></span>
<span style="color:#b06;font-weight:bold"><Arg/></span>
<span style="color:#b06;font-weight:bold"><Set</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"filenameDateFormat"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.requestlog.filenameDateFormat"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"yyyy_MM_dd"</span><span style="color:#b06;font-weight:bold">/></span>
<span style="color:#b06;font-weight:bold"></Set></span>
<span style="color:#b06;font-weight:bold"><Set</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"retainDays"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.requestlog.retainDays"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"90"</span><span style="color:#b06;font-weight:bold">/></span>
<span style="color:#b06;font-weight:bold"></Set></span>
<span style="color:#b06;font-weight:bold"><Set</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"append"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.requestlog.append"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"false"</span><span style="color:#b06;font-weight:bold">/></span>
<span style="color:#b06;font-weight:bold"></Set></span>
<span style="color:#b06;font-weight:bold"><Set</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"timeZone"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><Property</span> <span style="color:#369">name=</span><span style="color:#d20;background-color:#fff0f0">"jetty.requestlog.timezone"</span> <span style="color:#369">default=</span><span style="color:#d20;background-color:#fff0f0">"GMT"</span><span style="color:#b06;font-weight:bold">/></span>
<span style="color:#b06;font-weight:bold"></Set></span>
<span style="color:#b06;font-weight:bold"></New></span>
<span style="color:#b06;font-weight:bold"></Arg></span>
<span style="color:#b06;font-weight:bold"></New></span>
<span style="color:#b06;font-weight:bold"></Set></span>
</code></pre></div><p>That’s all.</p>
Code Reviewshttps://www.endpointdev.com/blog/2022/03/code-reviews/2022-03-21T00:00:00+00:00Kevin Campusano
<p><img src="/blog/2022/03/code-reviews/20220219-194926-sm.webp" alt="Winter scene with pine trees behind snow on tall grasses around a winding stream crossed by a primitive bridge of 8 logs, below blue sky with white clouds"></p>
<!-- Photo by Jon Jensen -->
<p>Last week, a few End Point team members and I came together to prepare a presentation on code reviews for the whole company. We went through the basics of “what”, “why”, and “how”.</p>
<p>We also, and perhaps most interestingly, made several recommendations that we’ve discovered after years of doing code reviews in a variety of teams and project sizes. A series of “lessons learned” so to speak.</p>
<p>I thought it’d be useful to capture that discussion in written form. Let’s start with the basics.</p>
<h3 id="what-is-a-code-review">What is a code review?</h3>
<p><a href="https://en.wikipedia.org/wiki/code_review">Wikipedia’s article on code reviews</a> says that a code review is…</p>
<blockquote>
<p>A <strong>software quality assurance</strong> activity in which <strong>one or several people</strong> check a program mainly by viewing and <strong>reading parts of its source code</strong>, and they do so <strong>after implementation</strong> or as an interruption of implementation.</p>
</blockquote>
<p>That is a precise but frankly wordy way to say “having somebody look at the code you’ve written”. This definition, however, touches on a few aspects that give us good insight into what code reviews are and what their purpose is.</p>
<p>First up, it tells us that code reviews are a software quality assurance activity. That is, their main goal is to make sure that the code that’s being produced is of good quality.</p>
<p>Second, it tells us that they are carried out by one or several people, revealing that code reviews are a team exercise. It’s the opposite of coding in isolation. Coding becomes a communal task, with input from other team members.</p>
<p>It also tells us, maybe unsurprisingly, that the main focus of the review is the code itself. As the main deliverable artifact of the software development process, we should strive to make it as good as possible.</p>
<p>Finally, it tells us when code reviews should happen: when implementation is done, or there’s a logical interruption of it. Meaning, once a feature is done, a user story is complete, a bug fixed. That is, when there’s a cohesive chunk of code that has been written.</p>
<h3 id="why-should-we-do-code-reviews">Why should we do code reviews?</h3>
<p>So why are code reviews important? They provide many benefits.</p>
<p>First and foremost, code reviews can help improve the code’s internal quality. Productive discussion around an implementation can help improve maintainability and readability of the code when reviewers, with a fresh set of eyes, spot the potential for such improvements where the original author may have missed them.</p>
<p>Also, and just as important, external quality of the code can be improved. Reviewers can help find bugs or other types of defects like security and performance issues.</p>
<p>Code reviews can also serve as a knowledge sharing tool. Code written by team members of more seniority or who are more knowledgeable about the code area, business domain, or a specific tool or library, can be exemplary for other team members. They can learn from the code when conducting reviews. This has the added perk that it reduces the situations where a single person holds all the knowledge of a given system component or code area. Likewise, code review feedback provided by such an expert can have the same effect.</p>
<p>Another great benefit they can bring to the table is the distribution of code ownership among all of the team members, not only the author of the code in question. When projects have strong code review habits, the code becomes something that the whole team is producing, as every line that gets to production has been seen by, and incorporated input from, many of the members of the team. Everybody owns and can feel proud of the final product.</p>
<p>Finally, reviewers can sometimes just come up with better and/or simpler solutions than the ones the original author implemented. They can come up with these given their fresh perspective and maybe their experience with similar problems in other domains. Code reviews allow for these to be incorporated before the time comes to ship the code.</p>
<h3 id="what-should-reviewers-look-for">What should reviewers look for?</h3>
<p>Simply put, reviewers should look at every aspect of the code and offer suggestions for improvements where they see fit. <a href="https://google.github.io/eng-practices/review/#look_for">Google’s recommended practices</a> compile a somewhat comprehensive list of the elements that reviewers should look for:</p>
<ul>
<li><strong>Design:</strong> Is the code well-designed and appropriate for the system?</li>
<li><strong>Functionality:</strong> Does the code implement the requirements correctly?</li>
<li><strong>Complexity:</strong> Could the code be made simpler? Is it understandable?</li>
<li><strong>Tests:</strong> Does the code have correct and well-designed automated tests?</li>
<li><strong>Naming:</strong> Are clear names for variables, classes, methods, etc. being used?</li>
<li><strong>Comments:</strong> Are the comments clear, useful and necessary?</li>
<li><strong>Style:</strong> Does the code follow the project’s style guide?</li>
<li><strong>Documentation:</strong> Did the relevant documentation also get updated? Any public interface documentation or OpenAPI files for example.</li>
</ul>
<p>When it comes to code style, something to note is that this is where tools like linters are prettiers can be implemented to reduce human labor. Early in the project, if the team decides on the style, such a tool can be setup to automate the process of making sure that all code that gets written complies with the style guide. Some code repositories even allow for such tools to be automatically run upon every push. This makes style guide compliance not even a concern for the reviewers, because the tooling always makes sure that the code does comply.</p>
<h3 id="who-should-review-codebrwho-should-ask-for-their-code-to-be-reviewed">Who should review code?<br>Who should ask for their code to be reviewed?</h3>
<p>Everybody in the team should be regularly reviewing code and having their code reviewed. Regardless of seniority or experience in the specific project area, domain, framework, or language. More “junior” team members benefit from reviewing code by learning new techniques, principles, technologies, and the code base itself. More “senior” team members can provide valuable input that improves the code base and other team members' skills.</p>
<p>We also have to realize that the distinction between “junior” and “senior” is often blurry. Most teams have people with a variety of skill sets and experience; so everybody has the ability to offer good insight. One can always be a “senior” in one aspect of the project, and a “junior” in another.</p>
<p>Even if a single reviewer can be good enough, it is beneficial to include as many reviewers as possible, lest we fall into the trap of overloading a small number of individuals by having them be in charge of most or all of the reviews. Also, like I mentioned before, two great benefits that code reviews offer are knowledge sharing and code ownership. The more people you have regularly reviewing code, the bigger they impact will they have in these two aspects.</p>
<p>That said, it is always better to have somebody more experienced in the area of the code that’s changing be among the reviewers.</p>
<h3 id="when-should-code-be-reviewed">When should code be reviewed?</h3>
<p>As Wikipedia’s definition revealed, it is ideal for code reviews to be done when implementation is done on a feature; before merging the new code into the main development branch. Most modern software development that uses a Git-based code repository uses something like GitHub’s <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests">Pull Request</a> mechanism. (GitLab uses Merge Request instead of Pull Request: same concept, different terminology.)</p>
<p>The developer creates a Pull Request when they are done implementing a feature or fixing a bug. The PR compiles all the changes (a series of commits) and turns them into a nice digestible package. This adds great visibility to the changes and makes them very easy to review before merging. Ideally, no patch makes it into the system without first being reviewed.</p>
<p>Code reviews can also happen before the implementation is done. Maybe the developer wants to get the team’s input on a specific function, method, class, component or approach. If the developer asks for it specifically, it can happen at any point in their development process.</p>
<h3 id="some-recommendations">Some recommendations</h3>
<p>Now that we’ve gone over the basics, let’s discuss some recommendations and pitfalls to avoid, inspired by our experience in conducting code reviews over the years.</p>
<h4 id="keep-pull-requests-to-a-manageable-size">Keep pull requests to a manageable size</h4>
<p>Bigger patches make code reviews harder to perform because of the sheer volume of code that the reviewers need to work through. More code to read makes it tempting to skim over it rather than reading in depth, and makes it hard to gain a thorough understanding of the changes and offer good insight.</p>
<p>So we feel like it is best to keep them as small as possible. This desire to keep things small may need to affect the overall software development process upstream. For example, making sure user stories or change requests are granular enough so that they can be fulfilled with a reasonable amount of code changes. Consider splitting bigger features into smaller, bite-sized issues to make this possible.</p>
<h4 id="make-the-pull-request-cohesive">Make the pull request cohesive</h4>
<p>PRs are better when their size is manageable, but also we need to make sure that they contain all they need to allow the reviewers to understand them completely and as a whole. There’s no need to split the changes artificially (for example, between back-end and front-end) if ultimately, they need to be pushed together to fulfill the requirement that’s being worked on, and leave the system in a working state.</p>
<h4 id="if-you-would-like-a-preliminary-review-ask-specific-questions">If you would like a preliminary review, ask specific questions</h4>
<p>Sometimes we want to get early reviews even before the implementation is complete or we’re at a logical interruption point. For such cases, a good practice is to come to the reviewers with a specific question that we’d like them to focus on.</p>
<p>It can waste time if the intent of the review is not explicitly communicated: The reviewer could mistakenly do a thorough review and leave feedback on tiny details of code that’s not yet ready and not even address the specific larger aspects that the developer wants help with.</p>
<h4 id="pair-programming-has-the-first-code-review-baked-in">Pair programming has the first code review baked in</h4>
<p><a href="https://en.wikipedia.org/wiki/Pair_programming">Pair programming</a> can expedite the review process by closing the code review feedback loop, compressing it to its fastest form. In pair programming, code effectively gets reviewed as soon as it is written, bit by bit.</p>
<p>If the team has more than two developers, though, there’s still great value in having the other team members, the ones not involved in the pair programming activity, review the code. They will approach it later with fresh eyes and without the shared mental context the pair had.</p>
<h4 id="code-reviews-work-well-asynchronously">Code reviews work well asynchronously</h4>
<p>In general, don’t make code reviews a synchronous process. Publicly available Git repository cloud hosting services like GitHub and GitLab include great tools for reviewing pull/merge requests. We should use them to their full potential. There’s no need for a conference call or an in person meeting where everybody blocks a chunk of time to dedicate it to reviewing a PR. Everybody can do it on their own at their convenience.</p>
<p>But if a member of the team is new to the process of reviewing code, it can be good to work through a few pull requests at the same time, together, till they get the hang of it.</p>
<h4 id="give-code-reviews-high-priority">Give code reviews high priority</h4>
<p>Give code reviews a high priority within your daily tasks. It is counterproductive to let pull requests sit for a long time when a few minutes to an hour of our time can mean that a user story/ticket can move forward through the process. If you work by organizing your increments via sprints, remember that the goal is to complete the most stories as a team. Reviewing a pull request is actively supporting that goal, even if it isn’t one of the stories/issues you’re working on yourself.</p>
<h4 id="make-sure-your-prs-get-the-attention-they-need">Make sure your PRs get the attention they need</h4>
<p>Don’t just “fire and forget” a pull request. If it happens that any of our patches are taking too long to be seen by other people, we don’t just abandon them and think that they are somebody else’s problem now, that our work is done. In these cases we should feel free to reach out to the reviewers and bring the PRs to their attention.</p>
<p>To this end, we should leverage all the communication tools available, even outside of the code repository or code review tool, by chat, phone, issue tracking system, etc.</p>
<h4 id="get-as-many-reviewers-as-you-can">Get as many reviewers as you can</h4>
<p>A single reviewer on a pull request can be enough. However, it is always beneficial to try to get as many eyes as possible onto a change. It improves ownership and it allows for more effective knowledge sharing. It also has the potential for more improvements on the code base, as more people, with varying strengths, look at the code and offer their feedback.</p>
<p>Also try to avoid having a single person be the gatekeeper of merges. Like I said, everybody can and should participate in the activity of code reviews. Sometimes having a gatekeeper this may be desirable if, for example, CI/CD is in place in such a way that merges produce automatic production deployments. But we can always try to make sure that code that reaches the gatekeeper is already reviewed by other team members by the time it does. That way we avoid overloading them and eliminate the bottleneck or single point of failure.</p>
<p>GitHub, GitLab, and similar tools provide settings to limit which users can commit to certain branches. If a process like that is needed, it can be done with help of such tools while still having PRs that many team members can review and discuss.</p>
<h4 id="dont-let-the-perfect-be-the-enemy-of-the-good">Don’t let the perfect be the enemy of the good</h4>
<p>If the PR isn’t “perfect” or not “as good as it could be”, but it does not worsen the code base, and implements the changes competently, maybe there’s no need to block it. Code reviews are important, but it also is important to try to maintain momentum of delivery and avoid making developers feel nitpicked. So consider that when reviewing. There is a balancing act between deadlines, paying off and incurring technical debt, and what is “good enough”.</p>
<p>In the same train of thought, it is useful to clearly label code review comments. Specifying whether the reviewer considers each comment as a question, a simple nitpick, or an actually important change.</p>
<h4 id="top-down-imposition-of-process-is-often-inconvenient">Top-down imposition of process is often inconvenient</h4>
<p>Code reviews don’t need to include a ton of software process overhead. Regardless of your process style, it can be a very lightweight practice and done at the developers’ discretion. It can be as simple as sending a diff file to a fellow developer and asking them for feedback. The tools available today make them very accessible and easy to do.</p>
<p>As such, they can become very effective when the team itself manages and conducts them; as opposed to having it come to them as a predetermined specific process from management. Even a practice as beneficial as code reviews can be soured by a bad, overly strict, or dogmatic implementation.</p>
<h3 id="thats-all-for-now">That’s all for now</h3>
<p>At the end of the day, the concept of code reviews is simple: To have somebody else look at our code, in hopes that what eventually makes it to production is as good quality as possible.</p>
<p>In this article we’ve discussed many more aspects to clearly explain the promise of code reviews. We’ve given some details on how to conduct them, who is involved, and what benefits we can get from them. We also captured a series of recommendations that we have learned through experience.</p>
<p>If you’re not into the habit yet, hopefully this article convinced you to give it a try!</p>
Automating reading the screen and interacting with GUI programs on X Window Systemhttps://www.endpointdev.com/blog/2022/03/automate-read-screen-interact-gui-x11/2022-03-10T00:00:00+00:00Constante “Tino” Gonzalez
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/20220216_224654.webp" alt="Metal tower with cables in front of overcast sky and muted sun"></p>
<!-- Photo by Jon Jensen -->
<p>A while back, Google Earth made some changes to the layer select menu in the sidebar, which broke a program that toggles the 3D imagery on VisionPort systems. These run the X Window System (also known as X11, or just X) on Ubuntu Linux.</p>
<p>In looking for a workaround, I found that shell scripts can fully interact with GUI apps using the <code>xdotool</code>, <code>xwd</code>, and <code>convert</code> commands. This script would send a series of keystrokes to open the sidebar, navigate the layers menu, and toggle the box for the 3D buildings layer.</p>
<p>Changing the series of keystrokes to match the new number of layers should have fixed the issue, but there was more to this script. The next part of the script would take a screenshot, crop the checkbox, and compare it to saved files of other cropped boxes. Fixing this part of the script required correcting the positions of the captures and replacing the reference files with ones that pictured the updated Google Earth checkbox states.</p>
<p>Here I will explain how the script works and how we changed it so that it no longer needs these reference files and ultimately runs faster.</p>
<h3 id="overview-of-how-the-script-works">Overview of how the script works</h3>
<p><code>xwd</code> takes a screenshot of a window on the screen.
<code>convert</code> transforms the pixel data in the image into lines of text with location and color that we can easily search and read with <code>grep</code>, <code>sed</code>, or similar.
<code>xdotool</code> interacts with GUI windows. It can find, focus, and send keystrokes and mouse commands, among other things.</p>
<h3 id="preparing">Preparing</h3>
<p>To illustrate, let’s make a simple case of looking for a button and clicking on it from the terminal. We will skip making a script for this example as most of this can be accomplished on a single command line with <code>xdotool</code>. The commands here are for the Ubuntu operating system and may be a little different on other systems.</p>
<p>If you would like to try it as you read along, you will need an image editor to quickly look up pixel positions and colors. GIMP works great and is pictured in this example.</p>
<p>You will also need to install some packages. On a terminal:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">sudo apt install xdotool x11-apps imagemagick
</code></pre></div><h3 id="working-through-an-example">Working through an example</h3>
<p>1. To know which window to interact with, xdotool needs to know the window’s name. So let’s open a browser and navigate to endpointdev.com and then note the page title in the browser tab, “Secure Business Solutions”:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/1-scr-read.webp" alt="Screenshot of endpointdev.com website home page loaded in a browser"></p>
<p>2. On the terminal:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">xdotool search "secure business solutions"
Defaulting to search window name, class, and classname
69206019
</code></pre></div><p>This reminds us that it can search for windows in many ways and returns a window ID, in this example 69206019. Search results also get stored on the “window stack” and can be referenced as <code>%1</code>, <code>%2</code>, and so on.</p>
<p>3. Use this ID in the next terminal command. If you have more than one ID, you may need to refine your search:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">xwd -id <span style="color:#00d;font-weight:bold">69206019</span> -out endpointdev.xwd
</code></pre></div><p><code>xwd</code> creates a screenshot of the window that we want to see and saves it to the filename passed with the <code>-out</code> argument.</p>
<p>4. Then run a <code>convert</code> command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">convert endpointdev.xwd endpointdev.txt
</code></pre></div><p>By converting the image to text we can use any text tools like <code>grep</code>, <code>cut</code>, <code>diff</code>, or <code>sed</code> to find colors and coordinates on the images. We just need to know how to read it. The first few lines of the endpointdev.txt file look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># ImageMagick pixel enumeration: 1156,638,65535,srgb
0,0: (11565,11565,11565) #2D2D2D srgb(45,45,45)
1,0: (10537,10537,10537) #292929 grey16
2,0: (4369,4369,4369) #111111 srgb(17,17,17)
3,0: (0,0,0) #000000 black
4,0: (0,0,0) #000000 black
5,0: (0,0,0) #000000 black
</code></pre></div><p>Each line represents a pixel:</p>
<ul>
<li>The first two numbers are the x and y positions,</li>
<li>in parentheses are the decimal color values per RGB channel,</li>
<li>after <code>#</code> is the HTML RGB color value which we will be using in our example,</li>
<li>and last the <code>srgb</code> code or name for the color.</li>
</ul>
<p>We can use any of these color codes for matching. To decode these color codes visually we use an image editor.</p>
<p>5. Open the <code>endpointdev.xwd</code> file that we created in an image editor. Here we will use GIMP:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/2-scr-read.webp" alt="Screenshot of GIMP image editor open with a screenshot of endpointdev.com website home page loaded in a browser"></p>
<p>6. Select the zoom tool (see the mouse pointer in the screenshot above to know which that is). Use it to draw a box around the VisionPort button or something else that we want our script to “see” and zoom in on it:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/3-scr-read.webp" alt="Screenshot of GIMP image editor zoom tool selected on a screenshot of a tiny part of the endpointdev.com website with a VisionPort link"></p>
<p>7. Next, use the color picker tool (check the mouse pointer in the screenshot above for this tool) and click on the circle, or any other spot you want to pick for your script:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/4-scr-read.webp" alt="Screenshot of GIMP image editor color picker tool selected on a screenshot of a tiny part of the endpointdev.com website with a VisionPort link"></p>
<p>8. Notice by the pointer above how the color picker box changed colors. Now clicking on this box brings up a new window:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/5-scr-read.webp" alt="Screenshot of GIMP image editor “Change Foreground Color” dialog box"></p>
<p>Here we can find the HTML notation of the color we selected: <code>00ffcc</code> (see the pointer in the screenshot above). We’ll call this “VisionPort button green”.</p>
<p>In my experience even things that look like the same color can have a unique pixel caused by a different shade on an edge or something else. Our old script checked a 25×25 pixel image to guarantee what it had was a check mark, but finding one of these colors unique enough to use as a key ensures that the check mark is next to it, so we can check the state on a single pixel too.</p>
<p>If one pixel is not unique enough we can look for a pattern of pixels and still store them as variables on the script to avoid saving and comparing files.</p>
<p>9. With this HTML hex RGB color code we go back to the terminal and search our file for it. Use capital letters for the color or else <code>grep -i</code> to make it case-insensitive:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">grep 00FFCC endpointdev.txt
</code></pre></div><p>This will return a list of pixels with this color, such as:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">827,107: (0,65535,52428) #00FFCC srgb(0,255,204)
839,107: (0,65535,52428) #00FFCC srgb(0,255,204)
853,107: (0,65535,52428) #00FFCC srgb(0,255,204)
878,107: (0,65535,52428) #00FFCC srgb(0,255,204)
883,107: (0,65535,52428) #00FFCC srgb(0,255,204)
</code></pre></div><p>10. For our purpose we can use any of those, so let’s try the first result. In the terminal run:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">xdotool windowfocus <span style="color:#00d;font-weight:bold">69206019</span> mousemove <span style="color:#00d;font-weight:bold">827</span> <span style="color:#00d;font-weight:bold">107</span> click <span style="color:#00d;font-weight:bold">1</span>
</code></pre></div><p>Your browser should then follow the link to visionport.com and load that website’s home page in your browser:</p>
<p><img src="/blog/2022/03/automate-read-screen-interact-gui-x11/6-scr-read.webp" alt="Screenshot of visionport.com website loaded in a web browser"></p>
<p>Did it work for you too? 🎉</p>
<p>That is all we need to build a script that interacts with GUI apps. If you would like to practice, take another screenshot and use xdotool to look for and click the “contact” button. 😁</p>
<h3 id="how-it-works-in-a-little-more-detail">How it works in a little more detail</h3>
<p>The last xdotool command chain’s steps are:</p>
<ul>
<li><code>windowfocus 69206019</code> selects the ID that we searched for before,</li>
<li><code>mousemove</code> moves the mouse pointer to the specified absolute coordinates, in this case the first pixel that matched the VisionPort button green color in the <code>endpointdev.txt</code> file,</li>
<li><code>click 1</code> sends a mouse left click at the mouse’s current position.</li>
</ul>
<p>Updating our script to use this instead of the series of keystrokes took some investigation and testing but made it faster in the end.</p>
<p>One of the issues to overcome was that the VisionPort custom layers capture xdotool’s window and mouse commands, so to target a specific window on the VisionPort, we have to offset the coordinates of our commands according to the position of the window on the screen.</p>
<h3 id="chaining-xdotool-actions">Chaining xdotool actions</h3>
<p>Here is another example of chaining several xdotool actions in a single command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">xdotool search <span style="color:#d20;background-color:#fff0f0">"secure business solutions"</span> windowfocus key Alt+F4
</code></pre></div><p>That should close the browser window on most systems. If any of these commands failed, try replacing <code>windowfocus</code> with <code>windowactivate</code>; some xdotool commands depend on system support. Check xdotool’s manual for more options.</p>
<p>Command chain steps:</p>
<ul>
<li><code>search <name></code>: Finds windows with that name, and stores the results in the “window stack” memory.</li>
<li><code>windowfocus</code>: Selects a window. Defaults to the first window from xdotool’s window stack; no need to supply the window ID if following a search.</li>
<li><code>key Alt+F4</code>: Sends the Alt+F4 keypress to the selected window. The <code>+</code> here means at the same time. Use spaces to separate a list of keys. By default there is a delay between keystrokes, and that is configurable.</li>
</ul>
<p>Use the standard Unix <code>sleep <seconds></code> in between other xdotool commands that may need delays.</p>
<p>xdotool takes a list of commands naturally, which can even be stored and read directly from text files. They can also be executed directly by making the file executable (with <code>chmod +x</code>) and setting the shebang line (first line of the file) to:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">#!/usr/bin/xdotool
</code></pre></div><p>Such executable scripts are commonly named using a <code>.xdo</code> extension.</p>
<h3 id="fine-tuning">Fine-tuning</h3>
<p>While we can convert the whole screenshot to text and search on it like we did in the example above, the files are large, so cropping a smaller section before converting to text is recommended.</p>
<p>We can crop a section, save it to a file and compare it to another file, like our script did. We can also crop an image to 1 pixel wide across a menu bar if that’s what we’re aiming for, or a column of icons in our case, to find all the button positions without much overhead. Doing this instead of only checking the area hard-coded for a check box gave our script the ability to find the check box based on the icon next to it when changing positions due to other tabs expanding on the menu.</p>
<p>It is also possible to use the multiprocessing features of shell scripts: When working on multiple window screenshots or crops, we can have them all run at the same time using the <code>&</code> operator at the end of the commands, then use the <code>wait</code> command to let them all finish before performing the text searches. These are very fast on small cropped files.</p>
<h3 id="learning-more">Learning more</h3>
<p>The <a href="https://man.cx/xdotool">xdotool documentation</a> has information on other ways to identify and interact with windows and is a great place to go to learn more.</p>
Database Design: Using Documentshttps://www.endpointdev.com/blog/2022/03/database-design-with-documents/2022-03-09T00:00:00+00:00Emre Hasegeli
<p><img src="/blog/2022/03/database-design-with-documents/20220126_011303.webp" alt="Angle view of square paving stones in two colors, in a pattern"></p>
<!-- Photo by Jon Jensen -->
<p>Using documents in relational databases is increasingly popular. This
technique can be practical and efficient when used in fitting circumstances.</p>
<h3 id="example">Example</h3>
<p>Let’s start with an example. Imagine we are scraping web sites for external
URLs and store them in a table. We’ll have the web sites table to store
the scrape timestamp and another table to store all of the references.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>web_site_domain<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>last_scraped_at<span style="color:#bbb"> </span>timestamptz,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(web_site_domain)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>refs<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>web_site_domain<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>ref_location<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>link_url<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(web_site_domain,<span style="color:#bbb"> </span>ref_location,<span style="color:#bbb"> </span>link_url),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(web_site_domain)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>We do not need to bother adding an id to the <code>web_sites</code> table, because we
assume there won’t be too many of them. The domain is small and more
practical to use as an identifier. If you are curious about advantages
of using natural keys, see <a href="/blog/2021/03/database-design-using-natural-keys/">my previous article</a>.</p>
<h3 id="normalized-tables">Normalized Tables</h3>
<p>There may be many thousands of unique URLs for a single web site and other
web sites may refer to the same URLs. To try to minimize the storage,
we can keep the locations and the URLs in separate tables, give them integer
identifiers, and have another table for the many-to-many relations.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>locations<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>location_id<span style="color:#bbb"> </span><span style="color:#038">bigint</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>web_site_domain<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>ref_location<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(location_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(web_site_domain,<span style="color:#bbb"> </span>ref_location),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(web_site_domain)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>links<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>link_id<span style="color:#bbb"> </span><span style="color:#038">bigint</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>link_url<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(link_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(link_url)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>locations_links<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>location_id<span style="color:#bbb"> </span><span style="color:#038">bigint</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>link_id<span style="color:#bbb"> </span><span style="color:#038">bigint</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(location_id,<span style="color:#bbb"> </span>link_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(location_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>locations,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(link_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>links<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>The idea in here is to keep our biggest table narrow. It’d pay off to refer
to the <code>locations</code> and <code>links</code> just with integer identifiers when we’ll have
very many of them.</p>
<h3 id="table-sizes">Table Sizes</h3>
<p>We’ll have many web sites and many URLs, so our lookup tables would be big
and the relation table would be even bigger. That is going to be a major
problem because of many reasons, one of them being space efficiency. Narrow
tables with many rows are not very space efficient especially on Postgres
where there’s 24 bytes per-row overhead. To demonstrate it being significant,
let’s add one row to each of the tables to see the sizes compared with
the overhead.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span>(web_site_domain)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'example.com'</span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>refs<span style="color:#bbb"> </span>(web_site_domain,<span style="color:#bbb"> </span>ref_location,<span style="color:#bbb"> </span>link_url)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'example.com'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'/source'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'http://example.net/target.html'</span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>locations<span style="color:#bbb"> </span>(web_site_domain,<span style="color:#bbb"> </span>ref_location)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'example.com'</span>,<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'/source'</span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>links<span style="color:#bbb"> </span>(link_url)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'http://example.net/example.html'</span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>locations_links<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">VALUES</span><span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">1</span>,<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">table_name</span>,<span style="color:#bbb"> </span>tuple_len,<span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">100</span>.<span style="color:#00d;font-weight:bold">0</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">24</span><span style="color:#bbb"> </span>/<span style="color:#bbb"> </span>tuple_len)::<span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span>overhead_perc<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>information_schema.tables,<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LATERAL</span><span style="color:#bbb"> </span>pgstattuple(<span style="color:#080;font-weight:bold">table_name</span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>table_schema<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'public'</span>;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">table_name</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span>tuple_len<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span>overhead_perc<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">-----------------+-----------+---------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">36</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">67</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>refs<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">75</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">32</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>locations<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">52</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">46</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>links<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">64</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">38</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>locations_links<span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">40</span><span style="color:#bbb"> </span>|<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">60</span><span style="color:#bbb">
</span></code></pre></div><p>With just one row and such short domains and URLs, we wouldn’t be gaining
anything by having the links on a lookup table. In fact, it’s a lot less
efficient. Although, the situation may change with many rows.</p>
<h3 id="composite-types">Composite Types</h3>
<p>We can combine multiple fields in a column to avoid the overhead of many
rows in narrow tables. Postgres has good support for composite types and
arrays over them. They are useful if you need strict data type checking.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TYPE</span><span style="color:#bbb"> </span>web_site_page<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>ref_location<span style="color:#bbb"> </span><span style="color:#038">text</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>link_urls<span style="color:#bbb"> </span><span style="color:#038">text</span>[]<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>pages<span style="color:#bbb"> </span>web_site_page[];<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">UPDATE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SET</span><span style="color:#bbb"> </span>pages<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#038">ARRAY</span>[<span style="color:#080;font-weight:bold">ROW</span>(ref_location,<span style="color:#bbb"> </span><span style="color:#038">ARRAY</span>[link_url])::web_site_page]<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>refs;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>pg_column_size(pages)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>web_sites;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>pg_column_size<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">----------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">117</span><span style="color:#bbb">
</span></code></pre></div><p>As you see, the new column is not exactly small because composite types
and arrays come with their own overheads. Still it could be much better
when a web site has many references.</p>
<h3 id="json-document-column">JSON Document Column</h3>
<p>The composite types are not very easy to work with. We can store the pages
in another format. JSON is the most popular one nowadays. There are multiple
options to store JSON in a column in Postgres. Let’s start with
the text-based data type.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>pages_json<span style="color:#bbb"> </span>json;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">UPDATE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SET</span><span style="color:#bbb"> </span>pages_json<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span>to_json(pages);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>pg_column_size(pages_json)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>web_sites;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>pg_column_size<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">----------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">76</span><span style="color:#bbb">
</span></code></pre></div><p>What is perhaps also surprising is the JSON being smaller than the array over
composite type. This is because Postgres data types are a bit wasteful because
of alignment and storing oids of the enclosed data types repeatedly.</p>
<h3 id="binary-json">Binary JSON</h3>
<p>Another JSON data type, <code>jsonb</code> is available in Postgres. It is a binary
structure that allows faster access to the values inside the document.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>pages_jsonb<span style="color:#bbb"> </span>jsonb;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">UPDATE</span><span style="color:#bbb"> </span>web_sites<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SET</span><span style="color:#bbb"> </span>pages_jsonb<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span>to_jsonb(pages);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>pg_column_size(pages_jsonb)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>web_sites;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>pg_column_size<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">----------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">98</span><span style="color:#bbb">
</span></code></pre></div><p>As you see the text-based JSON is still smaller. This is due to the binary
offsets stored in the binary structure for faster access to the fields.
The text-based JSON also compresses better.</p>
<h3 id="size-comparison">Size Comparison</h3>
<p>Very simple single row data shows the text-based <code>json</code> as the smallest,
binary <code>jsonb</code> a bit larger, and the composite type to be the largest.
However, the differences would be a lot smaller with more realistic sample data
with many items inside the documents. I generated some data and gathered
these results (the sizes excluding the indexes).</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"> model | size
------------------|---------
single table | 436 MiB
normalized tables | 387 MiB
composite type | 324 MiB
json | 318 MiB
jsonb | 320 MiB
</code></pre></div><p>You may expect the normalized tables to be the smallest as otherwise
the metadata is stored repeatedly in the columns. Though, the results reveal
it’s the other way around. This is because of the per-row overhead in
Postgres. It becomes noticeable in narrow tables as in this example.</p>
<p>It’s also useful to notice that the most of the size is occupied by
the TOAST tables when the data is stored in a single column. TOAST is
a mechanism in Postgres that kicks in when a column is large enough which
would often be the case when you design your tables this way. In this case,
the table excluding the TOASTed part is pretty small which would help
any query that doesn’t touch the large column.</p>
<h3 id="usability">Usability</h3>
<p>What matters more than size is the usability of having it all in a single
column. It is really practical to send and receive this column as a whole
all the way from the database to the frontend. Compare this with dealing
with lots of small objects used with the normalized tables.</p>
<p>It’s also possible in Postgres to index the document columns to allow
efficient searches in them. There are many options to do so with advantages
and disadvantages over each other. Though this post is getting long. I’ll
leave indexing to another one.</p>
Using SSH tunnels to get around network limitationshttps://www.endpointdev.com/blog/2022/01/using-ssh-tunnels-network-limitations/2022-01-26T00:00:00+00:00Zed Jensen
<p><img src="/blog/2022/01/using-ssh-tunnels-network-limitations/banner.jpg" alt="Cliff dwelling in Arizona"></p>
<!-- Picture by Zed Jensen, 2021 -->
<p>SSH is an extremely useful way to use computers that you aren’t in front of physically.</p>
<p>It can also be used to overcome some unique networking challenges, particularly those where one computer needs to connect to another in an unorthodox way. Let me show you a couple of uses of SSH tunnels that have come in handy for me personally.</p>
<h3 id="serving-content-without-a-public-ip-address">Serving content without a public IP address</h3>
<p>In the past, I wrote about <a href="/blog/2020/07/automating-minecraft-server/">maintaining a Minecraft server</a> to play on with my friends. In that case I was dealing with being physically separated from the server hardware I intended to use, but once I got that machine back, I realized that I still had a problem: My ISP and their networking gear didn’t support port forwarding, meaning that I couldn’t connect to my server from outside my home network. But even if I could have, the public IP address I was assigned changed regularly.</p>
<p>One solution I found was to use a reverse SSH tunnel to forward traffic from a publicly-visible virtual server to my local server.</p>
<p>To set this up, you just need your local machine and a server with a publicly visible IP address. I used a virtual machine from <a href="https://upcloud.com/">UpCloud</a>, which costs $5 per month, but you could use any other server, as long as it has its own IP address. Setup is fairly simple.</p>
<p>First, on the local machine, we create a new SSH public & private key pair just for this connection. This serves as a way to authenticate without a password, but without using a personal SSH key that has access to many more places.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">ssh-keygen -t ed25519 -f ~/.ssh/ed25519 -q -N ""
</code></pre></div><p>Next, we create a new OS user <code>proxy</code> on the server. Creating a user specifically for this purpose lets us make sure that our password-free SSH key doesn’t give access to any sensitive data on the remote server. On Linux:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">useradd -m proxy
</code></pre></div><p>Then, we add the public key generated earlier to <code>authorized_keys</code> in the new proxy user’s <code>~/.ssh</code> folder.</p>
<p>Finally, on the local machine, we run this command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">ssh -fN -i ~/.ssh/id_ed25519 -R 0.0.0.0:25565:localhost:25565 proxy@myserver
</code></pre></div><p>Here’s what each part of that does:</p>
<ul>
<li><code>-f</code> tells ssh to run in the background.</li>
<li><code>-N</code> doesn’t run a command on the remote host, which is perfect since we’re just forwarding traffic.</li>
<li><code>-i</code> specifies the SSH key we’re using.</li>
<li><code>-R</code> is a bit more complicated. <code>0.0.0.0:25565</code> specifies which traffic to intercept on the remote host and <code>localhost:25565</code> where to forward it to. Note that <code>0.0.0.0</code> means to listen on all IP addresses, such as 127.0.0.1, your internal network NAT address, and any others. <code>25565</code> is a TCP port number (UDP isn’t supported by SSH out of the box) and will vary based on the application you’re using.</li>
<li><code>proxy@myserver</code> is the user and hostname of the remote server.</li>
</ul>
<p>And that’s it! The reverse tunnel will now forward traffic from our publicly visible server to the specified port to our machine.</p>
<p>Another good use for this setup is when you have an IPv4 address at the edge of a network but only IPv6 internally. This is becoming more common as IPv4 addresses become more scarce.</p>
<p>Also note that for more permanent situations, a VPN like WireGuard or OpenVPN might be a better choice than an SSH tunnel; however, for lower-volume traffic SSH works just fine, and it is often quicker to set up.</p>
<h3 id="splitting-a-multi-container-docker-app-between-multiple-machines">Splitting a multi-container Docker app between multiple machines</h3>
<p>There are many other uses for SSH tunnels. One of our clients has an application with a lot of different moving parts that all need to communicate with each other for the entire application to function. For instance, there’s a database container, a container for part of the backend that needs CUDA support, and several others, in addition to a React frontend. Applications like this can of course slow your computer down quite a bit. What if you only need to make modifications to the frontend or another light part of the application? I asked around and found that a couple of coworkers had already found a workaround.</p>
<p>The solution is simple but effective: If you have another computer available — in my case, a desktop computer that I don’t usually use for work — you can run the performance-intensive containers on that machine. This allows you to work on the lighter parts of the application without experiencing slowdown (or fan noise, or other things like that) on your laptop.</p>
<p>Because the different parts of the application were already using Docker, it wasn’t hard to run the different pieces on separate machines, but they needed to know to talk to each other across the network. SSH tunnels let us do that without modifying our Docker configuration!</p>
<p>For the rest of this example I’ll refer to the two computers in this scenario as “the laptop” and “the desktop”, running the frontend and backend portions respectively, but keep in mind that you could do this with other setups as well.</p>
<h4 id="1-collecting-some-info">1. Collecting some info</h4>
<p>Before we can set everything up, we need to know which ports our application is communicating on. Examining the Dockerfiles for the backend containers in this application, I found that it was using ports 9933, 9934, and 10004. Normally, our frontend application would be communicating with these containers by talking to <code>http://localhost:9933</code> and so on, but once we know which ports it’s going to use, we can set up SSH to forward the traffic to our desktop machine instead.</p>
<p>The other important piece of info we’ll need is the (private home network) IP address of our desktop machine. In this case, mine is <code>192.168.0.55</code>, so we’ll use that.</p>
<h4 id="2-setting-up-the-ssh-config">2. Setting up the SSH config</h4>
<p>The most important step to making the different machines talk to each other is going to be in our SSH config, <code>~/.ssh/config</code>.</p>
<p>We’ll start by coming up with an alias hostname for our desktop, like <code>backend</code>, and creating a section for it in our config file. We then add a <code>LocalForward</code> line for each port we want to forward like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Host backend
HostName 192.168.0.55
User zed
ForwardAgent yes
LocalForward 9933 127.0.0.1:9933
LocalForward 9934 127.0.0.1:9934
LocalForward 10004 127.0.0.1:10004
</code></pre></div><h4 id="3-connecting-the-computers">3. Connecting the computers</h4>
<p>For the different parts of the application to talk to each other, there needs to be an active SSH connection. I usually open an SSH connection like normal (make sure to use the hostname we set in step 2!), start a tmux session, and then run the backend portions of the app in the tmux session:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">[zed@laptop ~]$ ssh desktop-machine
[zed@desktop ~]$ get-stuff-started
</code></pre></div><p>However, if you’d prefer, you can also open a reverse SSH connection that will run in the background until killed:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">[zed@laptop ~]$ ssh -fN desktop-machine
[zed@laptop ~]$
</code></pre></div><h4 id="4-test-it">4. Test it!</h4>
<p>Now that we’ve got an SSH session running to forward traffic to the desktop machine, you can try spinning up the React app and see if it connects properly.</p>
<h3 id="conclusion">Conclusion</h3>
<p>There are many other uses for SSH tunnels. Feel free to let us know in the comments how you’ve used them!</p>
Building a search suggestions feature with Node.js and Vuehttps://www.endpointdev.com/blog/2021/11/search-suggestions-with-node-and-vue/2021-11-27T00:00:00+00:00Greg Davidson
<p><img src="/blog/2021/11/search-suggestions-with-node-and-vue/banner.jpg" alt="Old Dog">
<a href="https://unsplash.com/photos/L1jHI4ThA44">Photo</a> by <a href="https://unsplash.com/@kaspercph">Kasper Rasmussen</a> on Unsplash</p>
<h3 id="the-backstory">The backstory</h3>
<p>Some time ago, I worked on a project to improve the usability of a search component for our clients. Similar to Google and other search interfaces, the user was presented with a number of suggested search terms as they typed into the search box. We wanted to add keyboard support and give the component a visual facelift. When the customer used the up, down, <kbd>Esc</kbd>, or <kbd>Enter</kbd> or <kbd>Return</kbd> keys, the component would allow them to choose a particular search term, clear their search, or navigate to the results for their chosen search term.</p>
<p>This is what the new and improved UI looked like:
<img src="/blog/2021/11/search-suggestions-with-node-and-vue/search-suggestions-ui.jpg" alt="Search interface with suggested search terms"></p>
<p>As developers, it can sometimes feel like we’re stuck when working on older, well established projects. We gaze longingly at newer and shinier tools. Part of my objective while building this feature was to prove the viability of a newer approach (Node.js and Vue) to the other engineers on the project as well as the client.</p>
<p>The feature existed already but we wanted to improve the UX and performance. Having added several Vue-powered features to this site in the past, I was very comfortable with the idea and have written about that <a href="/blog/2017/12/enhancing-your-sites-with-vue/">previously</a>. It would also be very easy to roll back if needed since this project was limited in scope, and very easy to compare the new solution with the code it replaced.</p>
<h3 id="picking-a-route-and-configuring-it">Picking a route and configuring it</h3>
<p>This project was running on <a href="https://www.interchangecommerce.org/i/dev">Interchange</a>, nginx, and MySQL, but our approach would work in other stacks (e.g. with Apache). One key concept is that we’re using nginx as a reverse proxy to route requests to our various apps, serve static files, etc. Using nginx in this way allows us to stitch together the different services for a given project. In this case I created an endpoint for the search suggestions endpoint in our nginx config file. This enabled requests made to <code>/suggestions/</code> to be passed along to our Node.js app. You can read more about <a href="https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/">reverse proxying with nginx</a> if you like.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-nginx" data-lang="nginx"><span style="color:#888"># Node.js powered endpoint for search suggestions
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">location</span> <span style="color:#d20;background-color:#fff0f0">/suggestions/</span> {
<span style="color:#080;font-weight:bold">proxy_pass</span> <span style="color:#d20;background-color:#fff0f0">http://0.0.0.0:8741/</span>;
}
</code></pre></div><p>If your project is running on Apache or another platform there will likely be a similar configuration option to route the requests to your Node.js app as I did.</p>
<h3 id="nodejs-time">Node.js time</h3>
<p>I wrote a little app with the <a href="https://www.npmjs.com/package/mysql">MySQL driver for Node.js</a> and <a href="https://expressjs.com/">Express</a>. The app connects to MySQL and uses the nifty built-in connection pooling. It accepts POST requests from the site and responds with an array of search suggestion objects in JSON format.</p>
<h3 id="enter-vue">Enter Vue</h3>
<p>For the front-end part of the feature I created Vue components for the search input and for the display of the results. As the customer types, results are fetched and displayed. Using the arrow keys navigates up and down and <kbd>Esc</kbd> clears out the search. Once the customer has the search they want they can either press <kbd>Enter</kbd> or click on the Search button.</p>
<p>It’s worth mentioning that this search feature works without JavaScript. When you enter a search term and press <kbd>Enter</kbd> or click the Search button, you will get results. However, if you do have JavaScript enabled (and if our suggestions app is up and running) you’ll get suggestions as you type. This is a good thing.</p>
<h3 id="performance-wins">Performance wins</h3>
<p>The new endpoint returns results in less than 100ms — a 3x or 4x improvement over the Perl script it replaced. We get a response faster, meaning the experience is much more smooth for the user!</p>
<h3 id="managing-the-nodejs-process">Managing the Node.js process</h3>
<p>We used <a href="https://pm2.keymetrics.io/docs/usage/pm2-doc-single-page/">pm2</a> and <a href="https://systemd.io/">systemd</a> to manage the Node.js processes and ensure they are started up when the server is rebooted. In my experience this has been very stable and has not required any babysitting by our operations folks.</p>
<h3 id="great-success">Great success!</h3>
<p>Have you done anything similar in your projects? Do you have vintage/legacy app that could benefit from learning some new tricks? Let us know!</p>
Database Design: Using Composite Keyshttps://www.endpointdev.com/blog/2021/05/database-design-using-composite-keys/2021-05-20T00:00:00+00:00Emre Hasegeli
<p><img src="/blog/2021/05/database-design-using-composite-keys/shipping-containers.jpg" alt="">
<a href="https://unsplash.com/photos/kyCNGGKCvyw">Photo</a> by <a href="https://unsplash.com/@chuttersnap">Chuttersnap</a></p>
<p>Whether to use single-column or composite keys is another long-debated subject of database design. I previously wrote to support <a href="/blog/2021/03/database-design-using-natural-keys/">using natural keys</a> and here I want to make good arguments for using composite keys.</p>
<h3 id="single-column-vs-composite">Single-column vs. Composite</h3>
<p>Single-column keys are widely used nowadays. I wouldn’t be surprised if many developers today don’t even think database design with composite keys is possible, even though they were essential in the beginning. Relational databases make no assumption that the keys must be composed of a single column.</p>
<p>Let’s see the composite keys with the corporate database example again. First, we’d need departments and employees:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>departments<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_location<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>username<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(username),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>Then our database grows, and we need to split the departments into multiple teams. Here’s what they’ll look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">| id | department | team | members |
| -- | ----------- | -------------- | ------- |
| 1 | sysadmin | infrastructure | 5 |
| 2 | sysadmin | internal_tools | 3 |
| 3 | development | internal_tools | 4 |
| 4 | development | web_site | 8 |
</code></pre></div><p>As you noticed there are 2 teams named <code>internal_tools</code>, so we cannot use this as the primary key column. We can add a surrogate auto-increment column to use as the primary key, or make the department and team the primary key. Let’s go with the surrogate key option first to demonstrate the problem:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>teams<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_name<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_members<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(team_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>team_name),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>As you noticed, we used the surrogate column as the primary key, and added an additional unique index to ensure the team name to be unique in with the department. Now, let’s relate the employees with the teams:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>team_id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(team_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>teams;<span style="color:#bbb">
</span></code></pre></div><p>Now, we know both the department and the team of an employee, but the problem is that they can point to inconsistent rows. For example, I can INSERT myself as an employee:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">| username | department_id | team_id |
| -------- | ------------- | ------- |
| hasegeli | sysadmin | 3 |
</code></pre></div><p><code>team_id</code> 3 is in development department, so now you’d never know if I am in the sysadmin or development department. This is a very common source of data integrity problems in the databases. Applications have no good option to handle this. They would typically crash or sometimes show the employee in one department and sometimes in the other.</p>
<p>You cannot easily add a constraint to the database to prevent this. The best option would be to remove the <code>department_id</code> when the <code>team_id</code> is added as we know the department of the team anyway, but this option is expensive and not always possible, for example when the <code>team_id</code> can be NULL.</p>
<h3 id="relations-with-composite-keys">Relations with Composite Keys</h3>
<p>Now, let’s create the teams table with a composite key:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">DROP</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>teams<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CASCADE</span>;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>teams<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_members<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>team_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>team_id<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TYPE</span><span style="color:#bbb"> </span><span style="color:#038">text</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>team_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>teams;<span style="color:#bbb">
</span></code></pre></div><p>With this method, we ensure data integrity and don’t need to disturb the existing users of the employees table while adding the teams. They can still reliably use the <code>department_id</code> column. We can also set the <code>team_id</code> as NULL and still maintain the data integrity.</p>
<h3 id="ease-of-change">Ease of Change</h3>
<p>As we have already seen, one of the benefits of using composite keys is to respond to database model changes easier and without compromising data integrity. This becomes relevant in many real world scenarios. For example, let’s add the employees to rooms which must belong to the same department:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>rooms<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>room_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>room_location<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>room_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>room_id<span style="color:#bbb"> </span><span style="color:#038">text</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>room_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>rooms;<span style="color:#bbb">
</span></code></pre></div><p>Nothing special needed to be done in here, and it is often easy to respond to more complicated change requests. Composite keys play well with database constraints. For example, let’s add an employee <code>rank</code> which needs to be unique in every department:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>employee_rank<span style="color:#bbb"> </span><span style="color:#038">int</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(employee_rank<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>employee_rank);<span style="color:#bbb">
</span></code></pre></div><p>This was so easy because we have the <code>department_id</code> in here. Now, let’s imagine a more complicated change request. We need to divide the departments into sections:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>sections<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>section_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>section_location<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>teams<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>section_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">DROP</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CONSTRAINT</span><span style="color:#bbb"> </span>teams_pkey<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CASCADE</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id,<span style="color:#bbb"> </span>team_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(epartment_id,<span style="color:#bbb"> </span>section_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>sections;<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">COLUMN</span><span style="color:#bbb"> </span>section_id<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id,<span style="color:#bbb"> </span>team_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>teams;<span style="color:#bbb">
</span></code></pre></div><p>As you see, we can make this happen with minimal impact to the users of the database. As a final practice, let’s reserve employee ranks between 1 and 10 for the <code>main</code> section of every department:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">ALTER</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ADD</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">CHECK</span><span style="color:#bbb"> </span>(section_id<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'main'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">OR</span><span style="color:#bbb"> </span>employee_rank<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">10</span>);<span style="color:#bbb">
</span></code></pre></div><h3 id="querying">Querying</h3>
<p>Another advantage of using composite keys is to have more possibility for joining of tables. For example using the tables we created we can join employees to sections without using the teams tables. This would not be possible if we had used single-column keys everywhere.</p>
<p>Join conditions get complicated with composite keys. The <code>USING</code> clause helps. To demonstrate, let’s join all of the tables we created so far:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>deparments<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>sections<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>rooms<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>room_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>teams<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id,<span style="color:#bbb"> </span>team_id);<span style="color:#bbb">
</span></code></pre></div><p>The <code>USING</code> clause only works if you name the columns the same on all tables. I recommend doing so.</p>
<p>Another advantage of <code>USING</code> clause is to eliminate duplicate columns on the result set. If you run this query, you would not see the <code>department_id</code> column repeated 5 times.</p>
<h3 id="performance-considerations">Performance Considerations</h3>
<p>One disadvantage of using composite keys is to store more data on tables as references. You would also need more space for the indexes as the reference columns often need to be indexed. However storage is the cheapest of resources, and the performance advantages easily outweigh the extra storage.</p>
<p>The main performance advantage of using composite keys is eliminating the need for many joins as mentioned before. However, when you do need to join many tables, the query planner would have many different paths. It’s the query planners’ strong suit to find the best join order. Composite keys allow them to come up with better plans in many scenarios. To demonstrate this, let’s get our join-all-tables query and add some WHERE conditions:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>employees<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>deparments<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>sections<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>rooms<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>room_id);<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>teams<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">USING</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>section_id,<span style="color:#bbb"> </span>team_id)<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>username<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIKE</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'a%'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_location<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIKE</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'b%'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>section_location<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIKE</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'c%'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AND</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>room_location<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">LIKE</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'d%'</span>;<span style="color:#bbb">
</span><span style="color:#bbb"> </span>team_members<span style="color:#bbb"> </span>><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">3</span>;<span style="color:#bbb">
</span></code></pre></div><p>Now we added 5 conditions using 5 columns on 5 different tables. The query planner can estimate which conditions are more selective and plan to join the tables from the smaller one to bigger one.</p>
Database Design: Using Natural Keyshttps://www.endpointdev.com/blog/2021/03/database-design-using-natural-keys/2021-03-15T00:00:00+00:00Emre Hasegeli
<p><img src="/blog/2021/03/database-design-using-natural-keys/safe-boxes.jpg" alt="Bank safe deposit boxes"></p>
<p>Whether to use natural or surrogate keys is a long-debated subject of database design. I am a fan of using natural keys. I think there are even more compelling reasons to use natural keys in databases as the systems grow more complex and interdependent.</p>
<h3 id="natural-or-surrogate">Natural or Surrogate</h3>
<p>Let’s start by what we mean by natural. It’s not trivial to define this. In today’s world of APIs, someone’s surrogate key is another’s natural key. Wikipedia <a href="https://en.wikipedia.org/wiki/Natural_key">defines natural keys</a> as “a type of unique key in a database formed of attributes that exist and are used in the external world outside the database”. This makes it clear that the keys we get from APIs are our natural keys. But how about the ones generated by us to be used in the external world?</p>
<p>When applications expose the keys on the URLs and APIs, others start relying on them. This is where our choices become important. When all those different applications generate their own keys instead of using the keys they got from other places, life becomes difficult for no reason.</p>
<p>Let’s elaborate with an example corporate database where employees are identified by their usernames and the departments with their domains. So our data would look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">| department | username | job |
| ----------------- | --------- | ----------------------- |
| sysadm.corp-x.com | hasegeli | Database Administrator |
| sysadm.corp-x.com | john | System Administrator |
| dep1.corp-x.com | jane | Developer |
</code></pre></div><p>When we design this using surrogate keys, it’d look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>departments<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">domain</span><span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">domain</span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>employees<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>username<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(username)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>department_employees<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">GENERATED</span><span style="color:#bbb"> </span>ALWAYS<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">AS</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IDENTITY</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>department_id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>employee_id<span style="color:#bbb"> </span><span style="color:#038">int</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>job<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">UNIQUE</span><span style="color:#bbb"> </span>(department_id,<span style="color:#bbb"> </span>employee_id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(department_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>departments<span style="color:#bbb"> </span>(id),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FOREIGN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(employee_id)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">REFERENCES</span><span style="color:#bbb"> </span>employees<span style="color:#bbb"> </span>(id)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>Then, we would have to make a query joining them all together:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>d.<span style="color:#080;font-weight:bold">domain</span>,<span style="color:#bbb"> </span>e.username,<span style="color:#bbb"> </span>de.job<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>department_employees<span style="color:#bbb"> </span>de<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>departments<span style="color:#bbb"> </span>d<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>de.department_id<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span>d.id<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">JOIN</span><span style="color:#bbb"> </span>employees<span style="color:#bbb"> </span>e<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ON</span><span style="color:#bbb"> </span>de.employee_id<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span>e.id;<span style="color:#bbb">
</span></code></pre></div><p>Instead, we could just have it as we need it on a single table:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">CREATE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">TABLE</span><span style="color:#bbb"> </span>department_employees<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">domain</span><span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>username<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb"> </span>job<span style="color:#bbb"> </span><span style="color:#038">text</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NOT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">NULL</span>,<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">PRIMARY</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">KEY</span><span style="color:#bbb"> </span>(<span style="color:#080;font-weight:bold">domain</span>,<span style="color:#bbb"> </span>username)<span style="color:#bbb">
</span><span style="color:#bbb"></span>);<span style="color:#bbb">
</span></code></pre></div><p>And, we could just query it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">domain</span>,<span style="color:#bbb"> </span>username,<span style="color:#bbb"> </span>job<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>department_employees;<span style="color:#bbb">
</span></code></pre></div><h3 id="good-examples-of-natural-keys">Good Examples of Natural Keys</h3>
<p>A primary key must be always available (NOT NULL) and unique. It should also be compact and easy to represent. Here are some good examples of natural keys:</p>
<ul>
<li>Keys acquired from various APIs</li>
<li>Usernames</li>
<li>Domains</li>
<li>Email addresses</li>
<li>ISO country codes</li>
<li>IATA airport codes</li>
<li>Dates</li>
</ul>
<p>I think easy representation matters for unchanged use among many systems. For that reason, I don’t think names with spaces, symbols, or non-ASCII Unicode letters are good keys.</p>
<p>UUIDs and hashes/digests like MD5 and SHA are also popular as primary keys. I don’t think they are a good choice as surrogate keys. Auto-incrementing IDs are easier to use, generate, store, predict — although, when they are defined in the external world and we have to store them, there’s nothing wrong with using them as the primary keys.</p>
<p>Another consideration when selecting keys is their stability. Things that can change are not good keys. This is not only because it is more work to change the keys everywhere, but because different parts of the system cannot rely on them.</p>
<p>The context matters to evaluate a key’s stability. For example, a phone number is not a good key for the users table, because users may change their phone number. However, it is perfect for the phone numbers table.</p>
<p>Also, I don’t think we should discard some useful keys just because they can change in an edge case. It is a good trade-off to allow the rare change rather than to complicate the system with a surrogate key.</p>
<h3 id="enterprise-architecture">Enterprise Architecture</h3>
<p>I don’t know what you understand from the word “enterprise” but I have heard people advocating streamlined, unpractical solutions in the name of “enterprise” for a long time. I have seen people using layers of layers in complex frameworks and ORMs with caches and buffers, managers and supervisors, factories and workers, locks and semaphores, queues and consumers, all in the name of “enterprise architecture”.</p>
<p>Many of those frameworks have a rule like requiring all database tables to have a single-column auto-incremented primary key named “id”. Such a rule is easy to enforce and follow but it’s limiting. It would not be practical, maybe not even possible to use natural keys as the primary keys within such framework.</p>
<p>Even though I am writing this post to support using natural keys, strong conventions built around surrogate keys do certainly help developing complex systems in the “enterprise” environments. They help to keep systems consistent when the developers are more motivated to write code than to read what’s there, while working until they get a better offer from another company.</p>
<h3 id="performance-consideration">Performance Consideration</h3>
<p>Performance is often an argument in favour of surrogate keys. Surrogate keys can be small auto-incrementing integers which would keep the indexes small, next branches easy to predict, so well performing.</p>
<p>Surrogate keys would take a bit of extra space since otherwise we don’t need to have them at all, but the space savings of using them as foreign keys would easily outweigh this. However, using surrogate keys may require the queries to have more tables joined together.</p>
<p>Some relational databases maintain the tables indexed by the primary key. Such systems are more affected by the size of the primary key. On PostgreSQL all indexes are secondary including the primary key, therefore the primary key is not any more important than the other indexes on the table.</p>
<h3 id="composite-keys">Composite Keys</h3>
<p>Natural keys are often unique together with other keys. This forces the database designer to use multiple columns as keys. Such design has many advantages, which I may cover in future blog posts.</p>
Svelte: A compiled JavaScript front-end frameworkhttps://www.endpointdev.com/blog/2020/12/svelte-a-compiled-js-framework/2020-12-02T00:00:00+00:00Kevin Campusano
<p><img src="/blog/2020/12/svelte-a-compiled-js-framework/banner.png" alt="Svelte banner"></p>
<p>It seems like <a href="https://en.wikipedia.org/wiki/Comparison_of_JavaScript_frameworks">JavaScript frameworks</a> are <a href="https://dayssincelastjavascriptframework.com/">a dime a dozen these days</a>. In the JavaScript world, there are frameworks and libraries for almost every task that you can imagine, and there is much overlap. In the front-end web development area though, there are three who reign supreme: <a href="https://angularjs.org/">Angular</a>, <a href="https://reactjs.org/">React</a>, and <a href="https://vuejs.org/">Vue.js</a>. Whether you’re building <a href="https://en.wikipedia.org/wiki/Single-page_application">single-page apps</a> or traditional web apps with some front-end zest, you can’t go wrong with any of these three.</p>
<p>Recently though, I discovered a new player in this field that promises to offer a similar developer experience to existing frameworks with <a href="https://youtu.be/AdNJ3fydeao?t=1084">great performance gains</a>. Its name is <a href="https://svelte.dev/">Svelte</a>, and it achieves this promise in a very interesting way: <a href="https://svelte.dev/blog/svelte-3-rethinking-reactivity">it’s a compiler</a>.</p>
<p>This is a novel approach to the front-end web development problem. While traditional front-end frameworks are libraries with runtime components that get loaded and executed in the browser, Svelte itself only runs at development/build time, on your development machine. Svelte takes the JavaScript components that we write, compiles them into pure, lean JavaScript and emits only that to the browser. No big library, or runtime, or interpretation of components ever happens in the browser. It doesn’t do any of this, and yet still offers a robust component framework with all the features that you might expect like reactivity, templating capabilities, scoped CSS, data binding, etc.</p>
<p>If we can truly get much better performance while maintaining a familiar developer experience when compared to more traditional frameworks (or improving it in some cases), that sounds like we get to have our cake and eat it too.</p>
<p>After learning all that, I decided to take a closer look at Svelte and see how it works, how it’s similar to and different from other frameworks, and what new features it brings to the table. In this blog post, I’ll discuss some of my most interesting findings.</p>
<h3 id="simple-components">Simple components</h3>
<p>In Svelte, the concept and mechanics of components are similar to those of other frameworks. Components encapsulate the HTML, CSS, and JavaScript of a given portion of our applications in a reusable manner. Svelte components look very similar to plain old HTML files, <a href="https://svelte.dev/blog/write-less-code">minimal fluff</a>. Here’s what a “hello world” example could look like as a Svelte component:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">h1</span>>{ message }!</<span style="color:#b06;font-weight:bold">h1</span>>
<<span style="color:#b06;font-weight:bold">p</span>>{ author }</<span style="color:#b06;font-weight:bold">a</span>>
<<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#080;font-weight:bold">export</span> <span style="color:#080;font-weight:bold">let</span> author;
<span style="color:#080;font-weight:bold">let</span> message = <span style="color:#d20;background-color:#fff0f0">"Hello World"</span>;
</<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">style</span>>
<span style="color:#b06;font-weight:bold">h1</span> {
<span style="color:#080;font-weight:bold">color</span>: <span style="color:#080;font-weight:bold">royalblue</span>;
}
</<span style="color:#b06;font-weight:bold">style</span>>
</code></pre></div><p>Very simple, refreshing even. This content would live inside a <code>*.svelte</code> file which is what Svelte components should be named. As you can see, the <code><style></code> and <code><script></code> tags contain just plain old CSS and JavaScript, respectively. <code>export let author</code> is how we declare component <a href="https://svelte.dev/tutorial/declaring-props">props</a>. That is, values that are supplied as parameters by the component that’s parent to this one, or, if this was the top most component, by the client code that initializes the Svelte application.</p>
<p>The rest of the file, the view template, is also plain HTML with some curly brace goodness injected by Svelte that allows us to render the results from JavaScript expressions. In this case, we are just rendering the <code>message</code> and <code>author</code> variable that we defined in the <code><script></code> portion of our component.</p>
<p>Now, Svelte has a lot of features, so it does add some syntax to spice up HTML templates and it also uses some unconventional JavaScript to express some constructs. Nothing too revolutionary though, as we’ll see later.</p>
<h3 id="its-easy-to-set-up">It’s easy to set up</h3>
<p>In front-end web development, we spend most of the time authoring components like these, so we’ll talk more about them later. In order for the components to do anything, though, we need that compilation step that I mentioned earlier. That means setting up a new Svelte project in our development environment. Luckily for us, Svelte makes this easy.</p>
<p>All you need is to install <a href="https://nodejs.org">Node.js</a> so that you have <code>npm</code> and <code>npx</code> available and can run this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npx degit sveltejs/template my-svelte-project
</code></pre></div><p>This will use the Svelte project template hosted <a href="https://github.com/sveltejs/template">here</a> to get you up and running quickly with a working application and build pipeline.</p>
<p><code>cd</code> into the <code>my-svelte-project</code> directory that was just created and run <code>npm install</code> to install all dependencies needed by the app. Then, run <code>npm run dev</code> and that should start a development web server that you can reach with your browser via “localhost:5000”. You’ll be greeted by this:</p>
<p><img src="/blog/2020/12/svelte-a-compiled-js-framework/hello-world.png" alt="Hello to the Svelte world"></p>
<p>Feel free to explore the app files. You’ll see there’s an <code>App.svelte</code> file that has our sole app component. It contains some HTML, CSS, and JS. <code>main.js</code> is particularly interesting for us at this point because it shows us how a Svelte app is initiated.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">import</span> App from <span style="color:#d20;background-color:#fff0f0">'./App.svelte'</span>;
<span style="color:#080;font-weight:bold">const</span> app = <span style="color:#080;font-weight:bold">new</span> App({
target: <span style="color:#038">document</span>.body,
props: {
name: <span style="color:#d20;background-color:#fff0f0">'world'</span>
}
});
<span style="color:#080;font-weight:bold">export</span> <span style="color:#080;font-weight:bold">default</span> app;
</code></pre></div><p>This should be very familiar if you’re used to the likes of Vue. Here, we import the <code>App.svelte</code> file and instantiate the JavaScript component contained within by passing it its target (that is, where in the DOM it is going to be mounted) and some <a href="https://svelte.dev/tutorial/declaring-props">props</a>.</p>
<p>Notice how we import the component file directly and no Svelte library. We don’t need it because our app does not need Svelte to run. It only needs Svelte to build. Svelte, after all, is a compiler. It’s not a runtime dependency, but rather, a build time one.</p>
<p>You can learn more about the parameters that Svelte components expect in <a href="https://svelte.dev/docs#Client-side_component_API">the official documentation</a>.</p>
<p>Another interesting aspect of setting up a Svelte app is that it uses <a href="https://rollupjs.org">Rollup</a> as a module bundler by default. You can confirm this by looking at the <code>my-svelte-project/rollup.config.js</code> file that was created. If you prefer <a href="https://webpack.js.org/">Webpack</a>, Svelte also supports it. All you need to do to build with Webpack instead of Rollup is use this command when creating your app:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npx degit sveltejs/template-webpack svelte-app
</code></pre></div><p>This is just using the project template hosted on <a href="https://github.com/sveltejs/template-webpack">GitHub</a> and it will build out a Webpack-based application functionally identical to the one we’ve just created. You will see that such projects include a <code>webpack.config.js</code> file instead of <code>rollup.config.js</code>.</p>
<h3 id="the-templating-capabilities-are-powerful">The templating capabilities are powerful</h3>
<p>When it comes to templating, Svelte has everything that you would expect from a modern JavaScript front-end framework and then some. You’ve got your basics like interpolation and flow control, but there are also some more advanced features that look to be very useful. Here’s an example component that demonstrate some of the most common capabilities:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#080;font-weight:bold">let</span> title = <span style="color:#d20;background-color:#fff0f0">'Trying out some Svelte features'</span>;
<span style="color:#080;font-weight:bold">let</span> itsAllJustJavaScript = <span style="color:#d20;background-color:#fff0f0">"It's all just JavaScript"</span>;
<span style="color:#080;font-weight:bold">let</span> youCanAlsoUseHtml = <span style="color:#d20;background-color:#fff0f0">'You can also use <b><em>HTML</em></b>'</span>;
<span style="color:#080;font-weight:bold">let</span> showThis = <span style="color:#080;font-weight:bold">true</span>;
<span style="color:#080;font-weight:bold">let</span> things = [<span style="color:#d20;background-color:#fff0f0">'a thing'</span>, <span style="color:#d20;background-color:#fff0f0">'another thing'</span>, <span style="color:#d20;background-color:#fff0f0">'a thingy'</span>];
<span style="color:#080;font-weight:bold">let</span> complicatedThings = [
{id: <span style="color:#00d;font-weight:bold">1</span>, what: <span style="color:#d20;background-color:#fff0f0">'car'</span>, why: <span style="color:#d20;background-color:#fff0f0">'it can go fast'</span>},
{id: <span style="color:#00d;font-weight:bold">2</span>, what: <span style="color:#d20;background-color:#fff0f0">'plane'</span>, why: <span style="color:#d20;background-color:#fff0f0">'it can fly'</span>},
{id: <span style="color:#00d;font-weight:bold">3</span>, what: <span style="color:#d20;background-color:#fff0f0">'submarine'</span>, why: <span style="color:#d20;background-color:#fff0f0">'it can go underwater'</span>}
];
</<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">main</span>>
<span style="color:#888"><!-- We can use any variable that's declared in <script> --></span>
<<span style="color:#b06;font-weight:bold">h1</span>>{title}</<span style="color:#b06;font-weight:bold">h1</span>>
<span style="color:#888"><!-- We can use any valid JavaScript expression and its return value will be
</span><span style="color:#888"> rendered.--></span>
<<span style="color:#b06;font-weight:bold">p</span>>{itsAllJustJavaScript.toUpperCase()}</<span style="color:#b06;font-weight:bold">p</span>>
<span style="color:#888"><!-- Svelte will make sure to sanitize text so that no HTML is rendered by
</span><span style="color:#888"> mistake... --></span>
<<span style="color:#b06;font-weight:bold">p</span>>{youCanAlsoUseHtml}</<span style="color:#b06;font-weight:bold">p</span>>
<span style="color:#888"><!-- ...unless you explicitly tell it not to by using @html --></span>
<<span style="color:#b06;font-weight:bold">p</span>>{@html youCanAlsoUseHtml}</<span style="color:#b06;font-weight:bold">p</span>>
<span style="color:#888"><!-- Conditional logic works as you would expect. --></span>
{#if showThis}
<<span style="color:#b06;font-weight:bold">p</span>>Showing this</<span style="color:#b06;font-weight:bold">p</span>>
<span style="color:#888"><!-- There's also {:else if ...} if you need it. --></span>
{:else}
<<span style="color:#b06;font-weight:bold">p</span>>And not that</<span style="color:#b06;font-weight:bold">p</span>>
{/if}
<span style="color:#888"><!-- It can iterate over an array. --></span>
<<span style="color:#b06;font-weight:bold">h2</span>>Things</<span style="color:#b06;font-weight:bold">h2</span>>
<<span style="color:#b06;font-weight:bold">ul</span>>
{#each things as thing}
<<span style="color:#b06;font-weight:bold">li</span>>{thing}</<span style="color:#b06;font-weight:bold">li</span>>
{/each}
</<span style="color:#b06;font-weight:bold">ul</span>>
<span style="color:#888"><!-- It can also iterate over an array of objects and use their fields. --></span>
<<span style="color:#b06;font-weight:bold">h2</span>>Complicated things</<span style="color:#b06;font-weight:bold">h2</span>>
<<span style="color:#b06;font-weight:bold">ul</span>>
{#each complicatedThings as thing}
<<span style="color:#b06;font-weight:bold">li</span>>{thing.id}: a {thing.what} is complicated because {thing.why}</<span style="color:#b06;font-weight:bold">li</span>>
{/each}
</<span style="color:#b06;font-weight:bold">ul</span>>
</<span style="color:#b06;font-weight:bold">main</span>>
</code></pre></div><p>There’s a more advanced feature that I think can prove very useful. Svelte supports rendering of templates based on promises. Here’s an example demonstrating how easy it is to implement a very common pattern on front-end development: displaying a loading message while we fetch some information via an HTTP request:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888">// First, we define our promise and capture it in a variable.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">let</span> url = <span style="color:#d20;background-color:#fff0f0">'https://vpic.nhtsa.dot.gov/api/vehicles/getmodelsformake/subaru?format=json'</span>
<span style="color:#080;font-weight:bold">let</span> promiseToGetSubarus = fetch(url).then(response => {
<span style="color:#080;font-weight:bold">if</span> (response.ok) {
<span style="color:#080;font-weight:bold">return</span> response.json();
} <span style="color:#080;font-weight:bold">else</span> {
<span style="color:#080;font-weight:bold">throw</span> <span style="color:#080;font-weight:bold">new</span> <span style="color:#038">Error</span>(<span style="color:#d20;background-color:#fff0f0">"Something went wrong ☹️"</span>);
}
});
</<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">main</span>>
<span style="color:#888"><!-- Then, in the template, we await the promise and show a loading message
</span><span style="color:#888"> in the meantime. --></span>
{#await promiseToGetSubarus}
<<span style="color:#b06;font-weight:bold">p</span>>loading Subarus...</<span style="color:#b06;font-weight:bold">p</span>>
<span style="color:#888"><!-- When the promise resolves, we can render the result. The "data"
</span><span style="color:#888"> variable here captures what our promise resolves with, in this case,
</span><span style="color:#888"> it's "response.json()", like we defined in our JavaScript code. --></span>
{:then data}
<<span style="color:#b06;font-weight:bold">p</span>>Message: {data['Message']}</<span style="color:#b06;font-weight:bold">p</span>>
<<span style="color:#b06;font-weight:bold">h2</span>>Subaru Models</<span style="color:#b06;font-weight:bold">h2</span>>
<<span style="color:#b06;font-weight:bold">ul</span>>
{#each data['Results'] as model}
<<span style="color:#b06;font-weight:bold">li</span>>{model['Model_Name']}</<span style="color:#b06;font-weight:bold">li</span>>
{/each}
</<span style="color:#b06;font-weight:bold">ul</span>>
<span style="color:#888"><!-- We can also do some error handling in case our request fails. --></span>
{:catch error}
<<span style="color:#b06;font-weight:bold">p</span> <span style="color:#369">style</span>=<span style="color:#d20;background-color:#fff0f0">"color: red"</span>>{error.message}</<span style="color:#b06;font-weight:bold">p</span>>
{/await}
</<span style="color:#b06;font-weight:bold">main</span>>
</code></pre></div><p>This is a pretty neat feature. It embraces the modern JavaScript concept of promises and has the potential of reducing a good amount of boilerplate and, as a consequence, make our components easier to read.</p>
<h3 id="reactivity-is-built-in-and-leverages-a-quirky-javascript-feature">Reactivity is built in, and leverages a quirky JavaScript feature</h3>
<p>Svelte also checks the reactivity box, and it does so in some interesting ways.</p>
<p>First of all, any value that you include in a template is fully reactive. So, to take an example from Svelte’s own documentation, this works as you’d expect:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888">// Here's our variable.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">let</span> count = <span style="color:#00d;font-weight:bold">0</span>;
<span style="color:#888">// Here's a function that updates our "count" variable.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">function</span> handleClick() {
count += <span style="color:#00d;font-weight:bold">1</span>;
}
</<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888"><!-- This is how we tell Svelte to respond to the click event on this button by
</span><span style="color:#888"> calling the handleClick function. --></span>
<<span style="color:#b06;font-weight:bold">button</span> <span style="color:#369">on:click</span>=<span style="color:#d20;background-color:#fff0f0">{handleClick}</span>>
<span style="color:#888"><!-- We use the count variable to render some markup here. It is reactive by
</span><span style="color:#888"> default. That means that the view will be updated automatically
</span><span style="color:#888"> every time the value changes. That is, every time the button is clicked. --></span>
Clicked {count} {count === 1 ? 'time' : 'times'}
</<span style="color:#b06;font-weight:bold">button</span>>
</code></pre></div><p>Try it out in <a href="https://svelte.dev/tutorial/reactive-assignments">Svelte’s official tutorial</a>.</p>
<p>There’s also support for computed values. That is, values that are derived from other values and reactively change whenever the values they depend on are changed. Svelte solves this problem with so called <a href="https://svelte.dev/tutorial/reactive-declarations">reactive declarations</a>. They look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">$: evenness = count % <span style="color:#00d;font-weight:bold">2</span> == <span style="color:#00d;font-weight:bold">0</span> ? <span style="color:#d20;background-color:#fff0f0">'even'</span> : <span style="color:#d20;background-color:#fff0f0">'odd'</span>;
</code></pre></div><p>In the context of our count example, we could use it with something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#080;font-weight:bold">let</span> count = <span style="color:#00d;font-weight:bold">0</span>;
<span style="color:#888">// Here's a reactive declaration. It gets recalculated every time "count"
</span><span style="color:#888"></span> <span style="color:#888">// changes. That is, every time the button is clicked.
</span><span style="color:#888"></span> $: evenness = count % <span style="color:#00d;font-weight:bold">2</span> == <span style="color:#00d;font-weight:bold">0</span> ? <span style="color:#d20;background-color:#fff0f0">'even'</span> : <span style="color:#d20;background-color:#fff0f0">'odd'</span>;
<span style="color:#080;font-weight:bold">function</span> handleClick() {
count += <span style="color:#00d;font-weight:bold">1</span>;
}
</<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">button</span> <span style="color:#369">on:click</span>=<span style="color:#d20;background-color:#fff0f0">{handleClick}</span>>
Clicked {count} {count === 1 ? 'time' : 'times'}
</<span style="color:#b06;font-weight:bold">button</span>>
<span style="color:#888"><!-- And as expected, this portion of the view also gets updated when "evenness"
</span><span style="color:#888"> changes. --></span>
<<span style="color:#b06;font-weight:bold">p</span>>
Evenness of count: {evenness}
</<span style="color:#b06;font-weight:bold">p</span>>
</code></pre></div><p>The fist time I saw this <code>$:</code> syntax I thought it was weird, and it kind of is. However, it’s actually valid JavaScript. This is just a label (as explained in <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/label">MDN</a>). The Svelte magic comes into play when it finds a label like this one. The compiler looks at the statement to the right of the label and does what it needs to do in order to make it reactive.</p>
<p>Svelte can also reactively run arbitrary code. Pretty much any statement can be run reactively. For example, all of these work:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// This gets executed every time "evenness" changes.
</span><span style="color:#888"></span>$: console.log(<span style="color:#d20;background-color:#fff0f0">"The evenness changed to: "</span> + evenness);
<span style="color:#888">// This gets executed every time "evenness" or "count" changes.
</span><span style="color:#888"></span>$: {
console.log(<span style="color:#d20;background-color:#fff0f0">"The evenness changed to: "</span> + evenness);
console.log(<span style="color:#d20;background-color:#fff0f0">"The count is: "</span> + count)
}
<span style="color:#888">// This gets executed every time "count" changes.
</span><span style="color:#888"></span>$: <span style="color:#080;font-weight:bold">if</span> (count >= <span style="color:#00d;font-weight:bold">10</span>) {
alert(<span style="color:#d20;background-color:#fff0f0">"count is dangerously high!"</span>);
}
</code></pre></div><h3 id="you-can-go-low-level">You can go low level</h3>
<p>Svelte offers various mechanisms for going low level and directly handling the DOM. One such method is the “this” binding. With it, we can take a given HTML element and have Svelte assign a reference to it into a variable of our choosing. Here’s how it works:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888">// First, we need to import the "onMount" function which allows us to
</span><span style="color:#888"></span> <span style="color:#888">// execute some code when the Svelte component is fully mounted into the DOM
</span><span style="color:#888"></span> <span style="color:#888">// and ready to use.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">import</span> { onMount } from <span style="color:#d20;background-color:#fff0f0">'svelte'</span>;
<span style="color:#888">// Here we define this variable which we will use to capture a reference to
</span><span style="color:#888"></span> <span style="color:#888">// one of our DOM elements.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">let</span> myDiv;
<span style="color:#888">// The "myDiv" variable is initialized when the component is mounted. So any
</span><span style="color:#888"></span> <span style="color:#888">// code that uses it needs to be inside an "onMount" callback. The variable
</span><span style="color:#888"></span> <span style="color:#888">// contains a reference to our div element defined below.
</span><span style="color:#888"></span> onMount(() => {
console.log(<span style="color:#d20;background-color:#fff0f0">"Here's a reference to some div: "</span>, myDiv);
myDiv.innerText = <span style="color:#d20;background-color:#fff0f0">"The component has been mounted"</span>;
});
</<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888"><!-- Using this 'bind:this="{div}"' binding, we tell Svelte to store a reference
</span><span style="color:#888"> to the DOM element represented by this div into the myDiv variable. --></span>
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">bind:this</span>=<span style="color:#d20;background-color:#fff0f0">"{myDiv}"</span>>
This is some text that will go away.
</<span style="color:#b06;font-weight:bold">div</span>>
</code></pre></div><p>The only thing to keep in mind is that we should only use the <code>myDiv</code> variable inside the <code>onMount</code> life cycle hook (more info <a href="https://svelte.dev/tutorial/onmount">here</a>). This is the only way to make sure that the variable is correctly initialized, since it will be <code>undefined</code> before the component is mounted.</p>
<p>Another mechanism to perform low level DOM operations is via “actions” and the “use” directive. This is a more robust and reusable alternative to the “this” binding in that it allows us to define a separate function to augment a given DOM element with custom functionality and interact with the rest of the Svelte app ecosystem via firing events. Here’s an example illustrating the most unnecessarily roundabout way of making a div clickable using the DOM API directly:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888">// This function encapsulates all direct DOM manipulations. It receives a
</span><span style="color:#888"></span> <span style="color:#888">// "node" parameter which contains a reference to the DOM element that had
</span><span style="color:#888"></span> <span style="color:#888">// the "use" directive called on it with this "customClickable" as a
</span><span style="color:#888"></span> <span style="color:#888">// parameter. This function could have been defined in a separate file and
</span><span style="color:#888"></span> <span style="color:#888">// imported here for easy reusability.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">function</span> customClickable(node) {
<span style="color:#080;font-weight:bold">function</span> handleOnClick(event) {
<span style="color:#888">// We have full access to the DOM element. We can use members like
</span><span style="color:#888"></span> <span style="color:#888">// "id" or "dispatchEvent".
</span><span style="color:#888"></span> console.log(<span style="color:#d20;background-color:#fff0f0">`dispatching </span><span style="color:#33b;background-color:#fff0f0">${</span>node.id<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">'s customclick event`</span>);
<span style="color:#888">// Here we dispatch a custom event named "customclick" with some
</span><span style="color:#888"></span> <span style="color:#888">// arbitrary parameters.
</span><span style="color:#888"></span> node.dispatchEvent(<span style="color:#080;font-weight:bold">new</span> CustomEvent(<span style="color:#d20;background-color:#fff0f0">'customclick'</span>, {
detail: { x: event.clientX, y: event.clientY }
}));
}
<span style="color:#888">// Here we directly use the DOM API to set up an event handler for our
</span><span style="color:#888"></span> <span style="color:#888">// "node".
</span><span style="color:#888"></span> node.addEventListener(<span style="color:#d20;background-color:#fff0f0">'click'</span>, handleOnClick);
<span style="color:#888">// We should return an object with a "destroy" method to be called by
</span><span style="color:#888"></span> <span style="color:#888">// Svelte when the component is unmounted. In this case, all we do is
</span><span style="color:#888"></span> <span style="color:#888">// remove the event listener that we added above.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> {
destroy() {
node.removeEventListener(<span style="color:#d20;background-color:#fff0f0">'click'</span>, handleOnClick);
}
};
}
<span style="color:#888">// This function gets passed the custom event data, just as
</span><span style="color:#888"></span> <span style="color:#888">// "customClickable" defines it when it calls "dispatchEvent" on the "node".
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">function</span> logCoordinates(event) {
console.log(<span style="color:#d20;background-color:#fff0f0">"Div clicked in "</span>, {
x: event.detail.x, y: event.detail.y
});
}
</<span style="color:#b06;font-weight:bold">script</span>>
<span style="color:#888"><!-- When defining our custom clickable div, we use the "use:customClickable"
</span><span style="color:#888"> directive to tell Svelte to call the "customClickable" function and pass it
</span><span style="color:#888"> the div's underlying DOM element. "customClickable" in turn augments the div
</span><span style="color:#888"> so that it can dispatch a new type of event: "customclick". Then, using
</span><span style="color:#888"> Svelte's usual event handling mechanisms, we set the element up so that when
</span><span style="color:#888"> the div dispatches the "customclick" event, "logCoordinates" gets executed.
</span><span style="color:#888">--></span>
<<span style="color:#b06;font-weight:bold">div</span>
<span style="color:#369">id</span>=<span style="color:#d20;background-color:#fff0f0">"my_custom_clickable_div"</span>
<span style="color:#369">style</span>=<span style="color:#d20;background-color:#fff0f0">"border: 1px solid blue; padding: 10px; cursor: pointer;"</span>
<span style="color:#369">use:customClickable</span>
<span style="color:#369">on:customclick</span>=<span style="color:#d20;background-color:#fff0f0">{logCoordinates}</span>
>
Click here and see what happens.
</<span style="color:#b06;font-weight:bold">div</span>>
</code></pre></div><p>Again, this example is doing something completely unnecessary because divs are already clickable. For a more complex example you can study <a href="https://svelte.dev/tutorial/actions">this one from Svelte’s official documentation</a>.</p>
<p>And that’s it for now. These are the features that called my attention as I dove into Svelte for the first time. Of course, Svelte offers many other features that I didn’t discuss here like:</p>
<ul>
<li><a href="https://svelte.dev/tutorial/text-inputs">Two-way data binding</a> for input elements in forms and even between components.</li>
<li><a href="https://svelte.dev/tutorial/dom-events">Handling and dispatching DOM and custom events</a> to keep the flow of data through the component hierarchy clean and obvious.</li>
<li><a href="https://svelte.dev/tutorial/transition">Transitions</a> and <a href="https://svelte.dev/tutorial/animate">Animations</a> which in Svelte are built in, first-class citizens.</li>
<li><a href="https://svelte.dev/tutorial/slots">Component composition</a>: traditional nesting and via slots.</li>
<li><a href="https://svelte.dev/tutorial/writable-stores">Global application state management</a> à la <a href="https://vuex.vuejs.org/">Vuex</a> or <a href="https://redux.js.org/">Redux</a>.</li>
<li>A full-stack application framework in the form of <a href="https://sapper.svelte.dev/">Sapper</a>.</li>
</ul>
<p>These are features that other frameworks also include and Svelte offers similar implementations, with its own particular flavor. I just wanted to mention them so you know they are there. To learn more, I’d encourage you to work through the <a href="https://svelte.dev/tutorial/">excellent official tutorial</a>.</p>
<p>When talking about features, Svelte has one advantage in that it has more freedom than traditional frameworks to add a lot of functionality. They have this freedom because more features don’t mean any additional overhead (in space or performance) for those who don’t use them. Svelte is a compiler, so it has the ability to produce a bundle which includes only the features that are being used in your app. In other words, the framework can grow and not have to deal with the constraints of traditional library-based frameworks. The future seems bright.</p>
<p>All in all, I think Svelte is a great contender. It offers all the features that put it right up there with the big guys. It also promises a great uplift in performance when compared to said big guys. As a cherry on top, it offers a great developer/code authoring experience which is familiar and also arguably better than existing frameworks. I for one am looking forward to seeing how the project evolves and matures and what cool things people make with it. And of course, also trying it out in some projects of my own.</p>
Demonstrating the HotSwap JVMhttps://www.endpointdev.com/blog/2020/11/java-hotswap/2020-11-25T00:00:00+00:00Josh Tolley
<p><img src="/blog/2020/11/java-hotswap/zebras-scale-crush.jpg" alt="zebras"></p>
<p><a href="https://unsplash.com/photos/dgH8NSdEDv0">Photo</a> by <a href="https://unsplash.com/@valenciascott">Neil and Zulma Scott</a></p>
<p>For a recent Java development project I spent a while setting up an environment to take advantage of the HotSwap JVM, a Java virtual machine that automatically reloads classes when they change. This feature can potentially eliminate the need to redeploy each time code changes, reducing development cycle time considerably. While setting up the environment, I found I wanted a simple example of hot swapping available for my own experimentation, and I thought I’d share that example here.</p>
<p>First, let’s create a simple Java program. It needs to be slightly more complex than the ubiquitous “Hello, World!” application, because we need it to keep running for a while; if it just prints some message and exits immediately, we won’t have time to compile new code and see the hot swap feature in action. Here’s an example that uses a simple infinite loop, wherein it sleeps for one second, prints a message, and then repeats.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.lang.Thread</span>;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">HotSwapTest</span> {
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">main</span>(String[] args) {
<span style="color:#080;font-weight:bold">while</span> (<span style="color:#080;font-weight:bold">true</span>) {
<span style="color:#080;font-weight:bold">try</span> {
System.<span style="color:#369">out</span>.<span style="color:#369">println</span>(<span style="color:#d20;background-color:#fff0f0">"Hi"</span>);
Thread.<span style="color:#369">sleep</span>(1000);
} <span style="color:#080;font-weight:bold">catch</span> (InterruptedException e) {
<span style="color:#888">// Ignore this
</span><span style="color:#888"></span> }
}
}
}
~
</code></pre></div><p>If I build this into build/classes/java/main and run it, as expected it prints out “Hi” every second:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">josh@igtre:~/hotswaptest$ java -cp build/classes/java/main/ HotSwapTest
Hi
Hi
Hi
...
</code></pre></div><p>The usual JVM doesn’t include the HotSwap feature. For my purposes I downloaded <a href="https://dcevm.github.io/">DCEVM</a>, an alternative JVM which includes HotSwap. It’s also possible to patch some existing JVMs to add HotSwap, if you’d prefer. When I run the same code with DCEVM, it runs the code just like it did with the normal JVM, with additional debugging output:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:01:23.423 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:01:24.189 INFO (org.hotswap.agent.config.PluginRegistry) - Discovered plugins: [JdkPlugin, Hotswapper, WatchResources, ClassInitPlugin, AnonymousClassPatch, Hibernate, Hibernate3JPA, Hibernate3, Spring, Jersey1, Jersey2, Jetty, Tomcat, ZK, Logback, Log4j2, MyFaces, Mojarra, Omnifaces, ELResolver, WildFlyELResolver, OsgiEquinox, Owb, Proxy, WebObjects, Weld, JBossModules, ResteasyRegistry, Deltaspike, GlassFish, Vaadin, Wicket, CxfJAXRS, FreeMarker, Undertow, MyBatis]
</code></pre></div><p>To make hot swapping work automatically, we need to provide the JVM with a properties file in the JVM’s classpath. Mine looks like this, and lives in build/classes/java/main, next to the compiled class files:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">autoHotswap=true
LOGGER=debug
</code></pre></div><p>These properties are pretty self-explanatory: they tell the JVM to hot swap automatically when it finds new code, and turn up logging to DEBUG level.</p>
<p>So, with that all set up, let’s run the program again, change the code and rebuild it, and see what happens. For this test, I’ll just edit the message printed in each loop from “Hi” to “Hello”.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">josh@igtre:~/hotswaptest$ ./dcevm/bin/java -cp build/classes/java/main/ HotSwapTest
Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:36:22.132 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:36:22.543 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushIntrospectClassInfoCache(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.546 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
HOTSWAP AGENT: 15:36:22.549 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushObjectStreamCaches(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.550 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
...
</code></pre></div><p>The flurry of DEBUG messages tells me that it must have read my properties file correctly, and when I change the code and rebuild, I see the JVM respond with still more debug messages, saying it found and reloaded my code:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">HOTSWAP AGENT: 15:38:27.066 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.118 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.119 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.280 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
...
HOTSWAP AGENT: 15:38:27.439 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
Hi
HOTSWAP AGENT: 15:38:27.482 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Hi
Hi
Hi
</code></pre></div><p>But although it says it swapped in the new code successfully, it’s still printing “Hi”, not “Hello”. Why?</p>
<p>It turns out this ability to hot swap new code isn’t unlimited, and one important limitation is that methods you’re already running aren’t reloaded. Since the <code>main()</code> method was running, it didn’t get swapped out. What if I change the code so that instead of printing a string every second, it calls a method, and that method prints the string? Here’s some code to test that technique.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.lang.Thread</span>;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">HotSwapTest</span> {
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">printMsg</span>() {
System.<span style="color:#369">out</span>.<span style="color:#369">println</span>(<span style="color:#d20;background-color:#fff0f0">"Here is printMsg"</span>);
}
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">main</span>(String[] args) {
<span style="color:#080;font-weight:bold">while</span> (<span style="color:#080;font-weight:bold">true</span>) {
<span style="color:#080;font-weight:bold">try</span> {
HotSwapTest.<span style="color:#369">printMsg</span>();
Thread.<span style="color:#369">sleep</span>(1000);
} <span style="color:#080;font-weight:bold">catch</span> (InterruptedException e) {
<span style="color:#888">// Ignore this
</span><span style="color:#888"></span> }
}
}
}
</code></pre></div><p>Now, I start the JVM over again, and as expected, it prints “Here is printMsg” once every second. When I change to “Here is printMsg v2.0” and rebuild, this happens:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Here is printMsg
Here is printMsg
HOTSWAP AGENT: 15:48:34.923 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.007 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.008 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.174 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_CREATE
HOTSWAP AGENT: 15:48:35.188 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.175 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
HOTSWAP AGENT: 15:48:35.191 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.352 DEBUG (org.hotswap.agent.command.impl.SchedulerImpl) - Executing pluginManager.hotswap([class HotSwapTest])
HOTSWAP AGENT: 15:48:35.355 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [HotSwapTest] (autoHotswap)
HOTSWAP AGENT: 15:48:35.370 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from com.sun.beans.introspect.ClassInfo cache
HOTSWAP AGENT: 15:48:35.375 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from ObjectStreamClass caches
HOTSWAP AGENT: 15:48:35.376 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
HOTSWAP AGENT: 15:48:35.415 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Here is printMsg v2.0
Here is printMsg v2.0
</code></pre></div><p>As you can see, it swapped in the new code correctly, and now prints the new message. HotSwap was a success!</p>
<p>I imagine there are very few production environments where this feature would be applicable, and even in development, getting this to work properly for something like a JEE app deployed to some application container isn’t necessarily a simple task. But if it can cut down on redeployment cycles, it can certainly be a valuable developer tool.</p>
Rapid Test-Driven Development in Juliahttps://www.endpointdev.com/blog/2020/11/rapid-tdd-in-julia/2020-11-09T00:00:00+00:00Kamil Ciemniewski
<p><img src="/blog/2020/11/rapid-tdd-in-julia/automation.jpg" alt="Automation"></p>
<p>The Julia programming language has been rising in the ranks among the science-oriented programming languages lately. It has proven to be revolutionary in many ways. I’ve been watching its development for years now. It’s one of the most innovative of all the modern programming languages.</p>
<p>Julia’s design seems to be driven by two goals: to appeal to the scientific community and to achieve the best performance possible. This is an attempt to solve the “<a href="https://thebottomline.as.ucsb.edu/2018/10/julia-a-solution-to-the-two-language-programming-problem">two languages problem</a>” where data analysis and model building is performed using a slower interpreted language (like R or Python) while performance-critical parts are written in a faster language like C or C++.</p>
<p>The type-system is what allows Julia to meet its goals. The mix of strong and dynamic typing enables Python-like productivity with C++ or Rust-like performance. Julia is not an interpreted language. It compiles its code to native binary just like C, C++, Go, or Rust. The compilation and execution, though, are what sets it 1000 feet apart from all those other languages.</p>
<p>Here’s a simplified, brief outline of the steps in <a href="https://docs.julialang.org/en/v1/devdocs/eval/#Julia-Execution">Julia’s code execution model</a>:</p>
<ol>
<li>Julia process starts up.</li>
<li>Code is parsed.</li>
<li>For each code chunk:
<ul>
<li>If it hasn’t yet been compiled, decide whether to interpret or JIT compile it and then execute:
<ul>
<li>If compile then <strong>infer the types</strong> and <strong>use LLVM to produce native code</strong>.</li>
<li>Execute the newly-created native code.</li>
</ul>
</li>
<li>If it has been compiled, execute it.</li>
</ul>
</li>
<li>Repeat until the program ends or the user closes the REPL.</li>
</ol>
<p>It’s quite apparent that the compilation and type inference happen at a very different time compared to other compiled languages. Using Rust, you compile your code just once. Execution isn’t taxed by consecutive recompilation.</p>
<p>The result is quite a big negative surprise to Julia’s newcomers. Each time you run your app, there’s a significant slowdown before you see anything. It’s called the “time to first plot” issue. This is because, for example, a data scientist may want to generate some plots during her “exploratory data analysis”. Doing it in languages that are slower on paper — like R — makes those plots appear way quicker than in Julia.</p>
<h3 id="there-are-more-time-to-first-x-issues-in-julia">There are more “time to first X” issues in Julia</h3>
<p>Julia’s execution model makes more aspects trickier than just seeing the first plot. If you’re a software engineer who’s used to following the <a href="https://en.wikipedia.org/wiki/Test-driven_development">test-driven development</a> (TDD) approach, you’re in for a big surprise.</p>
<p>In languages like Ruby or Rust, it’s easy to have a tool watch for any file changes and respond by running the project’s testing suite. I often use the <a href="https://github.com/watchexec/watchexec">watchexec</a> tool which works with virtually any language, interpreter, or compiler. I run <code>watchexec -cw . "bundle exec rspec --fail-fast"</code> when working on a Ruby project, or <code>watchexec -cw . "cargo test"</code> with Rust.</p>
<p>With Julia this approach is not an option though — the “time to first test” is dramatically long. The wastefulness of continuous re-compilation steals my precious time, making me extremely unproductive.</p>
<h3 id="making-it-work-in-julia">Making it work in Julia</h3>
<p>The “time to first X” issue is only a problem if we’re closing the session in which our code has already been compiled. If we could move the file-watching and test-running steps all into the same session, the testing suite would run slowly only the first time. Julia’s standard library has built-in file watching functions that we could use to reproduce the <code>watchexec</code> in our code:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> using FileWatching
julia> watch_file
watch_file (generic function with 2 methods)
julia> watch_folder
watch_folder (generic function with 4 methods)
</code></pre></div><p>We can use those to get notified about the changes in our project’s files whenever they happen. Let’s imagine the following simple project’s structure:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ tree
.
└── src
├── App.jl
└── nested
└── Other.jl
<span style="color:#00d;font-weight:bold">2</span> directories, <span style="color:#00d;font-weight:bold">2</span> files
</code></pre></div><p>How do we watch for file changes in Julia? Let’s start up the REPL and see:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> using FileWatching
help?> watch_file
search: watch_file watch_folder unwatch_folder
watch_file(path::AbstractString, timeout_s::Real=-1)
Watch file or directory path for changes until a change occurs or timeout_s seconds have elapsed.
The returned value is an object with boolean fields changed, renamed, and timedout, giving the result of watching the file.
This behavior of this function varies slightly across platforms. See https://nodejs.org/api/fs.html#fs_caveats (https://nodejs.org/api/fs.html#fs_caveats) for more detailed information.
julia> watch_file("src")
</code></pre></div><p>The REPL didn’t return from the <code>watch_file</code> function.
We can now change the “src/App.jl” file and see what happens:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> watch_file("src")
FileWatching.FileEvent(true, true, false)
julia>
</code></pre></div><p>Good! The function returned a <code>FileEvent</code> struct. We can ask Julia for its definition:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">help?> FileWatching.FileEvent
No documentation found.
Summary
≡≡≡≡≡≡≡≡≡
struct FileWatching.FileEvent <: Any
Fields
≡≡≡≡≡≡≡≡
renamed :: Bool
changed :: Bool
timedout :: Bool
</code></pre></div><p>We can see it tells us whether the file’s been renamed, changed, or if the timeout happened.</p>
<p>So far so good, can we get it to notify us when the nested file changes too?</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> watch_file("src")
</code></pre></div><p>Now changing the “src/nested/Other.jl”:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> watch_file("src")
</code></pre></div><p>Nothing happened. We’ll need to be specific about the nested directory to make it work:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> watch_file("src/nested")
FileWatching.FileEvent(true, true, false)
</code></pre></div><p>With those experiments we can now conclude that:</p>
<ol>
<li>We’ll need to watch on all possible nested directories at the same time.</li>
<li>Watching blocks the current thread so for each folder to watch we need a separate thread.</li>
</ol>
<p>We’ll need a list of folders. My first idea was to use the <code>Glob</code> package:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> using Glob
julia> glob("**/*")
2-element Array{String,1}:
"src/App.jl"
"src/nested"
</code></pre></div><p>This seems legit but let’s nest another folder. Here’s how the project’s structure would look now:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ tree .
.
└── src
├── App.jl
└── nested
├── Other.jl
└── nested2
└── YetAnother.jl
<span style="color:#00d;font-weight:bold">3</span> directories, <span style="color:#00d;font-weight:bold">3</span> files
</code></pre></div><p>Trying the <code>Glob</code> package again:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> glob("**/*")
2-element Array{String,1}:
"src/App.jl"
"src/nested"
julia> glob("**/**/*")
2-element Array{String,1}:
"src/nested/Other.jl"
"src/nested/nested2"
</code></pre></div><p>Turns out that Julia’s <code>Glob</code> package doesn’t support extensions that allow “recursive” globbing. We’ll need to roll our own code to return all the possible nested folders:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> function subdirs(base="src")
ret = [base]
for (root, dirs, _) in walkdir(base)
fulldirs = map(d -> joinpath(root, d), dirs)
ret = vcat(vcat(vcat(map(subdirs, fulldirs)...), fulldirs), ret)
end
return ret |> unique
end
subdirs (generic function with 2 methods)
julia> subdirs("src")
3-element Array{Any,1}:
"src/nested/nested2"
"src/nested"
"src"
</code></pre></div><p>Being able to list all the nested directories, we can now work on the file-watching function. Here’s the plan of attack:</p>
<ol>
<li>Create a “channel” to receive the file changes from other threads watching each of those directories.</li>
<li>Spin up a new thread for working through the stream from the channel specifically.</li>
<li>Spin up threads for every nested directory found and watch for changes at the same time.</li>
<li>When the file change is detected, queue it into the channel.</li>
</ol>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">function onchange(f, basedirs=["src"])
channel = Channel()
function handle()
should_continue = true
for file in channel
try
f(file)
catch err
should_continue = typeof(err) != InterruptException
@warn("Error in the hanlder:\n$err")
end
end
end
function schedule(file)
put!(channel, file)
end
Threads.@spawn handle()
subs = vcat(map(basedir -> subdirs(basedir), basedirs)...)
@threads for dir in subs
should_continue = true
while true
(file, event) = watch_folder(dir, 1)
if event.changed
try
schedule(file)
catch err
should_continue = typeof(err) != InterruptException
@warn("Error in the scheduler:\n$err")
end
end
end
end
for dir in subs
unwatch_folder(dir)
end
end
</code></pre></div><p>Before we can run this code though, we need to mention one of other of Julia’s quirks. The <code>@threads</code> macro is cool and all, but it’s not going to work unless you start Julia with some predefined number of threads first:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#369">JULIA_NUM_THREADS</span>=<span style="color:#00d;font-weight:bold">4</span> julia
</code></pre></div><p>Let’s give it a go now:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> onchange(f -> println(f))
</code></pre></div><p>While the REPL is still “inside” the <code>onchange</code> function, let’s change some of those files in the dummy project and see what happens:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">julia> onchange(f -> println(f))
4913
App.jl
App.jl
4913
Other.jl
Other.jl
</code></pre></div><p>It works! The output is weird but we do get something here. For each file change, we get three messages here. After being puzzled for hours with how Julia implements this file watching I decided to just not mind it and add the throttling to make it work for my testing needs. The idea is that the throttling will only run the suite once per each of those triples.</p>
<p>Fortunately, the <code>Flux</code> package comes with the <code>throttle</code> function that we can reuse:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">function throttle(f, timeout; leading=true, trailing=false)
cooldown = true
later = nothing
result = nothing
function throttled(args...; kwargs...)
yield()
if cooldown
if leading
result = f(args...; kwargs...)
else
later = () -> f(args...; kwargs...)
end
cooldown = false
@async try
while (sleep(timeout); later != nothing)
later()
later = nothing
end
finally
cooldown = true
end
elseif trailing
later = () -> (result = f(args...; kwargs...))
end
return result
end
end
</code></pre></div><p>And the final version of our function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">function onchange(f, basedirs=["src", "test"], timeout=1)
channel = Channel()
function handle()
should_continue = true
for file in channel
try
f(file)
catch err
should_continue = typeof(err) != InterruptException
@warn("Error in the hanlder:\n$err")
end
end
end
function schedule(file)
put!(channel, file)
end
throttled_schedule = throttle(schedule, timeout)
Threads.@spawn handle()
subs = vcat(map(basedir -> subdirs(basedir), basedirs)...)
@threads for dir in subs
should_continue = true
while true
(file, event) = watch_folder(dir, 1)
if event.changed
try
throttled_schedule(file)
catch err
should_continue = typeof(err) != InterruptException
@warn("Error in the scheduler:\n$err")
end
end
end
end
for dir in subs
unwatch_folder(dir)
end
end
</code></pre></div><h3 id="putting-it-all-together">Putting it all together</h3>
<p>Armed with the helper <code>onchange</code> function we can now set up our nice auto-test runner. Let’s add the “test/runtests.jl” file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">using Test
function runtests()
@testset "the project" begin
include("test/test_one.jl")
include("test/test_two.jl")
end
nothing
end
function watchtest()
onchange(_ -> runtests())
end
</code></pre></div><p>Now, with the <code>watchtest</code> function running the whole testing suite re-runs whenever any of the project’s files changes.</p>
<h3 id="final-words">Final words</h3>
<p>I found it easy to have a love-hate relationship with Julia. I have all the respect for its creators. They’re doing an amazing job and are very bold with bringing in innovation. Once the code is compiled, it’s amazingly fast. The language’s ecosystem, along with amazing packages is one of its strongest points.</p>
<p>However, it’s awfully slow when you run your functions for the first time <strong>in the current session</strong>. Also, developers often have to rethink the workflows they’re so used to. This article touches on one of those issues.</p>
<p>The way the file watching is implemented in the standard library leaves a lot of room for improvement. As an example, I’m getting the “renamed” flag instead of “changed” in the <code>FileWatching.FileEvent</code> when I’m changing the file. That’s why in code I’m just checking for the absence of the timeout. It feels like a dirty hack but what can you do? The watcher was also not working consistently when the timeout was not given.</p>
<p>The immaturity of the standard library isn’t a show-stopper for many. Julia is developing rapidly and we can expect it to get better and better over time. Other issues will need engineers themselves to rethink their paradigms. I think it’s good though — challenges are what’s making us evolve after all and radical innovation doesn’t happen that often.</p>
The Pragmatic Programmer book, 20th anniversary editionhttps://www.endpointdev.com/blog/2020/10/the-pragmatic-programmer-20th-anniversary-edition/2020-10-16T00:00:00+00:00Jon Jensen
<p><img src="/blog/2020/10/the-pragmatic-programmer-20th-anniversary-edition/20201016_130906-sm.jpg" alt="Photo of the original and 20th anniversary editions of The Pragmatic Programmer book, atop lawn and fall leaves"></p>
<!-- Photo by Jon Jensen -->
<p>The Pragmatic Programmer is a now-classic book about software development, first published in 1999. It is old enough that it predates the famous <a href="https://agilemanifesto.org/">Agile Manifesto</a> of 2001. The authors of the book, Andy and Dave, were two of the 14 authors of that manifesto.</p>
<p>For its 20th anniversary in 2019, Dave and Andy created a new edition. They updated things that had become dated, such as mentions of programming languages, tools, operating systems, websites, etc. That was, I imagine, the easy part. They went on to extensively revise the entire text and incorporate new lessons learned in the past two decades.</p>
<h3 id="a-classic">A classic</h3>
<p>This book is part of our company’s <a href="/blog/2018/05/work-philosophy-canon/">“work philosophy canon”</a> that I ask every software developer at End Point to read, so for that reason and others I wanted to be familiar with the new edition and make sure it is still something I want to recommend so broadly.</p>
<p>The book is also required reading for university courses at Cornell (<a href="https://www.cs.cornell.edu/courses/cs3110/2020sp/reflections.html">CS 3110: Data Structures and Functional Programming</a>) and the University of Washington (<a href="https://courses.cs.washington.edu/courses/cse331/19au/quizzes.html">CSE 331: Software Design and Implementation</a>), probably among others!</p>
<p>I first read this book 19 years ago and really enjoyed it. I found that it clearly expressed many useful concepts I had stumbled upon in my own programming experience. More importantly, it gave me other helpful guidance and warned of pitfalls. Having that all collected in book form with concise, memorable chapter names and tips gave me more shared vocabulary to use with others in our work.</p>
<p>I also enjoyed reading the new 20th anniversary edition. It did not have the same personal impact, which is to be expected since I am at a very different point in my programming career, technical reading, and life experience. But I think the new edition has as great or even increased power for new readers.</p>
<h3 id="overview-of-changes">Overview of changes</h3>
<p>The authors estimate that around 75% of the text has changed, and that seems about right. A few areas that stood out to me were:</p>
<ul>
<li>
<p>The now widely-known adage “don’t repeat yourself” originally meant primarily not to duplicate knowledge, that each piece of data should appear in only one canonical location. Over time, “don’t repeat yourself” has been more often used to preach against copying and pasting code. While that is also a good general rule, it has been oversimplified and turned into a blanket prohibition. Dave & Andy mention cases where duplicating code is the right thing to do, because the problem space is not the same and factoring out all such code duplication will make later changes far more complicated as the code has to behave differently for different callers.</p>
</li>
<li>
<p>Computer security was not nearly as big of a concern when they were writing the first edition in the late 1990s, and now that almost every computer is networked and attackers are motivated to reach every system from anywhere in the world, they gave the topic more attention.</p>
</li>
<li>
<p>Concurrency gets much more discussion, as is sensible in a time where scaling up typically means using many more CPU cores and processors.</p>
</li>
<li>
<p>Unit testing was far less common in the late ’90s, and there were few good test frameworks, so they recommended you write your own. Now there are many good test frameworks for every language, so they of course recommend you choose one of those and use it.</p>
</li>
<li>
<p>Dave writes “A Confession” on page 223: He tells us that writing tests for 30+ years made testing so much a part of the way he thinks and programs, that he now writes testable interfaces even when he doesn’t write tests. He isn’t saying that testing is no longer important, but rather that it’s not a religion, that it can and should be examined by experienced programmers, and it has important effects on the programmer aside from the tests themselves.</p>
</li>
<li>
<p>The old edition had checklists scattered throughout, and collected in the pull-out card at the end. The new edition doesn’t have checklists anymore, except the Debugging Checklist on page 97. I think that’s ok. They weren’t something I ever referred back to.</p>
</li>
</ul>
<h3 id="tips">Tips</h3>
<p>The old edition had 70 tips that appear throughout the chapters. The new edition has 100 tips. In both cases they are extracted and collected on a pull-out card at the end of the book. What changed?</p>
<h4 id="removed-or-heavily-reworked-tips">Removed or heavily reworked tips</h4>
<ul>
<li>Write Code That Writes Code</li>
<li>Use Exceptions for Exceptional Problems</li>
<li>Design Using Services</li>
<li>Separate Views from Models</li>
<li>Don’t Use Wizard Code You Don’t Understand</li>
<li>Abstractions Live Longer than Details</li>
<li>Costly Tools Don’t Produce Better Designs</li>
</ul>
<h4 id="tips-that-changed-slightly">Tips that changed slightly</h4>
<ul>
<li>“Use a Single Editor Well” is now “Achieve Editor Fluency”</li>
<li>“Don’t Panic When Debugging” is now simply “Don’t Panic”</li>
<li>“Minimize Coupling Between Modules” is now “Decoupled Code Is Easier to Change”</li>
<li>“Configure, Don’t Integrate” is now “Parameterize Your App Using External Configuration”</li>
<li>“Don’t Gather Requirements—Dig for Them” is now “Requirements Are Learned in a Feedback Loop”</li>
<li>“Organize Teams Around Functionality” is now “Organize Fully Functional Teams”</li>
<li>“Gently Exceed Your Users’ Expectations” is now “Delight Users, Don’t Just Deliver Code”</li>
<li>“Put Abstractions in Code, Details in Metadata” is now “Policy Is Metadata”</li>
<li>“Always Design for Concurrency” is now “Random Failures Are Often Concurrency Issues”</li>
<li>“Listen to Nagging Doubts—Start When You’re Ready” is now “Listen to Your Inner Lizard”</li>
<li>“Don’t Be a Slave to Formal Methods” is now “Do What Works, Not What’s Fashionable”</li>
<li>“Some Things Are Better Done than Described” transformed to “Agile Is Not a Noun; Agile Is How You Do Things”</li>
</ul>
<h4 id="new-tips">New tips</h4>
<ul>
<li>You Have Agency</li>
<li>Good Design Is Easier to Change Than Bad Design</li>
<li>Forgo Following Fads</li>
<li>Failing Test Before Fixing Code</li>
<li>Read the Damn Error Message</li>
<li>Act Locally</li>
<li>Take Small Steps—Always</li>
<li>Avoid Fortune-Telling</li>
<li>Tell, Don’t Ask</li>
<li>Don’t Chain Method Calls</li>
<li>Avoid Global Data</li>
<li>If It’s Important Enough To Be Global, Wrap It in an API</li>
<li>Programming Is About Code, But Programs Are About Data</li>
<li>Don’t Hoard State; Pass It Around</li>
<li>Don’t Pay Inheritance Tax</li>
<li>Prefer Interfaces to Express Polymorphism</li>
<li>Delegate to Services: Has-A Trumps Is-A</li>
<li>Use Mixins to Share Functionality</li>
<li>Shared State Is Incorrect State</li>
<li>Use Actors For Concurrency Without Shared State</li>
<li>Testing Is Not About Finding Bugs</li>
<li>A Test Is the First User of Your Code</li>
<li>Build End-To-End, Not Top-Down or Bottom Up</li>
<li>Use Property-Based Tests to Validate Your Assumptions</li>
<li>Keep It Simple and Minimize Attack Surfaces</li>
<li>Apply Security Patches Quickly</li>
<li>Name Well; Rename When Needed</li>
<li>No One Knows Exactly What They Want</li>
<li>Programmers Help People Understand What They Want</li>
<li>Don’t Go into the Code Alone</li>
<li>Maintain Small Stable Teams</li>
<li>Schedule It to Make It Happen</li>
<li>Deliver When Users Need It</li>
<li>Use Version Control to Drive Builds, Tests, and Releases</li>
<li>First, Do No Harm</li>
<li>Don’t Enable Scumbags</li>
<li>It’s Your Life. Share it. Celebrate it. Build it. <strong>And have fun!</strong></li>
</ul>
<p>The new edition’s PDF version does not include the pull-out card with tips that comes with the printed book. That’s too bad, because the tips were what I most often referred to in the old edition after I finished reading it. At least Andy and Dave have published on their website a list of the 100 updated tips. It would be even nicer to have a PDF of the book’s pull-out card as they did for the old edition.</p>
<h3 id="upgrade">Upgrade?</h3>
<p>As you can see, there are many new tips. The text has been updated. It is still of excellent quality. So should you get the new edition?</p>
<p>I mentioned earlier two university courses that use the book. In both courses the professors allow students to use either the original 1999 version or the 2019 version. That seems like the right thing to do to me as well.</p>
<p>If you already have the old edition of the book, I don’t think it’s necessary to rush out and buy the new edition. The old one is still quite good and relevant, and you’ll be learning a bit of software development history by reading the older one.</p>
<p>If you do not yet have the book, or if you will be going over it with others who are new to it, it is probably best for you to get the new edition as well so you can be familiar with the new tips and the ways Dave & Andy describe things now.</p>
<h3 id="summing-up">Summing up</h3>
<p>I am glad that Dave and Andy updated their book. Their updates make sense and make the book more relevant to present-day readers.</p>
<p>Relatedly, it would be good to also see an update of the also excellent book <em>Practices of an Agile Developer</em> by Venkat Subramaniam and Andy. That is coming up on 15 years old itself. Perhaps some of its topics have been incorporated in the new edition of <em>The Pragmatic Programmer</em>, but as I review the table of contents, it looks like it is still mostly additive. I have cited its important maxim “Different Makes a Difference” so often that I thought it was from <em>The Pragmatic Programmer</em> instead!</p>
<p>I’ll close with this good observation and advice from Andy in their Changelog podcast interview:</p>
<blockquote>
<p>Was it Dijkstra who had the Turing Award lecture about the very humble programmer? That is a critical piece of early literature… And you talk about things that haven’t changed. This was 1972? … And he makes the very important point that complexity will overwhelm us if we don’t take a very humble, very measured approach. And it’s been 30–40 years and everyone—present company included—has ignored this wonderful advice.</p>
<p>Humility is difficult in our environment, in our culture, and it is probably, of all the human factorsy things that you need to be a good developer, I would submit that being humble, realizing you don’t know all the answers, that you need to find out, that you need to experiment, get feedback, try it.</p>
<p>This part of our headlong rush into the shiny new thing is this kind of faith that “Well, that’s gonna be better. I can do it better. I’m better than this/that.” Yeah, maybe… But you should validate that. You should try it. You should go back and read these things. You should try these other experiments.</p>
</blockquote>
<p>Thanks, Andy & Dave!</p>
<h3 id="reference">Reference</h3>
<h4 id="all-tips-from-each-edition">All tips from each edition</h4>
<ul>
<li><a href="https://blog.codinghorror.com/a-pragmatic-quick-reference/">Quick reference 70 tips (and checklists) from original edition</a></li>
<li><a href="https://pragprog.com/tips/">Quick reference 100 tips from new edition</a></li>
</ul>
<h4 id="interviews">Interviews</h4>
<ul>
<li><a href="https://www.functionalgeekery.com/episode-126-andy-hunt-and-dave-thomas/">Functional Geekery Episode 126 – Andy Hunt and Dave Thomas from July 2, 2019</a> by Steven Proctor</li>
<li><a href="https://changelog.com/podcast/352">The Changelog – Episode #352: The Pragmatic Programmers with Andy Hunt & Dave Thomas</a> by Adam Stacoviak & Jerod Santo</li>
<li><a href="https://www.codenewbie.org/podcast/why-you-should-read-the-new-edition-of-the-pragmatic-programmer">CodeNewbie Podcast season 9 episode 8 from August 26, 2019: Why you should read the new edition of the Pragmatic Programmer with Andy Hunt, Dave Thomas</a> by Saron Yitbarek (author of the new edition’s foreword)</li>
</ul>
<h4 id="other-comparisons-of-the-two-editions">Other comparisons of the two editions</h4>
<ul>
<li><a href="https://www.kevinhooke.com/2020/05/06/the-pragmatic-programmer-1st-edition-vs-20th-anniversary-edition-what-are-the-major-changes/">Sections added, changed significantly, and removed</a> by Kevin Hooke</li>
<li><a href="https://booksoncode.com/articles/pragmatic-programmer-comparison">The Pragmatic Programmer Book 2nd Edition Differences by Miranda Limonczenko</a> (very detailed, but covers only the first 3 chapters)</li>
</ul>
<h4 id="book-pages">Book pages</h4>
<ul>
<li><a href="https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/">The Pragmatic Programmer, 20th Anniversary Edition</a> book home page, errata, source code downloads, and links to buy</li>
<li><a href="https://pragprog.com/titles/pad/practices-of-an-agile-developer/">Practices of an Agile Developer</a> book home page, samples, pull-out card, and <a href="/blog/2006/06/review-practices-of-agile-developer/">review by Ethan Rowe</a></li>
</ul>
Jamstack Conf Virtual 2020: Thoughts & Highlightshttps://www.endpointdev.com/blog/2020/06/jamstack-conf-virtual-may-2020/2020-06-16T00:00:00+00:00Greg Davidson
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/conference.jpg" alt="Conference"></p>
<h3 id="welcome-to-jamstack-conf-virtual-2020">Welcome to Jamstack Conf Virtual 2020</h3>
<p>Last week I attended <a href="https://jamstackconf.com/2020/may/">Jamstack Conf Virtual 2020</a>. It had originally been slated to take place in London, UK but was later transformed into a virtual event in light of the COVID-19 pandemic. The conference began at 2pm London time (thankfully I double-checked this the night before!)—6am for those of us in the Pacific Time Zone.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Up early for <a href="https://twitter.com/hashtag/jamstackconf?src=hash&ref_src=twsrc%5Etfw">#jamstackconf</a> 😎☕️ <a href="https://t.co/ydjvrHWCZH">pic.twitter.com/ydjvrHWCZH</a></p>— Greg Davidson (@syncopated) <a href="https://twitter.com/syncopated/status/1265637434638778368?ref_src=twsrc%5Etfw">May 27, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Before getting too much further I wanted to mention that if you are not familiar with the Jamstack, You can read more about it at <a href="https://jamstack.org/">jamstack.org</a>.</p>
<p>To virtually participate in the conference we used an app called <a href="https://hopin.com/">Hopin</a>. I had not heard of it before but was impressed with how well it worked. There were over 3000 attendees from 130+ countries one of the times I checked. <a href="https://www.hawksworx.com/">Phil Hawksworth</a> was the Host/MC for the event and did a great job. There were virtual spaces for the stage, sessions, expo (vendors), and networking. If you opted to, the networking feature paired you with a random attendee for a video chat. I’m not sure what I expected going into it but I thought it was fun. I met a fellow developer from the Dominican Republic. The experience was very similar though more serendipitous than the hallway track or lunch line at an in-person conference.</p>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/phil-welcome.png" alt="Phil Hawksworth welcoming the attendees"></p>
<h3 id="keynote">Keynote</h3>
<p><a href="https://twitter.com/biilmann">Matt Biilmann</a> opened the conference with a keynote address about the challenges we face as a developer community trying to improve access to accurate, timely and locally relevant information to a global audience. Many billions of users with all kinds of devices and varying levels of connectivity. He moved on to share how Netlify is trying to enable developers to “build websites instead of infrastructure” and “ensure all the best practices become common practices” through features like git-based deployments, build plugins, and edge handlers (more on those later).</p>
<h3 id="state-of-the-jamstack-survey-results">State of the Jamstack Survey results</h3>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/laurie-voss-talk.png" alt="Laurie Voss reporting findings from the Jamstack Survey 2020"></p>
<p><a href="https://seldo.com/">Laurie Voss</a> <a href="https://slides.com/seldo/jamstack-survey-2020#/">walked us through</a> the results of the <a href="https://www.netlify.com/blog/2020/05/27/state-of-the-jamstack-survey-2020-first-results/">Jamstack Survey 2020</a>. There were some interesting findings and surprises. Later on I read <a href="https://seldo.com/posts/you-will-never-be-a-full-stack-developer/">Laurie’s post</a> (which Matt had mentioned in his talk) and found that very interesting as well.</p>
<h3 id="fireside-chat-with-harper-reed">Fireside chat with Harper Reed</h3>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/harper-phae-talk.png" alt="Frances Berriman interviewed Harper Reed - fireside chat style"></p>
<p><a href="https://fberriman.com/">Frances Berriman</a> chatted with <a href="https://harperreed.com/">Harper Reed</a> and asked him about his application of Jamstack principles from years ago when he led the technology team for Barack Obama’s election campaign. He described the need to “get far with very limited resources” and spoke about experimenting with serving HTML from Google App Engine and Amazon S3. Using pre-built HTML allowed them to scale very efficiently and he opted to use message queues rather than interacting with the database to keep things very quick for users.</p>
<p>Another benefit Harper noted was how quickly and easily new team members could be onboarded. Folks who knew HTML, CSS and JavaScript would be up to speed and productive in no time. He admitted it’s more complicated today 😀. Speed is another benefit—he just loves when he comes across a super-fast web site (often mostly static).</p>
<h3 id="lightning-launch-netlify-">Lightning Launch: Netlify ⚡</h3>
<p><a href="https://twitter.com/calavera">David Calavera</a> gave a demo of Netlify’s new Edge Handlers feature which lets developers add logic to their code at the edge (e.g. the CDN servers geographically closest to the user). He demonstrated how Edge Handlers make it possible to examine the headers of the request to tailor the response to that unique request. Check out the <a href="https://www.youtube.com/watch?v=D44n8YVb5iI">video of his talk</a> to watch him live code an example. I believe <a href="https://workers.cloudflare.com/">Cloudflare Workers</a> and Fastly’s <a href="https://www.fastly.com/products/edge-compute/serverless">Edge Compute</a> are operating in a similar problem space. I plan to look into each of these offerings more thoroughly in future.</p>
<h3 id="lightning-launch-prismic-">Lightning Launch: Prismic ⚡</h3>
<p><a href="https://twitter.com/RenaudBressand">Renaud Bressand</a> from Prismic demoed <a href="https://www.slicemachine.dev/">Slicemachine</a>—a new feature from Prismic which lets you combine <a href="https://nuxtjs.org/">nuxt</a>/Vue components with content managed in Prismic. It looked like a very compelling way of enabling better collaboration between developers and content creators.</p>
<h3 id="lightning-launch-redwoodjs-">Lightning Launch: RedwoodJS ⚡</h3>
<p><a href="https://tom.preston-werner.com/">Tom Preston-Werner</a> demoed his latest project <a href="https://redwoodjs.com/">RedwoodJS</a>. I had heard Tom speak about this on a few podcasts recently and it was nice to see him demo it for us. It looks interesting and feels reminiscent of <a href="https://rubyonrails.org/">Rails</a>. RedwoodJS simplifies wiring up your database to a <a href="https://graphql.org/">GraphQL</a> API (they use <a href="https://www.prisma.io/">Prisma</a> for this) and integrating it into a React application. Tom is also the creator of <a href="https://jekyllrb.com/">Jekyll</a>—a Jamstack-style tool which has been around for many years. It was nice to see several speakers give him some recognition for his work on that project.</p>
<h3 id="the-covid-tracking-project-0-to-2m-api-requests-in-3-months">The COVID Tracking Project: 0 to 2M API requests in 3 months</h3>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/kissane.png" alt="Erin Kissane presenting The COVID Tracking Project"></p>
<p><a href="http://incisive.nu/">Erin Kissane</a> gave an inspiring talk about her work on <a href="https://covidtracking.com">The COVID Tracking Project</a>. She described it as an “interim public data rescue project”. Erin and some friends, journalists and volunteers worked together to create the site. They started by scraping COVID data from each state’s web site and storing it in a Google Sheet. They used <a href="https://www.gatsbyjs.com/">Gatsby</a>, <a href="https://www.contentful.com/">Contentful</a>, and <a href="https://css-tricks.com/introducing-sass-modules/">Sass modules</a> and hosted the site on Netlify. Using the Jamstack approach allowed the site to remain performant and continue to be responsive under some huge traffic spikes. Over time, they iterated on the design and continue to improve the site daily. I highly recommend checking out the <a href="https://www.youtube.com/watch?v=ryngYoHXNfQ">video of the talk</a> when you get a chance.</p>
<h3 id="jamstack-for-emerging-markets">Jamstack for emerging markets</h3>
<p><a href="https://www.codebeast.dev/">Christian Nwamba</a> described some of the challenges of building sites for users in Nigeria (low power devices, spotty connectivity, unreliable power). He shared that 55% of the most visited sites in Nigeria are global companies (Google, Facebook/Instagram, Netflix, Stack Overflow etc.). Christian reviewed a large, popular banking site in Nigeria and noted its many shortcomings.</p>
<p>To demonstrate how the bank might do better he built <a href="https://proud-flower-060c1e01e.azurestaticapps.net/">an app</a> for transferring money between friends & family built in the Jamstack style and using serverless functions. The most interesting thing I picked up from this was his method of using serverless functions to protect the app secrets (API keys, etc.). The front end of the application did not need to concern itself with this—the serverless functions kept the secrets safe and acted as a proxy between the frontend and the backend APIs. Be sure to take a look at <a href="https://github.com/christiannwamba/quickbank">Christian’s code</a> if you are interested.</p>
<h3 id="managing-diabetes-with-jamstack">Managing diabetes with Jamstack</h3>
<p><a href="https://jamiebradley.dev/">Jamie Bradley</a> taught us about diabetes and the Jamstack app he built to help himself and others manage it. He built <a href="https://heysugar.health/">HeySugar</a> with Gatsby and Sanity, hosted it on Netlify. He’s making it easier for others to deploy their own instances as well.</p>
<h3 id="selling-tickets-without-servers-or-frameworks-or-cookies">Selling tickets without servers. Or frameworks. Or cookies.</h3>
<p><a href="https://jvhellemond.nl/">Jan van Hellemond</a> has volunteered for years at a very popular conference in Europe (<a href="https://fronteers.nl/congres">Fronteers</a> I think). In previous years tickets sold out in 6 minutes one year, and 30 seconds in another! Their ticket vendor was struggling to handle to the load and this caused them to oversell early bird tickets. Jan built a Jamstack site to sell their tickets and was very pleased with the performance. He used simple, single-purpose vanilla JavaScript components and simple, single-purpose API handlers (serverless functions).</p>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/tickets.png" alt="Jan presenting his Jamstack ticket selling app"></p>
<p>Jan prerendered as much as possible and seeded a relational database with the tickets available for sale. As the tickets were sold, they were marked sold with a database update. Webhooks were used as customers stepped through the checkout flow. Jan joked about deploying to production on a Friday afternoon because of how safe and simple the deployment process was. There were no services to restart, etc. because “it’s just files”. It was cool to see a practical example and that Jan used the basic building blocks of the web (HTML, JavaScript, CSS, old school links) successfully without reaching for a large JavaScript framework.</p>
<h3 id="the-business-side-of-the-jamstack">The business side of the Jamstack</h3>
<p><a href="https://twitter.com/_anarossetto_">Ana Rossetto</a> shared how her company has been having great success with the Jamstack. Previously, the agency had been building projects for clients with Drupal. She walked through a project she and the team built to encourage people to buy from small, locally-owned businesses. She was impressed with what they were able to create in a very short amount of time.</p>
<h3 id="build-plugin-authors-session">Build plugin authors’ session</h3>
<p>After the main talks there were several sessions. In the Hopin app, I was able to peek into some of these and the presenters and attendees were chatting (both text chat and video). This was very much like the experience of peeking into conference rooms at a physical venue and choosing whether to stay and participate or move on. After some wandering I chose to attend a session with three Netlify build plugin authors.</p>
<p><img src="/blog/2020/06/jamstack-conf-virtual-may-2020/subfont.png" alt="Peter telling us about subfont"></p>
<p><a href="https://mntr.dk/">Peter Müller</a> built <a href="https://github.com/Munter/netlify-plugin-subfont">Subfont</a> with a friend. He demonstrated how subsetting web fonts (i.e. only loading the characters & glyphs you really need) can dramatically improve frontend performance. He compared the <a href="https://webpagetest.org/">WebPageTest</a> results for Google Fonts with and without Subfont and Subfont was seconds faster! I have subset and locally hosted webfonts in several client projects here at End Point. It takes time and is a manual process. Peter’s plugin makes this excellent performance improvement relatively painless.</p>
<p><a href="https://darn.es/">David Darnes</a> demoed his build plugin to <a href="https://github.com/daviddarnes/netlify-plugin-ghost-markdown">turn Ghost content into Markdown files</a>. Very interesting and it seemed flexible enough to work with other tools as well.</p>
<p><a href="https://twitter.com/bahmutov">Gleb Bahmutov</a> presented the build plugin he created to run <a href="https://www.cypress.io/">Cypress</a> tests as part of your Netlify build process. It was amazing to see how simple it was (single devDependency and single line in the netlify.toml config file).</p>
<h3 id="videos-from-the-conference">Videos from the conference</h3>
<p>Netlify has already put all of the <a href="https://www.youtube.com/playlist?list=PL58Wk5g77lF8jzqp_1cViDf-WilJsAvqT">Jamstack Conf Virtual 2020 talks</a> on YouTube so head over there and check those out if you’d like to. Thanks very much to the team at Netlify for organizing the conf, all of the speakers, vendors and attendees!</p>
An Introduction to webpack 4: Setting Up a Modern, Modular JavaScript Front-End Applicationhttps://www.endpointdev.com/blog/2020/03/an-intro-to-webpack-4/2020-03-26T00:00:00+00:00Kevin Campusano
<p><img src="/blog/2020/03/an-intro-to-webpack-4/banner.png" alt="Banner"></p>
<p>Image taken from <a href="https://webpack.js.org/">webpack.js.org</a></p>
<p>I’ve got a confession to make: Even though I’ve developed many JavaScript-heavy, client side projects with complex build pipelines, I’ve always been somewhat confused by the engine that drives these pipelines under the hood: <a href="https://webpack.js.org">webpack</a>.</p>
<p>Up until now, when it came to setting up a build system for front-end development, I always deferred to some framework’s default setup or some recipes discovered after some Googling or StackOverflow-ing. I never really understood webpack at a level where I felt comfortable reading, understanding and modifying a config file.</p>
<p>This “learn enough to be effective” approach has served me well so far and it works great for being able to get something working, while also spending time efficiently. When everything works as it should, that is. This approach starts to fall apart when weird, more obscure issues pop up and you don’t know enough about the underlying system concepts to get a good idea of what could’ve gone wrong. Which can sometimes lead to frustrating Googling sessions accompanied with a healthy dose of trial and error. Ask me how I know…</p>
<p>Well, all that ends today. I’ve decided to go back to basics with webpack and learn about the underlying concepts, components and basic configuration. Spoiler alert: it’s all super simple stuff.</p>
<p>Let’s dive in.</p>
<h3 id="the-problem-that-webpack-solves">The problem that webpack solves</h3>
<p>webpack is a module bundler. That means that its main purpose is taking a bunch of disparate files and “bundling” them together into single, aggregated files. Why would we want to do this? Well, for one, to be able to write code that’s modular.</p>
<p>Writing modular code is not as easy in JavaScript that runs in a browser as it is in other languages or environments. Traditionally, the way to achieve good modularity in the web front-end has been via including separate scripts via multiple <code><script></code> tags within HTML files. This approach comes with its own host of problems. Things like the order in which the scripts are included suddenly matter, because the browser executes them top to bottom, which means that you have to be very careful to include them in an order where dependencies of the later files are included first. Also, this approach encourages the pollution of the global scope, where every script declares some global variables which are then used by other scripts. This is problematic because it is not clear which scripts depend on which ones. In other words, dependencies become implicit and hard to track. Unit testing becomes harder as well, since you need to mock these dependencies at a global scope. You also run the risk of some scripts overriding these global resources anytime. It’s just not very clean.</p>
<p>Over the years, solutions have been devised by the community to tackle this issue, which have taken the form of module loaders like <a href="http://www.commonjs.org/">CommonJS</a> and <a href="https://github.com/amdjs/amdjs-api/blob/master/AMD.md">AMD</a>. Finally, <a href="https://babeljs.io/docs/en/learn/">ES2015</a> introduced a native module loading system into the language itself via the <code>import</code> and <code>export</code> statements. The life of JavaScript in the browser is not so easy though, as these new specifications take time to implement by the various browser vendors. This is where webpack comes in. Part of its offerings is the ability to use this standard, modern module loading mechanism without having to worry about browser compatibility. This unlocks the potential for front-end developers to write modern, beautiful, modularized JavaScript. This is huge.</p>
<h3 id="concepts-entry-points-and-output">Concepts: Entry points and output</h3>
<p>Now, what does that look like in webpack terms? Well, we define an entry point, an output specification, and let webpack do its thing.</p>
<p>This is a good time to formally introduce two of webpack’s core concepts. First, we have entry points.</p>
<p>Entry points represent the starting point of execution for your app or page, and as such, serve as the starting file for webpack to begin building up your bundle. These are normally JavaScript files that bootstrap the execution of whatever application or page that you are developing. From a software design perspective, they tend to import many other modules and start up the scripts and instantiate the objects that actually run the application logic.</p>
<p>So, for example, say you have a file named <code>index.js</code>, which has some front-end logic and uses some utility classes living in separate files like <code>SomeServiceClass.js</code> or <code>ApiClient.js</code>. This <code>index.js</code> is a great candidate for being an entry point for your application or page because it is the one singular file that calls upon all of the other dependencies/modules.</p>
<p>We also have output. Output is the result of a webpack bundling operation. When webpack takes our entry points, it builds and compiles their corresponding dependencies and produces a single JavaScript file that can be directly included into our page or app via a <code><script></code> tag. This file is our output. This is the only file that needs to be included in the final page, because webpack took all the separate dependencies and bundled them together in one single package.</p>
<h3 id="introducing-our-demo-app-and-its-problems">Introducing our demo app and its problems</h3>
<p>But let me show rather than tell. Consider a simple calculator application, whose source code file structure looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">.
├── index.html
└── js
├── calculator
│ ├── calculator.js
│ └── operations
│ ├── addition.js
│ ├── division.js
│ ├── multiplication.js
│ ├── operation.js
│ └── subtraction.js
└── index.js
</code></pre></div><p>You can explore the source code for this small application <a href="https://github.com/megakevin/end-point-blog-webpack-intro/tree/master/original">here</a>. Feel free to go ahead and download it if you want to work along with me.</p>
<p>I’ve called the app the “Automatic Calculator” (not to be confused with its much less powerful alternative, the “manual” calculator!) and you will be able to figure out its general architecture pretty quickly.</p>
<p>In the root directory, we’ve got <code>index.html</code> which contains the GUI for our app. Then, all the behavior is inside the <code>js</code> directory.</p>
<p><code>index.html</code> is pretty straightforward. It’s got a simple form with two fields for typing in numbers, and a button that will automatically run a few arithmetic operations on those numbers. The results are presented right there in the page just a few pixels below the form.</p>
<p>For the purposes of this post, the interesting part of that file comes near the bottom, where we include all of our JavaScript logic. It looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/operations/operation.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/operations/addition.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/operations/subtraction.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/operations/multiplication.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/operations/division.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/calculator/calculator.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"js/index.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
</code></pre></div><p>As you can see, our little Automatic Calculator logic is separated into a series of files. And right now we start seeing some of the drawbacks of not using any sort of module loading for our app. This page has to include all of the script files that it needs separately. What’s worse, the order in which they are included matters. For example, since the <code>js/calculator/calculator.js</code> file depends on the <code>js/calculator/operations/multiplication.js</code> file to work, <code>multiplication.js</code> needs to be included first. Otherwise, the page will break. From this page’s perspective, it would be much easier and cleaner if it could just include one file, one “bundle” that has everything it needs.</p>
<p>If we look at our script files, we see more related problems. Consider <code>js/index.js</code>, for example. This is the file that starts up the app. It defines an App class which it then instantiates and runs. Here’s what it looks like (explore the source code in the git repo if you want to see the whole thing):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">class</span> App {
constructor() {
<span style="color:#080;font-weight:bold">this</span>.calculator = <span style="color:#080;font-weight:bold">new</span> Calculator();
<span style="color:#888">/* Omitted */</span>
}
run() {
<span style="color:#888">/* Omitted */</span>
}
}
console.log(<span style="color:#d20;background-color:#fff0f0">"Starting application"</span>);
<span style="color:#080;font-weight:bold">let</span> app = <span style="color:#080;font-weight:bold">new</span> App();
app.run();
console.log(<span style="color:#d20;background-color:#fff0f0">"Application started"</span>);
</code></pre></div><p>The <code>App</code> class’ constructor is creating a new instance of the <code>Calculator</code> class. The problem is that, from the point of view of the reader of this file, it’s doing so out of thin air. There’s no indication whatsoever that this file depends on and uses the <code>Calculator</code> class. It is available here only because the file that contains that class happens to be included in a <code><script></code> tag in the same page that is using <code>js/index.js</code>. That’s hard to read and maintain as the dependency is implicit in a place where it should be explicit. Imagine if <code>js/index.js</code> was a couple hundred lines bigger and had a few dozen more dependencies. That’d be very hard to manage. You would have to read the entire file to get an idea of how the code is structured. To be able to reason about the code from a higher level, you would have to go to the really low level. Too much cognitive overhead.</p>
<p>The same thing happens in the <code>js/calculator/calculator.js</code> file. It defines the <code>Calculator</code> class and that class depends on other classes (<code>Addition</code>, <code>Subtraction</code>, etc). Which are also defined in other JavaScript files but the <code>js/calculator/calculator.js</code> does not explicitly say that those are dependencies that it needs. They are called up out of thin air. Everyone is trusting in <code>index.html</code> to include all the separate script files where these classes are defined and that it does so in the proper order. Too much responsibility for little old <code>index.html</code>. And also, what would happen if somebody wanted to reuse the <code>Calculator</code> class in another page for instance? That developer would have to know all of the dependencies for that class and include them manually in the new page. What if the file were much bigger, with more dependencies? That can get really ugly really quickly.</p>
<p>Luckily for us, those are exactly the kinds of problems that webpack helps us deal with. So let’s start refactoring this little app so that it can take advantage of a webpack-based build process.</p>
<h3 id="installing-webpack-into-our-project">Installing webpack into our project</h3>
<blockquote>
<p>If you are following along and downloaded the source code from the GitHub repo, note that whenever I say “root”, I mean that repo’s <code>original</code> directory. That’s where the original version of the Automatic Calculator app’s source code lives as it is before introducing webpack. You can work directly from there or take the contents of that directory and put them wherever it is most comfortable to you. The <code>final</code> directory contains the source code as it will be by the end of this post.</p>
</blockquote>
<p>The first thing we need to do is install webpack into our project. That’s super easy if you already have <a href="https://nodejs.org/en/">Node.js</a> and <a href="https://www.npmjs.com/">NPM</a> installed. How to install Node.js and NPM is out of the scope of this discussion, so I would recommend following Node.js’s documentation to get them installed.</p>
<p>Once you have that, go to our project’s root and run <code>npm init -y</code>. This will create a <code>package.json</code> file with some default configuration. This effectively makes our code a proper Node.js project.</p>
<p>After that, installing webpack is as easy as going to our project’s root and running:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npm install webpack webpack-cli --save-dev
</code></pre></div><p>That will create a new <code>node_modules</code> directory and install both the <code>webpack</code> and <code>webpack-cli</code> packages as development dependencies. They are installed as dev dependencies because, in production, our app won’t actually need webpack to run properly. webpack will only help us build the deployment assets for our app, and that happens at development time.</p>
<h3 id="the-webpack-configuration-file">The webpack configuration file</h3>
<p>Now, we need to create a <code>webpack.config.js</code> file which tells webpack how to take our files and produce said deployment assets, i.e. the compiled bundle. Here’s what a simple config file tailored to our app would look like.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">const</span> path = require(<span style="color:#d20;background-color:#fff0f0">'path'</span>);
module.exports = {
mode: <span style="color:#d20;background-color:#fff0f0">"development"</span>,
entry: <span style="color:#d20;background-color:#fff0f0">"./src/index.js"</span>,
output: {
filename: <span style="color:#d20;background-color:#fff0f0">"index.js"</span>,
path: path.resolve(<span style="color:#00d;font-weight:bold">__</span>dirname, <span style="color:#d20;background-color:#fff0f0">"dist"</span>),
},
};
</code></pre></div><p>Let’s discuss it line by line.</p>
<p>First, there’s <code>const path = require('path');</code> which is just including the Node.js native <code>path</code> package that the config file uses further down below.</p>
<p>Then there’s the <code>module.exports</code> definition. Conveniently, webpack’s configuration values are organized within a JavaScript object. Setting <code>module.exports</code> equal to that object makes it available to other JavaScript files that <code>import</code> this file.</p>
<p>Now we have the actual webpack build settings. The <code>mode</code> field can be either <code>development</code>, <code>production</code>, or <code>none</code>. Not super important for us right now, but depending on what you set here, webpack will apply different optimizations to the bundles.</p>
<p>The <code>entry</code> field defines the entry point for the build process. As discussed before, this is the file that webpack will start from when figuring out the entire <a href="https://webpack.js.org/concepts/dependency-graph/">dependency graph</a>. That is, all of the files that specify that they need one another to work via <code>import</code> and <code>export</code> statements (more on that later). In our app, the dependency graph looks like this:</p>
<p><img src="/blog/2020/03/an-intro-to-webpack-4/dependency-graph.png" alt="Image"></p>
<p>In other words, <code>index.js</code> depends on <code>calculator.js</code>. <code>calculator.js</code> in turn depends on <code>addition.js</code>, <code>subtraction.js</code>, <code>multiplication.js</code>, and <code>division.js</code>. Finally, all of the latter four depend on <code>operation.js</code>. Here, we have specified <code>./src/index.js</code> as our entry point. But wait, our <code>index.js</code> file lives inside our <code>js</code> dir. What gives? We’ll change that soon, when we prepare our files to use ES2015 modules and have an overall more conventional organization, as far as webpack is concerned.</p>
<p>Finally, with the <code>output</code> field, we tell webpack what we want it to produce after bundling together all those files. In this case, we’ve configured it to produce a file called <code>index.js</code> inside the <code>dist</code> directory.</p>
<h3 id="refactoring-our-project-to-use-es2015-modules">Refactoring our project to use ES2015 modules</h3>
<p>Like I discussed before, webpack allows us to express our dependencies using the now-standard <code>import</code> and <code>export</code> statements, which are natively supported as of JavaScript’s language level ES2015. Let’s do that. This is super easy, and you will understand it immediately if you have used any other language that supports modules like C#, Java, Python, PHP… Yeah, pretty much any other language out there BUT JavaScript.</p>
<p>Anyway, we have to add this line at the beginning of <code>index.js</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">import</span> Calculator from <span style="color:#d20;background-color:#fff0f0">"./calculator/calculator.js"</span>;
</code></pre></div><p>Nice, this is an explicit dependency declaration for our <code>index.js</code> file. Now webpack (and readers of our code!) can know that this file uses the Calculator class. And like that, we go file by file adding dependencies. In <code>calculator.js</code>, we add:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">import</span> Addition from <span style="color:#d20;background-color:#fff0f0">"./operations/addition.js"</span>;
<span style="color:#080;font-weight:bold">import</span> Subtraction from <span style="color:#d20;background-color:#fff0f0">"./operations/subtraction.js"</span>;
<span style="color:#080;font-weight:bold">import</span> Multiplication from <span style="color:#d20;background-color:#fff0f0">"./operations/multiplication.js"</span>;
<span style="color:#080;font-weight:bold">import</span> Division from <span style="color:#d20;background-color:#fff0f0">"./operations/division.js"</span>;
</code></pre></div><p>In all of those four files, we add:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">import</span> Operation from <span style="color:#d20;background-color:#fff0f0">"./operation.js"</span>;
</code></pre></div><p>It’s all pretty self-explanatory. The syntax is <code>import <CLASS_NAME> from "<RELATIVE_FILE_PATH>";</code> where the path is relative to the location of the file in which we’re adding the <code>import</code> statement.</p>
<p>Now, <code>import</code> statements are for specifying which other files a given file needs to work. To allow the code elements that are defined within a file to be imported by others though, they need to be exported. We do that by annotating the code elements that we want to make available with the <code>export</code> statement. <code>export</code> basically specifies what parts of a module are available for others to use.</p>
<p>Our code is factored in a very simple way, where the only thing defined in a given file is a class. So, in order for all the <code>import</code> statements that we added to work, we just need to go file by file making changes like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#000;background-color:#fdd">-class Calculator {
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+export default class Calculator {
</span></code></pre></div><p>All we did was add the <code>export</code> and <code>default</code> keywords to the definition of the <code>Calculator</code> class in <code>calculator.js</code>. <code>export</code> makes it so we are allowed to import that class elsewhere, and <code>default</code> allows us to use the style of import that we used before: the <code>import Calculator from "./calculator/calculator.js";</code> one.</p>
<blockquote>
<p>By the way, yes, there are other styles of <code>import</code> and the <code>default</code> keyword is optional. To learn more about <code>import</code> and <code>export </code> statements and JavaScript modules in general, I’d recommend a few resources from MDN: <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules">modules</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import">import</a>, and <a href="https://developer.mozilla.org/en-US/docs/web/javascript/reference/statements/export">export</a>.</p>
</blockquote>
<p>Alright, so after adding <code>export default</code> to all of our class definitions (except for the <code>App</code> class defined in <code>index.js</code>, we don’t need to export it because we don’t import it anywhere) we have almost all the pieces of the puzzle ready for our webpack build to work.</p>
<p>The only thing left is to rename our <code>js</code> directory to <code>src</code> to match our webpack configuration. There’s no particular reason for using <code>src</code> instead of <code>js</code>, other than it being a good way to clearly separate the location of the source code versus the location of the compiled assets, Which will live in <code>dist</code>.</p>
<p>Anyway, with that rename done, you can go ahead and run</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npx webpack --config webpack.config.js
</code></pre></div><p>As a result of that command, you should see a new <code>dist</code> directory with a single <code>index.js</code> file in it. Just like we specified as the output in our webpack configuration file.</p>
<p>And that’s it! We have set up a webpack build process for our app. Don’t forget to change the <code>index.html</code> page so that it only includes this new compiled asset. That should look something like this, at the end of the <code><body></code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"dist/index.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
</code></pre></div><p>You should be able to open up the page in your browser and marvel at what we’ve accomplished:</p>
<p><img src="/blog/2020/03/an-intro-to-webpack-4/the-app.png" alt="Image"></p>
<h3 id="the-other-problem-that-webpack-solves">The other problem that webpack solves</h3>
<p>Ok, we’ve made a great accomplishment here. We’ve managed to leverage webpack to write modular JavaScript. The importance of this should not be understated. However, webpack offers us much more. Another problem that has historically plagued front-end web development is compatibility across multiple browsers. webpack helps us with that as well via loaders.</p>
<h3 id="concept-loaders">Concept: Loaders</h3>
<p>In a nutshell, loaders are components that webpack uses to transform source code files during its bundling process, in order to make them available to the app. They basically allow webpack to turn things other than JavaScript into modules that can be used by the application via <code>import</code>. By itself, webpack only knows how to process JavaScript and JSON files. Using loaders, however, webpack can also process other types of files like CSS and its various flavors like LESS or SASS, or even JavaScript derived languages like TypeScript or CoffeeScript.</p>
<h3 id="writing-modern-javascript-with-babel">Writing modern JavaScript with Babel</h3>
<p>Here’s where the browser compatibility solution that I alluded to earlier comes into play: <code>babel-loader</code>. This loader makes it so we can write bleeding edge JavaScript, with all the newest features of the language, without having to worry if our target browsers support them.</p>
<p><a href="https://babeljs.io/">Babel</a> is a compiler that takes code written with later versions of the language and turns it into backwards-compatible JavaScript that can run in older browsers. <code>babel-loader</code> is how Babel integrates with webpack. I like being able to take advantage of JavaScript’s latest features, so, for me, setting up Babel in any new project is a must. Luckily for us, with webpack, setting it up is easy.</p>
<blockquote>
<p>You can learn more about Babel and the latest JavaScript features <a href="https://babeljs.io/docs/en/">here</a> and <a href="https://babeljs.io/docs/en/learn">here</a>.</p>
</blockquote>
<p>Like most JavaScript packages, Babel is distributed as an NPM package. So let’s go ahead and install it with</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npm install --save-dev babel-loader @babel/core @babel/preset-env
</code></pre></div><p>This will install the core Babel package as well as the loader for webpack, and the <code>env</code> preset.</p>
<blockquote>
<p>The concept of a “preset” is actually a Babel-related one, not really having anything to do with webpack. You can learn more about them <a href="https://babeljs.io/docs/en/babel-preset-env">here</a>, but, for our purposes, suffice it to say that Babel presets are a specification of which features are available to use. There are many presets (i.e. feature sets) against which you can configure Babel to compile. <code>env</code> is just a very convenient one that provides support for all the latest language features.</p>
</blockquote>
<p>Again, we’re installing these as dev-only dependencies because they are not needed for the app at runtime, only at build time.</p>
<p>Now, we go to our <code>webpack.config.js</code> file and add these lines:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">module.exports = {
<span style="color:#888">// ...
</span><span style="color:#888"></span> module: {
rules: [
{ test: <span style="color:#080;background-color:#fff0ff">/\.js$/</span>, exclude: <span style="color:#080;background-color:#fff0ff">/node_modules/</span>, loader: <span style="color:#d20;background-color:#fff0f0">"babel-loader"</span> },
]
}
};
</code></pre></div><p>This is how we specify that we want <code>babel-loader</code> to transform our JavaScript files before webpack bundles them together. Inside <code>module.rules</code>, there’s an array of objects. Each of the elements of that array is a rule specifying on one hand which files it applies to, via the regular expression in the <code>test</code> field, and on the other hand which loader will be used to process them via the <code>loader</code> field. The <code>exclude</code> field makes sure that files under our <code>node_modules</code> directory are not affected by <code>babel-loader</code>. We don’t want to be transforming the packages we downloaded from NPM after all, those are ready to use as they are. We only want to transform our own code.</p>
<p>In summary, this rule tells webpack to “run all <code>.js</code> files by <code>babel-loader</code> except for those inside <code>node_modules</code>”. <code>babel-loader</code> will make sure to transform them into plain old JavaScript before giving them back to webpack for bundling.</p>
<p>Finally, Babel itself requires a little bit of configuration. So let’s give it what it needs by creating a <code>.babelrc</code> file in our project’s root with these contents:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript">{
<span style="color:#d20;background-color:#fff0f0">"presets"</span>: [<span style="color:#d20;background-color:#fff0f0">"@babel/preset-env"</span>]
}
</code></pre></div><p>Pretty self-explanatory. This configuration tells Babel to use the <code>env</code> preset when processing our files.</p>
<p>Now run <code>npx webpack --config webpack.config.js</code> and hope for the best. Just kidding! Everything should work like a charm and with that, we have just unlocked JavaScript’s full potential for our project. Open the page again and you will see nothing has changed. What we have gained is the ability to write modern JavaScript without having to worry about compatibility.</p>
<h3 id="bonus-multiple-entry-points">Bonus: Multiple entry points</h3>
<p>When building an <a href="https://en.wikipedia.org/wiki/Single-page_application">SPA</a>, a single entry point is appropriate. However, we often have apps with several pages, each one of which is a sort of advanced front-end application in its own right. From a webpack perspective, apps like that will have multiple entry points, one for each page. This is because each page has its own set of client side code that runs independently from the other pages. They may share common parts under the hood (via class or library reuse, for example) but the main, initial script is different.</p>
<p>Let’s consider our calculator. So far, that app has only one page. Imagine we want to add a new one. Let’s say, an admin control panel. To make that happen, let’s add new <code>admin.html</code> and <code>src/admin.js</code> files to the project. Their contents don’t matter. All I need is for them to exist in order to illustrate the multiple entry points capability.</p>
<p>As far as webpack is concerned, we can configure the build process to support that style of application by updating our <code>webpack.config.js</code> like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"> module.exports = {
mode: "development",
<span style="color:#000;background-color:#fdd">- entry: "./src/index.js",
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ entry : {
</span><span style="color:#000;background-color:#dfd">+ index: "./src/index.js",
</span><span style="color:#000;background-color:#dfd">+ admin: "./src/admin.js"
</span><span style="color:#000;background-color:#dfd">+ },
</span><span style="color:#000;background-color:#dfd"></span> output: {
<span style="color:#000;background-color:#fdd">- filename: "index.js",
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ filename: "[name].js",
</span><span style="color:#000;background-color:#dfd"></span> path: path.resolve(__dirname, "dist"),
},
}
</code></pre></div><p>As you can see, we’ve changed our config object’s <code>entry</code> field to include two separate entry points, one for our existing <code>index.js</code> entry point and another for the new <code>admin.js</code> one. We’ve also tweaked the output configuration; instead of a static name for the resulting output bundle, we use a pattern to describe them. In this case, <code>[name].js</code> makes it so we end up with two bundle files named after their corresponding entry points. The <code>[name]</code> part is where the magic happens. When creating the compiled bundle files, webpack knows to substitute that variable with the corresponding value from the entry point configuration.</p>
<p>Go ahead and run <code>npx webpack --config webpack.config.js</code> again and inspect the <code>dist</code> directory. You will see that we now have two bundles: <code>index.js</code> and <code>admin.js</code>.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">dist/
├── admin.js
└── index.js
</code></pre></div><p>The new <code>admin.js</code> file can be added to the new <code>admin.html</code> page like usual via a <code><script></code> tag, just like we did with <code>index.html</code>.</p>
<h3 id="bonus-css-can-also-be-a-module">Bonus: CSS can also be a module</h3>
<p>When discussing loaders, I mentioned the possibility of making things other than JavaScript behave like modules and be available to the application. Let’s demonstrate how we can make that happen.</p>
<p>First, let’s install the loaders that we need with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">npm install --save-dev css-loader style-loader
</code></pre></div><p>Then, create a new CSS file under <code>/css/index.css</code> with some fabulous styling:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css"><span style="color:#b06;font-weight:bold">body</span> {
<span style="color:#080;font-weight:bold">background-color</span>: <span style="color:#080;font-weight:bold">aquamarine</span>;
}
<span style="color:#b06;font-weight:bold">button</span>, <span style="color:#b06;font-weight:bold">input</span> {
<span style="color:#080;font-weight:bold">border</span>: <span style="color:#080;font-weight:bold">grey</span> <span style="color:#080;font-weight:bold">solid</span> <span style="color:#00d;font-weight:bold">1</span><span style="color:#888;font-weight:bold">px</span>;
<span style="color:#080;font-weight:bold">background-color</span>: <span style="color:#080;font-weight:bold">white</span>;
<span style="color:#080;font-weight:bold">padding</span>: <span style="color:#00d;font-weight:bold">5</span><span style="color:#888;font-weight:bold">px</span>;
}
<span style="color:#b06;font-weight:bold">input</span>:<span style="color:#555">hover</span>, <span style="color:#b06;font-weight:bold">button</span>:<span style="color:#555">hover</span> {
<span style="color:#080;font-weight:bold">background-color</span>: <span style="color:#080;font-weight:bold">teal</span>;
}
</code></pre></div><p>Now import it into our <code>index.js</code> file with a line like this near the top of the file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">import</span> <span style="color:#d20;background-color:#fff0f0">"../css/index.css"</span>;
</code></pre></div><p>And finally, let’s configure webpack so that it knows what to do when it finds this weird import css line. We do so by updating our <code>module.rules</code> config:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"> module: {
rules: [
{ test: /\.js$/, exclude: /node_modules/, loader: "babel-loader" },
<span style="color:#000;background-color:#dfd">+ { test: /\.css$/, use: ['style-loader', 'css-loader'] }
</span><span style="color:#000;background-color:#dfd"></span> ]
}
</code></pre></div><p>What we did here was add the new loaders to the webpack build pipeline. With this, it now knows how to handle CSS files and apply whatever styling rules are defined in that CSS to the page in question. Pretty neat trick, huh?</p>
<blockquote>
<p>These two particular loaders are doing more than meets the eye. You can learn more about them <a href="https://github.com/webpack-contrib/css-loader">here</a> and <a href="https://github.com/webpack-contrib/style-loader">here</a>.</p>
</blockquote>
<h3 id="further-reading-plugins">Further reading: Plugins</h3>
<p>There’s one core concept that we haven’t discussed yet: plugins. Plugins offer yet another avenue for customizing webpack’s behavior. I won’t go into too much detail on them here because I think that with the understanding we have gained on entry points, outputs, and loaders, we’ve added really powerful tools into our toolboxes. These tools will be enough in most cases, or at least allow us to get new projects up and running, but with more insight into what’s actually happening under webpack’s hood. If you are so inclined, you can learn more about them <a href="https://webpack.js.org/concepts/plugins/">here</a>. In a nutshell, plugins are for “doing anything else that a loader cannot do”, as webpack’s own documentation puts it.</p>
<p>~</p>
<p>And that’s it for now. Thanks for joining me in exploring webpack, a pretty neat piece of software that makes many a front-end developer’s life easier, mine included.</p>
Web Projects for a Rainy Dayhttps://www.endpointdev.com/blog/2020/03/web-projects-for-rainy-day/2020-03-25T00:00:00+00:00Elizabeth Garrett Christensen
<p><img src="/blog/2020/03/web-projects-for-rainy-day/image-0.jpg" alt="raindrops on a plant"></p>
<p><a href="https://www.flickr.com/photos/yellowstonenps/32984582893/">Image</a> by <a href="https://www.flickr.com/photos/yellowstonenps/">Yellowstone NPS on Flickr</a></p>
<p>With the COVID-19 quarantine disrupting life for many of us, I thought I’d put together a list of things you can do with your website on a rainy day. These are things to keep your business moving even if you’re at home and some of your projects are stuck waiting on things to reopen. If you’re looking for some useful things to do to fill your days over the next few months, this post is for you!</p>
<h3 id="major-version-updates">Major Version Updates</h3>
<p>Make a list of your entire stack, from OS to database to development frameworks. Note the current version and research the current supported versions. I find Wikipedia pages to be fairly reliable for this (e.g. <a href="https://en.wikipedia.org/wiki/CentOS">en.wikipedia.org/wiki/CentOS</a>). Ok, so what things need to be updated, or will need to be in the next year? Start on those now and use some downtime to get ahead of your updates.</p>
<h4 id="sample-of-a-clients-stack-review">Sample of a client’s stack review</h4>
<div class="table-scroll">
<table>
<thead>
<td>Software</td>
<td>Purpose</td>
<td>Our version</td>
<td>Release date</td>
<td>End of support</td>
<td>Next update</td>
<td>Newest version</td>
<td>Notes</td>
</thead>
<tr>
<td>CentOS</td>
<td>OS for e-commerce server</td>
<td>7</td>
<td>July 2014</td>
<td>June 2024</td>
<td>Not imminent</td>
<td>8</td>
<td><a href="https://wiki.centos.org/About/Product">https://wiki.centos.org/About/Product</a></td>
</tr>
<tr>
<td>Nginx</td>
<td>Web server</td>
<td>1.16.0</td>
<td>March 2020</td>
<td>Unclear</td>
<td>Not imminent</td>
<td>1.16.1</td>
<td><a href="https://nginx.org/">https://nginx.org/</a></td>
</tr>
<tr>
<td>PostgreSQL</td>
<td>Database server</td>
<td>9.5.20</td>
<td>January 2016</td>
<td>Feb 2020</td>
<td>Medium term, to version 11</td>
<td>12</td>
<td><a href="https://www.postgresql.org/support/versioning/">https://www.postgresql.org/support/versioning/</a></td>
</tr>
<tr>
<td>Rails</td>
<td>App framework for store</td>
<td>5.1</td>
<td>February 2017</td>
<td>Current</td>
<td>Long Term, to version 6</td>
<td>6</td>
<td><a href="https://rubygems.org/gems/rails/versions>https://rubygems.org/gems/rails/versions</a></td>
</tr>
<tr>
<td>Spree</td>
<td>Ecommerce and admin gem</td>
<td>3.3</td>
<td>April 2017</td>
<td>Current</td>
<td>Long Term, to version 4</td>
<td>4</td>
<td><a href="https://rubygems.org/gems/spree/versions">https://rubygems.org/gems/spree/versions</a></td>
</tr>
<tr>
<td>Elasticsearch</td>
<td>Search platform for product import/search</td>
<td>5.6.x</td>
<td>September 2017</td>
<td>March 2019</td>
<td>Immediate, to version 6.8</td>
<td>7.4</td>
<td><a href="https://www.elastic.co/support/eol">https://www.elastic.co/support/eol</a></td>
</tr>
<tr>
<td>WordPress</td>
<td>Info site</td>
<td>5.2.3</td>
<td>September 2019</td>
<td>Unclear</td>
<td>5.2.4 shipped recently</td>
<td>5.2</td>
<td><a href="https://codex.wordpress.org/Supported_Versions">https://codex.wordpress.org/Supported_Versions</a></td>
</tr>
</table>
</div>
<h3 id="content-cleanup--seo-review">Content Cleanup & SEO Review</h3>
<p>Everyone’s website gets cluttered with outdated content. Take a look at your pages, review, and update what needs to be changed. Pay attention to search engine optimization (SEO) concerns as you go through it. Make sure your content has headers, accurate keywords, and good meta-descriptions. Research SEO best practices if you need a refresher.</p>
<p>Nowadays, reducing repeated content has huge benefits for SEO so we recommend any content review includes a review of duplication. If you have a small site, you can go through your content and SEO manually. Larger projects can utilize tools on the market such as <a href="https://www.siteliner.com/">Siteliner</a> or <a href="https://wordpress.org/plugins/wp-optimize/">WPOptimize</a>.</p>
<p>While you’re taking a dive into content, don’t forget to review your Google Analytics and understand what content is being used and what isn’t. Google has added many new features to Analytics and Ads, so it’s a good idea to refresh yourself on the updated documentation and new features.</p>
<h3 id="reporting">Reporting</h3>
<p>A lot of clients with big ecommerce data sets or other applications that collect data benefit from a separate reporting or business analytics tool. A rainy day can be a good time to think about what reports you want on last year’s business, what data will help you plan for the future. End Point has worked with a few different reporting tools that easily add on to your database, like <a href="https://www.hitachivantara.com/en-us/products/data-management-analytics/pentaho.html">Pentaho</a> or <a href="https://www.jaspersoft.com/reporting-software">Jasper</a> and those can be really useful.</p>
<h3 id="documentation">Documentation</h3>
<p>I wouldn’t be a good project manager if I didn’t throw this one in the list. Documentation is so, so important, yet really we can always do more. End Point uses a few different tools, including wikis running <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> and Google Docs, for keeping track of project details. Now’s a good time to set up a nice documentation system or do a big review and make sure everything is updated and back in order. Maybe dream of a vacation you <em>might</em> be able to take when this is over and make sure everything’s ready for you to do that.</p>
<h3 id="disaster-recovery-tests">Disaster Recovery Tests</h3>
<p>For anyone with business-critical infrastructure, you need to ensure you know how to get everything back up and running in the case of a major failure, either with on-premises or cloud hosting. Now’s a good time to clarify with your hosting vendor things like: What are your backups like? What is your disaster recovery plan? What is the timeline for recovering the application in the event of a major failure? If you can, take time to do a simulation and make sure all the pieces are there if they need to be. Simply said, we also need to test our backups in order to ensure that they actually work.</p>
<h3 id="redesign">Redesign</h3>
<p>If you’ve been meaning to refresh your website, a rainy day is prime time to do it. Designers and developers are looking for projects and you’ll have extra time on your hands to oversee the process, spend time reviewing and testing, and get things done just the way you want.</p>
<h3 id="automated-testing">Automated Testing</h3>
<p>Good developers want an automated test suite as part of their application. Not all applications were built with this from the beginning and many didn’t have the budget or time to get it done. With extra time on your hands, this can be a great time to start building your test suite or to improve the coverage of your existing one.</p>
<p>Unit tests in particular are a good place to start. Unit tests are great not only because they help validate software correctness and protect against regressions, but also because they require a well-factored and modular system. This means that, while writing your unit tests, you will often be forced to go back to your application’s code base and refactor it to make it testable, to make it better. Investing in creating a solid unit test suite is a great bang for your buck. You can also look at implementing continuous integration—having a pipeline to let multiple developers deploy code throughout the day and configure your automated tests into the workflow.</p>
<h3 id="versioning--deployment-tools">Versioning & Deployment Tools</h3>
<p>When you’re cleaning house, take a look at your Git version control repository and make sure everything important is in there. We have a few projects that have a main project in Git, but sometimes the smaller projects and one-offs can go astray. This is a good time to get everything organized into one repository, or make sure external repositories are connected and integrated.</p>
<p>Automated DevOps deployment tools can also be nice to work on. Tools like Ansible and Chef can take a lot of time to set up and test, but they have some great time-saving and accuracy advantages down the line. Our in-house security experts also recommend tools like AIDE and OSSEC which automate monitoring file changes daily.</p>
<h3 id="security-audit-and-monitoring">Security Audit and Monitoring</h3>
<p>Taking a few days to review your personal security and that of your application is something you should do regularly and now’s a good time to plan for it. Charlie’s got <a href="/blog/2020/02/end-point-security-tips/">a great security post</a> that’s a good top-level review. For application security, End Point uses some tools for vulnerability scanning. We also have a checklist of basic security items that include password handling, PII data, and other common security holes. For certain projects/clients we must also take HIPAA or PCI DSS compliance into account. Also, don’t neglect to review your TLS status and ensure that web applications run on TLS 1.2 and are TLS 1.3 ready. This also may relate to the underlying operating systems—whether they are able to support the latest TLS version natively.</p>
<h3 id="optimization-and-performance">Optimization and Performance:</h3>
<p>Most of the time new features have higher priority than improving the performance of an existing system. It could be the right time to review core functionalities and list out the areas that need improvement in serving a better experience to customers by optimization. The areas can be focused on optimizing code, database queries, image size, data compression over network, adding cache, CDN, and so on. We’ve been moving quite a few clients to the <a href="https://www.cloudflare.com/">Cloudflare</a> DNS and CDN service and we’ve been really happy with it. Optimization work will definitely influence the customer retention rate which helps to increase profitability long term.</p>
<h3 id="refactoring">Refactoring</h3>
<p>Along the same lines as optimization, code refactoring can have long term gains in performance and ease of future development. Think of this like house cleaning. It is always easier to find any item in the house when things are arranged in an orderly manner. Similarly, the organized and clean code base will play a vital role in future code changes and development, helping to reduce the chances of unexpected bugs, save time making changes at one place and improving code readability. Disciplined refactoring delivers readable, reusable, non-redundant code. Refactoring can be applied to your databases and user interfaces as well.</p>
<p>Want to get started on some background projects for your website? <a href="/contact/">Talk to us today</a>.</p>
A Career Talk for 1st Gradershttps://www.endpointdev.com/blog/2019/12/1st-grade-career-talk/2019-12-06T00:00:00+00:00Steph Skardal
<p><img src="/blog/2019/12/1st-grade-career-talk/steph.jpg" alt="Steph giving a career talk">
Giving a Career Talk to 1st Graders</p>
<p>This week, I gave a career talk to my daughter’s 1st grade class and I talked about my job as a software engineer. I started with <a href="https://www.youtube.com/watch?v=Ct-lOOUqmyY">this video</a>, which depicts two kids explaining to their dad how to make a peanut butter sandwich (called the “Exact Instructions Challenge”), but he takes them very literally and acts as though he has no context on how to work with peanut butter, jelly, and bread. The video got some giggles!</p>
<p>After the video, I talked about how the video was similar to what I do: I give computers instructions, and like that silly dad, computers don’t know anything about what they are being told to do. I hope they understood the analogy!</p>
<h3 id="i-have-5-alexas">I have 5 Alexas!</h3>
<p>We talked about what a computer is and how we all have a lot of computers at our house (“I have 5 Alexas at my house!”, “I have a PS4!”, “I have a PS2!”), even some that can turn the lights on and off now. I didn’t show them code because I didn’t think it would mean much to them, but a couple of the kids in the class had worked on kid-friendly coding projects.</p>
<p>I talked a little bit about my education (1st graders aren’t quite sure what this “college” thing is), and how I work from home. We talked about how I problem-solve, just like they do — right before the talk started, their teacher asked them to problem-solve so that everyone could have a chair! I explained that sometimes I problem-solve and figure out what the fastest way to do something is, or what another solution might be, but it might last longer. We also talked about how communication (reading and writing) are important in my job, so they should keep working on that!</p>
<h3 id="qa">Q&A</h3>
<p>The best part of the talk was Q&A, because I was able to understand what was in their big kid brains! Here are some of the questions I fielded:</p>
<ul>
<li>Do you give computers instructions in 1s and 0s? How do you do that? How do computers understand 1s and 0s? <em>I tried to explain this a little bit by talking about different languages (a couple of the kids speak additional languages), but I think I failed on that front. :shrug:</em></li>
<li>Do you spend “like all the time on computers”? Or “all the time staring at a computer ALL DAY”?!!</li>
<li>Do you stop hackers trying to get into computers? How do you stop hackers? <em>I explained that I had to think about how hackers might try to work and ways to stop them, and made an analogy that it’s like locking the door to keep your little sister out of your room.</em></li>
<li>How many hours a day do you work? <em>I explained that right now I work half-time, but that before Astrid (my 1st grader in the class) was born, I worked 8 hours per day, like many of their parents might do now.</em></li>
<li>Do you find bugs in code? <em>I answered that we call it “debugging”, and they knew what that meant because they use that terminology in library! I also said I squashed bugs, and they thought that was hilarious.</em></li>
</ul>
<p>Overall, it was a fun experience! A couple of the kids at the end said that, “they still didn’t understand what [I do]”. I hope I added to their huge kid imaginations in answering questions about hackers and binary code and encouraged them to problem solve in the future!</p>
Reviewing the Codehttps://www.endpointdev.com/blog/2019/11/reviewing-the-code/2019-11-26T00:00:00+00:00Kürşat Kutlu Aydemir
<p><img src="/blog/2019/11/reviewing-the-code/review_magnifier.png" alt="Magnifying glass"></p>
<!-- Photo from https://pixabay.com/illustrations/magnifying-glass-search-to-find-1019982/ -->
<p>Code review is analysis of source code of a software project by reading the code itself or sometimes using automated tools to analyze it, and it is part of the Software Quality Assurance (SQA) activities which define the quality assurance of the whole software development lifecycle.</p>
<p>As we go through the flow of a code-review, we’ll understand what to assure by doing code review. Code review lifecycle covers the guidelines and the code review process itself.</p>
<h3 id="code-review-guidelines">Code review guidelines</h3>
<h4 id="preparing-for-code-review">Preparing for code review</h4>
<ul>
<li>Pick the right reviewers. The reviewer should be an experienced software architect. Sometimes, less experienced groups can join the review team. Communication is also important, since frequent round table code review meetings with the developers are not usually productive.</li>
<li>Let authors annotate the code before review indicating which files have precedence for review. Annotation by the developers to orient the reviewer so they know where changes have taken place and where to review is a good practice.</li>
<li>Request code review. Requesting code review just before testing is another good practice.</li>
</ul>
<h4 id="tips-for-reviewer-during-the-code-review">Tips for reviewer during the code review</h4>
<ul>
<li>Review small pieces of code at a time, for instance around 400 lines of code.</li>
<li>Review for around 60 minutes at a time.</li>
<li>Set goals and metrics.</li>
<li>Use <a href="https://en.wikipedia.org/wiki/SMART_criteria">SMART criteria</a>:</li>
</ul>
<blockquote>
<ul>
<li>Specific: target one area clearly and precisely.</li>
<li>Measurable: quantify progress toward success by using metrics.</li>
<li>Actionable: ready to start and possible to accomplish.</li>
<li>Relevant: connected to what’s being done, considered, and/or resourced.</li>
<li>Timely: occurring at a favorable or useful time and opportune.</li>
</ul>
</blockquote>
<ul>
<li>Measure inspection rate and defect rate.</li>
<li>Use checklists.</li>
</ul>
<h3 id="code-review-flow">Code review flow</h3>
<p>Regardless of what programming language is used to develop, code review should cover the following common areas:</p>
<ul>
<li>Maintainability</li>
<li>Reusability</li>
<li>Performance</li>
<li>Security</li>
<li>Bug analysis</li>
<li>Knowledge sharing</li>
</ul>
<p>Other practices can also be reviewed, like extensibility or OOP (Object Oriented Programming) software design patterns. These depend on possible future plans and the nature of the software being developed.</p>
<h3 id="an-example-checklist-for-java-projects">An Example Checklist for Java projects</h3>
<p>I’d like to share a checklist below that I used for a project recently. I didn’t add my comments on the checklist items. The comments should be added to each item and if there are important details in the code, they should be added in the detailed report indicating the code blocks with the comments.</p>
<h4 id="clean-code--general">Clean Code & General</h4>
<ul>
<li>Use solution-problem domain names</li>
<li>Classes should be small</li>
<li>Functions should be small</li>
<li>Do one thing</li>
<li>Don’t repeat yourself (avoid duplication)—reusability</li>
<li>Explain yourself in your code</li>
<li>Use exceptions rather than return codes, nulls</li>
<li>Configurable</li>
</ul>
<h4 id="security">Security</h4>
<ul>
<li>Minimize the accessibility of classes and members</li>
<li>Avoid excessive logs for unusual behavior</li>
<li>Release resources (streams, connections, etc.) in all cases</li>
<li>Validate inputs</li>
<li>Validate outputs</li>
<li>Make public static fields final</li>
<li>Authorize and authenticate</li>
</ul>
<h4 id="performance">Performance</h4>
<ul>
<li>Avoid excessive synchronization</li>
<li>Keep synchronized sections small</li>
<li>String concatenation</li>
<li>Avoid creating unnecessary objects</li>
</ul>
<h4 id="extensibility">Extensibility</h4>
<ul>
<li>Make it easy to add enhancements with minimum changes</li>
<li>Components should easily be replaceable</li>
</ul>
<h4 id="scalability">Scalability</h4>
<ul>
<li>Consider large user base and data</li>
<li>Can be deployed in distributed systems</li>
</ul>
<h4 id="a-few-possible-checklist-item-comments">A few possible checklist item comments:</h4>
<ul>
<li>For naming checklist items, the below code piece complies with the conventions as it uses the solution domains as packaging name.</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">package</span> <span style="color:#b06;font-weight:bold">com.google.search.common</span>;
</code></pre></div><ul>
<li>On the other hand, the following line would violate the Java package naming conventions since its letter capitalizations and domain usage are inconsistent.</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">package</span> <span style="color:#b06;font-weight:bold">COM.google_search.common</span>;
</code></pre></div><ul>
<li>
<p>Sometimes a very long line of code could be used which usually violates the readability standards.</p>
</li>
<li>
<p>Or sometimes a variable used in a class scope in a concurrently run singleton Java object violates thread safety and could be a security issue.</p>
</li>
</ul>
<h3 id="static-code-analysis">Static Code Analysis</h3>
<p>Reviewing is viewing the software by reviewer and the reviewers’ own comments and best practice notes will be used as the final output.</p>
<p>There are also static code analysis tools like PMD for Java which generally checks the code for bugs and clean code. This static code analysis, usually attached to the final code review report of the reviewer, is most helpful for an overview of the code.</p>
<p>Sometimes software development IDEs also provide best practices while writing the code. I like the PyCharm IDE’s smart clean code suggestions for Python, for example.</p>
<h3 id="automated-code-review">Automated Code Review</h3>
<p>Automated testing is subject to measurable reports and static code analysis. Static code analysis tools can be scheduled for code checking and reporting.</p>
<p>Testing the code as a preparation for code review is another process that can be automated and already part of the SDLC processes. Some overlooked issues, like releasing resources and concurrency issues, can be also checked by automated tests and can be reported as automated code review results.</p>
<h3 id="conclusion">Conclusion</h3>
<p>I have outlined the general code review cycle of a project.</p>
<p>Depending on the programming language and the architecture of the software to be developed there can be additional best practices like functional, non-functional, OOP design principles, etc.</p>
<p>Reviewers may extend or lessen their checklist items according to future plans, architecture or review request coverage.</p>
Winning with CSS Grid and Autoprefixerhttps://www.endpointdev.com/blog/2019/11/winning-with-css-grid-and-autoprefixer/2019-11-04T00:00:00+00:00Greg Davidson
<p><img src="/blog/2019/11/winning-with-css-grid-and-autoprefixer/grid.jpg" alt="Aerial view of St. Vitus Cathedral, Prague"></p>
<p>Photo by <a href="https://unsplash.com/photos/BWtyq5fn6Ng">Stijn te Strake</a> on <a href="https://unsplash.com/">Unsplash</a></p>
<h3 id="using-css-grid-today">Using CSS Grid Today</h3>
<p><a href="https://caniuse.com/#search=css%20grid">Support for CSS Grid</a> is excellent (over 90% globally! 💯). I have been using it in a variety of client projects for the past few years with great success. The combination of CSS Grid and Flexbox has made layout on the web much simpler, more fun and less frustrating compared to the <del>hacks</del> techniques which were used for CSS layout in the past. Definitely give it a try if you haven’t already done so.</p>
<p>IE10 and IE11 support an earlier version of the CSS Grid spec and with help from Autoprefixer we can reap the benefits of CSS Grid in those old browsers with a little bit of extra work. I’ve long been a huge fan and happy user of <a href="https://github.com/postcss/autoprefixer">Autoprefixer</a>. It’s a <a href="https://postcss.org/">PostCSS</a> plugin that examines your CSS and augments it with <a href="https://developer.mozilla.org/en-US/docs/Glossary/Vendor_Prefix">vendor prefixes</a> and alternative syntaxes. Although vendor prefixes (e.g. <code>-webkit</code>, <code>-moz</code>, <code>-o</code>, etc.) are much less common these days, there are a few still in use.</p>
<h3 id="css-grid-basics">CSS Grid Basics</h3>
<p>With CSS Grid the basic idea is to define the grid, i.e. the rows and columns that make it up, and then go about placing elements on the grid. Think of a website layout with a masthead, navigation, hero image, main content sidebar, and footer as an example. Your grid styles can specify where each of these elements should be displayed on the grid: how many rows or columns they span, etc. In addition to this method of explicitly placing elements on the grid, <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Auto-placement_in_CSS_Grid_Layout">auto-placement</a> places elements on the grid for you automatically.</p>
<h3 id="autoprefixer--css-grid">Autoprefixer + CSS Grid</h3>
<p>I recently learned how to use <a href="https://github.com/postcss/autoprefixer#control-comments">control comments in Autoprefixer</a> to get fine-grained control over the way it operates on and enhances my CSS Grid styles. Rather than specifying a single (global) behavior via config options, control comments let you tell Autoprefixer how you would like it to process individual blocks of CSS code. For the scenario where I was explicitly placing elements on my grid I used the following control comment in my CSS file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css"><span style="color:#888">/* autoprefixer grid: no-autoplace */</span>
</code></pre></div><p>This allowed me to lay out the elements and Autoprefixer generated the <code>-ms-grid</code> styles required to make my modern CSS Grid styles work in IE 10 and IE 11.</p>
<p>In another scenario I <em><strong>did</strong></em> want the auto-placement support and enabled it with the following control comment in a different block of the same CSS file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css"><span style="color:#888">/* autoprefixer grid: autoplace */</span>
</code></pre></div><p>This was for a group of uniformly sized tile elements that I wanted to lay out in a grid that adapted as the viewport size varied. E.g.: from 4 tiles per row, to 3 tiles, to 2 and finally to 1 tile per row for the smallest viewports.</p>
<p>In both cases I was pleasantly surprised that my CSS Grid styles worked in modern browsers <strong>and</strong> also in IE 11 and 10! I added some fallback styles using flexbox for those browsers which do not yet support CSS Grid to ensure the content was accessible there as well.</p>
<h3 id="test-your-autoprefixd-grid-styles">Test Your Autoprefix’d Grid Styles</h3>
<p>Please note the Autoprefixer docs warn its grid support may not work in <em>all</em> cases and must be tested. For my purposes this worked you great but make sure to test because your mileage may vary.</p>
<h3 id="learning-more">Learning More</h3>
<p>If you’d like to learn more about CSS Grid I would recommend checking out <a href="https://jensimmons.com/">Jen Simmons’</a> <a href="https://www.youtube.com/playlist?list=PLbSquHt1VCf0b43dfLKTrCriXdlZcmgoi">CSS Grid Basics</a> series on YouTube and the <a href="https://gridbyexample.com/">Grid by Example</a> site by <a href="https://rachelandrew.co.uk/">Rachel Andrew</a>. To learn more about Autoprefixer, check out <a href="https://github.com/postcss/autoprefixer">the docs</a> at GitHub or you can <a href="https://autoprefixer.github.io/">try it out interactively</a>.</p>
Project On-ramping: Infrastructure and Codebasehttps://www.endpointdev.com/blog/2019/10/codebase-on-ramp/2019-10-21T00:00:00+00:00Steph Skardal
<p><img src="/blog/2019/10/codebase-on-ramp/nasa.jpg" alt="Aerial view of New York City from satellite">
Photo by NASA via Unsplash</p>
<p>Living in the consulting world, we often jump into codebases and environments to get quickly up to speed to provide estimates or start fixing bugs right away. Here are some important landmarks I visit along my way to get up to speed on a new project.</p>
<h3 id="dev-stack">Dev stack</h3>
<p>First up, I learn as much as I can about the dev stack a codebase runs on. It used to be the case that LAMP (Linux, Apache, MySQL, PHP) was a pretty common setup in our world, but the number of dev tools and variety in stacks has expanded greatly over time. Many of my End Point clients are now running nginx on Linux, with Ruby on Rails and MySQL or PostgreSQL, but as a company we have both breadth and depth in web stack technologies (hover over the Expertise dropdown above).</p>
<p>Layered on top of the base infrastructure might be a JavaScript framework (e.g. <a href="https://reactjs.org/">React</a>, <a href="https://emberjs.com/">Ember.js</a>), and other abstractions to improve the dev process (e.g. <a href="https://sass-lang.com/">Sass</a>, <a href="https://nodejs.org/en/">Node.js</a>). And layered on top of that further is the possibility of other services running on the server locally (e.g. search using <a href="https://www.elastic.co/">Elasticsearch</a> or <a href="https://lucene.apache.org/solr/">Solr</a>) and 3rd-party tools running on a server elsewhere (e.g. content delivery networks, monitoring, <a href="https://stripe.com/">Stripe</a>).</p>
<p>The web stack and 3rd-party tools can be so complex these days that there is a lot of ground to cover before jumping into the code.</p>
<h3 id="personnel-infrastructure">Personnel infrastructure</h3>
<p>Next up, I like to understand how our client works with the website and more about their limitations (time, technical, other). Especially in the consulting space where we are supporting existing developers, I think it’s important to exercise empathy over how they’ve gotten to the point they are at, and why they’ve reached out for additional development resources.</p>
<p>Does the client want a solution that enables them to make their own changes moving forward, i.e., a content management solution? Do they want to hand the project off entirely due to time limitations? Is there a hybrid solution that will enable them to make changes and defer to us as a resource for bigger decisions? Is the existing development staff who we should be best supporting and advising on decisions?</p>
<h3 id="are-there-dev-instances">Are there dev instances?</h3>
<p>After we have a high-level understanding of the dev stack and personnel involved, the next bridge we cross is to determine dev instance architecture (and how to get access to a dev instance!). It goes without saying (though I’m saying it anyways) that it’s a much lower risk to experiment and play around on a dev instance rather than a production server.</p>
<p>Here at End Point, we have our own in-house tool for spinning up dev instances (<a href="https://www.devcamps.org/">DevCamps</a>), and we also work with clients who have their own setup for dev environments (e.g. <a href="https://www.docker.com/">Docker</a>, <a href="https://aws.amazon.com/">AWS</a>) that we can jump into.</p>
<h3 id="codebase-where-do-i-go">Codebase: Where do I go?</h3>
<p>I like to equate learning a new code base to moving to a new city. When you first arrive, you don’t know what you are looking at or where to go, but as you begin to make frequent trips to one location or another, you become familiar with those paths and you become most familiar with the paths that you take most often.</p>
<p>One of my first stops is the database. I review the tables and understand how the application models interface with them, as well as how they reference each other. Another stop (in the Rails world) is to visit <code>config/routes.rb</code>, which <a href="https://guides.rubyonrails.org/routing.html">maps URLs and dispatches them to a Rails controller</a>. This immediately informs me of the complexity of the application by the number of different URL requests it handles.</p>
<p>A third favorite stop upon gaining access to the codebase is the code history (hopefully there is one!), which can be telling about the organization as well as frequent code destinations visited.</p>
<p>Another place to visit is the (again, in the Rails space) <a href="https://bundler.io/gemfile.html">Bundler Gemfile</a>, which tells me how many (or how few) dependencies are included in a project.</p>
<p>While I don’t think it’s important to get into the nitty gritty of Rails models, controllers, or views at a first pass, my observations of the database, routing, Git history, and Gemfile can paint a picture of the complexity and dependencies of one project. Outside of the Rails space, many modern web frameworks have comparable code paradigm that I think are similarly revealing.</p>
<h3 id="digging-deeper">Digging deeper</h3>
<p>From there, I look to client priorities for where to dig deeper into the code. I have found that in some cases in my consulting experience, the client priorities and convention use in codebase (e.g. with Ruby on Rails) overlaps nicely with my skill set and I’m able to jump right in to make changes, but other client work requires further research into third parties or historical context of the code and changes needed.</p>