https://www.endpointdev.com/blog/tags/api/2023-10-21T00:00:00+00:00End Point DevJSforce: A Quick Path to Salesforce Developmenthttps://www.endpointdev.com/blog/2023/10/jsforce-quick-path-to-salesforce-development/2023-10-21T00:00:00+00:00Couragyn Chretien
<p><img src="/blog/2023/10/jsforce-quick-path-to-salesforce-development/desert-sky.webp" alt="A completely clear blue sky is broken by a desert mountain with exposed light rock, covered partly by striking green trees and bushes. Above the mountain is a half-moon."></p>
<!-- Photo by Seth Jensen, 2023. -->
<p>Using JavaScript with JSforce can get you working on a Salesforce project quickly if you don’t have a Salesforce expert on hand. It provides easy access to Salesforce’s API, which will allow you to focus on development instead of learning a new system.</p>
<h3 id="no-salesforce-learning-curve">No Salesforce learning curve</h3>
<p>Apex is a platform-specific language created so that developers can interact with Salesforce classes/objects and write custom code. Apex allows you to do some cool things such as directly triggering custom Apex code based on an action in Salesforce.</p>
<p>The problem with Apex is that it is its own world, with its own IDEs, deployment processes, etc. There’s a steep learning curve to getting up to speed with the Apex ecosphere.</p>
<p>JSforce is a wrapper/abstraction of the Salesforce API. It allows you to do a lot, like search, perform CRUD actions, and even send emails. These functions aren’t as streamlined as their built-in Apex counterpart, but JSforce allows any JS developer to jump right into the code without wasting costly training time.</p>
<h3 id="using-jsforce-cli">Using JSforce CLI</h3>
<p>Below are some examples of connecting and performing basic CRUD operations.</p>
<h4 id="connecting">Connecting</h4>
<p>Installation:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ npm install jsforce -g
</code></pre></div><p>Connection:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ jsforce
> login('user@example.org', 'password123');
{ id: '00550000000vwsFAAQ',
organizationId: '00D500000006xKGEAY',
url: 'https://login.salesforce.com/id/00D500000006xKGEAY/00550000000vwsFAAQ' }
>
</code></pre></div><h4 id="get">GET</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js">conn.sobject(<span style="color:#d20;background-color:#fff0f0">"Account"</span>).retrieve(<span style="color:#d20;background-color:#fff0f0">"0017000000hOMChAAO"</span>, <span style="color:#080;font-weight:bold">function</span>(err, account) {
<span style="color:#080;font-weight:bold">if</span> (err) { <span style="color:#080;font-weight:bold">return</span> console.error(err); }
console.log(<span style="color:#d20;background-color:#fff0f0">"Name : "</span> + account.Name);
<span style="color:#888">// ...
</span><span style="color:#888"></span>});
</code></pre></div><h4 id="post">POST</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js">conn.sobject(<span style="color:#d20;background-color:#fff0f0">"Account"</span>).create({ Name : <span style="color:#d20;background-color:#fff0f0">'My Account #1'</span> }, <span style="color:#080;font-weight:bold">function</span>(err, ret) {
<span style="color:#080;font-weight:bold">if</span> (err || !ret.success) { <span style="color:#080;font-weight:bold">return</span> console.error(err, ret); }
console.log(<span style="color:#d20;background-color:#fff0f0">"Created record id : "</span> + ret.id);
<span style="color:#888">// ...
</span><span style="color:#888"></span>});
</code></pre></div><h4 id="put">PUT</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js">conn.sobject(<span style="color:#d20;background-color:#fff0f0">"Account"</span>).update({
Id : <span style="color:#d20;background-color:#fff0f0">'0017000000hOMChAAO'</span>,
Name : <span style="color:#d20;background-color:#fff0f0">'Updated Account #1'</span>
}, <span style="color:#080;font-weight:bold">function</span>(err, ret) {
<span style="color:#080;font-weight:bold">if</span> (err || !ret.success) { <span style="color:#080;font-weight:bold">return</span> console.error(err, ret); }
console.log(<span style="color:#d20;background-color:#fff0f0">'Updated Successfully : '</span> + ret.id);
<span style="color:#888">// ...
</span><span style="color:#888"></span>});
</code></pre></div><h4 id="delete">DELETE</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js">conn.sobject(<span style="color:#d20;background-color:#fff0f0">"Account"</span>).destroy(<span style="color:#d20;background-color:#fff0f0">'0017000000hOMChAAO'</span>, <span style="color:#080;font-weight:bold">function</span>(err, ret) {
<span style="color:#080;font-weight:bold">if</span> (err || !ret.success) { <span style="color:#080;font-weight:bold">return</span> console.error(err, ret); }
console.log(<span style="color:#d20;background-color:#fff0f0">'Deleted Successfully : '</span> + ret.id);
});
</code></pre></div><p>For a deeper dive into setup and uses of JSforce check out <a href="/blog/2020/03/salesforce-integration-with-node/">this post</a> by my coworker Dylan Wooters.</p>
Middleware: Is that still a thing?https://www.endpointdev.com/blog/2022/05/middleware-is-that-still-a-thing/2022-05-26T00:00:00+00:00Richard Templet
<p><img src="/blog/2022/05/middleware-is-that-still-a-thing/20220401-013133.webp" alt="Photo looking from ground level up at concrete and crushed stone building against a blue sky with some white clouds"></p>
<!-- photo by Jon Jensen -->
<p>The simple answer to the question in the title is simply, yes! Despite the term being many decades old and well past its hype peak, middleware is still very much a thing and has become a key part of the technical landscape that is critical in day-to-day functioning of systems.</p>
<p>So there are still some questions to be answered: What is middleware? What does it do? And maybe most importantly: Why do we care?</p>
<h3 id="what-is-it">What is it?</h3>
<p>In its simplest meaning, middleware is an application that sits between other applications and shuffles data between them. There is normally one system requesting the information and the middleware figures out where to get that requested data and makes the request to another system.</p>
<p>An easy example of this is buying something at a retail store using your credit or debit card. When you swipe your card, the business makes a request to a service (some middleware) to ask if there’s enough room on the card for that purchase. Then that system makes a request to the appropriate bank or card holding company to ask the same question. The bank or card holding company replies with either a yes or no and that answer is then relayed back to the terminal where you swiped your card.</p>
<p>You might be thinking, “Why do we need the middleware? Just have the terminal ask the right bank!” Well, that’s a benefit of middleware: The terminal software doesn’t need to have all the same logic coded into it to know whether or how to talk to Visa or Mastercard as well as Citibank or Capital One or Bank of America to get the answer. It just talks to the middleware and lets it figure all of that out. This way you can have many different types of terminals or credit card taking applications available but only one middleware application needs to have the smarts and authorization to know where and how to get the answer.</p>
<p>The term middleware is most often used to describe business systems internal to a company. It is a subset of the broader concept “API” (application programming interface), which includes services provided for external users, either the general public or specific authorized customers or business partners. If a service is used directly by humans of the general public, it is typically called SaaS (software as a service).</p>
<h3 id="what-does-it-do">What does it do?</h3>
<p>As I explained in the example above, the simplistic definition of middleware is an application that shuffles data between two systems. In reality, middleware is sometimes way more complicated than that!</p>
<p>In that example of a credit card charge terminal, using middleware also allows for the retailer to set centralized company-wide rules for fraud screening, add or remove supported payment methods, or change which payment gateway is used, without having to update each terminal individually.</p>
<p>Middleware also can do data manipulation and mapping between two systems. Let’s consider a more complicated case like booking a cross-country flight.</p>
<p>You’ve probably used sites like Google Flights or Expedia to search for ticket costs. You go to their site, put in your starting airport(s) and your destination airport(s), select the dates when you’d like to travel, and maybe check the box saying you’ve got some wiggle room on when you leave and come back. After that you click the search button and within a few seconds, you’ve got tons of flight options to sort through! The search results can be sorted by price, number of stops, and airlines, to name a few.</p>
<p>So taking a step back to marvel at what just happened, how do you think they got that much data from that many sources so quickly? Well, the answer is middleware.</p>
<p>At some point, a middleware system made a request to each of the airlines to ask for their available flights with their dates, costs, amenities, etc. and stored that information on their own system to be used for your search. There are many different ways that data could be captured. We won’t delve into all that excitement, but what we do know is some system (middleware) had to fetch this data from each of the airlines and store it to be used for your search. It has to organize and store that data in a way that makes sense to the search engine to use so you can get back your search results in a speedy fashion.</p>
<p>There’s a good chance that the data from American Airlines will differ in its structure and data points compared to Delta. It’s the middleware’s job to take each of those data feeds and organize and store them somewhere for the search engine to use. In this part of the example, the middleware is taking a request to fetch the Delta flight data, turning that request into a request to another system for the data then doing whatever data manipulation and storage of that data to satisfy the search engine.</p>
<h3 id="why-do-we-care">Why do we care?</h3>
<p>The main reason that we care is we’ve become used to having a single location (website, application etc.) that allows us access to data from multiple places.</p>
<p>In our credit card example, there are many different types of credit cards we can use to check out at our local gas station. Having them require us to only use a Shell gas card would be bad for their business! We’ve grown accustomed to being able to use our Visa, Mastercard, American Express, or Shell gas card at that same gas station. This is all made possible due to middleware behind the scenes doing the hard work.</p>
<p>In many cases, the function of gathering data from many sources in one place, keeping it current, and allowing it to be searched, sorted, etc. is not just one means to an end, but it actually <em>is</em> the whole business! It is why services like Google Flights and Expedia exist.</p>
<p>Middleware can serve many other purposes even when only used internally by a single company:</p>
<ul>
<li><strong>authentication</strong> — making sure who- or whatever made a request is supposed to be able to</li>
<li><strong>authorization</strong> — making sure the kind of request being made is one this user is allowed to</li>
<li><strong>auditing</strong> — logging or sampling traffic in middleware is often more comprehensive and convenient than at each service individually</li>
<li><strong>rate-limiting</strong> — ensuring the user is not using too many resources too fast, or that one service isn’t overused and interfering with access to other services</li>
<li><strong>caching</strong> — storing answers to frequent requests to reuse for a limited time, reducing the number of live requests to busy internal applications and databases</li>
<li><strong>billing</strong> — tracking usage so it can be paid for or at least each user’s usage is fairly recorded</li>
<li><strong>fault-tolerance</strong> — retrying a request if an internal system is temporarily unavailable, or rerouting requests to a different system, to shield users from outages</li>
<li><strong>load balancing</strong> — spreading requests across multiple services to allow greater total throughput</li>
<li><strong>translation</strong> — modern APIs typically use JSON to structure their data, and middleware can translate between a nice JSON interface and older systems using a variety of more complex and archaic data formats</li>
<li><strong>migrations</strong> — transparently moving from an old system to a newer one, possibly in phases or temporarily for testing, without users having to change anything or even be aware</li>
</ul>
<p>In closing, while middleware isn’t commonly mentioned or even thought about, it is a very important part of what makes the technological world function.</p>
Using Postman to Test APIshttps://www.endpointdev.com/blog/2021/12/using-postman-to-test-apis/2021-12-16T00:00:00+00:00Couragyn Chretien
<p><img src="/blog/2021/12/using-postman-to-test-apis/20071221_144412000-sm.jpg" alt="Photo of large rusty iron chain with blurry sea in background"></p>
<!-- photo by Josh Ausborne -->
<p>Postman is an easy-to-use tool that facilitates testing APIs. The GUI avoids the pain of old-school command-line tools and other time-consuming practices. Postman is helpful during development, for collaborating, and integral in automated testing.</p>
<p>It’s one of the most popular and longest-running API tools out there, so there’s a lot of documentation and community forums to provide whatever assistance is needed.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/postman.png" alt="Postman logo"></p>
<h3 id="use-in-development">Use in development</h3>
<p>The days of only manually testing APIs with curl are gone.</p>
<p>Instead we can use Postman to create reusable API calls. Postman will remember information such as the header configuration and data contained in the body. The API call can be sent from the GUI client, and it will display the response in a human-readable format.</p>
<p>Saved requests can be grouped in Collections. These can be imported and exported for collaborative use in a team.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/gui.png" alt="Postman GUI screenshot"></p>
<h3 id="automated-testing">Automated Testing</h3>
<p>Full API testing can be done with the click of a button. Investing time now into building up the Postman infastructure will save development time down the line.</p>
<p>Custom Test Suites can be written in JavaScript and run manually or by a script. Periodically running these will ensure your application’s API experiences limited regression. Below are a few simple tests that can be run using Postman’s JavaScript <code>pm</code> object.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// Post request
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">const</span> postRequest = {
url: <span style="color:#d20;background-color:#fff0f0">'https://endpointdev.com/post'</span>,
method: <span style="color:#d20;background-color:#fff0f0">'POST'</span>,
header: {
<span style="color:#d20;background-color:#fff0f0">'Content-Type'</span>: <span style="color:#d20;background-color:#fff0f0">'application/json'</span>
},
body: {
mode: <span style="color:#d20;background-color:#fff0f0">'raw'</span>,
raw: JSON.stringify({ key: <span style="color:#d20;background-color:#fff0f0">'posting this text'</span> })
}
};
pm.sendRequest(postRequest, (error, response) => {
console.log(error ? error : response.json());
});
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#888">// Get request
</span><span style="color:#888"></span>pm.sendRequest(<span style="color:#d20;background-color:#fff0f0">'https://endpointdev.com/get'</span>, (error, response) => {
<span style="color:#080;font-weight:bold">if</span> (error) {
console.log(error);
}
pm.test(<span style="color:#d20;background-color:#fff0f0">'response should be okay to process'</span>, () => {
pm.expect(response).to.have.property(<span style="color:#d20;background-color:#fff0f0">'status'</span>, <span style="color:#d20;background-color:#fff0f0">'OK'</span>);
pm.expect(response).to.have.property(<span style="color:#d20;background-color:#fff0f0">'code'</span>, <span style="color:#00d;font-weight:bold">200</span>);
pm.expect(error).to.equal(<span style="color:#080;font-weight:bold">null</span>);
});
});
</code></pre></div><p>These tests can be hooked into your CI/CD pipeline easily for truly automated testing. Postman integrates with Jenkins with help from Newman, Postman’s command-line Collection Runner. For a step-by-step guide getting this up and running see their <a href="https://learning.postman.com/docs/running-collections/using-newman-cli/integration-with-jenkins/">Integrating with Jenkins</a> documentation.</p>
<h3 id="alternatives">Alternatives</h3>
<p>Insomnia and Paw are two of the top alternatives to Postman. They have mostly the same capabilities with slight differences.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/insomnia.png" alt="Insomnia logo"></p>
<p>Insomnia, like Postman, has a free version and an upgraded paid version. The UI tends to be a bit easier on the eyes. Its lightweight nature allows it to run faster and be a bit more responsive than Postman typically is. The main downside is the inability to write testing for standard requests—it needs to be in OpenAPI format.</p>
<p><img src="/blog/2021/12/using-postman-to-test-apis/paw.png" alt="Paw logo"></p>
<p>Paw is Apple to Postman’s Android. It was originally a Mac-only application and has only recently become available on other platforms. Like most Mac apps its UI is streamlined and will be very easy to pick up for any Mac users. The original Mac app performs better than the cross-platform version, and it has many extensions to expand its capabilities. The main downside is that there is no free version.</p>
<h3 id="reference">Reference</h3>
<ul>
<li><a href="https://www.postman.com/">Postman</a></li>
<li><a href="https://insomnia.rest/">Insomnia</a></li>
<li><a href="https://paw.cloud/">Paw</a></li>
</ul>
Integrating the Estes Freight Shipping SOAP API as a Spree Shipping Calculatorhttps://www.endpointdev.com/blog/2021/09/estes-shipping-spree/2021-09-28T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2021/09/estes-shipping-spree/20190617-084228-small.jpg" alt="Cargo ship on sea with dark clouds"></p>
<!-- photo by Jon Jensen -->
<p>One of our clients with a Spree-based e-commerce site was interested in providing automated shipping quotes to their customers using their freight carrier <a href="https://www.estes-express.com/">Estes</a>. After doing some research I found that Estes provided a variety of SOAP APIs and determined a method for extending Spree with custom shipping rate calculators. This presented an interesting challenge to me on several levels: most of my previous API integration experience was with <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a>, not <a href="https://en.wikipedia.org/wiki/SOAP">SOAP</a> APIs, and I had not previously worked on custom shipping calculators for Spree. Fortunately, the Estes SOAP API documentation and some code examples of other Spree shipping calculators were all I needed to create a successful integration of the freight shipping API for this client.</p>
<h3 id="estes-api-documentation">Estes API Documentation</h3>
<p>The Estes <a href="https://www.estes-express.com/resources/digital-services/api/rate-quote-web-service-v4-0">Rate Quote Web Service</a> API is the one that I relied on for being able to generate shipping quotes based on a combination of source address, destination address, and package weight. I found the developer documentation to be thorough and helpful, and was able to create working client code to send a request and receive a response relatively quickly. Many optional fields can be provided when making requests but I found that I only needed to use a small subset of these, as shown in the example code below.</p>
<p>The one aspect of the API that tripped me up a bit was their use of <code>CN</code> as the country code for Canada; Spree and most other codebases I have encountered use the international standard <a href="https://www.iso.org/iso-3166-country-codes.html">ISO 3166 country codes</a> with <code>CA</code> for Canada, so I had to add a small workaround for that in my client code when requesting shipping quotes to Canadian addresses. Another limitation I encountered is that the API expected to receive only 5-digit US and 6-character Canadian postal codes, so I had to do a bit of manipulation in my shipping calculator to account for that.</p>
<h3 id="ruby-soap-client">Ruby SOAP Client</h3>
<p>I researched Ruby SOAP clients and soon found <a href="https://www.savonrb.com/">Savon</a>, the “Heavy metal SOAP client”, which proved to be very easy to integrate into my existing Rails/Spree application. I added the savon gem to my project’s Gemfile and I was quickly able to instantiate a Savon client and configure it using the WSDL provided by Estes. After that, most of the integration work involved crafting a valid XML payload for my request and then parsing the response.</p>
<h3 id="spree-shipping-calculators">Spree Shipping Calculators</h3>
<p>The final piece of the puzzle was implementing the Savon client into a Spree shipping calculator class. This process allowed me to retrieve details about the current user’s order and then return the calculated shipping estimates within the context of the Spree checkout pages. Looking at existing shipping calculator code helped set me on the right path here; in the end, it was just a matter of defining a new class that inherited from the base <code>Spree::ShippingCalculator</code> class and then defining the <code>#compute_package</code> method expected by Spree for returning the shipping cost of a given package.</p>
<h3 id="code-example">Code Example</h3>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/models/spree/calculator/shipping/estes_calculator.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Spree::Calculator::Shipping</span>
<span style="color:#888"># Custom freight shipping rate API integration</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">EstesCalculator</span> < <span style="color:#036;font-weight:bold">Spree</span>::<span style="color:#036;font-weight:bold">ShippingCalculator</span>
<span style="color:#036;font-weight:bold">ESTES_API_URL</span> = <span style="color:#d20;background-color:#fff0f0">'https://www.estes-express.com/tools/rating/ratequote/v4.0/services/RateQuoteService?wsdl'</span>.freeze
<span style="color:#080;font-weight:bold">def</span> <span style="color:#b06;font-weight:bold">self</span>.<span style="color:#06b;font-weight:bold">description</span>
<span style="color:#d20;background-color:#fff0f0">'Estes Freight'</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">compute_package</span>(package)
country_code = package.order.ship_address.country.iso
<span style="color:#080;font-weight:bold">return</span> <span style="color:#00d;font-weight:bold">0</span> <span style="color:#080;font-weight:bold">unless</span> country_code.in?([<span style="color:#d20;background-color:#fff0f0">'US'</span>, <span style="color:#d20;background-color:#fff0f0">'CA'</span>, <span style="color:#d20;background-color:#fff0f0">'MX'</span>])
client = <span style="color:#036;font-weight:bold">Savon</span>.client(
<span style="color:#a60;background-color:#fff0f0">filters</span>: %i[user password account],
<span style="color:#a60;background-color:#fff0f0">log</span>: <span style="color:#080">true</span>,
<span style="color:#a60;background-color:#fff0f0">log_level</span>: <span style="color:#a60;background-color:#fff0f0">:debug</span>,
<span style="color:#a60;background-color:#fff0f0">logger</span>: <span style="color:#036;font-weight:bold">Logger</span>.new(<span style="color:#036;font-weight:bold">Rails</span>.root.join(<span style="color:#d20;background-color:#fff0f0">'log'</span>, <span style="color:#d20;background-color:#fff0f0">'savon.log'</span>)),
<span style="color:#a60;background-color:#fff0f0">pretty_print_xml</span>: <span style="color:#080">true</span>,
<span style="color:#a60;background-color:#fff0f0">wsdl</span>: <span style="color:#036;font-weight:bold">ESTES_API_URL</span>
)
country_code = <span style="color:#d20;background-color:#fff0f0">'CN'</span> <span style="color:#080;font-weight:bold">if</span> country_code == <span style="color:#d20;background-color:#fff0f0">'CA'</span> <span style="color:#888"># Estes uses CN for Canada</span>
postal_code = package.order.ship_address.zipcode
postal_code =
<span style="color:#080;font-weight:bold">if</span> country_code == <span style="color:#d20;background-color:#fff0f0">'CN'</span>
postal_code.delete(<span style="color:#d20;background-color:#fff0f0">' '</span>).first(<span style="color:#00d;font-weight:bold">6</span>)
<span style="color:#080;font-weight:bold">else</span>
postal_code.first(<span style="color:#00d;font-weight:bold">5</span>)
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> xml = <span style="color:#a61717;background-color:#e3d2d2"><<</span>~XML
<span style="color:#b06;font-weight:bold"><soapenv:Envelope</span>
<span style="color:#369">xmlns:soapenv=</span><span style="color:#d20;background-color:#fff0f0">"http://schemas.xmlsoap.org/soap/envelope/"</span>
<span style="color:#369">xmlns:rat=</span><span style="color:#d20;background-color:#fff0f0">"http://ws.estesexpress.com/ratequote"</span>
<span style="color:#369">xmlns:rat1=</span><span style="color:#d20;background-color:#fff0f0">"http://ws.estesexpress.com/schema/2019/01/ratequote"</span><span style="color:#b06;font-weight:bold">></span>
<span style="color:#b06;font-weight:bold"><soapenv:Header></span>
<span style="color:#b06;font-weight:bold"><rat:auth></span>
<span style="color:#b06;font-weight:bold"><rat:user></span>#{ENV['ESTES_USER']}<span style="color:#b06;font-weight:bold"></rat:user></span>
<span style="color:#b06;font-weight:bold"><rat:password></span>#{ENV['ESTES_PASSWORD']}<span style="color:#b06;font-weight:bold"></rat:password></span>
<span style="color:#b06;font-weight:bold"></rat:auth></span>
<span style="color:#b06;font-weight:bold"></soapenv:Header></span>
<span style="color:#b06;font-weight:bold"><soapenv:Body></span>
<span style="color:#b06;font-weight:bold"><rat1:rateRequest></span>
<span style="color:#b06;font-weight:bold"><rat1:requestID></span>#{package.order.number}<span style="color:#b06;font-weight:bold"></rat1:requestID></span>
<span style="color:#b06;font-weight:bold"><rat1:account></span>#{ENV['ESTES_ACCOUNT']}<span style="color:#b06;font-weight:bold"></rat1:account></span>
<span style="color:#b06;font-weight:bold"><rat1:originPoint></span>
<span style="color:#b06;font-weight:bold"><rat1:countryCode></span>US<span style="color:#b06;font-weight:bold"></rat1:countryCode></span>
<span style="color:#b06;font-weight:bold"><rat1:postalCode></span>10001<span style="color:#b06;font-weight:bold"></rat1:postalCode></span>
<span style="color:#b06;font-weight:bold"></rat1:originPoint></span>
<span style="color:#b06;font-weight:bold"><rat1:destinationPoint></span>
<span style="color:#b06;font-weight:bold"><rat1:countryCode></span>#{country_code}<span style="color:#b06;font-weight:bold"></rat1:countryCode></span>
<span style="color:#b06;font-weight:bold"><rat1:postalCode></span>#{postal_code}<span style="color:#b06;font-weight:bold"></rat1:postalCode></span>
<span style="color:#b06;font-weight:bold"></rat1:destinationPoint></span>
<span style="color:#b06;font-weight:bold"><rat1:payor></span>S<span style="color:#b06;font-weight:bold"></rat1:payor></span>
<span style="color:#b06;font-weight:bold"><rat1:terms></span>C<span style="color:#b06;font-weight:bold"></rat1:terms></span>
<span style="color:#b06;font-weight:bold"><rat1:baseCommodities></span>
<span style="color:#b06;font-weight:bold"><rat1:commodity></span>
<span style="color:#b06;font-weight:bold"><rat1:class></span>50<span style="color:#b06;font-weight:bold"></rat1:class></span>
<span style="color:#b06;font-weight:bold"><rat1:weight></span>#{package_weight(package)}<span style="color:#b06;font-weight:bold"></rat1:weight></span>
<span style="color:#b06;font-weight:bold"></rat1:commodity></span>
<span style="color:#b06;font-weight:bold"></rat1:baseCommodities></span>
<span style="color:#b06;font-weight:bold"></rat1:rateRequest></span>
<span style="color:#b06;font-weight:bold"></soapenv:Body></span>
<span style="color:#b06;font-weight:bold"></soapenv:Envelope></span>
XML
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"> response = client.call(<span style="color:#a60;background-color:#fff0f0">:get_quote</span>, <span style="color:#a60;background-color:#fff0f0">xml</span>: xml)
quotes = response.body.dig(<span style="color:#a60;background-color:#fff0f0">:rate_quote</span>, <span style="color:#a60;background-color:#fff0f0">:quote_info</span>, <span style="color:#a60;background-color:#fff0f0">:quote</span>)
<span style="color:#080;font-weight:bold">if</span> quotes.is_a?(<span style="color:#038">Array</span>)
quotes.first.dig(<span style="color:#a60;background-color:#fff0f0">:pricing</span>, <span style="color:#a60;background-color:#fff0f0">:total_price</span>).to_f
<span style="color:#080;font-weight:bold">elsif</span> quotes.is_a?(<span style="color:#036;font-weight:bold">Hash</span>)
quotes.dig(<span style="color:#a60;background-color:#fff0f0">:pricing</span>, <span style="color:#a60;background-color:#fff0f0">:total_price</span>).to_f
<span style="color:#080;font-weight:bold">else</span>
<span style="color:#00d;font-weight:bold">0</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">rescue</span> <span style="color:#036;font-weight:bold">Savon</span>::<span style="color:#036;font-weight:bold">Error</span>
<span style="color:#888"># Record shipping rate as 0 if an API error is caught, 0 amount will indicate need to show user an error message on the shipping rate page</span>
<span style="color:#00d;font-weight:bold">0</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080">private</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">package_weight</span>(package)
weight = package.contents.sum { |content| content.line_item.weight }
<span style="color:#080;font-weight:bold">if</span> weight < <span style="color:#00d;font-weight:bold">5</span> <span style="color:#888"># enforce minimum package weight</span>
<span style="color:#00d;font-weight:bold">5</span>
<span style="color:#080;font-weight:bold">else</span>
weight.round
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><h3 id="conclusion">Conclusion</h3>
<p>I was pleased that I was able to quickly build a custom shipping calculator in Spree that used the Estes Rate Quote API to provide accurate shipping estimates for large freight packages. The high quality documentation of the Estes API and the Savon SOAP Client gem made for a pleasant development experience, and the client was happy to gain the new functionality for their Spree store.</p>
Building REST APIs with .NET 5, ASP.NET Core, and PostgreSQLhttps://www.endpointdev.com/blog/2021/07/dotnet-5-web-api/2021-07-09T00:00:00+00:00Kevin Campusano
<p><img src="/blog/2021/07/dotnet-5-web-api/market-cropped.jpg" alt="A market at night">
<a href="https://unsplash.com/photos/cpbWNtkKoiU">Photo</a> by <a href="https://unsplash.com/@sam_beasley">Sam Beasley</a></p>
<p>This is old news by now, but I’m still amazed by the fact that nowadays <a href="https://dotnet.microsoft.com/platform/open-source">.NET is open source and can run on Linux</a>. I truly believe that this new direction can help the technology realize its true potential, since it’s no longer shackled to Windows-based environments. I’ve personally been outside the .NET game for a good while, but with <a href="https://docs.microsoft.com/en-us/dotnet/core/dotnet-five">the milestone release that is .NET 5</a>, I think now is a great time to dive back in.</p>
<p>So I thought of taking some time to do just that, really dive in, see what’s new, and get a sense of the general developer experience that the current incarnation of .NET offers. So in this blog post, I’m going to chronicle my experience developing a simple but complete <a href="https://www.redhat.com/en/topics/api/what-is-a-rest-api">REST API</a> application. Along the way, I’ll touch on the most common problems that one runs into when developing such applications and how they are solved in the .NET world. So think of this piece as a sort of tutorial or overview of the most common framework features when it comes to developing REST APIs.</p>
<blockquote>
<p>There’s a <a href="#table-of-contents">table of contents</a> at the bottom.</p>
</blockquote>
<p>First, let’s get familiar with what we’re building.</p>
<h3 id="what-were-building">What we’re building</h3>
<h4 id="the-demo-application">The demo application</h4>
<blockquote>
<p>You can find the finished product on my <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api">GitHub</a>.</p>
</blockquote>
<p>The application that we’ll be building throughout this article will address a request from a hypothetical car junker business. Our client wants to automate the process of calculating how much money to offer their customers for their vehicles, given certain information about them. And they want an app to do that. We are building the back-end component that will support that app. It is a REST API that allows users to provide vehicle information (year, make, model, condition, etc.) and will produce a quote of how much money our hypothetical client would be willing to pay for it.</p>
<p>Here’s a short list of features that we need to implement in order to fulfill that requirement:</p>
<ol>
<li>Given a vehicle model and condition, calculate a price.</li>
<li>Store and manage rules that are used to calculate vehicle prices.</li>
<li>Store and manage pricing overrides on a vehicle model basis. Price overrides are used regardless of the current rules.</li>
<li>CRUD vehicle models so that overrides can be specified for them.</li>
</ol>
<h4 id="the-data-model">The data model</h4>
<p>Here’s what our data model looks like:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/data-model.png" alt="Data Model"></p>
<p>The main table in our model is the <code>quotes</code> table. It stores all the requests for quotes received from our client’s customers. It captures all the relevant vehicle information in terms of model and condition. It also captures the offered quote; that is, the money value that our system calculates for their vehicle.</p>
<p>The <code>quotes</code> table includes all the fields that identify a vehicle: year, make, model, body style and size. It also includes a <code>model_style_year_id</code> field which is an optional foreign key to another table. This FK points to the <code>model_style_years</code> table which contains specific vehicle models that our system can store explicitly.</p>
<p>The idea of this is that, when a customer submits a request for a quote, if we have their vehicle registered in our database, then we can populate this foreign key and link the quote with the specific vehicle that it’s quoting. If we don’t have their vehicle registered, then we leave that field unpopulated. Either way, we can offer a quote. The only difference is the level or certainty of the quote.</p>
<p>The records in the <code>model_style_years</code> table represent specific vehicles. That whole hierarchy works like this: A vehicle make (e.g. Honda, Toyota, etc. in the <code>makes</code> table) has many models (e.g. Civic, Corolla, etc. in the <code>models</code> table), each model has many styles (the <code>model_styles</code> table). Styles are combinations of body types (the <code>body_types</code> table) and sizes (e.g. Mid-size Sedan, Compact Coupe, etc. in the <code>sizes</code> table). And finally, each model style has many years in which they were being produced (via the <code>model_style_years</code> table).</p>
<p>This model allows us very fine-grained differentiation between vehicles. For example, we can have a “2008 Honda Civic Hatchback which is a Compact car” and also a “1990 Honda Civic Hatchback which is a Sub-compact”. That is, same model, different year, size or body type.</p>
<p>We also have a <code>quote_rules</code> table which stores the rules that are applied when it comes to calculating a vehicle quote. The rules are pairs of key-values with an associated monetary value. So for example, rules like “a vehicle that has alloy wheels is worth $10 more” can be expressed in the table with a record where <code>feature_type</code> is “has_alloy_wheels”, <code>feature_value</code> is “true” and <code>price_modifier</code> is “10”.</p>
<p>Finally, we have a <code>quote_overrides</code> table which specifies a flat, static price for specific vehicles (via the link to the <code>model_style_years</code> table). The idea here is that if some customer requests a quote for a vehicle for which we have an override, no price calculation rules are applied and they are offered what is specified in the override record.</p>
<h3 id="the-development-environment">The development environment</h3>
<h4 id="setting-up-the-postgresql-database-with-docker">Setting up the PostgreSQL database with Docker</h4>
<p>For this project, our database of choice is <a href="https://www.postgresql.org/">PostgreSQL</a>. Luckily for us, getting a PostgreSQL instance up and running is very easy thanks to <a href="https://www.docker.com/">Docker</a>.</p>
<blockquote>
<p>If you want to learn more about dockerizing a typical web application, take a look at <a href="/blog/2020/08/containerizing-magento-with-docker-compose-elasticsearch-mysql-and-magento/">this article</a> that explains the process in detail.</p>
</blockquote>
<p>Once you have <a href="https://docs.docker.com/get-docker/">Docker installed</a> in your machine, getting a PostgreSQL instance is as simple as running the following command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ docker run -d <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> --name vehicle-quote-postgres <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -p 5432:5432 <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> --network host <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -e <span style="color:#369">POSTGRES_DB</span>=vehicle_quote <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -e <span style="color:#369">POSTGRES_USER</span>=vehicle_quote <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -e <span style="color:#369">POSTGRES_PASSWORD</span>=password <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> postgres
</code></pre></div><p>Here we’re asking Docker to run a new <a href="https://docs.docker.com/get-started/#what-is-a-container">container</a> based on the latest <code>postgres</code> <a href="https://docs.docker.com/get-started/#what-is-a-container-image">image</a> from <a href="https://hub.docker.com/_/postgres">DockerHub</a>, name it <code>vehicle-quote-postgres</code>, specify the port to use the default PostgreSQL one, make it accessible to the local network (with the <code>--network host</code> option) and finally, specify a few environment variables that the <code>postgres</code> image uses when building our new instance to set up the default database name, user and password (with the three <code>-e</code> options).</p>
<p>After Docker is done working its magic, you should be able to access the database with something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">$ docker exec -it vehicle-quote-postgres psql -U vehicle_quote
psql (13.2 (Debian 13.2-1.pgdg100+1))
Type "help" for help.
vehicle_quote=#
</code></pre></div><p>This command is connecting to our new <code>vehicle-quote-postgres</code> container and then, from within the container, using the <a href="https://www.postgresql.org/docs/current/app-psql.html">command line client psql</a> in order to connect to the database.</p>
<p>If you have <a href="https://www.compose.com/articles/postgresql-tips-installing-the-postgresql-client/">psql installed</a> on your own machine, you can use it directly to connect to the PostgreSQL instance running inside the container:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ psql -h localhost -U vehicle_quote
</code></pre></div><p>This is possible because we specified in our <code>docker run</code> command that the container would be accepting traffic over port 5432 (<code>-p 5432:5432</code>) and that it would be accesible within the same network as our actual machine (<code>--network host</code>).</p>
<h4 id="installing-the-net-5-sdk">Installing the .NET 5 SDK</h4>
<p>Ok, with that out of the way, let’s install .NET 5.</p>
<p>.NET 5 truly is multi-platform, so whatever environment you prefer to work with, they’ve got you covered. You can go to <a href="https://dotnet.microsoft.com/download/dotnet/5.0">the .NET 5 download page</a> and pick your desired flavor of the SDK.</p>
<p>On Ubuntu 20.10, which is what I’m running, installation is painless. It’s your typical process with <a href="https://en.wikipedia.org/wiki/APT_(software)">APT</a> and <a href="https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#2010-">this page from the official docs</a> has all the details.</p>
<p>First step is to add the Microsoft package repository:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ wget https://packages.microsoft.com/config/ubuntu/20.10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
$ sudo dpkg -i packages-microsoft-prod.deb
</code></pre></div><p>Then, install .NET 5 with APT like one would any other software package:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ sudo apt-get update; <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> sudo apt-get install -y apt-transport-https && <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> sudo apt-get update && <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> sudo apt-get install -y dotnet-sdk-5.0
</code></pre></div><p>Run <code>dotnet --version</code> in your console and you should see something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet --version
5.0.301
</code></pre></div><h3 id="setting-up-the-project">Setting up the project</h3>
<h4 id="creating-our-aspnet-core-rest-api-project">Creating our ASP.NET Core REST API project</h4>
<p>Ok now that we have our requirements, database and SDK, let’s start setting up our project. We do so with the following command:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet new webapi -o VehicleQuotes
</code></pre></div><p>This instructs the <code>dotnet</code> command line tool to create a new REST API web application project for us in a new <code>VehicleQuotes</code> directory.</p>
<p>As a result, <code>dotnet</code> will give you some messages, including this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">The template "ASP.NET Core Web API" was created successfully.
</code></pre></div><p>A new directory was created with our web application files. The newly created <code>VehicleQuotes</code> project looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">.
├── appsettings.Development.json
├── appsettings.json
├── Controllers
│ └── WeatherForecastController.cs
├── obj
│ ├── project.assets.json
│ ├── project.nuget.cache
│ ├── VehicleQuotes.csproj.nuget.dgspec.json
│ ├── VehicleQuotes.csproj.nuget.g.props
│ └── VehicleQuotes.csproj.nuget.g.targets
├── Program.cs
├── Properties
│ └── launchSettings.json
├── Startup.cs
├── VehicleQuotes.csproj
└── WeatherForecast.cs
</code></pre></div><p>Important things to note here are the <code>appsettings.json</code> and <code>appsettings.Development.json</code> files which contain environment specific configuration values; the <code>Controllers</code> directory where we define our application controllers and action methods (i.e. our REST API endpoints); the <code>Program.cs</code> and <code>Startup.cs</code> files that contain our application’s entry point and bootstrapping logic; and finally <code>VehicleQuotes.csproj</code> which is the file that contains project-wide configuration that the framework cares about like references, compilation targets, and other options. Feel free to explore.</p>
<p>The <code>dotnet new</code> command has given us quite a bit. These files make up a fully working application that we can run and play around with. It even has a <a href="https://swagger.io/tools/swagger-ui/">Swagger UI</a>, as I’ll demonstrate shortly. It’s a great place to get started from.</p>
<blockquote>
<p>You can also get a pretty comprehensive <code>.gitignore</code> file by running the <code>dotnet new gitignore</code> command.</p>
</blockquote>
<p>From inside the <code>VehicleQuotes</code> directory, you can run the application with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet run
</code></pre></div><p>Which will start up a development server and give out the following output:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet run
Building...
info: Microsoft.Hosting.Lifetime[0]
Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
Content root path: /home/kevin/projects/endpoint/blog/VehicleQuotes
</code></pre></div><p>Open up a browser window and go to <code>https://localhost:5001/swagger</code> to find a Swagger UI listing our API’s endpoints:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/initial-swagger.png" alt="Initial Swagger UI"></p>
<p>As you can see we’ve got a <code>GET /WeatherForecast</code> endpoint in our app. This is included by default in the <code>webapi</code> project template that we specified in our call to <code>dotnet new</code>. You can see it defined in the <code>Controllers/WeatherForecastController.cs</code> file.</p>
<h4 id="installing-packages-well-need">Installing packages we’ll need</h4>
<p>Now let’s install all the tools and libraries we will need for our application. First, we install the <a href="https://www.nuget.org/packages/dotnet-aspnet-codegenerator/">ASP.NET Code Generator</a> tool which we’ll use later for scaffolding controllers:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet tool install --global dotnet-aspnet-codegenerator
</code></pre></div><p>We also need to install the <a href="https://www.nuget.org/packages/dotnet-ef/">Entity Framework command line tools</a> which help us with creating and applying database migrations:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet tool install --global dotnet-ef
</code></pre></div><p>Now, we need to install a few libraries that we’ll use in our project. First are all the packages that allow us to use <a href="https://docs.microsoft.com/en-us/ef/">Entity Framework Core</a>, provide scaffolding support and give us a detailed debugging page for database errors:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet add package Microsoft.VisualStudio.Web.CodeGeneration.Design
$ dotnet add package Microsoft.EntityFrameworkCore.Design
$ dotnet add package Microsoft.EntityFrameworkCore.SqlServer
$ dotnet add package Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore
</code></pre></div><p>We also need the <a href="https://www.npgsql.org/efcore/">EF Core driver for PostgreSQL</a> which will allow us to interact with our database:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL
</code></pre></div><p>Finally, we need <a href="https://github.com/efcore/EFCore.NamingConventions">another package</a> that will allow us to use the <a href="https://en.wikipedia.org/wiki/Snake_case">snake case</a> naming convention for our database tables, fields, etc. We need this because EF Core uses <a href="https://wiki.c2.com/?UpperCamelCase">capitalized camel case</a> by default, which is not very common in the PostgreSQL world, so this will allow us to play nice. This is the package:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet add package EFCore.NamingConventions
</code></pre></div><h4 id="connecting-to-the-database-and-performing-initial-app-configuration">Connecting to the database and performing initial app configuration</h4>
<p>In order to connect to, query, and modify a database using EF Core, we need to create a <a href="https://docs.microsoft.com/en-us/ef/core/dbcontext-configuration/"><code>DbContext</code></a>. This is a class that serves as the entry point into the database. Create a new directory called <code>Data</code> in the project root and add this new <code>VehicleQuotesContext.cs</code> file to it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">Microsoft.EntityFrameworkCore</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">VehicleQuotesContext</span> : DbContext
{
<span style="color:#080;font-weight:bold">public</span> VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
: <span style="color:#080;font-weight:bold">base</span>(options)
{
}
}
}
</code></pre></div><p>As you can see this is just a simple class that inherits from EF Core’s <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.dbcontext?view=efcore-5.0"><code>DbContext</code></a> class. That’s all we need for now. We will continue building on this class as we add new tables and configurations.</p>
<p>Now, we need to add this class into <a href="https://docs.microsoft.com/en-us/aspnet/core/?view=aspnetcore-5.0">ASP.NET Core’s</a> built-in <a href="https://martinfowler.com/articles/injection.html">IoC (inversion of control) container</a> so that it’s available to controllers and other classes via <a href="https://en.wikipedia.org/wiki/Dependency_injection">Dependency Injection</a>, and tell it how to find our database. Go to <code>Startup.cs</code> and add the following using statement near the top of the file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">Microsoft.EntityFrameworkCore</span>;
</code></pre></div><p>That will allow us to do the following change in the <code>ConfigureServices</code> method:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"> public void ConfigureServices(IServiceCollection services)
{
// ...
<span style="color:#000;background-color:#dfd">+ services.AddDbContext<VehicleQuotesContext>(options =>
</span><span style="color:#000;background-color:#dfd">+ options
</span><span style="color:#000;background-color:#dfd">+ .UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
</span><span style="color:#000;background-color:#dfd">+ );
</span><span style="color:#000;background-color:#dfd"></span> }
</code></pre></div><blockquote>
<p><code>UseNpgsql</code> is an <a href="https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods">extension method</a> made available to us by the <code>Npgsql.EntityFrameworkCore.PostgreSQL</code> package that we installed in the previous step.</p>
</blockquote>
<p>The <code>services</code> variable contains all the objects (known as “services”) that are available in the app for Dependency Injection. So here, we’re adding our newly created <code>DbContext</code> to it, specifying that it will connect to a PostgreSQL database (via the <code>options.UseNpgsql</code> call), and that it will use a connection string named <code>VehicleQuotesContext</code> from the app’s default configuration file. So let’s add the connection string then. To do so, change the <code>appsettings.json</code> like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"> {
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
<span style="color:#000;background-color:#fdd">- "AllowedHosts": "*"
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ "AllowedHosts": "*",
</span><span style="color:#000;background-color:#dfd">+ "ConnectionStrings": {
</span><span style="color:#000;background-color:#dfd">+ "VehicleQuotesContext": "Host=localhost;Database=vehicle_quote;Username=vehicle_quote;Password=password"
</span><span style="color:#000;background-color:#dfd">+ }
</span><span style="color:#000;background-color:#dfd"></span> }
</code></pre></div><p>This is your typical PostgreSQL connection string. The only gotcha is that it needs to be specified under the <code>ConnectionStrings</code> -> <code>VehicleQuotesContext</code> section so that our call to <code>Configuration.GetConnectionString</code> can find it.</p>
<p>Now let’s put the <code>EFCore.NamingConventions</code> package to good use and configure EF Core to use snake case when naming database objects. Add the following to the <code>ConfigureServices</code> method in <code>Startup.cs</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddDbContext<VehicleQuotesContext>(options =>
options
.UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
<span style="color:#000;background-color:#dfd">+ .UseSnakeCaseNamingConvention()
</span><span style="color:#000;background-color:#dfd"></span> );
}
</code></pre></div><blockquote>
<p><code>UseSnakeCaseNamingConvention</code> is an extension method made available to us by the <code>EFCore.NamingConventions</code> package that we installed in the previous step.</p>
</blockquote>
<p>Now let’s make logging a little bit more verbose with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddDbContext<VehicleQuotesContext>(options =>
options
.UseNpgsql(Configuration.GetConnectionString("VehicleQuotesContext"))
.UseSnakeCaseNamingConvention()
<span style="color:#000;background-color:#dfd">+ .UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))
</span><span style="color:#000;background-color:#dfd">+ .EnableSensitiveDataLogging()
</span><span style="color:#000;background-color:#dfd"></span> );
}
</code></pre></div><p>This will make sure full database queries appear in the log in the console, including parameter values. This could expose sensitive data so be careful when using <code>EnableSensitiveDataLogging</code> in production.</p>
<p>We can also add the following service configuration to have the app display detailed error pages when something related to the database or migrations goes wrong:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">public void ConfigureServices(IServiceCollection services)
{
// ...
<span style="color:#000;background-color:#dfd">+ services.AddDatabaseDeveloperPageExceptionFilter();
</span><span style="color:#000;background-color:#dfd"></span>}
</code></pre></div><blockquote>
<p><code>AddDatabaseDeveloperPageExceptionFilter</code> is an extension method made available to us by the <code>Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore</code> package that we installed in the previous step.</p>
</blockquote>
<p>Finally, one last configuration I like to do is have the Swagger UI show up at the root URL, so that instead of using <code>https://localhost:5001/swagger</code>, we’re able to just use <code>https://localhost:5001</code>. We do so by by updating the <code>Configure</code> method this time, in the same <code>Startup.cs</code> file that we’ve been working on:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
app.UseSwagger();
<span style="color:#000;background-color:#fdd">- app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1"));
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ app.UseSwaggerUI(c => {
</span><span style="color:#000;background-color:#dfd">+ c.SwaggerEndpoint("/swagger/v1/swagger.json", "VehicleQuotes v1");
</span><span style="color:#000;background-color:#dfd">+ c.RoutePrefix = "";
</span><span style="color:#000;background-color:#dfd">+ });
</span><span style="color:#000;background-color:#dfd"></span> }
</code></pre></div><p>The magic is done by the <code>c.RoutePrefix = "";</code> line which makes it so there’s no need to put any prefix in order to access the auto generated Swagger UI.</p>
<p>Try it out. Do <code>dotnet run</code> and navigate to <code>https://localhost:5001</code> and you should see the Swagger UI there.</p>
<h3 id="building-the-application">Building the application</h3>
<h4 id="creating-model-entities-migrations-and-updating-the-database">Creating model entities, migrations and updating the database</h4>
<p>Alright, with all that configuration out of the way, let’s implement some of our actual application logic now. Refer back to our data model. We’ll start by defining our three simplest tables: <code>makes</code>, <code>sizes</code> and <code>body_types</code>. With EF Core, we define tables via so-called <a href="https://en.wikipedia.org/wiki/Plain_old_CLR_object">POCO</a> entities, which are simple C# classes with some properties. The classes become tables and the properties become the tables’ fields. Instances of these classes represent records in the database.</p>
<p>So, create a new <code>Models</code> directory in our project’s root and add these three files:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/BodyType.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">BodyType</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Name { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/Make.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Make</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Name { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/Size.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Size</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Name { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><p>Now, we add three corresponding <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.dbset-1?view=efcore-5.0"><code>DbSet</code></a>s to our <code>DbContext</code> in <code>Data/VehicleQuoteContext.cs</code>. Here’s the diff:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">using Microsoft.EntityFrameworkCore;
<span style="color:#000;background-color:#dfd">+using VehicleQuotes.Models;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes
{
public class VehicleQuotesContext : DbContext
{
public VehicleQuotesContext (DbContextOptions<VehicleQuotesContext> options)
: base(options)
{
}
<span style="color:#000;background-color:#dfd">+ public DbSet<Make> Makes { get; set; }
</span><span style="color:#000;background-color:#dfd">+ public DbSet<Size> Sizes { get; set; }
</span><span style="color:#000;background-color:#dfd">+ public DbSet<BodyType> BodyTypes { get; set; }
</span><span style="color:#000;background-color:#dfd"></span> }
}
</code></pre></div><p>This is how we tell EF Core to build tables in our database for our entities. You’ll see later how we use those <code>DbSet</code>s to access the data in those tables. For now, let’s create a <a href="https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/?tabs=dotnet-core-cli">migration</a> script that we can later run to apply changes to our database. Run the following to have EF Core create it for us:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef migrations add AddLookupTables
</code></pre></div><p>Now take a look at the newly created <code>Migrations</code> directory. It contains a few new files, but the one we care about right now is <code>Migrations/{TIMESTAMP}_AddLookupTables.cs</code>. In its <code>Up</code> method, it’s got some code that will modify the database structure when run. The EF Core tooling has inspected our project, identified the new entities, and automatically generated a migration script for us that creates tables for them. Notice also how the tables and fields use the snake case naming convention, just as we specified with the call to <code>UseSnakeCaseNamingConvention</code> in <code>Startup.cs</code>.</p>
<p>Now, to actually run the migration script and apply the changes to the database, we do:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef database update
</code></pre></div><p>That command inspects our project to find any migrations that haven’t been run yet, and applies them. In this case, we only have one, so that’s what it runs. Look at the output in the console to see it working its magic step by step:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef database update
Build started...
Build succeeded.
warn: Microsoft.EntityFrameworkCore.Model.Validation[10400]
Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data; this mode should only be enabled during development.
...
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (4ms) [<span style="color:#369">Parameters</span>=[], <span style="color:#369">CommandType</span>=<span style="color:#d20;background-color:#fff0f0">'Text'</span>, <span style="color:#369">CommandTimeout</span>=<span style="color:#d20;background-color:#fff0f0">'30'</span>]
CREATE TABLE sizes (
id integer GENERATED BY DEFAULT AS IDENTITY,
name text NULL,
CONSTRAINT pk_sizes PRIMARY KEY (id)
);
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
Executed DbCommand (1ms) [<span style="color:#369">Parameters</span>=[], <span style="color:#369">CommandType</span>=<span style="color:#d20;background-color:#fff0f0">'Text'</span>, <span style="color:#369">CommandTimeout</span>=<span style="color:#d20;background-color:#fff0f0">'30'</span>]
INSERT INTO <span style="color:#d20;background-color:#fff0f0">"__EFMigrationsHistory"</span> (migration_id, product_version)
VALUES (<span style="color:#d20;background-color:#fff0f0">'20210625212939_AddLookupTables'</span>, <span style="color:#d20;background-color:#fff0f0">'5.0.7'</span>);
Done.
</code></pre></div><p>Notice how it warns us about potential exposure of sensitive data because of that <code>EnableSensitiveDataLogging</code> option we opted into in <code>Startup.cs</code>. Also, EF Core related logs are extra verbose showing all database operations because of another configuration option that we applied there: the <code>UseLoggerFactory(LoggerFactory.Create(builder => builder.AddConsole()))</code> one.</p>
<p>You can connect to the database with the <code>psql</code> command line client and see that the changes took effect:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ psql -h localhost -U vehicle_quote
...
<span style="color:#369">vehicle_quote</span>=<span style="color:#888"># \c vehicle_quote </span>
psql (12.7 (Ubuntu 12.7-0ubuntu0.20.10.1), server 13.2 (Debian 13.2-1.pgdg100+1))
You are now connected to database <span style="color:#d20;background-color:#fff0f0">"vehicle_quote"</span> as user <span style="color:#d20;background-color:#fff0f0">"vehicle_quote"</span>.
<span style="color:#369">vehicle_quote</span>=<span style="color:#888"># \dt</span>
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+---------------
public | __EFMigrationsHistory | table | vehicle_quote
public | body_types | table | vehicle_quote
public | makes | table | vehicle_quote
public | sizes | table | vehicle_quote
(<span style="color:#00d;font-weight:bold">4</span> rows)
<span style="color:#369">vehicle_quote</span>=<span style="color:#888"># \d makes</span>
Table <span style="color:#d20;background-color:#fff0f0">"public.makes"</span>
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+----------------------------------
id | integer | | not null | generated by default as identity
name | text | | |
Indexes:
<span style="color:#d20;background-color:#fff0f0">"pk_makes"</span> PRIMARY KEY, btree (id)
</code></pre></div><p>There are our tables in all their normalized, snake-cased glory. The <code>__EFMigrationsHistory</code> table is used internally by EF Core to keep track of which migrations have been applied.</p>
<h4 id="creating-controllers-for-cruding-our-tables">Creating controllers for CRUDing our tables</h4>
<p>Now that we have that, let’s add a few endpoints to support basic CRUD of those tables. We can use the <code>dotnet-aspnet-codegenerator</code> scaffolding tool that we installed earlier. For the three tables that we have, we would do:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet aspnet-codegenerator controller <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -name MakesController <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m Make <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -dc VehicleQuotesContext <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -async <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -api <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -outDir Controllers
$ dotnet aspnet-codegenerator controller <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -name BodyTypesController <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m BodyType <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -dc VehicleQuotesContext <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -async <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -api <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -outDir Controllers
$ dotnet aspnet-codegenerator controller <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -name SizesController <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m Size <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -dc VehicleQuotesContext <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -async <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -api <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -outDir Controllers
</code></pre></div><p>Those commands tell the scaffolding tool to create new controllers that:</p>
<ol>
<li>Are named as given by the <code>-name</code> option.</li>
<li>Use the model class specified in the <code>-m</code> option.</li>
<li>Use our <code>VehicleQuotesContext</code> to talk to the database. As per the <code>-dc</code> option.</li>
<li>Define the methods using <code>async</code>/<code>await</code> syntax. Given by the <code>-async</code> option.</li>
<li>Are API controllers. Specified by the <code>-api</code> option.</li>
<li>Are created in the <code>Controllers</code> directory. Via the <code>-outDir</code> option.</li>
</ol>
<p>Explore the new files that got created in the <code>Controllers</code> directory: <code>MakesController.cs</code>, <code>BodyTypesController.cd</code> and <code>SizesController.cs</code>. The controllers have been generated with the necessary <a href="https://docs.microsoft.com/en-us/aspnet/mvc/overview/older-versions-1/controllers-and-routing/aspnet-mvc-controllers-overview-cs#understanding-controller-actions">Action Methods</a> to fetch, create, update and delete their corresponding entities. Try <code>dotnet run</code> and navigate to <code>https://localhost:5001</code> to see the new endpoints in the Swagger UI:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/swagger-lookup-tables.png" alt="Swagger UI with lookup tables"></p>
<p>Try it out! You can interact with each of the endpoints from the Swagger UI and it all works as you’d expect.</p>
<h4 id="adding-unique-constraints-via-indexes">Adding unique constraints via indexes</h4>
<p>Ok, our app is coming along well. Right now though, there’s an issue with the tables that we’ve created. It’s possible to create vehicle makes with the same name. The same is true for body types and sizes. This doesn’t make much sense for these tables. So let’s fix that by adding a uniqueness constraint. We can do it by creating a unique database index using EF Core’s <code>Index</code> attribute. For example, we can modify our <code>Models/Make.cs</code> like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#000;background-color:#dfd">+using Microsoft.EntityFrameworkCore;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.Models
{
<span style="color:#000;background-color:#dfd">+ [Index(nameof(Name), IsUnique = true)]
</span><span style="color:#000;background-color:#dfd"></span> public class Make
{
public int ID { get; set; }
public string Name { get; set; }
}
}
</code></pre></div><p>In fact, do the same for our other entities in <code>Models/BodyType.cs</code> and <code>Models/Size.cs</code>. Don’t forget the <code>using Microsoft.EntityFrameworkCore</code> statement.</p>
<p>With that, we can create a new migration:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef migrations add AddUniqueIndexesToLookupTables
</code></pre></div><p>That will result in a new migration script in <code>Migrations/{TIMESTAMP}_AddUniqueIndexesToLookupTables.cs</code>. Its <code>Up</code> method looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">override</span> <span style="color:#080;font-weight:bold">void</span> Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateIndex(
name: <span style="color:#d20;background-color:#fff0f0">"ix_sizes_name"</span>,
table: <span style="color:#d20;background-color:#fff0f0">"sizes"</span>,
column: <span style="color:#d20;background-color:#fff0f0">"name"</span>,
unique: <span style="color:#080;font-weight:bold">true</span>);
migrationBuilder.CreateIndex(
name: <span style="color:#d20;background-color:#fff0f0">"ix_makes_name"</span>,
table: <span style="color:#d20;background-color:#fff0f0">"makes"</span>,
column: <span style="color:#d20;background-color:#fff0f0">"name"</span>,
unique: <span style="color:#080;font-weight:bold">true</span>);
migrationBuilder.CreateIndex(
name: <span style="color:#d20;background-color:#fff0f0">"ix_body_types_name"</span>,
table: <span style="color:#d20;background-color:#fff0f0">"body_types"</span>,
column: <span style="color:#d20;background-color:#fff0f0">"name"</span>,
unique: <span style="color:#080;font-weight:bold">true</span>);
}
</code></pre></div><p>As you can see, new unique indexes are being created on the tables and fields that we specified. Like before, apply the changes to the database structure with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef database update
</code></pre></div><p>Now if you try to create, for example, a vehicle make with a repeated name, you’ll get an error. Try doing so by <code>POST</code>ing to <code>/api/Makes</code> via the Swagger UI:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/unique-constraint-violation.png" alt="Unique constraint violation"></p>
<h4 id="responding-with-specific-http-error-codes-409-conflict">Responding with specific HTTP error codes (409 Conflict)</h4>
<p>The fact that we can now enforce unique constraints is all well and good. But the error scenario is not very user friendly. Instead of returning a “500 Internal Server Error” status code with a wall of text, we should be responding with something more sensible. Maybe a “409 Conflict” would be more appropriate for this kind of error. We can easily update our controllers to handle that scenario. What we need to do is update the methods that handle the <code>POST</code> and <code>PUT</code> endpoints so that they catch the <code>Microsoft.EntityFrameworkCore.DbUpdateException</code> exception and return the proper response. Here’s how we would do it for the <code>MakesController</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">// ...
namespace VehicleQuotes.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class MakesController : ControllerBase
{
// ...
[HttpPut("{id}")]
public async Task<IActionResult> PutMake(int id, Make make)
{
// ...
try
{
await _context.SaveChangesAsync();
}
// ...
<span style="color:#000;background-color:#dfd">+ catch (Microsoft.EntityFrameworkCore.DbUpdateException)
</span><span style="color:#000;background-color:#dfd">+ {
</span><span style="color:#000;background-color:#dfd">+ return Conflict();
</span><span style="color:#000;background-color:#dfd">+ }
</span><span style="color:#000;background-color:#dfd"></span>
return NoContent();
}
[HttpPost]
public async Task<ActionResult<Make>> PostMake(Make make)
{
_context.Makes.Add(make);
<span style="color:#000;background-color:#fdd">- await _context.SaveChangesAsync();
</span><span style="color:#000;background-color:#fdd"></span>
<span style="color:#000;background-color:#dfd">+ try
</span><span style="color:#000;background-color:#dfd">+ {
</span><span style="color:#000;background-color:#dfd">+ await _context.SaveChangesAsync();
</span><span style="color:#000;background-color:#dfd">+ }
</span><span style="color:#000;background-color:#dfd">+ catch (Microsoft.EntityFrameworkCore.DbUpdateException)
</span><span style="color:#000;background-color:#dfd">+ {
</span><span style="color:#000;background-color:#dfd">+ return Conflict();
</span><span style="color:#000;background-color:#dfd">+ }
</span><span style="color:#000;background-color:#dfd"></span>
return CreatedAtAction("GetMake", new { id = make.ID }, make);
}
// ...
}
}
</code></pre></div><p>Go ahead and do the same for the other two controllers, and try again to POST a repeated make name via the Swagger UI. You should see this now instead:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/http-409-conflict.png" alt="HTTP 409 Conflict"></p>
<p>Much better now, don’t you think?</p>
<h4 id="adding-a-more-complex-entity-to-the-model">Adding a more complex entity to the model</h4>
<p>Now let’s work on an entity that’s a little bit more complex: the one we will use to represent vehicle models.</p>
<p>For this entity, we don’t want our API to be as low level as the one for the other three, where it was basically a thin wrapper over database tables. We want it to be a little bit more abstract and not expose the entire database structure verbatim.</p>
<p>Refer back to the data model. We’ll add <code>models</code>, <code>model_styles</code> and <code>model_style_years</code>. Let’s start by adding the following classes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/Model.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Collections.Generic</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Model</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Name { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> MakeID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> Make Make { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> ICollection<ModelStyle> ModelStyles { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/ModelStyle.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Collections.Generic</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ModelStyle</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ModelID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> BodyTypeID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> SizeID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> Model Model { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> BodyType BodyType { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> Size Size { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> ICollection<ModelStyleYear> ModelStyleYears { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/ModelStyleYear.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ModelStyleYear</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Year { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ModelStyleID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> ModelStyle ModelStyle { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><p>Notice how some of these entities now include properties whose types are other entities. Some of them are collections even. These are called Navigation Properties and are how we tell EF Core that our entities are related to one another. These will result in foreign keys being created in the database.</p>
<p>Take the <code>Model</code> entity for example. It has a property <code>Make</code> of type <code>Make</code>. It also has a <code>MakeID</code> property of type <code>int</code>. EF Core sees this and figures out that there’s a relation between the <code>makes</code> and <code>models</code> tables. Specifically, that <code>models</code> have a <code>make</code>. A many-to-one relation where the <code>models</code> table stores a foreign key to the <code>makes</code> table.</p>
<p>Similarly, the <code>Model</code> entity has a <code>ModelStyles</code> property of type <code>ICollection<ModelStyleYear></code>. This tells EF Core that <code>models</code> have many <code>model_styles</code>. This one is a one-to-many relation from the perspective of the <code>models</code> table. The foreign key lives in the <code>model_styles</code> table and points back to <code>models</code>.</p>
<blockquote>
<p>The <a href="https://docs.microsoft.com/en-us/ef/core/modeling/relationships?tabs=fluent-api%2Cfluent-api-simple-key%2Csimple-key#single-navigation-property-1">official documentation</a> is a great resource to learn more details about how relationships work in EF Core.</p>
</blockquote>
<p>After that, same as before, we have to add the corresponding <code>DbSet</code>s to our <code>DbContext</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">// Data/VehicleQuotesContext.cs
// ...
namespace VehicleQuotes
{
public class VehicleQuotesContext : DbContext
{
// ...
<span style="color:#000;background-color:#dfd">+ public DbSet<Model> Models { get; set; }
</span><span style="color:#000;background-color:#dfd">+ public DbSet<ModelStyle> ModelStyles { get; set; }
</span><span style="color:#000;background-color:#dfd">+ public DbSet<ModelStyleYear> ModelStyleYears { get; set; }
</span><span style="color:#000;background-color:#dfd"></span> }
}
</code></pre></div><p>Don’t forget the migration script. First create it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef migrations add AddVehicleModelTables
</code></pre></div><p>And then apply it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef database update
</code></pre></div><h4 id="adding-composite-unique-indexes">Adding composite unique indexes</h4>
<p>These vehicle model related tables also need some uniqueness enforcement. This time, however, the unique keys are composite. Meaning that they involve multiple fields. For vehicle models, for example, it makes no sense to have multiple records with the same make and name. But it does make sense to have multiple models with the same name, as long as they belong to different makes. We can solve for that with a composite index. Here’s how we create one of those in <code>Model</code> with EF Core:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">using System.Collections.Generic;
<span style="color:#000;background-color:#dfd">+using Microsoft.EntityFrameworkCore;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.Models
{
<span style="color:#000;background-color:#dfd">+ [Index(nameof(Name), nameof(MakeID), IsUnique = true)]
</span><span style="color:#000;background-color:#dfd"></span> public class Model
{
// ...
}
}
</code></pre></div><p>Very similar to what we did with the <code>Make</code>, <code>BodyType</code>, and <code>Size</code> entities. The only difference is that this time we included multiple fields in the parameters for the <code>Index</code> attribute.</p>
<p>We should do the same for <code>ModelStyle</code> and <code>ModelStyleYear</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">using System.Collections.Generic;
<span style="color:#000;background-color:#dfd">+using Microsoft.EntityFrameworkCore;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.Models
{
<span style="color:#000;background-color:#dfd">+ [Index(nameof(ModelID), nameof(BodyTypeID), nameof(SizeID), IsUnique = true)]
</span><span style="color:#000;background-color:#dfd"></span> public class ModelStyle
{
// ...
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#000;background-color:#dfd">+using Microsoft.EntityFrameworkCore;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.Models
{
<span style="color:#000;background-color:#dfd">+ [Index(nameof(Year), nameof(ModelStyleID), IsUnique = true)]
</span><span style="color:#000;background-color:#dfd"></span> public class ModelStyleYear
{
// ...
}
}
</code></pre></div><p>Don’t forget the migrations:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef migrations add AddUniqueIndexesForVehicleModelTables
$ dotnet ef database update
</code></pre></div><h4 id="adding-controllers-with-custom-routes">Adding controllers with custom routes</h4>
<p>Our data model dictates that vehicle models belong in a make. In other words, a vehicle model has no meaning by itself. It only has meaning within the context of a make. Ideally, we want our API routes to reflect this concept. In other words, instead of URLs for models to look like this: <code>/api/Models/{id}</code>; we’d rather them look like this: <code>/api/Makes/{makeId}/Models/{modelId}</code>. Let’s go ahead and scaffold a controller for this entity:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet aspnet-codegenerator controller <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -name ModelsController <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -m Model <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -dc VehicleQuotesContext <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -async <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -api <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -outDir Controllers
</code></pre></div><p>Now let’s change the resulting <code>Controllers/ModelsController.cs</code> to use the URL structure that we want. To do so, we modify the <code>Route</code> attribute that’s applied to the <code>ModelsController</code> class to this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[Route("api/Makes/{makeId}/[controller]</span>/<span style="color:#d20;background-color:#fff0f0">")]
</span></code></pre></div><p>Do a <code>dotnet run</code> and take a peek at the Swagger UI on <code>https://localhost:5001</code> to see what the <code>Models</code> endpoint routes look like now:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/nested-routes.png" alt="Nested routes"></p>
<p>The vehicle model routes are now nested within makes, just like we wanted.</p>
<p>Of course, this is just eye candy for now. We need to actually use this new <code>makeId</code> parameter for the logic in the endpoints. For example, one would expect a <code>GET</code> to <code>/api/Makes/1/Models</code> to return all the vehicle models that belong to the make with <code>id</code> 1. But right now, all vehicle models are returned regardless. All other endpoints behave similarly, there’s no limit to the operations on the vehicle models. The given <code>makeId</code> is not taken into consideration at all.</p>
<p>Let’s update the <code>ModelsController</code>’s <code>GetModels</code> method (which is the one that handles the <code>GET /api/Makes/{makeId}/Models</code> endpoint) to behave like one would expect. It should look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpGet]</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<IEnumerable<Model>>> GetModels([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId)
{
<span style="color:#888;font-weight:bold">var</span> make = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Makes.FindAsync(makeId);
<span style="color:#080;font-weight:bold">if</span> (make == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Models.Where(m => m.MakeID == makeId).ToListAsync();
}
</code></pre></div><p>See how we’ve included a new parameter to the method: <code>[FromRoute] int makeId</code>. This <code>[FromRoute]</code> <a href="https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/attributes/">attribute</a> is how we tell ASP.NET Core that this endpoint will use that <code>makeId</code> parameter coming from the URL route. Then, we use our <code>DbContext</code> to try to find the make that corresponds to the given identifier. This is done in <code>_context.Makes.FindAsync(makeId)</code>. Then, if we can’t find the given make, we return a <code>404 Not Found</code> HTTP status code with the <code>return NotFound();</code> line. Finally, we query the <code>models</code> table for all the records whose <code>make_id</code> matches the given parameter. That’s done in the last line of the method.</p>
<blockquote>
<p>We have access to the <code>DbContext</code> because it has been injected as a dependency into the controller via its constructor by the framework.</p>
</blockquote>
<blockquote>
<p><a href="https://docs.microsoft.com/en-us/ef/core/querying/">The official documentation</a> is a great resource to learn about all the possibilities when querying data with EF Core.</p>
</blockquote>
<p>Let’s update the <code>GetModel</code> method, which handles the <code>GET /api/Makes/{makeId}/Models/{id}</code> endpoint, similarly.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">[HttpGet("{id}")]
<span style="color:#000;background-color:#fdd">-public async Task<ActionResult<Model>> GetModel(int id)
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+public async Task<ActionResult<Model>> GetModel([FromRoute] int makeId, int id)
</span><span style="color:#000;background-color:#dfd"></span>{
<span style="color:#000;background-color:#fdd">- var model = await _context.Models.FindAsync(id);
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ var model = await _context.Models.FirstOrDefaultAsync(m =>
</span><span style="color:#000;background-color:#dfd">+ m.MakeID == makeId && m.ID == id
</span><span style="color:#000;background-color:#dfd">+ );
</span><span style="color:#000;background-color:#dfd"></span>
if (model == null)
{
return NotFound();
}
return model;
}
</code></pre></div><p>We’ve once again included the <code>makeId</code> as a parameter to the method and modified the EF Core query to use both the make ID and the vehicle model ID when looking for the record.</p>
<p>And that’s the gist of it. Other methods would need to be updated similarly. The next section will include these methods in their final form, so I won’t go through each one of them here.</p>
<h4 id="using-resource-models-as-dtos-for-controllers">Using resource models as DTOs for controllers</h4>
<p>Now, I did say at the beginning that we wanted the vehicle model endpoint to be a bit more abstract. Right now it’s operating directly over the EF Core entities and our table. As a result, creating new vehicle models via the <code>POST /api/Makes/{makeId}/Models</code> endpoint is a pain. Take a look at the Swagger UI request schema for that endpoint:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/raw-model-request-schema.png" alt="Raw model request schema"></p>
<p>This is way too much. Let’s make it a little bit more user friendly by making it more abstract.</p>
<p>To do that, we will introduce what I like to call a Resource Model (or <a href="https://martinfowler.com/eaaCatalog/dataTransferObject.html">DTO, Data Transfer Object</a>, or View Model). This is a class whose only purpose is to streamline the API contract of the endpoint by defining a set of fields that clients will use to make requests and interpret responses. Something that’s simpler than our actual database structure, but still captures all the information that’s important for our application. We will update the <code>ModelsController</code> so that it’s able to receive objects of this new class as requests, operate on them, translate them to our EF Core entities and actual database records, and return them as a response. The hope is that, by hiding the details of our database structure, we make it easier for clients to interact with our API.</p>
<p>So let’s create a new <code>ResourceModels</code> directory in our project’s root and add these two classes:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// ResourceModels/ModelSpecification.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.ResourceModels</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ModelSpecification</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Name { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> ModelSpecificationStyle[] Styles { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// ResourceModels/ModelSpecificationStyle.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.ResourceModels</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ModelSpecificationStyle</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> BodyType { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Size { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span>[] Years { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><p>Thanks to these two, instead of that mess from above, clients <code>POST</code>ing to <code>/api/Makes/{makeId}/Models</code> will be able to use a request body like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"string"</span>,
<span style="color:#b06;font-weight:bold">"styles"</span>: [
{
<span style="color:#b06;font-weight:bold">"bodyType"</span>: <span style="color:#d20;background-color:#fff0f0">"string"</span>,
<span style="color:#b06;font-weight:bold">"size"</span>: <span style="color:#d20;background-color:#fff0f0">"string"</span>,
<span style="color:#b06;font-weight:bold">"years"</span>: [
<span style="color:#d20;background-color:#fff0f0">"string"</span>
]
}
]
}
</code></pre></div><p>Which is much simpler. We have the vehicle model name and an array of styles. Each style has a body type and a size, which we can specify by their names because those are unique keys. We don’t need their integer IDs (i.e. primary keys) in order to find to them. Then, each style has an array of strings that contain the years in which those styles are available for that model. The make is part of the URL already, so we don’t need to also specify it in the request payload.</p>
<p>Let’s update our <code>ModelsController</code> to use these Resource Models instead of the <code>Model</code> EF Core entity. Be sure to include the namespace where the Resource Models are defined by adding the following using statement: <code>using VehicleQuotes.ResourceModels;</code>. Now, let’s update the <code>GetModels</code> method (which handles the <code>GET /api/Makes/{makeId}/Models</code> endpoint) so that it looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpGet]</span>
<span style="color:#888">// Return a collection of `ModelSpecification`s and expect a `makeId` from the URL.
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<IEnumerable<ModelSpecification>>> GetModels([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId)
{
<span style="color:#888">// Look for the make identified by `makeId`.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> make = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Makes.FindAsync(makeId);
<span style="color:#888">// If we can't find the make, then we return a 404.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (make == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#888">// Build a query to fetch the relevant records from the `models` table and
</span><span style="color:#888"></span> <span style="color:#888">// build `ModelSpecification` with the data.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> modelsToReturn = <span style="color:#00d;font-weight:bold">_</span>context.Models
.Where(m => m.MakeID == makeId)
.Select(m => <span style="color:#080;font-weight:bold">new</span> ModelSpecification {
ID = m.ID,
Name = m.Name,
Styles = m.ModelStyles.Select(ms => <span style="color:#080;font-weight:bold">new</span> ModelSpecificationStyle {
BodyType = ms.BodyType.Name,
Size = ms.Size.Name,
Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
}).ToArray()
});
<span style="color:#888">// Execute the query and respond with the results.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> modelsToReturn.ToListAsync();
}
</code></pre></div><p>The first thing that we changed was the return type. Instead of <code>Task<ActionResult<IEnumerable<Model>>></code>, the method now returns <code>Task<ActionResult<IEnumerable<ModelSpecification>>></code>. We’re going to use our new Resource Models as these endpoints’ contract, so we need to make sure we are returning those. Next, we considerably changed the LINQ expression that searches the database for the vehicle model records we want. The filtering logic (given by the <code>Where</code>) is the same. That is, we’re still searching for vehicle models within the given make ID. What we changed was the projection logic in the <code>Select</code>. Our Action Method now returns a collection of <code>ModelSpecification</code> objects, so we updated the <code>Select</code> to produce such objects, based on the records from the <code>models</code> table that match our search criteria. We build <code>ModelSpecification</code>s using the data coming from <code>models</code> records and their related <code>model_styles</code> and <code>model_style_years</code>. Finally, we asynchronously execute the query to fetch the data from the database and return it.</p>
<p>Next, let’s move on to the <code>GetModel</code> method, which handles the <code>GET /api/Makes/{makeId}/Models/{id}</code> endpoint. This is what it should look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpGet("{id}")]</span>
<span style="color:#888">// Return a `ModelSpecification`s and expect `makeId` and `id` from the URL.
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<ModelSpecification>> GetModel([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId, [FromRoute] <span style="color:#888;font-weight:bold">int</span> id)
{
<span style="color:#888">// Look for the model specified by the given identifiers and also load
</span><span style="color:#888"></span> <span style="color:#888">// all related data that we care about for this method.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> model = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Models
.Include(m => m.ModelStyles).ThenInclude(ms => ms.BodyType)
.Include(m => m.ModelStyles).ThenInclude(ms => ms.Size)
.Include(m => m.ModelStyles).ThenInclude(ms => ms.ModelStyleYears)
.FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);
<span style="color:#888">// If we couldn't find it, respond with a 404.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (model == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#888">// Use the fetched data to construct a `ModelSpecification` to use in the response.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> ModelSpecification {
ID = model.ID,
Name = model.Name,
Styles = model.ModelStyles.Select(ms => <span style="color:#080;font-weight:bold">new</span> ModelSpecificationStyle {
BodyType = ms.BodyType.Name,
Size = ms.Size.Name,
Years = ms.ModelStyleYears.Select(msy => msy.Year).ToArray()
}).ToArray()
};
}
</code></pre></div><p>Same as before, we changed the return type of the method to be <code>ModelSpecification</code>. Then, we modified the query so that it loads all the related data for the <code>Model</code> entity via its navigation properties. That’s what the <code>Include</code> and <code>ThenInclude</code> calls do. We need this data loaded because we use it in the method’s return statement to build the <code>ModelSpecification</code> that will be included in the response. The logic to build it is very similar to that of the previous method.</p>
<blockquote>
<p>You can learn more about the various available approaches for loading data with EF Core in <a href="https://docs.microsoft.com/en-us/ef/core/querying/related-data/">the official documentation</a>.</p>
</blockquote>
<p>Next is the <code>PUT /api/Makes/{makeId}/Models/{id}</code> endpoint, handled by the <code>PutModel</code> method:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpPut("{id}")]</span>
<span style="color:#888">// Expect `makeId` and `id` from the URL and a `ModelSpecification` from the request payload.
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<IActionResult> PutModel([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId, <span style="color:#888;font-weight:bold">int</span> id, ModelSpecification model)
{
<span style="color:#888">// If the id in the URL and the request payload are different, return a 400.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (id != model.ID)
{
<span style="color:#080;font-weight:bold">return</span> BadRequest();
}
<span style="color:#888">// Obtain the `models` record that we want to update. Include any related
</span><span style="color:#888"></span> <span style="color:#888">// data that we want to update as well.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> modelToUpdate = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Models
.Include(m => m.ModelStyles)
.FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);
<span style="color:#888">// If we can't find the record, then return a 404.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (modelToUpdate == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#888">// Update the record with what came in the request payload.
</span><span style="color:#888"></span> modelToUpdate.Name = model.Name;
<span style="color:#888">// Build EF Core entities based on the incoming Resource Model object.
</span><span style="color:#888"></span> modelToUpdate.ModelStyles = model.Styles.Select(style => <span style="color:#080;font-weight:bold">new</span> ModelStyle {
BodyType = <span style="color:#00d;font-weight:bold">_</span>context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
Size = <span style="color:#00d;font-weight:bold">_</span>context.Sizes.Single(size => size.Name == style.Size),
ModelStyleYears = style.Years.Select(year => <span style="color:#080;font-weight:bold">new</span> ModelStyleYear {
Year = year
}).ToList()
}).ToList();
<span style="color:#080;font-weight:bold">try</span>
{
<span style="color:#888">// Try saving the changes. This will run the UPDATE statement in the database.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.SaveChangesAsync();
}
<span style="color:#080;font-weight:bold">catch</span> (Microsoft.EntityFrameworkCore.DbUpdateException)
{
<span style="color:#888">// If there's an error updating, respond accordingly.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> Conflict();
}
<span style="color:#888">// Finally return a 204 if everything went well.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> NoContent();
}
</code></pre></div><p>The purpose of this endpoint is to update existing resources. So, it receives a representation of said resource as a parameter that comes from the request body. Before, it expected an instance of the <code>Model</code> entity, but now, we’ve changed it to receive a <code>ModelSpecification</code>. The rest of the method is your usual structure of first obtaining the record to update by the given IDs, then changing its values according to what came in as a parameter, and finally, saving the changes.</p>
<p>You probably get the idea by now: since the API is using the Resource Model, we need to change input and output values for the methods and run some logic to translate between Resource Model objects and Data Model objects that EF Core can understand so that it can perform its database operations.</p>
<p>That said, here’s what the <code>PostModel</code> Action Method, handler of the <code>POST /api/Makes/{makeId}/Models</code> endpoint, should look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpPost]</span>
<span style="color:#888">// Return a `ModelSpecification`s and expect `makeId` from the URL and a `ModelSpecification` from the request payload.
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<ModelSpecification>> PostModel([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId, ModelSpecification model)
{
<span style="color:#888">// First, try to find the make specified by the incoming `makeId`.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> make = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Makes.FindAsync(makeId);
<span style="color:#888">// Respond with 404 if not found.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (make == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#888">// Build out a new `Model` entity, complete with all related data, based on
</span><span style="color:#888"></span> <span style="color:#888">// the `ModelSpecification` parameter.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> modelToCreate = <span style="color:#080;font-weight:bold">new</span> Model {
Make = make,
Name = model.Name,
ModelStyles = model.Styles.Select(style => <span style="color:#080;font-weight:bold">new</span> ModelStyle {
<span style="color:#888">// Notice how we search both body type and size by their name field.
</span><span style="color:#888"></span> <span style="color:#888">// We can do that because their names are unique.
</span><span style="color:#888"></span> BodyType = <span style="color:#00d;font-weight:bold">_</span>context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType),
Size = <span style="color:#00d;font-weight:bold">_</span>context.Sizes.Single(size => size.Name == style.Size),
ModelStyleYears = style.Years.Select(year => <span style="color:#080;font-weight:bold">new</span> ModelStyleYear {
Year = year
}).ToArray()
}).ToArray()
};
<span style="color:#888">// Add it to the DbContext.
</span><span style="color:#888"></span> <span style="color:#00d;font-weight:bold">_</span>context.Add(modelToCreate);
<span style="color:#080;font-weight:bold">try</span>
{
<span style="color:#888">// Try running the INSERTs.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.SaveChangesAsync();
}
<span style="color:#080;font-weight:bold">catch</span> (Microsoft.EntityFrameworkCore.DbUpdateException)
{
<span style="color:#888">// Return accordingly if an error happens.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> Conflict();
}
<span style="color:#888">// Get back the autogenerated ID of the record we just INSERTed.
</span><span style="color:#888"></span> model.ID = modelToCreate.ID;
<span style="color:#888">// Finally, return a 201 including a location header containing the newly
</span><span style="color:#888"></span> <span style="color:#888">// created resource's URL and the resource itself in the response payload.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> CreatedAtAction(
nameof(GetModel),
<span style="color:#080;font-weight:bold">new</span> { makeId = makeId, id = model.ID },
model
);
}
</code></pre></div><p>All that should be pretty self explanatory by now. Moving on to the <code>DeleteModel</code> method which handles the <code>DELETE /api/Makes/{makeId}/Models/{id}</code> endpoint:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#369">[HttpDelete("{id}")]</span>
<span style="color:#888">// Expect `makeId` and `id` from the URL.
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<IActionResult> DeleteModel([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId, <span style="color:#888;font-weight:bold">int</span> id)
{
<span style="color:#888">// Try to find the record identified by the ids from the URL.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> model = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Models.FirstOrDefaultAsync(m => m.MakeID == makeId && m.ID == id);
<span style="color:#888">// Respond with a 404 if we can't find it.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (model == <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">return</span> NotFound();
}
<span style="color:#888">// Mark the entity for removal and run the DELETE.
</span><span style="color:#888"></span> <span style="color:#00d;font-weight:bold">_</span>context.Models.Remove(model);
<span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.SaveChangesAsync();
<span style="color:#888">// Respond with a 204.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> NoContent();
}
</code></pre></div><p>And that’s all for that controller. Hopefully that demonstrated what it looks like to have endpoints that operate using objects other than the EF Core entities. Fire up the app with <code>dotnet run</code> and explore the Swagger UI and you’ll see the changes that we’ve made reflected in there. Try it out. Try CRUDing some vehicle models. And don’t forget to take a look at our POST endpoint specification which looks much more manageable now:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/post-model-endpoint.png" alt="POST Models endpoint"></p>
<p>Which means that you can send in something like this, for example:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Corolla"</span>,
<span style="color:#b06;font-weight:bold">"styles"</span>: [
{
<span style="color:#b06;font-weight:bold">"bodyType"</span>: <span style="color:#d20;background-color:#fff0f0">"Sedan"</span>,
<span style="color:#b06;font-weight:bold">"size"</span>: <span style="color:#d20;background-color:#fff0f0">"Compact"</span>,
<span style="color:#b06;font-weight:bold">"years"</span>: [ <span style="color:#d20;background-color:#fff0f0">"2000"</span>, <span style="color:#d20;background-color:#fff0f0">"2001"</span> ]
}
]
}
</code></pre></div><blockquote>
<p>This will work assuming you’ve created at least one make to add the vehicle model to, as well as a body type whose name is <code>Sedan</code> and a size whose name is <code>Compact</code>.</p>
</blockquote>
<blockquote>
<p>There’s also a <code>ModelExists</code> method in that controller which we don’t need anymore. You can delete it.</p>
</blockquote>
<h4 id="validation-using-built-in-data-annotations">Validation using built-in Data Annotations</h4>
<p>Depending on how “creative” you were in the previous section when trying to CRUD models, you may have run into an issue or two regarding the data that’s allowed into our database. We solve that by implementing input validation. In ASP.NET Core, the easiest way to implement validation is via Data Annotation attributes on the entities or other objects that controllers receive as request payloads. So let’s see about adding some validation to our app. Since our <code>ModelsController</code> uses the <code>ModelSpecification</code> and <code>ModelSpecificationStyle</code> Resource Models to talk to clients, let’s start there. Here’s the diff:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#000;background-color:#dfd">+using System.ComponentModel.DataAnnotations;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.ResourceModels
{
public class ModelSpecification
{
public int ID { get; set; }
<span style="color:#000;background-color:#dfd">+ [Required]
</span><span style="color:#000;background-color:#dfd"></span> public string Name { get; set; }
<span style="color:#000;background-color:#dfd">+ [Required]
</span><span style="color:#000;background-color:#dfd"></span> public ModelSpecificationStyle[] Styles { get; set; }
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><span style="color:#000;background-color:#dfd">+using System.ComponentModel.DataAnnotations;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.ResourceModels
{
public class ModelSpecificationStyle
{
<span style="color:#000;background-color:#dfd">+ [Required]
</span><span style="color:#000;background-color:#dfd"></span> public string BodyType { get; set; }
<span style="color:#000;background-color:#dfd">+ [Required]
</span><span style="color:#000;background-color:#dfd"></span> public string Size { get; set; }
<span style="color:#000;background-color:#dfd">+ [Required]
</span><span style="color:#000;background-color:#dfd">+ [MinLength(1)]
</span><span style="color:#000;background-color:#dfd"></span> public string[] Years { get; set; }
}
}
</code></pre></div><p>And just like that, we get a good amount of functionality. We use the <code>Required</code> and <code>MinLength</code> attributes from the <code>System.ComponentModel.DataAnnotations</code> namespace to specify that some fields are required, and that our <code>Years</code> array needs to contain at least one element. When the app receives a request to the PUT or POST endpoints — which are the ones that expect a <code>ModelSpecification</code> as the payload — validation kicks in. If it fails, the action method is never executed and a 400 status code is returned as a response. Try POSTing to <code>/api/Makes/{makeId}/Models</code> with a payload that violates some of these rules to see for yourself. I tried for example sending this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#080;font-weight:bold">null</span>,
<span style="color:#b06;font-weight:bold">"styles"</span>: [
{
<span style="color:#b06;font-weight:bold">"bodyType"</span>: <span style="color:#d20;background-color:#fff0f0">"Sedan"</span>,
<span style="color:#b06;font-weight:bold">"size"</span>: <span style="color:#d20;background-color:#fff0f0">"Full size"</span>,
<span style="color:#b06;font-weight:bold">"years"</span>: []
}
]
}
</code></pre></div><p>And I got back a 400 response with this payload:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"https://tools.ietf.org/html/rfc7231#section-6.5.1"</span>,
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"One or more validation errors occurred."</span>,
<span style="color:#b06;font-weight:bold">"status"</span>: <span style="color:#00d;font-weight:bold">400</span>,
<span style="color:#b06;font-weight:bold">"traceId"</span>: <span style="color:#d20;background-color:#fff0f0">"00-0fd4f00eeb9f2f458ccefc180fcfba1c-79a618f13218394b-00"</span>,
<span style="color:#b06;font-weight:bold">"errors"</span>: {
<span style="color:#b06;font-weight:bold">"Name"</span>: [
<span style="color:#d20;background-color:#fff0f0">"The Name field is required."</span>
],
<span style="color:#b06;font-weight:bold">"Styles[0].Years"</span>: [
<span style="color:#d20;background-color:#fff0f0">"The field Years must be a string or array type with a minimum length of '1'."</span>
]
}
}
</code></pre></div><p>Pretty neat, huh? With minimal effort, we have some basic validation rules in place and a pretty usable response for when errors occur.</p>
<blockquote>
<p>To learn more about model validation, including all the various validation attributes included in the framework, check the official documentation: <a href="https://docs.microsoft.com/en-us/aspnet/core/mvc/models/validation?view=aspnetcore-5.0">Model validation</a> and <a href="https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.dataannotations?view=net-5.0">System.ComponentModel.DataAnnotations Namespace</a>.</p>
</blockquote>
<h4 id="validation-using-custom-attributes">Validation using custom attributes</h4>
<p>Of course, the framework is never going to cover all possible validation scenarios with the built-in attributes. Case in point, it’d be great to validate that the <code>Years</code> array contains values that look like actual years. That is, four-character, digit-only strings. There are no validation attributes for that. So, we need to create our own. Let’s add this file into a new <code>Validations</code> directory:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Validations/ContainsYearsAttribute.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.ComponentModel.DataAnnotations</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Linq</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Runtime.CompilerServices</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Validation</span>
{
<span style="color:#888">// In the .NET Framework, attribute classes need to have their name suffixed with the word "Attribute".
</span><span style="color:#888"></span> <span style="color:#888">// Validation attributes need to inherit from `System.ComponentModel.DataAnnotations`'s `ValidationAttribute` class
</span><span style="color:#888"></span> <span style="color:#888">// and override the `IsValid` method.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ContainsYearsAttribute</span> : ValidationAttribute
{
<span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">string</span> propertyName;
<span style="color:#888">// This constructor is called by the framework when the attribute is applied to some member. In this specific
</span><span style="color:#888"></span> <span style="color:#888">// case, we define a `propertyName` parameter annotated with a `CallerMemberName` attribute. This makes it so
</span><span style="color:#888"></span> <span style="color:#888">// the framework sends in the name of the member to which our `ContainsYears` attribute is applied to.
</span><span style="color:#888"></span> <span style="color:#888">// We store the value to use it later when constructing our validation error message.
</span><span style="color:#888"></span> <span style="color:#888">// Check https://docs.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.callermembernameattribute?view=net-5.0
</span><span style="color:#888"></span> <span style="color:#888">// for more info on `CallerMemberName`.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> ContainsYearsAttribute([CallerMemberName] <span style="color:#888;font-weight:bold">string</span> propertyName = <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#080;font-weight:bold">this</span>.propertyName = propertyName;
}
<span style="color:#888">// This method is called by the framework during validation. `value` is the actual value of the field that this
</span><span style="color:#888"></span> <span style="color:#888">// attribute will validate.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">override</span> ValidationResult IsValid(<span style="color:#888;font-weight:bold">object</span> <span style="color:#080;font-weight:bold">value</span>, ValidationContext validationContext)
{
<span style="color:#888">// By only applying the validation checks when the value is not null, we make it possible for this
</span><span style="color:#888"></span> <span style="color:#888">// attribute to work on optional fields. In other words, this attribute will skip validation if there is no
</span><span style="color:#888"></span> <span style="color:#888">// value to validate.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">if</span> (<span style="color:#080;font-weight:bold">value</span> != <span style="color:#080;font-weight:bold">null</span>)
{
<span style="color:#888">// Check if all the elements of the string array are valid years. Check the `IsValidYear` method below
</span><span style="color:#888"></span> <span style="color:#888">// to see what checks are applied for each of the array elements.
</span><span style="color:#888"></span> <span style="color:#888;font-weight:bold">var</span> isValid = (<span style="color:#080;font-weight:bold">value</span> <span style="color:#080;font-weight:bold">as</span> <span style="color:#888;font-weight:bold">string</span>[]).All(IsValidYear);
<span style="color:#080;font-weight:bold">if</span> (!isValid)
{
<span style="color:#888">// If not, return an error.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> ValidationResult(GetErrorMessage());
}
}
<span style="color:#888">// Return a successful validation result if no errors were detected.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> ValidationResult.Success;
}
<span style="color:#888">// Determines if a given value is valid by making sure it's not null, nor empty, that its length is 4 and that
</span><span style="color:#888"></span> <span style="color:#888">// all its characters are digits.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">bool</span> IsValidYear(<span style="color:#888;font-weight:bold">string</span> <span style="color:#080;font-weight:bold">value</span>) =>
!String.IsNullOrEmpty(<span style="color:#080;font-weight:bold">value</span>) && <span style="color:#080;font-weight:bold">value</span>.Length == <span style="color:#00d;font-weight:bold">4</span> && <span style="color:#080;font-weight:bold">value</span>.All(Char.IsDigit);
<span style="color:#888">// Builds a user friendly error message which includes the name of the field that this validation attribute has
</span><span style="color:#888"></span> <span style="color:#888">// been applied to.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#888;font-weight:bold">string</span> GetErrorMessage() =>
<span style="color:#d20;background-color:#fff0f0">$"The {propertyName} field must be an array of strings containing four digits."</span>;
}
}
</code></pre></div><p>Check the comments in the code for more details into how that class works. Then, we apply our custom attribute to our <code>ModelSpecificationStyle</code> class in the same way that we applied the built in ones:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">using System.ComponentModel.DataAnnotations;
<span style="color:#000;background-color:#dfd">+using VehicleQuotes.Validation;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.ResourceModels
{
public class ModelSpecificationStyle
{
// ...
[Required]
[MinLength(1)]
<span style="color:#000;background-color:#dfd">+ [ContainsYears]
</span><span style="color:#000;background-color:#dfd"></span> public string[] Years { get; set; }
}
}
</code></pre></div><p>Now do a <code>dotnet run</code> and try to POST to <code>/api/Makes/{makeId}/Models</code> this payload:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Rav4"</span>,
<span style="color:#b06;font-weight:bold">"styles"</span>: [
{
<span style="color:#b06;font-weight:bold">"bodyType"</span>: <span style="color:#d20;background-color:#fff0f0">"SUV"</span>,
<span style="color:#b06;font-weight:bold">"size"</span>: <span style="color:#d20;background-color:#fff0f0">"Mid size"</span>,
<span style="color:#b06;font-weight:bold">"years"</span>: [ <span style="color:#d20;background-color:#fff0f0">"not_a_year"</span> ]
}
]
}
</code></pre></div><p>That should make the API respond with this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"https://tools.ietf.org/html/rfc7231#section-6.5.1"</span>,
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"One or more validation errors occurred."</span>,
<span style="color:#b06;font-weight:bold">"status"</span>: <span style="color:#00d;font-weight:bold">400</span>,
<span style="color:#b06;font-weight:bold">"traceId"</span>: <span style="color:#d20;background-color:#fff0f0">"00-9980325f3e388f48a5975ef382d5b137-2d55da1bb9613e4f-00"</span>,
<span style="color:#b06;font-weight:bold">"errors"</span>: {
<span style="color:#b06;font-weight:bold">"Styles[0].Years"</span>: [
<span style="color:#d20;background-color:#fff0f0">"The Years field must be an array of strings containing four numbers."</span>
]
}
}
</code></pre></div><p>That’s our custom validation attribute doing its job.</p>
<p>There’s another aspect that we could validate using a custom validation attribute. What happens if we try to POST a payload with a body type or size that doesn’t exist? These queries from the <code>PostModel</code> method would throw an <code>InvalidOperationException</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs">BodyType = <span style="color:#00d;font-weight:bold">_</span>context.BodyTypes.Single(bodyType => bodyType.Name == style.BodyType)
</code></pre></div><p>and</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs">Size = <span style="color:#00d;font-weight:bold">_</span>context.Sizes.Single(size => size.Name == style.Size)
</code></pre></div><p>They do so because we used the <code>Single</code> method, which is designed like that. It tries to find a body type or size whose name is the given value, can’t find it, and thus, throws an exception.</p>
<blockquote>
<p>If, for example, we wanted not-founds to return <code>null</code>, we could have used <code>SingleOrDefault</code> instead.</p>
</blockquote>
<p>This unhandled exception results in a response that’s quite unbecoming:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/invalid-operation-exception.png" alt="InvalidOperationException during POST"></p>
<p>So, to prevent that exception and control the error messaging, we need a couple of new validation attributes that go into the <code>body_types</code> and <code>sizes</code> tables and check if the given values exist. Here’s what one would look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Validations/VehicleBodyTypeAttribute.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.ComponentModel.DataAnnotations</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">Microsoft.EntityFrameworkCore</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Linq</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Validation</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">VehicleBodyTypeAttribute</span> : ValidationAttribute
{
<span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">override</span> ValidationResult IsValid(<span style="color:#888;font-weight:bold">object</span> <span style="color:#080;font-weight:bold">value</span>, ValidationContext validationContext)
{
<span style="color:#080;font-weight:bold">if</span> (<span style="color:#080;font-weight:bold">value</span> == <span style="color:#080;font-weight:bold">null</span>) <span style="color:#080;font-weight:bold">return</span> ValidationResult.Success;
<span style="color:#888;font-weight:bold">var</span> dbContext = validationContext.GetService(<span style="color:#080;font-weight:bold">typeof</span>(VehicleQuotesContext)) <span style="color:#080;font-weight:bold">as</span> VehicleQuotesContext;
<span style="color:#888;font-weight:bold">var</span> bodyTypes = dbContext.BodyTypes.Select(bt => bt.Name).ToList();
<span style="color:#080;font-weight:bold">if</span> (!bodyTypes.Contains(<span style="color:#080;font-weight:bold">value</span>))
{
<span style="color:#888;font-weight:bold">var</span> allowed = String.Join(<span style="color:#d20;background-color:#fff0f0">", "</span>, bodyTypes);
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> ValidationResult(
<span style="color:#d20;background-color:#fff0f0">$"Invalid vehicle body type {value}. Allowed values are {allowed}."</span>
);
}
<span style="color:#080;font-weight:bold">return</span> ValidationResult.Success;
}
}
}
</code></pre></div><blockquote>
<p>You can find the other one here: <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Validations/VehicleSizeAttribute.cs">Validations/VehicleSizeAttribute.cs</a>.</p>
</blockquote>
<p>These two are very similar to one another. The most interesting part is how we use the <code>IsValid</code> method’s second parameter (<code>ValidationContext</code>) to obtain an instance of <code>VehicleQuotesContext</code> that we can use to query the database. The rest should be pretty self-explanatory. These attributes are classes that inherit from <code>System.ComponentModel.DataAnnotations</code>’s <code>ValidationAttribute</code> and implement the <code>IsValid</code> method. The method then checks that the value under scrutiny exists in the corresponding table and if it does not, raises a validation error. The validation error includes a list of all allowed values. They can be applied to our <code>ModelSpecificationStyle</code> class like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">// ...
namespace VehicleQuotes.ResourceModels
{
public class ModelSpecificationStyle
{
[Required]
<span style="color:#000;background-color:#dfd">+ [VehicleBodyType]
</span><span style="color:#000;background-color:#dfd"></span> public string BodyType { get; set; }
[Required]
<span style="color:#000;background-color:#dfd">+ [VehicleSize]
</span><span style="color:#000;background-color:#dfd"></span> public string Size { get; set; }
//...
}
}
</code></pre></div><p>Now, a request like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Rav4"</span>,
<span style="color:#b06;font-weight:bold">"styles"</span>: [
{
<span style="color:#b06;font-weight:bold">"bodyType"</span>: <span style="color:#d20;background-color:#fff0f0">"not_a_body_type"</span>,
<span style="color:#b06;font-weight:bold">"size"</span>: <span style="color:#d20;background-color:#fff0f0">"Mid size"</span>,
<span style="color:#b06;font-weight:bold">"years"</span>: [ <span style="color:#d20;background-color:#fff0f0">"2000"</span> ]
}
]
}
</code></pre></div><p>Produces a response like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"https://tools.ietf.org/html/rfc7231#section-6.5.1"</span>,
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"One or more validation errors occurred."</span>,
<span style="color:#b06;font-weight:bold">"status"</span>: <span style="color:#00d;font-weight:bold">400</span>,
<span style="color:#b06;font-weight:bold">"traceId"</span>: <span style="color:#d20;background-color:#fff0f0">"00-9ad59a7aff60944ab54c19a73be73cc7-eeabafe03df74e40-00"</span>,
<span style="color:#b06;font-weight:bold">"errors"</span>: {
<span style="color:#b06;font-weight:bold">"Styles[0].BodyType"</span>: [
<span style="color:#d20;background-color:#fff0f0">"Invalid vehicle body type not_a_body_type. Allowed values are Coupe, Sedan, Convertible, Hatchback, SUV, Truck."</span>
]
}
}
</code></pre></div><h4 id="implementing-endpoints-for-quote-rules-and-overrides">Implementing endpoints for quote rules and overrides</h4>
<p>At this point we’ve explored many of the most common features available to us for developing Web APIs. So much so that implementing the next two pieces of functionality for our app doesn’t really introduce any new concepts. So, I wont discuss that here in great detail.</p>
<p>Feel free to browse the source code <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api">on GitHub</a> if you want though. These are the relevant files:</p>
<ul>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Controllers/QuoteOverridesController.cs">Controllers/QuoteOverridesController.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Controllers/QuoteRulesController.cs">Controllers/QuoteRulesController.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Models/QuoteOverride.cs">Models/QuoteOverride.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Models/QuoteRule.cs">Models/QuoteRule.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/ResourceModels/QuoteOverrideSpecification.cs">ResourceModels/QuoteOverrideSpecification.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Validations/FeatureTypeAttribute.cs">Validation/FeatureTypeAttribute.cs</a></li>
<li><a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/Migrations/20210627204444_AddQuoteRulesAndOverridesTables.cs">Migrations/20210627204444_AddQuoteRulesAndOverridesTables.cs</a></li>
</ul>
<p>The <code>FeatureTypeAttribute</code> class is interesting in that it provides another example of a validation attribute. This time is one that makes sure the value being validated is included in an array of strings that’s defined literally in the code.</p>
<p>Other than that, it’s all stuff we’ve already covered: models, migrations, scaffolding controllers, custom routes, resource models, etc.</p>
<p>If you are following along, be sure to add those files and run a <code>dotnet ef database update</code> to apply the migration.</p>
<h4 id="implementing-the-quote-model">Implementing the quote model</h4>
<p>Let’s now start implementing the main capability of our app: calculating quotes for vehicles. Let’s start with the <code>Quote</code> entity. This is what the new <code>Models/Quote.cs</code> file containing the entity class will look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">// Models/Quote.cs
</span><span style="color:#888"></span><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.ComponentModel.DataAnnotations.Schema</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Quote</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> ID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#888">// Directly tie this quote record to a specific vehicle that we have
</span><span style="color:#888"></span> <span style="color:#888">// registered in our db, if we have it.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int?</span> ModelStyleYearID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#888">// If we don't have the specific vehicle in our db, then store the
</span><span style="color:#888"></span> <span style="color:#888">// vehicle model details independently.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Year { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Make { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Model { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> BodyTypeID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> SizeID { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> ItMoves { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasAllWheels { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasAlloyWheels { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasAllTires { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasKey { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasTitle { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> RequiresPickup { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasEngine { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasTransmission { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">bool</span> HasCompleteInterior { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">int</span> OfferedQuote { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> <span style="color:#888;font-weight:bold">string</span> Message { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> DateTime CreatedAt { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> ModelStyleYear ModelStyleYear { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> BodyType BodyType { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
<span style="color:#080;font-weight:bold">public</span> Size Size { <span style="color:#080;font-weight:bold">get</span>; <span style="color:#080;font-weight:bold">set</span>; }
}
}
</code></pre></div><p>This should be pretty familiar by now. It’s a plain old class that defines a number of properties, one for each of the fields in the resulting table and a few navigation properties that serve to access related data.</p>
<p>The only aspect worth noting is that we’ve defined the <code>ModelStyleYearID</code> property as a nullable integer (with <code>int?</code>). This is because, like we discussed at the beginning, the foreign key from <code>quotes</code> to <code>vehicle_style_years</code> is actually optional. The reason being that we may receive a quote request for a vehicle that we don’t have registered in our database. We need to be able to support quoting those vehicles too, so if we don’t have the requested vehicle registered, then that foreign key will stay unpopulated and we’ll rely on the other fields (i.e. <code>Year</code>, <code>Make</code>, <code>Model</code>, <code>BodyTypeID</code> and <code>SizeID</code>) to identify the vehicle and calculate the quote for it.</p>
<h4 id="using-dependency-injection">Using Dependency Injection</h4>
<p>So far we’ve been putting a lot of logic in our controllers. That’s generally not ideal, but fine as long as the logic is simple. The problem is that a design like that can quickly become a hindrance for maintainability and testing as our application grows more complex. For the logic that calculates a quote, we’d be better served by implementing it in its own class, outside of the controller that should only care about defining endpoints and handling HTTP concerns. Then, the controller can be given access to that class and delegate to it all the quote calculation logic. Thankfully, ASP.NET Core includes an IoC container by default, which allows us to use Dependency Injection to solve these kinds of problems. Let’s see what that looks like.</p>
<p>For working with quotes, we want to offer two endpoints:</p>
<ol>
<li>A <code>POST api/Quotes</code> that captures the vehicle information, calculates the quote, keeps record of the request, and responds with the calculated value.</li>
<li>A <code>GET api/Quotes</code> that returns all the currently registered quotes in the system.</li>
</ol>
<p>Using the Dependency Injection capabilities, a controller that implements those two could look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Collections.Generic</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Threading.Tasks</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">Microsoft.AspNetCore.Mvc</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.ResourceModels</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Services</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Controllers</span>
{
<span style="color:#369"> [Route("api/[controller]</span><span style="color:#d20;background-color:#fff0f0">")]
</span><span style="color:#d20;background-color:#fff0f0"></span><span style="color:#369"> [ApiController]</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">QuotesController</span> : ControllerBase
{
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">readonly</span> QuoteService <span style="color:#00d;font-weight:bold">_</span>service;
<span style="color:#888">// When intiating the request processing logic, the framework recognizes
</span><span style="color:#888"></span> <span style="color:#888">// that this controller has a dependency on QuoteService and expects an
</span><span style="color:#888"></span> <span style="color:#888">// instance of it to be injected via the constructor. The framework then
</span><span style="color:#888"></span> <span style="color:#888">// does what it needs to do in order to provide that dependency.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> QuotesController(QuoteService service)
{
<span style="color:#00d;font-weight:bold">_</span>service = service;
}
<span style="color:#888">// GET: api/Quotes
</span><span style="color:#888"></span><span style="color:#369"> [HttpGet]</span>
<span style="color:#888">// This method returns a collection of a new resource model instead of just the `Quote` entity directly.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<IEnumerable<SubmittedQuoteRequest>>> GetAll()
{
<span style="color:#888">// Instead of directly implementing the logic in this method, we call on
</span><span style="color:#888"></span> <span style="color:#888">// the service class and let it take care of the rest.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>service.GetAllQuotes();
}
<span style="color:#888">// POST: api/Quotes
</span><span style="color:#888"></span><span style="color:#369"> [HttpPost]</span>
<span style="color:#888">// This method receives as a paramater a `QuoteRequest` of just the `Quote` entity directly.
</span><span style="color:#888"></span> <span style="color:#888">// That way callers of this endpoint don't need to be exposed to the details of our data model implementation.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<SubmittedQuoteRequest>> Post(QuoteRequest request)
{
<span style="color:#888">// Instead of directly implementing the logic in this method, we call on
</span><span style="color:#888"></span> <span style="color:#888">// the service class and let it take care of the rest.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>service.CalculateQuote(request);
}
}
}
</code></pre></div><p>As you can see, we’ve once again opted to abstract away clients from the implementation details of our data model and used Resource Models for the API contract instead of the <code>Quote</code> entity directly. We have one for input data that’s called <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/ResourceModels/QuoteRequest.cs"><code>QuoteRequest</code></a> and another one for output: <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/blob/master/ResourceModels/SubmittedQuoteRequest.cs"><code>SubmittedQuoteRequest</code></a>. Not very remarkable by themselves, but feel free to explore the source code in <a href="https://github.com/megakevin/end-point-blog-dotnet-5-web-api/tree/master/ResourceModels">the GitHub repo</a>.</p>
<p>This controller has a dependency on <code>QuoteService</code>, which it uses to perform all of the necessary logic. This class is not defined yet so let’s do that next:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Collections.Generic</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Linq</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Threading.Tasks</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">Microsoft.EntityFrameworkCore</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Models</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.ResourceModels</span>;
<span style="color:#080;font-weight:bold">namespace</span> <span style="color:#b06;font-weight:bold">VehicleQuotes.Services</span>
{
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">QuoteService</span>
{
<span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">readonly</span> VehicleQuotesContext <span style="color:#00d;font-weight:bold">_</span>context;
<span style="color:#888">// This constructor defines a dependency on VehicleQuotesContext, similar to most of our controllers.
</span><span style="color:#888"></span> <span style="color:#888">// Via the built in dependency injection features, the framework makes sure to provide this parameter when
</span><span style="color:#888"></span> <span style="color:#888">// creating new instances of this class.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> QuoteService(VehicleQuotesContext context)
{
<span style="color:#00d;font-weight:bold">_</span>context = context;
}
<span style="color:#888">// This method takes all the records from the `quotes` table and constructs `SubmittedQuoteRequest`s with them.
</span><span style="color:#888"></span> <span style="color:#888">// Then returns that as a list.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<List<SubmittedQuoteRequest>> GetAllQuotes()
{
<span style="color:#888;font-weight:bold">var</span> quotesToReturn = <span style="color:#00d;font-weight:bold">_</span>context.Quotes.Select(q => <span style="color:#080;font-weight:bold">new</span> SubmittedQuoteRequest
{
ID = q.ID,
CreatedAt = q.CreatedAt,
OfferedQuote = q.OfferedQuote,
Message = q.Message,
Year = q.Year,
Make = q.Make,
Model = q.Model,
BodyType = q.BodyType.Name,
Size = q.Size.Name,
ItMoves = q.ItMoves,
HasAllWheels = q.HasAllWheels,
HasAlloyWheels = q.HasAlloyWheels,
HasAllTires = q.HasAllTires,
HasKey = q.HasKey,
HasTitle = q.HasTitle,
RequiresPickup = q.RequiresPickup,
HasEngine = q.HasEngine,
HasTransmission = q.HasTransmission,
HasCompleteInterior = q.HasCompleteInterior,
});
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> quotesToReturn.ToListAsync();
}
<span style="color:#888">// This method takes an incoming `QuoteRequest` and calculates a quote based on the vehicle described by it.
</span><span style="color:#888"></span> <span style="color:#888">// To calculate this quote, it looks for any overrides before trying to use the currently existing rules defined
</span><span style="color:#888"></span> <span style="color:#888">// in the `quote_rules` table. It also stores a record on the `quotes` table with all the incoming data and the
</span><span style="color:#888"></span> <span style="color:#888">// quote calculation result. It returns back the quote value as well as a message explaining the conditions of
</span><span style="color:#888"></span> <span style="color:#888">// the quote.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
{
<span style="color:#888;font-weight:bold">var</span> response = <span style="color:#080;font-weight:bold">this</span>.CreateResponse(request);
<span style="color:#888;font-weight:bold">var</span> quoteToStore = <span style="color:#080;font-weight:bold">await</span> <span style="color:#080;font-weight:bold">this</span>.CreateQuote(request);
<span style="color:#888;font-weight:bold">var</span> requestedModelStyleYear = <span style="color:#080;font-weight:bold">await</span> <span style="color:#080;font-weight:bold">this</span>.FindModelStyleYear(request);
QuoteOverride quoteOverride = <span style="color:#080;font-weight:bold">null</span>;
<span style="color:#080;font-weight:bold">if</span> (requestedModelStyleYear != <span style="color:#080;font-weight:bold">null</span>)
{
quoteToStore.ModelStyleYear = requestedModelStyleYear;
quoteOverride = <span style="color:#080;font-weight:bold">await</span> <span style="color:#080;font-weight:bold">this</span>.FindQuoteOverride(requestedModelStyleYear);
<span style="color:#080;font-weight:bold">if</span> (quoteOverride != <span style="color:#080;font-weight:bold">null</span>)
{
response.OfferedQuote = quoteOverride.Price;
}
}
<span style="color:#080;font-weight:bold">if</span> (quoteOverride == <span style="color:#080;font-weight:bold">null</span>)
{
response.OfferedQuote = <span style="color:#080;font-weight:bold">await</span> <span style="color:#080;font-weight:bold">this</span>.CalculateOfferedQuote(request);
}
<span style="color:#080;font-weight:bold">if</span> (requestedModelStyleYear == <span style="color:#080;font-weight:bold">null</span>)
{
response.Message = <span style="color:#d20;background-color:#fff0f0">"Offer subject to change upon vehicle inspection."</span>;
}
quoteToStore.OfferedQuote = response.OfferedQuote;
quoteToStore.Message = response.Message;
<span style="color:#00d;font-weight:bold">_</span>context.Quotes.Add(quoteToStore);
<span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.SaveChangesAsync();
response.ID = quoteToStore.ID;
response.CreatedAt = quoteToStore.CreatedAt;
<span style="color:#080;font-weight:bold">return</span> response;
}
<span style="color:#888">// Creates a `SubmittedQuoteRequest`, initialized with default values, using the data from the incoming
</span><span style="color:#888"></span> <span style="color:#888">// `QuoteRequest`. `SubmittedQuoteRequest` is what gets returned in the response payload of the quote endpoints.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> SubmittedQuoteRequest CreateResponse(QuoteRequest request)
{
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> SubmittedQuoteRequest
{
OfferedQuote = <span style="color:#00d;font-weight:bold">0</span>,
Message = <span style="color:#d20;background-color:#fff0f0">"This is our final offer."</span>,
Year = request.Year,
Make = request.Make,
Model = request.Model,
BodyType = request.BodyType,
Size = request.Size,
ItMoves = request.ItMoves,
HasAllWheels = request.HasAllWheels,
HasAlloyWheels = request.HasAlloyWheels,
HasAllTires = request.HasAllTires,
HasKey = request.HasKey,
HasTitle = request.HasTitle,
RequiresPickup = request.RequiresPickup,
HasEngine = request.HasEngine,
HasTransmission = request.HasTransmission,
HasCompleteInterior = request.HasCompleteInterior,
};
}
<span style="color:#888">// Creates a `Quote` based on the data from the incoming `QuoteRequest`. This is the object that gets eventually
</span><span style="color:#888"></span> <span style="color:#888">// stored in the database.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">async</span> Task<Quote> CreateQuote(QuoteRequest request)
{
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">new</span> Quote
{
Year = request.Year,
Make = request.Make,
Model = request.Model,
BodyTypeID = (<span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.BodyTypes.SingleAsync(bt => bt.Name == request.BodyType)).ID,
SizeID = (<span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.Sizes.SingleAsync(s => s.Name == request.Size)).ID,
ItMoves = request.ItMoves,
HasAllWheels = request.HasAllWheels,
HasAlloyWheels = request.HasAlloyWheels,
HasAllTires = request.HasAllTires,
HasKey = request.HasKey,
HasTitle = request.HasTitle,
RequiresPickup = request.RequiresPickup,
HasEngine = request.HasEngine,
HasTransmission = request.HasTransmission,
HasCompleteInterior = request.HasCompleteInterior,
CreatedAt = DateTime.Now
};
}
<span style="color:#888">// Tries to find a registered vehicle that matches the one for which the quote is currently being requested.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">async</span> Task<ModelStyleYear> FindModelStyleYear(QuoteRequest request)
{
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.ModelStyleYears.FirstOrDefaultAsync(msy =>
msy.Year == request.Year &&
msy.ModelStyle.Model.Make.Name == request.Make &&
msy.ModelStyle.Model.Name == request.Model &&
msy.ModelStyle.BodyType.Name == request.BodyType &&
msy.ModelStyle.Size.Name == request.Size
);
}
<span style="color:#888">// Tries to find an override for the vehicle for which the quote is currently being requested.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">async</span> Task<QuoteOverride> FindQuoteOverride(ModelStyleYear modelStyleYear)
{
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.QuoteOverides
.FirstOrDefaultAsync(qo => qo.ModelStyleYear == modelStyleYear);
}
<span style="color:#888">// Uses the rules stored in the `quote_rules` table to calculate how much money to offer for the vehicle
</span><span style="color:#888"></span> <span style="color:#888">// described in the incoming `QuoteRequest`.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">private</span> <span style="color:#080;font-weight:bold">async</span> Task<<span style="color:#888;font-weight:bold">int</span>> CalculateOfferedQuote(QuoteRequest request)
{
<span style="color:#888;font-weight:bold">var</span> rules = <span style="color:#080;font-weight:bold">await</span> <span style="color:#00d;font-weight:bold">_</span>context.QuoteRules.ToListAsync();
<span style="color:#888">// Given a vehicle feature type, find a rule that applies to that feature type and has the value that
</span><span style="color:#888"></span> <span style="color:#888">// matches the condition of the incoming vehicle being quoted.
</span><span style="color:#888"></span> Func<<span style="color:#888;font-weight:bold">string</span>, QuoteRule> theMatchingRule = featureType =>
rules.FirstOrDefault(r =>
r.FeatureType == featureType &&
r.FeatureValue == request[featureType]
);
<span style="color:#888">// For each vehicle feature that we care about, sum up the the monetary values of all the rules that match
</span><span style="color:#888"></span> <span style="color:#888">// the given vehicle condition.
</span><span style="color:#888"></span> <span style="color:#080;font-weight:bold">return</span> QuoteRule.FeatureTypes.All
.Select(theMatchingRule)
.Where(r => r != <span style="color:#080;font-weight:bold">null</span>)
.Sum(r => r.PriceModifier);
}
}
}
</code></pre></div><p>Finally, we need to tell the framework that this class is available for Dependency Injection. Similarly to how we did with our <code>VehicleQuotesContext</code>, we do so in the <code>Startup.cs</code> file’s <code>ConfigureServices</code> method. Just add this line at the top:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs">services.AddScoped<Services.QuoteService>();
</code></pre></div><blockquote>
<p>The core tenet of Inversion of Control is to depend on abstractions, not on implementations. So ideally, we would not have our controller directly call for a <code>QuoteService</code> instance. Instead, we would have it reference an abstraction, e.g. an interface like <code>IQuoteService</code>. The statement on <code>Startup.cs</code> would then look like this instead: <code>services.AddScoped<Services.IQuoteService, Services.QuoteService>();</code>.</p>
<p>This is important because it would allow us to unit test the component that depends on our service class (i.e. the controller in this case) by passing it a <a href="https://en.wikipedia.org/wiki/Mock_object">mock object</a> — one that also implements <code>IQuoteService</code> but does not really implement all the functionality of the actual <code>QuoteService</code> class. Since the controller only knows about the interface (that is, it “depends on an abstraction”), the actual object that we give it as a dependency doesn’t matter to it, as long as it implements that interface. This ability to inject mocks as dependencies is invaluable during testing. Testing is beyond the scope of this article though, so I’ll stick with the simpler approach with a static dependency on a concrete class. Know that this is not a good practice when it comes to actual production systems.</p>
</blockquote>
<p>And that’s all it takes. Once you add a few rules via <code>POST /api/QuoteRules</code>, you should be able to get some vehicles quoted with <code>POST /api/Quotes</code>. And also see what the system has stored via <code>GET /api/Quotes</code>.</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/a-quote.png" alt="A Quote"></p>
<p>And that’s all the functionality that we set out to build into our REST API! There are a few other neat things that I thought I’d include though.</p>
<h4 id="adding-seed-data-for-lookup-tables">Adding seed data for lookup tables</h4>
<p>Our vehicle size and body type data isn’t meant to really change much. In fact, we could even preload that data when our application starts. EF Core provides a data seeding feature that we can access via configurations on the <code>DbContext</code> itself. For our case, we could add this method to our <code>VehicleQuotesContext</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">protected</span> <span style="color:#080;font-weight:bold">override</span> <span style="color:#080;font-weight:bold">void</span> OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Size>().HasData(
<span style="color:#080;font-weight:bold">new</span> Size { ID = <span style="color:#00d;font-weight:bold">1</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Subcompact"</span> },
<span style="color:#080;font-weight:bold">new</span> Size { ID = <span style="color:#00d;font-weight:bold">2</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Compact"</span> },
<span style="color:#080;font-weight:bold">new</span> Size { ID = <span style="color:#00d;font-weight:bold">3</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Mid Size"</span> },
<span style="color:#080;font-weight:bold">new</span> Size { ID = <span style="color:#00d;font-weight:bold">5</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Full Size"</span> }
);
modelBuilder.Entity<BodyType>().HasData(
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">1</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Coupe"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">2</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Sedan"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">3</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Hatchback"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">4</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Wagon"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">5</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Convertible"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">6</span>, Name = <span style="color:#d20;background-color:#fff0f0">"SUV"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">7</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Truck"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">8</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Mini Van"</span> },
<span style="color:#080;font-weight:bold">new</span> BodyType { ID = <span style="color:#00d;font-weight:bold">9</span>, Name = <span style="color:#d20;background-color:#fff0f0">"Roadster"</span> }
);
}
</code></pre></div><p><a href="https://docs.microsoft.com/en-us/ef/core/modeling/#use-fluent-api-to-configure-a-model"><code>OnModelCreating</code></a> is a hook that we can define to run some code at the time the model is being created for the first time. Here, we’re using it to seed some data. In order to apply that, a migration needs to be created and executed. If you’ve added some data to the database, be sure to wipe it before running the migration so that we don’t run into unique constraint violations. Here are the migrations:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ dotnet ef migrations add AddSeedDataForSizesAndBodyTypes
$ dotnet ef database update
</code></pre></div><p>After that’s done, it no longer makes sense to allow creating, updating, deleting and fetching individual sizes and body types, so I would delete those endpoints from the respective controllers.</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/body-types-get-all-only.png" alt="Body Types, GET all only"></p>
<p><img src="/blog/2021/07/dotnet-5-web-api/sizes-get-all-only.png" alt="Sizes, GET all only"></p>
<blockquote>
<p>There are other options for data seeding in EF Core. Take a look: <a href="https://docs.microsoft.com/en-us/ef/core/modeling/data-seeding">Data Seeding</a>.</p>
</blockquote>
<h4 id="improving-the-swagger-ui-via-xml-comments">Improving the Swagger UI via XML comments</h4>
<p>Our current auto-generated Swagger UI is pretty awesome. Especially considering that we got it for free. It’s a little lacking when it comes to more documentation about specific endpoint summaries or expected responses. The good news is that there’s a way to leverage <a href="https://docs.microsoft.com/en-us/dotnet/csharp/codedoc">C# XML Comments</a> in order to improve the Swagger UI.</p>
<p>We can add support for that by configuring our project to produce, at build time, an XML file with the docs that we write. In order to do so, we need to update the <code>VehicleQuotes.csproj</code> like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff"><Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<!-- ... -->
<span style="color:#000;background-color:#dfd">+ <GenerateDocumentationFile>true</GenerateDocumentationFile>
</span><span style="color:#000;background-color:#dfd">+ <NoWarn>$(NoWarn);1591</NoWarn>
</span><span style="color:#000;background-color:#dfd"></span> </PropertyGroup>
<!-- ... -->
</Project>
</code></pre></div><p><code>GenerateDocumentationFile</code> is the flag that tells the .NET 5 build tools to generate the documentation file. The <code>NoWarn</code> element prevents our build output from getting cluttered with a lot of warnings saying that some classes and methods are not properly documented. We don’t want that because we just want to write enough documentation for the Swagger UI. And that includes only the controllers.</p>
<p>You can run <code>dotnet build</code> and look for the new file in <code>bin/Debug/net5.0/VehicleQuotes.xml</code>.</p>
<p>Then, we need to update <code>Startup.cs</code>. First we need to add the following <code>using</code> statements:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.IO</span>;
<span style="color:#080;font-weight:bold">using</span> <span style="color:#b06;font-weight:bold">System.Reflection</span>;
</code></pre></div><p>And add the following code to the <code>ConfigureServices</code> method on <code>Startup.cs</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">public void ConfigureServices(IServiceCollection services)
{
// ...
services.AddSwaggerGen(c =>
{
c.SwaggerDoc("v1", new OpenApiInfo { Title = "VehicleQuotes", Version = "v1" });
<span style="color:#000;background-color:#dfd">+ c.IncludeXmlComments(
</span><span style="color:#000;background-color:#dfd">+ Path.Combine(
</span><span style="color:#000;background-color:#dfd">+ AppContext.BaseDirectory,
</span><span style="color:#000;background-color:#dfd">+ $"{Assembly.GetExecutingAssembly().GetName().Name}.xml"
</span><span style="color:#000;background-color:#dfd">+ )
</span><span style="color:#000;background-color:#dfd"></span> );
});
// ...
}
</code></pre></div><p>This makes it so the <code>SwaggerGen</code> service knows to look for the XML documentation file when building up the Open API specification file used for generating the Swagger UI.</p>
<p>Now that all of that is set up, we can actually write some XML comments and attributes that will enhance our Swagger UI. As an example, put this on top of <code>ModelsController</code>’s <code>Post</code> method:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-cs" data-lang="cs"><span style="color:#888">/// <summary>
</span><span style="color:#888">/// Creates a new vehicle model for the given make.
</span><span style="color:#888">/// </summary>
</span><span style="color:#888">/// <param name="makeId">The ID of the vehicle make to add the model to.</param>
</span><span style="color:#888">/// <param name="model">The data to create the new model with.</param>
</span><span style="color:#888">/// <response code="201">When the request is valid.</response>
</span><span style="color:#888">/// <response code="404">When the specified vehicle make does not exist.</response>
</span><span style="color:#888">/// <response code="409">When there's already another model in the same make with the same name.</response>
</span><span style="color:#888"></span><span style="color:#369">[HttpPost]</span>
<span style="color:#369">[ProducesResponseType(StatusCodes.Status201Created)]</span>
<span style="color:#369">[ProducesResponseType(StatusCodes.Status404NotFound)]</span>
<span style="color:#369">[ProducesResponseType(StatusCodes.Status409Conflict)]</span>
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">async</span> Task<ActionResult<ModelSpecification>> Post([FromRoute] <span style="color:#888;font-weight:bold">int</span> makeId, ModelSpecification model)
{
<span style="color:#888">// ...
</span><span style="color:#888"></span>}
</code></pre></div><p>Then, the Swagger UI now looks like this for this endpoint:</p>
<p><img src="/blog/2021/07/dotnet-5-web-api/fully-documented-post-models.png" alt="Fully documented POST Models endpoint"></p>
<h4 id="configuring-the-app-via-settings-files-and-environment-variables">Configuring the app via settings files and environment variables</h4>
<p>Another aspect that’s important to web applications is having them be configurable via things like configuration files or environment variables. The framework already has provision for this, we just need to use it. I’m talking about the <code>appsettings</code> files.</p>
<p>We have two of them created for us by default: <code>appsettings.json</code> which is applied in all environments, and <code>appsettings.Development.json</code> that is applied only under development environments. The environment is given by the <code>ASPNETCORE_ENVIRONMENT</code> environment variable, and it can be set to either <code>Development</code>, <code>Staging</code>, or <code>Production</code> by default. That means that if we had, for example, an <code>appsettings.Staging.json</code> file, the settings defined within would be loaded if the <code>ASPNETCORE_ENVIRONMENT</code> environment variable were set to <code>Staging</code>. You get the idea.</p>
<blockquote>
<p>You can learn more about <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-5.0">configuration</a> and <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/environments?view=aspnetcore-5.0">environments</a> in the official documentation.</p>
</blockquote>
<p>Anyway, let’s add a new setting on <code>appsettings.json</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">{
// ...
<span style="color:#000;background-color:#dfd">+ "DefaultOffer": 77
</span><span style="color:#000;background-color:#dfd"></span>}
</code></pre></div><p>We’ll use this setting to give default offers when we’re not able to calculate appropriate quotes for vehicles. This can happen if we don’t have rules, or if the ones we have don’t match any of the incoming vehicle features or if for some other reason the final sum ends up in zero or negative number. We can use this setting in our <code>QuoteService</code> like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-diff" data-lang="diff">// ...
<span style="color:#000;background-color:#dfd">+using Microsoft.Extensions.Configuration;
</span><span style="color:#000;background-color:#dfd"></span>
namespace VehicleQuotes.Services
{
public class QuoteService
{
// ...
<span style="color:#000;background-color:#dfd">+ private readonly IConfiguration _configuration;
</span><span style="color:#000;background-color:#dfd"></span>
<span style="color:#000;background-color:#fdd">- public QuoteService(VehicleQuotesContext context)
</span><span style="color:#000;background-color:#fdd"></span><span style="color:#000;background-color:#dfd">+ public QuoteService(VehicleQuotesContext context, IConfiguration configuration)
</span><span style="color:#000;background-color:#dfd"></span> {
_context = context;
<span style="color:#000;background-color:#dfd">+ _configuration = configuration;
</span><span style="color:#000;background-color:#dfd"></span> }
// ...
public async Task<SubmittedQuoteRequest> CalculateQuote(QuoteRequest request)
{
// ...
<span style="color:#000;background-color:#dfd">+ if (response.OfferedQuote <= 0)
</span><span style="color:#000;background-color:#dfd">+ {
</span><span style="color:#000;background-color:#dfd">+ response.OfferedQuote = _configuration.GetValue<int>("DefaultOffer", 0);
</span><span style="color:#000;background-color:#dfd">+ }
</span><span style="color:#000;background-color:#dfd"></span>
quoteToStore.OfferedQuote = response.OfferedQuote;
// ...
}
// ...
}
}
</code></pre></div><p>Here, we’ve added a new parameter to the constructor to specify that <code>VehicleQuotesContext</code> has a dependency on <code>IConfiguration</code>. This prompts the framework to provide an instance of that when instantiating the class. We can use that instance to access the settings that we defined in the <code>appsettings.json</code> file via its <code>GetValue</code> method, like I demonstrated above.</p>
<p>The value of the settings in <code>appsettings.json</code> can be overridden by environment variables as well. On Linux, for example, we can run the app and set an environment value with a line like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ <span style="color:#369">DefaultOffer</span>=<span style="color:#00d;font-weight:bold">123</span> dotnet run
</code></pre></div><p>This will make the application use <code>123</code> instead of <code>77</code> when it comes to the <code>DefaultOffer</code> setting. This flexibility is great from a DevOps perspective. And we had to do minimal work in order to get that going.</p>
<h3 id="thats-all-for-now">That’s all for now</h3>
<p>And that’s it! In this article we’ve gone through many of the features offered in <a href="https://docs.microsoft.com/en-us/dotnet/core/dotnet-five">.NET 5</a>, <a href="https://docs.microsoft.com/en-us/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-5.0">ASP.NET Core</a>, and <a href="https://docs.microsoft.com/en-us/ef/core/">Entity Framework Core</a> to support some of the most common use cases when it comes to developing Web API applications.</p>
<p>We’ve installed .NET 5 and created an ASP.NET Core Web API project with EF Core and a few bells and whistles, created controllers to support many different endpoints, played a little bit with routes and response codes, created and built upon a data model and updated a database via entities and migrations, implemented database constraints using unique indexes, implemented input validation using both built-in and custom validation attributes, implemented resource models as DTOs for defining the contract of some of our API endpoints, tapped into the built-in dependency injection capabilities, explored and improved the auto-generated Swagger UI, added seed data for our database, learned about configuration via settings files and environment variables.</p>
<p>.NET 5 is looking great.</p>
<h3 id="table-of-contents">Table of contents</h3>
<ul>
<li><a href="#what-were-building">What we’re building</a>
<ul>
<li><a href="#the-demo-application">The demo application</a></li>
<li><a href="#the-data-model">The data model</a></li>
</ul>
</li>
<li><a href="#the-development-environment">The development environment</a>
<ul>
<li><a href="#setting-up-the-postgresql-database-with-docker">Setting up the PostgreSQL database with Docker</a></li>
<li><a href="#installing-the-net-5-sdk">Installing the .NET 5 SDK</a></li>
</ul>
</li>
<li><a href="#setting-up-the-project">Setting up the project</a>
<ul>
<li><a href="#creating-our-aspnet-core-rest-api-project">Creating our ASP.NET Core REST API project</a></li>
<li><a href="#installing-packages-well-need">Installing packages we’ll need</a></li>
<li><a href="#connecting-to-the-database-and-performing-initial-app-configuration">Connecting to the database and performing initial app configuration</a></li>
</ul>
</li>
<li><a href="#building-the-application">Building the application</a>
<ul>
<li><a href="#creating-model-entities-migrations-and-updating-the-database">Creating model entities, migrations and updating the database</a></li>
<li><a href="#creating-controllers-for-cruding-our-tables">Creating controllers for CRUDing our tables</a></li>
<li><a href="#adding-unique-constraints-via-indexes">Adding unique constraints via indexes</a></li>
<li><a href="#responding-with-specific-http-error-codes-409-conflict">Responding with specific HTTP error codes (409 Conflict)</a></li>
<li><a href="#adding-a-more-complex-entity-to-the-model">Adding a more complex entity to the model</a></li>
<li><a href="#adding-composite-unique-indexes">Adding composite unique indexes</a></li>
<li><a href="#adding-controllers-with-custom-routes">Adding controllers with custom routes</a></li>
<li><a href="#using-resource-models-as-dtos-for-controllers">Using resource models as DTOs for controllers</a></li>
<li><a href="#validation-using-built-in-data-annotations">Validation using built-in Data Annotations</a></li>
<li><a href="#validation-using-custom-attributes">Validation using custom attributes</a></li>
<li><a href="#implementing-endpoints-for-quote-rules-and-overrides">Implementing endpoints for quote rules and overrides</a></li>
<li><a href="#implementing-the-quote-model">Implementing the quote model</a></li>
<li><a href="#using-dependency-injection">Using Dependency Injection</a></li>
<li><a href="#adding-seed-data-for-lookup-tables">Adding seed data for lookup tables</a></li>
<li><a href="#improving-the-swagger-ui-via-xml-comments">Improving the Swagger UI via XML comments</a></li>
<li><a href="#configuring-the-app-via-settings-files-and-environment-variables">Configuring the app via settings files and environment variables</a></li>
</ul>
</li>
<li><a href="#thats-all-for-now">That’s all for now</a></li>
</ul>
Availity: An API for Health Insurancehttps://www.endpointdev.com/blog/2020/11/availity-api-health-insurance/2020-11-16T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2020/11/availity-api-health-insurance/banner.jpg" alt="Stethoscope">
<a href="https://flic.kr/p/25e3v5L">Photo</a> by <a href="https://www.flickr.com/people/143707811@N07/">Sergio Santos</a>, used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a>, cropped from original.</p>
<p>I have been working on a tele-therapy application for a client in the health care industry over the past few months and had the opportunity to do some interesting work in the area of health insurance coverages and claims.</p>
<p>I was tasked with creating an integration of the <a href="https://www.availity.com/">Availity</a> API for insurance coverages which provides the ability to make requests for details about a patient’s health insurance coverage and returns responses containing information like the patient’s primary care doctor, their copay amounts, and their deductibles.</p>
<p>The ability to query this health insurance information from an API in an automated fashion helps streamline the process of billing clients by validating their health insurance details and also determining what a patient’s financial responsibility will be for their online therapy sessions. Availity provides information for over 11,000 insurance companies; a <a href="https://apps.availity.com/public-web/payerlist-ui/payerlist-ui/#/">full list of supported payers</a> is available on their site via a web interface and a downloadable CSV file.</p>
<p>Availity provides both REST and SOAP APIs in addition to a batch processing system that functions over SFTP. For the purposes of this article I will be focusing on the REST API which was the primary focus of my work for this project.</p>
<p>The <a href="https://developer.availity.com/partner/documentation">developer documentation</a> for the REST API was mostly self-serve; after signing up for an account on the Availity site I was able to rely on their publicly available documentation for all of the API details that I needed to start making requests.</p>
<p>Unfortunately, the development process for integrating the SOAP API was not nearly as smooth; the <a href="https://developer.availity.com/partner/">SOAP APIs</a> link on Availity’s main developer portal page currently comes up blank, and I had to register a separate account in order to create a support request to obtain documentation on the SOAP API. Even with that documentation in hand, I found it difficult to determine things like the correct WSDL to use, and the process for generating X12 strings was much more complicated than making a more traditional REST request with a parameter hash. The large majority of payers supported by Availity are covered by the REST API, but there are some that are only supported by SOAP API requests and necessitate this more difficult process.</p>
<p>The application we were developing featured a Rails backend, so I used the Ruby <a href="https://github.com/rest-client/rest-client">rest-client</a> gem when making requests to the Availity REST API.</p>
<p>The request payload was surprisingly small. Most payers only require this combination of patient/provider details when making a request:</p>
<ul>
<li>Patient birth date</li>
<li>Payer ID (Assigned by Availity to the health insurance company in the payer list linked above)</li>
<li>Member ID (The patient’s membership ID with their insurer)</li>
<li>Provider NPI</li>
<li>Service type</li>
</ul>
<p>Here is an excerpt of the API client code I created for making requests to the API:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Availity</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Client</span>
<span style="color:#080">extend</span> <span style="color:#036;font-weight:bold">Limiter</span>::<span style="color:#036;font-weight:bold">Mixin</span>
<span style="color:#888"># Rate limit API requests to 100 per second</span>
<span style="color:#888"># https://developer.availity.com/partner/node/503</span>
limit_method <span style="color:#a60;background-color:#fff0f0">:coverages</span>, <span style="color:#a60;background-color:#fff0f0">rate</span>: <span style="color:#00d;font-weight:bold">100</span>, <span style="color:#a60;background-color:#fff0f0">interval</span>: <span style="color:#00d;font-weight:bold">1</span>
<span style="color:#036;font-weight:bold">BASE_URL</span> = <span style="color:#d20;background-color:#fff0f0">'https://api.availity.com/availity/v1'</span>
<span style="color:#888"># ...</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">coverages</span>(<span style="color:#a60;background-color:#fff0f0">payer_id</span>:, <span style="color:#a60;background-color:#fff0f0">member_id</span>:, <span style="color:#a60;background-color:#fff0f0">patient_birth_date</span>:, <span style="color:#a60;background-color:#fff0f0">patient_first_name</span>:, <span style="color:#a60;background-color:#fff0f0">patient_last_name</span>:)
url = <span style="color:#d20;background-color:#fff0f0">"</span><span style="color:#33b;background-color:#fff0f0">#{</span><span style="color:#036;font-weight:bold">BASE_URL</span><span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">/coverages"</span>
params = {
<span style="color:#a60;background-color:#fff0f0">payerID</span>: payer_id,
<span style="color:#a60;background-color:#fff0f0">memberId</span>: member_id,
<span style="color:#a60;background-color:#fff0f0">providerNpi</span>: <span style="color:#d20;background-color:#fff0f0">'123456789'</span>,
<span style="color:#a60;background-color:#fff0f0">patientBirthDate</span>: patient_birth_date,
<span style="color:#a60;background-color:#fff0f0">patientFirstName</span>: patient_first_name,
<span style="color:#a60;background-color:#fff0f0">patientLastName</span>: patient_last_name,
<span style="color:#a60;background-color:#fff0f0">serviceType</span>: <span style="color:#d20;background-color:#fff0f0">'30'</span>
}
headers = {
<span style="color:#a60;background-color:#fff0f0">Authorization</span>: <span style="color:#d20;background-color:#fff0f0">"Bearer </span><span style="color:#33b;background-color:#fff0f0">#{</span>token<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">"</span>,
<span style="color:#a60;background-color:#fff0f0">params</span>: params
}
<span style="color:#036;font-weight:bold">RestClient</span>::<span style="color:#036;font-weight:bold">Request</span>.execute(
<span style="color:#038">method</span>: <span style="color:#a60;background-color:#fff0f0">:get</span>,
<span style="color:#a60;background-color:#fff0f0">url</span>: url,
<span style="color:#a60;background-color:#fff0f0">headers</span>: headers,
<span style="color:#a60;background-color:#fff0f0">log</span>: <span style="color:#33b">@log</span>
)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The insurance coverage API then returns a link to a fairly long JSON response with a variety of information on the patient’s health insurance plan. I added an additional class to parse that response for specific details such as name of primary care doctor, total and remaining deductibles for the year, copay amounts, etc.</p>
<p>In addition to this insurance coverage API, Availity provides various other <a href="https://developer.availity.com/partner/node/503">API end points</a> related to health care claims and costs that could be equally valuable to the development of applications in the health care industry.</p>
GraphQL Server Librarieshttps://www.endpointdev.com/blog/2019/07/graphql-server-libraries/2019-07-12T00:00:00+00:00Zed Jensen
<p><img src="/blog/2019/07/graphql-server-libraries/image-0.jpg" alt="Eroded Icelandic mountain" /><br>Photo by <a href="https://unsplash.com/photos/t07FAEn9wAA">Jon Flobrant</a> on Unsplash</p>
<p>This post is a followup to my previous post, <a href="/blog/2019/05/graphql-an-alternative-to-rest/">GraphQL — An Alternative to REST</a>. Please check that out for an introduction to GraphQL and what makes it different from other API solutions. I’ve collected a list of some of the currently-maintained GraphQL libraries for a few different languages, along with some examples (most of which aren’t fully functional on their own, they’d need more configuration) so you can see what it might be like to use GraphQL in your project. I’ll be focusing on the ways each of these libraries implement GraphQL and what you’d need to do to start a project with each of them, so if you have questions about GraphQL itself, please check out my other blog post.</p>
<h3 id="apollo-server-javascripttypescript">Apollo Server (JavaScript/TypeScript)</h3>
<p><a href="https://www.apollographql.com/">Apollo GraphQL</a> has libraries for both a GraphQL server and client (which I’ll discuss later). <a href="https://www.apollographql.com/docs/apollo-server/">Apollo Server</a> can be used both as a standalone server as well as with libraries like <a href="https://expressjs.com/">Express</a>. Apollo Server is the server library I have the most experience with—I wrote a server last year using Express and Apollo Server, along with a client that used Apollo Client. I’m a fan of the flexibiliy of Apollo, but it takes more work to set up than some of the alternatives.</p>
<p>Setting up Apollo Server as a standalone can be done fairly simply following the directions on <a href="https://www.apollographql.com/docs/apollo-server/getting-started/">their website</a>. However, I’m going to go over the basics of integrating with Express. There are two main parts to writing a server with Apollo: your GraphQL schema and your resolvers. These stay more or less the same whether you’re using Apollo as a standalone server or combining it with Express. You do have to have a database set up separately; I’ll show examples with MongoDB, but you could easily swap it out with PostgreSQL or another database. I’ll show an example resolver along with its GraphQL schema for a blog post. The schema follows the GraphQL schema rules and might look like the following:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#080;font-weight:bold">const</span> typeDefs = [gql<span style="color:#d20;background-color:#fff0f0">`
</span><span style="color:#d20;background-color:#fff0f0"> type Post {
</span><span style="color:#d20;background-color:#fff0f0"> id: String!
</span><span style="color:#d20;background-color:#fff0f0"> body: String!
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">
</span><span style="color:#d20;background-color:#fff0f0"> query {
</span><span style="color:#d20;background-color:#fff0f0"> post(id: String!): Post
</span><span style="color:#d20;background-color:#fff0f0"> }
</span><span style="color:#d20;background-color:#fff0f0">`</span>];
</code></pre></div><p>Now for the resolver. Resolvers are functions that take information from the query (like arguments) and return the relevant data, usually from a database. For our blog post, a resolver might look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#080;font-weight:bold">const</span> resolvers = {
post: (root, args, context, info) => {
<span style="color:#080;font-weight:bold">return</span> Post.findById(args.id);
}
};
</code></pre></div><p>Simple! We just get the data from the database and return it—as long as the property names match those of our schema, Apollo will automatically format it according to the frontend’s request and return it to them.</p>
<p>OK, next we create the server:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-js" data-lang="js"><span style="color:#080;font-weight:bold">import</span> express from <span style="color:#d20;background-color:#fff0f0">'express'</span>;
<span style="color:#080;font-weight:bold">import</span> { ApolloServer } from <span style="color:#d20;background-color:#fff0f0">'apollo-server-express'</span>;
<span style="color:#080;font-weight:bold">const</span> PORT = <span style="color:#00d;font-weight:bold">3000</span>;
<span style="color:#080;font-weight:bold">const</span> app = express();
<span style="color:#080;font-weight:bold">const</span> server = ApolloServer({
typeDefs,
resolvers
});
server.applyMiddleware(app);
app.listen(PORT, () => {
console.log(
<span style="color:#d20;background-color:#fff0f0">`Server running at http://localhost:</span><span style="color:#33b;background-color:#fff0f0">${</span>PORT<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">/graphql`</span>
);
});
</code></pre></div><p>And that’s it! Note that these examples are missing a few things like imports, and we didn’t add authentication of any kind, but this is the general format for creating a server with Apollo.</p>
<h3 id="prisma">Prisma</h3>
<p>Prisma is a cool library developed by the same people as Graph.cool that does much of the work for you in enabling GraphQL access to your database.</p>
<p>Prisma offers configuration for existing databases, but unfortunately I had trouble getting it to work on my Ubuntu system—I ran into issues getting the Docker container to connect to my local Postgres and MongoDB databases. However, following the quick guide found <a href="https://www.prisma.io/docs/get-started/01-setting-up-prisma-new-database-JAVASCRIPT-a002/">here</a> on the Prisma website, I was able to get a GraphQL server up and running inside a Docker container with a new database. The process was simple:</p>
<p>First, you have to install the Prisma command line utility:</p>
<pre tabindex="0"><code>npm install -g prisma
</code></pre><p>You also need to have Docker installed. Documentation for Docker can be found <a href="https://docs.docker.com/get-started/">here</a>.</p>
<p>Next, you need to configure Prisma. Create a directory for your Prisma server, and create a new file named <code>docker-compose.yml</code>:</p>
<pre tabindex="0"><code>mkdir hello-world
cd hello-world
touch docker-compose.yml
</code></pre><p>Then, paste the following into it:</p>
<pre tabindex="0"><code>version: '3'
services:
prisma:
image: prismagraphql/prisma:1.34
restart: always
ports:
- '4466:4466'
environment:
PRISMA_CONFIG: |
port: 4466
databases:
default:
connector: mongo
uri: mongodb://prisma:prisma@mongo
mongo:
image: mongo:3.6
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: prisma
MONGO_INITDB_ROOT_PASSWORD: prisma
ports:
- '27017:27017'
volumes:
- mongo:/var/lib/mongo
volumes:
mongo: ~
</code></pre><p>I used Mongo here, but Prisma’s site has guides for PostgreSQL and MySQL as well. It’s important to make sure now that you don’t have any conflicts with currently running databases—on my machine I had already had a MongoDB server running on port 27017. I fixed this by just stopping my local MongoDB server, but I’m sure you could configure the Docker containers to work with different ports as well. Running Ubuntu, I just ran <code>sudo service mongodb stop</code> and then the Prisma Docker containers worked just fine. When I was done, I ran <code>sudo service mongodb start</code> to start it up again.</p>
<p>Next, you’ll start the Prisma containers and initialize the Prisma server configuration:</p>
<pre tabindex="0"><code>docker-compose up -d
prisma init --endpoint http://localhost:4466
</code></pre><p>The final step is to deploy the service:</p>
<pre tabindex="0"><code>prisma deploy
</code></pre><p>If all goes well, you’ll see a message that includes a URL to the Prisma Admin, which is a browser tool to interact with your GraphQL endpoints. I used it for a little while when I was testing Prisma out, and it seems to work well and is easy to use.</p>
<p>All in all, Prisma seems like a great way to start if you don’t want to handle the messy details of setup. However, I did have issues getting it to play nice with my already-existing databases (including both PostgreSQL and MongoDB). However, it is still relatively new, so I would expect support to get better over time.</p>
<h3 id="graphene-python">Graphene (Python)</h3>
<p>Graphene is a GraphQL framework for Python. It has integrations for a few different server frameworks (a list can be found <a href="https://github.com/graphql-python/graphene">here</a>), but I’ll show examples from <code>graphene-django</code>, since Django is fairly common and something that we use fairly often at End Point.</p>
<p>Because you’re also setting up a Django project, the tutorial for graphene-django is a little more involved, so I’ll just share the relevant GraphQL sections so you can compare to the other libraries in this post. The most important part, the schema, is defined in Python with a similar format to Django models:</p>
<pre tabindex="0"><code>import graphene
from graphene_django.types import DjangoObjectType
from app.models import Category, Ingredient
class CategoryType(DjangoObjectType):
class Meta:
model = Category
class IngredientType(DjangoObjectType):
class Meta:
model = Ingredient
class Query(object):
all_categories = graphene.List(CategoryType)
all_ingredients = graphene.List(IngredientType)
def resolve_all_categories(self, info, **kwargs):
return Category.objects.all()
def resolve_all_ingredients(self, info, **kwargs):
# We can easily optimize query count in the resolve method
return Ingredient.objects.select_related('category').all()
</code></pre><p>As you can see, the format for defining your GraphQL schema is quite different from some other libraries, but you have the advantage of it looking similar to Django’s model definitions. You’ll also need a higher-level Query definition:</p>
<pre tabindex="0"><code>import graphene
import cookbook.ingredients.schema
class Query(cookbook.ingredients.schema.Query, graphene.ObjectType):
# This class will inherit from multiple Queries
# as we begin to add more apps to our project
pass
schema = graphene.Schema(query=Query)
</code></pre><p>Now that we have a schema defined, we need to add a few things to <code>settings.py</code>:</p>
<pre tabindex="0"><code>INSTALLED_APPS = [
...
# This will also make the `graphql_schema` management command available
'graphene_django',
]
GRAPHENE = {
'SCHEMA': 'cookbook.schema.schema'
}
</code></pre><p>The last piece needed to use your GraphQL schema is in <code>urls.py</code>:</p>
<pre tabindex="0"><code>from graphene_django.views import GraphQLView
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^graphql$', GraphQLView.as_view(graphiql=True)),
]
</code></pre><p>And finally we run the server:</p>
<pre tabindex="0"><code>$ python manage.py runserver
</code></pre><p>Now you should be able to use your GraphQL schema at http://localhost:8000/graphql just like with any other GraphQL server.</p>
<p>Graphene for Django seems like a good solution in that it uses a similar format to other aspects of Django, like the model definitions. However, its format (especially for schema definition) is rather different-looking from the GraphQL standard used by most other libraries, and it seems like it might make it more work to keep your frontend and backend in sync.</p>
<h3 id="graphcool">Graph.cool</h3>
<p>I won’t discuss Graph.cool in detail here, because I went over it in my previous blog post. However, it still merits mention here as an option for your GraphQL server. Essentially, Graph.cool lets you define a GraphQL schema and then handles the work of setting up a database and even hosting for you. If you just want to get a basic GraphQL server set up for testing, or if you don’t need too many features beyond data storage and retrieval, Graph.cool is a great choice.</p>
<h3 id="additional-links">Additional links</h3>
<p>For server libraries in other languages, these seem like good options:</p>
<ul>
<li><a href="https://graphql-ruby.org/">GraphQL Ruby</a></li>
<li><a href="https://www.graphql-java.com/">GraphQL Java</a></li>
<li><a href="https://github.com/graph-gophers/graphql-go">graphql-go</a></li>
</ul>
<p>Thanks for reading! Keep an eye out next week for a second post which will cover GraphQL client libraries.</p>
GraphQL — An Alternative to RESThttps://www.endpointdev.com/blog/2019/05/graphql-an-alternative-to-rest/2019-05-11T00:00:00+00:00Zed Jensen
<p><img src="/blog/2019/05/graphql-an-alternative-to-rest/banner.jpg" alt="Banner"></p>
<p><a href="https://graphql.org/">GraphQL</a> has become more and more popular recently as an alternative to traditional RESTful APIs since it was released as open source by Facebook in 2015. According to the GraphQL website, it is “a query language for APIs and a runtime for fulfilling those queries with your existing data”. In this blog post, I’ll go over some of what makes GraphQL different from other API solutions, and then show how to get a GraphQL API up and running so you can try it out yourself!</p>
<p>GraphQL is designed to fit on top of your database layer. With the help of libraries like <a href="https://www.apollographql.com/">Apollo GraphQL</a>, it can be used with many different databases. Some of the main differences between GraphQL and more traditional RESTful APIs include:</p>
<ul>
<li>
<p>GraphQL uses one endpoint. Most traditional APIs use an endpoint for each type of data; in my example, you’d probably have one each for users (<code>/user</code>), posts (<code>/post</code>) and comments (<code>/comment</code>). Each of these would return some JSON with the data you want. GraphQL, on the other hand, lives at one endpoint (usually <code>/graphql</code>) and changes what it returns based on what you ask for, as detailed in the next point.</p>
</li>
<li>
<p>You can get multiple types in one request. For instance, if you want to get information about an author plus all of their posts, instead of making a request for the author and a request for posts, you do just one request for the author and specify that you’d like their posts as well:</p>
</li>
</ul>
<pre tabindex="0"><code>query {
user(id: "12345") {
name
posts {
title
body
}
}
}
</code></pre><ul>
<li>You decide which parts of the data you want. Traditional REST APIs give you data based on which endpoint you’re querying (<code>/post/:id</code>, <code>/user/:id</code>, etc.), and the format of the data is generally the same. For instance, no matter which <code>id</code> you ask for at <code>/posts/:id</code>, you’ll always get something that looks like this back:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"id"</span>:<span style="color:#d20;background-color:#fff0f0">"123"</span>,
<span style="color:#b06;font-weight:bold">"name"</span>:<span style="color:#d20;background-color:#fff0f0">"Smash Mouth"</span>,
<span style="color:#b06;font-weight:bold">"joined"</span>:<span style="color:#d20;background-color:#fff0f0">"1994"</span>
}
</code></pre></div><p>But what if we don’t need to know when they joined right now? Another example that better illustrates this problem (and GraphQL’s solution) is getting a list of blog posts. The home page of a blog usually shows several of the most recent posts, but you can also view a list of just post names that will go farther back. With a traditional REST API, you would have to create separate endpoints for each of these scenarios, for example <code>/first_posts</code> and <code>/posts</code>, or add GET parameters or similar. With GraphQL, you can just specify exactly what you want. The query to replace <code>/first_posts</code> might look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">query {
posts {
title
body
author {
name
}
}
}
</code></pre></div><p>The data returned might look like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"data"</span>: {
<span style="color:#b06;font-weight:bold">"posts"</span>: [
{
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"All Star"</span>,
<span style="color:#b06;font-weight:bold">"body"</span>: <span style="color:#d20;background-color:#fff0f0">"Somebody once told me the world is gonna roll me\n I ain't the sharpest tool in the shed\n She was looking kind of dumb with her finger and her thumb\n In the shape of an \"L\" on her forehead\n Well the years start coming and they don't stop coming\n Fed to the rules and I hit the ground running\n Didn't make sense not to live for fun\n Your brain gets smart but your head gets dumb\n So much to do, so much to see\n So what's wrong with taking the back streets?\n You'll never know if you don't go\n You'll never shine if you don't glow\n Hey now, you're an all-star, get your game on, go play\n Hey now, you're a rock star, get the show on, get paid\n And all that glitters is gold\n Only shooting stars break the mold\n Only shooting stars break the mold"</span>,
<span style="color:#b06;font-weight:bold">"author"</span>: {
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Smash Mouth"</span>
}
},
{
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"Less-interesting Stuff"</span>,
<span style="color:#b06;font-weight:bold">"body"</span>: <span style="color:#d20;background-color:#fff0f0">"Sed ut perspiciatis, unde omnis iste natus error sit voluptatem\n accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore\n veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam\n voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur\n magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui\n dolorem ipsum, quia dolor sit amet consectetur adipisci[ng] velit, sed quia non-numquam\n [do] eius modi tempora inci[di]dunt, ut labore et dolore magnam aliquam quaerat\n voluptatem. Ut enim ad minima veniam, quis nostrum[d] exercitationem ullam corporis\n suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure\n reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel\n illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?"</span>,
<span style="color:#b06;font-weight:bold">"author"</span>: {
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Some Latin-speaking Guy"</span>
}
}
]
}
}
</code></pre></div><p>That’s great for <code>/first_posts</code>, but we probably don’t need to bloat the response with the entire post body if we’re just using the names for a list, right? Let’s try again, but this time we’ll remove <code>body</code> from the request:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">query {
posts {
title
author {
name
}
}
}
</code></pre></div><p>Now we just get the title and author, exactly like we want. Notice also that these queries are pulling in data from a related model, the User model, via the <code>author</code> property.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"data"</span>: {
<span style="color:#b06;font-weight:bold">"posts"</span>: [
{
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"All Star"</span>,
<span style="color:#b06;font-weight:bold">"author"</span>: {
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Smash Mouth"</span>
}
},
{
<span style="color:#b06;font-weight:bold">"title"</span>: <span style="color:#d20;background-color:#fff0f0">"Less-interesting Stuff"</span>,
<span style="color:#b06;font-weight:bold">"author"</span>: {
<span style="color:#b06;font-weight:bold">"name"</span>: <span style="color:#d20;background-color:#fff0f0">"Some Latin-speaking Guy"</span>
}
}
]
}
}
</code></pre></div><p>Although you can get similar functionality by using GET parameters or creating extra endpoints, with GraphQL, you can just make that one simple change without needing to write extra code.</p>
<h3 id="try-it">Try it!</h3>
<p>OK, so there are some of the things that sets GraphQL apart. However, reading about it is only so helpful—now I’ll show how to get a GraphQL backend up and running quickly using <a href="https://www.graph.cool/">Graph.cool</a>, a Node.js backend service that does most of the work of setting up a server, including setting up a database based on your schema. (Graphcool was retired in mid-2020, so you should instead check out their more flexible server option called <a href="https://www.prisma.io/">Prisma</a>.)</p>
<p>First, you need to install graphcool by running:</p>
<p><code>npm install -g graphcool</code></p>
<p>You’ll also need to sign up for a (free) account, but you can do that later. Once graphcool is installed, create a directory to use for your code and initialize it.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ mkdir gql-blog
$ <span style="color:#038">cd</span> gql-blog
$ graphcool init
</code></pre></div><p>Open the file <code>types.graphql</code> and uncomment the lines describing a model called Post, as well as the related line in User; aftewards, it should look something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">type User @model {
# id is of type ID, must be not null (!), and must be unique
id: ID! @isUnique
name: String
dateOfBirth: DateTime
# posts contains an array of Post objects
posts: [Post!]! @relation(name: "UserPosts")
}
type Post @model {
id: ID! @isUnique
title: String!
# Graphcool relations have to be defined both ways so the service knows
# whether it's a one-to-many, many-to-many, or many-to-one relation
author: User! @relation(name: "UserPosts")
}
</code></pre></div><p>Note that there are a few extra descriptors here that aren’t part of the core GraphQL schema; <code>@isUnique</code>, <code>@relation</code>, <code>@model</code>, and <code>ID</code> are all part of the Graphcool service’s additions. However, they are allowed within GraphQL’s spec and provide some extra functionality to the API. <a href="https://graphql.org/learn/schema/#type-system">GraphQL’s website</a> has more information about the type system.</p>
<p>Now that you have a basic schema, it’s time to run your server. In the directory where you ran <code>graphcool init</code>, run <code>graphcool deploy</code>. First, it’ll open a browser window and ask you to log in or create an account. Once you do that, you should see a prompt asking you which server cluster you want to deploy to. Pick any of them (but note that <code>local</code> requires a bit more setup before working). Push enter on the next few prompts to use the default options and you’ll get a big wall of text telling you that your server is up and running! To try it out, find the line near the end that looks like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">Here are your GraphQL Endpoints:
Simple API: https://api.graph.cool/simple/v1/bbd987478jjhhds902k2l
</code></pre></div><p>Copy and paste that URL into a browser and you’ll have access to the GraphQL Playground, where you can run queries and mutations (like queries, but for creating/updating data). You need some data to play with, so copy the following into the Playground one at a time and run them to create a User and a Post. Note that you’ll need to look at the ID returned by the first mutation and paste it into the second mutation so that the two are linked.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">mutation {
createUser(
name: "SmashMouth"
) {
id
name
}
}
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">mutation {
createPost(
title: "All Star",
authorId: "USER_ID_HERE"
) {
id
title
}
}
</code></pre></div><p>Once those are finished, you can run queries to get the data you want. Here’s an example of a query to return the author you created earlier plus any posts associated with it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">query {
User(id: "USER_ID_HERE") {
id
posts {
id
}
}
}
</code></pre></div><p>You’ll notice that this query only returns the ID of the user and their posts. That’s because we only asked for the ID—if you want to get the name of the user and the title of the post, you’ll have to add that to the query:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-text" data-lang="text">query {
User(id: "USER_ID_HERE") {
id
name
posts {
id
title
}
}
}
</code></pre></div><p>Pretty cool, huh? You get exactly what you ask for. Next, try getting your post (you’ll need its ID) and asking for its author along with it..</p>
<p>On the far right of the GraphQL Playground, there’s a tab labeled Schema that shows the entire schema of your app. If you look in there, you’ll see that there are far more types defined than were in the <code>types.graphql</code> file we edited earlier—this is because graphcool automatically generates them for you. That’s the main reason why it’s easier to use than other GraphQL libraries. In particular, you might try getting all posts or all users.</p>
<p>If you want to keep messing around with GraphQL, I’d suggest trying to add your own new <code>Comment</code> type to types.graphql, emulating the way User and Post are related to each other. Keep in mind that you’ll have to run <code>graphcool deploy</code> and reload your browser to see your changes show up in the Playground.</p>
<p>For more information and thorough documentation, see the <a href="https://graphql.org/learn/">GraphQL website</a>.</p>
<p>If you want to try GraphQL without installing anything, visit <a href="http://graphql.nodaljs.com/">graphql.nodaljs.com</a> which lets you try it on their own server.</p>
Eliminating Resolvers in GraphQL Rubyhttps://www.endpointdev.com/blog/2019/03/eliminating-resolvers-in-graphql-ruby/2019-03-29T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2019/03/eliminating-resolvers-in-graphql-ruby/banner.png" alt="GraphQL Ruby code" /></p>
<p>In this follow-up to my post from last month about <a href="/blog/2019/02/converting-graphql-ruby-resolvers-to-the-class-based-api/">Converting GraphQL Ruby Resolvers to the Class-based API</a> I’m going to show how I took the advice of the GraphQL gem’s <a href="https://graphql-ruby.org/fields/resolvers.html">documentation on Resolvers</a> and started replacing the GraphQL-specific Resolver classes with plain old Ruby classes to facilitate easier testing and code reuse.</p>
<p>The current documentation for the <code>GraphQL::Schema::Resolver</code> class essentially recommends that it not be used, except for cases with specific requirements as detailed in the documentation.</p>
<blockquote>
<p>Do you really need a Resolver? Putting logic in a Resolver has some downsides:</p>
</blockquote>
<blockquote>
<p>Since it’s coupled to GraphQL, it’s harder to test than a plain ol’ Ruby object in your app
Since the base class comes from GraphQL-Ruby, it’s subject to upstream changes which may require updates in your code</p>
</blockquote>
<blockquote>
<p>Here are a few alternatives to consider:</p>
</blockquote>
<blockquote>
<ul>
<li>Put display logic (sorting, filtering, etc.) into a plain ol’ Ruby class in your app, and test that class</li>
</ul>
</blockquote>
<blockquote>
<ul>
<li>Hook up that object with a method</li>
</ul>
</blockquote>
<p>I found that I was indeed having trouble testing my Resolvers that inherited from <code>GraphQL::Schema::Resolver</code> due to the GraphQL-specific overhead and context that they contained. Fortunately, it turned out to be a pretty simple process to convert a Resolver class to a plain Ruby class and test it with RSpec.</p>
<p>This was my starting point:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/resolvers/instructor_names.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Resolvers</span>
<span style="color:#888"># Return collections of instructor names based on query arguments</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">InstructorNames</span> < <span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">Base</span>
type [<span style="color:#038">String</span>], <span style="color:#a60;background-color:#fff0f0">null</span>: <span style="color:#080">false</span>
argument <span style="color:#a60;background-color:#fff0f0">:semester</span>, <span style="color:#036;font-weight:bold">Inputs</span>::<span style="color:#036;font-weight:bold">SemesterInput</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">true</span>
argument <span style="color:#a60;background-color:#fff0f0">:past_years</span>, <span style="color:#038">Integer</span>, <span style="color:#d20;background-color:#fff0f0">'Include instructors for this number of past years'</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">false</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">resolve</span>(<span style="color:#a60;background-color:#fff0f0">semester</span>:, <span style="color:#a60;background-color:#fff0f0">past_years</span>: <span style="color:#00d;font-weight:bold">0</span>)
term_year_range = determine_term_year_range(semester, past_years)
<span style="color:#036;font-weight:bold">CourseInstructor</span>
.where(<span style="color:#a60;background-color:#fff0f0">term_year</span>: term_year_range)
.group(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.pluck(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.map { |<span style="color:#038">name</span>| <span style="color:#038">name</span>.join(<span style="color:#d20;background-color:#fff0f0">' '</span>) }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080">private</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">determine_term_year_range</span>(semester, past_years)
term_year_max = semester[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]
term_year_min = term_year_max - past_years
term_year_min..term_year_max
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>I started the conversion process by rewriting my <code>Resolvers::InstructorNames</code> class to be a plain Ruby object:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/resolvers/instructor_names.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Resolvers</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">InstructorNames</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#b06;font-weight:bold">self</span>.<span style="color:#06b;font-weight:bold">run</span>(<span style="color:#a60;background-color:#fff0f0">semester</span>:, <span style="color:#a60;background-color:#fff0f0">past_years</span>:)
term_year_range = determine_term_year_range(semester, past_years)
<span style="color:#036;font-weight:bold">CourseInstructor</span>
.where(<span style="color:#a60;background-color:#fff0f0">term_year</span>: term_year_range)
.group(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.pluck(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.map { |<span style="color:#038">name</span>| <span style="color:#038">name</span>.join(<span style="color:#d20;background-color:#fff0f0">' '</span>) }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#b06;font-weight:bold">self</span>.<span style="color:#06b;font-weight:bold">determine_term_year_range</span>(semester, past_years)
term_year_max = semester[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]
term_year_min = term_year_max - past_years
term_year_min..term_year_max
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The removal of all GraphQL-specific code made this an easy class to test with RSpec:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># spec/graphql/resolvers/instructor_names_spec.rb</span>
<span style="color:#038">require</span> <span style="color:#d20;background-color:#fff0f0">'rails_helper'</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Resolvers</span>
<span style="color:#036;font-weight:bold">RSpec</span>.describe <span style="color:#036;font-weight:bold">InstructorNames</span> <span style="color:#080;font-weight:bold">do</span>
let!(<span style="color:#a60;background-color:#fff0f0">:instructors_2018</span>) { create_pair(<span style="color:#a60;background-color:#fff0f0">:course_instructor</span>, <span style="color:#a60;background-color:#fff0f0">term_year</span>: semester_2018[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]) }
let!(<span style="color:#a60;background-color:#fff0f0">:instructors_2019</span>) { create_pair(<span style="color:#a60;background-color:#fff0f0">:course_instructor</span>, <span style="color:#a60;background-color:#fff0f0">term_year</span>: semester_2019[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]) }
let(<span style="color:#a60;background-color:#fff0f0">:outcome</span>) { described_class.run(inputs) }
let(<span style="color:#a60;background-color:#fff0f0">:semester_2018</span>) { { <span style="color:#a60;background-color:#fff0f0">term_year</span>: <span style="color:#00d;font-weight:bold">2018</span> } }
let(<span style="color:#a60;background-color:#fff0f0">:semester_2019</span>) { { <span style="color:#a60;background-color:#fff0f0">term_year</span>: <span style="color:#00d;font-weight:bold">2019</span> } }
context <span style="color:#d20;background-color:#fff0f0">'with a single year'</span> <span style="color:#080;font-weight:bold">do</span>
let(<span style="color:#a60;background-color:#fff0f0">:inputs</span>) { { <span style="color:#a60;background-color:#fff0f0">semester</span>: semester_2019, <span style="color:#a60;background-color:#fff0f0">past_years</span>: <span style="color:#00d;font-weight:bold">0</span> } }
it <span style="color:#d20;background-color:#fff0f0">'returns the expected list of instructor names'</span> <span style="color:#080;font-weight:bold">do</span>
expect(outcome).to match_array(instructors_2019.map(&<span style="color:#a60;background-color:#fff0f0">:full_name</span>))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
context <span style="color:#d20;background-color:#fff0f0">'with multiple years'</span> <span style="color:#080;font-weight:bold">do</span>
let(<span style="color:#a60;background-color:#fff0f0">:inputs</span>) { { <span style="color:#a60;background-color:#fff0f0">semester</span>: semester_2019, <span style="color:#a60;background-color:#fff0f0">past_years</span>: <span style="color:#00d;font-weight:bold">1</span> } }
let(<span style="color:#a60;background-color:#fff0f0">:instructors</span>) { instructors_2018 + instructors_2019 }
it <span style="color:#d20;background-color:#fff0f0">'returns the expected list of instructor names'</span> <span style="color:#080;font-weight:bold">do</span>
expect(outcome).to match_array(instructors.map(&<span style="color:#a60;background-color:#fff0f0">:full_name</span>))
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>Finally, I updated my query type to hook up the GraphQL field with the return value of the new plain <code>InstructorNames</code> class:</p>
<p>Old <code>QueryType</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/types/query_type.rb</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">QueryType</span> < <span style="color:#036;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">BaseObject</span>
description <span style="color:#d20;background-color:#fff0f0">'Queries'</span>
field <span style="color:#a60;background-color:#fff0f0">:instructor_names</span>,
<span style="color:#a60;background-color:#fff0f0">description</span>: <span style="color:#d20;background-color:#fff0f0">'Returns a collection of instructor names for a given range of years'</span>,
<span style="color:#a60;background-color:#fff0f0">resolver</span>: <span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">InstructorNames</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>New <code>QueryType</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/types/query_type.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Types</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Query</span> < <span style="color:#036;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">BaseObject</span>
description <span style="color:#d20;background-color:#fff0f0">'Queries'</span>
field <span style="color:#a60;background-color:#fff0f0">:instructor_names</span>, [<span style="color:#038">String</span>], <span style="color:#a60;background-color:#fff0f0">null</span>: <span style="color:#080">false</span>, <span style="color:#a60;background-color:#fff0f0">description</span>: <span style="color:#d20;background-color:#fff0f0">'Returns a collection of instructor names for a given range of years'</span> <span style="color:#080;font-weight:bold">do</span>
argument <span style="color:#a60;background-color:#fff0f0">:semester</span>, <span style="color:#036;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">Inputs</span>::<span style="color:#036;font-weight:bold">Semester</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">true</span>
argument <span style="color:#a60;background-color:#fff0f0">:past_years</span>, <span style="color:#038">Integer</span>, <span style="color:#d20;background-color:#fff0f0">'Include instructors for this number of past years'</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">false</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">instructor_names</span>(<span style="color:#a60;background-color:#fff0f0">semester</span>:, <span style="color:#a60;background-color:#fff0f0">past_years</span>: <span style="color:#00d;font-weight:bold">0</span>)
<span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">InstructorNames</span>.run(<span style="color:#a60;background-color:#fff0f0">semester</span>: semester, <span style="color:#a60;background-color:#fff0f0">past_years</span>: past_years)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>Note that the <code>instructor_names</code> method matches the <code>instructor_names</code> field definition, and is responsible for providing the value returned by that field. The argument and field type definitions have been moved out of the Resolver (because it no longer contains anything specific to GraphQL) and into the field definition.</p>
<p>I considered moving my updated “Resolver” logic out of the <code>app/graphql/</code> hierarchy entirely, and that might have made more sense if I anticipated wanting to reuse that code elsewhere in my application. But since this particular Rails application is running in API mode and really only exists to serve the GraphQL API, I decided to leave it in place and maintain the naming convention while removing the actual GraphQL inheritance. For a larger application it might make sense to move these files into a directory under <code>lib/</code>.</p>
Switching from Google Maps to Leaflethttps://www.endpointdev.com/blog/2019/03/switching-google-maps-leaflet/2019-03-23T00:00:00+00:00Juan Pablo Ventoso
<p><img src="/blog/2019/03/switching-google-maps-leaflet/leaflet-weather-map-us.jpg" alt="Leaflet Weather map example" /><br>Photo: <a href="https://www.extendedforecast.net/radsat">RadSat HD</a></p>
<p>It’s no news for anyone who has Google Maps running on their websites that Google started charging for using their API. We saw it coming when, back in 2016, they started requiring a key to add a map using their JavaScript API. And on June 11, 2018, they did a major upgrade to their API and billing system.</p>
<p><b>The consequence?</b> Any website with more than 25,000 page loads per day will have to pay. And if you are using a dynamic map (a map with custom styling and/or content) you only have roughly 28,000 free monthly page loads. We must create a billing account, <em>even if we have a small website with a couple of daily visitors</em>, hand credit card information to Google, and monitor our stats to make sure we won’t be charged. And if we don’t do that, our map will be dark and will have a “For development only” message in the background.</p>
<p>So what are your options? You can either pay or completely remove Google Maps from your websites. Even enterprise weather websites like <a href="https://weather.com/weather/radar/interactive/l/USNY0996:1:US">The Weather Channel</a> or <a href="https://www.wunderground.com/wundermap">Weather Underground</a> have now replaced their Google Maps API calls with an alternative like Leaflet or MapBox (in some cases, they even gained some functionality in the process).</p>
<p>I have a <a href="https://www.extendedforecast.net">personal weather website</a>, and when I heard big changes were coming, I started to move away from Google Maps as well. My choice at that moment was Leaflet: It has everything you may need to build a robust tile-based map, add layers, markers, animations, custom tiles, etc. And it’s BSD-licensed <b>open source and free</b>.</p>
<h3 id="creating-a-basic-map">Creating a basic map</h3>
<p><img src="/blog/2019/03/switching-google-maps-leaflet/google-vs-leaflet-look-and-feel.jpg" /><br><small>Google Map conversion to Leaflet can be almost seamless if the same tiles are used.</small></p>
<p>Google Maps API and Leaflet share a similar way of doing most things, but they have some key differences we need to take into account. As a general rule, Google used the “google.maps” prefix to name most classes and interfaces, while Leaflet uses the “L” prefix instead.</p>
<p>First thing we need to do is to remove the Google Maps API reference from our website(s). So we need to replace the reference:</p>
<pre tabindex="0"><code><script type="text/javascript" src="https://maps.googleapis.com/maps/api/js?key=[your_api_key]"></script>
</code></pre><p>With the references to the Leaflet map JavaScript and stylesheet URIs.</p>
<pre tabindex="0"><code><script src="https://unpkg.com/leaflet@1.0.2/dist/leaflet.js"></script>
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.0.2/dist/leaflet.css" />
</code></pre><p>Now let’s take a look at the code needed to create a Google Map vs. a Leaflet map.</p>
<ul>
<li>Google:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> map = <span style="color:#080;font-weight:bold">new</span> google.maps.Map(<span style="color:#038">document</span>.getElementById(<span style="color:#d20;background-color:#fff0f0">"map"</span>), {
center: <span style="color:#080;font-weight:bold">new</span> google.maps.LatLng(<span style="color:#00d;font-weight:bold">40.7401</span>, -<span style="color:#00d;font-weight:bold">73.9891</span>),
zoom: <span style="color:#00d;font-weight:bold">12</span>,
mapTypeId: google.maps.MapTypeId.ROADMAP
});
</code></pre></div><p>Leaflet:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> map = <span style="color:#080;font-weight:bold">new</span> L.Map(<span style="color:#d20;background-color:#fff0f0">"map"</span>, {
center: <span style="color:#080;font-weight:bold">new</span> L.LatLng(<span style="color:#00d;font-weight:bold">40.7401</span>, -<span style="color:#00d;font-weight:bold">73.9891</span>),
zoom: <span style="color:#00d;font-weight:bold">12</span>,
layers: <span style="color:#080;font-weight:bold">new</span> L.TileLayer(<span style="color:#d20;background-color:#fff0f0">"https://tile.openstreetmap.org/{z}/{x}/{y}.png"</span>)
});
</code></pre></div><p>Quite similar, isn’t it? The main difference is that, in Leaflet, we need to provide a tile layer for the base map because there isn’t one by default. There are a lot of excellent tile layers available to use at no cost. Here are some of them:</p>
<ul>
<li><b>Bright</b>: <code>https://a.tiles.mapbox.com/v3/mapbox.world-bright/{z}/{x}/{y}.png</code></li>
<li><b>Topographic</b>: <code>https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png</code></li>
<li><b>Black and white</b>: <code>https://stamen-tiles-{s}.a.ssl.fastly.net/toner/{z}/{x}/{y}{r}.png</code></li>
</ul>
<p>You can browse other free tile layer providers for Leaflet on <a href="https://leaflet-extras.github.io/leaflet-providers/preview/">this link</a>. And of course, if you want to pay there’s a lot of affordable paid tiles out there too.</p>
<h3 id="adding-a-marker">Adding a marker</h3>
<p>Adding a marker is quite straightforward as well. It even looks easier on Leaflet than Google.</p>
<p>Google:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> marker = <span style="color:#080;font-weight:bold">new</span> google.maps.Marker({
position: <span style="color:#080;font-weight:bold">new</span> google.maps.LatLng(<span style="color:#00d;font-weight:bold">40.7401</span>, -<span style="color:#00d;font-weight:bold">73.9891</span>),
map: map,
title: <span style="color:#d20;background-color:#fff0f0">"End Point Corporation"</span>
});
</code></pre></div><p>Leaflet:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> marker = <span style="color:#080;font-weight:bold">new</span> L.Marker(<span style="color:#080;font-weight:bold">new</span> L.LatLng(<span style="color:#00d;font-weight:bold">40.7401</span>, -<span style="color:#00d;font-weight:bold">73.9891</span>));
marker.bindPopup(<span style="color:#d20;background-color:#fff0f0">"End Point Corporation"</span>);
map.addLayer(marker);
</code></pre></div><p>And that’s it: we have a working Leaflet map with a marker that displays a text when we click on it.</p>
<p><img src="/blog/2019/03/switching-google-maps-leaflet/leaflet-example-working.jpg" /><br><small>Screenshot of the Leaflet example. Code below, if you want to try it live:</small></p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">title</span>>Leaflet map example — End Point Corporation</<span style="color:#b06;font-weight:bold">title</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">src</span>=<span style="color:#d20;background-color:#fff0f0">"https://unpkg.com/leaflet@1.0.2/dist/leaflet.js"</span>></<span style="color:#b06;font-weight:bold">script</span>>
<<span style="color:#b06;font-weight:bold">link</span> <span style="color:#369">rel</span>=<span style="color:#d20;background-color:#fff0f0">"stylesheet"</span> <span style="color:#369">href</span>=<span style="color:#d20;background-color:#fff0f0">"https://unpkg.com/leaflet@1.0.2/dist/leaflet.css"</span> />
</<span style="color:#b06;font-weight:bold">head</span>>
<<span style="color:#b06;font-weight:bold">body</span>>
<<span style="color:#b06;font-weight:bold">style</span>>
<span style="color:#b06;font-weight:bold">body</span> { <span style="color:#080;font-weight:bold">margin</span>: <span style="color:#00d;font-weight:bold">0</span> };
#<span style="color:#b06;font-weight:bold">map</span> { <span style="color:#080;font-weight:bold">height</span>: <span style="color:#00d;font-weight:bold">100</span><span style="color:#888;font-weight:bold">%</span> };
</<span style="color:#b06;font-weight:bold">style</span>>
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">id</span>=<span style="color:#d20;background-color:#fff0f0">"map"</span>></<span style="color:#b06;font-weight:bold">div</span>>
<<span style="color:#b06;font-weight:bold">script</span> <span style="color:#369">type</span>=<span style="color:#d20;background-color:#fff0f0">"text/javascript"</span>>
<span style="color:#080;font-weight:bold">var</span> endPointLocation = <span style="color:#080;font-weight:bold">new</span> L.LatLng(<span style="color:#00d;font-weight:bold">40.7401</span>, -<span style="color:#00d;font-weight:bold">73.9891</span>);
<span style="color:#080;font-weight:bold">var</span> map = <span style="color:#080;font-weight:bold">new</span> L.Map(<span style="color:#d20;background-color:#fff0f0">"map"</span>, {
center: endPointLocation,
zoom: <span style="color:#00d;font-weight:bold">12</span>,
layers: <span style="color:#080;font-weight:bold">new</span> L.TileLayer(<span style="color:#d20;background-color:#fff0f0">"https://tile.openstreetmap.org/{z}/{x}/{y}.png"</span>)
});
<span style="color:#080;font-weight:bold">var</span> marker = <span style="color:#080;font-weight:bold">new</span> L.Marker(endPointLocation);
marker.bindPopup(<span style="color:#d20;background-color:#fff0f0">"End Point Corporation"</span>);
map.addLayer(marker);
</<span style="color:#b06;font-weight:bold">script</span>>
</<span style="color:#b06;font-weight:bold">body</span>>
</code></pre></div><h3 id="layers-and-controls">Layers and controls</h3>
<p>From this point, we can start doing more complex things if we need to:</p>
<ul>
<li><b>Display images on the map</b>: <a href="https://leafletjs.com/reference-1.4.0.html#imageoverlay">ImageOverlay</a>.</li>
<li><b>Display a custom tile layer</b>: <a href="https://leafletjs.com/reference-1.4.0.html#tilelayer">TileLayer</a>.</li>
<li><b>Draw polygons, rectangles, circles</b>: <a href="https://leafletjs.com/reference-1.4.0.html#polygon">Polygon</a> - <a href="https://leafletjs.com/reference-1.4.0.html#rectangle">Rectangle</a> - <a href="https://leafletjs.com/reference-1.4.0.html#circle">Circle</a>.</li>
<li><b>Display GeoJSON data on the map</b>: <a href="https://leafletjs.com/reference-1.4.0.html#geojson">GeoJSON</a>.</li>
</ul>
<p>You can browse the <a href="https://leafletjs.com/reference-1.4.0.html">Leaflet API reference</a> for further details.</p>
<h3 id="plugins-and-tools">Plugins and tools</h3>
<p>There is some extended functionality in Google Maps that is not available in Leaflet by default unless we use a plugin. For example, if we want to add the “fullscreen” button to the top right corner, just as Google has it, or if we want to let the user draw polygons on top of the map, we’ll need to download and add the reference to the required plugins. Here is a list of the ones I’ve already used:</p>
<ul>
<li><b>“Fullscreen” button plugin</b>: <a href="https://github.com/Leaflet/Leaflet.fullscreen">Leaflet.fullscreen</a>.</li>
<li><b>Vector drawing and editing plugin</b>: <a href="https://github.com/Leaflet/Leaflet.draw">Leaflet.draw</a>.</li>
<li><b>Heatmap plugin</b>: <a href="https://github.com/Leaflet/Leaflet.heat">Leaflet.heat</a>.</li>
</ul>
<p>You can find more plugins at the <a href="https://github.com/Leaflet/">Leaflet GitHub account</a>. And of course, you can (and should!) contribute to improve them.</p>
<p>There is also some alternatives to additional services offered by Google like geocoding or routing. They might have some limitations involved, so it would be wise to take a look at their usage policies first.</p>
<ul>
<li><b>Geocoding API</b>: <a href="https://wiki.openstreetmap.org/wiki/Nominatim">Nominatim</a>.</li>
<li><b>Routing</b>: <a href="http://project-osrm.org/">Project ORSM</a> (free version has limited use).</li>
</ul>
<p>More services can be found at <a href="https://switch2osm.org/other-uses/">switch2osm.org/other-uses</a>.</p>
<h3 id="putting-it-all-together">Putting it all together</h3>
<p>I’ve been using Leaflet for almost a year now in an interactive weather map originally made with the Google Maps API. Of course, I’ve had some minor hiccups along the way, but having full control of the source code and resources allows you to add functionality, fix things or even rewrite whatever you need.</p>
<p>The Leaflet source code is well organized, modularized and easy to understand. I’ve created custom grid layers using different tile sources, with different coordinate systems, animations with frame transitions, custom controls, clickable polygons, popups with dynamic content from AJAX calls, and more. And all works smoothly. So I recommend that you <b>go ahead and start using Leaflet right away</b>.</p>
<p><img src="/blog/2019/03/switching-google-maps-leaflet/leaflet-map-radsat-hd.jpg" /><br><small>Example of a fully-functional Leaflet map with custom controls, overlays, animations and polygons.</small></p>
<p>And this is the repository with my weather map source code: <a href="https://github.com/juanpabloventoso/RadSat-HD">RadSat HD</a>. Feel free to leave any comments or suggestions!</p>
Converting GraphQL Ruby Resolvers to the Class-based APIhttps://www.endpointdev.com/blog/2019/02/converting-graphql-ruby-resolvers-to-the-class-based-api/2019-02-28T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2019/02/converting-graphql-ruby-resolvers-to-the-class-based-api/banner.png" alt="GraphQL Ruby code" /></p>
<p>The <a href="https://graphql-ruby.org/">GraphQL gem</a> is a great tool for any Rails developer who wants to add a full-featured GraphQL API to their Rails applications.</p>
<p>I have been using GraphQL to serve an API in one of my Rails applications since late 2017 and have been very happy with the features and performance provided by the gem, but some of the domain-specific syntax for building out my API schema never felt quite right when compared to the other Ruby code I was writing in my projects. Fortunately, the 1.8.0 release of the GraphQL Ruby gem brought with it a new default class-based syntax while remaining compatible with existing code that predated the change.</p>
<p>The <a href="https://graphql-ruby.org/schema/class_based_api">Class-based API guide</a> that accompanied the changes does a good job of describing the upgrade path for developers who need to convert their existing schemas. The old <code>.define</code> syntax is eventually going to be removed with version 2.0 of the gem, so I was interested in converting my existing API over to the new style, both to see what benefits the newer syntax provides and to ensure that the API schema remains compatible with future releases of the gem.</p>
<p>The GraphQL gem provides some rake tasks like <code>graphql:upgrade:schema</code> and <code>graphql:upgrade:member</code> for automatic conversion of the older-style <code>.define</code> files to the newer class-based syntax. It worked quite well for updating my <a href="https://graphql-ruby.org/type_definitions/objects.html">type definitions</a>, but I also make heavy use of <a href="https://graphql-ruby.org/fields/resolvers.html">resolvers</a> for containing the logic needed to return values in my GraphQL fields, and there was no way to automatically convert those files.</p>
<p>I found that the process of manually converting my resolvers was pretty straightforward and provided some benefits by cleaning up my <code>QueryType</code> file that was starting to look a little unwieldy.</p>
<p>Here is a before and after for comparison:</p>
<p>Old pre-1.8 ‘.define’ syntax for types:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/types/query_type.rb</span>
<span style="color:#036;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">QueryType</span> = <span style="color:#036;font-weight:bold">GraphQL</span>::<span style="color:#036;font-weight:bold">ObjectType</span>.define <span style="color:#080;font-weight:bold">do</span>
description <span style="color:#d20;background-color:#fff0f0">'Queries'</span>
field <span style="color:#a60;background-color:#fff0f0">:instructor_names</span>, types[types.String] <span style="color:#080;font-weight:bold">do</span>
description <span style="color:#d20;background-color:#fff0f0">'Returns a collection of instructor names for a given range of years'</span>
argument <span style="color:#a60;background-color:#fff0f0">:semester</span>, !<span style="color:#036;font-weight:bold">Inputs</span>::<span style="color:#036;font-weight:bold">SemesterInput</span>
argument <span style="color:#a60;background-color:#fff0f0">:past_years</span>, types.Int, <span style="color:#d20;background-color:#fff0f0">'Include instructors for this number of past years'</span>
resolve <span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">InstructorNamesResolver</span>.new
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>New class-based syntax for types:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/types/query_type.rb</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">QueryType</span> < <span style="color:#036;font-weight:bold">Types</span>::<span style="color:#036;font-weight:bold">BaseObject</span>
description <span style="color:#d20;background-color:#fff0f0">'Queries'</span>
field <span style="color:#a60;background-color:#fff0f0">:instructor_names</span>,
<span style="color:#a60;background-color:#fff0f0">description</span>: <span style="color:#d20;background-color:#fff0f0">'Returns a collection of instructor names for a given range of years'</span>,
<span style="color:#a60;background-color:#fff0f0">resolver</span>: <span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">InstructorNamesResolver</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>Old pre-1.8 ‘.define’ syntax for resolvers:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/resolvers/instructor_names_resolver.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Resolvers</span>
<span style="color:#888"># Return collections of instructor names based on query arguments</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">InstructorNamesResolver</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(_obj, args, _ctx)
semester = args[<span style="color:#a60;background-color:#fff0f0">:semester</span>]
past_years = args[<span style="color:#a60;background-color:#fff0f0">:past_years</span>] || <span style="color:#00d;font-weight:bold">0</span>
term_year_range = determine_term_year_range(semester, past_years)
<span style="color:#036;font-weight:bold">CourseInstructor</span>
.where(<span style="color:#a60;background-color:#fff0f0">term_year</span>: term_year_range)
.group(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.pluck(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.map { |<span style="color:#038">name</span>| <span style="color:#038">name</span>.join(<span style="color:#d20;background-color:#fff0f0">' '</span>) }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080">private</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">determine_term_year_range</span>(semester, past_years)
term_year_max = semester[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]
term_year_min = term_year_max - past_years
term_year_min..term_year_max
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>New class-based syntax for resolvers:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># app/graphql/resolvers/instructor_names_resolver.rb</span>
<span style="color:#080;font-weight:bold">module</span> <span style="color:#b06;font-weight:bold">Resolvers</span>
<span style="color:#888"># Return collections of instructor names based on query arguments</span>
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">InstructorNamesResolver</span> < <span style="color:#036;font-weight:bold">Resolvers</span>::<span style="color:#036;font-weight:bold">Base</span>
type [<span style="color:#038">String</span>], <span style="color:#a60;background-color:#fff0f0">null</span>: <span style="color:#080">false</span>
argument <span style="color:#a60;background-color:#fff0f0">:semester</span>, <span style="color:#036;font-weight:bold">Inputs</span>::<span style="color:#036;font-weight:bold">SemesterInput</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">true</span>
argument <span style="color:#a60;background-color:#fff0f0">:past_years</span>, <span style="color:#038">Integer</span>, <span style="color:#d20;background-color:#fff0f0">'Include instructors for this number of past years'</span>, <span style="color:#a60;background-color:#fff0f0">required</span>: <span style="color:#080">false</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">resolve</span>(<span style="color:#a60;background-color:#fff0f0">semester</span>:, <span style="color:#a60;background-color:#fff0f0">past_years</span>: <span style="color:#00d;font-weight:bold">0</span>)
term_year_range = determine_term_year_range(semester, past_years)
<span style="color:#036;font-weight:bold">CourseInstructor</span>
.where(<span style="color:#a60;background-color:#fff0f0">term_year</span>: term_year_range)
.group(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.pluck(<span style="color:#a60;background-color:#fff0f0">:first_name</span>, <span style="color:#a60;background-color:#fff0f0">:last_name</span>)
.map { |<span style="color:#038">name</span>| <span style="color:#038">name</span>.join(<span style="color:#d20;background-color:#fff0f0">' '</span>) }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080">private</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">determine_term_year_range</span>(semester, past_years)
term_year_max = semester[<span style="color:#a60;background-color:#fff0f0">:term_year</span>]
term_year_min = term_year_max - past_years
term_year_min..term_year_max
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The main things to note here are:</p>
<ul>
<li>The class definitions for <code>Types::QueryType</code> and <code>Resolvers::InstructorNamesResolver</code>, which now inherit from base GraphQL classes.</li>
<li>The <code>type</code> and <code>argument</code> definitions are moved out of the <code>query_type</code> file, cleaning it up greatly, especially for larger schemas with many fields.</li>
<li>The <code>call</code> method in <code>InstructorNamesResolver</code> is now <code>resolve</code>, and accepts arguments using named parameters (or <code>**args</code> if you want to retain the previous syntax).</li>
</ul>
<p>Overall, I’m really happy with the newer GraphQL API syntax and think that its more idiomatic Ruby is easier to work with and feels more familiar than the older style. I look forward to building out my API schema further and attempting things like replacing my use of Resolvers with plain old Ruby objects in order to make it even easier to test the logic they contain.</p>
Speech Recognition from scratch using Dilated Convolutions and CTC in TensorFlowhttps://www.endpointdev.com/blog/2019/01/speech-recognition-with-tensorflow/2019-01-08T00:00:00+00:00Kamil Ciemniewski
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/25928285337_50483f3619_o.jpg" alt="Sound visualization" /><br><a href="https://www.flickr.com/photos/williamismael/25928285337/">Image by WILL POWER · CC BY 2.0, cropped</a></p>
<p>In this blog post, I’d like to take you on a journey. We’re going to get a speech recognition project from its architecting phase, through coding and training. In the end, we’ll have a fully working model. You’ll be able to take it and run the model serving app, exposing a nice HTTP API. Yes, you’ll even be able to use it in your projects.</p>
<p>Speech recognition has been amongst one of the hardest tasks in Machine Learning. Traditional approaches involve meticulous crafting and extracting of the audio features that separate one phoneme from another. To be able to do that, one needs a deep background in data science and signal processing. The complexity of the training process prompted teams of researchers to look for alternative, more automated approaches.</p>
<p>With the growing development of Deep Learning, the need for handcrafted features declined. The training process for a neural network is much more streamlined. You can feed the signals either in their raw form or as their spectrograms and watch the model improve.</p>
<p>Did this get you excited? Let’s start!</p>
<h3 id="project-plan-of-attack">Project Plan of Attack</h3>
<p>Let’s build a web service that exposes an API. Let it be able to receive audio signals, encoded as an array of floating point numbers. In return, we’re going to get the recognized text.</p>
<p>Here’s a rough plan of the stages we’re going to go through:</p>
<ol>
<li>Get the dataset to train the model on</li>
<li>Architect the model</li>
<li>Implement it along with the unit tests</li>
<li>Train it on the dataset</li>
<li>Measure its accuracy</li>
<li>Serve it as a web service</li>
</ol>
<h4 id="the-dataset">The dataset</h4>
<p>The open-source community has a lot to be thankful for the <a href="https://foundation.mozilla.org/">Mozilla Foundation</a> for. It’s a host of many projects with a wonderful, free Firefox browser at its forefront. One of its other projects, called <a href="https://voice.mozilla.org">Common Voice</a>, focuses on gathering large data sets to be used by anyone in speech recognition projects.</p>
<p>The datasets consist of wave files and their text transcriptions. There’s no notion of time-alignment. It’s just the audio and text for each utterance.</p>
<p>If you want to code along, head up to <a href="https://voice.mozilla.org/pl/datasets">the Common Voice Datasets download page</a>. Be warned that it weighs roughly around 12GB.</p>
<p>After the download, simply extract the files from the archive into the <code>./data</code> directory of the root of the project. The files, in the end, should reside under the <code>./data/cv_corpus_v1/</code> path.</p>
<p>How much data should we have? It always depends on the challenge at hand. Roughly speaking, the more difficult the task, the more powerful your neural network needs to be. It will need to be capable of expressing more complex patterns in data. The more powerful the network, the easier it is to have it just memorize the training examples. This is highly undesirable and results in overfitting. To lessen its aptitude to do so, you need to either augment your data on the fly randomly or gather more “real” examples. On this project, we’re going to do both. Data augmentation will be covered in the coding section. Additional datasets we’ll use are well known <a href="http://www.openslr.org/12/">LibriSpeech</a> (<a href="http://www.openslr.org/resources/12/train-clean-360.tar.gz">the file to download, around 23GB</a>) and <a href="http://voxforge.org">VoxForge</a> (<a href="https://s3.us-east-2.amazonaws.com/common-voice-data-download/voxforge_corpus_v1.0.0.tar.gz">the file to download</a>).</p>
<p>Those two datasets are among the most popular that are freely available. There are others I chose to omit as they weigh quite a lot. I was already almost out of free space after the download and preprocessing of the three sets chosen above.</p>
<p>You need to download both Libri and Vox and extract them under <code>./data/LibriSpeech/</code> and <code>./data/voxforge/</code>.</p>
<h3 id="background-on-audio-processing">Background on audio processing</h3>
<p>In order to build a working model, we need some background in signal processing. Although a lot of the traditional work is going to be done by the neural network automatically, we still need to understand what is going on in order to reason about its various hyperparameters.</p>
<p>Additionally, we’re going to process audio into a form that’s easier to train. This is going to lower the memory requirements. It’s also going to lower the time needed for model’s parameters to <em>converge</em> to ones that work well.</p>
<h4 id="how-is-audio-represented">How is audio represented?</h4>
<p>Let’s have a quick look at what the audio data looks like when we load it from a wave file.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">librosa</span>
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">librosa.display</span>
SAMPLING_RATE=<span style="color:#00d;font-weight:bold">16000</span>
<span style="color:#888"># ...</span>
wave, _ = librosa.load(path_to_file, sr=SAMPLING_RATE)
librosa.display.waveplot(wave, sr=SAMPLING_RATE)
</code></pre></div><p>The above code specifies that we want to load the audio data with a <em>sampling rate</em> of a 16k (more about it later). It then loads it and plots it along the time axis:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/wave-plot.png" alt="" title="Plot of a raw audio signal"></p>
<p>The X-axis obviously represents the time. The Y axis is often called the <a href="https://en.wikipedia.org/wiki/Amplitude">amplitude</a>. A quick look at the plot above makes it obvious that we have negative values in the signal. How come those values are called amplitudes then? Amplitude is said to represent the maximum difference of displacements of a physical object as it vibrates. What does it mean to have a negative amplitude? To make those values a bit more clear, let’s call it just displacement for now. Audio is nothing else than the vibration of the air. If you were to build an electrical recorder, you might come up with one that gives you output in voltages at each point in time. As the air vibrates, you need a <strong>reference point</strong> obviously. This, in turn, allows you to catch the exact specifics of the vibration — how it “rises” above the reference point and then gets back way below it. Imagine that your electrical circuit gives you output within the range of <code>-1V</code> and <code>1V</code>. To load it into your computer and into the plot like above, you’d need to capture those values at discrete points in time. The <strong>sampling rate</strong> is nothing else than a number of times within one second when the value from your sound-meter would be measured and stored — to be loaded later. Next time, when you read that your CD from the ’90 contains audio sampled at a frequency of 44,100 Hz, you’ll know that the raw “air displacement” values were sampled 44,100 times each second.</p>
<p>Let’s do a simple thought experiment to prepare for the next section. What would you hear if all the above values were constant, e.g. 1.0? We saw that the values given by <code>librosa</code> are floating points. In the example file they ranged between -0.6 and 0.6. The value of 1.0 is certainly much higher — would you hear “more” of “something” then? Because the definition of a sound is that <strong>it’s a vibration</strong>: you wouldn’t hear anything! The amplitudes of the audio signal must periodically change — this is how we detect or hear sounds. This implies that in order to distinguish between different sounds, those sounds have to “vibrate differently”. The difference that makes sounds different is the <strong>frequency</strong> of the vibration.</p>
<h4 id="decomposing-the-signal-with-the-fourier-transform">Decomposing the signal with the Fourier Transform</h4>
<p>Let’s create a signal generating machine, that will output a sinusoidal of a given frequency and amplitude:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">gen_sin</span>(freq, amplitude, sr=<span style="color:#00d;font-weight:bold">1000</span>):
<span style="color:#080;font-weight:bold">return</span> np.sin(
(freq * <span style="color:#00d;font-weight:bold">2</span> * np.pi * np.linspace(<span style="color:#00d;font-weight:bold">0</span>, sr, sr)) / sr
) * amplitude
</code></pre></div><p>Here’s how 1000 points signal looks like for a frequency of 30 and an amplitude of 1:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">seaborn</span> <span style="color:#080;font-weight:bold">as</span> <span style="color:#b06;font-weight:bold">sns</span>
sns.lineplot(data=gen_sin(<span style="color:#00d;font-weight:bold">30</span>, <span style="color:#00d;font-weight:bold">1</span>))
</code></pre></div><p><img src="/blog/2019/01/speech-recognition-with-tensorflow/signal-1000-30-1.png" alt="" title="Sinusoidal signal"></p>
<p>Here’s one for 10 and 0.6:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/signal-1000-10-0.6.png" alt="" title="Sinusoidal signal"></p>
<p>You can count the number of times the values in plots approach their maximum. Knowing that sine has only one maximum within its period and that we’re showing just one second, that number shows that we have frequencies 30 and 10.</p>
<p>What would we get if we were to sum such sinusoidal signals of different frequencies and amplitudes? Let’s see — below you can see 3 different sine waves plotted on top of each other. The fourth — and last one — shows the signal that is the sum of all of them:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/wave-decomposition-2.png" alt="" title="Wave composition / decomposition"></p>
<p>Here’s another example, with the last plot showing the sum of 5 different waves:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/wave-decomposition-1.png" alt="" title="Wave composition / decomposition"></p>
<p>It isn’t that regular anymore, is it? It turns out that <strong>you can construct any signal by summing up some number of sine waves of different frequencies and amplitudes</strong> (and phases, their translation in time). The converse is also true: <strong>any signal can be represented as a sum of some number of sine waves of different frequencies and amplitudes</strong> (and phases). This is extremely important to our speech recognition task. Frequencies are the real difference between sounds that make up the phonemes and words that we want to be able to recognize.</p>
<p>This is where the <a href="https://en.wikipedia.org/wiki/Fourier_transform">Fourier Transform</a> comes into play. It takes our data points that represent intensity per each point in time and produces data points representing intensity per each <em>frequency bin</em>. It’s said that it transforms the domain of the signal from <em>time</em> into <em>frequency</em>. Now, what exactly is a <em>frequency bin</em>? Imagine the physical audio signal being constructed from frequencies between 0Hz and 8000Hz. The FFT algorithm (Fast Fourier Transform) is going to split that full spectrum into <em>bins</em>. If you were to split it into 10 bins, you’d end up having the following ranges: 0Hz–800Hz, 800Hz–1600Hz, 1600Hz–2400Hz, 2400Hz–3200Hz, 3200Hz–4000Hz, 4000Hz–4800Hz, 4800Hz–5600Hz, 5600Hz–6400Hz, 6400Hz–7200Hz, 7200Hz–8000Hz.</p>
<p>Let’s see how the FFT works on the example of the signal given above. The waves and plots were produced by the following Python function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">plot_wave_composition</span>(defs, hspace=<span style="color:#00d;font-weight:bold">1.0</span>):
fig_size = plt.rcParams[<span style="color:#d20;background-color:#fff0f0">"figure.figsize"</span>]
plt.rcParams[<span style="color:#d20;background-color:#fff0f0">"figure.figsize"</span>] = [<span style="color:#00d;font-weight:bold">14.0</span>, <span style="color:#00d;font-weight:bold">10.0</span>]
waves = [
gen_sin(freq, amp)
<span style="color:#080;font-weight:bold">for</span> freq, amp <span style="color:#080">in</span> defs
]
fig, axs = plt.subplots(nrows=<span style="color:#038">len</span>(defs) + <span style="color:#00d;font-weight:bold">1</span>)
<span style="color:#080;font-weight:bold">for</span> ix, wave <span style="color:#080">in</span> <span style="color:#038">enumerate</span>(waves):
sb.lineplot(data=wave, ax=axs[ix])
axs[ix].set_ylabel(<span style="color:#d20;background-color:#fff0f0">'</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(defs[ix]))
<span style="color:#080;font-weight:bold">if</span> ix != <span style="color:#00d;font-weight:bold">0</span>:
axs[ix].set_title(<span style="color:#d20;background-color:#fff0f0">'+'</span>)
plt.subplots_adjust(hspace = hspace)
sb.lineplot(data=<span style="color:#038">sum</span>(waves), ax=axs[<span style="color:#038">len</span>(defs)])
axs[<span style="color:#038">len</span>(defs)].set_ylabel(<span style="color:#d20;background-color:#fff0f0">'sum'</span>)
axs[<span style="color:#038">len</span>(defs)].set_xlabel(<span style="color:#d20;background-color:#fff0f0">'time'</span>)
axs[<span style="color:#038">len</span>(defs)].set_title(<span style="color:#d20;background-color:#fff0f0">'='</span>)
plt.rcParams[<span style="color:#d20;background-color:#fff0f0">"figure.figsize"</span>] = fig_size
<span style="color:#080;font-weight:bold">return</span> waves, <span style="color:#038">sum</span>(waves)
</code></pre></div><p>We can plot the signals and grab them at the same time with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">wave_defs = [
(<span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">1</span>),
(<span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">0.8</span>),
(<span style="color:#00d;font-weight:bold">5</span>, <span style="color:#00d;font-weight:bold">0.2</span>),
(<span style="color:#00d;font-weight:bold">7</span>, <span style="color:#00d;font-weight:bold">0.1</span>),
(<span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">0.25</span>)
]
waves, the_sum = plot_wave_composition(wave_defs)
</code></pre></div><p>Next, let’s compute the FFT values along with the frequencies:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">ffts = np.fft.fft(the_sum)
freqs = np.fft.fftfreq(<span style="color:#038">len</span>(the_sum))
frequencies, coeffs = <span style="color:#038">zip</span>(
*<span style="color:#038">list</span>(
<span style="color:#038">filter</span>(
<span style="color:#080;font-weight:bold">lambda</span> row: row[<span style="color:#00d;font-weight:bold">1</span>] > <span style="color:#00d;font-weight:bold">10</span>, <span style="color:#888"># arbitrary threshold but let’s not make it too complex for now</span>
[ (<span style="color:#038">int</span>(<span style="color:#038">abs</span>(freq * <span style="color:#00d;font-weight:bold">1000</span>)), coef) <span style="color:#080;font-weight:bold">for</span> freq, coef <span style="color:#080">in</span> <span style="color:#038">zip</span>(freqs[<span style="color:#00d;font-weight:bold">0</span>:(<span style="color:#038">len</span>(ffts) // <span style="color:#00d;font-weight:bold">2</span>)], np.abs(ffts)[<span style="color:#00d;font-weight:bold">0</span>:(<span style="color:#038">len</span>(ffts) // <span style="color:#00d;font-weight:bold">2</span>)]) ]
)
)
)
sns.barplot(x=<span style="color:#038">list</span>(frequencies), y=coeffs)
</code></pre></div><p>The last call produces the following plot:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/fft-results.png" alt="" title="Detected frequencies"></p>
<p>The X-axis represents now the frequency in Hz, while the Y-axis is the intensity.</p>
<p>There’s one missing part before we can use it with our speech data. As you can see, FFT gives us frequencies <strong>for the whole signal, assuming that it’s periodic and spans in time into infinity</strong>. Obviously, when I say “hello”, the air vibrates differently in the beginning, changes in between and is even more different at the end. We need to <strong>split</strong> that audio into small “windows” of data points. By feeding them into FFT, we can get the frequencies for each one of them. This turns the data domain from time into frequency within the scope of the window. It remains the info about the time at the global level, making our data represent: <code>time x frequency x intensity</code>.</p>
<h4 id="scaling-frequencies">Scaling frequencies</h4>
<p>The human perception is a vastly complex phenomenon. Taking that into account can take us a long way when working on the recognition model emulating the work of our brains when we’re listening to each other.</p>
<p>Let’s make another experiment. What sound is produced by the 800Hz sine?</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">IPython.display</span> <span style="color:#080;font-weight:bold">import</span> Audio
Audio(data=gen_sin(<span style="color:#00d;font-weight:bold">800</span>, <span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">16000</span>), rate=<span style="color:#00d;font-weight:bold">16000</span>)
</code></pre></div><div>
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/800Hz.wav" type="audio/wav">
</audio>
</div><br />
<p>Let’s now generate 900Hz and 1000Hz to get a sense of the difference:</p>
<p>900Hz:</p>
<div>
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/900Hz.wav" type="audio/wav">
</audio>
</div><br />
<p>1000Hz:</p>
<div>
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/1000Hz.wav" type="audio/wav">
</audio>
</div><br />
<p>Let us now ante up the frequencies and generate 7000Hz, 7100Hz and 7200Hz:</p>
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/7000Hz.wav" type="audio/wav">
</audio>
<br />
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/7100Hz.wav" type="audio/wav">
</audio>
<br />
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/7200Hz.wav" type="audio/wav">
</audio>
<br />
<p>Can you hear the difference being smaller in the case of the last three? It’s a well-known phenomenon. We sense a greater difference in sounds for lower frequencies and as it increases that difference becomes less and less.</p>
<p>Because of this, three gentlemen—Stevens, Volkmann, and Newman—created a so-called <a href="https://en.wikipedia.org/wiki/Mel_scale">Mel scale</a> in 1937. You can think of it as a simple rescaling of the frequencies that roughly follows the relationship shown below:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/Mel-Hz_plot.svg.png" alt=""></p>
<p>Although not mandatory, lots of models that deal with human speech also decrease the importance of the intensity by taking the log of the re-scaled data. The resulting <code>time x frequency (mels) x log-intensity</code> is called the <strong>log-Mel spectrogram</strong>.</p>
<h3 id="background-on-deep-learning-techniques-in-use-for-this-project">Background on deep learning techniques in use for this project</h3>
<p>We’ve just gone through the necessary basics of signal processing. Let’s now focus on the Deep Learning concepts we’ll use to construct and train the model.</p>
<p>While this article assumes that the reader already knows a lot, there are less common techniques we’ll use that deserve at least a quick go through.</p>
<h4 id="dilated-convolutions-as-a-faster-alternative-to-recurrent-networks">Dilated convolutions as a faster alternative to recurrent networks</h4>
<p>Traditionally, the sequence processing in Deep Learning is tackled by the <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">recurrent neural networks</a>.</p>
<p>No matter the choice of their flavor, the basic scheme is always the same: the computations are done <strong>sequentially</strong> going through examples <strong>in time</strong>. In our case, we’d need to split the <code>time x frequency x intensity</code> into <code>time</code> length of <code>frequency x intensity</code> chunks. As the chunks would be processed one by one, the recurrent network internal state would “remember” the previous chunk’s specifics, incorporating them into their future outputs. The output shape would be <code>time x frequency x recurrent units</code>.</p>
<p>The fact that the computations are done sequentially, makes them quite slow overall. Later in-pipeline computations spend most of the time waiting on the previous ones to finish because of the direct dependency. The problem is even more severe with the use of GPUs. We use them because of their ability to do math in parallel on huge chunks of data. With recurrent networks, lots of that power is being wasted.</p>
<p>The premise of RNNs is that in theory, they can have the capacity for keeping very long contexts in their “memory”. This has recently been put into test and falsified in practice by <a href="https://arxiv.org/pdf/1803.01271.pdf">Bai et al</a>. Also, when you stop and think about the task at hand: does it really matter to “remember” the beginning of the sentence to know that it ends with the word “dog”? Some context is obviously needed — but not as wide as it might seem at first.</p>
<p>I have an Nvidia GTX 1070Ti with 8GB of memory to train my models on. I don’t really feel like waiting a month for my recurrent network to converge. In this project, let’s use a very performant alternative — <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional neural network</a>.</p>
<h5 id="expanding-the-context-of-the-convolutional-network">Expanding the context of the convolutional network</h5>
<p>Simple convolutional layers weren’t used for sequence processing much for a good reason. The crux of the sequence processing is to be able to take bigger contexts into account. Depending on the job, we might want to constrain the context only to the <em>past</em> — learning the <strong>causal</strong> relations in data. We might sometimes want to incorporate both <em>past</em> and <em>future</em> in it as well. The go-to solution for doing OCR at the moment is to use bidirectional recurrent layers. Their one pass learns the relations from left to right while another learns from right to left. The results are then concatenated.</p>
<p>By applying proper padding, we can easily include one or two-sided contexts in 1D convolutions. The challenge is that in order to make the outputs depend on bigger contexts, the size of the filters needs to become bigger and bigger. This, in turn, requires more and more memory.</p>
<p>Because our aim is to create a model that we’ll be able to train on a quite cheap (given the GPUs used in this field usually) GTX 1070Ti (around $500 at the moment), we want the memory requirements to be as low as possible.</p>
<p>Thanks to the success of the <a href="https://arxiv.org/pdf/1609.03499.pdf">WaveNet</a> (among others), a specific class of convolutional layers gained a lot of attention lately. The variation is called <strong>Dilated Convolutions</strong> or sometimes <strong>Atrous Convolutions</strong>. So what are they?</p>
<p>Let’s first have a look at how the outputs depend on their context for simple convolutional layers:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/causal-conv-3-1.png" alt=""></p>
<p>Imagine that you originally have just the top-most row of numbers. You are going to use 1D convolutions and to make the reasoning easiest, the number of filters is 1. Also for simplicity, all filter values are set to 1. You can see the cross-correlation (because that’s what convolutional layers are in fact computing) operator taking 3 values in the context, multiplying by the filter and summing up to <code>2 * 1 + 3 * 1 + 4 * 1 = 9</code>.</p>
<p>The <em>atrous</em> convolutions are really the same, except they <strong>dilate</strong> their focus without increasing the size of the filter by introducing holes. It’s shown below with the convolution of the size 2 and dilation of 2:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/causal-conv-2-2.png" alt=""></p>
<p>Here’s yet another example for the size of 2 and dilation of 3:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/causal-conv-2-3.png" alt=""></p>
<h4 id="gated-activations">Gated activations</h4>
<p>Traditionally, convolutional layers are followed by the *elu family of activations (ReLu, Elu, PRelu, Selu). They fit in well within the “match pattern” paradigm of the conv nets. On the contrary, recurrent units operate the “remember/forget” approach. Two of their most commonly used implementations, GRU and LSTM, include explicit “forget” gates.</p>
<p>We want to mimic their ability to “forget” parts of the context within our dilated convolutions based model too. To do that, we’re going to use the “gated activations” approach, explained by <a href="https://arxiv.org/pdf/1712.09444.pdf">Liptchinsky et al.</a></p>
<p>The idea is very simple: we pass the input through Conv1D separately and apply tanh and sigmoid respectively. The result is the element-wise product. We’re going to go one step further in our approach, by applying tanh one more time in the end.</p>
<h4 id="others">Others</h4>
<p>The full explanation of all of the details of our neural network’s architecture is beyond the scope of an article like this. Let me point you at additional pieces along with the reading they come from:</p>
<ul>
<li><a href="https://arxiv.org/pdf/1502.03167.pdf">Batch Normalization</a></li>
<li><a href="ftp://ftp.idsia.ch/pub/juergen/icml2006.pdf">Connectionist Temporal Classification</a></li>
<li><a href="https://arxiv.org/pdf/1512.03385.pdf">Residual Learning</a></li>
</ul>
<h3 id="lets-code-it">Let’s code it</h3>
<p>The architecture of our choice in this project is going to heavily rely on the great success of residual-style networks as well as dilated convolutions. You might see similarities to the famous WaveNet, although it’s going to be a bit different.</p>
<p>Here is the bird-eye view of the SpeechNet neural network:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/speech-net.png" alt=""></p>
<p>The residual stacks, being at the heart of it, are structured the following way:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/residual-stack.png" alt=""></p>
<p>The residual blocks, doing all the heavy lifting, can be seen as shown below:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/residual-block.png" alt=""></p>
<h4 id="the-most-important-aspect-of-coding-of-the-deep-learning-models">The most important aspect of coding of the Deep Learning models</h4>
<p>Developing Deep Learning models doesn’t really differ that much from any other type of coding. It does require specific background knowledge, but the good coding practices remain the same. In fact, good coding habits are 10× more relevant here than in e.g. a web-app project.</p>
<p>Training a speech-to-text model is bound to require days if not weeks. Imagine having a small bug in your code, preventing the process from finding a good local minimum. It’s extremely frustrating to find out about it days into the training, with the model trainable parameters not being improved much.</p>
<p>Let’s start by adding some unit tests then. In this project, we’re using the Jupyter notebook as we don’t intend to package it anywhere. The code’s intent is to be for educational purposes mainly.</p>
<p>Adding unit tests within the Jupyter notebook is possible with the following “hack” (notice the value for <code>argv</code>):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">unittest</span>
RUN_TESTS = TRUE
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">TestNotebook</span>(unittest.TestCase):
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_it_works</span>(self):
self.assertEqual(<span style="color:#00d;font-weight:bold">2</span> + <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">4</span>)
<span style="color:#080;font-weight:bold">if</span> __name__ == <span style="color:#d20;background-color:#fff0f0">'__main__'</span> <span style="color:#080">and</span> RUN_TESTS:
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">doctest</span>
doctest.testmod()
unittest.main(
argv=[<span style="color:#d20;background-color:#fff0f0">'first-arg-is-ignored'</span>],
failfast=<span style="color:#080;font-weight:bold">True</span>,
exit=<span style="color:#080;font-weight:bold">False</span>
)
</code></pre></div><p>You can notice the import of the <code>doctest</code> module which adds support for <a href="https://docs.python.org/2/library/doctest.html">doc-string level tests</a> which may come in handy as well.</p>
<p>I also hugely recommend the <a href="https://hypothesis.readthedocs.io/en/latest/">hypothesis library</a> for testing the QuickCheck way <a href="/blog/2016/03/quickcheck-property-based-testing-in/">as I blogged about it before</a>.</p>
<h5 id="data-pipeline">Data pipeline</h5>
<p>A place that’s surprisingly very bug-potent is the data pipeline. It’s easy to e.g. shuffle the labels independently of input vectors if you’re not careful. There’s also always a chance to introduce input vectors including <code>NaN</code> or <code>inf</code> values, which a few steps later produce <code>NaN</code> or <code>inf</code> loss values. Let’s add a simple test to check for the first condition:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">
<span style="color:#888"># assuming test path will look like: 1/file.wav</span>
<span style="color:#888"># the input and output types are driven by the input_fn shown later</span>
<span style="color:#888"># here, we’re just generating values based on the “path”</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">dummy_load_wave</span>(example):
row, params = example
path = row.filename
<span style="color:#080;font-weight:bold">return</span> np.ones((SAMPLING_RATE)) * <span style="color:#038">float</span>(path.split(<span style="color:#d20;background-color:#fff0f0">'/'</span>)[<span style="color:#00d;font-weight:bold">0</span>]), row
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">TestNotebook</span>(unittest.TestCase):
<span style="color:#888"># (...)</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_dataset_returns_data_in_order</span>(self):
params = experiment_params(
dataset_params(
batch_size=<span style="color:#00d;font-weight:bold">2</span>,
epochs=<span style="color:#00d;font-weight:bold">1</span>,
augment=<span style="color:#080;font-weight:bold">False</span>
)
)
data = pd.DataFrame(
data={
<span style="color:#d20;background-color:#fff0f0">'text'</span>: [ <span style="color:#038">str</span>(i) <span style="color:#080;font-weight:bold">for</span> i <span style="color:#080">in</span> <span style="color:#038">range</span>(<span style="color:#00d;font-weight:bold">10</span>) ],
<span style="color:#d20;background-color:#fff0f0">'filename'</span>: [ <span style="color:#d20;background-color:#fff0f0">'</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">/wav'</span>.format(i) <span style="color:#080;font-weight:bold">for</span> i <span style="color:#080">in</span> <span style="color:#038">range</span>(<span style="color:#00d;font-weight:bold">10</span>) ]
}
)
dataset = input_fn(data, params[<span style="color:#d20;background-color:#fff0f0">'data'</span>], dummy_load_wave)()
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
<span style="color:#080;font-weight:bold">with</span> tf.Session() <span style="color:#080;font-weight:bold">as</span> session:
<span style="color:#080;font-weight:bold">try</span>:
<span style="color:#080;font-weight:bold">while</span> <span style="color:#080;font-weight:bold">True</span>:
audio, label = session.run(next_element)
audio, length = audio
<span style="color:#080;font-weight:bold">for</span> _audio, _label <span style="color:#080">in</span> <span style="color:#038">zip</span>(<span style="color:#038">list</span>(audio), <span style="color:#038">list</span>(label)):
self.assertEqual(_audio[<span style="color:#00d;font-weight:bold">0</span>], <span style="color:#038">float</span>(_label))
<span style="color:#080;font-weight:bold">for</span> _length <span style="color:#080">in</span> length:
self.assertEqual(_length, SAMPLING_RATE)
<span style="color:#080;font-weight:bold">except</span> tf.errors.OutOfRangeError:
<span style="color:#080;font-weight:bold">pass</span>
</code></pre></div><p>The above code assumes having the <code>input_fn</code> function in scope. If you’re not familiar with the concept yet, please go ahead and read the introduction to the <a href="https://www.tensorflow.org/guide/estimators">TensorFlow Estimators API</a>.</p>
<p>Here’s our implementation:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">multiprocessing</span> <span style="color:#080;font-weight:bold">import</span> Pool
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">input_fn</span>(input_dataset, params, load_wave_fn=load_wave):
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">_input_fn</span>():
<span style="color:#d20;background-color:#fff0f0">"""
</span><span style="color:#d20;background-color:#fff0f0"> Returns raw audio wave along with the label
</span><span style="color:#d20;background-color:#fff0f0"> """</span>
dataset = input_dataset
<span style="color:#038">print</span>(params)
<span style="color:#080;font-weight:bold">if</span> <span style="color:#d20;background-color:#fff0f0">'max_text_length'</span> <span style="color:#080">in</span> params <span style="color:#080">and</span> params[<span style="color:#d20;background-color:#fff0f0">'max_text_length'</span>] <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
<span style="color:#038">print</span>(<span style="color:#d20;background-color:#fff0f0">'Constraining dataset to the max_text_length'</span>)
dataset = input_dataset[input_dataset.text.str.len() < params[<span style="color:#d20;background-color:#fff0f0">'max_text_length'</span>]]
<span style="color:#080;font-weight:bold">if</span> <span style="color:#d20;background-color:#fff0f0">'min_text_length'</span> <span style="color:#080">in</span> params <span style="color:#080">and</span> params[<span style="color:#d20;background-color:#fff0f0">'min_text_length'</span>] <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
<span style="color:#038">print</span>(<span style="color:#d20;background-color:#fff0f0">'Constraining dataset to the min_text_length'</span>)
dataset = input_dataset[input_dataset.text.str.len() >= params[<span style="color:#d20;background-color:#fff0f0">'min_text_length'</span>]]
<span style="color:#080;font-weight:bold">if</span> <span style="color:#d20;background-color:#fff0f0">'max_wave_length'</span> <span style="color:#080">in</span> params <span style="color:#080">and</span> params[<span style="color:#d20;background-color:#fff0f0">'max_wave_length'</span>] <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
<span style="color:#038">print</span>(<span style="color:#d20;background-color:#fff0f0">'Constraining dataset to the max_wave_length'</span>)
<span style="color:#038">print</span>(<span style="color:#d20;background-color:#fff0f0">'Resulting dataset length: </span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(<span style="color:#038">len</span>(dataset)))
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">generator_fn</span>():
pool = Pool()
buffer = []
<span style="color:#080;font-weight:bold">for</span> epoch <span style="color:#080">in</span> <span style="color:#038">range</span>(params[<span style="color:#d20;background-color:#fff0f0">'epochs'</span>]):
<span style="color:#080;font-weight:bold">for</span> _, row <span style="color:#080">in</span> dataset.sample(frac=<span style="color:#00d;font-weight:bold">1</span>).iterrows():
buffer.append((row, params))
<span style="color:#080;font-weight:bold">if</span> <span style="color:#038">len</span>(buffer) >= params[<span style="color:#d20;background-color:#fff0f0">'batch_size'</span>]:
<span style="color:#080;font-weight:bold">if</span> params[<span style="color:#d20;background-color:#fff0f0">'parallelize'</span>]:
audios = pool.map(
load_wave_fn,
buffer
)
<span style="color:#080;font-weight:bold">else</span>:
audios = <span style="color:#038">map</span>(
load_wave_fn,
buffer
)
<span style="color:#080;font-weight:bold">for</span> audio, row <span style="color:#080">in</span> audios:
<span style="color:#080;font-weight:bold">if</span> audio <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
<span style="color:#080;font-weight:bold">if</span> np.isnan(audio).any():
<span style="color:#038">print</span>(<span style="color:#d20;background-color:#fff0f0">'SKIPPING! NaN coming from the pipeline!'</span>)
<span style="color:#080;font-weight:bold">else</span>:
<span style="color:#080;font-weight:bold">yield</span> (audio, <span style="color:#038">len</span>(audio)), row.text.encode()
buffer = []
<span style="color:#080;font-weight:bold">return</span> tf.data.Dataset.from_generator(
generator_fn,
output_types=((tf.float32, tf.int32), (tf.string)),
output_shapes=((<span style="color:#080;font-weight:bold">None</span>,()), (()))
) \
.padded_batch(
batch_size=params[<span style="color:#d20;background-color:#fff0f0">'batch_size'</span>],
padded_shapes=(
(tf.TensorShape([<span style="color:#080;font-weight:bold">None</span>]), tf.TensorShape(())),
tf.TensorShape(())
)
)
<span style="color:#080;font-weight:bold">return</span> _input_fn
</code></pre></div><p>This depends on the <code>load_wave</code> function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">librosa</span>
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">hickle</span> <span style="color:#080;font-weight:bold">as</span> <span style="color:#b06;font-weight:bold">hkl</span>
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">os.path</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">to_path</span>(filename):
<span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">'./data/cv_corpus_v1/'</span> + filename
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">load_wave</span>(example, absolute=<span style="color:#080;font-weight:bold">False</span>):
row, params = example
_path = row.filename <span style="color:#080;font-weight:bold">if</span> absolute <span style="color:#080;font-weight:bold">else</span> to_path(row.filename)
<span style="color:#080;font-weight:bold">if</span> os.path.isfile(_path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>):
wave = hkl.load(_path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>).astype(np.float32)
<span style="color:#080;font-weight:bold">else</span>:
wave, _ = librosa.load(_path, sr=SAMPLING_RATE)
hkl.dump(wave, _path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>)
<span style="color:#080;font-weight:bold">if</span> <span style="color:#038">len</span>(wave) <= params[<span style="color:#d20;background-color:#fff0f0">'max_wave_length'</span>]:
<span style="color:#080;font-weight:bold">if</span> params[<span style="color:#d20;background-color:#fff0f0">'augment'</span>]:
wave = random_noise(
random_stretch(
random_shift(
wave,
params
),
params
),
params
)
<span style="color:#080;font-weight:bold">else</span>:
wave = <span style="color:#080;font-weight:bold">None</span>
<span style="color:#080;font-weight:bold">return</span> wave, row
</code></pre></div><p>Which depends on three other functions used to augment the data on the fly to improve the model’s generalization:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">random</span>
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">glob</span>
noise_files = glob.glob(<span style="color:#d20;background-color:#fff0f0">'./data/*.wav'</span>)
noises = {}
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">random_stretch</span>(audio, params):
rate = random.uniform(params[<span style="color:#d20;background-color:#fff0f0">'random_stretch_min'</span>], params[<span style="color:#d20;background-color:#fff0f0">'random_stretch_max'</span>])
<span style="color:#080;font-weight:bold">return</span> librosa.effects.time_stretch(audio, rate)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">random_shift</span>(audio, params):
_shift = random.randrange(params[<span style="color:#d20;background-color:#fff0f0">'random_shift_min'</span>], params[<span style="color:#d20;background-color:#fff0f0">'random_shift_max'</span>])
<span style="color:#080;font-weight:bold">if</span> _shift < <span style="color:#00d;font-weight:bold">0</span>:
pad = (_shift * -<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">0</span>)
<span style="color:#080;font-weight:bold">else</span>:
pad = (<span style="color:#00d;font-weight:bold">0</span>, _shift)
<span style="color:#080;font-weight:bold">return</span> np.pad(audio, pad, mode=<span style="color:#d20;background-color:#fff0f0">'constant'</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">random_noise</span>(audio, params):
_factor = random.uniform(
params[<span style="color:#d20;background-color:#fff0f0">'random_noise_factor_min'</span>],
params[<span style="color:#d20;background-color:#fff0f0">'random_noise_factor_max'</span>]
)
<span style="color:#080;font-weight:bold">if</span> params[<span style="color:#d20;background-color:#fff0f0">'random_noise'</span>] > random.uniform(<span style="color:#00d;font-weight:bold">0</span>, <span style="color:#00d;font-weight:bold">1</span>):
_path = random.choice(noise_files)
<span style="color:#080;font-weight:bold">if</span> _path <span style="color:#080">in</span> noises:
wave = noises[_path]
<span style="color:#080;font-weight:bold">else</span>:
<span style="color:#080;font-weight:bold">if</span> os.path.isfile(_path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>):
wave = hkl.load(_path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>).astype(np.float32)
noises[_path] = wave
<span style="color:#080;font-weight:bold">else</span>:
wave, _ = librosa.load(_path, sr=SAMPLING_RATE)
hkl.dump(wave, _path + <span style="color:#d20;background-color:#fff0f0">'.wave.hkl'</span>)
noises[_path] = wave
noise = random_shift(
wave,
{
<span style="color:#d20;background-color:#fff0f0">'random_shift_min'</span>: -<span style="color:#00d;font-weight:bold">16000</span>,
<span style="color:#d20;background-color:#fff0f0">'random_shift_max'</span>: <span style="color:#00d;font-weight:bold">16000</span>
}
)
max_noise = np.max(noise[<span style="color:#00d;font-weight:bold">0</span>:<span style="color:#038">len</span>(audio)])
max_wave = np.max(audio)
noise = noise * (max_wave / max_noise)
<span style="color:#080;font-weight:bold">return</span> _factor * noise[<span style="color:#00d;font-weight:bold">0</span>:<span style="color:#038">len</span>(audio)] + (<span style="color:#00d;font-weight:bold">1.0</span> - _factor) * audio
<span style="color:#080;font-weight:bold">else</span>:
<span style="color:#080;font-weight:bold">return</span> audio
</code></pre></div><p>Notice that we’re making almost everything into a configurable parameter. We want the code to allow the greatest freedom of searching for just the right set of hyperparameters.</p>
<p>The data pipeline as shown above randomly shuffles the <a href="https://pandas.pydata.org">Pandas</a> data frame once for each epoch. It also creates a pool of background workers to parallelize the data loading as much as possible. We’re doing the data loading and augmentation on the CPU. It also uses the <a href="https://github.com/telegraphic/hickle">hickle</a> library for caching audio signals on the disk. Loading a wave file with a given sampling rate isn’t <strong>that</strong> fast as one might think. In my experiments, loading the resulting array of floating points via <code>hickle</code> was 10x faster. We need the best speed of feeding the data into the network or else our GPU is going to stay underutilized.</p>
<p>In my experiments also, turning data augmentation on <strong>made a real difference</strong>. I’ve run the training without it and the network overfit was disastrous: with the normalized <a href="https://en.wikipedia.org/wiki/Edit_distance">edit distance</a> for the training set revolving around 0.01 and 0.53 for the validation.</p>
<p>The <code>random_noise</code> function uses the noise sounds included in the <a href="http://download.tensorflow.org/data/speech_commands_v0.01.tar.gz">Speech Commands: A public dataset for single-word speech recognition</a> dataset. Please go ahead and download it, extracting just the noise files under the <code>./data</code> directory.</p>
<p>The last function in use we haven’t seen yet is the <code>experiment_params</code>. It’s just a helper that allows an easy params hash construction for our experiments:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">dataset_params</span>(batch_size=<span style="color:#00d;font-weight:bold">32</span>,
epochs=<span style="color:#00d;font-weight:bold">50000</span>,
parallelize=<span style="color:#080;font-weight:bold">True</span>,
max_text_length=<span style="color:#080;font-weight:bold">None</span>,
min_text_length=<span style="color:#080;font-weight:bold">None</span>,
max_wave_length=<span style="color:#00d;font-weight:bold">80000</span>,
shuffle=<span style="color:#080;font-weight:bold">True</span>,
random_shift_min=-<span style="color:#00d;font-weight:bold">4000</span>,
random_shift_max= <span style="color:#00d;font-weight:bold">4000</span>,
random_stretch_min=<span style="color:#00d;font-weight:bold">0.7</span>,
random_stretch_max= <span style="color:#00d;font-weight:bold">1.3</span>,
random_noise=<span style="color:#00d;font-weight:bold">0.75</span>,
random_noise_factor_min=<span style="color:#00d;font-weight:bold">0.2</span>,
random_noise_factor_max=<span style="color:#00d;font-weight:bold">0.5</span>,
augment=<span style="color:#080;font-weight:bold">False</span>):
<span style="color:#080;font-weight:bold">return</span> {
<span style="color:#d20;background-color:#fff0f0">'parallelize'</span>: parallelize,
<span style="color:#d20;background-color:#fff0f0">'shuffle'</span>: shuffle,
<span style="color:#d20;background-color:#fff0f0">'max_text_length'</span>: max_text_length,
<span style="color:#d20;background-color:#fff0f0">'min_text_length'</span>: min_text_length,
<span style="color:#d20;background-color:#fff0f0">'max_wave_length'</span>: max_wave_length,
<span style="color:#d20;background-color:#fff0f0">'random_shift_min'</span>: random_shift_min,
<span style="color:#d20;background-color:#fff0f0">'random_shift_max'</span>: random_shift_max,
<span style="color:#d20;background-color:#fff0f0">'random_stretch_min'</span>: random_stretch_min,
<span style="color:#d20;background-color:#fff0f0">'random_stretch_max'</span>: random_stretch_max,
<span style="color:#d20;background-color:#fff0f0">'random_noise'</span>: random_noise,
<span style="color:#d20;background-color:#fff0f0">'random_noise_factor_min'</span>: random_noise_factor_min,
<span style="color:#d20;background-color:#fff0f0">'random_noise_factor_max'</span>: random_noise_factor_max,
<span style="color:#d20;background-color:#fff0f0">'epochs'</span>: epochs,
<span style="color:#d20;background-color:#fff0f0">'batch_size'</span>: batch_size,
<span style="color:#d20;background-color:#fff0f0">'augment'</span>: augment
}
</code></pre></div><h5 id="labels-encoder-and-decoder">Labels encoder and decoder</h5>
<p>When working with the CTC loss, we need a way to code each letter as a numerical value. Conversely, the neural network is going to give us probabilities for each letter, given by its index within the output matrix.</p>
<p>The idea behind this project’s approach is to push the encoding and decoding into the network graph itself. We want two functions: <code>encode_labels</code> and <code>decode_codes</code>. We want the first to turn a string into an array of integers. The second one should complement it, turning the array of integers into the resulting string.</p>
<p>It’s a good idea to use our <code>hypothesis</code> library for this unit test. It’s going to come up with many input examples, trying to falsify our assumptions:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(st.text(alphabet=<span style="color:#d20;background-color:#fff0f0">"abcdefghijk1234!@#$%^&*"</span>, max_size=<span style="color:#00d;font-weight:bold">10</span>))
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_encode_and_decode_work</span>(self, text):
assume(text != <span style="color:#d20;background-color:#fff0f0">''</span>)
params = { <span style="color:#d20;background-color:#fff0f0">'alphabet'</span>: <span style="color:#d20;background-color:#fff0f0">'abcdefghijk1234!@#$%^&*'</span> }
label_ph = tf.placeholder(tf.string, shape=(<span style="color:#00d;font-weight:bold">1</span>), name=<span style="color:#d20;background-color:#fff0f0">'text'</span>)
codes_op = encode_labels(label_ph, params)
decode_op = decode_codes(codes_op, params)
<span style="color:#080;font-weight:bold">with</span> tf.Session() <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
session.run(tf.tables_initializer(name=<span style="color:#d20;background-color:#fff0f0">'init_all_tables'</span>))
codes, decoded = session.run(
[codes_op, decode_op],
{
label_ph: np.array([text])
}
)
note(codes)
note(decoded)
self.assertEqual(text, <span style="color:#d20;background-color:#fff0f0">''</span>.join(<span style="color:#038">map</span>(<span style="color:#080;font-weight:bold">lambda</span> s: s.decode(<span style="color:#d20;background-color:#fff0f0">'UTF-8'</span>), decoded.values)))
self.assertEqual(codes.values.dtype, np.int32)
self.assertEqual(<span style="color:#038">len</span>(codes.values), <span style="color:#038">len</span>(text))
</code></pre></div><p>Here is the implementation that passes the above test:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">encode_labels</span>(labels, params):
characters = <span style="color:#038">list</span>(params[<span style="color:#d20;background-color:#fff0f0">'alphabet'</span>])
table = tf.contrib.lookup.HashTable(
tf.contrib.lookup.KeyValueTensorInitializer(
characters,
<span style="color:#038">list</span>(<span style="color:#038">range</span>(<span style="color:#038">len</span>(characters)))
),
-<span style="color:#00d;font-weight:bold">1</span>,
name=<span style="color:#d20;background-color:#fff0f0">'char2id'</span>
)
<span style="color:#080;font-weight:bold">return</span> table.lookup(
tf.string_split(labels, delimiter=<span style="color:#d20;background-color:#fff0f0">''</span>)
)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">decode_codes</span>(codes, params):
characters = <span style="color:#038">list</span>(params[<span style="color:#d20;background-color:#fff0f0">'alphabet'</span>])
table = tf.contrib.lookup.HashTable(
tf.contrib.lookup.KeyValueTensorInitializer(
<span style="color:#038">list</span>(<span style="color:#038">range</span>(<span style="color:#038">len</span>(characters))),
characters
),
<span style="color:#d20;background-color:#fff0f0">''</span>,
name=<span style="color:#d20;background-color:#fff0f0">'id2char'</span>
)
<span style="color:#080;font-weight:bold">return</span> table.lookup(codes)
</code></pre></div><h5 id="log-mel-spectrogram-layer">Log-Mel Spectrogram layer</h5>
<p>Another piece we need is a way to turn raw audio signals into the log-Mel spectrograms. The idea, again, is to push it into the network graph. This way it’s going to work way faster on GPUs and also the model’s API is going to be much simpler.</p>
<p>In the following unit test, we’re testing our custom TensorFlow layer against values coming from known-to-be-valid librosa:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(
st.sampled_from([<span style="color:#00d;font-weight:bold">22000</span>, <span style="color:#00d;font-weight:bold">16000</span>, <span style="color:#00d;font-weight:bold">8000</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">1024</span>, <span style="color:#00d;font-weight:bold">512</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">1024</span>, <span style="color:#00d;font-weight:bold">512</span>]),
npst.arrays(
np.float32,
(<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">16000</span>),
elements=st.floats(-<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">1</span>)
)
)
<span style="color:#555">@settings</span>(max_examples=<span style="color:#00d;font-weight:bold">10</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_log_mel_conversion_works</span>(self, sampling_rate, n_fft, frame_step, audio):
lower_edge_hertz=<span style="color:#00d;font-weight:bold">0.0</span>
upper_edge_hertz=sampling_rate / <span style="color:#00d;font-weight:bold">2.0</span>
num_mel_bins=<span style="color:#00d;font-weight:bold">64</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">librosa_melspectrogram</span>(audio_item):
spectrogram = np.abs(
librosa.core.stft(
audio_item,
n_fft=n_fft,
hop_length=frame_step,
center=<span style="color:#080;font-weight:bold">False</span>
)
)**<span style="color:#00d;font-weight:bold">2</span>
<span style="color:#080;font-weight:bold">return</span> np.log(
librosa.feature.melspectrogram(
S=spectrogram,
sr=sampling_rate,
n_mels=num_mel_bins,
fmin=lower_edge_hertz,
fmax=upper_edge_hertz,
) + <span style="color:#00d;font-weight:bold">1e-6</span>
)
audio_ph = tf.placeholder(tf.float32, (<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">16000</span>))
librosa_log_mels = np.transpose(
np.stack([
librosa_melspectrogram(audio_item)
<span style="color:#080;font-weight:bold">for</span> audio_item <span style="color:#080">in</span> audio
]),
(<span style="color:#00d;font-weight:bold">0</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">1</span>)
)
log_mel_op = tf.check_numerics(
LogMelSpectrogram(
sampling_rate=sampling_rate,
n_fft=n_fft,
frame_step=frame_step,
lower_edge_hertz=lower_edge_hertz,
upper_edge_hertz=upper_edge_hertz,
num_mel_bins=num_mel_bins
)(audio_ph),
message=<span style="color:#d20;background-color:#fff0f0">"log mels"</span>
)
<span style="color:#080;font-weight:bold">with</span> tf.Session() <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
log_mels = session.run(
log_mel_op,
{
audio_ph: audio
}
)
np.testing.assert_allclose(
log_mels,
librosa_log_mels,
rtol=<span style="color:#00d;font-weight:bold">1e-1</span>,
atol=<span style="color:#00d;font-weight:bold">0</span>
)
</code></pre></div><p>The implementation of the layer, that passes the above unit test reads as follows:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">LogMelSpectrogram</span>(tf.layers.Layer):
<span style="color:#080;font-weight:bold">def</span> __init__(self,
sampling_rate,
n_fft,
frame_step,
lower_edge_hertz,
upper_edge_hertz,
num_mel_bins,
**kwargs):
<span style="color:#038">super</span>(LogMelSpectrogram, self).__init__(**kwargs)
self.sampling_rate = sampling_rate
self.n_fft = n_fft
self.frame_step = frame_step
self.lower_edge_hertz = lower_edge_hertz
self.upper_edge_hertz = upper_edge_hertz
self.num_mel_bins = num_mel_bins
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(self, inputs, training=<span style="color:#080;font-weight:bold">True</span>):
stfts = tf.contrib.signal.stft(
inputs,
frame_length=self.n_fft,
frame_step=self.frame_step,
fft_length=self.n_fft,
pad_end=<span style="color:#080;font-weight:bold">False</span>
)
power_spectrograms = tf.real(stfts * tf.conj(stfts))
num_spectrogram_bins = power_spectrograms.shape[-<span style="color:#00d;font-weight:bold">1</span>].value
linear_to_mel_weight_matrix = tf.constant(
np.transpose(
librosa.filters.mel(
sr=self.sampling_rate,
n_fft=self.n_fft + <span style="color:#00d;font-weight:bold">1</span>,
n_mels=self.num_mel_bins,
fmin=self.lower_edge_hertz,
fmax=self.upper_edge_hertz
)
),
dtype=tf.float32
)
mel_spectrograms = tf.tensordot(
power_spectrograms,
linear_to_mel_weight_matrix,
<span style="color:#00d;font-weight:bold">1</span>
)
mel_spectrograms.set_shape(
power_spectrograms.shape[:-<span style="color:#00d;font-weight:bold">1</span>].concatenate(
linear_to_mel_weight_matrix.shape[-<span style="color:#00d;font-weight:bold">1</span>:]
)
)
<span style="color:#080;font-weight:bold">return</span> tf.log(mel_spectrograms + <span style="color:#00d;font-weight:bold">1e-6</span>)
</code></pre></div><h5 id="converted-data-lengths-function">Converted data lengths function</h5>
<p>In order to use the CTC loss and decoder efficiently, we need to pass it the length of the data effectively representing audio for each batch. This is because not all audio files are of the same length but we need to pad them with zeros to do mini-batch.</p>
<p>Here’s the unit test:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(
npst.arrays(
np.float32,
(st.integers(min_value=<span style="color:#00d;font-weight:bold">16000</span>, max_value=<span style="color:#00d;font-weight:bold">16000</span>*<span style="color:#00d;font-weight:bold">5</span>)),
elements=st.floats(-<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">1</span>)
),
st.sampled_from([<span style="color:#00d;font-weight:bold">22000</span>, <span style="color:#00d;font-weight:bold">16000</span>, <span style="color:#00d;font-weight:bold">8000</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">1024</span>, <span style="color:#00d;font-weight:bold">512</span>, <span style="color:#00d;font-weight:bold">640</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">1024</span>, <span style="color:#00d;font-weight:bold">512</span>, <span style="color:#00d;font-weight:bold">160</span>]),
)
<span style="color:#555">@settings</span>(max_examples=<span style="color:#00d;font-weight:bold">10</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_compute_lengths_works</span>(self,
audio_wave,
sampling_rate,
n_fft,
frame_step
):
assume(n_fft >= frame_step)
original_wave_length = audio_wave.shape[<span style="color:#00d;font-weight:bold">0</span>]
audio_waves_ph = tf.placeholder(tf.float32, (<span style="color:#080;font-weight:bold">None</span>, <span style="color:#080;font-weight:bold">None</span>), name=<span style="color:#d20;background-color:#fff0f0">"audio_waves"</span>)
original_lengths_ph = tf.placeholder(tf.int32, (<span style="color:#080;font-weight:bold">None</span>), name=<span style="color:#d20;background-color:#fff0f0">"original_lengths"</span>)
lengths_op = compute_lengths(
original_lengths_ph,
{
<span style="color:#d20;background-color:#fff0f0">'frame_step'</span>: frame_step,
<span style="color:#d20;background-color:#fff0f0">'n_fft'</span>: n_fft
}
)
self.assertEqual(lengths_op.dtype, tf.int32)
log_mel_op = LogMelSpectrogram(
sampling_rate=sampling_rate,
n_fft=n_fft,
frame_step=frame_step,
lower_edge_hertz=<span style="color:#00d;font-weight:bold">0.0</span>,
upper_edge_hertz=<span style="color:#00d;font-weight:bold">8000.0</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">13</span>
)(audio_waves_ph)
<span style="color:#080;font-weight:bold">with</span> tf.Session() <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
lengths, log_mels = session.run(
[lengths_op, log_mel_op],
{
audio_waves_ph: np.array([audio_wave]),
original_lengths_ph: np.array([original_wave_length])
}
)
note(original_wave_length)
note(lengths)
note(log_mels.shape)
self.assertEqual(lengths[<span style="color:#00d;font-weight:bold">0</span>], log_mels.shape[<span style="color:#00d;font-weight:bold">1</span>])
</code></pre></div><p>And here’s the implementation:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">compute_lengths</span>(original_lengths, params):
<span style="color:#d20;background-color:#fff0f0">"""
</span><span style="color:#d20;background-color:#fff0f0"> Computes the length of data for CTC
</span><span style="color:#d20;background-color:#fff0f0"> """</span>
<span style="color:#080;font-weight:bold">return</span> tf.cast(
tf.floor(
(tf.cast(original_lengths, dtype=tf.float32) - params[<span style="color:#d20;background-color:#fff0f0">'n_fft'</span>]) /
params[<span style="color:#d20;background-color:#fff0f0">'frame_step'</span>]
) + <span style="color:#00d;font-weight:bold">1</span>,
tf.int32
)
</code></pre></div><h5 id="atrous-1d-convolutions-layer">Atrous 1D Convolutions layer</h5>
<p>It’s also a good idea to ensure that our dilated convolutions layer behaves as in theory. TensorFlow already includes an ability to specify the dilations. The end result though may differ wildly based on the choice of other parameters.</p>
<p>Let’s ensure at least that it works as intended when we choose it to work in the “causal” mode. The unit test:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_causal_conv1d_works</span>(self):
conv_size2_dilation_1 = AtrousConv1D(
filters=<span style="color:#00d;font-weight:bold">1</span>,
kernel_size=<span style="color:#00d;font-weight:bold">2</span>,
dilation_rate=<span style="color:#00d;font-weight:bold">1</span>,
kernel_initializer=tf.ones_initializer(),
use_bias=<span style="color:#080;font-weight:bold">False</span>
)
conv_size3_dilation_1 = AtrousConv1D(
filters=<span style="color:#00d;font-weight:bold">1</span>,
kernel_size=<span style="color:#00d;font-weight:bold">3</span>,
dilation_rate=<span style="color:#00d;font-weight:bold">1</span>,
kernel_initializer=tf.ones_initializer(),
use_bias=<span style="color:#080;font-weight:bold">False</span>
)
conv_size2_dilation_2 = AtrousConv1D(
filters=<span style="color:#00d;font-weight:bold">1</span>,
kernel_size=<span style="color:#00d;font-weight:bold">2</span>,
dilation_rate=<span style="color:#00d;font-weight:bold">2</span>,
kernel_initializer=tf.ones_initializer(),
use_bias=<span style="color:#080;font-weight:bold">False</span>
)
conv_size2_dilation_3 = AtrousConv1D(
filters=<span style="color:#00d;font-weight:bold">1</span>,
kernel_size=<span style="color:#00d;font-weight:bold">2</span>,
dilation_rate=<span style="color:#00d;font-weight:bold">3</span>,
kernel_initializer=tf.ones_initializer(),
use_bias=<span style="color:#080;font-weight:bold">False</span>
)
data = np.array(<span style="color:#038">list</span>(<span style="color:#038">range</span>(<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">31</span>)))
data_ph = tf.placeholder(tf.float32, (<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">30</span>, <span style="color:#00d;font-weight:bold">1</span>))
size2_dilation_1_1 = conv_size2_dilation_1(data_ph)
size2_dilation_1_2 = conv_size2_dilation_1(size2_dilation_1_1)
size3_dilation_1_1 = conv_size3_dilation_1(data_ph)
size3_dilation_1_2 = conv_size3_dilation_1(size3_dilation_1_1)
size2_dilation_2_1 = conv_size2_dilation_2(data_ph)
size2_dilation_2_2 = conv_size2_dilation_2(size2_dilation_2_1)
size2_dilation_3_1 = conv_size2_dilation_3(data_ph)
size2_dilation_3_2 = conv_size2_dilation_3(size2_dilation_3_1)
<span style="color:#080;font-weight:bold">with</span> tf.Session() <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
outputs = session.run(
[
size2_dilation_1_1,
size2_dilation_1_2,
size3_dilation_1_1,
size3_dilation_1_2,
size2_dilation_2_1,
size2_dilation_2_2,
size2_dilation_3_1,
size2_dilation_3_2
],
{
data_ph: np.reshape(data, (<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">30</span>, <span style="color:#00d;font-weight:bold">1</span>))
}
)
<span style="color:#080;font-weight:bold">for</span> ix, out <span style="color:#080">in</span> <span style="color:#038">enumerate</span>(outputs):
out = np.squeeze(out)
outputs[ix] = out
self.assertEqual(out.shape[<span style="color:#00d;font-weight:bold">0</span>], <span style="color:#038">len</span>(data))
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">0</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">5</span>, <span style="color:#00d;font-weight:bold">7</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">11</span>, <span style="color:#00d;font-weight:bold">13</span>, <span style="color:#00d;font-weight:bold">15</span>, <span style="color:#00d;font-weight:bold">17</span>, <span style="color:#00d;font-weight:bold">19</span>, <span style="color:#00d;font-weight:bold">21</span>, <span style="color:#00d;font-weight:bold">23</span>, <span style="color:#00d;font-weight:bold">25</span>, <span style="color:#00d;font-weight:bold">27</span>, <span style="color:#00d;font-weight:bold">29</span>, <span style="color:#00d;font-weight:bold">31</span>, <span style="color:#00d;font-weight:bold">33</span>, <span style="color:#00d;font-weight:bold">35</span>, <span style="color:#00d;font-weight:bold">37</span>, <span style="color:#00d;font-weight:bold">39</span>, <span style="color:#00d;font-weight:bold">41</span>, <span style="color:#00d;font-weight:bold">43</span>, <span style="color:#00d;font-weight:bold">45</span>, <span style="color:#00d;font-weight:bold">47</span>, <span style="color:#00d;font-weight:bold">49</span>, <span style="color:#00d;font-weight:bold">51</span>, <span style="color:#00d;font-weight:bold">53</span>, <span style="color:#00d;font-weight:bold">55</span>, <span style="color:#00d;font-weight:bold">57</span>, <span style="color:#00d;font-weight:bold">59</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">1</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">8</span>, <span style="color:#00d;font-weight:bold">12</span>, <span style="color:#00d;font-weight:bold">16</span>, <span style="color:#00d;font-weight:bold">20</span>, <span style="color:#00d;font-weight:bold">24</span>, <span style="color:#00d;font-weight:bold">28</span>, <span style="color:#00d;font-weight:bold">32</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">40</span>, <span style="color:#00d;font-weight:bold">44</span>, <span style="color:#00d;font-weight:bold">48</span>, <span style="color:#00d;font-weight:bold">52</span>, <span style="color:#00d;font-weight:bold">56</span>, <span style="color:#00d;font-weight:bold">60</span>, <span style="color:#00d;font-weight:bold">64</span>, <span style="color:#00d;font-weight:bold">68</span>, <span style="color:#00d;font-weight:bold">72</span>, <span style="color:#00d;font-weight:bold">76</span>, <span style="color:#00d;font-weight:bold">80</span>, <span style="color:#00d;font-weight:bold">84</span>, <span style="color:#00d;font-weight:bold">88</span>, <span style="color:#00d;font-weight:bold">92</span>, <span style="color:#00d;font-weight:bold">96</span>, <span style="color:#00d;font-weight:bold">100</span>, <span style="color:#00d;font-weight:bold">104</span>, <span style="color:#00d;font-weight:bold">108</span>, <span style="color:#00d;font-weight:bold">112</span>, <span style="color:#00d;font-weight:bold">116</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">2</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">6</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">12</span>, <span style="color:#00d;font-weight:bold">15</span>, <span style="color:#00d;font-weight:bold">18</span>, <span style="color:#00d;font-weight:bold">21</span>, <span style="color:#00d;font-weight:bold">24</span>, <span style="color:#00d;font-weight:bold">27</span>, <span style="color:#00d;font-weight:bold">30</span>, <span style="color:#00d;font-weight:bold">33</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">39</span>, <span style="color:#00d;font-weight:bold">42</span>, <span style="color:#00d;font-weight:bold">45</span>, <span style="color:#00d;font-weight:bold">48</span>, <span style="color:#00d;font-weight:bold">51</span>, <span style="color:#00d;font-weight:bold">54</span>, <span style="color:#00d;font-weight:bold">57</span>, <span style="color:#00d;font-weight:bold">60</span>, <span style="color:#00d;font-weight:bold">63</span>, <span style="color:#00d;font-weight:bold">66</span>, <span style="color:#00d;font-weight:bold">69</span>, <span style="color:#00d;font-weight:bold">72</span>, <span style="color:#00d;font-weight:bold">75</span>, <span style="color:#00d;font-weight:bold">78</span>, <span style="color:#00d;font-weight:bold">81</span>, <span style="color:#00d;font-weight:bold">84</span>, <span style="color:#00d;font-weight:bold">87</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">3</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">10</span>, <span style="color:#00d;font-weight:bold">18</span>, <span style="color:#00d;font-weight:bold">27</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">45</span>, <span style="color:#00d;font-weight:bold">54</span>, <span style="color:#00d;font-weight:bold">63</span>, <span style="color:#00d;font-weight:bold">72</span>, <span style="color:#00d;font-weight:bold">81</span>, <span style="color:#00d;font-weight:bold">90</span>, <span style="color:#00d;font-weight:bold">99</span>, <span style="color:#00d;font-weight:bold">108</span>, <span style="color:#00d;font-weight:bold">117</span>, <span style="color:#00d;font-weight:bold">126</span>, <span style="color:#00d;font-weight:bold">135</span>, <span style="color:#00d;font-weight:bold">144</span>, <span style="color:#00d;font-weight:bold">153</span>, <span style="color:#00d;font-weight:bold">162</span>, <span style="color:#00d;font-weight:bold">171</span>, <span style="color:#00d;font-weight:bold">180</span>, <span style="color:#00d;font-weight:bold">189</span>, <span style="color:#00d;font-weight:bold">198</span>, <span style="color:#00d;font-weight:bold">207</span>, <span style="color:#00d;font-weight:bold">216</span>, <span style="color:#00d;font-weight:bold">225</span>, <span style="color:#00d;font-weight:bold">234</span>, <span style="color:#00d;font-weight:bold">243</span>, <span style="color:#00d;font-weight:bold">252</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">4</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">6</span>, <span style="color:#00d;font-weight:bold">8</span>, <span style="color:#00d;font-weight:bold">10</span>, <span style="color:#00d;font-weight:bold">12</span>, <span style="color:#00d;font-weight:bold">14</span>, <span style="color:#00d;font-weight:bold">16</span>, <span style="color:#00d;font-weight:bold">18</span>, <span style="color:#00d;font-weight:bold">20</span>, <span style="color:#00d;font-weight:bold">22</span>, <span style="color:#00d;font-weight:bold">24</span>, <span style="color:#00d;font-weight:bold">26</span>, <span style="color:#00d;font-weight:bold">28</span>, <span style="color:#00d;font-weight:bold">30</span>, <span style="color:#00d;font-weight:bold">32</span>, <span style="color:#00d;font-weight:bold">34</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">38</span>, <span style="color:#00d;font-weight:bold">40</span>, <span style="color:#00d;font-weight:bold">42</span>, <span style="color:#00d;font-weight:bold">44</span>, <span style="color:#00d;font-weight:bold">46</span>, <span style="color:#00d;font-weight:bold">48</span>, <span style="color:#00d;font-weight:bold">50</span>, <span style="color:#00d;font-weight:bold">52</span>, <span style="color:#00d;font-weight:bold">54</span>, <span style="color:#00d;font-weight:bold">56</span>, <span style="color:#00d;font-weight:bold">58</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">5</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">5</span>, <span style="color:#00d;font-weight:bold">8</span>, <span style="color:#00d;font-weight:bold">12</span>, <span style="color:#00d;font-weight:bold">16</span>, <span style="color:#00d;font-weight:bold">20</span>, <span style="color:#00d;font-weight:bold">24</span>, <span style="color:#00d;font-weight:bold">28</span>, <span style="color:#00d;font-weight:bold">32</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">40</span>, <span style="color:#00d;font-weight:bold">44</span>, <span style="color:#00d;font-weight:bold">48</span>, <span style="color:#00d;font-weight:bold">52</span>, <span style="color:#00d;font-weight:bold">56</span>, <span style="color:#00d;font-weight:bold">60</span>, <span style="color:#00d;font-weight:bold">64</span>, <span style="color:#00d;font-weight:bold">68</span>, <span style="color:#00d;font-weight:bold">72</span>, <span style="color:#00d;font-weight:bold">76</span>, <span style="color:#00d;font-weight:bold">80</span>, <span style="color:#00d;font-weight:bold">84</span>, <span style="color:#00d;font-weight:bold">88</span>, <span style="color:#00d;font-weight:bold">92</span>, <span style="color:#00d;font-weight:bold">96</span>, <span style="color:#00d;font-weight:bold">100</span>, <span style="color:#00d;font-weight:bold">104</span>, <span style="color:#00d;font-weight:bold">108</span>, <span style="color:#00d;font-weight:bold">112</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">6</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">5</span>, <span style="color:#00d;font-weight:bold">7</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">11</span>, <span style="color:#00d;font-weight:bold">13</span>, <span style="color:#00d;font-weight:bold">15</span>, <span style="color:#00d;font-weight:bold">17</span>, <span style="color:#00d;font-weight:bold">19</span>, <span style="color:#00d;font-weight:bold">21</span>, <span style="color:#00d;font-weight:bold">23</span>, <span style="color:#00d;font-weight:bold">25</span>, <span style="color:#00d;font-weight:bold">27</span>, <span style="color:#00d;font-weight:bold">29</span>, <span style="color:#00d;font-weight:bold">31</span>, <span style="color:#00d;font-weight:bold">33</span>, <span style="color:#00d;font-weight:bold">35</span>, <span style="color:#00d;font-weight:bold">37</span>, <span style="color:#00d;font-weight:bold">39</span>, <span style="color:#00d;font-weight:bold">41</span>, <span style="color:#00d;font-weight:bold">43</span>, <span style="color:#00d;font-weight:bold">45</span>, <span style="color:#00d;font-weight:bold">47</span>, <span style="color:#00d;font-weight:bold">49</span>, <span style="color:#00d;font-weight:bold">51</span>, <span style="color:#00d;font-weight:bold">53</span>, <span style="color:#00d;font-weight:bold">55</span>, <span style="color:#00d;font-weight:bold">57</span>], dtype=np.float32)
)
np.testing.assert_equal(
outputs[<span style="color:#00d;font-weight:bold">7</span>],
np.array([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">6</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">12</span>, <span style="color:#00d;font-weight:bold">16</span>, <span style="color:#00d;font-weight:bold">20</span>, <span style="color:#00d;font-weight:bold">24</span>, <span style="color:#00d;font-weight:bold">28</span>, <span style="color:#00d;font-weight:bold">32</span>, <span style="color:#00d;font-weight:bold">36</span>, <span style="color:#00d;font-weight:bold">40</span>, <span style="color:#00d;font-weight:bold">44</span>, <span style="color:#00d;font-weight:bold">48</span>, <span style="color:#00d;font-weight:bold">52</span>, <span style="color:#00d;font-weight:bold">56</span>, <span style="color:#00d;font-weight:bold">60</span>, <span style="color:#00d;font-weight:bold">64</span>, <span style="color:#00d;font-weight:bold">68</span>, <span style="color:#00d;font-weight:bold">72</span>, <span style="color:#00d;font-weight:bold">76</span>, <span style="color:#00d;font-weight:bold">80</span>, <span style="color:#00d;font-weight:bold">84</span>, <span style="color:#00d;font-weight:bold">88</span>, <span style="color:#00d;font-weight:bold">92</span>, <span style="color:#00d;font-weight:bold">96</span>, <span style="color:#00d;font-weight:bold">100</span>, <span style="color:#00d;font-weight:bold">104</span>, <span style="color:#00d;font-weight:bold">108</span>], dtype=np.float32)
)
</code></pre></div><p>And the layer’s code:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">AtrousConv1D</span>(tf.layers.Layer):
<span style="color:#080;font-weight:bold">def</span> __init__(self,
filters,
kernel_size,
dilation_rate,
use_bias=<span style="color:#080;font-weight:bold">True</span>,
kernel_initializer=tf.glorot_normal_initializer(),
causal=<span style="color:#080;font-weight:bold">True</span>
):
<span style="color:#038">super</span>(AtrousConv1D, self).__init__()
self.filters = filters
self.kernel_size = kernel_size
self.dilation_rate = dilation_rate
self.causal = causal
self.conv1d = tf.layers.Conv1D(
filters=filters,
kernel_size=kernel_size,
dilation_rate=dilation_rate,
padding=<span style="color:#d20;background-color:#fff0f0">'valid'</span> <span style="color:#080;font-weight:bold">if</span> causal <span style="color:#080;font-weight:bold">else</span> <span style="color:#d20;background-color:#fff0f0">'same'</span>,
use_bias=use_bias,
kernel_initializer=kernel_initializer
)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(self, inputs):
<span style="color:#080;font-weight:bold">if</span> self.causal:
padding = (self.kernel_size - <span style="color:#00d;font-weight:bold">1</span>) * self.dilation_rate
inputs = tf.pad(inputs, tf.constant([(<span style="color:#00d;font-weight:bold">0</span>, <span style="color:#00d;font-weight:bold">0</span>,), (<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">0</span>), (<span style="color:#00d;font-weight:bold">0</span>, <span style="color:#00d;font-weight:bold">0</span>)]) * padding)
<span style="color:#080;font-weight:bold">return</span> self.conv1d(inputs)
</code></pre></div><h5 id="residual-block-layer">Residual Block layer</h5>
<p>One aspect that wasn’t covered yet is the heavy usage of batch normalization. When coding the residual block layer, ensuring that batch normalization is properly applied when training and when inferring is one of the most important tasks.</p>
<p>Here’s the unit test:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(
npst.arrays(
np.float32,
(<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">16000</span>),
elements=st.floats(-<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">1</span>)
),
st.sampled_from([<span style="color:#00d;font-weight:bold">64</span>, <span style="color:#00d;font-weight:bold">32</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">7</span>, <span style="color:#00d;font-weight:bold">3</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">4</span>]),
)
<span style="color:#555">@settings</span>(max_examples=<span style="color:#00d;font-weight:bold">10</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_residual_block_works</span>(self, audio_waves, filters, size, dilation_rate):
<span style="color:#080;font-weight:bold">with</span> tf.Graph().as_default() <span style="color:#080;font-weight:bold">as</span> g:
audio_ph = tf.placeholder(tf.float32, (<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#080;font-weight:bold">None</span>))
log_mel_op = LogMelSpectrogram(
sampling_rate=<span style="color:#00d;font-weight:bold">16000</span>,
n_fft=<span style="color:#00d;font-weight:bold">512</span>,
frame_step=<span style="color:#00d;font-weight:bold">256</span>,
lower_edge_hertz=<span style="color:#00d;font-weight:bold">0</span>,
upper_edge_hertz=<span style="color:#00d;font-weight:bold">8000</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">10</span>
)(audio_ph)
expanded_op = tf.layers.Dense(filters)(log_mel_op)
_, block_op = ResidualBlock(
filters=filters,
kernel_size=size,
causal=<span style="color:#080;font-weight:bold">True</span>,
dilation_rate=dilation_rate
)(expanded_op, training=<span style="color:#080;font-weight:bold">True</span>)
<span style="color:#888"># really dumb loss function just for the sake</span>
<span style="color:#888"># of testing:</span>
loss_op = tf.reduce_sum(block_op)
variables = tf.trainable_variables()
self.assertTrue(<span style="color:#038">any</span>([<span style="color:#d20;background-color:#fff0f0">"batch_normalization"</span> <span style="color:#080">in</span> var.name <span style="color:#080;font-weight:bold">for</span> var <span style="color:#080">in</span> variables]))
grads_op = tf.gradients(
loss_op,
variables
)
<span style="color:#080;font-weight:bold">for</span> grad, var <span style="color:#080">in</span> <span style="color:#038">zip</span>(grads_op, variables):
<span style="color:#080;font-weight:bold">if</span> grad <span style="color:#080">is</span> <span style="color:#080;font-weight:bold">None</span>:
note(var)
self.assertTrue(grad <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>)
<span style="color:#080;font-weight:bold">with</span> tf.Session(graph=g) <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
result, expanded, grads, _ = session.run(
[block_op, expanded_op, grads_op, loss_op],
{
audio_ph: audio_waves
}
)
self.assertFalse(np.array_equal(result, expanded))
self.assertEqual(result.shape, expanded.shape)
self.assertEqual(<span style="color:#038">len</span>(grads), <span style="color:#038">len</span>(variables))
self.assertFalse(<span style="color:#038">any</span>([np.isnan(grad).any() <span style="color:#080;font-weight:bold">for</span> grad <span style="color:#080">in</span> grads]))
</code></pre></div><p>And here’s the implementation:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ResidualBlock</span>(tf.layers.Layer):
<span style="color:#080;font-weight:bold">def</span> __init__(self, filters, kernel_size, dilation_rate, causal, **kwargs):
<span style="color:#038">super</span>(ResidualBlock, self).__init__(**kwargs)
self.dilated_conv1 = AtrousConv1D(
filters=filters,
kernel_size=kernel_size,
dilation_rate=dilation_rate,
causal=causal
)
self.dilated_conv2 = AtrousConv1D(
filters=filters,
kernel_size=kernel_size,
dilation_rate=dilation_rate,
causal=causal
)
self.out = tf.layers.Conv1D(
filters=filters,
kernel_size=<span style="color:#00d;font-weight:bold">1</span>
)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(self, inputs, training=<span style="color:#080;font-weight:bold">True</span>):
data = tf.layers.batch_normalization(
inputs,
training=training
)
filters = self.dilated_conv1(data)
gates = self.dilated_conv2(data)
filters = tf.nn.tanh(filters)
gates = tf.nn.sigmoid(gates)
out = tf.nn.tanh(
self.out(
filters * gates
)
)
<span style="color:#080;font-weight:bold">return</span> out + inputs, out
</code></pre></div><h5 id="residual-stack-layer">Residual Stack layer</h5>
<p>Testing the residual stack follows the same kind of logic:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(
npst.arrays(
np.float32,
(<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">16000</span>),
elements=st.floats(-<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">1</span>)
),
st.sampled_from([<span style="color:#00d;font-weight:bold">64</span>, <span style="color:#00d;font-weight:bold">32</span>]),
st.sampled_from([<span style="color:#00d;font-weight:bold">7</span>, <span style="color:#00d;font-weight:bold">3</span>])
)
<span style="color:#555">@settings</span>(max_examples=<span style="color:#00d;font-weight:bold">10</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_residual_stack_works</span>(self, audio_waves, filters, size):
dilation_rates = [<span style="color:#00d;font-weight:bold">1</span>,<span style="color:#00d;font-weight:bold">2</span>,<span style="color:#00d;font-weight:bold">4</span>]
<span style="color:#080;font-weight:bold">with</span> tf.Graph().as_default() <span style="color:#080;font-weight:bold">as</span> g:
audio_ph = tf.placeholder(tf.float32, (<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#080;font-weight:bold">None</span>))
log_mel_op = LogMelSpectrogram(
sampling_rate=<span style="color:#00d;font-weight:bold">16000</span>,
n_fft=<span style="color:#00d;font-weight:bold">512</span>,
frame_step=<span style="color:#00d;font-weight:bold">256</span>,
lower_edge_hertz=<span style="color:#00d;font-weight:bold">0</span>,
upper_edge_hertz=<span style="color:#00d;font-weight:bold">8000</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">10</span>
)(audio_ph)
expanded_op = tf.layers.Dense(filters)(log_mel_op)
stack_op = ResidualStack(
filters=filters,
kernel_size=size,
causal=<span style="color:#080;font-weight:bold">True</span>,
dilation_rates=dilation_rates
)(expanded_op, training=<span style="color:#080;font-weight:bold">True</span>)
<span style="color:#888"># really dumb loss function just for the sake</span>
<span style="color:#888"># of testing:</span>
loss_op = tf.reduce_sum(stack_op)
variables = tf.trainable_variables()
self.assertTrue(<span style="color:#038">any</span>([<span style="color:#d20;background-color:#fff0f0">"batch_normalization"</span> <span style="color:#080">in</span> var.name <span style="color:#080;font-weight:bold">for</span> var <span style="color:#080">in</span> variables]))
grads_op = tf.gradients(
loss_op,
variables
)
<span style="color:#080;font-weight:bold">for</span> grad, var <span style="color:#080">in</span> <span style="color:#038">zip</span>(grads_op, variables):
<span style="color:#080;font-weight:bold">if</span> grad <span style="color:#080">is</span> <span style="color:#080;font-weight:bold">None</span>:
note(var)
self.assertTrue(grad <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>)
<span style="color:#080;font-weight:bold">with</span> tf.Session(graph=g) <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
result, expanded, grads, _ = session.run(
[stack_op, expanded_op, grads_op, loss_op],
{
audio_ph: audio_waves
}
)
self.assertFalse(np.array_equal(result, expanded))
self.assertEqual(result.shape, expanded.shape)
self.assertEqual(<span style="color:#038">len</span>(grads), <span style="color:#038">len</span>(variables))
self.assertFalse(<span style="color:#038">any</span>([np.isnan(grad).any() <span style="color:#080;font-weight:bold">for</span> grad <span style="color:#080">in</span> grads]))
</code></pre></div><p>With the layer’s code looking as follows:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ResidualStack</span>(tf.layers.Layer):
<span style="color:#080;font-weight:bold">def</span> __init__(self, filters, kernel_size, dilation_rates, causal, **kwargs):
<span style="color:#038">super</span>(ResidualStack, self).__init__(**kwargs)
self.blocks = [
ResidualBlock(
filters=filters,
kernel_size=kernel_size,
dilation_rate=dilation_rate,
causal=causal
)
<span style="color:#080;font-weight:bold">for</span> dilation_rate <span style="color:#080">in</span> dilation_rates
]
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(self, inputs, training=<span style="color:#080;font-weight:bold">True</span>):
data = inputs
skip = <span style="color:#00d;font-weight:bold">0</span>
<span style="color:#080;font-weight:bold">for</span> block <span style="color:#080">in</span> self.blocks:
data, current_skip = block(data, training=training)
skip += current_skip
<span style="color:#080;font-weight:bold">return</span> skip
</code></pre></div><h5 id="the-speechnet">The SpeechNet</h5>
<p>Finally, let’s add a very similar test for the SpeechNet itself:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#555">@given</span>(
npst.arrays(
np.float32,
(<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#00d;font-weight:bold">16000</span>),
elements=st.floats(-<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">1</span>)
)
)
<span style="color:#555">@settings</span>(max_examples=<span style="color:#00d;font-weight:bold">10</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test_speech_net_works</span>(self, audio_waves):
<span style="color:#080;font-weight:bold">with</span> tf.Graph().as_default() <span style="color:#080;font-weight:bold">as</span> g:
audio_ph = tf.placeholder(tf.float32, (<span style="color:#00d;font-weight:bold">4</span>, <span style="color:#080;font-weight:bold">None</span>))
logits_op = SpeechNet(
experiment_params(
{},
stack_dilation_rates= [<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">2</span>, <span style="color:#00d;font-weight:bold">4</span>],
stack_kernel_size= <span style="color:#00d;font-weight:bold">3</span>,
stack_filters= <span style="color:#00d;font-weight:bold">32</span>,
alphabet= <span style="color:#d20;background-color:#fff0f0">'abcd'</span>
)
)(audio_ph)
<span style="color:#888"># really dumb loss function just for the sake</span>
<span style="color:#888"># of testing:</span>
loss_op = tf.reduce_sum(logits_op)
variables = tf.trainable_variables()
self.assertTrue(<span style="color:#038">any</span>([<span style="color:#d20;background-color:#fff0f0">"batch_normalization"</span> <span style="color:#080">in</span> var.name <span style="color:#080;font-weight:bold">for</span> var <span style="color:#080">in</span> variables]))
grads_op = tf.gradients(
loss_op,
variables
)
<span style="color:#080;font-weight:bold">for</span> grad, var <span style="color:#080">in</span> <span style="color:#038">zip</span>(grads_op, variables):
<span style="color:#080;font-weight:bold">if</span> grad <span style="color:#080">is</span> <span style="color:#080;font-weight:bold">None</span>:
note(var)
self.assertTrue(grad <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>)
<span style="color:#080;font-weight:bold">with</span> tf.Session(graph=g) <span style="color:#080;font-weight:bold">as</span> session:
session.run(tf.global_variables_initializer())
result, grads, _ = session.run(
[logits_op, grads_op, loss_op],
{
audio_ph: audio_waves
}
)
self.assertEqual(result.shape[<span style="color:#00d;font-weight:bold">2</span>], <span style="color:#00d;font-weight:bold">5</span>)
self.assertEqual(<span style="color:#038">len</span>(grads), <span style="color:#038">len</span>(variables))
self.assertFalse(<span style="color:#038">any</span>([np.isnan(grad).any() <span style="color:#080;font-weight:bold">for</span> grad <span style="color:#080">in</span> grads]))
</code></pre></div><p>And let’s provide the code that passes it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">SpeechNet</span>(tf.layers.Layer):
<span style="color:#080;font-weight:bold">def</span> __init__(self, params, **kwargs):
<span style="color:#038">super</span>(SpeechNet, self).__init__(**kwargs)
self.to_log_mel = LogMelSpectrogram(
sampling_rate=params[<span style="color:#d20;background-color:#fff0f0">'sampling_rate'</span>],
n_fft=params[<span style="color:#d20;background-color:#fff0f0">'n_fft'</span>],
frame_step=params[<span style="color:#d20;background-color:#fff0f0">'frame_step'</span>],
lower_edge_hertz=params[<span style="color:#d20;background-color:#fff0f0">'lower_edge_hertz'</span>],
upper_edge_hertz=params[<span style="color:#d20;background-color:#fff0f0">'upper_edge_hertz'</span>],
num_mel_bins=params[<span style="color:#d20;background-color:#fff0f0">'num_mel_bins'</span>]
)
self.expand = tf.layers.Conv1D(
filters=params[<span style="color:#d20;background-color:#fff0f0">'stack_filters'</span>],
kernel_size=<span style="color:#00d;font-weight:bold">1</span>,
padding=<span style="color:#d20;background-color:#fff0f0">'same'</span>
)
self.stacks = [
ResidualStack(
filters=params[<span style="color:#d20;background-color:#fff0f0">'stack_filters'</span>],
kernel_size=params[<span style="color:#d20;background-color:#fff0f0">'stack_kernel_size'</span>],
dilation_rates=params[<span style="color:#d20;background-color:#fff0f0">'stack_dilation_rates'</span>],
causal=params[<span style="color:#d20;background-color:#fff0f0">'causal_convolutions'</span>]
)
<span style="color:#080;font-weight:bold">for</span> _ <span style="color:#080">in</span> <span style="color:#038">range</span>(params[<span style="color:#d20;background-color:#fff0f0">'stacks'</span>])
]
self.out = tf.layers.Conv1D(
filters=<span style="color:#038">len</span>(params[<span style="color:#d20;background-color:#fff0f0">'alphabet'</span>]) + <span style="color:#00d;font-weight:bold">1</span>,
kernel_size=<span style="color:#00d;font-weight:bold">1</span>,
padding=<span style="color:#d20;background-color:#fff0f0">'same'</span>
)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call</span>(self, inputs, training=<span style="color:#080;font-weight:bold">True</span>):
data = self.to_log_mel(inputs)
data = tf.layers.batch_normalization(
data,
training=training
)
<span style="color:#080;font-weight:bold">if</span> <span style="color:#038">len</span>(data.shape) == <span style="color:#00d;font-weight:bold">2</span>:
data = tf.expand_dims(data, <span style="color:#00d;font-weight:bold">0</span>)
data = self.expand(data)
<span style="color:#080;font-weight:bold">for</span> stack <span style="color:#080">in</span> self.stacks:
data = stack(data, training=training)
data = tf.layers.batch_normalization(
data,
training=training
)
<span style="color:#080;font-weight:bold">return</span> self.out(data) + <span style="color:#00d;font-weight:bold">1e-8</span>
</code></pre></div><h5 id="the-model-function">The model function</h5>
<p>We have only one last piece of code to cover before we’ll be able to start the training. It’s the <code>model_fn</code> that adheres to the TensorFlow Estimators API:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">model_fn</span>(features, labels, mode, params):
<span style="color:#080;font-weight:bold">if</span> <span style="color:#038">isinstance</span>(features, <span style="color:#038">dict</span>):
audio = features[<span style="color:#d20;background-color:#fff0f0">'audio'</span>]
original_lengths = features[<span style="color:#d20;background-color:#fff0f0">'length'</span>]
<span style="color:#080;font-weight:bold">else</span>:
audio, original_lengths = features
lengths = compute_lengths(original_lengths, params)
<span style="color:#080;font-weight:bold">if</span> labels <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
codes = encode_labels(labels, params)
network = SpeechNet(params)
is_training = mode==tf.estimator.ModeKeys.TRAIN
logits = network(audio, training=is_training)
text, predicted_codes = decode_logits(logits, lengths, params)
<span style="color:#080;font-weight:bold">if</span> mode == tf.estimator.ModeKeys.PREDICT:
predictions = {
<span style="color:#d20;background-color:#fff0f0">'logits'</span>: logits,
<span style="color:#d20;background-color:#fff0f0">'text'</span>: tf.sparse_tensor_to_dense(
text,
<span style="color:#d20;background-color:#fff0f0">''</span>
)
}
export_outputs = {
<span style="color:#d20;background-color:#fff0f0">'predictions'</span>: tf.estimator.export.PredictOutput(predictions)
}
<span style="color:#080;font-weight:bold">return</span> tf.estimator.EstimatorSpec(
mode,
predictions=predictions,
export_outputs=export_outputs
)
<span style="color:#080;font-weight:bold">else</span>:
loss = tf.reduce_mean(
tf.nn.ctc_loss(
labels=codes,
inputs=logits,
sequence_length=lengths,
time_major=<span style="color:#080;font-weight:bold">False</span>,
ignore_longer_outputs_than_inputs=<span style="color:#080;font-weight:bold">True</span>
)
)
mean_edit_distance = tf.reduce_mean(
tf.edit_distance(
tf.cast(predicted_codes, tf.int32),
codes
)
)
distance_metric = tf.metrics.mean(mean_edit_distance)
<span style="color:#080;font-weight:bold">if</span> mode == tf.estimator.ModeKeys.EVAL:
<span style="color:#080;font-weight:bold">return</span> tf.estimator.EstimatorSpec(
mode,
loss=loss,
eval_metric_ops={ <span style="color:#d20;background-color:#fff0f0">'edit_distance'</span>: distance_metric }
)
<span style="color:#080;font-weight:bold">elif</span> mode == tf.estimator.ModeKeys.TRAIN:
global_step = tf.train.get_or_create_global_step()
tf.summary.text(
<span style="color:#d20;background-color:#fff0f0">'train_predicted_text'</span>,
tf.sparse_tensor_to_dense(text, <span style="color:#d20;background-color:#fff0f0">''</span>)
)
tf.summary.scalar(<span style="color:#d20;background-color:#fff0f0">'train_edit_distance'</span>, mean_edit_distance)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
<span style="color:#080;font-weight:bold">with</span> tf.control_dependencies(update_ops):
train_op = tf.contrib.layers.optimize_loss(
loss=loss,
global_step=global_step,
learning_rate=params[<span style="color:#d20;background-color:#fff0f0">'lr'</span>],
optimizer=(params[<span style="color:#d20;background-color:#fff0f0">'optimizer'</span>]),
update_ops=update_ops,
clip_gradients=params[<span style="color:#d20;background-color:#fff0f0">'clip_gradients'</span>],
summaries=[
<span style="color:#d20;background-color:#fff0f0">"learning_rate"</span>,
<span style="color:#d20;background-color:#fff0f0">"loss"</span>,
<span style="color:#d20;background-color:#fff0f0">"global_gradient_norm"</span>,
]
)
<span style="color:#080;font-weight:bold">return</span> tf.estimator.EstimatorSpec(
mode,
loss=loss,
train_op=train_op
)
</code></pre></div><p>Using the API, we’ll get lots of stats in TensorBoard for free. It will also make it very easy to validate the model and to export it to a <code>SavedModel</code> format.</p>
<p>In order to easily experiment with different hyperparameters, I’ve also created a helper function as listed below:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">copy</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">experiment</span>(data_params=dataset_params(), **kwargs):
params = experiment_params(
data_params,
**kwargs
)
<span style="color:#038">print</span>(params)
estimator = tf.estimator.Estimator(
model_fn=model_fn,
model_dir=<span style="color:#d20;background-color:#fff0f0">'stats/</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(experiment_name(params)),
params=params
)
<span style="color:#888">#import pdb; pdb.set_trace()</span>
train_spec = tf.estimator.TrainSpec(
input_fn=input_fn(
train_data,
params[<span style="color:#d20;background-color:#fff0f0">'data'</span>]
)
)
features = {
<span style="color:#d20;background-color:#fff0f0">"audio"</span>: tf.placeholder(dtype=tf.float32, shape=[<span style="color:#080;font-weight:bold">None</span>]),
<span style="color:#d20;background-color:#fff0f0">"length"</span>: tf.placeholder(dtype=tf.int32, shape=[])
}
serving_input_receiver_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(
features
)
best_exporter = tf.estimator.BestExporter(
name=<span style="color:#d20;background-color:#fff0f0">"best_exporter"</span>,
serving_input_receiver_fn=serving_input_receiver_fn,
exports_to_keep=<span style="color:#00d;font-weight:bold">5</span>
)
eval_params = copy.deepcopy(params[<span style="color:#d20;background-color:#fff0f0">'data'</span>])
eval_params[<span style="color:#d20;background-color:#fff0f0">'augment'</span>] = <span style="color:#080;font-weight:bold">False</span>
eval_spec = tf.estimator.EvalSpec(
input_fn=input_fn(
eval_data,
eval_params
),
throttle_secs=<span style="color:#00d;font-weight:bold">60</span>*<span style="color:#00d;font-weight:bold">30</span>,
exporters=best_exporter
)
tf.estimator.train_and_evaluate(
estimator,
train_spec,
eval_spec
)
</code></pre></div><p>As well as two more to test the model’s accuracy and to get the test set predictions:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">test</span>(data_params=dataset_params(), **kwargs):
params = experiment_params(
data_params,
**kwargs
)
<span style="color:#038">print</span>(params)
estimator = tf.estimator.Estimator(
model_fn=model_fn,
model_dir=<span style="color:#d20;background-color:#fff0f0">'stats/</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(experiment_name(params)),
params=params
)
eval_params = copy.deepcopy(params[<span style="color:#d20;background-color:#fff0f0">'data'</span>])
eval_params[<span style="color:#d20;background-color:#fff0f0">'augment'</span>] = <span style="color:#080;font-weight:bold">False</span>
eval_params[<span style="color:#d20;background-color:#fff0f0">'epochs'</span>] = <span style="color:#00d;font-weight:bold">1</span>
eval_params[<span style="color:#d20;background-color:#fff0f0">'shuffle'</span>] = <span style="color:#080;font-weight:bold">False</span>
estimator.evaluate(
input_fn=input_fn(
test_data,
eval_params
)
)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">predict_test</span>(**kwargs):
params = experiment_params(
dataset_params(
augment=<span style="color:#080;font-weight:bold">False</span>,
shuffle=<span style="color:#080;font-weight:bold">False</span>,
batch_size=<span style="color:#00d;font-weight:bold">1</span>,
epochs=<span style="color:#00d;font-weight:bold">1</span>,
parallelize=<span style="color:#080;font-weight:bold">False</span>
),
**kwargs
)
<span style="color:#038">print</span>(<span style="color:#038">len</span>(test_data))
estimator = tf.estimator.Estimator(
model_fn=model_fn,
model_dir=<span style="color:#d20;background-color:#fff0f0">'stats/</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(experiment_name(params)),
params=params
)
<span style="color:#080;font-weight:bold">return</span> <span style="color:#038">list</span>(
estimator.predict(
input_fn=input_fn(
test_data,
params[<span style="color:#d20;background-color:#fff0f0">'data'</span>]
)
)
)
</code></pre></div><p>Which depends on the following other functions:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">experiment_params</span>(data,
optimizer=<span style="color:#d20;background-color:#fff0f0">'Adam'</span>,
lr=<span style="color:#00d;font-weight:bold">1e-4</span>,
alphabet=<span style="color:#d20;background-color:#fff0f0">" 'abcdefghijklmnopqrstuvwxyz"</span>,
causal_convolutions=<span style="color:#080;font-weight:bold">True</span>,
stack_dilation_rates=[<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">27</span>, <span style="color:#00d;font-weight:bold">81</span>],
stacks=<span style="color:#00d;font-weight:bold">2</span>,
stack_kernel_size=<span style="color:#00d;font-weight:bold">3</span>,
stack_filters=<span style="color:#00d;font-weight:bold">32</span>,
sampling_rate=<span style="color:#00d;font-weight:bold">16000</span>,
n_fft=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">4</span>,
frame_step=<span style="color:#00d;font-weight:bold">160</span>,
lower_edge_hertz=<span style="color:#00d;font-weight:bold">0</span>,
upper_edge_hertz=<span style="color:#00d;font-weight:bold">8000</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">160</span>,
clip_gradients=<span style="color:#080;font-weight:bold">None</span>,
codename=<span style="color:#d20;background-color:#fff0f0">'regular'</span>,
**kwargs):
params = {
<span style="color:#d20;background-color:#fff0f0">'optimizer'</span>: optimizer,
<span style="color:#d20;background-color:#fff0f0">'lr'</span>: lr,
<span style="color:#d20;background-color:#fff0f0">'data'</span>: data,
<span style="color:#d20;background-color:#fff0f0">'alphabet'</span>: alphabet,
<span style="color:#d20;background-color:#fff0f0">'causal_convolutions'</span>: causal_convolutions,
<span style="color:#d20;background-color:#fff0f0">'stack_dilation_rates'</span>: stack_dilation_rates,
<span style="color:#d20;background-color:#fff0f0">'stacks'</span>: stacks,
<span style="color:#d20;background-color:#fff0f0">'stack_kernel_size'</span>: stack_kernel_size,
<span style="color:#d20;background-color:#fff0f0">'stack_filters'</span>: stack_filters,
<span style="color:#d20;background-color:#fff0f0">'sampling_rate'</span>: sampling_rate,
<span style="color:#d20;background-color:#fff0f0">'n_fft'</span>: n_fft,
<span style="color:#d20;background-color:#fff0f0">'frame_step'</span>: frame_step,
<span style="color:#d20;background-color:#fff0f0">'lower_edge_hertz'</span>: lower_edge_hertz,
<span style="color:#d20;background-color:#fff0f0">'upper_edge_hertz'</span>: upper_edge_hertz,
<span style="color:#d20;background-color:#fff0f0">'num_mel_bins'</span>: num_mel_bins,
<span style="color:#d20;background-color:#fff0f0">'clip_gradients'</span>: clip_gradients,
<span style="color:#d20;background-color:#fff0f0">'codename'</span>: codename
}
<span style="color:#888">#import pdb; pdb.set_trace()</span>
<span style="color:#080;font-weight:bold">if</span> kwargs <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span> <span style="color:#080">and</span> <span style="color:#d20;background-color:#fff0f0">'data'</span> <span style="color:#080">in</span> kwargs:
params[<span style="color:#d20;background-color:#fff0f0">'data'</span>] = { **params[<span style="color:#d20;background-color:#fff0f0">'data'</span>], **kwargs[<span style="color:#d20;background-color:#fff0f0">'data'</span>] }
<span style="color:#080;font-weight:bold">del</span> kwargs[<span style="color:#d20;background-color:#fff0f0">'data'</span>]
<span style="color:#080;font-weight:bold">if</span> kwargs <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>:
params = { **params, **kwargs }
<span style="color:#080;font-weight:bold">return</span> params
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">experiment_name</span>(params, excluded_keys=[<span style="color:#d20;background-color:#fff0f0">'alphabet'</span>, <span style="color:#d20;background-color:#fff0f0">'data'</span>, <span style="color:#d20;background-color:#fff0f0">'lr'</span>, <span style="color:#d20;background-color:#fff0f0">'clip_gradients'</span>]):
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">represent</span>(key, value):
<span style="color:#080;font-weight:bold">if</span> key <span style="color:#080">in</span> excluded_keys:
<span style="color:#080;font-weight:bold">return</span> <span style="color:#080;font-weight:bold">None</span>
<span style="color:#080;font-weight:bold">else</span>:
<span style="color:#080;font-weight:bold">if</span> <span style="color:#038">isinstance</span>(value, <span style="color:#038">list</span>):
<span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">'</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">_</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(key, <span style="color:#d20;background-color:#fff0f0">'_'</span>.join([<span style="color:#038">str</span>(v) <span style="color:#080;font-weight:bold">for</span> v <span style="color:#080">in</span> value]))
<span style="color:#080;font-weight:bold">else</span>:
<span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">'</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">_</span><span style="color:#33b;background-color:#fff0f0">{}</span><span style="color:#d20;background-color:#fff0f0">'</span>.format(key, value)
parts = <span style="color:#038">filter</span>(
<span style="color:#080;font-weight:bold">lambda</span> p: p <span style="color:#080">is</span> <span style="color:#080">not</span> <span style="color:#080;font-weight:bold">None</span>,
[
represent(k, params[k])
<span style="color:#080;font-weight:bold">for</span> k <span style="color:#080">in</span> <span style="color:#038">sorted</span>(params.keys())
]
)
<span style="color:#080;font-weight:bold">return</span> <span style="color:#d20;background-color:#fff0f0">'/'</span>.join(parts)
</code></pre></div><p>Each new set of hyperparameters constitutes a different “experiment”. It will output separate statistics in TensorBoard that are going to be easily filterable.</p>
<p>The <code>experiment</code> function uses the <code>train_and_validate</code> TensorFlow function which will periodically test the model against the validation set. This is our tool of gauging how well it generalizes. It also uses the <code>tf.estimator.BestExporter</code> class to automatically export <code>SavedModel</code> files for best performing versions.</p>
<h5 id="other-aspects">Other aspects</h5>
<p>The coverage of the full code listing wouldn’t be very practical for an article like this. We’ve covered the most important of them above. I invite you to have a look at the Jupyter notebook itself which is hosted on GitHub: <a href="https://github.com/kamilc/speech-recognition">kamilc/speech-recognition</a>.</p>
<h3 id="lets-train-it">Let’s train it</h3>
<p>Before we can dive in and start training the model using the code above, we need to set a few things up.</p>
<p>First of all, I’m using Docker. This way I’m not constrained e.g. by the version of Cuda to install.</p>
<p>Here’s the Dockerfile for this project:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-dockerfile" data-lang="dockerfile"><span style="color:#080;font-weight:bold">FROM</span><span style="color:#d20;background-color:#fff0f0"> tensorflow/tensorflow:latest-devel-gpu-py3</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> apt-get update<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> apt-get install -y ffmpeg git cmake<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> pip install matplotlib pandas scikit-learn librosa seaborn hickle hypothesis[pandas]<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> mkdir -p /home/data-science/projects<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">VOLUME</span><span style="color:#d20;background-color:#fff0f0"> /home/data-science/projects</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> <span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"c.NotebookApp.token = ''"</span> >> ~/.jupyter/jupyter_notebook_config.py<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> <span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"c.NotebookApp.password = ''"</span> >> ~/.jupyter/jupyter_notebook_config.py<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">WORKDIR</span><span style="color:#d20;background-color:#fff0f0"> /home/data-science/projects</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">RUN</span> pip install git+https://github.com/Supervisor/supervisor && <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> mkdir -p /var/log/supervisor<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">ADD</span> supervisor.conf /etc/supervisor.conf<span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">EXPOSE</span><span style="color:#d20;background-color:#fff0f0"> 80</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">EXPOSE</span><span style="color:#d20;background-color:#fff0f0"> 6006</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2">
</span><span style="color:#a61717;background-color:#e3d2d2"></span><span style="color:#080;font-weight:bold">CMD</span> supervisord -c /etc/supervisor.conf<span style="color:#a61717;background-color:#e3d2d2">
</span></code></pre></div><p>I also like to make my life easier and provide the Makefile that automates common project-related tasks:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-makefile" data-lang="makefile"><span style="color:#06b;font-weight:bold">build</span>:
nvidia-docker build -t speech-recognition:latest .
<span style="color:#06b;font-weight:bold">run</span>:
nvidia-docker run -p 80:80 -p 6006:6006 --shm-size 16G --mount <span style="color:#369">type</span>=bind,source=/home/kamil/projects/speech-recognition,target=/home/data-science/projects -it speech-recognition
<span style="color:#06b;font-weight:bold">bash</span>:
nvidia-docker run --mount <span style="color:#369">type</span>=bind,source=/home/kamil/projects/speech-recognition,target=/home/data-science/projects -it speech-recognition bash
</code></pre></div><p>We’ll use TensorBoard to visualize the progress. At the same time, we need Jupyter notebooks server to be running as well. We’ll need a supervisor daemon to run both at the same time in a container. Here’s its config file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ini" data-lang="ini"><span style="color:#080;font-weight:bold">[supervisord]</span>
<span style="color:#369">nodaemon</span>=<span style="color:#d20;background-color:#fff0f0">true</span>
<span style="color:#080;font-weight:bold">[program:jupyter]</span>
<span style="color:#369">command</span>=<span style="color:#d20;background-color:#fff0f0">bash -c "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/home/data-science/projects --ip 0.0.0.0 --no-browser --allow-root --port=80"</span>
<span style="color:#080;font-weight:bold">[program:tensorboard]</span>
<span style="color:#369">command</span>=<span style="color:#d20;background-color:#fff0f0">tensorboard --logdir /home/data-science/projects/stats</span>
</code></pre></div><p>In order to run the Jupyter notebook and start experimenting you’ll need to run the following in the command line:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">make build
</code></pre></div><p>And then to start the container with TensorFlow, Jupyter, and Tensorboard:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">make run
</code></pre></div><p>The notebook includes a helper function for running experiments. Here’s the invocation, whose set of parameters worked best for me:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">experiment(
dataset_params(
batch_size=<span style="color:#00d;font-weight:bold">18</span>,
epochs=<span style="color:#00d;font-weight:bold">10</span>,
max_wave_length=<span style="color:#00d;font-weight:bold">320000</span>,
augment=<span style="color:#080;font-weight:bold">True</span>,
random_noise=<span style="color:#00d;font-weight:bold">0.75</span>,
random_noise_factor_min=<span style="color:#00d;font-weight:bold">0.1</span>,
random_noise_factor_max=<span style="color:#00d;font-weight:bold">0.15</span>,
random_stretch_min=<span style="color:#00d;font-weight:bold">0.8</span>,
random_stretch_max=<span style="color:#00d;font-weight:bold">1.2</span>
),
codename=<span style="color:#d20;background-color:#fff0f0">'deep_max_20_seconds'</span>,
alphabet=<span style="color:#d20;background-color:#fff0f0">' !"&</span><span style="color:#04d;background-color:#fff0f0">\'</span><span style="color:#d20;background-color:#fff0f0">,-.01234:;</span><span style="color:#04d;background-color:#fff0f0">\\</span><span style="color:#d20;background-color:#fff0f0">abcdefghijklmnopqrstuvwxyz'</span>, <span style="color:#888"># !"&',-.01234:;\abcdefghijklmnopqrstuvwxyz</span>
causal_convolutions=<span style="color:#080;font-weight:bold">False</span>,
stack_dilation_rates=[<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">27</span>],
stacks=<span style="color:#00d;font-weight:bold">6</span>,
stack_kernel_size=<span style="color:#00d;font-weight:bold">7</span>,
stack_filters=<span style="color:#00d;font-weight:bold">3</span>*<span style="color:#00d;font-weight:bold">128</span>,
n_fft=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">8</span>,
frame_step=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">4</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">160</span>,
optimizer=<span style="color:#d20;background-color:#fff0f0">'Momentum'</span>,
lr=<span style="color:#00d;font-weight:bold">0.00001</span>,
clip_gradients=<span style="color:#00d;font-weight:bold">20.0</span>
)
</code></pre></div><p>The training process takes lots of time. On my machine, it took it more than 2 weeks. Searching for the best set of parameters is very difficult (and not fun).</p>
<p>The function accepts the <code>max_text_length</code> as one of its parameters. I first ran the experiments setting it to some small value (e.g. 15 characters). It constrains the data set to a narrow set of “easy” files. The reason is that it’s easy to spot any issues with the architecture on an easy set: if it’s not converging here, then we surely have a bug.</p>
<p>For the main training procedure, this parameter is kept unset.</p>
<h3 id="results">Results</h3>
<p>By using TensorBoard, we get a handy tool for monitoring the progress. I made the <code>model_fn</code> output statistics for the training set <a href="https://en.wikipedia.org/wiki/Edit_distance">edit distance</a> as well as the one for the evaluation set.</p>
<p>The statistics for the <a href="https://en.wikipedia.org/wiki/Connectionist_temporal_classification">CTC Loss</a> are included by default.</p>
<p>Here are the charts for the final model included in the GitHub repo:</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/training-1.png" alt=""></p>
<p>A thing to notice is that I paused the training between the 20th and 30th December.</p>
<p>The above chart presents the <strong>training time</strong> edit distance. Because of the pretty aggressive data augmentation, I noticed that throughout the whole process the training and validation edit distances didn’t differ hugely.</p>
<p>Following image shows the CTC loss with the orange line representing the evaluation runs.</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/training-2.png" alt=""></p>
<p>The evaluation edit distance is shown below. I stopped the training once the further gain for a whole day was dropping by less than <code>0.005</code>.</p>
<p><img src="/blog/2019/01/speech-recognition-with-tensorflow/training-3.png" alt=""></p>
<p>Every machine learning model should be rigorously measured against meaningful accuracy statistics. Let’s see how we did:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">test(
dataset_params(
batch_size=<span style="color:#00d;font-weight:bold">18</span>,
epochs=<span style="color:#00d;font-weight:bold">10</span>,
max_wave_length=<span style="color:#00d;font-weight:bold">320000</span>,
augment=<span style="color:#080;font-weight:bold">True</span>,
random_noise=<span style="color:#00d;font-weight:bold">0.75</span>,
random_noise_factor_min=<span style="color:#00d;font-weight:bold">0.1</span>,
random_noise_factor_max=<span style="color:#00d;font-weight:bold">0.15</span>,
random_stretch_min=<span style="color:#00d;font-weight:bold">0.8</span>,
random_stretch_max=<span style="color:#00d;font-weight:bold">1.2</span>
),
codename=<span style="color:#d20;background-color:#fff0f0">'deep_max_20_seconds'</span>,
alphabet=<span style="color:#d20;background-color:#fff0f0">' !"&</span><span style="color:#04d;background-color:#fff0f0">\'</span><span style="color:#d20;background-color:#fff0f0">,-.01234:;</span><span style="color:#04d;background-color:#fff0f0">\\</span><span style="color:#d20;background-color:#fff0f0">abcdefghijklmnopqrstuvwxyz'</span>, <span style="color:#888"># !"&',-.01234:;\abcdefghijklmnopqrstuvwxyz</span>
causal_convolutions=<span style="color:#080;font-weight:bold">False</span>,
stack_dilation_rates=[<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">27</span>],
stacks=<span style="color:#00d;font-weight:bold">6</span>,
stack_kernel_size=<span style="color:#00d;font-weight:bold">7</span>,
stack_filters=<span style="color:#00d;font-weight:bold">3</span>*<span style="color:#00d;font-weight:bold">128</span>,
n_fft=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">8</span>,
frame_step=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">4</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">160</span>,
optimizer=<span style="color:#d20;background-color:#fff0f0">'Momentum'</span>,
lr=<span style="color:#00d;font-weight:bold">0.00001</span>,
clip_gradients=<span style="color:#00d;font-weight:bold">20.0</span>
)
</code></pre></div><p>The output:</p>
<pre tabindex="0"><code>(...)
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Finished evaluation at 2019-01-07-10:51:09
INFO:tensorflow:Saving dict for global step 1525345: edit_distance = 0.07922124, global_step = 1525345, loss = 13.410753
(...)
</code></pre><p>This shows that for the test set, we’ve scored <code>0.079</code> in edit distance. We could invert it to call accuracy (somewhat naively though), which gives <code>92.1%</code> — not too bad. The result would be officially reported as <code>7.9 LER</code>.</p>
<p>What’s even nicer is the size of the model:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ls stats/causal_convolutions_False/codename_deep_max_20_seconds/frame_step_640/lower_edge_hertz_0/n_fft_1280/num_mel_bins_160/optimizer_Momentum/sampling_rate_16000/stack_dilation_rates_1_3_9_27/stack_filters_384/stack_kernel_size_7/stacks_6/upper_edge_hertz_8000/export/best_exporter/1546198558/variables -lh
total 204M
</code></pre></div><p>That’s <code>204MB</code> for the model trained on the 375k+ dataset with aggressive augmentation (which makes the resulting dataset size effectively a couple times bigger).</p>
<p>It’s always nice to <strong>see</strong> what the results look like. Here’s the code that runs the model through the whole test sets and gathers the predicted transcriptions:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">test_results = predict_test(
codename=<span style="color:#d20;background-color:#fff0f0">'deep_max_20_seconds'</span>,
alphabet=<span style="color:#d20;background-color:#fff0f0">' !"&</span><span style="color:#04d;background-color:#fff0f0">\'</span><span style="color:#d20;background-color:#fff0f0">,-.01234:;</span><span style="color:#04d;background-color:#fff0f0">\\</span><span style="color:#d20;background-color:#fff0f0">abcdefghijklmnopqrstuvwxyz'</span>, <span style="color:#888"># !"&',-.01234:;\abcdefghijklmnopqrstuvwxyz</span>
causal_convolutions=<span style="color:#080;font-weight:bold">False</span>,
stack_dilation_rates=[<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">27</span>],
stacks=<span style="color:#00d;font-weight:bold">6</span>,
stack_kernel_size=<span style="color:#00d;font-weight:bold">7</span>,
stack_filters=<span style="color:#00d;font-weight:bold">3</span>*<span style="color:#00d;font-weight:bold">128</span>,
n_fft=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">8</span>,
frame_step=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">4</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">160</span>,
optimizer=<span style="color:#d20;background-color:#fff0f0">'Momentum'</span>,
lr=<span style="color:#00d;font-weight:bold">0.00001</span>,
clip_gradients=<span style="color:#00d;font-weight:bold">20.0</span>
)
[ <span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">''</span>.join(t[<span style="color:#d20;background-color:#fff0f0">'text'</span>]) <span style="color:#080;font-weight:bold">for</span> t <span style="color:#080">in</span> test_results ]
</code></pre></div><p>And the excerpt of the above is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">[<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'without the dotaset the artice suistles'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"i've got to go to him"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'and you know it'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'down below in the darknes were hundrededs of people sleping in peace'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'strange images pased through my mind'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'the shep had taught him that'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'it was glaringly hot not a clou in hesky nor a breath of wind'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'your son went to serve at a distant place and became a cinturion'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'they made a boy continue tiging but he found nothing'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'the shoreas in da'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'fol the instructions here'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"the're caling to u not to give up and to kep on fighting"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'the shop was closed on monis'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'even coming down on the train together she wrote me'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"i'm going away he said"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"he wasn't asking for help"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'some of the grynsh was faling of the circular edge'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"i'd like to think"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'the alchemist robably already knew al that'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"you 'l take fiftly and like et"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'it was droping of in flakes and raining down on the sand'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"what's your name he asked"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"it's because you were not born"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'what do you think of that'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"if i had told tyo o you wouldn't have sen the pyramids"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"i havn't hert the baby complain yet"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'i told him wit could teach hr to ignore people who was had tend'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"the one you're blocking"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'henderson stod up with a spade in his hand'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"he didn't ned to sek out the old woman for this"</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'only a minority of literature is reaten this way'</span>,
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">"i wish you wouldn't"</span>,
...]
</code></pre></div><p>Seems quite okay. You can immediately notice that some words are misspelled. This stems from the nature of the CTC algorithm itself. We’re <strong>predicting letters</strong> instead of words here. The good side is that the problem of out-of-vocabulary words is lessened. The worse part is that you’ll get e.g. ‘sek’ sometimes instead of ‘seek’. Because we’re outputting the logits for each example, it’s possible to use e.g. the <a href="https://github.com/githubharald/CTCWordBeamSearch">CTCWordBeamSearch</a> to constrain the output’s tokens to ones known within the corpus — making it predict the words instead.</p>
<p>Here’s the last little fun test: speech to text on the utterance I created on my laptop:</p>
<audio controls="controls">
<source src="/blog/2019/01/speech-recognition-with-tensorflow/test-me.m4a" type="audio/wav">
</audio>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">results = predict(
<span style="color:#d20;background-color:#fff0f0">'cv_corpus_v1/test-me.m4a'</span>,
codename=<span style="color:#d20;background-color:#fff0f0">'deep_max_20_seconds'</span>,
alphabet=<span style="color:#d20;background-color:#fff0f0">' !"&</span><span style="color:#04d;background-color:#fff0f0">\'</span><span style="color:#d20;background-color:#fff0f0">,-.01234:;</span><span style="color:#04d;background-color:#fff0f0">\\</span><span style="color:#d20;background-color:#fff0f0">abcdefghijklmnopqrstuvwxyz'</span>, <span style="color:#888"># !"&',-.01234:;\abcdefghijklmnopqrstuvwxyz</span>
causal_convolutions=<span style="color:#080;font-weight:bold">False</span>,
stack_dilation_rates=[<span style="color:#00d;font-weight:bold">1</span>, <span style="color:#00d;font-weight:bold">3</span>, <span style="color:#00d;font-weight:bold">9</span>, <span style="color:#00d;font-weight:bold">27</span>],
stacks=<span style="color:#00d;font-weight:bold">6</span>,
stack_kernel_size=<span style="color:#00d;font-weight:bold">7</span>,
stack_filters=<span style="color:#00d;font-weight:bold">3</span>*<span style="color:#00d;font-weight:bold">128</span>,
n_fft=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">8</span>,
frame_step=<span style="color:#00d;font-weight:bold">160</span>*<span style="color:#00d;font-weight:bold">4</span>,
num_mel_bins=<span style="color:#00d;font-weight:bold">160</span>,
optimizer=<span style="color:#d20;background-color:#fff0f0">'Momentum'</span>,
lr=<span style="color:#00d;font-weight:bold">0.00001</span>,
clip_gradients=<span style="color:#00d;font-weight:bold">20.0</span>
)
<span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">''</span>.join(results[<span style="color:#00d;font-weight:bold">0</span>][<span style="color:#d20;background-color:#fff0f0">'text'</span>])
</code></pre></div><p>The result:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#d20;background-color:#fff0f0">b</span><span style="color:#d20;background-color:#fff0f0">'it semed to work just fine'</span>
</code></pre></div><h3 id="project-on-github">Project on GitHub</h3>
<p>The full Jupyter notebook’s code for this article can be found on GitHub: <a href="https://github.com/kamilc/speech-recognition">kamilc/speech-recognition</a>.</p>
<p>The repository includes the bz2 archive of the best performing model I’ve trained. You can download it and run it as a web service via <a href="https://www.tensorflow.org/serving/">TensorFlow Serving</a>, which we will cover in the next and last section here.</p>
<h3 id="serving-the-model-with-the-tensorflow-serving">Serving the model with the TensorFlow Serving</h3>
<p>The last step in this project is to serve our trained model as a web service. Thankfully, the TensorFlow project includes a ready to use “model server” that’s free to use: <a href="https://www.tensorflow.org/serving/">TensorFlow Serving</a>.</p>
<p>The idea behind it is that we can run it, pointing it at the directory containing the models saved in the TensorFlow’s SavedModel format.</p>
<p>The deployment is extremely straightforward if you’re okay with running it from a Docker container. Let’s first pull the image:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker pull tensorflow/serving
</code></pre></div><p>Next, we need to download the saved model we’ve trained in this article from GitHub:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ wget https://github.com/kamilc/speech-recognition/raw/master/best.tar.bz2
$ tar xvjf best.tar.bz2
</code></pre></div><p>In the next step, we need to start a container for the TensorFlow Serving image making it:</p>
<ul>
<li>open its port to outside</li>
<li>mount the directory containing our model</li>
<li>set the <code>MODEL_NAME</code> environment variable</li>
</ul>
<p>As follows:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker run -t --rm -p 8501:8501 -v <span style="color:#d20;background-color:#fff0f0">"/home/kamil/projects/speech-recognition/best/1546646971:/models/speech/1"</span> -e <span style="color:#369">MODEL_NAME</span>=speech tensorflow/serving
</code></pre></div><p>The service communicates via JSON payloads. Let’s prepare a payload.json file containing our request payload:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{<span style="color:#b06;font-weight:bold">"inputs"</span>: {<span style="color:#b06;font-weight:bold">"audio"</span>: <span style="color:#a61717;background-color:#e3d2d2"><audio-data-here></span>, <span style="color:#b06;font-weight:bold">"length"</span>: <span style="color:#a61717;background-color:#e3d2d2"><audio-raw-signal-length-here></span>}}
</code></pre></div><p>We can now easily query the web service with the prepared request audio data:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">curl -d @payload.json <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> -X POST http://localhost:8501/v1/models/speech:predict
</code></pre></div><p>Here’s what our intelligent web service responds with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"outputs"</span>: {
<span style="color:#b06;font-weight:bold">"text"</span>: [
[
<span style="color:#d20;background-color:#fff0f0">"c"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">"v"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">"r"</span>,
<span style="color:#d20;background-color:#fff0f0">"y"</span>,
<span style="color:#d20;background-color:#fff0f0">"t"</span>,
<span style="color:#d20;background-color:#fff0f0">"h"</span>,
<span style="color:#d20;background-color:#fff0f0">"i"</span>,
<span style="color:#d20;background-color:#fff0f0">"n"</span>,
<span style="color:#d20;background-color:#fff0f0">"g"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"i"</span>,
<span style="color:#d20;background-color:#fff0f0">"n"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"t"</span>,
<span style="color:#d20;background-color:#fff0f0">"h"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"u"</span>,
<span style="color:#d20;background-color:#fff0f0">"n"</span>,
<span style="color:#d20;background-color:#fff0f0">"i"</span>,
<span style="color:#d20;background-color:#fff0f0">"v"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">"r"</span>,
<span style="color:#d20;background-color:#fff0f0">"s"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"o"</span>,
<span style="color:#d20;background-color:#fff0f0">"v"</span>,
<span style="color:#d20;background-color:#fff0f0">"a"</span>,
<span style="color:#d20;background-color:#fff0f0">"l"</span>,
<span style="color:#d20;background-color:#fff0f0">"s"</span>,
<span style="color:#d20;background-color:#fff0f0">"h"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"t"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>,
<span style="color:#d20;background-color:#fff0f0">"d"</span>,
<span style="color:#d20;background-color:#fff0f0">"i"</span>,
<span style="color:#d20;background-color:#fff0f0">"n"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"a"</span>,
<span style="color:#d20;background-color:#fff0f0">"w"</span>,
<span style="color:#d20;background-color:#fff0f0">"i"</span>,
<span style="color:#d20;background-color:#fff0f0">"t"</span>,
<span style="color:#d20;background-color:#fff0f0">" "</span>,
<span style="color:#d20;background-color:#fff0f0">"j"</span>,
<span style="color:#d20;background-color:#fff0f0">"g"</span>,
<span style="color:#d20;background-color:#fff0f0">"m"</span>,
<span style="color:#d20;background-color:#fff0f0">"f"</span>,
<span style="color:#d20;background-color:#fff0f0">"t"</span>,
<span style="color:#d20;background-color:#fff0f0">"a"</span>,
<span style="color:#d20;background-color:#fff0f0">"r"</span>,
<span style="color:#d20;background-color:#fff0f0">"y"</span>,
<span style="color:#d20;background-color:#fff0f0">"s"</span>,
<span style="color:#d20;background-color:#fff0f0">"e"</span>
]
],
<span style="color:#b06;font-weight:bold">"logits"</span>: [
[
<span style="color:#a61717;background-color:#e3d2d2"><logits-here></span>
]
]
}
}
</code></pre></div>
Currency exchange rates with exchangeratesapi.iohttps://www.endpointdev.com/blog/2018/07/currency-rates-exchangeratesapi-io/2018-07-14T00:00:00+00:00Jon Jensen
<p><img src="/blog/2018/07/currency-rates-exchangeratesapi-io/38037392525_3501fee918_o-sm.jpg" alt="city street with currency exchange signs" /></p>
<p>Several of our clients run ecommerce sites, built on custom software, that allow customers to pay in their choice of a few different currencies.</p>
<p>Some have set up separate merchant bank accounts for each currency and use separate payment gateway accounts to accept payments natively in each currency. But more commonly they use a payment gateway that allows them to accept payment in several currencies, but receive funds converted into their specified native currency in their merchant bank account.</p>
<p>In either case, it is common to store prices for products, shipping, etc. in one base currency (say, USD for companies based in the U.S.) and dynamically convert prices for customers. Non-native prices may need to be marked up to cover the cost of conversion into the native currency, depending on the terms of the agreement with the payment gateway or bank.</p>
<p>Because currency exchange rates change often, and because payment gateways generally do not offer a way to retrieve the exchange rates in advance, we need our own source for ongoing retrieval and storage of our exchange rates (also known as forex rates).</p>
<p>For a while we were very pleased with <a href="https://fixer.io/">Fixer.io</a>, which was a free service that collected exchange rates from the European Central Bank (ECB) and provided current or historical rates via a simple JSON API. We were sad to find that in March 2018 they deprecated that API and in June 2018 they discontinued it entirely, as <a href="https://github.com/fixerAPI/fixer#readme">described to their users</a>. Fixer.io has transitioned to a paid service and they appear to have improved their operation to retrieve exchange rate data from at least a dozen more sources than the ECB, and to store far more frequent rate updates. Those are nice features, but not something our clients need.</p>
<p>Fixer.io still offers a free plan that allows 1000 API calls per month but in that plan the exchange rates are all based on the Euro. You need to sign up for at least the $10/month plan to retrieve exchange rates from the other currencies as base. As our customers needed exchange rates based on USD and CAD, the free plan wouldn’t work for us. They offer a limited-time Legacy Plan which would work, but that would just be “kicking the can down the road” until that plan too is discontinued. Their lowest paid plan is inexpensive, but with one of our clients retrieving exchange rates just once per week for only 4 currencies, it felt like overkill to track another paid service and API key, deal with renewals and updated credit card payment details, etc.</p>
<p>We quickly found another service called <a href="https://exchangeratesapi.io/">Exchange Rates API</a>, run by <a href="https://github.com/madisvain">Madis Väin</a>, which is free to use without any account setup or API key needed. It is also based on the published ECB exchange rates, and uses the same API that Fixer.io did. The <a href="https://github.com/madisvain/exchangeratesapi">server side is open source</a> written in Python using the <a href="https://github.com/channelcat/sanic">Sanic</a> web server library (similar to Flask, but focused on speed and async request handlers).</p>
<p>The ExchangeRatesAPI.io service runs on <a href="https://www.heroku.com/">Heroku</a> and the open source code is easy to deploy to your own Heroku instance if you want to run your own private service.</p>
<p>Thanks to Madis Väin for providing it, and to Fixer.io for the good service in the past!</p>
Using GitHub for Blog Commentshttps://www.endpointdev.com/blog/2017/11/using-github-for-blog-comments/2017-11-14T00:00:00+00:00Phineas Jensen
<p>Last Saturday, November 11, we rolled out a new website that we’ve been working on for a few months. Part of this update was moving from Blogger as our blogging platform to static HTML generated by <a href="https://middlemanapp.com">Middleman</a>. We were more than happy to move away from Blogger for a variety of reasons, including its lack of HTTPS support for custom domains and how difficult it was to keep its templating and styling up to date with our main website. We were also able to move from blog.endpoint.com to www.endpoint.com/blog.</p>
<p>The most obvious thing that is missing from Middleman’s blog extension is the lack of a commenting system. After exploring some options for comments, we settled on using GitHub issues and comments, inspired by <a href="http://donw.io/post/github-comments/">Don Williamson’s post</a> about doing the same thing. It’s a bit of an unconventional approach, so this post will explain how to use our commenting system and how we implemented it.</p>
<p>Commenting requires a GitHub account, which is easy to sign up for and free, and the general target audience of our blog will often already have a GitHub account. At the bottom of each post will be a link to a GitHub issue at the top of the list of comments, if there are any. Click on the issue, write and post your comment, and it’ll appear on the blog post when you reload the page. It’s pretty simple.</p>
<p>Now, for the gritty details. Don’s post shows the JavaScript he uses to find the correct GitHub issue and fetch its comments, but it required a bit of modification to work in our Middleman project. First, we needed a way to get the right GitHub issue number. Don’s example has that worked into the JavaScript with Hugo, fetching it from some parameter value, but I opted to create a hidden input on the page that would provide the right number:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">input</span> <span style="color:#369">type</span>=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> <span style="color:#369">name</span>=<span style="color:#d20;background-color:#fff0f0">"gh_issue"</span> <span style="color:#369">value</span>=<span style="color:#d20;background-color:#fff0f0">"<%= current_page.data.gh_issue_number =>"</span>>
</code></pre></div><p>Or in HAML:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">%input{type: "hidden", name: "gh_issue", value: current_page.data.gh_issue_number}
</code></pre></div><p>And we can fetch that value with jQuery:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">var</span> issue_id = $(<span style="color:#d20;background-color:#fff0f0">'[name="gh_issue"]'</span>).val();
</code></pre></div><p>From there, we just had to modify the IDs referenced in Don’s JavaScript to match ours, and comments were working perfectly! Well, new comments were.</p>
<p>Our blog is over 9 years old now, with over 1,300 posts and comments on many of those, and we needed to pull those existing comments over into GitHub as well. Actually copying the data wasn’t too difficult. I wrote a simple Python script to use the Blogger API to fetch posts and their comments, sort them by date, create an appropriately-named GitHub issue, and add the comments to them. Aside from GitHub’s anti-abuse detection system getting in the way a few times, it was an easy process.</p>
<p>We wanted to preserve original author and timestamp information, so I had my script prepend the comment body with that information in a code block so it could be easily read by people reading comments on GitHub and parsed by JavaScript:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">original author: ...
date: ...
</code></pre></div><p>To parse that out, I wrote a quick and dirty regex that runs if I’m the user who created the comment:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">if</span> (data.user.login == <span style="color:#d20;background-color:#fff0f0">"phinjensen"</span>) {
<span style="color:#080;font-weight:bold">var</span> regex = <span style="color:#080;background-color:#fff0ff">/^```\r?\noriginal author: ([^\r\n]+)\r?\ndate: ([^\r\n]+)\r?\n```/</span>;
<span style="color:#080;font-weight:bold">var</span> info = data.body.match(regex);
<span style="color:#080;font-weight:bold">if</span> (info) {
author_name = info[<span style="color:#00d;font-weight:bold">1</span>];
date = <span style="color:#080;font-weight:bold">new</span> <span style="color:#038">Date</span>(info[<span style="color:#00d;font-weight:bold">2</span>]);
data.body_html = data.body_html.replace(<span style="color:#080;background-color:#fff0ff">/^<pre><code>.+\r?\n.+\r?\n<\/code><\/pre>/</span>, <span style="color:#d20;background-color:#fff0f0">''</span>);
}
}
</code></pre></div><p>If that regex isn’t matched, then the author and date data is left as-is and parsed as normal. Here’s the full JavaScript we’re using:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#080;font-weight:bold">function</span> formatDate(date) {
<span style="color:#080;font-weight:bold">var</span> monthNames = [
<span style="color:#d20;background-color:#fff0f0">"January"</span>, <span style="color:#d20;background-color:#fff0f0">"February"</span>, <span style="color:#d20;background-color:#fff0f0">"March"</span>,
<span style="color:#d20;background-color:#fff0f0">"April"</span>, <span style="color:#d20;background-color:#fff0f0">"May"</span>, <span style="color:#d20;background-color:#fff0f0">"June"</span>, <span style="color:#d20;background-color:#fff0f0">"July"</span>,
<span style="color:#d20;background-color:#fff0f0">"August"</span>, <span style="color:#d20;background-color:#fff0f0">"September"</span>, <span style="color:#d20;background-color:#fff0f0">"October"</span>,
<span style="color:#d20;background-color:#fff0f0">"November"</span>, <span style="color:#d20;background-color:#fff0f0">"December"</span>
];
<span style="color:#080;font-weight:bold">var</span> day = date.getDate();
<span style="color:#080;font-weight:bold">var</span> monthIndex = date.getMonth();
<span style="color:#080;font-weight:bold">var</span> year = date.getFullYear();
<span style="color:#080;font-weight:bold">return</span> monthNames[monthIndex] + <span style="color:#d20;background-color:#fff0f0">' '</span> + day + <span style="color:#d20;background-color:#fff0f0">', '</span> + year;
}
$(<span style="color:#038">document</span>).ready(<span style="color:#080;font-weight:bold">function</span> () {
<span style="color:#080;font-weight:bold">var</span> issue_id = $(<span style="color:#d20;background-color:#fff0f0">'[name="gh_issue"]'</span>).val();
<span style="color:#080;font-weight:bold">var</span> url = <span style="color:#d20;background-color:#fff0f0">"https://github.com/EndPointCorp/end-point-blog/issues/"</span> + issue_id
<span style="color:#080;font-weight:bold">var</span> api_url = <span style="color:#d20;background-color:#fff0f0">"https://api.github.com/repos/EndPointCorp/end-point-blog/issues/"</span> + issue_id + <span style="color:#d20;background-color:#fff0f0">"/comments"</span>
$.ajax(api_url, {
headers: { Accept: <span style="color:#d20;background-color:#fff0f0">"application/vnd.github.v3.full+json"</span> },
dataType: <span style="color:#d20;background-color:#fff0f0">"json"</span>,
success: <span style="color:#080;font-weight:bold">function</span>(comments) {
$(<span style="color:#d20;background-color:#fff0f0">".comments"</span>).append(<span style="color:#d20;background-color:#fff0f0">"Visit the <b><a href='"</span> + url + <span style="color:#d20;background-color:#fff0f0">"'>GitHub Issue</a></b> to comment on this post."</span>);
$.each(comments, <span style="color:#080;font-weight:bold">function</span>(i, data) {
<span style="color:#080;font-weight:bold">var</span> date, author_name;
<span style="color:#080;font-weight:bold">if</span> (data.user.login == <span style="color:#d20;background-color:#fff0f0">"phinjensen"</span>) {
<span style="color:#080;font-weight:bold">var</span> regex = <span style="color:#080;background-color:#fff0ff">/^```\r?\noriginal author: ([^\r\n]+)\r?\ndate: ([^\r\n]+)\r?\n```/</span>;
<span style="color:#080;font-weight:bold">var</span> info = data.body.match(regex);
<span style="color:#080;font-weight:bold">if</span> (info) {
author_name = info[<span style="color:#00d;font-weight:bold">1</span>];
date = <span style="color:#080;font-weight:bold">new</span> <span style="color:#038">Date</span>(info[<span style="color:#00d;font-weight:bold">2</span>]);
data.body_html = data.body_html.replace(<span style="color:#080;background-color:#fff0ff">/^<pre><code>.+\r?\n.+\r?\n<\/code><\/pre>/</span>, <span style="color:#d20;background-color:#fff0f0">''</span>);
}
}
<span style="color:#080;font-weight:bold">if</span> (!date) {
date = <span style="color:#080;font-weight:bold">new</span> <span style="color:#038">Date</span>(data.created_at);
}
<span style="color:#080;font-weight:bold">var</span> $comment = $(<span style="color:#d20;background-color:#fff0f0">"<div/>"</span>, {<span style="color:#d20;background-color:#fff0f0">'class'</span>: <span style="color:#d20;background-color:#fff0f0">'comment'</span>});
<span style="color:#080;font-weight:bold">if</span> (!author_name) {
$comment.append($(<span style="color:#d20;background-color:#fff0f0">"<img/>"</span>, {
<span style="color:#d20;background-color:#fff0f0">'src'</span>: data.user.avatar_url,
<span style="color:#d20;background-color:#fff0f0">'class'</span>: <span style="color:#d20;background-color:#fff0f0">'avatar-image'</span>
}));
}
<span style="color:#080;font-weight:bold">var</span> $body = $(<span style="color:#d20;background-color:#fff0f0">"<div/>"</span>, {<span style="color:#d20;background-color:#fff0f0">'class'</span>: <span style="color:#d20;background-color:#fff0f0">'comment-body'</span>});
<span style="color:#080;font-weight:bold">var</span> $header = $(<span style="color:#d20;background-color:#fff0f0">"<span/>"</span>, {<span style="color:#d20;background-color:#fff0f0">'class'</span>: <span style="color:#d20;background-color:#fff0f0">'comment-header'</span>});
<span style="color:#080;font-weight:bold">if</span> (author_name) {
$header.append(author_name);
} <span style="color:#080;font-weight:bold">else</span> {
$header.append($(<span style="color:#d20;background-color:#fff0f0">"<a/>"</span>, {<span style="color:#d20;background-color:#fff0f0">'href'</span>: data.user.html_url, <span style="color:#d20;background-color:#fff0f0">'text'</span>: data.user.login}));
}
$header.append(<span style="color:#d20;background-color:#fff0f0">" commented on "</span> + formatDate(date));
$body.append($header);
$body.append($(<span style="color:#d20;background-color:#fff0f0">"<div>"</span> + data.body_html + <span style="color:#d20;background-color:#fff0f0">"</div>"</span>, {<span style="color:#d20;background-color:#fff0f0">'class'</span>: <span style="color:#d20;background-color:#fff0f0">'comment-body'</span>,}));
$comment.append($body);
$(<span style="color:#d20;background-color:#fff0f0">".comments"</span>).append($comment);
});
},
error: <span style="color:#080;font-weight:bold">function</span>() {
$(<span style="color:#d20;background-color:#fff0f0">".comments"</span>).append(<span style="color:#d20;background-color:#fff0f0">"Comments are not open for this post yet."</span>);
}
});
});
</code></pre></div><p>Huge thanks to <a href="http://donw.io/">Don Williamson</a> for sharing his implementation of this idea! If you have any feedback for us and how we’re doing it, let us know in the comments!</p>
Hue’s on First: How we used responsive bulbs to join software and hardware for a busy medical practicehttps://www.endpointdev.com/blog/2016/03/hues-on-first/2016-03-14T00:00:00+00:00Liz Flyntz
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="/blog/2016/03/hues-on-first/FastTrackoffice.jpg"/></div>
<p>In 2014 we began working with a busy bariatric surgery office in Long Island to create a system that would allow the practice to better manage doctor paging and patient wait time. By placing a responsive, color-coded light bulb and tablet outside each examination room, the staff could see which rooms were empty, which were occupied by a patient waiting on a specific doctor, and in which a doctor-patient consultation was in process. Outside each room is a tablet with information including the patient number, the attending doctor’s name, and the wait time.</p>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="/blog/2016/03/hues-on-first/FastTrackmapmonitor.jpg"/></div>
<p>In addition to providing a comprehensive, granular paging service for doctors, Fast Track also provides feedback to the practice. This feedback includes average patient wait times per doctor, per time of day, and per procedure. This allows the practice to make necessary changes and increase patient satisfaction and peace of mind.</p>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="/blog/2016/03/hues-on-first/FastTrackapp.jpg"/></div>
<p>I asked Danny Divita, one of the main developers on this project, to tell us more about the <a href="http://www2.meethue.com/en-us/">Hue</a>/ <a href="https://web.archive.org/web/20180107115334/http://www.fasttrackmed.com/">FastTrack</a> interface.</p>
<p><em>LF: Describe the project for which we used Hue bulbs. What were all the pieces that needed fitting together?</em></p>
<p>DD: The Hue bulbs are being used for a bariatric clinic to alert the staff when pages are sent to of pages to specific rooms. Alongside the lights the API also has to work with a RF device-locating API. The RF API had to be coordinated with the Hue light API to coordinate changing colors, alerts, and states.</p>
<p><em>LF: Why were Hue bulbs a good solution for the problem of creating an easy-to-understand, integratable visual alert system? Did you consider any alternatives?</em></p>
<p>DD: From a cost perspective and ease of integration the bulbs were a good solution for this project. There are other solutions, but they are more costly and the integration would have taken longer.</p>
<p><em>LF: How did you hear about Hue bulbs? Was the API easy to access and work with? How was using Hue’s testing environment?</em></p>
<p>DD: The client suggested the bulbs during their research before having us develop their application. The API is extremely easy to work with and well documented. The Hue API has a built-in testing feature that made it easy to understand the API and develop around it.</p>
<p><em>LF: What was the most difficult thing about integrating this technology? Can you think of any other commercial implementations of these bulbs? Specifically in the medical industry?</em></p>
<p>DD: The most difficult aspect was trying to come up with a good alert (blinking) feature. The API has some built in alerts, but they are limited to a set timed interval. We had to develop around this to ensure the alerts stayed constant and did not time out. Another application for these lights in the medical field could be to indicate severity of a situation or condition. Because the lights can change color, the applications can signal the proper response for the staff.</p>
<p><em>LF: Did you write any new software for this project? Is that work available to other developers?</em></p>
<p>DD: We used a third party library that helps to manage stateless workflows. Our application expanded upon this library to design a workflow engine that manages paging the staff, Hue lights, and RF device locating.</p>
Google Maps JavaScript API LatLng Property Name Changeshttps://www.endpointdev.com/blog/2015/06/google-maps-javascript-api-latlng/2015-06-11T00:00:00+00:00Greg Davidson
<h3 id="debugging-broken-maps">Debugging Broken Maps</h3>
<p>A few weeks ago I had to troubleshoot some Google Maps related code that had suddenly stopped working. Some debugging revealed the issue: the code adding markers to the page was attempting to access properties that did not exist. This seemed odd because the latitude and longitude values were the result of a geocoding request which was completing successfully. The other thing which stood out to me were the property names themselves:</p>
<pre tabindex="0"><code>var myLoc = new google.maps.LatLng(results[0].geometry.location.k, results[0].geometry.location.D);
</code></pre><p>It looked like the original author had inspected the geocoded response, found the ‘k’ and ‘D’ properties which held latitude and longitude values and used them in their maps code. This had all been working fine until Google released a <a href="https://groups.google.com/forum/#!topic/google-maps-js-api-v3-notify/tYp4JKtkDg0">new version</a> of their JavaScript API. Sites that did not <a href="https://developers.google.com/maps/documentation/javascript/basics#Versioning">specify a particular version</a> of the API were upgraded to the new version automatically. If you have Google Maps code which stopped working recently this might be the reason why.</p>
<h3 id="the-solution-use-the-built-in-methods-in-the-latlng-class">The Solution: Use the built-in methods in the LatLng class</h3>
<img alt="Screen Shot 2015 06 10 at 3 47 32 PM" border="0" height="308" src="/blog/2015/06/google-maps-javascript-api-latlng/image-0.png" title="Screen Shot 2015-06-10 at 3.47.32 PM.png" width="419"/>
<p>I recalled there being some helper methods for LatLng objects and confirmed this with a visit to the <a href="https://developers.google.com/maps/documentation/javascript/3.exp/reference#LatLng">docs for the LatLng class</a> which had recently been updated and given the <a href="https://material.io/guidelines/material-design/introduction.html">Material design</a> treatment—thanks Google! The lat() and lng() methods were what I needed and updating the code with them fixed the issue. The fixed code was similar to this:</p>
<pre tabindex="0"><code>var myLoc = new google.maps.LatLng(results[0].geometry.location.lat(), results[0].geometry.location.lng());
</code></pre><h3 id="digging-deeper">Digging Deeper</h3>
<p>I was curious about this so I mapped out the differences between the three latest versions of the API:</p>
<div class="table-scroll">
<table><thead>
<tr> <th>API Version</th> <th>Latitude Property</th> <th>Longitude Property</th> <th>Constructor Name</th> </tr>
</thead> <tbody>
<tr> <td>3.21.x (experimental)</td> <td>A</td> <td>F</td> <td>rf</td> </tr>
<tr> <td>3.20.x (release)</td> <td>A</td> <td>F</td> <td>pf</td> </tr>
<tr> <td>3.19.x (frozen)</td> <td>k</td> <td>D</td> <td>pf</td> </tr>
</tbody> </table>
</div>
<p>It seems to me that the property name changes are a result of running the Google Maps API code through the <a href="https://developers.google.com/closure/compiler/">Closure Compiler</a>. Make sure to use the built-in lat() and lng() methods as these property names are very likely to change again in future!</p>
Integrate Twilio in Djangohttps://www.endpointdev.com/blog/2014/11/integrate-twilio-in-django/2014-11-24T00:00:00+00:00Kulbir Singh
<h3 id="twilio">Twilio</h3>
<p><a href="https://www.twilio.com/">Twilio</a> is a powerful HTTP API that allows you to build powerful voice and SMS apps. The goal of this blog post is to help make building the SMS applications as simple as possible in django.</p>
<p>There is a already Twilio Python help library available. The open source <a href="https://github.com/twilio/twilio-python">twilio-python</a> library lets us to write Python code to make HTTP requests to the Twilio API.</p>
<h3 id="installation">Installation</h3>
<p>The easiest way to install twilio-python library is using <a href="http://pip.readthedocs.org/en/latest/quickstart.html">pip</a>. Pip is a package manager for Python.</p>
<p>Simply run following command in terminal.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ pip install twilio
</code></pre></div><h3 id="twilio-api-credentails">Twilio API Credentails</h3>
<p>To Integrate twilio API in django application, we need <strong>TWILIO_ACCOUNT_SID</strong> and <strong>TWILIO_AUTH_TOKEN</strong> variables. These variables can be found by logging into your Twilio account dashboard. These variables are used to communicate with the Twilio API.</p>
<p>You’ll need to add them to your <strong>settings.py</strong> file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">TWILIO_ACCOUNT_SID = <span style="color:#d20;background-color:#fff0f0">'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'</span>
TWILIO_AUTH_TOKEN = <span style="color:#d20;background-color:#fff0f0">'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'</span>
</code></pre></div><h3 id="create-a-new-app">Create a New App</h3>
<p>We are going to interact with people using SMS, so I prefer to create an app named <strong>communication</strong>. I am assuming you’ve already installed Django.</p>
<p>Run following command in terminal.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ django-admin.py startapp communcation
</code></pre></div><p>We will need to register the new app in our django project.</p>
<p>Add it to your INSTALLED_APPS tuple in your settings.py file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">INSTALLED_APPS = (
<span style="color:#d20;background-color:#fff0f0">'django.contrib.auth'</span>,
<span style="color:#d20;background-color:#fff0f0">'django.contrib.contenttypes'</span>,
<span style="color:#d20;background-color:#fff0f0">'django.contrib.sessions'</span>,
<span style="color:#d20;background-color:#fff0f0">'django.contrib.sites'</span>,
<span style="color:#d20;background-color:#fff0f0">'django.contrib.messages'</span>,
<span style="color:#d20;background-color:#fff0f0">'django.contrib.staticfiles'</span>,
<span style="color:#a61717;background-color:#e3d2d2">‘</span>communication<span style="color:#a61717;background-color:#e3d2d2">’</span>,
...
)
</code></pre></div><h3 id="create-the-model">Create the Model</h3>
<p>Now we’ll open up <strong>communication/models.py</strong> to start creating models for our app.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">SendSMS</span>(models.Model):
to_number = models.CharField(max_length=<span style="color:#00d;font-weight:bold">30</span>)
from_number = models.CharField(max_length=<span style="color:#00d;font-weight:bold">30</span>)
sms_sid = models.CharField(max_length=<span style="color:#00d;font-weight:bold">34</span>, default=<span style="color:#d20;background-color:#fff0f0">""</span>, blank=<span style="color:#080;font-weight:bold">True</span>)
account_sid = models.CharField(max_length=<span style="color:#00d;font-weight:bold">34</span>, default=<span style="color:#d20;background-color:#fff0f0">""</span>, blank=<span style="color:#080;font-weight:bold">True</span>)
created_at = models.DateTimeField(auto_now_add=<span style="color:#080;font-weight:bold">True</span>)
sent_at = models.DateTimeField(null=<span style="color:#080;font-weight:bold">True</span>, blank=<span style="color:#080;font-weight:bold">True</span>)
delivered_at = models.DateTimeField(null=<span style="color:#080;font-weight:bold">True</span>, blank=<span style="color:#080;font-weight:bold">True</span>)
status = models.CharField(max_length=<span style="color:#00d;font-weight:bold">20</span>, default=<span style="color:#d20;background-color:#fff0f0">""</span>, blank=<span style="color:#080;font-weight:bold">True</span>)
</code></pre></div><p>and run the <strong>syncdb</strong> command after defining the model:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">$ python manage.py syncdb
</code></pre></div><p>It will create the necessary database tables for our app.</p>
<h3 id="create-utilspy-file">Create utils.py file</h3>
<p>Create a new file named utils.py and save in <strong>communication/utils.py</strong>.</p>
<p>Put the following code in <strong>communication/utils.py</strong>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">django.conf</span> <span style="color:#080;font-weight:bold">import</span> settings
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">twilio</span>
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">twilio.rest</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">send_twilio_message</span>(to_number, body):
client = twilio.rest.TwilioRestClient(
settings.TWILIO_ACCOUNT_SID, settings.TWILIO_AUTH_TOKEN)
<span style="color:#080;font-weight:bold">return</span> client.messages.create(
body=body,
to=to_number,
from_=settings.TWILIO_PHONE_NUMBER
)
</code></pre></div><h3 id="testing-send_twilio_message">Testing send_twilio_message</h3>
<p>Open the <strong>shell</strong> and run following commands.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">>>> <span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">communication.utils</span> <span style="color:#080;font-weight:bold">import</span> send_twilio_message
>>> sms = send_twilio_message(<span style="color:#d20;background-color:#fff0f0">'+15005550006'</span>, <span style="color:#d20;background-color:#fff0f0">'Hello Endpointer,'</span>)
>>> <span style="color:#038">print</span> sms.sid
SM97f8ac9321114af1b7fd4463ff8bd038
</code></pre></div><p>Having the sid means that everything in the backend is working fine. And we can proceed to work on the front end.</p>
<h3 id="create-form">Create Form</h3>
<p>Lets create a form to gather the data. Now open/create up <strong>communication/forms.py</strong> to start creating forms for our app. And paste the following code into it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">SendSMSForm</span>(forms.ModelForm):
<span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Meta</span>:
model = SendSMS
fields = (<span style="color:#d20;background-color:#fff0f0">'to_number'</span>, <span style="color:#d20;background-color:#fff0f0">'body'</span>)
</code></pre></div><h3 id="the-view-createview">The View CreateView</h3>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">SendSmsCreateView</span>(CreateView):
model = SendSMS
form_class = SendSMSForm
template_name = <span style="color:#d20;background-color:#fff0f0">'communication/sendsms_form.html'</span>
success_url = reverse_lazy(<span style="color:#d20;background-color:#fff0f0">'send_sms'</span>)
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">form_valid</span>(self, form):
number = self.cleaned_data[<span style="color:#d20;background-color:#fff0f0">'to_number'</span>]
body = self.cleaned_data[<span style="color:#d20;background-color:#fff0f0">'body'</span>]
<span style="color:#888"># call twilio</span>
sent = send_twilio_message(number, body)
<span style="color:#888"># save form</span>
send_sms = form.save(commit=<span style="color:#080;font-weight:bold">False</span>)
send_sms.from_number = settings.TWILIO_PHONE_NUMBER
send_sms.sms_sid = sent.sid
send_sms.account_sid = sent.account_sid
send_sms.status = sent.status
send_sms.sent_at = now()
<span style="color:#080;font-weight:bold">if</span> sent.price:
send_sms.price = Decimal(force_text(sent.price))
send_sms.price_unit = sent.price_unit
send_sms.save()
<span style="color:#080;font-weight:bold">return</span> <span style="color:#038">super</span>(SendSmsCreateView, self).form_valid(form)
</code></pre></div><h3 id="defining-urls">Defining URLS</h3>
<p>The URL configuration tells Django how to match a request’s path to your Python code. Django looks for the URL configuration, defined as urlpatterns, in the <strong>urls.py</strong> file in your project:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">django.conf.urls</span> <span style="color:#080;font-weight:bold">import</span> patterns, url
<span style="color:#080;font-weight:bold">from</span> <span style="color:#b06;font-weight:bold">.views</span> <span style="color:#080;font-weight:bold">import</span> SendSmsCreateView
urlpatterns = patterns(<span style="color:#d20;background-color:#fff0f0">''</span>,
url(
regex=<span style="color:#d20;background-color:#fff0f0">r</span><span style="color:#d20;background-color:#fff0f0">'^communication/send/sms/$'</span>,
view=SendSmsCreateView.as_view(),
name=<span style="color:#d20;background-color:#fff0f0">'send_sms'</span>
),
)
</code></pre></div><h3 id="creating-the-template">Creating the Template</h3>
<p>Now that we’ve defined a URL for our list view, we can try it out. Django includes a server suitable for development purposes that you can use to easily test your project:</p>
<p>If you visit the <strong>http://127.0.0.1:8000/communication/send/sms/</strong> in your browser, though, you’ll see an error: <strong>TemplateDoesNotExist</strong>.</p>
<p>This is because we have not defined the template file yet. So now create <strong>sendsms_form.html</strong> file in <strong>templates/communication/</strong> and put the following code in it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">form</span> <span style="color:#369">action</span>=<span style="color:#d20;background-color:#fff0f0">"."</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"form-horizontal"</span> <span style="color:#369">method</span>=<span style="color:#d20;background-color:#fff0f0">"post"</span> <span style="color:#369">role</span>=<span style="color:#d20;background-color:#fff0f0">"form"</span>>{% csrf_token %}
{% for field in form %}
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"form-group"</span>> <<span style="color:#b06;font-weight:bold">label</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"col-sm-2 control-label"</span> <span style="color:#369">for</span>=<span style="color:#d20;background-color:#fff0f0">"inputEmail3"</span>>{{ field.label_tag }}</<span style="color:#b06;font-weight:bold">label</span>>
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"col-sm-10"</span>> {{ field }}
{{ field.errors }}
</<span style="color:#b06;font-weight:bold">div</span>> </<span style="color:#b06;font-weight:bold">div</span>> {% endfor %}
<<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"form-group"</span>> <<span style="color:#b06;font-weight:bold">div</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"col-sm-offset-2 col-sm-10"</span>> <<span style="color:#b06;font-weight:bold">button</span> <span style="color:#369">class</span>=<span style="color:#d20;background-color:#fff0f0">"btn btn-default"</span> <span style="color:#369">type</span>=<span style="color:#d20;background-color:#fff0f0">"submit"</span>>Submit</<span style="color:#b06;font-weight:bold">button</span>>
</<span style="color:#b06;font-weight:bold">div</span>> </<span style="color:#b06;font-weight:bold">div</span>></<span style="color:#b06;font-weight:bold">form</span>>
</code></pre></div><p>Now reload the <strong>http://127.0.0.1:8000/communication/send/sms/</strong> in your browser. Assuming everything is okay, you should then see the following form:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/11/integrate-twilio-in-django/image-0-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/11/integrate-twilio-in-django/image-0.png"/></a></div>
<p>Fill out the form, and hit the submit button to send your SMS.</p>
<h3 id="conclusion">CONCLUSION</h3>
<p>Congratulations, your SMS is successfully sent. Good luck!</p>
Facebook, Twitter, Google+ sharing with the URLhttps://www.endpointdev.com/blog/2014/07/facebook-twitter-google-plus-sharing/2014-07-10T00:00:00+00:00Marina Lohova
<p>This blog post is intended for the folks who spent way more time displaying social sharing buttons on their websites than originally planned. My buttons were supposed to bring up the Share Dialog for Facebook, Twitter and Google+ platforms. I had the requirement to display a custom logo and a custom description. It seemed easy… until it turned out to be rather difficult.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="/blog/2014/07/facebook-twitter-google-plus-sharing/image-0-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="/blog/2014/07/facebook-twitter-google-plus-sharing/image-0.png"/></a></div>
<p>The appropriate format for Twitter is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><a href=<span style="color:#d20;background-color:#fff0f0">"http://twitter.com/share?text=<%= desc %>&url=<%= url %>"</span>><span style="color:#036;font-weight:bold">Twitter</span><<span style="color:#080;background-color:#fff0ff">/a>
</span></code></pre></div><p>Note that Twitter dialog does not include the picture, only the description.</p>
<p>Trouble started with Facebook when I learned that custom parameters in Facebook’s sharer.php url are officially not supported anymore: <a href="https://developers.facebook.com/bugs/357750474364812">https://developers.facebook.com/bugs/357750474364812</a>. I tried the format widely suggested on forums, and while custom image was successfully displayed, neither title, not description showed.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><a href=<span style="color:#d20;background-color:#fff0f0">"http://www.facebook.com/sharer.php?u=<%= url %>&p[images][0]=<%= img %>&description=<%= desc %>"</span>><span style="color:#036;font-weight:bold">Facebook</span><<span style="color:#080;background-color:#fff0ff">/a>
</span></code></pre></div><p>Facebook documentation hinted that addition of the OpenGraph Protocol standard tags to the of the page may help:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-html" data-lang="html"><<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">content</span>=<span style="color:#d20;background-color:#fff0f0">"http://samples.ogp.me/136756249803614"</span> <span style="color:#369">property</span>=<span style="color:#d20;background-color:#fff0f0">"og:url"</span>/>
<<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">content</span>=<span style="color:#d20;background-color:#fff0f0">"Chocolate Pecan Pie"</span> <span style="color:#369">property</span>=<span style="color:#d20;background-color:#fff0f0">"og:title"</span>/>
<<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">content</span>=<span style="color:#d20;background-color:#fff0f0">"This pie is delicious!"</span> <span style="color:#369">property</span>=<span style="color:#d20;background-color:#fff0f0">"og:description"</span>/>
<<span style="color:#b06;font-weight:bold">meta</span> <span style="color:#369">content</span>=<span style="color:#d20;background-color:#fff0f0">"https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-prn1/851565_496755187057665_544240989_n.jpg"</span> <span style="color:#369">property</span>=<span style="color:#d20;background-color:#fff0f0">"og:image"</span>/>
</code></pre></div><p>I wasn’t able to get it up and working with sharer.php. After spending considerable amount of time on this I had to give up and acknowledge that the only way to fully customize the dialog would require registering an app and utilizing APP_ID <a href="https://developers.facebook.com/docs/sharing/reference/share-dialog">https://developers.facebook.com/docs/sharing/reference/share-dialog</a>.</p>
<p>I anticipated the same kind of trouble with the last button for Google+. And I wasn’t mistaken. The only allowed format for G+ is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><a href=<span style="color:#d20;background-color:#fff0f0">"https://plus.google.com/share?url=<%= url %>"</span>><span style="color:#036;font-weight:bold">Google</span>+<<span style="color:#080;background-color:#fff0ff">/a>
</span></code></pre></div><p>Despite a few mentions on the web this does not work anymore:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><meta content=<span style="color:#d20;background-color:#fff0f0">"<%= desc %>"</span> itemprop=<span style="color:#d20;background-color:#fff0f0">"description"</span>/>
</code></pre></div><p>Same here. Doesn’t work:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><meta content=<span style="color:#d20;background-color:#fff0f0">"<%= title %>"</span> property=<span style="color:#d20;background-color:#fff0f0">"og:title"</span>/>
<meta content=<span style="color:#d20;background-color:#fff0f0">"article"</span> property=<span style="color:#d20;background-color:#fff0f0">"og:type"</span>/>
<meta content=<span style="color:#d20;background-color:#fff0f0">"<%= url %>"</span> property=<span style="color:#d20;background-color:#fff0f0">"og:url"</span>/>
<meta content=<span style="color:#d20;background-color:#fff0f0">"<%= img %>"</span> property=<span style="color:#d20;background-color:#fff0f0">"og:image"</span>/>
<meta content=<span style="color:#d20;background-color:#fff0f0">"<%= desc %>"</span> property=<span style="color:#d20;background-color:#fff0f0">"og:description"</span>/>
</code></pre></div><p>Eventually it turned out that it wasn’t possible to use parameters for G+ link unless you sign up for the API key and use one of the API methods. I wasn’t planning to obtain a key at that time, so I had to simply drop the custom logo and text for G+.</p>
<p>Looks like both Facebook and Google+ took steps to restrict the free usage of their share urls so more people would register their apps with them.</p>
Spree Active Shipping Gem “We are unable to calculate shipping rates for the selected items.” Errorhttps://www.endpointdev.com/blog/2014/02/spree-active-shipping-gem-we-are-unable/2014-02-12T00:00:00+00:00Matt Galvin
<p>I was recently working on a <a href="http://spreecommerce.org/">Spree</a> site and setting up the <a href="https://github.com/spree/spree_active_shipping">Spree Active Shipping Gem</a> along with the UPS API. For those not familiar, the Spree Active Shipping Gem interacts with various shipping APIs like UPS, USPS, and FedEx. Due to the nature of Spree—where it does so much for you, and the interaction between the Active Shipping Gem and a shipping API also being “auto-magic”, it is often difficult to debug. As I was recently undertaking the task of setting this up I found a few “gotchas” that I hope, through this blog post, may be able to save others a lot of time.</p>
<p>I have found that there wasn’t a lot of instruction for setting up the Active Shipping Gem and a shipping carrier API like the UPS Shipping API. Ostensibly, there isn’t much to it—the Active Shipping Gem handles much of the interaction between the shipping API of choice and Spree.</p>
<p>First, you’re going to go the <a href="https://github.com/spree/spree_active_shipping">Spree Active Shipping Gem</a> GitHub repo and follow the instructions for installing the Active Shipping Gem. It is very straightforward, but do proceed in the order mentioned in the Spree Active Shipping Gem documentation as some steps depend on the successful completion of others.</p>
<p>Second, you’re going to go to the shipper of your choice, in this case UPS, and follow their directions for using their <a href="https://web.archive.org/web/20160101071458/http://www.ups.com/content/us/en/bussol/browse/cat/developer_kit.html">API</a>. I do recommend actually reading, or at least skimming, the pages and pages of documentation. Why? Because there are some important agreements explaining how the API is to be used (basically legal requirements for the UPS logo).</p>
<p>The Active Shipping Gem makes a call to the API, the API returns a hash of various shipping methods and prices based on the parameters you’ve sent it (such as shipment origin and destination), and then it automatically displays in the UI as an adjustment. How great is that?!</p>
<p>Well, it would be great if it all worked out exactly as planned. However, if you are running Spree 2-0-stable you may find yourself battling an unusual circumstance. Namely, Spree 2-0-stable will create your core/app/views/spree/checkout/edit.html.erb as</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><<span style="color:#2b2;background-color:#f0fff0">% content_for </span> <span style="color:#a60;background-color:#fff0f0">:head</span> <span style="color:#080;font-weight:bold">do</span> <span style="color:#d20;background-color:#fff0f0">%>
</span><span style="color:#d20;background-color:#fff0f0"> <%= javascript_include_tag '/states' %></span>
<<span style="color:#2b2;background-color:#f0fff0">% end </span> %>
</code></pre></div><p>This will provide the incorrect path. It is intended to hit the StatesController, so update it like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><<span style="color:#2b2;background-color:#f0fff0">% content_for </span> <span style="color:#a60;background-color:#fff0f0">:head</span> <span style="color:#080;font-weight:bold">do</span> <span style="color:#d20;background-color:#fff0f0">%>
</span><span style="color:#d20;background-color:#fff0f0"> <%= javascript_include_tag states_url %></span>
<<span style="color:#2b2;background-color:#f0fff0">% end </span> %>
</code></pre></div><p>Now, once this correction has been made you may find that you are still having an error, “We are unable to calculate shipping rates for the selected items.”</p>
<p><a href="/blog/2014/02/spree-active-shipping-gem-we-are-unable/image-0.png" imageanchor="1"><img border="0" src="/blog/2014/02/spree-active-shipping-gem-we-are-unable/image-0.png"/></a></p>
<p>At this point Chrome Dev Tools will not show any errors. When I had this error, a number of Google searches returned results of the kind “make sure you have set up your shipping zones correctly and added shipping methods to these zones”. I verified this again and again in the console, as did many others who were equally perplexed by this message on <a href="http://stackoverflow.com/questions/18277367/spree-commerce-error-on-checkout-we-are-unable-to-ship-the-selected-items-to-y">StackOverflow</a> as well as in Google Groups, like <a href="https://groups.google.com/forum/#!msg/spree-user/aCJz5iNemfo/3v4uJ8hPBVsJ">here</a> and <a href="https://groups.google.com/forum/#!topic/spree-user/aCJz5iNemfo">here</a>. Some got this error when they added an item to the cart that had a count_on_hand of 0 and backorderable equal to false, like you can see here at <a href="https://github.com/spree/spree/issues/3521">Spree GitHub issues</a>. If a 0 count_on_hand is what is giving you this error, but you want a product to be backorderable, make sure to also check the “Propagate All Variants” in the Spree admin as seen below. This will loop through all of the product’s variants with a count_on_hand of 0, and allow them to be backorderable.</p>
<p><a href="/blog/2014/02/spree-active-shipping-gem-we-are-unable/image-1-big.png" imageanchor="1"><img border="0" src="/blog/2014/02/spree-active-shipping-gem-we-are-unable/image-1.png"/></a></p>
<p>After a long while of searching and wondering, is it the API? Is it the Active Shipping Gem? Is it a blacklisted zip code? I went through and changed one setting at a time in the Spree admin until finally arriving at the source of this error for me. Missing product weight. Because UPS needs the product weight in order to calculate shipping charges, make sure this is set.</p>
<p>The “We are unable to calculate shipping rates for the selected items” error message is misleading. If you encounter this error after correcting the javascript_include_tag, the cause is most likely a setting in the admin. Check for how insufficient inventory is handled, missing product weights, or incorrectly setup up or non-existent shipping zones & associated methods. I hope if this error message is what brought you here that this post has saved you some time.</p>
Verify Addresses the Easy Way with SmartyStreetshttps://www.endpointdev.com/blog/2012/12/verify-addresses-easy-way-with/2012-12-20T00:00:00+00:00Tim Case
<p>Adding an address form is a pretty common activity in web apps and even more so with ecommerce web apps. Validations on forms allow us to guide the user to filling out all required fields and to make sure the fields conform to basic formats. Up until now going further with addresses to verify they actually exist in the real world was a difficult enough task that most developers wouldn’t bother with it. Imagine though the cost to the merchant who ships something to the wrong state because the customer accidently selected “SD” (South Dakota) when they thought they were selecting “SC” (South Carolina), a simple enough mistake to make and one that wouldn’t be caught by most address forms. In today’s ecommerce world customers expect deliveries to be fast and reliable, and in this case the customer would have to wait until the package is returned to the merchant with “Address Unknown” only to have to wait even longer for the reshipment. Even worse for the merchant, maybe the package never gets returned.</p>
<p><a href="http://smartystreets.com/">SmartyStreets</a> is a new API web app that I implemented for our client <a href="http://mobixa.com">Mobixa</a>, a web app that allows people to sell their used mobile phones. <a href="http://mobixa.com">Mobixa</a> sends shipping labels and payment checks to customers so that they can send their phones to <a href="http://mobixa.com">Mobixa</a> and get paid for it. Wrong addresses can delay the whole process and <a href="http://mobixa.com">Mobixa</a> wanted to reduce the number of bad addresses that were being keyed in by the customers. <a href="http://smartystreets.com/">SmartyStreets</a> provides an easy way for developers to allow Address verification to their web forms so that addresses are verified against the US Postal Service’s valid addresses. <a href="http://smartystreets.com/">SmartyStreets</a> is <a href="http://en.wikipedia.org/wiki/Coding_Accuracy_Support_System">CASS</a> certified meaning that they meet the USPS accuracy requirements.</p>
<p>The big advantage of Smarty Streets is that adding address verification to a form can be as easy as adding a link to their jQuery based plugin and then a script tag with your <a href="http://smartystreets.com/">SmartyStreets</a> API key. The plugin autodetects address fields and when a minimum of 3 fields are entered (address, city, state), it will display an indicator sprite to the user and send an async request to the API for verification. The verification has three possible outcomes:</p>
<ol>
<li>
<p>The address is verified. A valid address will display a verified image to the side of the zip code and all of the address fields will be modified with a more “correct” address, with correct being defined as what matches the USPS official definition for the matched address. Zip codes will be modified with the carrier route code, so “92806” becomes “92806-3433”. An address modification would for example change “1735 pheasant” to “1731 N Pheasant St.”. Proper casing and spelling errors will also be enacted.</p>
</li>
<li>
<p>The address is ambiguous. An ambiguous address is one that returns multiple matches. Let’s say for example that “1 Rosedale St.”, could be “1 N Rosedale Street” or “1 S Rosedale Street”. In this case it displays a popup which allows the user to select the correct address or to override the suggestions and continue with the address they entered.</p>
</li>
<li>
<p>The address is invalid. An invalid address informs the user that it’s invalid and both the invalid address and the ambiguous address offer the user two additional choices. “Double checking the address” will rerun the address validation after the user has modified the address. “I certify that what I type is correct”, is a second choice which allows the user to continue with the address they typed. This last choice is important because it allows the user the power to continue with what they want instead of forcing them to conform to the address validation.</p>
</li>
</ol>
<p>Checks are performed when <a href="http://smartystreets.com/">SmartyStreets</a> senses that it has enough address information to run a search. During this time the submit button to the form is disabled until the check is completed. Once a check is performed once, it will not perform again unless the user elects to “Double check” the address, this is a good design choice to prevent the user from getting stuck in an infinite loop of sorts.</p>
<p>Our implementation of <a href="http://smartystreets.com/">SmartyStreets</a> into <a href="http://mobixa.com">Mobixa</a> included customizing it to do things a little bit differently than the out of the box defaults. The jQuery plugin comes with many events and hooks for adding customization and if you want to go your own way you can implement everything on the frontend save for the API call yourself. Documentation on the website is useful, and the developers of <a href="http://smartystreets.com/">SmartyStreets</a> conveniently answered my questions via an Olark chat window.</p>
<p>The costs of <a href="http://smartystreets.com/">SmartyStreets</a> is that you have to spend time to implement it in your app, a monthly fee based on number of API calls, and also that your UI flow will change slightly in that the user will need to wait for the API call to complete before submitting the form. I don’t always implement validation when I have an address form in an app, but when I do, I like to use <a href="http://smartystreets.com/">SmartyStreets</a>.</p>
The truth about Google Wallet integrationhttps://www.endpointdev.com/blog/2012/10/the-truth-about-google-wallet/2012-10-19T00:00:00+00:00Marina Lohova
<p>Google Wallet integration is quite a bumpy ride for every developer. I would like to describe one integration pattern that actually works. It is written in PHP for Google Wallet 2.5 API.</p>
<h3 id="google-merchant-account-settings">Google Merchant account settings</h3>
<p>First, one must <a href="https://developers.google.com/checkout/developer/Google_Checkout_Buy_Now_Button_How_To#signup">sign up</a> for Google Merchant account. Once this is done, it is very important to configure the service properly on the Settings > Integration tab</p>
<p><a href="/blog/2012/10/the-truth-about-google-wallet/image-0.png" imageanchor="1"><img border="0" src="/blog/2012/10/the-truth-about-google-wallet/image-0.png"/></a></p>
<h3 id="buy-now-button">Buy now button</h3>
<p><a href="https://developers.google.com/checkout/developer/Google_Checkout_Buy_Now_Button_How_To">Buy Now buttons</a> are the simplest form of integration. The code for the button can be obtained on the Tools > Buy Now Buttons tab.</p>
<img border="0" src="/blog/2012/10/the-truth-about-google-wallet/image-1.jpeg"/>
<p>I modified the code provided by Google to transfer information to Google Wallet server via the hidden fields on the form.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><form method=<span style="color:#d20;background-color:#fff0f0">"POST"</span> action=<span style="color:#d20;background-color:#fff0f0">"https://sandbox.google.com/checkout/api/checkout/v2/checkoutForm/Merchant/<merchant_id>"</span> accept-charset=<span style="color:#d20;background-color:#fff0f0">"utf-8"</span>>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"item_name_1"</span> value=<span style="color:#d20;background-color:#fff0f0">""</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"item_description_1"</span> value=<span style="color:#d20;background-color:#fff0f0">"Subscription Fees"</span>/>
Enter Amount to Deposit:<input type=<span style="color:#d20;background-color:#fff0f0">"text"</span> class=<span style="color:#d20;background-color:#fff0f0">"normal"</span> size=<span style="color:#d20;background-color:#fff0f0">"5"</span> name=<span style="color:#d20;background-color:#fff0f0">"item_price_1"</span> value=<span style="color:#d20;background-color:#fff0f0">""</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"item_currency_1"</span> value=<span style="color:#d20;background-color:#fff0f0">"USD"</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"item_quantity_1"</span> value=<span style="color:#d20;background-color:#fff0f0">"1"</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"shopping-cart.items.item-1.digital-content.display-disposition"</span> value=<span style="color:#d20;background-color:#fff0f0">"OPTIMISTIC"</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"shopping-cart.items.item-1.digital-content.description"</span> value=<span style="color:#d20;background-color:#fff0f0">"It may take up to 24 hours to process your deposit. Check your account balance and notify the Commissioner if it's not updated within 24 hours."</span>/>
<input type=<span style="color:#d20;background-color:#fff0f0">"hidden"</span> name=<span style="color:#d20;background-color:#fff0f0">"_charset_"</span> />
<!-- Button code -->
<input type=<span style="color:#d20;background-color:#fff0f0">"image"</span> name=<span style="color:#d20;background-color:#fff0f0">"Google Checkout"</span> alt=<span style="color:#d20;background-color:#fff0f0">"Fast checkout through Google"</span> src=<span style="color:#d20;background-color:#fff0f0">"http://sandbox.google.com/checkout/buttons/checkout.gif?merchant_id=<merchant_id>&w=180&h=46&style=white&variant=text&loc=en_US"</span> height=<span style="color:#d20;background-color:#fff0f0">"46"</span> width=<span style="color:#d20;background-color:#fff0f0">"180"</span> />>
</form></merchant_id></merchant_id>
</code></pre></div><h3 id="tweaking-google-merchant-libraries-for-php">Tweaking Google Merchant Libraries for PHP</h3>
<p>The business logic for my website includes:</p>
<ul>
<li>People enter amount they wish to deposit via Google Wallet and click “Buy Now” button.</li>
<li>Google sends back confirmation that card is pre-authorized and ready to charge</li>
<li>The script tells Google: Go ahead and charge it.</li>
<li>Google tells the site that charge went through.</li>
<li>Table “orders” in the database adds entry of transaction.</li>
</ul>
<p>The good news is that we have a very good PHP library for Google Wallet integration in place. The source code can be found <a href="http://code.google.com/p/google-checkout-php-sample-code/source/browse/trunk/">here</a>.</p>
<p>The bad news is that the sample code provided with the library does not entirely work with the newest 2.5 Google Wallet API. The library does have functions like SendChargeAndShipOrder (charge-and-ship-order) from 2.5 API implemented, but the demo example still uses the old function SendChargeOrder (charge-order).</p>
<p>I will tweak the sample code in responsehandlerdemo.php for my specific business requirements and 2.5 API version.</p>
<p>Customization is implemented with the help of useful Google Wallet <a href="https://developers.google.com/checkout/developer/Google_Checkout_Custom_Processing_How_To">callbacks</a> sent at various stages of order processing. First, my script will wait for the “authorization-amount-notification” notification that is sent after credit card data was verified and authorized. When the notification is received, I will issue the “charge-and-ship-order” command. The “charge-order” command used in the library’s sample code no longer works in 2.5 API, because charging and shipping is now done at once.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">switch</span> (<span style="color:#369">$root</span>) {
<span style="color:#080;font-weight:bold">case</span> <span style="color:#d20;background-color:#fff0f0">"authorization-amount-notification"</span>: {
<span style="color:#369">$google_order_number</span> = <span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'google-order-number'</span>][<span style="color:#d20;background-color:#fff0f0">'VALUE'</span>];
<span style="color:#369">$GChargeRequest</span> = <span style="color:#080;font-weight:bold">new</span> GoogleRequest(<span style="color:#369">$merchant_id</span>, <span style="color:#369">$merchant_key</span>, <span style="color:#369">$server_type</span>);
<span style="color:#369">$GChargeRequest</span>-><span style="color:#369">SendChargeAndShipOrder</span>(<span style="color:#369">$google_order_number</span>);
<span style="color:#369">$Gresponse</span>-><span style="color:#369">sendAck</span>(<span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'serial-number'</span>]);
<span style="color:#080;font-weight:bold">break</span>;
}
<span style="color:#080;font-weight:bold">default</span>:
<span style="color:#369">$Gresponse</span>-><span style="color:#369">sendAck</span>(<span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'serial-number'</span>]);
<span style="color:#080;font-weight:bold">break</span>;
}
</code></pre></div><p>I should receive the “charge-amount-notification” notification after the order is successfully charged. If we go to the Google Wallet dashboard, we will notice that the order is marked as charged and shipped. When the notification is received, the record is created in the “orders” table.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php"><span style="color:#080;font-weight:bold">case</span> <span style="color:#d20;background-color:#fff0f0">"charge-amount-notification"</span>: {
<span style="color:#369">$items</span> = get_arr_result(<span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'order-summary'</span>][<span style="color:#d20;background-color:#fff0f0">'shopping-cart'</span>][<span style="color:#d20;background-color:#fff0f0">'items'</span>][<span style="color:#d20;background-color:#fff0f0">'item'</span>]);
<span style="color:#369">$googleid</span> = <span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'google-order-number'</span>][<span style="color:#d20;background-color:#fff0f0">'VALUE'</span>];
<span style="color:#080;font-weight:bold">foreach</span>( <span style="color:#369">$items</span> <span style="color:#080;font-weight:bold">as</span> <span style="color:#369">$item</span> ) {
<span style="color:#369">$userid</span> =<span style="color:#369">$item</span>[<span style="color:#d20;background-color:#fff0f0">'item-name'</span>][<span style="color:#d20;background-color:#fff0f0">'VALUE'</span>];
<span style="color:#080;font-weight:bold">if</span>(isset(<span style="color:#369">$userid</span>)){
<span style="color:#369">$amount</span> = <span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'total-charge-amount'</span>][<span style="color:#d20;background-color:#fff0f0">'VALUE'</span>];
<span style="color:#369">$date</span> = <span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'timestamp'</span>][<span style="color:#d20;background-color:#fff0f0">'VALUE'</span>];
<span style="color:#369">$tmpsql</span> = <span style="color:#d20;background-color:#fff0f0">"INSERT into orders(google_id,amount,userid) VALUES('"</span>. <span style="color:#369">$googleid</span>.<span style="color:#d20;background-color:#fff0f0">"','"</span>.<span style="color:#369">$amount</span>.<span style="color:#d20;background-color:#fff0f0">"','"</span>.<span style="color:#369">$userid</span>.<span style="color:#d20;background-color:#fff0f0">"')"</span>;
<span style="color:#080;font-weight:bold">if</span>(! <span style="color:#369">$sqlresult</span>=mysql_query(<span style="color:#369">$tmpsql</span>)) {
<span style="color:#369">$Gresponse</span>-><span style="color:#369">log</span>-><span style="color:#369">LogError</span>(mysql_error());
}
}
}
<span style="color:#369">$Gresponse</span>-><span style="color:#369">sendAck</span>(<span style="color:#369">$data</span>[<span style="color:#369">$root</span>][<span style="color:#d20;background-color:#fff0f0">'serial-number'</span>]);
<span style="color:#080;font-weight:bold">break</span>;
}
</code></pre></div><h3 id="debugging-tools">Debugging tools</h3>
<p>Google provides a very useful Google Wallet <a href="http://support.google.com/checkout/sell/bin/answer.py?hl=en&answer=134469">Sandbox</a>, that the developers can use while testing their features. No actual orders are placed and no credit cards are charged.</p>
<p>All the low level warnings and exceptions are recorded to the Tools > Integration Console tab.</p>
<img border="0" src="/blog/2012/10/the-truth-about-google-wallet/image-2.jpeg"/>
<p>Finally, it can be very useful to check googlemessage.log file defined in the sample code.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-php" data-lang="php">define(<span style="color:#d20;background-color:#fff0f0">'RESPONSE_HANDLER_LOG_FILE'</span>, <span style="color:#d20;background-color:#fff0f0">'googlemessage.log'</span>);
</code></pre></div><h3 id="full-source-code">Full source code</h3>
<p>The full source code for the modified responsedemohandler.php can be found <a href="https://gist.github.com/eb63b05ab465c672c6ae">here</a>.</p>
Paginating API call with Radian6https://www.endpointdev.com/blog/2012/08/paginating-api-call-with-radian6/2012-08-24T00:00:00+00:00Marina Lohova
<p>I wrote about Radian6 in my <a href="/blog/2012/06/respecting-api-call-limit-with-radian6/">earlier blog post</a>. Today I will review one more aspect of Radian6 API - call pagination.</p>
<p>Most Radian6 requests return paginated data. This introduces extra complexity of making request several times in the loop in order to get all results. Here is one simple way to retrieve the paginated data from Radian6 using the powerful Ruby blocks.</p>
<p>I will use the following URL to fetch data:</p>
<p><em>/data/comparisondata/1338958800000/1341550800000/2777/8/9/6/</em></p>
<p>Let’s decypher this.</p>
<ul>
<li>
<p>1338958800000 is <strong>start_date</strong>, 1341550800000 is <strong>end_date</strong> for document search. It’s June, 06, 2012 - July, 06, 2012 formatted with date.to_time.to_i * 1000.</p>
</li>
<li>
<p>2777 is <strong>topic_id</strong>, a Radian6 term, denoting a set of search data for every customer.</p>
</li>
<li>
<p>8 stands for Twitter media type. There are various media types in Radian6. They reflect where the data came from. <strong>media_types</strong> parameter can include a list of values for different media types separated by commas.</p>
</li>
<li>
<p>9 and 6 are <strong>page</strong> and <strong>page_size</strong> respectively.</p>
</li>
</ul>
<p>First comes the method to fetch a single page.</p>
<p>In the Radian6 wrapper class:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">page</span>(index, &block)
data = block.call(index)
articles, count = data[<span style="color:#d20;background-color:#fff0f0">'article'</span>], data[<span style="color:#d20;background-color:#fff0f0">'article_count'</span>].to_i
[articles, count]
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>A data record in Radian6 is called an <strong>article</strong>. A call returns the ‘article’ field for a page of articles along other useful fields.</p>
<p>Now we will retrieve all pages of data from Radian6:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">paginated_call</span>(&block)
articles, index, count = [], <span style="color:#00d;font-weight:bold">0</span>, <span style="color:#00d;font-weight:bold">0</span>
<span style="color:#080;font-weight:bold">begin</span>
index += <span style="color:#00d;font-weight:bold">1</span>
batch, count = page(index, &block)
articles += batch
<span style="color:#080;font-weight:bold">end</span> <span style="color:#080;font-weight:bold">while</span> count > <span style="color:#00d;font-weight:bold">0</span>
articles
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>Time to enjoy the method! I’m using <a href="https://github.com/jnunemaker/httparty/">httparty</a> gem to make requests to API.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby">paginated_call <span style="color:#080;font-weight:bold">do</span> |page|
get(<span style="color:#d20;background-color:#fff0f0">"/data/comparisondata/1338958800000/1341550800000/2777/8/</span><span style="color:#33b;background-color:#fff0f0">#{</span>page<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">/1000/"</span>)
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>Thanks for flying!</p>
Respecting API Call Limit with Radian6https://www.endpointdev.com/blog/2012/06/respecting-api-call-limit-with-radian6/2012-06-29T00:00:00+00:00Marina Lohova
<p>A common practice in the online API world is to enforce the call limit. Twitter allows 150 API calls per hour. Shopify has 500 API calls per 5 minutes limit. You may learn how to work with Shopify call limit from <a href="https://web.archive.org/web/20120826132711/http://wiki.shopify.com/Learning_to_Respect_the_API_calls_limit">this great article</a>.</p>
<p>One of our projects was built around interaction with <a href="http://www.radian6.com/">Radian6</a> platform. Radian6 is a new and growing data mining service with the default limit of 100 calls per hour. I will describe our take on the call limit implementation.</p>
<h3 id="introducing-the-call-counter">Introducing the call counter</h3>
<p>First, we need to know how many calls have been executed in the current hour. Every external call increments the counter field on a special model until the counter reaches the limit. The counter is reset back to zero at the beginning of every hour.</p>
<pre tabindex="0"><code>script/rails g model RadianCallCount
</code></pre><p>In the migration:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">CreateRadianCallCounts</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Migration</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">change</span>
create_table <span style="color:#a60;background-color:#fff0f0">:radian_call_counts</span> <span style="color:#080;font-weight:bold">do</span> |t|
t.integer <span style="color:#a60;background-color:#fff0f0">:count</span>
t.timestamps
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>In db/seeds.rb file</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#038">puts</span> <span style="color:#d20;background-color:#fff0f0">"Initializing Radian6 counter"</span>
<span style="color:#036;font-weight:bold">RadianCallCount</span>.delete_all
<span style="color:#036;font-weight:bold">RadianCallCount</span>.create(<span style="color:#a60;background-color:#fff0f0">:count</span> => <span style="color:#00d;font-weight:bold">0</span>)
</code></pre></div><p>Let’s roll the counter!</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">rake db:migrate
rake db:seed
</code></pre></div><h3 id="scheduling-the-counter-reset">Scheduling the counter reset</h3>
<p>It is necessary to reset the counter back to zero in the beginning of each hour otherwise the subsequent calls will not be executed. The excellent ‘whenever’ gem will take care of this.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">gem install whenever
<span style="color:#038">cd</span> your_webapp
wheneverize .
</code></pre></div><p>In the model:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">RadianCallCount</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#b06;font-weight:bold">self</span>.<span style="color:#06b;font-weight:bold">reset</span>
<span style="color:#036;font-weight:bold">RadianCallCount</span>.first.update_attribute(<span style="color:#a60;background-color:#fff0f0">:count</span>, <span style="color:#00d;font-weight:bold">0</span>)
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>In config/schedule.rb:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby">every <span style="color:#a60;background-color:#fff0f0">:hour</span> <span style="color:#080;font-weight:bold">do</span>
runner <span style="color:#d20;background-color:#fff0f0">"RadianCallCount.reset"</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><h3 id="tracking-call-count">Tracking call count</h3>
<p>We will now use the <a href="https://rubygems.org/gems/rcapture">rcapture</a> gem to intercept a call to external API and increment the counter with it.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">gem install rcapture
</code></pre></div><p>In the module containing all Radian-specific methods:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#038">require</span> <span style="color:#d20;background-color:#fff0f0">'rcapture'</span>
<span style="color:#036;font-weight:bold">API_LIMIT</span> = <span style="color:#00d;font-weight:bold">100</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#b06;font-weight:bold">self</span>.<span style="color:#06b;font-weight:bold">included</span>(base)
base.extend <span style="color:#036;font-weight:bold">RCapture</span>::<span style="color:#036;font-weight:bold">Interceptable</span>
base.capture_pre <span style="color:#a60;background-color:#fff0f0">:methods</span> => [<span style="color:#a60;background-color:#fff0f0">:authenticate</span>,<span style="color:#a60;background-color:#fff0f0">:tweet_stats</span>] <span style="color:#080;font-weight:bold">do</span> |cs|
<span style="color:#036;font-weight:bold">RadianCallCount</span>.transaction <span style="color:#080;font-weight:bold">do</span>
calls_per_hour = <span style="color:#036;font-weight:bold">RadianCallCount</span>.first.count
allowed = (calls_per_hour < <span style="color:#036;font-weight:bold">Radian</span>::<span style="color:#036;font-weight:bold">API_LIMIT</span>)
cs.predicate = allowed
cs.return = <span style="color:#080">false</span>
<span style="color:#036;font-weight:bold">RadianCallCount</span>.first.increment!(<span style="color:#a60;background-color:#fff0f0">:count</span>) <span style="color:#080;font-weight:bold">if</span> allowed
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The code introduces a simple check before ‘authenticate’ and ‘tweets_stats’ methods. If call count exceeds the allowed limit the method is not executed and the method returns <strong>false</strong>. Otherwise, the counter increments after the successful method execution. We wrap the code in transaction because the actual count in the database may increase while we are making the API_LIMIT comparison.</p>
<h3 id="making-the-limit-aware-call">Making the limit-aware call</h3>
<p>Everything is ready to make the non-blocking API call. I scheduled a twitter statistics update to run every 3 hours:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby">every <span style="color:#00d;font-weight:bold">3</span>.hours <span style="color:#080;font-weight:bold">do</span>
runner <span style="color:#d20;background-color:#fff0f0">"Article.tweet_stats"</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The non-blocking calls are suitable for most situations. Sometimes there is a need to just keep trying…</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">call_with_timeout</span>(&block)
timeout = <span style="color:#00d;font-weight:bold">0</span>.minutes
results = <span style="color:#080">false</span>
<span style="color:#080;font-weight:bold">while</span> !results <span style="color:#080;font-weight:bold">do</span>
results = block.call
<span style="color:#080;font-weight:bold">if</span> !results
<span style="color:#080;font-weight:bold">break</span> <span style="color:#080;font-weight:bold">if</span> timeout >= <span style="color:#036;font-weight:bold">Radian</span>::<span style="color:#036;font-weight:bold">API_MAX_TIMEOUT</span>
<span style="color:#036;font-weight:bold">Rails</span>.logger.info(<span style="color:#d20;background-color:#fff0f0">"Sleeping for 5 minutes"</span>)
<span style="color:#038">sleep</span>(<span style="color:#036;font-weight:bold">Radian</span>::<span style="color:#036;font-weight:bold">API_CALL_TIMEOUT</span>)
timeout += <span style="color:#036;font-weight:bold">Radian</span>::<span style="color:#036;font-weight:bold">API_CALL_TIMEOUT</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
results
<span style="color:#080;font-weight:bold">end</span>
...
call_with_timeout { tweet_stats }
</code></pre></div><p>That is all for today. Thanks for your attention. Hope the post was useful.</p>
Handling Ecommerce Transactions with PayPalhttps://www.endpointdev.com/blog/2012/06/handling-ecommerce-transactions-with/2012-06-14T00:00:00+00:00Greg Davidson
<p><a href="https://www.flickr.com/photos/80083124@N08/7186533985/" title="IMG_0782.JPG by endpoint920, on Flickr"><img alt="IMG_0782.JPG" height="375" src="/blog/2012/06/handling-ecommerce-transactions-with/image-0.jpeg" width="500"/></a></p>
<h3 id="options">Options</h3>
<p>PayPal has several options for payment processing and Mark Johnson just shared his experiences working with saved credit cards using PayPal’s <a href="https://www.paypal.com/us/webapps/mpp/express-checkout">Express Checkout</a>.</p>
<h3 id="order-types">Order Types</h3>
<p>There are a couple of order types of transaction in Express Checkout:</p>
<ul>
<li>Standard: everything purchased in a single transaction</li>
<li>Custom: handles multiple shipments and multiple charges</li>
</ul>
<h3 id="whats-in-a-name">What’s in a name?</h3>
<p>PayPal generates something they call an “order” that is distinct from the order for a given merchant. This is typically confusing to merchants because their concept of an order simply refers to the order a customer has just placed in their ecommerce application. The PayPal “order” is created prior to any other transaction. If the authorization fails, the “order” is not removed (which would be nice) but lingers around for 29 days by default. When merchants ask about this, the response PayPal offers is to void the “order”.
For standard on API call (authorization) if that fails you have to do a second API call to void the order.</p>
<p>The “order” has little value except to specify a charge ceiling for a given ecommerce transactions. Although the ceiling is set by the “order”, there is the notion of an “order allowance”. By default this allowance or ceiling is set to 115% of the original charge but this can be negotiated and changed if required.</p>
<p>Merchants desire to have the “order” removed when it is no longer needed but this isn’t always possible. One of the limitations of PayPal from a developer’s perspective is that you cannot initiate any activity unless you are the customer. For example a customer service rep cannot enter an order on behalf of a customer. Also, telephone orders are not possible due to this limitation.</p>
<p>It is not possible to remove “order” if you might have subsequent changes to the order. This is common scenario in ecommerce apps where customers would like to add to or modify their order after it has been shipped. If the PayPal “order” is closed, subsequent transactions like this are not possible.</p>
<h3 id="api-calls-for-express-checkout">API Calls for Express Checkout</h3>
<p>When a customer initiates the check out process, a “Set Request” API call is made to PayPal to establish a session. The response includes a session ID. With Set Request, URLs need to be submitted which specify resources to redirect the customer to for either success for failure.</p>
<p>Get Request is another API method offered and the response will include the user id, shipping address and email of the customer. You can also force them to include a phone number and billing address if you like. PayPal by default likes to offer less information while ecommerce clients often like to have more information.</p>
<p>For standard orders, there is a “Do payment” call which combines the order setup and authorization into a single requests. For custom orders, the setup and authorization are performed separately. The developer has the option to either complete the order or leave it open. If you choose to leave it open, you can perform multiple capture or charges up to the ceiling amount that has been specified.</p>
<h3 id="pci-compliance">PCI Compliance</h3>
<p>The payment card industry has a set of regulations including:</p>
<ul>
<li>The full account number must be encrypted</li>
<li>The CVC cannot be written to permanent media</li>
<li>PAN displays must be masked</li>
</ul>
<p>Mark noted that Secure decryption is a challenge and he has worked on a secure decryption service for one of our ecommerce clients.</p>
<h3 id="saved-cards">Saved Cards</h3>
<p>Authorize.net has a customer information manager (CIM) which supports “profiles”. These profiles are referenced by tokens and the data storage includes PANs. There are three types of profiles: customer, payment and address. The default behaviours is to use them for “on-demand” transactions.</p>
<h3 id="pity-the-fool">Pity the fool…</h3>
<pre tabindex="0"><code>-->>:[]}}}}-O
</code></pre><p>Mr. T made several cameo appearances during the talk to keep us all engaged and interested!</p>
Our SoftLayer API toolshttps://www.endpointdev.com/blog/2012/01/our-softlayer-api-tools/2012-01-23T00:00:00+00:00Jon Jensen
<p>We do a lot of our hosting at <a href="http://www.softlayer.com/">SoftLayer</a>, which seems to be one of the <a href="http://www.datacenterknowledge.com/archives/2009/05/14/whos-got-the-most-web-servers/">hosts with the most servers</a> in the world – they claim to have over 100,000 servers as of last month. More important for us than sheer size are many other fine attributes that SoftLayer has, in no particular order:</p>
<ul>
<li>a strong track record of reliability</li>
<li>responsive support</li>
<li>datacenters around the U.S. and some in Europe and Asia</li>
<li>solid power backup</li>
<li>well-connected redundant networks with multiple 10 Gbps uplinks</li>
<li>gigabit Ethernet pipes all the way to the Internet</li>
<li>first-class IPv6 support</li>
<li>an internal private network with no data transfer charge</li>
<li>Red Hat Enterprise Linux offered at no extra charge</li>
<li>diverse dedicated server offerings at many price & performance points</li>
<li>some disk partitioning options (though more flexibility here would be nice, especially with LVM for the /boot and / filesystems)</li>
<li>fully automated provisioning, without salesman & quote hassles for standard offerings</li>
<li>3000 GB data transfer per month included standard with most servers</li>
<li>month-to-month contracts</li>
<li>reasonable prices (though we can of course always use lower prices, we’ll take quality over cheapness for most of our hosting needs!)</li>
<li>no arbitrary port blocks (some other providers rate-limit incoming TCP connections on port 22 to slow down ssh dictionary attacks, while others forbid IRC, etc.)</li>
<li>a web service API for monitoring and controlling many aspects of our account via REST/JSON or SOAP</li>
</ul>
<p>(No, they’re not paying me for writing this! But they really have nice offerings.)</p>
<p>It is this last item, the SoftLayer API, that I want to elaborate on here.</p>
<p>The <a href="http://sldn.softlayer.com/">SoftLayer Development Network</a> features API information and documentation and once you have an API account set up in the management website (quick and easy to do), you can start automating all sorts of tasks, from provisioning new hosts, monitoring your upcoming invoice or other accounting information, and much more.</p>
<p>I’ve released as open source two scripts we use: One is for managing secondary DNS domains in SoftLayer’s DNS servers, from a primary name server running BIND 9. The other is a Nagios check script for monitoring monthly data transfer used and alerting when over a set threshold or over the monthly allotment.</p>
<p>See the GitHub repository of <a href="https://github.com/jonjensen/endpoint-softlayer-api">endpoint-softlayer-api</a> if they would be useful to you, or to use as a starting point to interface with other SoftLayer APIs.</p>