https://www.endpointdev.com/blog/tags/solr/2019-09-09T00:00:00+00:00End Point DevCampendium: A Responsive, Fancy Detail Pagehttps://www.endpointdev.com/blog/2019/09/campendium-detail-page/2019-09-09T00:00:00+00:00Steph Skardal
<p><img src="/blog/2019/09/campendium-detail-page/banner.jpg" alt=""></p>
<p>This week, I was very excited to deploy a project for <a href="https://www.campendium.com/">Campendium</a>, one of our long-time clients. As noted in <a href="/blog/2019/08/campendium-updates/">my recent post on Campendium updates for the year</a>, Campendium has thousands of listings of places to camp and provides a great infrastructure for the development of a rich community of travelers around North America.</p>
<p>For the past few months, I’ve been working on a significant update to Campendium’s campground detail page, the page template where in-depth information is provided for each of Campendium’s locations. This is equivalent to a product detail page on an ecommerce site.</p>
<p>The “guts” of the update included a new detail page design with expanded responsiveness, the introduction of 360° videos, and expanded user driven content in the form of Question & Answer (Q&A or QnA), reviews, notes, nightly rates, etc. Read on to find out more and see video examples of several of the features!</p>
<h3 id="user-interface--responsiveness-updates">User Interface & Responsiveness Updates</h3>
<p>One of the things the Campendium team and I are most proud of here is the responsiveness of the new design. In the case of traveling and camping, responsiveness is important since a large amount of traffic comes from mobile devices, relative to what you might see in other industries.</p>
<p>User images are shown as “hero images” and the user interface updates depending on the browser device and width, as shown in the following videos for <a href="https://www.campendium.com/gilbert-ray-campground">Gilbert Ray Campground</a> and <a href="https://www.campendium.com/cayuga-lake-state-park">Cayuga Lake State Park</a>.</p>
<p style="text-align:center;font-weight:bold;"><iframe src="https://drive.google.com/file/d/1FFF8GcztnV-KGaJ4pjDYixCbkxw6kduu/preview" width="770" height="450"></iframe>A preview of responsive behavior on the campground detail page with user submitted photos.</p>
<p style="text-align:center;font-weight:bold;"><iframe src="https://drive.google.com/file/d/1caQZT9ogh_piuTX2UDIvWGEBZse0zwnx/preview" width="770" height="450"></iframe>A second preview of responsive behavior on the campground detail page with embedded maps.</p>
<p>Another updated design usability tweak was a sticky navigation bar to navigation throughout the page, which can get especially long with user submitted content. See how the “Overview”, “Video”, etc. links become sticky as you scroll down on the page, and the current region coming into view of the page is underlined:</p>
<p style="text-align:center;font-weight:bold;"><iframe src="https://drive.google.com/file/d/1w_MjP_HUXntYXHTG6FeREtMWwhKv4KgG/preview" width="770" height="450"></iframe>Navigation becomes fixed to the top of the browser as a user scrolls through the content.</p>
<p>Campendium uses Ruby on Rails as a backend, a bit of <a href="https://getbootstrap.com/">Bootstrap</a>, and <a href="https://sass-lang.com/">Sass</a> and best practice responsive design is used throughout this updated user interface.</p>
<h3 id="360-videos">360° Videos</h3>
<p>The Campendium team has been hard at work creating 360 degree videos of the various locations to provide significant value to their users. These videos are now embedded into the campground detail page. Video hosting is provided by YouTube and dropped into the Campendium campground HTML template.</p>
<p style="text-align:center;font-weight:bold;"><iframe src="https://drive.google.com/file/d/12XEu-95diRRMjlqb28jD0xdj9qIWXSb7/preview" width="770" height="450"></iframe>A video in a video—a 360 degree video example.</p>
<p>I personally think these 360° videos are awesome, especially to those visual learners out there! You can learn so much from a video that can be hard to capture in user reviews.</p>
<h3 id="community-qa">Community Q&A</h3>
<p>Another new feature with this deploy is the introduction of community driven Question & Answer. Users can submit questions about a campground, and users who have contributed to this campground are invited to respond (or they can opt-out site-wide to responding to questions). In addition to Q&A itself, user content can be “upvoted” (marked as Helpful) or flagged for an improved user content browsing experience.</p>
<p><img src="/blog/2019/09/campendium-detail-page/community.png" alt="Campendium community in Question & Answer"></p>
<p>All Q&A functionality is driven by JavaScript coupled with the Ruby on Rails backend.</p>
<h3 id="expansion-of-user-contributed-content-and-features">Expansion of User Contributed Content and Features</h3>
<p>Finally, this update includes expansion of user contributed content in addition to Q&A. This includes the ability for users to contribute notes, cell signal reports, and nightly rates. Reviews can now be filtered by user profile information and sorted by date or specific ranking. Reviews can also be searched by keywords and those keywords highlighted in the results. The following video demonstrates sorting and searching of reviews with highlighted keyword match:</p>
<p style="text-align:center;font-weight:bold;"><iframe src="https://drive.google.com/file/d/1G2KGfxCkOmuJm-AJE8BPXUC-OWdw1iQS/preview" width="770" height="450"></iframe>Review searchability with highlighting and filterability.</p>
<p>All search functionality driven by JavaScript on the frontend, and a Ruby on Rails backend coupled with <a href="https://github.com/sunspot/sunspot">Sunspot</a> and <a href="https://lucene.apache.org/solr/">Solr</a>.</p>
<h3 id="who-doesnt-love-stats">Who Doesn’t Love Stats?</h3>
<p>Just for the sake of sharing stats, from GitHub, this update included:</p>
<ul>
<li>215 changed files</li>
<li>6,284 additions</li>
<li>3,924 deletions (removal / cleanup of unneeded JavaScript files)</li>
</ul>
Campendium v2019: A Summary of Recent Updateshttps://www.endpointdev.com/blog/2019/08/campendium-updates/2019-08-05T00:00:00+00:00Steph Skardal
<p>This year has brought a handful of exciting changes for <a href="https://www.campendium.com/">Campendium</a>, one of End Point’s long-time clients, by yours truly. Created by campers for campers, Campendium has thousands of listings of places to camp, from swanky RV parks to free remote destinations, vetted by a team of full-time travelers and reviewed by over 200,000 members. I thought I would take some time to summarize these recent updates.</p>
<h3 id="maps-and-clustering">Maps and Clustering</h3>
<p><img src="/blog/2019/08/campendium-updates/map-clusters.png" alt="Campendium map clustering of campground locations"></p>
<p>Campendium uses <a href="https://www.mapbox.com/">Mapbox</a> for map rendering to display campgrounds and locations throughout North America. One of the new features added this year was <a href="https://docs.mapbox.com/mapbox-gl-js/example/cluster/">clustering</a> of campground locations, where campgrounds are grouped together and presented in a “cluster” with a size relative to how many campgrounds are in the cluster.</p>
<p>If a user is searching for campgrounds in a broad location, they can see where campgrounds might be more densely grouped by location. Once a user zooms in zoom in a couple of clicks, the campgrounds are no longer clustered and individual campgrounds locations can be seen. While working on this update, we spent a good amount of our time tweaking and troubleshooting the optimal clustering behavior to provide the most benefit to those searching for a campground. <a href="https://docs.mapbox.com/mapbox-gl-js/api/">Mapbox GL JS</a> works in parallel with <a href="https://reactjs.org/">ReactJS</a>, and runs with a Ruby on Rails back-end.</p>
<p><img src="/blog/2019/08/campendium-updates/map-non-clusters.jpg" alt="Campendium map non-clustering of campground locations after zooming in"></p>
<h3 id="advanced-filtering">Advanced Filtering</h3>
<p><img src="/blog/2019/08/campendium-updates/map-filtering.jpg" alt="Campendium advanced filtering"></p>
<p>Another exciting was the introduction of advanced filtering in the search interface, presented in combination with map display. Users can filter campgrounds by category (e.g. Public Land, RV Parking, Parking, Dump Station), filter by price (with a slider), hookups, campground policy (e.g. age or pet restrictions), discounts, recreation, and facilities. All of this search filtering is driven by <a href="https://github.com/sunspot/sunspot">Sunspot</a>, a Ruby on Rails gem for working with the popular <a href="https://lucene.apache.org/solr/">Solr</a> search engine. Results can be sorted by user provided reviews, price or distance from a specific GPS location. Here, much care was given to provide the best user interface for presenting this valuable functionality.</p>
<h3 id="supporters-only-features">“Supporters Only” Features</h3>
<p><img src="/blog/2019/08/campendium-updates/supporters.jpg" alt="Campendium Supporters only, Subscriptions"></p>
<p>Another recent update to Campendium includes functionality to offer user subscriptions. Registered users can sign up to support Campendium on a monthly or annual basis, and subscriptions are set to auto-renew at the end of their subscription period. This paid support hides advertisements throughout the site (advertisements are controlled by a third party), and advanced filtering on cell reception. There are plans to expand supporter features in the future. Ruby on Rails combined with <a href="https://stripe.com/docs/stripe-js/reference">StripeJS</a> is used to manage subscription payments, and Ruby on Rails also serves as a backend for <a href="https://developer.apple.com/in-app-purchase/">In-App Purchases</a> of subscriptions from the App store.</p>
<h3 id="always-responsive-and-latency-aware">Always Responsive and Latency-Aware</h3>
<p>Because a large portion of the Campendium visitors are on the road, it’s important to have both a responsive design and to build for bandwidth limitations for users. Throughout the development of these new features, responsive and mobile friendly designs were implemented leveraging <a href="https://sass-lang.com/">Sass</a>, sometimes requiring help from the rest of the knowledgeable <a href="/team/">End Point team</a>!</p>
<p>Many of the pages throughout the site are fully cached including the homepage, search result pages, and campground detail page, and cookies are used to indicate user status. In some cases, user submitted campground images are lazy-loaded to mitigate bandwidth limitations.</p>
<p><img src="/blog/2019/08/campendium-updates/mobile.jpg" alt="Mobile, responsive design for Campendium"></p>
<h3 id="whats-next">What’s Next?</h3>
<p>While I didn’t go into much technical depth on these updates, I am happy that the updates represent a broad spectrum of full-stack development skills featuring nginx, Ruby on Rails, 3rd party integration including StripeJS, MapboxGL and IAP (Apple), a JavaScript framework with ReactJS, and working with Ruby gems to leverage other tools, for example, Solr (Sunspot) and Sass.</p>
<p>In the future, Campendium plans to continue using these tools to see a more interactive, social campground detail page, and has plans to expand outside of North America. You can visit Campendium <a href="https://www.campendium.com/">here</a>, or find them on Instagram <a href="https://www.instagram.com/campendium/?hl=en">here</a> to follow their exciting announcements!</p>
Sunspot, Solr, Rails: Working with Resultshttps://www.endpointdev.com/blog/2011/12/sunspot-solr-rails-working-with-results/2011-12-12T00:00:00+00:00Steph Skardal
<p>Having worked with <a href="http://sunspot.github.io/">Sunspot</a> and <a href="https://lucene.apache.org/solr/">Solr</a> in several large Rails projects now, I’ve gained some knowledge about working with result sets optimally. Here’s a brief explanation on working with results or hits from a search object.</p>
<h3 id="mvc-setup">MVC Setup</h3>
<p>When working with Sunspot, searchable fields are defined in the model:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">Thing</span> < <span style="color:#036;font-weight:bold">ActiveRecord</span>::<span style="color:#036;font-weight:bold">Base</span>
searchable <span style="color:#080;font-weight:bold">do</span>
text <span style="color:#a60;background-color:#fff0f0">:field1</span>, <span style="color:#a60;background-color:#fff0f0">:stored</span> => <span style="color:#080">true</span>
text <span style="color:#a60;background-color:#fff0f0">:field2</span>
string <span style="color:#a60;background-color:#fff0f0">:field3</span>, <span style="color:#a60;background-color:#fff0f0">:stored</span> => <span style="color:#080">true</span>
integer <span style="color:#a60;background-color:#fff0f0">:field4</span>, <span style="color:#a60;background-color:#fff0f0">:multiple</span> => <span style="color:#080">true</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>The code block above will include field1, field2, field3, and field4 in the search index of <strong>things</strong> . A keyword or text search on things will search field1 and field2 for matches. field3 and field4 may be used for scoping, or limiting the search result set based to specific values of field3 or field4.</p>
<p>In your controller, a new search object is created with the appropriate scoping and keyword values, shown below. Pagination is also added inside the search block.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ThingsController</span> < <span style="color:#036;font-weight:bold">ApplicationController</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">index</span>
<span style="color:#33b">@search</span> = <span style="color:#036;font-weight:bold">Sunspot</span>.search(<span style="color:#036;font-weight:bold">Thing</span>) <span style="color:#080;font-weight:bold">do</span>
<span style="color:#888">#fulltext search</span>
fulltext params[<span style="color:#a60;background-color:#fff0f0">:keyword</span>]
<span style="color:#888">#scoping</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:field3</span>)
with <span style="color:#a60;background-color:#fff0f0">:field3</span>, params[<span style="color:#a60;background-color:#fff0f0">:field3</span>]
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:field4</span>)
with <span style="color:#a60;background-color:#fff0f0">:field3</span>, params[<span style="color:#a60;background-color:#fff0f0">:field4</span>]
<span style="color:#080;font-weight:bold">end</span>
paginate <span style="color:#a60;background-color:#fff0f0">:page</span> => params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#a60;background-color:#fff0f0">:per_page</span> => <span style="color:#00d;font-weight:bold">25</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#33b">@search</span>.execute!
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>In the view, one can iterate through the result set, where results is an array of Thing instances.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><% @search.results.each do |result| -%>
<h2><%= result.field3 %></h2>
<%= result.field1 %>
<% end -%>
</code></pre></div><h3 id="working-with-hits">Working with Hits</h3>
<p>The above code works. It works nicely until you display many results on one page where instantiation of things is not expensive. But the above code will call the query below for every search, and subsequently instantiate Ruby objects for each of the things found. This can become sluggish when the result set is large or the items themselves are expensive to instantiate.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># development.log
Thing Load (0.9ms) SELECT "things".* FROM "things" WHERE "things"."id" IN (6, 12, 7, 13, 8, ...)
</code></pre></div><p>An optimized way to work with search results sets is working directly with hits. @search.hits is an array of Sunspot::Search::Hits, which represent the raw information returned by Solr for a single returned item. Hit objects provide access to stored field values, identified by the :stored option in the model’s searchable definition. The model definition looks the same. The controller may now look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ThingsController</span> < <span style="color:#036;font-weight:bold">ApplicationController</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">index</span>
search = <span style="color:#036;font-weight:bold">Sunspot</span>.search(<span style="color:#036;font-weight:bold">Thing</span>) <span style="color:#080;font-weight:bold">do</span>
<span style="color:#888">#fulltext search</span>
fulltext params[<span style="color:#a60;background-color:#fff0f0">:keyword</span>]
<span style="color:#888">#scoping</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:field3</span>)
with <span style="color:#a60;background-color:#fff0f0">:field3</span>, params[<span style="color:#a60;background-color:#fff0f0">:field3</span>]
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:field4</span>)
with <span style="color:#a60;background-color:#fff0f0">:field3</span>, params[<span style="color:#a60;background-color:#fff0f0">:field4</span>]
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
search.execute!
<span style="color:#33b">@hits</span> = search.hits.paginate <span style="color:#a60;background-color:#fff0f0">:page</span> => params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#a60;background-color:#fff0f0">:per_page</span> => <span style="color:#00d;font-weight:bold">25</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>And working with the data in the view may look like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><% @hits.each do |result| -%>
<h2><%= hit.stored(:field3) %></h2>
<%= hit.stored(:field1) %>
<% end -%>
</code></pre></div><p>In some cases, you may want to introduce an additional piece of logic prior pagination, which is the case with the most recent Rails application I’ve been working on:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"> ...
search.execute!
filtered_results = []
search.hits.each <span style="color:#080;font-weight:bold">do</span> |hit|
<span style="color:#080;font-weight:bold">if</span> hit.stored(<span style="color:#a60;background-color:#fff0f0">:field3</span>) == <span style="color:#d20;background-color:#fff0f0">"some arbitrary value"</span>
filtered_results << hit
<span style="color:#080;font-weight:bold">elsif</span> hit.stored(<span style="color:#a60;background-color:#fff0f0">:field1</span>) == <span style="color:#d20;background-color:#fff0f0">"some other arbitrary value"</span>
filtered_results << hit
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#33b">@hits</span> = filtered_results.paginate <span style="color:#a60;background-color:#fff0f0">:page</span> => params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#a60;background-color:#fff0f0">:per_page</span> => <span style="color:#00d;font-weight:bold">25</span>
</code></pre></div><p>Sunspot and Solr are rich with functionality and features that can add value to a Rails application, but it’s important to identify areas of the application where database calls can be minimized and lazy loading can be optimized for better performance. The standard log file and database log file are good places to start looking.</p>
Rails Optimization: Advanced Techniques with Solrhttps://www.endpointdev.com/blog/2011/07/rails-optimization-advanced-techniques/2011-07-22T00:00:00+00:00Steph Skardal
<p>Recently, I’ve been involved in optimization on a Rails 2.3 application. The application had pre-existing fragment caches throughout the views with the use of Rails sweepers. Fragment caches are used throughout the site (rather than action or page caches) because the application has a fairly complex role management system that manages edit access at the instance, class, and site level. In addition to server-side optimization with more fragment caching and query clean-up, I did significant asset-related optimization including extensive use of CSS sprites, combining JavaScript and CSS requests where ever applicable, and optimizing images with tools like pngcrush and jpegtran. Unfortunately, even with the server-side and client-side optimization, my response times were still sluggish, and the server response was the most time consuming part of the request for a certain type of page that’s expected to be hit frequently:</p>
<img alt="" border="0" id="BLOGGER_PHOTO_ID_5628578619063114610" src="/blog/2011/07/rails-optimization-advanced-techniques/image-0.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;"/>
<p>A first stop in optimization was to investigate if memcached would speed up the site, described <a href="/blog/2011/07/raw-caching-performance-in-rubyrails/">in this article</a> Unfortunately, that did not improve the speed much.</p>
<p>Next, I re-examined the debug log to see what was taking so much time. The debug log looked like this (note that table names have been changed):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Processing ThingsController#index (for 174.111.14.48 at 2011-07-12 16:32:04) [GET]
Parameters: {"action"=>"index", "controller"=>"things"}
Thing Load (441.2ms) SELECT * FROM "things" WHERE ("things"."id" IN (22,6,23,7,35,24,36,25,14,9,37,26,15,...))
Rendering template within layouts/application
Rendering things/index
Cached fragment hit: views/all-tags (1.6ms)
Rendered things/_nav_search (3.3ms)
Rendered shared/_sort (0.2ms)
Cached fragment hit: ...
Cached fragment hit: ...
Cached fragment hit: ...
Rendered ...
Rendered ...
Completed in 821ms (View: 297, DB: 443) | 200 OK [http://www.mysite.com/things]
</code></pre></div><p>From the debug log, we can point out:</p>
<ul>
<li>The page loads in 821ms according to Rails, which is similar to the time reported in the waterfall shown above.</li>
<li>The page is loading several cached fragments, which is good.</li>
<li>The biggest time-suck of the page loading is a SELECT * FROM things …</li>
</ul>
<p>To rule out any database slowness due to missing indexes, I examined the query speed via console (note that this application runs on PostgreSQL):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql">=><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">EXPLAIN</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ANALYZE</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"things"</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">WHERE</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">"things"</span>.<span style="color:#d20;background-color:#fff0f0">"id"</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">IN</span><span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">22</span>,<span style="color:#00d;font-weight:bold">6</span>,<span style="color:#00d;font-weight:bold">23</span>,<span style="color:#00d;font-weight:bold">7</span>,<span style="color:#00d;font-weight:bold">35</span>,<span style="color:#00d;font-weight:bold">24</span>,<span style="color:#00d;font-weight:bold">36</span>,<span style="color:#00d;font-weight:bold">25</span>,<span style="color:#00d;font-weight:bold">14</span>,<span style="color:#00d;font-weight:bold">9</span>,<span style="color:#00d;font-weight:bold">37</span>,<span style="color:#00d;font-weight:bold">26</span>,<span style="color:#00d;font-weight:bold">15</span>,...));<span style="color:#bbb">
</span><span style="color:#bbb"> </span>QUERY<span style="color:#bbb"> </span>PLAN<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#888">------------------------------------------------------------------------------------------------------------
</span><span style="color:#888"></span><span style="color:#bbb"> </span>Seq<span style="color:#bbb"> </span>Scan<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">on</span><span style="color:#bbb"> </span>things<span style="color:#bbb"> </span>(cost=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">00</span>..<span style="color:#00d;font-weight:bold">42</span>.<span style="color:#00d;font-weight:bold">19</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">24</span><span style="color:#bbb"> </span>width=<span style="color:#00d;font-weight:bold">760</span>)<span style="color:#bbb"> </span>(actual<span style="color:#bbb"> </span>time=<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">023</span>..<span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">414</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>=<span style="color:#00d;font-weight:bold">25</span><span style="color:#bbb"> </span>loops=<span style="color:#00d;font-weight:bold">1</span>)<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Filter:<span style="color:#bbb"> </span>(id<span style="color:#bbb"> </span>=<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">ANY</span><span style="color:#bbb"> </span>(<span style="color:#d20;background-color:#fff0f0">'{22,6,23,7,35,24,36,25,14,9,37,26,15,...}'</span>::<span style="color:#038">integer</span>[]))<span style="color:#bbb">
</span><span style="color:#bbb"> </span>Total<span style="color:#bbb"> </span>runtime:<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">0</span>.<span style="color:#00d;font-weight:bold">452</span><span style="color:#bbb"> </span>ms<span style="color:#bbb">
</span><span style="color:#bbb"></span>(<span style="color:#00d;font-weight:bold">3</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">rows</span>)<span style="color:#bbb">
</span></code></pre></div><p>The query here is on the scale of 1000 times faster than the loading of the objects from the ThingsController. It’s well known that object instantiation in Ruby is slow. There’s not much I can do to speed up the pure performance of object instantation except possibly 1) upgrade to Ruby 1.9 or 2) try something like JRuby or Rubinius, which are both out of the scope of this project.</p>
<p>My next best option is to investigate using Rails low-level caching here to cache my objects pulled from the database, but there are a few challenges with this:</p>
<ul>
<li>The object instantiation is happening as part of a Solr (via sunspot) query, not a standard ActiveRecord lookup.</li>
<li>The Solr object that’s retrieved is used for pagination with the will_paginate gem.</li>
<li>Rails low-level caches can only store serializable objects. The Solr search object and WillPaginate:Collection object (a wrapper around an array of elements that can be paginated) are not serializable, so I must determine a suitable structure to store in the cache.</li>
</ul>
<h3 id="controller">Controller</h3>
<p>After troubleshooting, here’s what I came up with:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#33b">@things</span> = <span style="color:#036;font-weight:bold">Rails</span>.cache.fetch(<span style="color:#d20;background-color:#fff0f0">"things-search-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:page</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:tag</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:sort</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">"</span>) <span style="color:#080;font-weight:bold">do</span>
things = <span style="color:#036;font-weight:bold">Sunspot</span>.new_search(<span style="color:#036;font-weight:bold">Thing</span>)
things.build <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:tag</span>)
with <span style="color:#a60;background-color:#fff0f0">:tag_list</span>, <span style="color:#036;font-weight:bold">CGI</span>.unescape(params[<span style="color:#a60;background-color:#fff0f0">:tag</span>])
<span style="color:#080;font-weight:bold">end</span>
with <span style="color:#a60;background-color:#fff0f0">:active</span>, <span style="color:#080">true</span>
paginate <span style="color:#a60;background-color:#fff0f0">:page</span> => params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#a60;background-color:#fff0f0">:per_page</span> => <span style="color:#00d;font-weight:bold">25</span>
order_by params[<span style="color:#a60;background-color:#fff0f0">:sort</span>].to_sym, <span style="color:#a60;background-color:#fff0f0">:asc</span>
<span style="color:#080;font-weight:bold">end</span>
things.execute!
t = things.hits.inject([]) { |arr, h| arr.push(h.result); arr }
{ <span style="color:#a60;background-color:#fff0f0">:results</span> => t,
<span style="color:#a60;background-color:#fff0f0">:count</span> => things.total }
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#33b">@things</span> = <span style="color:#036;font-weight:bold">WillPaginate</span>::<span style="color:#036;font-weight:bold">Collection</span>.create(params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#00d;font-weight:bold">25</span>, <span style="color:#33b">@things</span>[<span style="color:#a60;background-color:#fff0f0">:count</span>]) { |pager| pager.replace(<span style="color:#33b">@things</span>[<span style="color:#a60;background-color:#fff0f0">:results</span>]) }
</code></pre></div><p>Here’s how it breaks down:</p>
<ul>
<li>My cache key is based on the page #, tag information, and sort type, shown in the argument passed into the low-level cache build:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#33b">@things</span> = <span style="color:#036;font-weight:bold">Rails</span>.cache.fetch(<span style="color:#d20;background-color:#fff0f0">"things-search-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:page</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:tag</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">-</span><span style="color:#33b;background-color:#fff0f0">#{</span>params[<span style="color:#a60;background-color:#fff0f0">:sort</span>]<span style="color:#33b;background-color:#fff0f0">}</span><span style="color:#d20;background-color:#fff0f0">"</span>) <span style="color:#080;font-weight:bold">do</span>
<span style="color:#888">###</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><ul>
<li>All this stuff creates a Solr object, sets the Solr object details, and builds the result set. In this particular Solr object, we are pulling things that have an :active value of true, may or may not have a specific tag, limiting the result set to 25, and ordering by the :sort parameter:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"> things = <span style="color:#036;font-weight:bold">Sunspot</span>.new_search(<span style="color:#036;font-weight:bold">Thing</span>)
things.build <span style="color:#080;font-weight:bold">do</span>
<span style="color:#080;font-weight:bold">if</span> params.has_key?(<span style="color:#a60;background-color:#fff0f0">:tag</span>)
with <span style="color:#a60;background-color:#fff0f0">:tag_list</span>, <span style="color:#036;font-weight:bold">CGI</span>.unescape(params[<span style="color:#a60;background-color:#fff0f0">:tag</span>])
<span style="color:#080;font-weight:bold">end</span>
with <span style="color:#a60;background-color:#fff0f0">:active</span>, <span style="color:#080">true</span>
paginate <span style="color:#a60;background-color:#fff0f0">:page</span> => params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#a60;background-color:#fff0f0">:per_page</span> => <span style="color:#00d;font-weight:bold">25</span>
order_by params[<span style="color:#a60;background-color:#fff0f0">:sort</span>].to_sym, <span style="color:#a60;background-color:#fff0f0">:asc</span>
<span style="color:#080;font-weight:bold">end</span>
things.execute!
</code></pre></div><ul>
<li><strong>things</strong> is my Sunspot/Solr object. I build an array of the Solr result set items and record the total number of things found. A hash that contains an array of “things” and a total count is my serializable cacheable object.</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"> t = things.hits.inject([]) { |arr, h| arr.push(h.result); arr }
{ <span style="color:#a60;background-color:#fff0f0">:results</span> => t,
<span style="color:#a60;background-color:#fff0f0">:count</span> => things.total }
</code></pre></div><ul>
<li>The tricky part here is building a WillPaginate::Collection object after pulling the cached data, since a WillPaginate object is also not serializable. This needs to know what the current page is, things per page, and total number of things found to correctly build the pagination links, but it doesn’t require that you have all the other “things” available:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#33b">@things</span> = <span style="color:#036;font-weight:bold">WillPaginate</span>::<span style="color:#036;font-weight:bold">Collection</span>.create(params[<span style="color:#a60;background-color:#fff0f0">:page</span>], <span style="color:#00d;font-weight:bold">25</span>, <span style="color:#33b">@things</span>[<span style="color:#a60;background-color:#fff0f0">:count</span>]) { |pager| pager.replace(<span style="color:#33b">@things</span>[<span style="color:#a60;background-color:#fff0f0">:results</span>]) }
</code></pre></div><h3 id="view">View</h3>
<p>My view contains the standard will_paginate reference:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">There are <%= pluralize @things.total_entries, 'Thing' %> Total
<%= will_paginate @things %>
</code></pre></div><p>And I pass the result set in a partial as a collection to display my listed items:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"><%= render :partial => 'shared/single_thing', :collection => @things %>
</code></pre></div><h3 id="sweepers">Sweepers</h3>
<p>Another thing to get right here is clearing the low-level cache with Rails sweepers. I have a fairly standard Sweeper setup similar to the one <a href="https://apidock.com/rails/ActionController/Caching/Sweeping">described here</a>. I utilize two ActiveRecord callbacks (after_save, before_destroy) in my sweeper to clear the cache, shown below.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">ThingSweeper</span> < <span style="color:#036;font-weight:bold">ActionController</span>::<span style="color:#036;font-weight:bold">Caching</span>::<span style="color:#036;font-weight:bold">Sweeper</span>
observe <span style="color:#036;font-weight:bold">Thing</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">after_save</span>(record)
<span style="color:#036;font-weight:bold">Rails</span>.cache.delete_matched(<span style="color:#080;background-color:#fff0ff">%r{things-search*}</span>)
<span style="color:#888"># expire_fragment ...</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">def</span> <span style="color:#06b;font-weight:bold">before_destroy</span>(record)
<span style="color:#036;font-weight:bold">Rails</span>.cache.delete_matched(<span style="color:#080;background-color:#fff0ff">%r{things-search*}</span>)
<span style="color:#888"># expire_fragment ...</span>
<span style="color:#080;font-weight:bold">end</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>With the changes described here (caching a serializable hash with the Solr results and total count, generating a WillPaginate:Collection object, and defining the Sweepers to clear the cache), I saw great improvements in performance. The standard “index” page request does not hit the database at all for users not logged in nor does it experience the sluggish object instantiation. My waterfall now looks like this:</p>
<img alt="" border="0" id="BLOGGER_PHOTO_ID_5628578620851292354" src="/blog/2011/07/rails-optimization-advanced-techniques/image-1.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;"/>
<p>And running 100 requests at a concurrency of 1 on the system (running in production on a development server) shows the requests are averaging 165ms, which is decent. After I wrote this post, I did a even more optimization on a different page type in the application that I hope to share in a future blog post.</p>
<p><strong>Note:</strong> Ideally, it would be better to cache individual objects so that I would not have expire entire search caches on every save or delete. However, I could not find methods in Solr that allows us to pull a list of ids of the result set without building the result set.</p>