https://www.endpointdev.com/blog/tags/backups/2022-05-31T00:00:00+00:00End Point DevBacking up your SaaS data with Google Takeouthttps://www.endpointdev.com/blog/2022/05/backing-up-your-saas-data-with-google-takeout/2022-05-31T00:00:00+00:00Seth Jensen
<p><img src="/blog/2022/05/backing-up-your-saas-data-with-google-takeout/banner.webp" alt="A concrete building with rectangular windows at sunset"></p>
<!-- Photo by Seth Jensen -->
<p>Keeping backups is extremely important.</p>
<p>Losing important files can feel like a far-off problem, but the chance of misplacing a drive, theft, drive failure, accidental deletion, house fire, flood, etc., is much greater than we may think. The benefits outweigh the cost of backups, for files that matter at all. So everyone should make regular backups of data that they care about and that can’t be replaced.</p>
<p>Even among people who regularly make backups, there is one area many of us neglect: all of that data on various online services, also called software as a service or SaaS: Google Drive, Apple iCloud, Microsoft OneDrive, Dropbox, Box, etc.</p>
<p>It’s true, the most volatile files are the ones sitting in a single location on your laptop or thumb drive, not those on Google, WordPress, or iCloud servers. The danger of losing files is not nearly as present with SaaS. You can’t drop Google on the floor and lose a couple terabytes of data, like you can a hard drive, but you can be locked out of your account, accidentally delete files, and lose data by missing a notice about a service shutting down. Not to mention the possibility that your SaaS provider is hacked and loses your data.</p>
<p>I recently realized I had about five years of photos, nine years of Google Drive “stuff,” and who knows what else, not backed up from my Google account. I decided to download it all with Google Takeout, the service Google provides for making account backups.</p>
<h3 id="google-takeout">Google Takeout</h3>
<p>Google introduced Takeout in 2011 as a service to export and download your data stored in Google products. It seems like the perfect option to easily back up all your files from Google’s servers. But how is the process, and how useful is the actual downloaded data?</p>
<p><a href="https://support.google.com/accounts/answer/3024190?hl=en">Downloading from Takeout</a> is quite painless. You select the services you want to back up, the formats you want them in (when there are multiple options), and start an export. When it’s ready, you get an email linking to a compressed file containing your data. You can export either to a <code>.zip</code> archive or a <code>.tgz</code> (<code>.tar.gz</code>) archive. Zip is more universally accessible, so if you don’t have or want extra software (such as 7-Zip), it is probably the better option.</p>
<p>One of the hardest things about keeping SaaS backups, even when they are easy to manage, is just remembering to do it. Backups become less useful if they’re six months or even years out of date.</p>
<p>Fortunately, Google Takeout has an option to automatically create a new full export 6 times over the course of a year and email you a download link. I’m not great at remembering to renew backups, so I’m letting them email me every two months with a new export. This is a more important feature than I originally thought, as researching for this post dragged on over nearly two months, despite feeling like I had very recently backed my files up.</p>
<p>One concern I’ve had with downloading SaaS data is whether it could be used in other apps, or imported again. I would prefer that my backups aren’t buried among thousands of lines of cruft. So I’d like to dive into the data and get a feel for how useful it would be if I actually needed it.</p>
<h3 id="how-useful-is-the-exported-data">How useful is the exported data?</h3>
<p>Most of Google Takeout’s data is reasonably well organized and usable — YouTube videos in MP4, calendars in ICS, photos in JPEG sorted by year as well as albums you’ve created. But there is plenty of inefficiency, and a few gotchas you need to watch for.</p>
<p>For some services, like Google Drive, you can select between a common usable format (DOCX for Docs, XLSX for sheets, etc) and a PDF render. There’s not a ton of variety in choices, but the formats are generally common enough that you could easily open them in your program of choice.</p>
<p>My download came in three <code>.tgz</code> files, two around 50GB and one around 5GB. That’s not awful, despite some odd choices on Google’s part for space management. For example, Google Keep exports in a nice, usable JSON format, but also in HTML with a huge <code>style</code> tag. My JSON takes 500KB, while the HTML takes 1500KB. Luckily they don’t seem to do this with YouTube, Google Photos, or Drive, or else I would be more concerned about bloat.</p>
<p>Some exports are somewhere in the middle, like YouTube playlists, which only include the ID of the YouTube video. Helpful in the case of losing your account, but not so much if any of those videos are deleted from YouTube.</p>
<h3 id="what-does-takeout-exclude">What does Takeout exclude?</h3>
<p>Takeout includes most of the data you would care about: Gmail (in MBOX format, including attachments), Google Photos, Blogger, saved Maps places, your uploaded YouTube videos, etc. You can see a full list of services and formats in <a href="https://takeout.google.com/settings/takeout">Takeout itself</a>.</p>
<p>A major flaw with Takeout is that it only backs up data you are the owner of. That means that, for example, if a co-worker creates a document with nothing but a title and invites you to help work on it, you may have added dozens of pages of painfully earned writing, but Takeout doesn’t consider it yours, so it won’t get exported. You have to manually save your own copy, separately, for every shared file.</p>
<p>Sharing is one of the most useful things about Google’s SaaS options, so not having shared files backed up could largely defeat the purpose of the backup. You can download shared files separately, but any added effort to make a complete backup quickly becomes a hassle, and defeats part of the purpose of using Takeout.</p>
<p>Be on the lookout for shared files that might not be backed up. For me, that’s mostly Drive, Photos, and Calendar, but pay attention to other shared files you may want backed up.</p>
<p>While the data exports aren’t perfect, the potential loss is so great that finding some way to back up SaaS data is a no-brainer.</p>
<p>Signing off with an image I found in my Blogger backup: my initials, created using the GIMP, circa 2009.</p>
<p><img src="/blog/2022/05/backing-up-your-saas-data-with-google-takeout/srj.jpg" alt="S. R. J. initials with artificial sun background"></p>
Introduction to BorgBackuphttps://www.endpointdev.com/blog/2020/09/introduction-to-borg-backup/2020-09-10T00:00:00+00:00Kannan Ponnusamy
<p><img src="/blog/2020/09/introduction-to-borg-backup/image-1.jpg" alt="Black and silver hard drive"></p>
<p><a href="https://unsplash.com/photos/ShHkXuZdpTw">Photo</a> by <a href="https://unsplash.com/@frank041985">Frank R</a></p>
<h3 id="what-is-borg">What is Borg?</h3>
<p><a href="https://www.borgbackup.org/">BorgBackup</a> (Borg for short) is a ‘deduplicating’ backup program that eliminates duplicate or redundant information. It optionally supports compression and authenticated encryption.</p>
<p>The main objective of Borg is to provide an efficient and secure way to backup data. The deduplication technique utilized to produce the backup process is very quick and effective.</p>
<h4 id="step-1-install-the-borg-backups">Step 1: Install the Borg backups</h4>
<p>On Ubuntu/Debian:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">apt install borgbackup
</code></pre></div><p>On RHEL/CentOS/Fedora:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">dnf install borgbackup
</code></pre></div><h4 id="step-2-initialize-local-borg-repository">Step 2: Initialize Local Borg repository</h4>
<p>Firstly, the system that is going to be backed up needs a new designated backup directory. Name the parent directory ‘backup’ and then create a child directory called ‘borgdemo’, which serves as the repository.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mkdir -p /mnt/backup
borg init --encryption=repokey /mnt/backup/borgdemo
</code></pre></div><h4 id="step-3-lets-create-the-first-backup-archive">Step 3: Let’s create the first backup (archive)</h4>
<p>In Borg terms, each backup instance will be called an archive. The following demonstrates how to backup the ‘photos’ directory and designate the archive as ‘archive_1’.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg create --stats --progress /mnt/backup/borgdemo::archive_1 /home/kannan/photos
</code></pre></div><p>Note: the archive label for each backup run needs to be specified.</p>
<h4 id="step-4-next-backup-incremental">Step 4: Next backup (Incremental)</h4>
<p>In order to see if the run was successful, the same command will be executed again. However, this time, with the different unique archive label.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg create --stats --progress /mnt/backup/borgdemo::archive_2 /home/kannan/photos
</code></pre></div><p>The following backup is mostly identical to the previous one. Because of deduplication, the process will not only run faster this time, it will be incremental as well. The <code>--stats</code> flag will provide statistics regarding the size of deduplication.</p>
<h4 id="step-5-list-all-the-archives">Step 5: List all the archives</h4>
<p>The ‘borg list’ command lists all of the archives stored within the Borg repository.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg list /mnt/backup/borgdemo
</code></pre></div><h4 id="step-6-remote-borg-repository">Step 6: Remote Borg Repository</h4>
<p>Take the scenario where the backups of many servers need to be maintained in a separate server. In this instance, a directory needs to be created for each of the systems that will be backed up. For this backup repository, create a folder named ‘backup’, and then within ‘backup’ a folder called ‘linode_01’. This folder will be initialized as a Borg repository.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mkdir -p /mnt/backup/linode_01
borg init --encryption=repokey user@backup_server:/mnt/backup/linode_01
</code></pre></div><p>The username, backup_server, repo can of course all be customized at the user’s discretion.</p>
<p>While initialising the repo, a passphrase for each backup repository can be set for authentication.</p>
<h4 id="step-7-create-an-initial-backup-to-the-remote-borg-repository">Step 7: Create an initial backup to the remote Borg repository</h4>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg create --stats ssh://user@backup_server/mnt/backup/linode_01::archive_1 /home/kannan/photos
</code></pre></div><p>To enable the remote backups, the following three environment variables can be used to simplify the automation process:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#038">export</span> <span style="color:#369">BORG_REPO</span>=<span style="color:#d20;background-color:#fff0f0">'ssh://user@backup_server/mnt/backup/linode_01'</span>
<span style="color:#038">export</span> <span style="color:#369">BORG_PASSPHRASE</span>=<span style="color:#d20;background-color:#fff0f0">'set_your_passpharase'</span>
<span style="color:#038">export</span> <span style="color:#369">BORG_RSH</span>=<span style="color:#d20;background-color:#fff0f0">'ssh -i /home/kannan/.ssh/id_rsa_backups'</span>
</code></pre></div><p>With those environment variables set, the ‘borg create’ command can be shortened to the following:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg create --stats ::archive_1 /home/kannan/photos
</code></pre></div><h4 id="step-8-excluding-certain-directories-or-files">Step 8: Excluding certain directories or files</h4>
<p>In order to exclude certain directories or files, the create command has an <code>--exclude</code> option or an exclude file/directory pattern can be generated. For example, the following command demonstrates how to exclude <code>/dev and /opt</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg create --stats ::archive_1 / --exclude /dev /opt
</code></pre></div><h4 id="step-9-restoring-an-archive-through-extraction">Step 9: Restoring an archive through extraction</h4>
<p>The ‘borg extract’ command extracts the contents of an archive. As a preset default, the entire archive will be extracted. However, the extraction can be limited by passing the directory path or file path as arguments to the command. For example, this is how a single photo can be extracted from the Photos archive:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg extract ::archive_1 /home/kannan/photos/sunrise.jpg
</code></pre></div><h4 id="step-10-pruning-older-backups">Step 10: Pruning older backups</h4>
<p>Every backup solution should have a way to maintain the older backups. Borg offers us <code>borg prune</code> for this. It prunes a repository by deleting all archives not matching any of the specified retention options.</p>
<p>For example, retain the final 10 archives from the day, another 6 end-of-week archives, and 3 of the end-of-month archive for every month using the following syntax:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">borg prune -v --list --keep-daily=<span style="color:#00d;font-weight:bold">10</span> --keep-weekly=<span style="color:#00d;font-weight:bold">6</span> --keep-monthly=<span style="color:#00d;font-weight:bold">3</span> ::
</code></pre></div><p>Note that the double colons <code>::</code> are required in order to automatically use the environment variables that were set prior.</p>
<p>For more in-depth documentation on Borg backup, <a href="https://borgbackup.readthedocs.io/en/stable/quickstart.html">read the docs</a>.</p>
End Point Security Tips: Securing your Infrastructurehttps://www.endpointdev.com/blog/2020/02/end-point-security-tips/2020-02-05T00:00:00+00:00Charles Chang
<img src="/blog/2020/02/end-point-security-tips/image-4.jpg" alt="phishing key on keyboard">
<p><a href="https://flic.kr/p/24YXTiY">Photo</a> from <a href="https://www.comparitech.com/blog/information-security/common-phishing-scams-how-to-avoid/">comparitech.com</a></p>
<h3 id="implement-security-measures-to-protect-your-organization--employees">Implement Security Measures to Protect Your Organization & Employees</h3>
<p>In this post, I’ll address what I believe are the three important initiatives every organization should implement to protect your organization and employees:</p>
<ol>
<li>Train employees on security culture.</li>
<li>Implement the best technical tools to aid with organizational security.</li>
<li>Implement recovery tools in case you need to recover from a security breach.</li>
</ol>
<h3 id="habits-of-a-security-culture">Habits of a Security Culture</h3>
<p>Train everyone in your organization on these fundamentals:</p>
<ol>
<li>The only time you should be requested to reset your password by email is when you initiate it. There are rare exceptions to this rule, such as when accounts are compromised and providers request all users reset their passwords, but those events should be publicly announced. Staff can confirm with security personnel before acting on such requests.</li>
<li>If you are asked to reset your password, it will typically be after you successfully logged into a website and the old one has expired.</li>
<li>If you receive an email and do not know the sender, do not trust the contents or open attachments. Get advice from security personnel if needed.</li>
<li>If you think the email is from your bank, keep in mind that banks do not ask their clients for private information via email.</li>
<li>If you think the social security office emailed you to obtain your personal information, keep in mind that they do not initiate or solicit private information via email.</li>
<li>Companies should not solicit private information unless you initiate first.</li>
<li>Online retailers should not ask for your private information unless you initiate first.</li>
</ol>
<h3 id="a-security-concern-going-phishing">A Security Concern: Going Phishing!</h3>
<div style="float: right; padding: 20px;"><img src="/blog/2020/02/end-point-security-tips/image-1.jpg" alt="phishing fraud" align="right" hspace="10"><p><a href="https://flic.kr/p/2gLaNqk">Photo</a> by <a href="https://www.epictop10.com/">Epic Top 10 Site</a></div>
<p>One of the more common ways to steal someone’s private information is through phishing. Phishing is like fishing: you try to catch something. In this case, the ‘fish’ is your data. Someone with malicious intent sends you email to attempt to get you to click on the link, picture, content, etc. within the email. Once you click the link or content within the email, it might take you to a website to enter or reset your password, even ask for your social security number or other personal information. The person with malicious intent would use the information collect to open accounts, purchase items online or resell your personal data. The links within the phishing email might even redirect you to a fake website that mimics a real website to collect your personal data. The goal is to confuse the email recipient into believing that the email message is legit in its attempt to collect personal information from the user.</p>
<h4 id="phishing-exercises">Phishing Exercises</h4>
<div style="float: left; padding: 20px;"><img src="/blog/2020/02/end-point-security-tips/image-2.jpg" alt="phishing attack" hspace="10"><p><a href="https://flic.kr/p/x9zZ4A">Photo</a> by <a href="https://www.flickr.com/photos/christiaancolen/">Christiaan Colen</a></p></div>
<p>One way to help the staff better their understanding of a phishing email is to practice phishing exercises by setting up an experiment to see which users would click on your test phishing email. If the user clicks on the experiment phishing email, the email administrator would notify the compliance officer and he or she would re-train the staff on how to properly differentiate real emails from phishing emails. Phishing exercises make a great activity for the employees to avoid the email scams and exert more caution in the future.</p>
<h3 id="essential-security-tools">Essential Security Tools</h3>
<h4 id="firewall">Firewall</h4>
<p>A firewall should be a mandatory technology for all consumers or businesses. This is basically your main door. The office or organization’s technology is what the firewall is protecting. When you sit in your office working on the computer, just imagine that you are in a fort surrounded by walls. If someone needs to come in, they must be given permission by a gatekeeper to enter the fort. The firewall is very similar. The firewall allows network traffic to go in and out of the office based on the configuration set by a network engineer. The engineer determines what network traffic comes into the office and goes out.</p>
<p>Companies typically test their firewall with penetration tests and network scans to determine if there are any security concerns after implementing the firewall. The testing of the firewall verifies that good security practices are implemented and the firewall is setup properly and securely.</p>
<p>Some hardware firewall devices to consider include:</p>
<ul>
<li><a href="https://www.watchguard.com/wgrd-products/rack-mount/firebox-m270-m370" target="_blank">WatchGuard</a></li>
<li><a href="https://www.cisco.com/c/en/us/products/security/firewalls/index.html" target="_blank">Cisco Firepower</a></li>
<li><a href="https://www.ui.com/unifi-routing/unifi-security-gateway-pro-4/" target="_blank">UniFi Security Gateway Pro</a></li>
</ul>
<h4 id="network-assessment--internal-threat-protection">Network Assessment & Internal Threat Protection</h4>
<p>Why is security vulnerability testing necessary? Many organizations have legacy servers, or desktops/laptops with operating systems that are no longer supported. For example, outdated Microsoft Windows XP and 7 can be compromised by malware while browsing the Internet due to <a href="https://www.pcworld.com/article/3400698/nsa-warns-that-bluekeep-vulnerability-in-windows-xp-and-windows-7-is-especially-dangerous.html">the BlueKeep vulnerability</a>. Systems that are not patched could expose your organization to be infected with malware such as ransomware which would hold your data ransom until you pay to unlock the data.</p>
<p>Security vulnerability testing typically results in a report outlining problematic areas such as outdated operating systems, private data that does not belong on a file server, such as social security or credit card number (personally identifying information, or PII). If PII is needed for the organization to operate, then higher security standards and an encryption system to store the private data is needed. The security vulnerabilities testing could also check if your environment is HIPAA or PCI DSS compliant if those security standards apply to you.</p>
<p>Some security vulnerability testing technology includes:</p>
<ul>
<li><a href="https://www.rapidfiretools.com/products/network-detective/" target="_blank">Rapid Fire Tools Network Detective</a></li>
<li><a href="https://www.rapidfiretools.com/products/cyber-hawk/" target="_blank">Rapid Fire Tools Cyberhawk</a></li>
</ul>
<h4 id="enterprise-antivirus-systems">Enterprise Antivirus Systems</h4>
<p>Antivirus software is the last line of defense if malware enters your computer. The antivirus itself would not protect you 100% from being infected by malware or virus but if you have multiple layers of security in place, then the chances are much lower that your organization’s systems would not be compromised. A company called <a href="https://www.av-comparatives.org/tests/business-security-test-2019-march-june/">AV-Comparatives assessed some of the popular antivirus software</a> in the market.</p>
<p>The battle with malware is endless. Case in point: The <a href="https://techcrunch.com/2019/05/12/wannacry-two-years-on/">WannaCry</a> malware affected over 200,000 machines across the world and spread quickly. Security researchers quickly realized the malware was spreading like a computer worm, across computers and over the network, using the Windows SMB protocol.</p>
<p>New variants of viruses and malware are developed every day, so the antivirus companies are also daily hard at work developing a way to block and remove malware.</p>
<p>Some antivirus software includes:</p>
<ul>
<li><a href="https://www.webroot.com/us/en/business/smb/endpoint-protection" target="_blank">Webroot Antivirus</a></li>
<li><a href="https://www.broadcom.com/products/cyber-security/endpoint/end-user" target="_blank">Symantec Endpoint Security</a></li>
</ul>
<h4 id="web-filtering-technology">Web Filtering Technology</h4>
<p>Web filtering technology blocks websites that are malicious or deemed not appropriate to visit from an organization’s network. For example, websites for gambling could be blocked to reduce employee distractions, but also to reduce access to sites popularly infected with malware, reducing the possibility of malware coming into your network.</p>
<p>There are many competitors out there in this competitive market, and some vendors offer free proof-of-concept testing with their product before you make a big investment. Take a look at:</p>
<ul>
<li><a href="https://www.forcepoint.com/product/url-filtering" target="_blank">Forcepoint Web Filtering</a></li>
<li><a href="https://www.webtitan.com/webtitan-gateway/" target="_blank">Web Titan Gateway</a></li>
</ul>
<h4 id="data-loss-prevention">Data Loss Prevention</h4>
<p>Employees’ and businesses’ private information is sensitive and should be protected. Businesses, whether audited or not, should always protect their employee private information. 20 years ago paper was used to store private information and locked in a file cabinet, but in 2020 most private information is stored digitally. How do companies keep private information from leaving their office?</p>
<p>Physically you probably can’t stop someone from walking out with private information, but digitally there is technology called digital loss prevention (DLP) that can help keep confidential information from leaving the office. For example, if someone in the office decides to copy private information onto USB storage, or tries to send a social security number or credit card information via email, this can often be prevented. Prepare your business with solutions tailored for regulations involving personal information using DLP software.</p>
<p>Some DLP solutions to consider include:</p>
<ul>
<li><a href="https://www.forcepoint.com/product/dlp-data-loss-prevention" target="_blank">Forcepoint DLP</a></li>
<li><a href="https://www.broadcom.com/products/cyber-security/information-protection/data-loss-prevention" target="_blank">Symantec Data Loss Prevention</a></li>
</ul>
<h4 id="email-filtering">Email Filtering</h4>
<p>Probably one of the easiest and oldest methods to infect or phish someone is via email. There are multiple mechanisms email filtering uses to stop malicious emails: SPF and DKIM, blacklists, etc. On top of these configurable items, email filtering software vendors often release updates throughout the day to block the latest known malware or spam based on heuristics.</p>
<p>Cloud email services such as Microsoft Office 365 are for many businesses superior to on-premise email servers due to having all the bells and whistles to proactively protect your email environment, such as spam & virus filtering, and email archiving for retention purposes. If you need email encryption to protect sensitive emails, this feature is also available.</p>
<p>The distinct advantage of using hosted email service is that the cost is predictable and includes maintenance, so your email administrators do not have to worry about updating the email server software or hardware.</p>
<p>Email filtering technology available includes:</p>
<ul>
<li><a href="https://www.microsoft.com/en-us/security/business/threat-protection/office-365-defender" target="_blank">Microsoft Defender for Office 365</a></li>
<li><a href="https://www.forcepoint.com/product/email-security" target="_blank">Forcepoint Email Filtering</a></li>
<li><a href="https://www.spamtitan.com/email-filtering-service/" target="_blank">SpamTitan</a></li>
</ul>
<h4 id="two-factor-authentication-2fa">Two Factor Authentication (2FA)</h4>
<p>Companies like Duo (owned by Cisco) and Google’s two-factor authentication technology are great tools to implement to improve your overall security. Beyond the usual user name and password, your smartphone or a hardware token are used to authorize the access to a website, an application, or a network. This technology is now well-established and available in most online services, and can be added to your own custom business applications.</p>
<p>Some starting points:</p>
<ul>
<li><a href="https://duo.com/product/multi-factor-authentication-mfa" target="_blank">Duo Multi-Factor Authentication</a></li>
<li><a href="https://www.google.com/landing/2step/" target="_blank">Google Authenticator</a></li>
</ul>
<h4 id="vpn-virtual-private-network">VPN (Virtual Private Network)</h4>
<p>Virtual private network technology allows businesses to provide secure access to office systems or applications to employees who travel or work from a remote location. That is important so that data traveling to and from the source, and many details of traffic patterns, are encrypted, so that the data cannot be captured and decrypted. VPN solutions have been around for years and are an important tool to securely and safely protect data in transit.</p>
<h4 id="password-reset-systems">Password Reset Systems</h4>
<p>Why are self-service password reset systems necessary, and are they secure? They eliminate many manual password assignment mistakes, keep passwords private to the user alone, and allow an organization to integrate two-factor authentication to securely reset user passwords, such as Active Directory for Windows, or other single sign-on password. Onboarding is needed for each user, which is done by sending them an email with a link to register.</p>
<p>The password reset system could be an internal system only available to your organization. Another possibility is to access the system via VPN or even through a proxy server in the DMZ to allow password reset from anywhere.</p>
<p>Some password reset systems include:</p>
<ul>
<li><a href="https://www.manageengine.com/products/self-service-password/" target="_blank">ManageEngine ADSelfService Plus</a></li>
<li><a href="https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-sspr-howitworks" target="_blank">Azure AD Self-Service Password Reset</a></li>
</ul>
<h3 id="recovery-tools">Recovery Tools</h3>
<h4 id="data-and-system-backupsdisaster-recovery">Data and System Backups/Disaster Recovery</h4>
<p>Data backup and disaster recovery practice is a critical component for a business to speed up the process of recovery if malware attacks your systems or if a system becomes inoperable.</p>
<p>If you are attacked by ransomware and all the critical systems and desktops are compromised and rendered useless, then the only way to recover is from backups. At that point, the disaster recovery preparation you made will be well worth what you invested into it. If you did not have an adequate backup or disaster recovery plan, your organization or company would have to start from scratch and rebuild all systems and desktops which could take weeks if not months to recover from. The lost wages, and possibly clients, due to unavailable systems and desktop to operate, probably would cost you far more than preparation does. The more options you have, the better the chances that you will get out of your bind and not prolong the situation.</p>
<p>Some backup solutions I have worked with have the flexibility to mix and match to fit your configuration needs:</p>
<ul>
<li><a href="https://www.acronis.com/en-us/products/cloud/cyber-protect/backup/" target="_blank">Acronis Backup Solution</a></li>
<li><a href="https://www.rubrik.com/solutions/backup-recovery" target="_blank">Rubrik</a></li>
<li><a href="https://azure.microsoft.com/en-us/services/backup/" target="_blank">Azure Backup</a></li>
</ul>
<p>Acronis Cloud Backup allows you to back up to local storage, the cloud, and off-premise, recover a system using a USB drive, back up Office 365 emails, or backup VMware or Hyper-V environments. It also allows you to recover a single file if needed.</p>
<h3 id="additional-security-recommendations">Additional Security Recommendations</h3>
<h4 id="ssltls-certificates">SSL/TLS Certificates</h4>
<p>What are SSL and TLS? SSL is outdated and replaced by TLS, but people often still use the familiar SSL name. They are an encryption system to keep sensitive information sent across the Internet encrypted so that only the intended recipient can access it.</p>
<p>When an SSL certificate is validated, the transmitted data is not only unreadable by anyone except for the intended server, but you are relatively well assured you are communicating with the organization you expected and not with a malicious intermediary. Read more details in <a href="https://www.sslshopper.com/why-ssl-the-purpose-of-using-ssl-certificates.html">SSL Shopper’s Why SSL?</a> article.</p>
<p>SSL certificates were historically used primarily for higher-security web servers, but now are commonly used on almost all web servers. They are also used for other remote server access, B2B server-to-server communication, VPN access, email systems, etc.</p>
<h3 id="recent-destructive-incidents">Recent Destructive Incidents</h3>
<p>These incidents from just a few weeks in fall 2019 show that we have a long way to go in protecting our technology usage:</p>
<ol>
<li><a href="https://www.nbcnewyork.com/news/local/long-island-schools-hacked-district-forced-to-pay-88000-in-ransom/1491924/" target="_blank">Long Island Schools Hacked; District Forced to Pay $88,000 in Ransom</a></li>
<li><a href="https://www.nbcnewyork.com/news/local/ny-school-delays-start-of-year-after-ransomware-attack/1990459/" target="_blank">NY School Delays Start of Year After Ransomware Attack</a></li>
<li><a href="https://www.newsweek.com/tortoiseshell-hacker-hire-military-heroes-fake-website-1461320" target="_blank">Veterans Targeted by Hackers Through Fake Military Heroes Hiring Website</a></li>
<li><a href="https://www.zdnet.com/article/hackers-target-transportation-and-shipping-industries-in-new-trojan-malware-campaign/" target="_blank">Hackers Target Transportation and Shipping Companies in New Trojan Malware Campaign</a></li>
<li><a href="https://inc42.com/buzz/hacking-routers-webcams-printers-are-the-most-searched-keywords-on-the-dark-web/" target="_blank">Hacking of IoT – Internet of Things (webcams, security cams, printers, home routers)</a></li>
</ol>
<h3 id="speak-with-us">Speak With Us!</h3>
<p>Keeping up with security can be a full-time job. If you need professional consulting on security tools and implementing them, <a href="/contact/">contact</a> our team today.</p>
Defense in Depthhttps://www.endpointdev.com/blog/2012/09/defense-in-depth/2012-09-28T00:00:00+00:00Zed Jensen
<p>“Defense in depth” is a way to build security systems so that when one layer of defense fails, there is another to take its place. If breaking in is hard enough for an attacker, it’s likely that they’ll abandon their assault, deciding it’s not worth the effort. Making the various layers different types of defense also makes it harder to get in, so that an attacker is less likely to get through all the layers. It can also keep one mistake from causing total security failure.</p>
<p>For example, if you have an office with one door and leave it unlocked by accident, then anyone can just walk in. However, if you have an office building with a main door, then a lobby and hallways, and then additional inner locked doors, then the chance of accidentally leaving both unlocked is small. If you accidentally leave the inner door open or unlocked, someone can only get into the office if they first get through the outer door.</p>
<p>Another example of defense in depth is making and maintaining offsite computer backups. The chance of an office being destroyed and all your data with it is low, but not zero. If you maintain offsite backups, then your losses in a catastrophe are reduced.</p>
<p>Another way to decrease risk of data loss is to keep multiple backups in widely separated locations, held by different people or companies. This ensures you still have access to data even when one account is locked out, a storage facility goes out of business, or an account is hijacked by an attacker.</p>
<p>What about the rogue insider threat? You can protect each backup with different credentials, and ensure few or no employees have access to more than one backup.</p>
<p>A security scheme is only as strong as its weakest link, so your various defenses need to be on par with each other. If your office is locked up nice and tight, but anyone can get into your computer over the network, then all your office security isn’t going to do much good and vice versa.</p>
<p>When considering security, it’s important to think about the cost compared to the potential loss. Picture a house. With a normal house, it’s not too hard to get in; all you have to do is break a window. Considering that, is it really worth spending lots of money on fancy locks for the doors? You can add bars to the windows and install multiple fancy door locks, but is it worth the inconvenience and ugliness? Perhaps moving to a new neighborhood is a better solution.</p>
<p>Recognizing that no defense is foolproof, and that a combination of cheap defenses may be more effective than one expensive defense, consider the concept of defense in depth whenever looking at protection, whether it be of an office building, website, or castle.</p>
<p>These articles go into more detail on the topic:</p>
<ul>
<li><a href="http://en.wikipedia.org/wiki/Defense_in_depth_(computing)">http://en.wikipedia.org/wiki/Defense_in_depth_(computing)</a></li>
<li><a href="https://www.owasp.org/index.php/Defense_in_depth">https://www.owasp.org/index.php/Defense_in_depth</a></li>
<li><a href="http://www.nsa.gov/ia/_files/support/defenseindepth.pdf">http://www.nsa.gov/ia/_files/support/defenseindepth.pdf</a></li>
</ul>
Restoring individual table data from a Postgres dumphttps://www.endpointdev.com/blog/2010/04/restoring-individual-table-data-from/2010-04-20T00:00:00+00:00Greg Sabino Mullane
<p>Recently, one of our clients needed to restore the data in a specific table from the previous night’s PostgreSQL dump file. Basically, there was a UPDATE query that did not do what it was supposed to, and some of the columns in the table were irreversibly changed. So, the challenge was to quickly restore the contents of that table.</p>
<p>The SQL dump file was generated by the <a href="https://www.postgresql.org/docs/current/static/app-pg-dumpall.html">pg_dumpall</a> command, and thus there was no easy way to extract individual tables. If you are using the <a href="https://www.postgresql.org/docs/current/static/app-pgdump.html">pg_dump</a> command, you can specify a “custom” dump format by adding the <strong>-Fc</strong> option. Then, pulling out the data from a single table becomes as simple as adding a few flags to the <a href="https://www.postgresql.org/docs/current/static/app-pgrestore.html">pg_restore</a> command like so:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ pg_restore --data-only --table=alpha large.custom.dumpfile.pg > alpha.data.pg
</code></pre></div><p>One of the drawbacks of using the custom format is that it is only available on a per-database basis; you cannot use it with pg_dumpall. That was the case here, so we needed to extract the data of that one table from within the large dump file. If you know me well, you might suspect at this point that I’ve written yet another handy perl script to tackle the problem. As tempting as that may have been, time was of the essence, and the wonderful array of Unix command line tools already provided me with everything I needed.</p>
<p>Our goal at this point was to pull the data from a single table (“alpha”) from a very large dump file (“large.dumpfile.pg”) into a separate and smaller file that we could use to import directly into the database.</p>
<p>The first step was to find exactly where in the file the data was. We knew the name of the table, and we also know that a dump file inserts data by using the <a href="https://www.postgresql.org/docs/current/static/sql-copy.html">COPY</a> command, so there should be a line like this in the dump file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">COPY</span><span style="color:#bbb"> </span>alpha<span style="color:#bbb"> </span>(a,b,<span style="color:#080;font-weight:bold">c</span>,d)<span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">stdin</span>;<span style="color:#bbb">
</span></code></pre></div><p>Because all the COPYs are done together, we can be pretty sure that the command after “COPY alpha” is another copy. So the first thing to try is:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -n COPY large.dumpfile.pg | grep -A1 <span style="color:#d20;background-color:#fff0f0">'COPY alpha '</span>
</code></pre></div><p>This uses grep’s handy <strong>-n</strong> option (aka <strong>–line-number</strong>) to output the line number that each match appears on. Then we pipe that back to grep, search for our table name, and print the line after it with the <strong>-A</strong> option (aka <strong>–after-context</strong>). The output looked like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -n COPY large.dumpfile.pg | grep -A1 <span style="color:#d20;background-color:#fff0f0">'COPY alpha '</span>
1233889:COPY alpha (cdate, who, state, add, remove) FROM stdin;
12182851:COPY alpha_sequence (sname, value) FROM stdin;
</code></pre></div><p>Note that many of the options here are GNU specific. If you are using an operating system that doesn’t support the common GNU tools, you are going to have a much harder time doing this (and many other shell tasks)!</p>
<p>We now have a pretty good guess at the starting and ending lines for our data: 1233889 to lines 12182850 (we subtract 1 as we don’t want the next COPY). We can now use <strong>head</strong> and <strong>tail</strong> to extract the lines we want, once we figure out how many lines our data spans:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ <span style="color:#038">echo</span> <span style="color:#00d;font-weight:bold">12182851</span> - <span style="color:#00d;font-weight:bold">1233889</span> | bc
<span style="color:#00d;font-weight:bold">10948962</span>
$ head -1233889 large.dumpfile.pg | tail -10948962 > alpha.data.pg
</code></pre></div><p>However, what if the next command was not a COPY? We’ll have to scan forward for the end of the COPY section, which is always a backslash and a single dot at the start of a new line. The new command becomes (all one line, but broken down for readability):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -n COPY large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | grep -m1 <span style="color:#d20;background-color:#fff0f0">'COPY alpha'</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | cut -d: -f1 <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | xargs -Ix tail --lines=+x large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | grep -n -m1 <span style="color:#d20;background-color:#fff0f0">'^\\\.'</span>
</code></pre></div><p>That’s a lot, but in the spirit of Unix tools doing one thing and one thing well, it’s easy to break down. First, we grab the line numbers where COPY occurs in our file, then we find the first occurrence of our table (using the <strong>-m</strong> aka <strong>–max-count</strong> option). We cut out the first field from that output, using a colon as the delimiter. This gives is the line number where the COPY begins. We pass this to xargs, and tail the file with a <strong>–lines=+x</strong> argument, which outputs all lines from that file <em>starting</em> at the given line number. Finally, we pipe that output to grep and look for the end of copy indicator, stopping at the first one, and also outputting the line number. Here’s what we get:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -n COPY large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | grep -m1 <span style="color:#d20;background-color:#fff0f0">'COPY alpha'</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | cut -d: -f1 <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | xargs -Ix tail --lines=+x large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | grep -n -m1 <span style="color:#d20;background-color:#fff0f0">'^\\\.'</span>
148956:<span style="color:#04d;background-color:#fff0f0">\.</span>
xargs: tail: terminated by signal <span style="color:#00d;font-weight:bold">13</span>
</code></pre></div><p>This tells us that 148956 lines after the COPY, we encountered the string “.”. (The complaint from xargs can be ignored). Now we can create our data file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -n COPY large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | grep -m1 <span style="color:#d20;background-color:#fff0f0">'COPY alpha'</span> <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | cut -d: -f1 <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | xargs -Ix tail --lines=+x large.dumpfile.pg <span style="color:#04d;background-color:#fff0f0">\
</span><span style="color:#04d;background-color:#fff0f0"></span> | head -148956 > alpha.data.pg
</code></pre></div><p>Now that the file is there, we should do a quick sanity check on it. If the file is small enough, we could simply call it up in your favorite editor or run it through <strong>less</strong> or <strong>more</strong>. You can also check things out by knowing that a Postgres dump file separates the data in columns by a tab character when using the COPY command. So we can view all lines that don’t have a tab, and make sure there is nothing except comments and the <strong>COPY</strong> and <strong>.</strong> lines:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ grep -v -P <span style="color:#d20;background-color:#fff0f0">'\t'</span> alpha.data.pg
</code></pre></div><p>The grep option <strong>-P</strong> (aka <strong>–perl-regexp</strong>) instructs grep to interpret the argument (“backslash t” in this case) as a Perl regular expression. You could also simply input a literal tab there: on most systems this can be done with the <strong><ctrl-v><TAB></strong> key combination.</p>
<p>It’s time to replace that bad data. We’ll need to truncate the existing table, then COPY our data back in. To do this, we’ll create a file that we’ll feed to <strong>psql -X -f</strong>. Here’s the top of the file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ cat > alpha.restore.pg
<span style="color:#04d;background-color:#fff0f0">\s</span>et ON_ERROR_STOP on
<span style="color:#04d;background-color:#fff0f0">\t</span>iming
<span style="color:#04d;background-color:#fff0f0">\c</span> mydatabase someuser
BEGIN;
CREATE SCHEMA backup;
CREATE TABLE backup.alpha AS SELECT * FROM public.alpha;
TRUNCATE TABLE alpha;
</code></pre></div><p>From the top: we tell psql to stop right away if it encounters any problems, and then turn on the timing of all queries. We explicitly connect to the correct database as the correct user. Putting it here in the script is a safety feature. Then we start a new transaction, create a backup schema, and make a copy of the existing data into a backup table before truncating the original table. The next step is to add in the data, then wrap things up:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ cat alpha.data.pg >> alpha.restore.pg
</code></pre></div><p>Now we run it and check for any errors. We use the <strong>-X</strong> argument to ensure control of exactly which psql options are in effect, bypassing any psqlrc files that may be in use.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ psql -X -f alpha.restore.pg
</code></pre></div><p>If everything looks good, the final step is to add a COMMIT and run the file again:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">$ <span style="color:#038">echo</span> <span style="color:#d20;background-color:#fff0f0">"COMMIT;"</span> >> alpha.restore.pg
$ psql -X -f alpha.restore.pg
</code></pre></div><p>And we are done! All of this is a little simplified, as in real life there was actually more than one table to be restored, and each had some foreign key dependencies that had to be worked around, but the basic idea remains the same. (and yes, I know you can do the extraction in a Perl one-liner)</p>
PostgreSQL EC2/EBS/RAID 0 snapshot backuphttps://www.endpointdev.com/blog/2010/02/postgresql-ec2-ebs-raid0-snapshot/2010-02-23T00:00:00+00:00Jon Jensen
<p>One of our clients uses <a href="https://aws.amazon.com/">Amazon Web Services</a> to host their production application and database servers on EC2 with EBS (Elastic Block Store) storage volumes. Their main database is <a href="/expertise/postgresql/">PostgreSQL</a>.</p>
<p>A big benefit of Amazon’s cloud services is that you can easily add and remove virtual server instances, storage space, etc. and pay as you go. One known problem with Amazon’s EBS storage is that it is much more I/O limited than, say, a nice SAN.</p>
<p>To partially mitigate the I/O limitations, they’re using 4 EBS volumes to back a Linux software RAID 0 block device. On top of that is the xfs filesystem. This gives roughly 4x the I/O throughput and has been effective so far.</p>
<p>They ship WAL files to a secondary server that serves as warm standby in case the primary server fails. That’s working fine.</p>
<p>They also do nightly backups using pg_dumpall on the master so that there’s a separate portable (SQL) backup not dependent on the server architecture. The problem that led to this article is that extra I/O caused by pg_dumpall pushes the system beyond its I/O limits. It adds both reads (from the PostgreSQL database) and writes (to the SQL output file).</p>
<p>There are several solutions we are considering so that we can keep both binary backups of the database and SQL backups, since both types are valuable. In this article I’m not discussing all the options or trying to decide which is best in this case. Instead, I want to consider just one of the tried and true methods of backing up the binary database files on another host to offload the I/O:</p>
<ol>
<li>Create an atomic snapshot of the block devices</li>
<li>Spin up another virtual server</li>
<li>Mount the backup volume</li>
<li>Start Postgres and allow it to recover from the apparent “crash” the server had (since there wasn’t a clean shutdown of the database before the snapshot</li>
<li>Do whatever pg_dump or other backups are desired</li>
<li>Make throwaway copies of the snapshot for QA or other testing</li>
</ol>
<p>The benefit of such snapshots is that you get an exact backup of the database, with whatever table bloat, indexes, statistics, etc. exactly as they are in production. That’s a big difference from a freshly created database and import from pg_dump.</p>
<p>The difference here is that we’re using 4 EBS volumes with RAID 0 striped across them, and there isn’t currently a way to do an atomic snapshot of all 4 volumes at the same time. So it’s no longer “atomic” and who knows what state the filesystem metadata and the file data itself would be in?</p>
<p>Well, why not try it anyway? Filesystem metadata doesn’t change that often, especially in the controlled environment of a Postgres data volume. Snapshotting within a relatively short timeframe would be pretty close to atomic, and probably look to the software (operating system and database) like some kind of strange crash since some EBS volumes would have slightly newer writes than others. But aren’t all crashes a little unpredictable? Why shouldn’t the software be able to deal with that? Especially if we have Postgres make a checkpoint right before we snapshot.</p>
<p>I wanted to know if it was crazy or not, so I tried it on a new set of services in a separate AWS account. Here are the notes and some details of what I did:</p>
<p>1. Created one EC2 image:</p>
<p>Amazon EC2 Debian 5.0 lenny AMI built by Eric Hammond<br>
Debian AMI ID ami-4ffe1926 (x86_64)<br>
Instance Type: High-CPU Extra Large (c1.xlarge) — 7 GB RAM, 8 CPU cores</p>
<p>2. Created 4 x 10 GB EBS volumes</p>
<p>3. Attached volumes to the image</p>
<p>4. Created software RAID 0 device:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mdadm -C /dev/md0 -n <span style="color:#00d;font-weight:bold">4</span> -l <span style="color:#00d;font-weight:bold">0</span> -z max /dev/sdf /dev/sdg /dev/sdh /dev/sdi
</code></pre></div><p>5. Created XFS filesystem on top of RAID 0 device:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mkfs -t xfs -L /pgdata /dev/md0
</code></pre></div><p>6. Set up in /etc/fstab and mounted:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mkdir /pgdata
<span style="color:#888"># edit /etc/fstab, with noatime</span>
mount /pgdata
</code></pre></div><p>7. Installed PostgreSQL 8.3</p>
<p>8. Configured postgresql.conf to be similar to primary production database server</p>
<p>9. Created empty new database cluster with data directory in /pgdata</p>
<p>10. Started Postgres and imported a play database (from public domain census name data and Project Gutenberg texts), resulting in about 820 MB in data directory</p>
<p>11. Ran some bulk inserts to grow database to around 5 GB</p>
<p>12. Rebooted EC2 instance to confirm everything came back up correctly on its own</p>
<p>13. Set up two concurrent data-insertion processes:</p>
<ul>
<li>
<p>50 million row insert based on another local table (INSERT INTO … SELECT …), in a single transaction (hits disk hard, but nothing should be visible in the snapshot because the transaction won’t have committed before the snapshot is taken)</p>
</li>
<li>
<p>Repeated single inserts in autocommit mode (Python script writing INSERT statements using random data from /usr/share/dict/words piped into psql), to verify that new inserts made it into the snapshot, and no partial row garbage leaked through</p>
</li>
</ul>
<p>14. Started those “beater” jobs, which mostly consumed 2-3 CPU cores</p>
<p>15. Manually inserted a known test row and created a known view that should appear in the snapshot</p>
<p>16. Started Postgres’s backup mode that allows for copying binary data files in a non-atomic manner, which also does a CHECKPOINT and thus also a filesystem sync:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>pg_start_backup(<span style="color:#d20;background-color:#fff0f0">'raid_backup'</span>);<span style="color:#bbb">
</span></code></pre></div><p>17. Manually inserted a 2nd known test row & 2nd known test view that I don’t want to appear in the snapshot after recovery</p>
<p>18. Ran snapshot script which calls ec2-create-snapshot on each of the 4 EBS volumes—during first run, run serially quite slowly taking about 1 minute total; during second run, run in parallel such that the snapshot point was within 1 second for all 4 volumes</p>
<p>19. Tell Postgres the backup’s over:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb"> </span>pg_stop_backup();<span style="color:#bbb">
</span></code></pre></div><p>20. Ran script to create new EBS volumes derived from the 4 snapshots (which aren’t directly usable and always go into S3), using <code>ec2-create-volume --snapshot</code></p>
<p>21. Run script to attach new EBS volumes to devices on the new EC2 instance using <code>ec2-attach-volume</code></p>
<p>22. Then, on the new EC2 instance for doing backups:</p>
<ul>
<li><code>mdadm --assemble --scan</code></li>
<li><code>mount /pgdata</code></li>
<li>Start Postgres</li>
<li>Count rows on the 2 volatile tables; confirm that the table with the in-process transaction doesn’t show any new rows, and that the table getting individual rows committed to reads correctly</li>
<li><code>VACUUM VERBOSE</code> — and confirm no errors or inconsistencies detected</li>
<li><code>pg_dumpall</code> # confirmed no errors and data looks sound</li>
</ul>
<p>It worked! No errors or problems, and pretty straightforward to do.</p>
<p>Actually before doing all the above I first did a simpler trial run with no active database writes happening, and didn’t make any attempt for the 4 EBS snapshots to happen simultaneously. They were actually spread out over almost a minute, and it worked fine. With the confidence that the whole thing wasn’t a fool’s errand, I then put together the scripts to do lots of writes during the snapshot and made the snapshots run in parallel so they’d be close to atomic.</p>
<p>There are lots of caveats to note here:</p>
<ul>
<li>This is an experiment in progress, not a how-to for the general public.</li>
<li>The data set that was snapshotted was fairly small.</li>
<li>Two successful runs, even with no failures, is not a very big sample set. :)</li>
<li>I didn’t use Postgres’s point-in-time recovery (PITR) here at all—I just started up the database and let Postgres recover from an apparent crash. Shipping over the few WAL logs from the master collected during the pg_backup run <em>after</em> the snapshot copying is complete would allow a theoretically fully reliable recovery to be made, not just a practically non-failing recovery as I did above.</li>
</ul>
<p>So there’s more work to be done to prove this technique viable in production for a mission-critical database, but it’s a promising start worth further investigation. It shows that there <em>is</em> a way to back up a database across multiple EBS volumes without adding noticeably to its I/O load by utilizing the Amazon EBS data store’s snapshotting and letting a separate EC2 server offload the I/O of backups or anything else we want to do with the data.</p>