Implementing Azure Blob Storage in .NET 9

2025-06-02T00:00:00+00:00

Businesses keep moving toward scalable and cloud-based architectures. With this in mind, a client that was dealing with random errors in a .NET app when saving files locally on the web server decided to get rid of that process and replace it with an Azure Blob Storage implementation.

Why use Azure Blob Storage? It’s an efficient cloud object storage solution from Microsoft, designed to store unstructured data, optimized for storing and serving documents, media, logs, or binary data, especially in applications that expose this data through an API. The key features are high performance, redundancy, reliability, and scalability. There’s an SDK that we can use for easy integration and development, be it in .NET or other languages.

Let’s take a look at what that change involves. For this example, we will set up the integration in a .NET 9 application:

Install the NuGet package required to connect to Azure Blob Storage. We can do it with the dotnet CLI, or through the NuGet package manager.

dotnet add package Azure.Storage.Blobs

Then, we need to configure our connection in our appsettings.json file. We will use the connection string that Azure provides us when we create the new storage account.

{
  "AzureBlobStorage": {
    "ConnectionString": "{your-connection-string}",
    "ContainerName": "my-container"
  }
}

In Azure Blob Storage, containers are like folders that group blobs (or files) together within a storage account. In this setting, the ContainerName value tells our application which container inside that account will be used for storing and retrieving blobs.

Create a BlobService class to read and write to the container.

using Azure.Storage.Blobs;
using Microsoft.Extensions.Configuration;

public class BlobService
{
    private readonly BlobContainerClient _containerClient;

    /// <summary>
    /// Initializes a new instance of the <see cref="BlobService"/> class.
    /// </summary>
    /// <param name="configuration">Application configuration containing Azure Blob Storage settings.</param>
    public BlobService(IConfiguration configuration)
    {
        var connectionString = configuration["AzureBlobStorage:ConnectionString"];
        var containerName = configuration["AzureBlobStorage:ContainerName"];

        _containerClient = new BlobContainerClient(connectionString, containerName);
    }

    /// <summary>
    /// Uploads a file stream to Azure Blob Storage with the specified file name.
    /// </summary>
    /// <param name="fileName">The name of the file to be stored in blob storage.</param>
    /// <param name="fileStream">The stream representing the file content.</param>
    /// <returns>A task that represents the asynchronous upload operation.</returns>
    public async Task UploadFileAsync(string fileName, Stream fileStream)
    {
        var blobClient = _containerClient.GetBlobClient(fileName);
        await blobClient.UploadAsync(fileStream, overwrite: true);
    }

    /// <summary>
    /// Downloads a file from Azure Blob Storage.
    /// </summary>
    /// <param name="fileName">The name of the file to download from blob storage.</param>
    /// <returns>A task that returns a stream containing the blob's content.</returns>
    public async Task<Stream> DownloadFileAsync(string fileName)
    {
        var blobClient = _containerClient.GetBlobClient(fileName);
        var response = await blobClient.DownloadAsync();
        return response.Value.Content;
    }
}

Quite simple, isn’t it? Now, let’s see a way to easily expose file upload/download endpoints via ASP.NET Core minimal APIs. Introduced in .NET 6 and evolving further in .NET 9, minimal APIs provide a lightweight way to build HTTP APIs with a few lines of code. Instead of requiring full controller classes and attributes, minimal APIs let you define routes directly in your main Program.cs file, using simple lambda expressions. This makes them ideal for microservices, lightweight APIs, and internal tools where fast development and low overhead are essential.

For this example, we will create two endpoints that will use our BlobService class to communicate with Azure Blob Storage to save and retrieve files. In our Program.cs file, let’s add:

/// <summary>
/// Endpoint to upload a file to Azure Blob Storage.
/// </summary>
/// <param name="file">The uploaded file sent from the client (via multipart/form-data).</param>
/// <param name="blobService">Injected service used to handle blob storage operations.</param>
/// <returns>An HTTP 200 OK response when the upload succeeds.</returns>
app.MapPost("/upload", async (IFormFile file, BlobService blobService) =>
{
    using var stream = file.OpenReadStream();
    await blobService.UploadFileAsync(file.FileName, stream);
    return Results.Ok("File uploaded successfully");
});

/// <summary>
/// Endpoint to download a file from Azure Blob Storage.
/// </summary>
/// <param name="fileName">The name of the file to download (provided in the URL path).</param>
/// <param name="blobService">Injected service used to access blob storage.</param>
/// <returns>The file stream as a binary response with a download filename.</returns>
app.MapGet("/download/{fileName}", async (string fileName, BlobService blobService) =>
{
    var stream = await blobService.DownloadFileAsync(fileName);
    return Results.File(stream, "application/octet-stream", fileName);
});

Of course, we would want to configure authentication, add validations, and other security measures on these endpoints to prevent spammers from filling out our storage capacity 🙂. But the basics are there.

Also, there are recommended practices for any Azure integrations to consider, as we increase the robustness of our application:

Monitor and log all storage operations using Azure Application Insights. Integrate Application Insights into our .NET app to get detailed logging of the different HTTP calls to the Azure services.
Use Managed Identities: Instead of storing secrets in appsettings.json, we can assign a Managed Identity to our application service, and grant it the Storage Blob Data Contributor role at the storage account. That will allow the application to gain access to the container automatically, without storing credentials in our settings files.
Containers should be private by default. Avoid using “Blob” or “Container” public access levels, unless necessary. When public access is enabled, any person with a link to the container will be able to open or download the blob.

We will cover more details about using Managed Identities and Shared Access Signatures (SAS) for other Azure services in an upcoming blog post. Stay tuned!

Media erasure in the time of SSD

2020-12-10T00:00:00+00:00

Photo by Alex Fu from Pexels

How valuable is your data? Losing it to a third party is usually a business’s worst nightmare—and can cause legal or even criminal repercussions, depending on the drive’s contents and the business’s jurisdiction.

Every system adminstrator worth their salt knows that running “rm” (or equivalent delete operations) doesn’t actually remove data, it simply removes the file name from the filesystem and leaves the data in place on the disk.

When dealing with traditional storage, destroying (intentionally or otherwise) your data used to be relatively easy. A wise system admin could simply run:

shred /dev/sda

And be fairly certain of the result. A cautious one might run a demagnetizing wand over the drive. Only the most paranoid might destroy it physically.

The Age of SSDs

Nowadays, most servers have switched away from storing data on rotating metal or glass platters. Solid state drives, or SSDs, are faster, less prone to errors from physical impact, and generally more sought after.

SSDs have issues with speed if the drives are too full, and have a limited lifespan—only a certain number of write operations can be achieved. This is less of an issue with modern drives thanks to wear leveling built into the firmware of the drives. However, this leads to some issues as well.

Complicated systems introduce issues

Because SSDs manage which blocks of storage they write to, a simple shred won’t do. There could be hundreds of bytes, or even kilobytes or megabytes, of data that the shred doesn’t reach.

Even some “traditional” storage can run into such issues these days. Hybrid drives offer some speed advantages: By leveraging a small amount of SSD storage, these drives save data to SSD first, then write it at slower speeds to the actual magnetic platters. The same issues with SSD storage can affect this cache of data.

So how to be sure?

Ideally, I would recommend using a combination of methods for security. Here are the main methods that are used at present:

Run shred and hope for the best

This is an option, for sure. Writing to every block of the disk will generally wipe the data securely enough. However, if there are sectors that have been marked bad by the drive firmware, these won’t be covered.

nwipe, DBAN, and other free options

Free software exists to securely run shred over operating systems. The old gold standard for this was Darik’s Boot and Nuke (DBAN), written by Darik Horn, but this software was acquired by Blancco, a for-profit data erasure company (more on them later). DBAN is still available, but lacks features necessary for a 100% wipe.

A fork called nwipe exists, and is available on many live operating systems. nwipe does a more thorough job than shred, but it still can’t get to sectors the firmware hides from the operating system.

Wikipedia has a list of data-erasing software. Most of these are open source or freeware, but it includes a few paid options.

SATA secure erase with hdparm

Drive manufacturers have thought of this issue as well. Most drive manufacturers offer a secure erase program that works with their drives, and the SATA standard has a procedure in place for securely erasing drive contents.

hdparm --user-master u --security-set-pass TheEnd /dev/X
hdparm -I /dev/X  # Check that the master password is enabled
time hdparm --user-master u --security-erase TheEnd /dev/X

This will show some output indicating success or failure. Once this is done…

hdparm -I /dev/X  # Check that the master password is not enabled, which indicates the wipe was successful

This is likely the best option for a savvy home user, combined with shred/nwipe. However, if you are going to attempt this, I highly recommend reading the full instructions!

Commercial software

Blancco, aforementioned as the purchasers of DBAN, offer an enterprise level product for destroying data called Drive Eraser. Importantly, for business users, it provides certification that the data are gone for good.

There are also many other options. Ask your favourite security vendor and they will be happy to sell you a product for this, though you usually have to take them at their word.

Be prepared—encrypt your disk!

Encrypt your disk. This can be achieved by encrypting either your home folder or by encrypting your whole disk. One downside is that your password is required after every reboot—a problem especially for servers, but even cloud providers offer virtual terminals these days, so for secure operations, this is the best option.

A secure, long password is usually enough to ensure the disk can’t be cracked, though be aware that cybersecurity changes quickly—what’s impossible to brute-force today might not be in five years.

So how can we be really, really sure?

Thermite

Physical destruction of the media is always the most secure way to destroy a disk—crushing, drilling, or in the fanciful, dangerous dreams of some systems administrators, covering the disk in thermite and lighting it with a magnesium flare. (Do not actually use thermite.)

At home, disassembling the drive and taking a hammer to the data bearing chips will do the trick, though again, combine this with the above options to be sure.

For spinning disks, a similar procedure is advised, though there are other options, of course…

Check your local laws and follow all safety procedures before engaging in creative drive destruction techniques!

Final thoughts

Use any or all of the above options and you’ll be ahead of the game. Taking the time to sanitize your data, or better, encrypting it from the beginning, is always a good investment.

Introduction to BorgBackup

2020-09-10T00:00:00+00:00

Photo by Frank R

What is Borg?

BorgBackup (Borg for short) is a ‘deduplicating’ backup program that eliminates duplicate or redundant information. It optionally supports compression and authenticated encryption.

The main objective of Borg is to provide an efficient and secure way to backup data. The deduplication technique utilized to produce the backup process is very quick and effective.

Step 1: Install the Borg backups

On Ubuntu/Debian:

apt install borgbackup

On RHEL/CentOS/Fedora:

dnf install borgbackup

Step 2: Initialize Local Borg repository

Firstly, the system that is going to be backed up needs a new designated backup directory. Name the parent directory ‘backup’ and then create a child directory called ‘borgdemo’, which serves as the repository.

mkdir -p /mnt/backup
borg init --encryption=repokey /mnt/backup/borgdemo

Step 3: Let’s create the first backup (archive)

In Borg terms, each backup instance will be called an archive. The following demonstrates how to backup the ‘photos’ directory and designate the archive as ‘archive_1’.

borg create --stats --progress /mnt/backup/borgdemo::archive_1 /home/kannan/photos

Note: the archive label for each backup run needs to be specified.

Step 4: Next backup (Incremental)

In order to see if the run was successful, the same command will be executed again. However, this time, with the different unique archive label.

borg create --stats --progress /mnt/backup/borgdemo::archive_2 /home/kannan/photos

The following backup is mostly identical to the previous one. Because of deduplication, the process will not only run faster this time, it will be incremental as well. The --stats flag will provide statistics regarding the size of deduplication.

Step 5: List all the archives

The ‘borg list’ command lists all of the archives stored within the Borg repository.

borg list /mnt/backup/borgdemo

Step 6: Remote Borg Repository

Take the scenario where the backups of many servers need to be maintained in a separate server. In this instance, a directory needs to be created for each of the systems that will be backed up. For this backup repository, create a folder named ‘backup’, and then within ‘backup’ a folder called ‘linode_01’. This folder will be initialized as a Borg repository.

mkdir -p /mnt/backup/linode_01
borg init --encryption=repokey user@backup_server:/mnt/backup/linode_01

The username, backup_server, repo can of course all be customized at the user’s discretion.

While initialising the repo, a passphrase for each backup repository can be set for authentication.

Step 7: Create an initial backup to the remote Borg repository

borg create --stats ssh://user@backup_server/mnt/backup/linode_01::archive_1 /home/kannan/photos

To enable the remote backups, the following three environment variables can be used to simplify the automation process:

export BORG_REPO='ssh://user@backup_server/mnt/backup/linode_01'
export BORG_PASSPHRASE='set_your_passpharase'
export BORG_RSH='ssh -i /home/kannan/.ssh/id_rsa_backups'

With those environment variables set, the ‘borg create’ command can be shortened to the following:

borg create --stats ::archive_1 /home/kannan/photos

Step 8: Excluding certain directories or files

In order to exclude certain directories or files, the create command has an --exclude option or an exclude file/directory pattern can be generated. For example, the following command demonstrates how to exclude /dev and /opt:

borg create --stats ::archive_1 / --exclude /dev /opt

Step 9: Restoring an archive through extraction

The ‘borg extract’ command extracts the contents of an archive. As a preset default, the entire archive will be extracted. However, the extraction can be limited by passing the directory path or file path as arguments to the command. For example, this is how a single photo can be extracted from the Photos archive:

borg extract ::archive_1 /home/kannan/photos/sunrise.jpg

Step 10: Pruning older backups

Every backup solution should have a way to maintain the older backups. Borg offers us borg prune for this. It prunes a repository by deleting all archives not matching any of the specified retention options.

For example, retain the final 10 archives from the day, another 6 end-of-week archives, and 3 of the end-of-month archive for every month using the following syntax:

borg prune -v --list --keep-daily=10 --keep-weekly=6 --keep-monthly=3 ::

Note that the double colons :: are required in order to automatically use the environment variables that were set prior.

For more in-depth documentation on Borg backup, read the docs.

Rclone: upload to the cloud from your command line and much more

2020-09-09T00:00:00+00:00

The Swiss army knife of storage

Cloud storage providers like Google Drive are great solutions for storing files. You can upload your data and not worry about maintaining a separate system to host it, or all the security hassles that can bring. However, very few major cloud storage providers offer a command line interface or any other official way to upload without using their web interface or closed-source binary tools, if they even offer that.

This obviously makes uploading files from servers difficult, but not impossible if you know the right tools.

About a year ago Jon Jensen penned a blog post about gdrive, a Google Drive command-line tool. However, due to changes with Google’s Drive security, that tool no longer works. This led me to look for a replacement.

Our use case

Recently I had to put some large files in to long term storage on Google Drive, since we needed the local space back. We wanted to retain the data, but didn’t foresee needing to access it for some time, if ever. Google Drive was a good solution for us, but the problem became how to get it there.

The files were too big, and some of them were not stored sparsely—empty space was tacked on to the disk images. We wanted to encrypt them, as the drives potentially contained customer information. So we had to sequentially process the files, encrypt them, and upload them. I felt like this would take quite a bit of time.

Enter rclone. Rclone can connect to many different kinds of cloud storage providers, DIY cloud storage solutions, and even things like FTP and WebDAV. You can use rclone to copy files directly like rsync, or even use it to mount the remote storage as a local drive. We chose to do the latter.

Rclone connects to a dizzying array of remote web services including Dropbox, Box, Amazon S3, Mega, SugarSync, and even homebrew cloud like ownCloud! This example uses Google Drive, but the instructions for many cloud providers are similar. The setup wizard can guide you through each step.

We decided to mount the remote storage, then encrypt and compress the files with gpg outputting directly to the remote mount point. This would allow us to set several different keys for decryption. Rclone offers a crypt module for encrypting backups, but this requires keeping a password. We prefer each person who may need to decrypt this be able to use a key. This also allowed me to run the command and walk away rather than try to shepherd the process along manually.

Once the rclone mount was in place, encrypting the files for upload was very simple. Having the storage mounted locally negated the need to encrypt it locally before uploading. We simply ran the following:

gpg --output /mnt/rclone/drive/sensitive_file.gpg --recipient recipient@email.com --encrypt ~/sensitive_file

Does this sound useful? Well, if it does…

How to do this yourself

Installing rclone is as simple as downloading the binary from rclone.org or cloning the Git repository and compiling it yourself. They also offer a script to automatically download the most recent version. Packages are also available in most operating system repositories, though as usual those tend to be a little older.

Whichever method you choose, once you have rclone installed, you’ll need to configure it to use your remote storage.

$ rclone config
2020/08/28 11:33:19 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

Choose ‘n’ for a new remote, and name it something memorable.

n/s/q> n
name> gdrive
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Google Drive
   \ "drive"
[snip]
Storage> drive

(At this point you could also choose a different storage type, and the rest of these instructions should still work.)

** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>

At this point, follow the instructions on how to make your own client id from rclone’s website or keep reading. If you leave this blank, the app will use a shared client id and there is global rate limiting in effect for every rclone instance.

client_id> [snip].apps.googleusercontent.com
Google Application Client Secret
Setting your own is recommended.
Enter a string value. Press Enter for the default ("").
client_secret> [snip]
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 3

At this point, choose a scope for the drive access. I’m not sure what the use cases are for the higher numbers, but 1–3 seem to be the most useful. For this demonstration, I chose 3—I want the app to be able to create new files but not read existing ones. For backups, I recommend 1 or 2.

ID of the root folder
Leave blank normally.

Fill in to access ‘Computers’ folders (see docs), or for rclone to use
a non root folder as its starting point.

Note that if this is blank, the first time rclone runs it will fill it
in with the ID of the root folder.

Enter a string value. Press Enter for the default ("").
root_folder_id>

You can also jail the app to a specific subfolder. You’ll need to use a folder ID for this, rather than a folder name. See the documentation if you need to do this.

Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file>  # I left this blank
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No

For our use case, we are setting this up on a remote machine, but you can use the interactive login if you have a graphical user interface on the machine in question.

y/n> n
Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=[snip]
Log in and authorize rclone for access
Enter verification code>

At this point copy the URL and follow the instructions on screen. Agree to the data sharing—If you have created your own API key, this data will be only shared with yourself, another reason to create your own.

Configure this as a team drive?
y) Yes
n) No (default)
y/n> n
--------------------
[gdrive]
type = drive
client_id = [snip]
client_secret = [snip]
scope = drive.file
root_folder_id = rclone
token = [snip]
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
gdrive               drive

At this point, you can quit the configuration mode, and your new rclone remote is ready to use!

Mounting the remote drive (putting it all together)

$ sudo mkdir -p /mnt/rclone/drive
$ sudo chown -R $USER /mnt/rclone/
$ rclone mount gdrive: /mnt/rclone/drive/

Note: If you wish to have the above command run in the background, add & to the end of the command. You’ll have to unmount it later with fusermount -u /mnt/rclone/drive

At this point you can copy any files you like into the mount above. However, you could just do that with rclone directly: rclone sourcefile gdrive:/. will work. Instead, I used the following command, which I showed at the very beginning of the blog post:

for i in $( ls /drives/source/directory ); do
  gpg --output /mnt/rclone/drive/$i.gpg --recipient recipient@email.com --encrypt /drives/source/directory/$i
done

Quite a bit of setup. But the result is a mounted folder into which you can copy encrypted files directly, without any intermediary steps. The above command took a while, but by automating the task, the total work time was still less than it would have been trying to do it manually, especially given that we didn’t have the spare disk space to store a separate encrypted copy of our files.

Want to do more? You could take a whole drive image with dd, run it through gpg, and pipe the output to the remote storage. You could use the crypt module for your own backups, rather than relying on gpg. The mount point acts as though it was a drive on your system, so any system utility like Timeshift could back up to the drive.

If you have any other suggestions for how this might be used, please leave a comment!

DIY Google API key

These instructions can also be found on the rclone website.

For this blog post I used my personal account and went through the steps below. Some of the steps may be different for a G Suite account. It’s a little cumbersome but will only take 5–10 minutes.

Once you’ve created the project, click on ‘Enable APIs and Services’.

Search for ‘Drive’ and click on the Google Drive API, which should be the first option.

Enable the Google Drive API.

Once you have enabled this, do not click ‘Create Credentials’ directly — this takes you to the wrong wizard. Instead, click on Credentials in the left navigation pane, then ‘Create Credentials’, which gives you the option to just create OAuth credentials, which is what we need.

Choose ‘OAuth credentials’ and fill in the OAuth consent screen if necessary. You can fill in limited information here, and just use ‘rclone’ for the name. If you are using a G Suite account you can choose ‘Internal’ rather than ‘External’.

Once you have done that, create your OAuth ID. Again, you can just name this rclone. Choose the application type of ‘Desktop App’.

You’re done! You can now use the OAuth ID displayed on this screen with rclone. Or go back and see it again from the dashboard under ‘Credentials’, then clicking the name of the OAuth ID you created.

When creating the ID, it’ll say you will need to verify the app. This is only true if you plan on heavy use. You can use it quite easily without this step for smaller use cases.

(These instructions were originally written by GitHub user @balazer, and are adapted here with many thanks.)

Google Drive for virtual machine images

2019-09-30T00:00:00+00:00

Recently we decommissioned an old database server. We wanted to keep a copy of its 53.7 GB virtual machine disk image in an archive in case there is ever any need to revive it. It is really unlikely that we will need it any time soon, or maybe ever, so we thought of putting it in one of the cloud storage services.

Cloud storage

Cloud service pricing is often metered by storage, network, operations, and other fees, making it complicated to calculate what you will pay. We already use Amazon S3, Azure Storage, and Google Cloud Storage and they are all no exception. For our example 53.7 GB disk image, the Google Cloud Storage Standard Storage pricing is currently:

No charge for network ingress when we upload it
About $1.08 to $1.40 USD per month to store the file in the US or EU
About $6.50 each time we download the file to most places in the world!

These days a ~50 GB disk image is on the small side, so you can imagine the cost going up rapidly with larger disk sizes.

Some cloud storage providers offer lower prices for slow-availability semi-offline storage, such as Amazon S3 Glacier or Google Coldline Storage. Those would indeed save us some money for monthly storage, but the most expensive aspect is the network egress cost, which is the same.

Local storage

External USB disk drives have gotten very inexpensive in recent years, so we considered just spending roughly $180 for a 6 TB hard disk and storing our disk image there. We can fit over 100 VM images of this size on such a disk, so the cost works out to a one-time cost of about $1.80 each.

But there are downsides to local storage, including:

It is a single point of failure. The disk can:
- fail
- be lost or stolen
- be damaged by water, an impact such as someone dropping it or a horse kicking it 🐴😀, or a power surge or static electricity
- be forgotten if the employee who has it leaves and nobody remembers to take it over.
The data would likely be inaccessible while the employee is out of the office.
Retrieval speed is limited by the upstream network bandwidth which is often fairly slow for consumer Internet connections and even crowded office networks.

For “just in case” old archival data, these risks are pretty reasonable and the price is good. But it would be nice to decouple the storage from any individual employee’s network speed.

Google Drive

Then we remembered that we have Google Drive as part of G Suite, and even though we don’t normally use it this way, it offers a lot of space and could be a place to store infrequently-needed large archival data such as this.

Many of us think of Google Drive primarily as its browser-based folder and file interface, a graphical consumer or business user service. So our instinct is to first download a file our desktop and then upload it from there to Google Drive using the browser.

For our large, multi-gigabyte files on a server, that is cumbersome and slow, at least twice as slow as it should be. We really would prefer to upload them directly from the server to Google Drive. And to use the old archived images, we would want to download them from Drive directly to a remote server, not first to a desktop.

Uploading

Conveniently for us, Google Drive has a web service API we can use! Oliver Marshall’s nice article shows how to use the open source gdrive command-line tool to access it.

The easiest thing to do after we have gdrive authenticate with our Google account is:

gdrive upload /path/to/file

which will put it in the base folder of My Drive.

You can also use gdrive list … to find a specific folder you would like to upload it into.

To give an idea of the speed, I had several files to upload from a Linux server, and measured speed of about 12 MB/sec. for each of 3 parallel uploads on a 100 Mbps server connection.

Verifying

Did the huge files make it without error? Google Drive doesn’t show them until they’re complete, which is good, so we will not see any partial files. To confirm, let’s check file size and MD5 hash value, which isn’t visible in the Drive web interface, but is via the API, along with the view & download URLs:

# gdrive info 1YMEoEWWOvuBGSk0GJBv7c0E4cSEOoc6n
Id: 1YMEoEWWOvuBGSk0GJBv7c0E4cSEOoc6n
Name: prod-db-20190731
Path: VM images/Database server/prod-db-20190731
Mime: application/octet-stream
Size: 53.7 GB
Created: 2019-08-08 21:20:50
Modified: 2019-08-08 21:20:50
Md5sum: 43f7e727047cb2bbbfb54b413752c229
Shared: True
Parents: 1HBkixF0UxraH6nHyD1Fiwx5X5k7dfaKm
ViewUrl: https://drive.google.com/a/your.domain/file/d/1YMEoEWWOvuBGSk0GJBv7c0E4cSEOoc6n/view?usp=drivesdk
DownloadUrl: https://drive.google.com/a/your.domain/uc?id=1YMEoEWWOvuBGSk0GJBv7c0E4cSEOoc6n&export=download

When we run md5sum (or md5 on BSD systems such as macOS) on the original file or block device on our server, the resulting hash value should match the Md5sum value gdrive shows.

Cautions

Regular Google Drive accounts are tied to the user, so if you leave the organization and they remove your account, your files will need to be deleted or reassigned to another user.

Use a Shared Drive (formerly called Team Drive) to avoid having the files tied to a specific user. There is even an API for G Suite admins to search all shared drives in their account.

I haven’t seen anything to make me think this API is any less reliable than anything in Google Cloud Platform, but it certainly is not promoted as being something to use for a busy production web application. It seems safest to use it for archival storage like this that will be accessed rarely, by a human.

Once authorized, your gdrive command-line tool has access to all your Google Drive files. That isn’t something you want to leave sitting around for ill-intentioned people to perhaps access. So when you’re done uploading your files, remove the app authorization until the next time you need it. In your browser go to: Google Drive > Settings (gear icon) > Manage Apps > GDrive > Options > Remove app.

Data Center Relocation Initiatives: Daunting But Achievable

2018-12-20T00:00:00+00:00

Photo by Seeweb · CC BY-SA 2.0, cropped

Introduction

A customer asked for our help dealing with logistical nightmares they encountered during a hardware update and data center relocation project. The customer had two active data centers, and wanted to relocate one of them to a modern Tier 4 facility to improve its performance and provide redundancy for critical systems. They also wanted to consolidate or decommission equipment to reduce recurring expenses and reduce their carbon footprint. The client decided to move to a Tier 4 data center because they provide across-the-board redundancy within the data center.

Moving forward with Tier IV

Enterprises typically transition to Tier 4 data centers because they offer the highest uptime guarantees, and have no single points of failure. These facilities are fully redundant in terms of electrical circuits, cooling, and networking. This architecture is best able to withstand even the most serious technical incidents without server availability being affected. Tier IV facilities have contracts with disaster management companies who will provide them, for example, with fuel in the event that a natural disaster damages the power grid.

(More information about the four data center levels is at the end of this post.)

Key Issues

Some of the customer’s key concerns were:

1. To safely move their virtual environment to the new data center.

2. To protect assets during the relocation.

3. To completely shut down the current data center and migrate seamlessly to the new data center.

4. To update external DNS records without causing any downtime for the web applications used by their customers.

Project Planning

To manage data center relocations is challenging because there are many moving parts and variables. Typically, planning starts 6 months ahead of the scheduled relocation date. We seek to understand the role of each system by consulting in-depth with all the teams involved. We analyze each part of the client’s tech infrastructure, break it down, and strategize about the best approach for handling the migration of each component. Additional time to plan can help especially when a large group of stakeholders is involved. Successful planning requires hard and sometimes extensive work, but makes an otherwise overwhelming data center migration smooth and worry free.

Here are some of our procedures for pre-migration preparation:

1. Process the paperwork for new hardware. Identify, review, and approve all applications. Hire movers. Schedule and coordinate with internal management and key stakeholders like IT staff, system admins, contractors, circuit providers, etc.

2. Collect and analyze resource usage associated with the critical and non-critical systems and compare resource needs to resource availability at the new facility.

3. Design and schedule the migration of critical systems to the new data center. Discuss the plan with key stakeholders.

4. Make sure replicas of critical systems are up to date.

5. Set up a new environment with new hardware in the new data center and establish network connections.

6. Prepare hardware and virtualization technology in the new data center.

7. Test, retest, and burn in the new hardware (2–4 weeks of testing).

8. Create a non-essential server within the virtualization environment and test it.

A Challenge: Eliminating Downtime

To minimize or eliminate downtime requires very careful preparation. With this particular client, we had the luxury of their already having a secondary data center, which could accommodate a majority of the critical systems during the migration.

We originally planned to leverage this by simply changing the external DNS records to point to this data center during the migration, but during planning determined that the external DNS record would not replicate worldwide quickly enough to meet the uptime needs of their global operation.

Solutions

After discussing various approaches, we ended up using a load balancer. A load balancer is a network device that routes incoming traffic destined for a single destination (web site, application, or service) and ‘shares’ the incoming connections across multiple destination devices or services. In this case, the data center we were to migrate to housed the client’s critical systems like web servers, database, etc. The secondary data center housed a replica of the critical systems using a replication technology, e.g. Actifio, Rubrik, SAN mirroring, VMWare SRM, or Zerto — we used Actifio in this scenario.

The customer had multiple systems with external DNS records and public IP addresses. We did not change the public DNS names for the critical systems. We did change the public IP addresses associated with the public DNS names, to point to the load balancer, which then pointed to the appropriate data center throughout the process. This setup enabled a gradual, flexible, and fully controlled transition to the new data center.

Summary

Our client wanted to reduce their servers’ power usage 40% by consolidating into a hyper-converged system. Other goals were to reduce floor space by 50%, and monthly recurring data center charges by 50%. We transitioned them to a new hardware stack (compute, storage, network, backup environment) in a Tier 4 data center, and saved them 50% percent on data center operation costs. Additionally, during the process we corrected issues associated with disaster recovery and business continuity, leaving the client’s operations more resilient, affordable, and more secure.

Appendix: Data Center Classifications

Tier IV (Best)

Tier IV data centers have redundancies for every process and data protection stream. No single outage or error can shut down the system. To keep Tier IV ranking, data centers must have 99.995% minimum uptime per year. They come equipped with 2N+1 infrastructure (two times the amount required for operation plus a backup), which is considered “fully redundant.”

Tier III (Average)

N+1 (the amount required for operation plus a backup) fault tolerance, which means a Tier III facility can undergo routine maintenance without a hiccup in operations. 99.982% minimum uptime per annum, used for maintenance and overwhelming emergencies.

Tier II (Minimal)

99.749% minimum uptime. Tier II facilities have considerably more downtime than Tier III facilities, primarily because Tier II facilities have less redundancy. They do however have partial cooling redundancy and multiple power redundancies.

Tier I (Basic, Cheapest)

99.671% minimum uptime, with 28.8 hours of downtime each year.

Half day GlusterFS training in Selangor, Malaysia

2017-04-02T00:00:00+00:00

On January 21, 2017, I had an opportunity to join a community-organized training on storage focused on GlusterFS. GlusterFS is an open source cloud-based filesharing network. The training was not a strictly structured training as the topic approached knowledge sharing from various experts and introduced GlusterFS to the ones who were new to it. The first session was delivered by Mr Adzmely Mansor from NexoPrima. He shared a bit of his view on GlusterFS and technologies that are related to it.

Mr Haris, a freelance Linux expert, later led a GlusterFS technical class. Here we created two virtual machines (we used Virtualbox) to understand how GlusterFS works in a hands-on scenario. We used Ubuntu 16.04 as the guest OS during technical training. We used Digital Ocean’s GlusterFS settings as a base of reference. The below commands detail roughly what we did during the training.

In GlusterFS the data section is called as “brick”. Hence we could have a lot of “bricks” if we have it more than once :) . As Ubuntu already had the related packages in its repository, we could simply run apt-get for the package installation. Our class notes were loosely based from Digital Ocean’s GlusterFS article here. (Note: the article was based on Ubuntu 12.04 so some of the steps could be omitted).

The GlusterFS packages could be installed as a superuser with the following command:

apt-get install glusterfs-server

Since we were using a bridged VM during the demo, we simply edited the /etc/hosts in the each VM so they could communicate between each other by using hostname instead of using typing the IP manually.

root@gluster2:~# grep gluster /etc/hosts
192.168.1.11 gluster1
127.0.0.1 gluster2

Here we will try to probe the remote host whether it is reachable:

root@gluster2:~# gluster peer probe gluster1
peer probe: success. Host gluster1 port 24007 already in peer list

The following commands create the storage volume. Later, whatever we put in the /data partition will be reachable on the other gluster node.

gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data
gluster volume create datastore1 replica 2 transport tcp gluster1:/data gluster2:/data force
gluster volume start datastore1

Most of the parts here could be retrieved from the link that I gave above. But let’s see what will happen later on when the mounting part is done.

cd /datastore1/
root@gluster2:/datastore1# touch blog
root@gluster2:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt

The same output could be retrieved from gluster1

root@gluster1:/datastore1# ls -lth
total 512
-rw-r--r-- 1 root root  0 Mar 14 21:33 blog
-rw-r--r-- 1 root root 28 Jan 21 12:15 ujian.txt

Mr. Adzmely gave explanation on the overall picture of GlusterFS

Mr. Haris explained on the technical implementation of GlusterFS

In terms of the application, the redundancy based storage is good for situations where you have a file being updated on several servers and you need to ensure the file is there for retrieval even if one of the servers is down. One audience member shared his experience deploying GlusterFS in his workplace (a university) for the purpose of new intake of student’s registration. If anyone ever uses Samba filesystem or NFS, this kind of similar, but GlusterFS is much more advanced. I recommend additional reading here.

Handling databases in dev environments for web development

2015-04-21T00:00:00+00:00

One of the biggest problems for web development environments is copying large amounts of data. Every time a new environment is needed, all that data needs to be copied. Source code should be tracked in version control software, and so copying it should be a simple matter of checking it out from the code repository. So that is usually not the problem. The main problem area is database data. This can be very large, take a long time to copy, and can impact the computers involved in the copy (usually the destination computer gets hammered with IO which makes load go high).

Often databases for development environments are created by copying a database dump from production and then importing that database dump. And since database dumps are text, they can be highly compressed, which can result in a relatively small file to copy over. But the import of the dump can still take lots of time and cause high load on the dev computer as it rebuilds tables and indexes. As long as your data is relatively small, this process may be perfectly acceptable.

Your database WILL get bigger

At some point though your database will get so big that this process will take too long and cause too much load to be acceptable.

To address the problem you can try to reduce the amount of data involved by only dumping a portion of the database data instead of all of it, or possibly using some “dummy sample data” instead. These techniques may work if you don’t care that development environments no longer have the same data as production. However, one serious problem with this is that a bug or behavior found in production can’t be replicated in a development environment because the data involved isn’t the same. For example, say a customer can’t checkout on the live site but you can’t replicate the bug in your development environment to fix the bug. In this example, the root cause of the problem could be a bug in the code handling certain products that are out of stock, and since the dev database didn’t have the same data it could make finding and fixing these types of problems a lot harder.

Snapshots

Another option is to use file system snapshots, like LVM snapshots, to quickly make clones of the database without needing to import the database dump each time. This works great if development environments live on the same server, or at least the development databases live on the same server. You would need to create a volume to hold a single copy of the database; this copy would be the origin for all snapshots. Then for each development environment, you could snapshot the origin volume, mount it read-write in a place accessible by the developer, customize the database configuration (like setting a unique port number to listen on), and then start up the database. This then provides a clone of the entire database in a tiny fraction of the time and uses less disk space and other system resources too.

In using snapshots there are some things you’ll need to be careful about. Snapshots are usually created using copy-on-write tables. The more snapshots mounted read-write, the more IO overhead is involved for the volumes involved. For this reason it is important that writes to the origin volume be avoided as much as possible while the snapshots are open. Also, snapshots that get a lot of writes can fill up their copy-on-write table, and depending on the file system and database that you are using this can be a big problem. So it is important to monitor each open snapshot for how full it is and increase their size if needed so they don’t fill up. Updating the origin database will require shutting down and removing all snapshots first, then update the origin database, then create and mount all the snapshots again. This is because all the copy-on-write tables would get full if you tried to update the origin while the snapshots are open.

Using snapshots like this may sound more complicated, and it is, but the processes involved can be scripted and automated and the benefits can be pretty significant if you have several developers and a lot of data to copy.

Shrink XFS partition: Almost possible with LVM

2015-01-29T00:00:00+00:00

If you happen to have reached this page because you’re trying to shrink an XFS filesystem let’s put things straight: sorry, that’s not possible.

But before you go away you should know there’s still hope and I’d like to show you how I used a little workaround to avoid reinstalling a Red Hat Enterprise Linux 7 or CentOS 7 VM using XFS dump/restore on-the-fly and LVM capabilities, both standard choices for the regular RHEL/CentOS 7 setup.

First of all let’s clarify the situation I found myself in. For various reasons I had a CentOS 7 VM with everything already configured and working, installed not many days ago to test new software we’re evaluating.

The VM itself is hosted on a dedicated server we manage on our own, so I had a certain degree of freedom to what I could do without the need of paying any additional fee. You may not be in this same situation, but you can probably try some similar solution for little money if you’re using an “hourly-billed” VPS provider.

The problem was that, even if everything was working and configured, the virtual hard disk device attached to the machine was too big and on the wrong storage area of the virtualization hosting server.

There was also another minor glitch: The VM itself was using an old virtualization device driver (IDE-based) instead of the new VIRTIO one. Since I knew that the virtualized OS CentOS 7 is capable of using VIRTIO based devices I also took the chance to fix this.

Unfortunately, XFS is not capable of being shrunk at the moment (and for the foreseeable future) so what I needed to do was to:

add a new virtual device correctly using VIRTIO and stored in the right storage area of the virtualization host server
migrate all the filesystems to the new virtual device
set the VM OS to be working from the newly-built partition
dismiss the old virtual device

In my specific case this translated to connect to the virtualization hosting server, create a new LVM logical volume to host the virtual disk device for the VM and then add the new virtual device to the VM configuration. Unfortunately in order to have the VM see the new virtual device I had to shut it down.

While connected to the virtualization host server I also downloaded and attached to the VM the latest ISO of SysRescueCD which is a data rescue specialized Linux distribution. I’m specifically using this distro since it’s one of the few which offers the XFS dump/restore tools on the live ISO.

Now the VM was ready to be booted with the SysRescueCD Live OS and then I could start working my way through all the needed fixes. If you’re doing something similar, of course please make sure you have offsite backups and have double-checked that they’re readable before doing anything else.

First of all inspect your dmesg output and find what is the source virtual device and what’s the new target virtual device. In my case the source was /dev/sda and the target was /dev/vda

dmesg | less

Then create a partition on the new device (eg: /dev/vda1) as Linux type for the /boot partition; this should be of the same size as the source /boot partition (eg: /dev/sda1) and dedicate all the remaining space to a new LVM type partition (eg: /dev/vda2)

fdisk /dev/vda
# [create /boot and LVM partitions]

You could also mount and copy the /boot files or re-create them entirely if you need to change the /boot partition size. Since I kept /boot exactly the same so I could use ddrescue (a more verbose version of classic Unix dd).

ddrescue /dev/sda1 /dev/vda1

The next step is supposed to migrate the MBR and should be working but in my case the boot phase kept failing so I also needed to reinstall the bootloader via the CentOS 7 rescue system (not covered in this tutorial but briefly mentioned near the end of the article).

ddrescue -i0 -s512 /dev/sda /dev/vda

Then create the target LVM volumes.

pvcreate /dev/vda2
vgcreate fixed_centos_VG /dev/vda2
lvcreate -L 1G -n swap fixed_centos_VG
lvcreate -l 100%FREE -n root fixed_centos_VG
vgchange -a y fixed_centos_VG

Create the target XFS filesystem.

mkfs.xfs -L root /dev/fixed_centos_VG/root

And then create the swap partition.

mkfs.swap /dev/fixed_centos_VG/swap

Next create the needed mountpoints and mount the old source and the new empty filesystems.

mkdir /mnt/disk_{wrong,fixed}
mount /dev/fixed_centos_VG/root /mnt/disk_fixed
vgchange -a y wrong_centos_VG
mount /dev/centos_VG/root /mnt/disk_wrong

Now here’s the real XFS magic. We’ll use xfsdump and xfsrestore to copy the filesystem content (files, directory, special files) without having to care about files permission, type, extended ACLs or anything else. Plus since it’s only moving the content of the filesystem we won’t need to have a partition of the same size and it won’t take as long as copying the entire block device as the process will just have to go through real used space.

xfs_dump -J - /mnt/disk_wrong | xfs_restore -J - /mnt/disk_fixed

If you want a more verbose output, leave out the -J option. After the process is done, be sure to carefully verify that everything is in place in the new partition.

ls -lhtra /mnt/disk_fixed/

Then unmount the disks and deactivate the LVM VGs.

umount /mnt/disk_{old,fixed}
vgchange -a n centos_VG
vgchange -a n fixed_centos_VG

At this point in order to avoid changing anything inside the virtualized OS (fstab, grub and so on), let’s remove the old VG and rename the newer one with the same name the old one had.

vgremove centos_VG
pvremove /dev/sda2
vgrename {fixed_,}centos_VG

You should be now able to shutdown the VM again, detach the old disk and start the new VM which will be using the new smaller virtual device.

If the boot phase keeps failing, boot the CentOS installation media in rescue mode and after chroot-ing inside your installation run grub-install /dev/vda (targeting your new main device) to reinstall grub.

Only after everything is working as expected, proceed to remove the old unneeded device and remove it from the virtualization host server.