• Home

  • Custom Ecommerce
  • Application Development
  • Database Consulting
  • Cloud Hosting
  • Systems Integration
  • Legacy Business Systems
  • Security & Compliance
  • GIS

  • Expertise

  • About Us
  • Our Team
  • Clients
  • Careers

  • Blog

  • EpiTrax

  • VisionPort

  • Contact
  • Our Blog

    Ongoing observations by End Point Dev people

    KISS: Slurping up File Attachments

    Steph Skardal

    By Steph Skardal
    June 26, 2012

    I’ve been heavily involved in an ecommerce project running on Rails 3, using Piggybak, RailsAdmin, Paperclip for file attachment management, nginx and unicorn. One thing that we’ve struggled with is handling large file uploads both in the RailsAdmin import process as well as from the standard RailsAdmin edit page. Nginx is configured to limit request size and duration, which is a problem for some of the large files that are uploaded, which are large purchasable, downloadable files.

    To allow these uploads, I brainstormed how to decouple the file upload from the import and update process. Phunk recently worked on integration of Resque, a popular Rails queueing tool which worked nicely. However, I ultimately decided that I wanted to go down a simpler route. The implementation is described below.

    Upload Status

    First, I created an UploadStatus model, to track the status of any file uploads. With RailsAdmin, there’s an automagic CRUD interface connected to this model. Here’s what the migration looked like:

    class CreateUploadStatuses < ActiveRecord::Migration
      def change
        create_table :upload_statuses do |t|
          t.string :filename, :nil => false
          t.boolean :success, :nil => false, :default => false
          t.string :message
    
          t.timestamps
        end
      end
    end
    

    RailsAdmin also leverages CanCan, so I updated my ability class to allow list, reads, and delete on the UploadStatus table only, since there is no need to edit these records:

          cannot [:create, :export, :edit], UploadStatus
          can [:delete, :read], UploadStatus
    

    KISS Script

    Here’s the simplified rake task that I used for the process:

    namespace :upload_files do
      task :run => :environment do
        files = Dir.glob("#{Rails.root}/to_upload/*.*")
        files.each do |full_filename|
          begin
            ext = File.extname(full_filename)
            name = File.basename(full_filename, ext)
    
            (klass_name, field, id) = name.split(':')
            klass = klass_name.classify.constantize
            item = klass.find(id)
    
            if item.nil?
              UploadStatus.create(:filename => "#{name}#{ext}", :message => "Could not find item from #{id}.")
              next
            end
    
            item.send("#{field}=", File.open(full_filename))
    
            if item.save
              FileUtils.rm(full_filename)
              UploadStatus.create(:filename => "#{name}#{ext}", :success => true)
            end
          rescue Exception => e
            UploadStatus.create(:filename => "#{name}#{ext}", :message => "#{e.inspect}")
          end
        end
      end
    end
    

    And here’s how the process breaks down:

    1. The script iterates through files in the #{Rails.root}/to_upload directory (lines 3-4).
    2. Based on the filename, in the format “class_name:field:id.extension”, the item to be updated is retrieved (line 11).
    3. If the item does not exist, an upload_status record is created with a message that notes the item could not be found (lines 13-16).
    4. If the file exists and the update occurs, the original file is deleted, and a successful upload status is recorded (lines 18-23).
    5. If the process fails anywhere, the exception is logged in a new upload status record (lines 24-26).

    This rake task is then called via a nightly cron job to slurp up the files. The simple script eliminates the requirement to upload large files via the admin interface, and decouples the upload from Paperclip/database management. It also has the added benefit of reporting the status to the administrators by leveraging RailsAdmin. Many features can be added to it, but it does the job that we need without much development overhead.

    rails


    Comments