Using FFI in Ruby

2018-04-16T00:00:00+00:00

Ruby for many years has been proving to be an amazing language. It’s one of the most popular for creating web applications but also DevOps / systems administration tools.

It seems that languages are naturally finding their own niches. For Ruby it’s what I listed above, while Python seems to work well for computer vision and machine learning.

For performance reasons, some code should never be coded in either one. Topic boundaries are being crossed typically though. You might be working on the web app that does some high performance math in the background. To marry the two worlds, people were successful with creating compiled “extensions”. These extensions are typically coded in C. Both languages along with their interpreters are able to use those using native-to-language constructs.

One good example of this is the great numpy package. It brings high-performance linear algebra constructs that would be impossible to code in pure Python with the same performance.

In the Ruby land, a similar example is the nokogiri gem. It allows very fast XML / HTML processing thanks to the core of its functionality being coded in C.

You might also be working on a Ruby app having a need for a functionality that has already been written as a library with C interface available. One way to use it would be to create an extension that would operate on those external C functions. This involves quite a lot of prep-work though. The RubyGems guides describe the process in detail.

There is one more option and I’m about to show you what it is.

Referencing external functions directly

FFI stands for “Foreign Function Interface”. Most languages have a way to operate with external code (written in another language). The interface for doing so is what’s called FFI. Examples:

Docs on FFI in Haskell
Docs on FFI in Python via the cffi package
Docs on FFI in Ruby
Docs on FFI in Rust

In this article, we’re interested in using FFI in Ruby.

No matter what the host language, the usage pattern is always the same:

Define what external library you want to link with
Define functions you want to make use of and how their arguments and return values map to your host language data types
(Optionally) Define the calling convention
Use those functions as if they were native to your host language

Let’s see how it looks in Ruby. The first step is to define a module you want to “attach” external functions to. This very often looks similar to the following:

module SomeLibrary
  extend FFI::Library
  # (...)
end

The FFI::Library module brings in useful methods for defining the linkage with the target, external library. We still need to “tell” Ruby the external library we want to work with. Here’s an example of working with fribidi, which contains lots of very useful functions for working with bidirectional Unicode text:

module Bidi
  extend FFI::Library
  ffi_lib ['libfribidi']
  # (...)
end

Let’s say that we want to use a very helpful fribidi_log2vis function. We’ll want it to give us a logical position index for each visual position index of the text, given a directionality of it.

Disclaimer: If the language you speak uses Latin-based script in writing, you might be scratching your head now. For most languages the logical positions of characters are the same as the “visual” ones (where those characters end up visually in the word / token). This is not always the case. Counterexamples include Arabic, Hebrew, and Syriac (among others). A line of text could also contain words / tokens written using different scripts (like e.g. names of places written in the Latin script, inside a RTL paragraph). Sometimes also e.g. the punctuation might appear in one place logically in the string, while visually somewhere else.

The signature of the function reads as follows:

fribidi_boolean fribidi_log2vis(
    /* input */
    FriBidiChar     *str,
    FriBidiStrIndex len,
    FriBidiCharType *pbase_dir,
    /* output */
    FriBidiChar     *visual_str,
    FriBidiStrIndex *position_L_to_V_list,
    FriBidiStrIndex *position_V_to_L_list,
    FriBidiLevel    *embedding_level_list
)

In order to use this function from Ruby, we need to tell it what those types mean in terms of types available in Ruby and the ffi gem:

module Bidi
  extend FFI::Library
  ffi_lib ['libfribidi']

  attach_function :fribidi_log2vis, [ :pointer, :int32, :pointer, :pointer, :pointer, :pointer, :pointer ], :bool
end

The full documentation on the types mappings can be found here.

We’re now ready to wrap our newly attached external function inside a convenient Ruby code. We’ll create a to_visual_indices method on the Bidi module that will take a string and a symbol representing the directionality. The directionality will expect :rtl for the RTL direction. Here’s the listing of how it looks:

module Bidi
  extend FFI::Library
  ffi_lib ['libfribidi']

  attach_function :fribidi_log2vis, [ :pointer, :int32, :pointer, :pointer, :pointer, :pointer, :pointer ], :bool

  def self.to_visual_indices(text, direction)
    null = FFI::Pointer::NULL

    t = FFI::MemoryPointer.new(:uint32, text.codepoints.count)
    t.put_array_of_uint32(0, text.codepoints)

    pos = FFI::MemoryPointer.new(:int, text.codepoints.count)

    dir_spec = FFI::MemoryPointer.new(:long)
    dir_spec.write_long( direction == :rtl ? 273 : 272)

    success = Lib.fribidi_log2vis(t, text.codepoints.count, dir_spec, null, null, pos, null)

    if success
      return pos.read_array_of_int(text.codepoints.count)
    else
      raise StandardError, "Failed to infer the visual ordering of the text"
    end
  end
end

You could now use your new module along with the new method that uses an external library:

>> Bidi.to_visual_indices "Friendship الصداقة", :rtl
=> [17, 16, 15, 14, 13, 12, 11, 10, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Using FFI in Ruby is a great way to bring in existing external functionality without the need to write an extension in C. It also spares you from the need to compile it, which in the case of Rails is very, very convenient.

A Collaborative Timezone Utility

2017-10-30T00:00:00+00:00

Try It Out Yourself

The code for this project is hosted on GitHub and can be cloned from here.

At End Point Corporation, our team is spread out across 10 time zones. This gives us the advantage of being able to work around the clock on projects. When one co-worker leaves for day, another can take over. Consider this scenario. It’s Monday evening and Martin needs to continue installing software on that Linux cluster, but it’s already 6pm and his wife is going to murder him if he’s not ready to go out for their anniversary dinner. Let’s see who can take over… Ah, yes, Sanjay in Bangalore can continue with the maintenance. Tuesday morning, the client wakes up to be surprised that 16 hours of work was completed in a day. With respect to software development, the same efficiencies can be realized by parallelizing tasks across time-zones. Code reviews and further development can be continued after normal business hours.

With all the blessings of a distributed engineering team, collaborating with co-workers can be, occasionally, challenging. Some of these challenges stem from complexities of our system of time. Every co-worker may be operating in a timezone that is different than yours. Time-zones have an associated offset relative to Coordinated Universal Time (UTC). These offsets are usually in whole hour increments but they may be any real-valued number.

For example, Eastern Standard Time (EST) has an offset of -5 (five hours behind UTC) and Indian Standard Time (IST) has an offset of 5.5 (five and half hours ahead of UTC). Furthermore, these UTC offsets can be completely arbitrary. In 1995, Kiribati, an island nation in the Pacific, changed its UTC offset from -10 to +14 so that all of its outlying islands can share the same time. To further complicate things, some regions may not observe daylight savings time (DST) while other regions do. In fact, in the United States, Indiana started observing DST on April 2, 2006. Some states like Arizona and Hawaii do not observe DST. Other countries, like Australia, have a similar situation where it’s left to local governments to decide whether DST is observed or not. Moreover, although DST usually accounts for adding or subtracting an hour of time, it isn’t always one hour. This has historically changed from time to time.

Now you may begin to imagine the headaches that arise when you need to coordinate with anything involving multiple time-zones. To make all of this easier, you can use a utility that we wrote to do all the time conversions for you. First, you have to add each co-worker’s information to a configuration file stored at ~/.timezoner. This configuration file will describe all of your co-worker’s contact information and their associated IANA time-zone. As an example, this is what the configuration file looks like:

# Timezone            Email                  Name              OfficePhone        MobilePhone
America/New_York      "edward@example.com"   "Edward Teach "   "n/a"              "+1 731 555 1234"
America/New_York      "henry@dexample.com"   "Henry Morgan"    "+1 646 555 5678"  "+1 954 555 5678"
America/New_York      "john@example.com"     "John Auger"      "n/a"              "+1 902 555 1234"
America/Denver        "sam@example.com"      "Samuel Bellamy"  "+1 347 535 1234"  "+1 994 555 5678"
America/Los_Angeles   "william@example.com"  "William Kidd"    "+1 330 555 5678"  "+1 305 555 1234"
America/Los_Angeles   "israel@example.com"   "Israel Hands"    "+1 507 555 1234"  "+1 208 555 5678"

Now when I need to coordinate a meeting, I can run the utility with the -T option to see each team member’s local time.

With the -U option, you can display each contact separated in groups based on UTC offset.

Let us know what you think and if you found this tool helpful.