Solving High-Resolution Video Stutter with GStreamer Hardware Acceleration

2026-05-18T00:00:00+00:00

In April, a client at Auburn University reached out to VisionPort Support to ask us to look into an issue with video playback for an upcoming presentation. They were running into performance issues attempting to play a video matching the full resolution of the seven-screen video wall, across all screens.

We have other clients playing video across all screens normally, on the same hardware and software, with no issue, so this was a strange report to the team. The seemingly simple request ended up leading us down a rabbit hole into how video playback works on the VisionPort platform and ultimately led to some serious modernization of our video rendering stack, the process of which we would like to share with you today.

Why was a system designed to play high-resolution video seamlessly across seven screens performing so poorly?

The System: A GStreamer Pipeline

The VisionPort video player is a custom application built on top of the GStreamer framework, initially developed to synchronize video across separate computers on our legacy Liquid Galaxy hardware configurations.

Each instance of the player renders a single video. This can be the full 7560x1920 canvas size of a seven-screen VisionPort, or often various subdivisions of the full resolution that cover two or three displays. Although the host hardware of our modern servers is theoretically sufficient, in practice the pipeline was overwhelmed.

High-level overview of the content playback flow for VisionPort systems

First Instinct: The Encoding

Our initial hypothesis was that video encoding was to blame. Encoding impacts file size, quality, and playback compatibility, and we have previously encountered issues with certain Apple workflows, videos exported from Final Cut Pro, for example. Although nothing about the original video stood out to our team as improperly configured, we re-encoded the source using an H.264 codec and a well-formed container, and tested reducing the resolutions of the video.

The stutter persisted consistently across the test videos, ruling out encoding as the root cause. In fact, the lower-resolution videos still had stuttering issues, furthering our confusion.

Hardware Decoding, Take One

We next considered that the pipeline was running entirely on the processor (CPU) instead of utilizing the ample graphics card resources (GPU). Although the video player was originally designed to support hardware decode elements, software decode had been the default configuration for a number of years.

Architecturally, the video player runs from within an application layer deployed to the host via a Docker container. The host features an NVIDIA graphics card, so our logic was that leveraging the GPU for this high-resolution video should, in theory, reduce bottlenecks induced by running everything off of the CPU.

Implementing hardware decode was a project in itself. We discovered that the original legacy decoders present in the video player, such as vaapidecodebin and vaapipostproc, were not available to our server’s CPU. NVIDIA’s accelerated decoders are not part of GStreamer’s default plugin set packaged on the Ubuntu base that we use, meaning we had to identify and install the correct package source and plugin in our Docker images. The necessary plugin package, via apt, ended up being gstreamer1.0-plugins-bad.

Lastly, we needed to resolve Docker-related resource-visibility issues to ensure the container could access the GPU. That meant using the NVIDIA Container Toolkit and passing the correct environment variables to the container so that it could utilize the GPU resources.

Once properly configured, and by enabling hardware decode in the launch command, we were able to determine that hardware decoding was functional.

A few useful commands to check whether the expected resources are present in the container:

nvidia-smi
# Checks if the container can access the GPU at all.

nvidia-smi dmon -s u
# Provides an ongoing report of what GPU features are being utilized over time,
# so we can see whether the decoder is being used.

gst-inspect-1.0 nvcodec
# Gets a list of supported features present for your current implementation of
# GStreamer with the NVIDIA codecs. You can also run the command without a
# specific feature to list every supported feature present.

After finally confirming that hardware decode was functioning, we discovered that playback had improved somewhat, but the stutter was not fully resolved.

The Real Culprit: Rendering, Not Decoding

At this point we were stumped, so we tried playing the video in another player we had available on the system, ffplay. The video appeared to play back somewhat normally.

We reported this to our developer, who decided to enable GStreamer’s debug output. The logs were filled with quality-of-service messages, indicating that one or more pipeline elements could not process frames quickly enough, causing downstream stages to drop or delay them.

The QoS pattern suggested the bottleneck was in the sink, not the decoder. The video player’s GStreamer autovideosink element was defaulting to xvimagesink, which relies on the CPU for color-space conversion and video texture handling. At 7560x1920px resolution and 60 fps, the CPU was unable to keep up with rendering the large amount of data. It turns out that the source of the issue had more to do with the framerate of the high-resolution video than with the resolution itself.

The host had glimagesink available, which is an alternative sink that uses OpenGL for GPU-based color-space conversion and texture upload. By explicitly specifying glimagesink instead of relying on autovideosink, we assigned rendering to the appropriate hardware.

This single change restored smooth playback at the target resolution.

Putting the Pieces Together

Even though we had a solution for the issue at hand, there were a few more details to sort out to make this universal for all media going forward. It turns out that the NVIDIA accelerated H.264 decoders for the hardware generation we use top out at 4K, meaning that H.264 encoded videos larger than 3840x2160 would not be able to run on the GPU even if we specified use of the decoder. Luckily, our implementation was robust enough to swap back to decoding on the CPU in the case of a fault.

Additionally, the H.265 decoder supports higher resolutions, comfortably handling 7560x1920 and allowing for future scalability.

The complete pipeline now runs the GL-backed sink with the option to enable hardware decode for H.264, H.265, VP8, VP9, or AV1, all common encoding formats. The format is auto-detected and will utilize the decoder if the content is acceptable. The CPU render is no longer a bottleneck for decoding or rendering, allowing videos to play back at the full 60 fps, even at higher resolutions.

Beyond One Engagement: Broader Implications

The Auburn University request prompted us to reevaluate our video player stack, particularly a legacy renderer choice and plugin-installation pattern from prior to our move to containerized deployments. Not mentioned in our process above were bug fixes to fix potential memory bugs and improvements to our testing process to simplify future changes.

These improvements are now incorporated into the upcoming VisionPort Library MQTT migration of the VisionPort application layer. New deployments will default to hardware-accelerated decoding via NVIDIA decoders and GL-backed rendering via glimagesink, rather than defaulting to the CPU path with the old renderer.

Key takeaways include:

Instrument first if a video pipeline is stuttering. GStreamer’s verbose debug output, particularly QoS messages, identifies pipeline bottlenecks.
Do not assume the autodetected sink is optimal. autovideosink is frequently adequate, but check the available sinks for your operating system.
Hardware acceleration in a container requires more than package installation. The container must have GPU visibility and the correct plugin set, which may be its own task.
Decoder and renderer can be independently CPU- or GPU-bound. Ensuring both are GPU-accelerated is crucial for high-resolution pipelines.

Working with End Point

If you’re facing video playback or multimedia pipeline performance challenges, whether with GStreamer, in a containerized environment, or across a multi-screen VisionPort-style installation, End Point’s Immersive and Geospatial division has extensive experience with these issues.

The GStreamer documentation is a valuable resource, and we are available to discuss our insights from similar deployments. Get in touch to set up a consultation.

Programming the Intel NDP in 1983

2023-03-12T00:00:00+00:00

The Beginning

I graduated from St. John’s College in Annapolis in 1980. It was an intensive four-year education in math, science, language, poetry, and philosophy. Two years later, I took four computer classes at a community college, and got my first IT job in 1983 at the beginning of the personal computer revolution.

There were two of us: Steve, the owner of the company, and I, working literally in his garage. I was just a fledgling, uncertain and doubtful of my own ability. The IBM PC had come out the summer before, a device IBM seemed to regard as little more than a toy. Steve was by profession a physicist.

Steve noticed the PC had an empty socket on the motherboard, next to the Intel 8088 CPU. He guessed it was for Intel’s 8087 Numeric Data Processor (NDP), also known as a math co-processor, that was designed as a companion to Intel’s 8088/86.

The CPU could operate perfectly well on its own, but if the NDP was installed, they would both read the same code stream. The CPU would ignore NDP instructions and let the NDP execute them. The NDP would ignore non-NDP instructions and let the CPU execute them. While the 8088/86 is running code, it can’t do anything else. With the NDP, true parallel processing is possible.

The Plan

IBM selected Microsoft to produce its systems software: operating system, compilers, linkers, libraries, macro assembler, etc. Microsoft not only ignored the 8087, it devised a floating-point format incompatible with it. Steve had signed a contract with Intel to be the exclusive distributor of the NDP for a period of years—Intel wasn’t selling any to speak of, so why not?

Steve’s business plan was to sell a package: an 8087 chip and modifications to IBM (Microsoft) systems software that builds programs (BASIC compiler, FORTRAN compiler, Pascal compiler, and Macro Assembler) so they can generate code for it, and libraries in which the math routines use the 8087 instead of complex, slow math routines in the CPU.

With the package we created, math-intensive applications would be sped up by anywhere from 2–3 times up to 40 times. The financial advantage to those who wrote their own applications in, say, BASIC, and used our package was compelling.

The Platform

The computer I was given was a standard IBM PC of early 1983: an Intel 8088 microprocessor running at 4.77 MHz, two 360 KB 5¼-inch full-height floppy disks, 640 KB of RAM, a monochrome monitor, five expansion slots, and PC-DOS 1.1.

The 8088’s 16-bit architecture internally was essentially the same as the 8086, with the difference that it used an 8-bit bus and was cheaper.

IBM designed an open architecture. In other words, it designed the PC to use all off-the-shelf equipment from other manufacturers, not IBM: chips from Intel, systems software from Microsoft, other chips from elsewhere. This was to be my saving grace, as we shall see below.

Two Problems, Two Solutions, one Package

Our guiding design idea was that our users could develop programs in the way they were accustomed to. They could, for example, use the BASIC Interpreter that came as part of the PC ROM BIOS for an iterative, live mode of development. When satisfied, they could save the resulting text file, build it with our patched version of the BASIC compiler and link it with our libraries. They would have bought an 8087 from us, plugged it into the coprocessor slot, and then could launch their executable.

One customer, for example, was in Kansas. They were commodity traders who had an algorithm written in BASIC. It would take data from all commodity trades during the day and at the end of the day, run the data through their algorithm, and use the result to place trades first thing the next morning. It took 5–6 hours to run, and they got home to their families at 10 or 11 o’clock at night.

With our software modifications and the 8087, it took only about 2 hours to run, so they were home in time for dinner.

Too Naive To Fail

When I first sat down at my PC to work, I didn’t know how to turn it on.

To do my job I had to program in Intel 16-bit assembly language. I not only had never seen any assembly language, I had never seen any computer language before besides FORTRAN and PL/I.

I was somewhat like the bumblebee. Mathematics has proven that the bumblebee is aerodynamically unable to fly. The bee, however, does not know math, so it flies anyway.

I sat at my desk and grappled with the complexity before me. At the end of the first week, I felt so hopeless and lost that I thought I would speak to Steve and say, “Steve, when we interviewed I misled you. I pretended I am not an idiot. Let’s say no more about it. I will slink away silently. You don’t even have to pay me.”

Then I thought, “Steve is a pretty smart man. I will leave it up to him. If he thinks I should leave, he can fire me. Meanwhile, I will stay here and give it everything I’ve got.”

My Tasks

The 8087 had a stack of eight ten-byte floating point registers. It could load and save either four-byte (single) or eight-byte (double) floating point. Steve had written a disassembler using the BASIC interpreter, so I had a printout of the code in the BASIC library. It did not show the source code, but it did show labels of routines and variables.

Steve had a list of the names of all the math routines in the library. I had to edit each math routine so that it would perform the same functionality as the original, but using the 8087.

A second requirement had to do with the floating-point format of constants in the source code. The Microsoft compilers generated them in MS-Binary floating point, in the structure exponent + sign bit + mantissa (or significand). The 8087 used IEEE floating-point, which was structured sign bit + exponent + mantissa. We patched the compilers so that at the place they would generate floating-point constants, they would generate IEEE-format constants instead.

My Advantages

IBM shipped each PC with two ring binders. One explained the BASIC language as implemented in the ROM BIOS. The other was a handbook for the PC, which told everything a person could possibly want to know about the machine, including a printout of the ROM BIOS assembly language source code at the end of the manual: page after page of working code in Intel 8088/86 assembly language.

I also had an Intel product manual on the 8088/86, including a section on the 8087. That was some of the most dense specifications on the planet, but it contained all the operations and registers, both for the CPU and the NDP.

Another advantage I had was the debugger, DEBUG.COM. I would edit a math routine to replace the operations in the CPU with a few simple opcodes for the NDP, since it did all its operations internally, allowing the programmer to use a few NDP instructions to carry out the functionality. I would then use the librarian to replace the given routine in the library with my routine, write a test program in BASIC and build it. Then I would load the executable in the debugger and step through the code. That had two big advantages: One, I got to see the assembly language in action, learning more than a book could ever teach me. Two, I verified that the code actually did what I thought it did. The code we shipped rarely had a bug in it and those were usually minor. Except one.

Whoops!

There was one major bug: error handling. I became aware of it when a customer called in about the failure of our code to catch a division by zero. Some research showed that the first 1024 bytes of RAM contained the Interrupt Vector Table (IVT), 256 four-byte entries containing the segment (two bytes) and offset (two bytes) of the Interrupt Service Routine (ISR). Interrupt INT 0 contained the address of the CPU divide by zero ISR.

Our software, since it did not use the CPU for division, did not catch a division by zero. This error would generate an IRQ (interrupt request) 13 which Intel had reserved for the math coprocessor. In Intel’s product spec, they marked IRQ 13 as Reserved. The IBM engineers, however, had always seen the word “Reserved” used for things reserved by IBM, so they used IRQ 13 for something else. So I had to write an ISR that would take over the IRQ 13 entry in the IVT, check to see if there was an error in the NDP, and if so, handle it. Otherwise, pass control onto the original ISR.

Eventually there was a DOS routine to create a chain of ISRs, each of which would determine if the interrupt was something it should handle or else pass it on to the next ISR in the chain. Good cyber-citizens would play nice, and use the DOS routine to get their place in the chain.

They Think They Didn’t Get the Right Test

The company expanded, leaving the garage far behind and moving into a brick-and-mortar building, one of many at a defunct rope-factory. It employed by this time 40–50 people.

Intel had marched forward, producing the 80286 (with its companion 80287) and the 80386 (with its companion 80387).

Steve asked me to write a test for the math chips. I wrote it in C and Intel 16-bit Assembler. The test performed the four standard arithmetic operations—add, subtract, multiply, divide—and also special operations the NDP was capable of, like exponents, logarithms, square roots, trigonometric functions, etc.

The code tested whether the answers calculated by the chip came out within a narrow range, since it is not possible to test for an exact result with floating-point numbers that can be “right” but differ in the least significant bits of the mantissa. I named it 87TEST.EXE. It shipped with every NDP we sold: 8087, 80287, 80387.

We began getting calls from customers who had bought an 80287 or 80387, complaining that they had received the wrong test. Someone would have to explain that the same executable with the same code tested all variants of the NDP. These calls were costing the company money to reassure the customers, so I wrote a new version of the test which determined which CPU and which NDP it was running on.

The OS was MS-DOS, a character-based system. I used the advanced ROM-BIOS character set to “draw” images representing both chips, each casting a “shadow” to seem three-dimensional, and each labeled with the number of the chip. For example, one “image” could be labeled 80286 and the other 80287.

In order to gain customer confidence these had to be accurate. Later models of these chips had instructions whereby you could ask the chip, “What are you?” and it would return its version number. Earlier variants did not have these instructions, but for the most part I found characteristics I could test for to determine the version.

With the 8088 and 8086, however, they both used exactly the same instruction set. I racked my brain to find a difference and could not. I turned to my dog-eared copy of the 8088/86 Intel product manual and pored through it. Finally, I found a difference: the 8088 had a four-byte prefetch queue and the 8086 had a six-byte one.

I solved the problem by writing self-modifying code, which was a huge no-no according to Intel. It used 6 bytes in the code segment, with a constant defined in the 6th byte. I cleared the prefetch queue by doing a jump to the lead instruction. Thus, the prefetch queue would contain four bytes and the sixth byte would still be in memory (8088) or all six bytes would be in the pre-fetch queue (8086). The first instruction moved a different constant into the 6th byte. In an 8088, the 6th byte in memory would be changed and in the 8086 the 6th byte would be modified in the prefetch queue and remain untouched in memory. Then I would test that 6th byte: changed = 8088 and unchanged = 8086.

At last my search to correctly identify the CPU and its companion NDP was over. The test program was complete.

Summary

I had to make do with the Intel product manual, the IBM PC manual, the IBM debugger, and tools my own company had created. Google did not exist, and for that matter, the Internet as we know it today did not exist. There was no plethora of instructional videos, as are widely available today.

Success required above all persistence and resourcefulness. I do not deny that native ability and education are important, but problem-solving in the IT world succeeds when we accept a challenge and strive to achieve what we never thought possible.

We adapt, we improvise, we overcome. Failure is not an option!

References

Data Center Relocation Initiatives: Daunting But Achievable

2018-12-20T00:00:00+00:00

Photo by Seeweb · CC BY-SA 2.0, cropped

Introduction

A customer asked for our help dealing with logistical nightmares they encountered during a hardware update and data center relocation project. The customer had two active data centers, and wanted to relocate one of them to a modern Tier 4 facility to improve its performance and provide redundancy for critical systems. They also wanted to consolidate or decommission equipment to reduce recurring expenses and reduce their carbon footprint. The client decided to move to a Tier 4 data center because they provide across-the-board redundancy within the data center.

Moving forward with Tier IV

Enterprises typically transition to Tier 4 data centers because they offer the highest uptime guarantees, and have no single points of failure. These facilities are fully redundant in terms of electrical circuits, cooling, and networking. This architecture is best able to withstand even the most serious technical incidents without server availability being affected. Tier IV facilities have contracts with disaster management companies who will provide them, for example, with fuel in the event that a natural disaster damages the power grid.

(More information about the four data center levels is at the end of this post.)

Key Issues

Some of the customer’s key concerns were:

1. To safely move their virtual environment to the new data center.

2. To protect assets during the relocation.

3. To completely shut down the current data center and migrate seamlessly to the new data center.

4. To update external DNS records without causing any downtime for the web applications used by their customers.

Project Planning

To manage data center relocations is challenging because there are many moving parts and variables. Typically, planning starts 6 months ahead of the scheduled relocation date. We seek to understand the role of each system by consulting in-depth with all the teams involved. We analyze each part of the client’s tech infrastructure, break it down, and strategize about the best approach for handling the migration of each component. Additional time to plan can help especially when a large group of stakeholders is involved. Successful planning requires hard and sometimes extensive work, but makes an otherwise overwhelming data center migration smooth and worry free.

Here are some of our procedures for pre-migration preparation:

1. Process the paperwork for new hardware. Identify, review, and approve all applications. Hire movers. Schedule and coordinate with internal management and key stakeholders like IT staff, system admins, contractors, circuit providers, etc.

2. Collect and analyze resource usage associated with the critical and non-critical systems and compare resource needs to resource availability at the new facility.

3. Design and schedule the migration of critical systems to the new data center. Discuss the plan with key stakeholders.

4. Make sure replicas of critical systems are up to date.

5. Set up a new environment with new hardware in the new data center and establish network connections.

6. Prepare hardware and virtualization technology in the new data center.

7. Test, retest, and burn in the new hardware (2–4 weeks of testing).

8. Create a non-essential server within the virtualization environment and test it.

A Challenge: Eliminating Downtime

To minimize or eliminate downtime requires very careful preparation. With this particular client, we had the luxury of their already having a secondary data center, which could accommodate a majority of the critical systems during the migration.

We originally planned to leverage this by simply changing the external DNS records to point to this data center during the migration, but during planning determined that the external DNS record would not replicate worldwide quickly enough to meet the uptime needs of their global operation.

Solutions

After discussing various approaches, we ended up using a load balancer. A load balancer is a network device that routes incoming traffic destined for a single destination (web site, application, or service) and ‘shares’ the incoming connections across multiple destination devices or services. In this case, the data center we were to migrate to housed the client’s critical systems like web servers, database, etc. The secondary data center housed a replica of the critical systems using a replication technology, e.g. Actifio, Rubrik, SAN mirroring, VMWare SRM, or Zerto — we used Actifio in this scenario.

The customer had multiple systems with external DNS records and public IP addresses. We did not change the public DNS names for the critical systems. We did change the public IP addresses associated with the public DNS names, to point to the load balancer, which then pointed to the appropriate data center throughout the process. This setup enabled a gradual, flexible, and fully controlled transition to the new data center.

Summary

Our client wanted to reduce their servers’ power usage 40% by consolidating into a hyper-converged system. Other goals were to reduce floor space by 50%, and monthly recurring data center charges by 50%. We transitioned them to a new hardware stack (compute, storage, network, backup environment) in a Tier 4 data center, and saved them 50% percent on data center operation costs. Additionally, during the process we corrected issues associated with disaster recovery and business continuity, leaving the client’s operations more resilient, affordable, and more secure.

Appendix: Data Center Classifications

Tier IV (Best)

Tier IV data centers have redundancies for every process and data protection stream. No single outage or error can shut down the system. To keep Tier IV ranking, data centers must have 99.995% minimum uptime per year. They come equipped with 2N+1 infrastructure (two times the amount required for operation plus a backup), which is considered “fully redundant.”

Tier III (Average)

N+1 (the amount required for operation plus a backup) fault tolerance, which means a Tier III facility can undergo routine maintenance without a hiccup in operations. 99.982% minimum uptime per annum, used for maintenance and overwhelming emergencies.

Tier II (Minimal)

99.749% minimum uptime. Tier II facilities have considerably more downtime than Tier III facilities, primarily because Tier II facilities have less redundancy. They do however have partial cooling redundancy and multiple power redundancies.

Tier I (Basic, Cheapest)

99.671% minimum uptime, with 28.8 hours of downtime each year.

Hue’s on First: How we used responsive bulbs to join software and hardware for a busy medical practice

2016-03-14T00:00:00+00:00

In 2014 we began working with a busy bariatric surgery office in Long Island to create a system that would allow the practice to better manage doctor paging and patient wait time. By placing a responsive, color-coded light bulb and tablet outside each examination room, the staff could see which rooms were empty, which were occupied by a patient waiting on a specific doctor, and in which a doctor-patient consultation was in process. Outside each room is a tablet with information including the patient number, the attending doctor’s name, and the wait time.

In addition to providing a comprehensive, granular paging service for doctors, Fast Track also provides feedback to the practice. This feedback includes average patient wait times per doctor, per time of day, and per procedure. This allows the practice to make necessary changes and increase patient satisfaction and peace of mind.

I asked Danny Divita, one of the main developers on this project, to tell us more about the Hue/ FastTrack interface.

LF: Describe the project for which we used Hue bulbs. What were all the pieces that needed fitting together?

DD: The Hue bulbs are being used for a bariatric clinic to alert the staff when pages are sent to of pages to specific rooms. Alongside the lights the API also has to work with a RF device-locating API. The RF API had to be coordinated with the Hue light API to coordinate changing colors, alerts, and states.

LF: Why were Hue bulbs a good solution for the problem of creating an easy-to-understand, integratable visual alert system? Did you consider any alternatives?

DD: From a cost perspective and ease of integration the bulbs were a good solution for this project. There are other solutions, but they are more costly and the integration would have taken longer.

LF: How did you hear about Hue bulbs? Was the API easy to access and work with? How was using Hue’s testing environment?

DD: The client suggested the bulbs during their research before having us develop their application. The API is extremely easy to work with and well documented. The Hue API has a built-in testing feature that made it easy to understand the API and develop around it.

LF: What was the most difficult thing about integrating this technology? Can you think of any other commercial implementations of these bulbs? Specifically in the medical industry?

DD: The most difficult aspect was trying to come up with a good alert (blinking) feature. The API has some built in alerts, but they are limited to a set timed interval. We had to develop around this to ensure the alerts stayed constant and did not time out. Another application for these lights in the medical field could be to indicate severity of a situation or condition. Because the lights can change color, the applications can signal the proper response for the staff.

LF: Did you write any new software for this project? Is that work available to other developers?

DD: We used a third party library that helps to manage stateless workflows. Our application expanded upon this library to design a workflow engine that manages paging the staff, Hue lights, and RF device locating.

Nvidia: Invalid or Corrupted Push Buffer Stream

2015-04-20T00:00:00+00:00

As a high-performance video rendering appliance, the Liquid Galaxy requires really good video cards—better than your typical on-board integrated video cards. Despite ongoing attempts by competitors to displace them, Nvidia remains the best choice for high-end video, if you use the proprietary Nvidia driver for Linux.

In addition to providing regular security and system updates, End Point typically provides advanced remote monitoring of our customers’ systems for issues such as unanticipated application behavior, driver issues, and hardware errors. One particularly persistent issue presents as an error with an Nvidia kernel module. Unfortunately, relying on proprietary Nvidia drivers so as to maintain an acceptable performance level limits the available diagnostic information and options for resolution.

The issue presents when the system ceases all video output functions as Xorg crashes. The kernel log contains the following error message:

2015-04-14T19:59:00.000083+00:00 lg2 kernel: [  719.850677] NVRM: Xid (0000:01:00): 32, Channel ID 00000003 intr 02000000

The message is repeated approximately 11000 times every second until the disk fills and the ability to log in to the system is lost. The only known resolution at this time is to power-cycle the affected machine. In the error state, the module cannot be removed from the kernel, which also prevents Linux from shutting down properly. All affected systems were running some version of Ubuntu x86-64. The issue seems to be independent of driver version, but is at least present in 343.36 and 340.65, and affects all Geforce cards. Quadro cards seem unaffected.

The Xid message in the kernel log contains an error code that provides a little more information. The Nvidia docs list the error as “Invalid or corrupted push buffer stream”. Possible causes listed include driver error, system memory corruption, bus error, thermal error, or frame buffer error. All affected systems were equipped with ECC RAM and were within normal operating temperature range when the issue presented.

Dealing with bugs like these can be arduous, but until they can be fixed, we cope by monitoring and responding to problems as quickly as possible.

Leap Motion Controller + Liquid Galaxy

2013-04-23T00:00:00+00:00

We’re constantly trying out new things with our Liquid Galaxy at End Point: new content types, new remote management scripts, new hardware. The good people at Leap Motion sent over one of their hand motion controllers and we couldn’t wait to get it unpacked and driving the Liquid Galaxy that sits in our office.

With access to developer tools, we were able to configure the Leap Motion and update the device firmware. Then, after a quick installation of the latest Google Earth build v 7.1.1.1550 that includes Leap Motion controller support, we were ready to test out the new device on our 7-screen Liquid Galaxy.

Watch the video and see for yourself—we think the combination is fantastic! That superhero feeling of flying around the world with a Liquid Galaxy is only enhanced that much more by now commanding the movement of the planet with subtle flexing of your fingers. (That’s Kiel in the video—personally, I think he looks like Magneto without the helmet.)

End Point continues to push the boundaries on what the Liquid Galaxy can do. We want to experiment some more and see what else might be possible with the Leap Motion controller for Google Street View, panoramic photos, videos, and maybe even games. If you’ve got an idea, or if you have a cool place where this might be deployed, please contact us!

Are you sure you want a MacBook Pro?

2012-07-30T00:00:00+00:00

The “UltraBooks” trend has been been made popular by Apple’s efforts at making the computer an appliance. There are clear benefits to this, but for enthusiasts, it means less control to upgrade to hardware of their choosing. Let’s contrast this experience with my own Dell Vostro 3500. It was purchased two years ago and yet through upgrades, it has many features the recently refreshed MacBook Pro with Retina display lacks, while offering the same processing power.

MacBook Pro Advantages

Let’s be clear, the MacBook Pro (MBP) has a number advantages over my Dell. The Retina display of course trounces my 1376×768 resolution on a 15" display. For designers, photographers, and video editors, Retina is a great asset. For someone who spends their day inside Vim and SSH, not so much. Similarly, the MBP’s seven hour battery life is impressive, although I’m sure with my processor intensive work load, it would be lower. But for me, I’m not terribly mobile. I need to be mobile from room to room perhaps, but I don’t find myself without power and needing to complete mission critical work less frequently than my replaceable 90 watt-hour battery can provide over 3 to 4 hours. Similarly, weight is not my number one concern; I simply stash the laptop in my backpack and go.

Incremental Upgrades

With my Dell, I purchased the baseline Vostro 3500 for $1,000 which included next business day service and accidental damage protection for three years. This means for parts which I cannot easily replace myself, like the display and motherboard, I’m covered. Over the years, I’ve upgraded the:

processor to an i5-580M, with Turbo Boost up 3.3GHz. Great for single threaded Ruby ($225).
wireless card to an Intel Wifi Link 5300 for better signal strength and 802.11n support ($42).
65whr battery at its end of life with a 90whr battery ($160 ouch!).
original HDD with a 160GB Intel 320 series SSD ($220).
optical drive with a 2.5” drive caddy ($52) and a 500GB Seagate Momentus XT Hybrid drive ($150).
RAM to 8GB ($99).

For $950 of upgrades over two years, I’ve got a machine with specs that are powerful in the places I care about, at a price I can afford and pay for incrementally. To get the same processor and warranty on the MacBook Pro, I need to shell out $3,150 and give up my eSATA, HDMI, VGA, PCI-Express, Gigabit Ethernet ports and 500GB of storage (upgradable to 1TB for $100). In short, I’m expected to pay a premium price for features which aren’t of my choosing, while giving up ports.

While this might not be possible for all people, developers should look long and hard at what it is they value in their workstation and not blindly accept whatever Apple hypes. You might find you don’t care quite as much about the “features” Apple wants you to.

Guide to Ubuntu 11.10 on a Samsung Netbook

2012-04-08T00:00:00+00:00

12.04 UPDATE: Unsurprisingly, after installing 12.04 which includes the 3.2.x kernel, brightness controls work perfectly out of the box. Sadly, it’s still necessary to tweak modprobe to get wireless working after a suspend/resume. Also, don’t forget to reboot after making the changes!

After reading a few too many reviews about netbook hardware and its compatibility with Linux, I settled on a Samsung NF310 netbook. However, like all things worth doing, it wasn’t nearly as easy as I’d hoped. This post highlights some of the lessons learned from my first days with Linux on my Samsung netbook.

Pre-Installation

I had been eagerly awaiting the arrival of my new hardware and had already gotten a fresh copy of Fedora 16 ready. The crippled Windows 7 Starter Edition was going to be off that netbook as a first order of business, as a matter of principle. If I had it to do over again, I would have installed the Windows-only BIOS update for the Netbook first. I’m working through using BartPE, but the installation onto the USB drive hasn’t gone well. Best to do these kinds of Windows-only activities while Windows is still easily available.

Additionally, I would have used YUMI multiboot USB creator to put on a copy of Ultimate Boot CD as well as a handful of live Linux distros for testing. When I later had issues installing, I found myself wondering, is it the hardware? I ended up going back and running the tests (which passed), but had I done this first, I would not have been distracted from the real problem.

Installation

Aside from choosing a distro, I found getting the installation to complete successfully to be a chore. It seems the disk’s partition table was GPT and so both Ubuntu and Fedora’s installation processes failed.

ext4 Filesystem check failure on/dev/mapper/vgsamsungnetbook-lvroot:

Operational error.

Errors like this usually mean there is a problem with the filesystem that will require user interaction to repair. Before restarting installation reboot to rescue mode or another system that allows you to repair the filesystem interactively. Restart installation after you have corrected the problems on the filesystem.

You can see how an error like this might prompt you to run hardware checks. To work around this, simply fire up one of the Live CD installations on your YUMI stick and offer Linux the following incantation:

parted
mklabel msdos
quit

Test Driving Fedora 16

After getting past installation issues, I loaded Fedora 16’s XCFE spin. Despite the dual core Atom processors and the 2GB RAM, I was looking for a lightweight distro; snappy UI and battery life were my first concerns and I knew less demands from the OS would leave more resources for the work I wanted to do.

The sparse XCFE desktop experience offered few bells and whistles but exactly what (I thought) I wanted. It was responsive and stayed out of the way. However, I quickly discovered that after resuming from suspend, my wireless connection would not work. Was this Fedora? Was this the kernel? Was this hardware failure? Only trial and error would help me figure it out…

Troubleshooting wireless connectivity after resume

The Samsung came with a Broadcom chip, which was recognized by Fedora, but I have a low opinion of Broadcom hardware and know that its driver support is relatively new in the kernel mainline. I noticed kernel errors related to the driver and decided to swap out my other laptop’s Intel Ultimate N Wifi Link 5300 card and see what happened. To Linux’s credit, it detected the new hardware without a hitch and connected right away after booting. However, upon another round of suspend/resume, wireless connectivity couldn’t be established.

Next, I decided to see if this could be a distro issue. I tried Ubuntu 11.10 to find this was not an issue with a particular distro. Of course with both attempts I fully patched all the software, so I didn’t think it a kernel issue; Fedora was running the 3.3 kernel!

Sadly, it’s 2012 and Linux still can’t handle simple power management issues out the box. Fortunately for you, lucky reader, I have the magic incantations you need to have wireless network connectivity after suspend/resume. I knew I couldn’t be the only one experiencing the issue but was simply blown away this wasn’t better documented. As always, log files tend to be the best source for searching Google; results seem more accurate and from more authoritative sources. Here’s what I was seeing from /var/log/kern.log after resuming:

kernel: [  326.394858] wlan0: direct probe to 00:13:10:9e:88:5f (try 1/3)
kernel: [  326.592242] wlan0: direct probe to 00:13:10:9e:88:5f (try 2/3)
kernel: [  326.792214] wlan0: direct probe to 00:13:10:9e:88:5f (try 3/3)
kernel: [  326.992198] wlan0: direct probe to 00:13:10:9e:88:5f timed out

Fixing Wireless Connectivity after suspend/resume

After a long and winding hunt through Launchpad bug reports and unhelpful forum threads, the first step is to identify which kernel module is providing your wifi driver.

lspci -k | grep -i -E "network|wireless|wifi|wlan|802.11" -A3

#my output...
05:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
 Subsystem: Intel Corporation Device 1101
 Kernel driver in use: iwlwifi
 Kernel modules: iwlwifi

On my machine, it was the iwlwifi module responsible for wireless. Armed with the module name, we now need just one simple line, in one simple file:

# /etc/modprobe.d/iwl.conf
options iwlwifi bt_coex_active=0

With that simple modification AND A REBOOT—Linux sanity is restored. The wireless works perfectly (and quite quickly) after resume.

For those who don’t happen to be using the iwlwifi module, it’s not obvious what the modprobe.d config file naming conventions are, but in general, Googling for your kernel module name plus bt_coex_active=0 should yield something useful.

Samsung screen brightness control, out of control

After having made the switch to Ubuntu to help troubleshoot my wireless issues, I found that the screen brightness would occasionally start randomly cycling through various levels repeatedly and rapidly. To stop this behavior temporarily, I found I could jump to a tty session by pressing ALT+F2 and return to X with ALT+F7.

For a more permanent fix, a kernel upgrade was required. While, Ubuntu 11.10’s stock 3.0.x kernel did not include the fixed “samsung_laptop” modules, the 3.2.x kernel did. Fortunately, with the upcoming 12.04 release, the 3.2.x kernel is actively being packaged for Ubuntu and can be downloaded from kernel.ubuntu.com. More specifically, you can browse the available kernel packages and choose your release of choice. Because I knew 3.2.x was going to be supported for the 12.04 release and I was looking for stability and didn’t see any must-have features in the 3.3.x kernel, I went with these downloads and installations:

# 64 bit installation
su
cd ~
mkdir src
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-headers-3.2.14-030214_3.2.14-030214.201204021356_all.deb http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-headers-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.14-precise/linux-image-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb
dpkg -i linux-headers-3.2.14-030214_3.2.14-030214.201204021356_all.deb linux-headers-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb linux-image-3.2.14-030214-generic_3.2.14-030214.201204021356_amd64.deb

reboot

After updating the kernel the screen brightness worked flawlessly and updating the kernel this way seemed more reasonable than compiling. However, I did notice that Update Manager wanted to reinstall the stock kernel. Running these lines as root will prevent that:

echo linux-generic hold | dpkg --set-selections
echo linux-headers-generic hold | dpkg --set-selections
echo linux-image-generic hold | dpkg --set-selections

Choosing a Distro and Desktop Shell

While I had originally planned to use Fedora 16 XFCE spin, I find myself wanting to stay with Ubuntu and Unity. The polish in the interface may be worth a bit of lag, and it seems it’s easy enough to switch shells at login with the little gear icon next to your username. You can easily add additional shells and experiment with what works for you.

Unity2D works reasonably well, despite a bit of delay. Moreover, adding lenses to Dash seems like it will make the interface even more usable. I guess I did want some of the bells and whistles, but not always, especially when I need to save battery life. But isn’t this kind of flexibility the point of Linux and Free software in general?

Closing thoughts: MacBook Air alternative?

Several times over the last few days of banging my head against the wall, I found myself thinking about Apple’s MacBook Air. It’s hard to argue the hardware is world class and OS X can be made to do most things a Linux distro can, while providing rock-solid power and device management with a slick UI. However, I just couldn’t bring myself to shell out $999 upfront, with the obligatory because-you’re-on-your-own-without AppleCare ($249), plus all the little Apple lock-in-gotchas that I knew were down the road.

Instead, I paid $315 for hardware that I can actually modify with additional batteries ($75/each/4hrs life) and a SSD upgrade ($85 for 40 GB). I guess the bottom line is, if you’re a developer, or a system administrator, it may seem like the Apple product is a way to avoid the pain of running Linux on something other than a server. However, by deciding to fight the good fight, I’ve saved $773 (after planned upgrades!) but spent 20 hours of time, which means I paid myself about $40/hour to learn a few things. That’s a great rate of return and now that the pain is over, it all seems worth it.

Two Cool Things about Liquid Galaxy

2010-10-04T00:00:00+00:00

I. It uses COTS Hardware.

Liquid Galaxy is suitable for using with COTS (Commodity Off The Shelf) hardware. Yes, Google Earth itself is rather resource intensive, so it helps performance to use feisty computers (including ones with SSDs) but it’s still COTS hardware. Of course the very cool thing about using COTS hardware is that the price is right and gets better all the time.

II. A simple, elegant and powerful master/slave configuration and communication approach

Liquid Galaxy works by configuring its “slave” systems to have offsets from the point of view of the master system that the system’s user navigates on. The slave systems “know” their locations relative to the master system. The master system broadcasts its location to the slaves via UDP packets. It’s then up to the slave systems to figure out what portion of a Google Earth globe they need to retrieve themselves relative to the coordinates broadcast from the master system.

With this approach it’s easy to scale to a large number of slave systems. An interesting extension to this configuration and communication approach that the Google engineering team for the project provided for is the ability to configure one or more remote Liquid Galaxies to remotely mirror the views being displayed in a given “master Galaxy”. This will allow teams of users to remotely view the same Google Earth views within the awesome environment of distinct Liquid Galaxies. The remote Liquid Galaxies essentially “play” the same views as are seen in the master Galaxy, but negligible network traffic is passed from system to system.

Sometimes it’s a silly hardware problem

2008-12-19T00:00:00+00:00

I’ve been using Twinkle and Ekiga for SIP VoIP on Ubuntu 8.10 x86_64. That’s been working pretty well.

However, I finally had to take some time to hunt down the source of a very annoying high-pitched noise coming from my laptop’s sound system (external speaker and headset both). I have an Asus M50SA laptop with Intel 82801H (ICH8 Family) audio on Realtek ALC883. I first thought perhaps it was the HDMI cable going to an external monitor, or some other RF interference from a cable, but turning things off or unplugging them didn’t make any difference.

Then I suspected there was some audio driver problem because the whine only started once the sound driver loaded at boot time. After trying all sorts of variations in the ALSA configuration, changing the options to the snd-hda-intel kernel module, I was at a loss and unplugged my USB keyboard and mouse.

It was the USB mouse! It’s a laser-tracked mouse with little shielding on the short cable. Plugging it into either of the USB ports near the front of the computer caused the noise. The keyboard didn’t matter.

At first I thought my other USB non-laser ball mouse didn’t add any noise, but it did, just a quieter and lower-pitch noise.

Then … I discovered a third USB port near the back of the computer that I hadn’t ever noticed. Plugging mice in there doesn’t interfere with the audio. Sigh. Maybe this tale will save someone else some trouble.

In the process I also fixed a problem that was in software: The external speakers didn’t mute when headphones are plugged in, as others have described as well. One of their solutions worked.

In /etc/modprobe.d/alsa-base add:

options snd-hda-intel model=targa-2ch-dig

and reboot. Or, if you dread rebooting as I do, exit all applications using audio, modprobe -r snd-hda-intel then modprobe snd-hda-intel. Finally, uncheck the “Headphones” checkbox in the sound control panel.