Our Blog

SELINUX=disabled? Read this and think twice!

By Emanuele “Lele” Calò
April 10, 2017

Not long ago, one of our customers had their website compromised because of a badly maintained, not-updated WordPress. At End Point we love WordPress, but it really needs to be configured and hardened the right way, otherwise it’s easy to end up in a real nightmare.

This situation is worsened even more if there’s no additional security enforcement system to protect the environment on which the compromised site lives. One of the basic ways to protect your Linux server, especially RHEL/Centos based ones, is using SELinux.

Sadly, most of the interaction people has with SELinux happens while disabling it, first on the running system:

setenforce disabled
# or
setenforce 0

and then permanently by manually editing the file /etc/sysconfig/selinux to change the variable SELINUX=enforcing to SELINUX=disabled.

Is that actually a good idea though? While SELinux can be a bit of a headache to tune appropriately and can easily be misconfigured, here’s something that could really convince you to think twice before disabling SELinux once and forever.

Back to our customer’s compromised site. While going through the customer’s system for some post-crisis cleaning, I found this hilarious piece of …

mysqldump issues after Percona 5.7 update

By Marco Matarazzo
April 7, 2017

During a recent CentOS 7 update, among other packages, we updated our Percona 5.7 installation to version 5.7.17-13.

Quickly after that, we discovered that mysqldump stopped working, thus breaking our local mysql backup script (that complained loudly).

What happened?

The error we received was:

mysqldump: Couldn't execute 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'rocksdb\_skip\_fill\_cache'': The 'INFORMATION_SCHEMA.SESSION_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56' (3167)

After a bit of investigation, we discovered this was caused by this regression bug, apparently already fixed but not yet available on CentOS:

Everything revolves around INFORMATION_SCHEMA being deprecated in version 5.7.6, when Performance Schema tables has been added as a replacement.

Basically, a regression caused mysqldump to try and use deprecated INFORMATION_SCHEMA tables instead of the new Performance Schema.

How to fix it?

Immediate workaround is to add this line to /etc/my.cnf or (more likely) /etc/percona-server.conf.d/mysqld.cnf, depending on how your configuration files are organized: …

Linode IPv6 issues with NetworkManager on CentOS 7

By Marco Matarazzo
April 4, 2017

In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is Linode.

During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.

IPv6 on Linode and SLAAC

Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using SLAAC.

In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.

This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.

Privacy stable addressing

Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.

This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to “stable-privacy”. Setting it to “eui64” …

Half day GlusterFS training in Selangor, Malaysia

By Muhammad Najmi bin Ahmad Zabidi
April 2, 2017

On January 21, 2017, I had an opportunity to join a community-organized training on storage focused on GlusterFS. GlusterFS is an open source cloud-based filesharing network. The training was not a strictly structured training as the topic approached knowledge sharing from various experts and introduced GlusterFS to the ones who were new to it. The first session was delivered by Mr Adzmely Mansor from NexoPrima. He shared a bit of his view on GlusterFS and technologies that are related to it.

Mr Haris, a freelance Linux expert, later led a GlusterFS technical class. Here we created two virtual machines (we used Virtualbox) to understand how GlusterFS works in a hands-on scenario. We used Ubuntu 16.04 as the guest OS during technical training. We used Digital Ocean’s GlusterFS settings as a base of reference. The below commands detail roughly what we did during the training.

In GlusterFS the data section is called as “brick”. Hence we could have a lot of “bricks” if we have it more than once :) . As Ubuntu already had the related packages in its repository, we could simply run apt-get for the package installation. Our class notes were loosely based from Digital Ocean’s GlusterFS …

DBA Revenge: How To Get Back at Developers

By Josh Williams
April 1, 2017

In the spirit of April 1st, resurrecting this old classic post:

Maybe you work at one of those large corporations that has a dedicated DBA staff, separate from the development team. Or maybe you’re lucky and just get to read about it on thedailywtf.com. But you’ve probably seen battles between database folk and the developers that “just want a table with "ID " VARCHAR(255), name VARCHAR(255), price VARCHAR(255), post_date VARCHAR(255). Is that so much to ask?!”

Well if you ever feel the need to get back at them, here’s a few things you can try. Quoted identifiers let you name your objects anything you want, even if they don’t look like a normal object name…

CREATE TABLE "; rollback; drop database postgres;--" ("'';
delete from table order_detail;commit;" INT PRIMARY KEY,
";commit;do $$`rm -rf *`$$ language plperlu;" TEXT NOT NULL);

COMMENT ON TABLE "; rollback; drop database postgres;--"
IS 'DON''T FORGET TO QUOTE THESE';

Good advice, that comment. Of course, assuming they learn, they’ll be quoting everything you give them. So, drop a quote right in the middle of it:

CREATE TABLE …

Postgres WAL files: best compression methods

By Greg Sabino Mullane
March 28, 2017

Turtle turtle by WO1 Larry Olson from US Army

The PostgreSQL database system uses the write-ahead logging method to ensure that a log of changes is saved before being applied to the actual data. The log files that are created are known as WAL (Write Ahead Log) files, and by default are 16 MB in size each. Although this is a small size, a busy system can generate hundreds or thousands of these files per hour, at which point disk space becomes an issue. Luckily, WAL files are extremely compressible. I examined different programs to find one that offered the best compression (as indicated by a smaller size) at the smallest cost (as indicated by wall clock time). All of the methods tested worked better than the venerable gzip program, which is suggested in the Postgres documentation for the archive_command option. The best overall solution was using the pxz program inside the archive_command setting, followed closely by use of the 7za program. Use of the built-in wal_compression option was an excellent solution as well, although not as space-saving as using external programs via archive_command.

A database system is a complex beast, involving many trade-offs. An important issue is …

wroc_love.rb 2017 part 2: The Elixir Hype

By Wojtek Ziniewicz
March 21, 2017

One of the main reasons I attend wroc_love.rb almost every year, is that it’s a great forum for confronting ideas. It’s almost a tradition to have at least 2 very enjoyful discussion panels during this conference. One of them was devoted to Elixir and why the Ruby [1] community is so hyping about it.

Why Elixir is “sold” to us as “new better Ruby” while its underlying principles are totally different? Won’t it result in Elixir programmers that do not understand Elixir (like Rails programmers that do not know Ruby)?

Panelists discussed briefly the history of Elixir:

Jose Valim (who created Elixir) was working on threading in Rails and he was searching for better approaches for threading in web frameworks. He felt like lots of things were lacking in Erlang and Elixir is his approach for better Exceptions, better developer experience.

Then they jumped to Elixir’s main goals which are:

• Compatibility with Erlang (all datatypes)
• Better tooling
• Improving developers’ experience

After that, they started speculating about problems that Elixir solves and RoR doesn’t:

Ruby on Rails addresses many problems in ways that may be somehow archaic to us in the ever-​scaling world of 2017. There are …

wroc_love.rb 2017 part 1

By Wojtek Ziniewicz
March 18, 2017

wroc_love.rb is a single-track 3-day conference that takes place in Wrocław, Poland, every year in March.

Here’s a subjective list of most interesting talks from the first day:

Kafka / Karafka by Maciej Mensfeld

Karafka is another library that simplifies Apache Kafka usage in Ruby. It lets Ruby on Rails apps benefit from horizontally scalable message busses in a pub-sub (or publisher/​consumer) type of network.

Why Kafka is (probably) better message/​task broker for your app:

• broadcasting is a real power feature of Kafka (HTTP lacks that)
• author claims that it’s easier to support than ZeroMQ/​RabbitMQ
• it’s namespaced with topics (similar to ROS, the Robot Operating System)
• great replacement for ruby-kafka and Poseidon

Karafka https://t.co/g9LQZiAV4i microframework to have #rails-like development performance with #kafka in #ruby @maciejmensfeld #wrocloverb

— Maciek Rząsa (@mjrzasa) 17 marzo 2017

Machine Learning to the Rescue by Mariusz Gil

This talk was devoted to Machine Learning success (and failure) story of the author.

Author underlined that Machine Learning is a process and proposed following workflow:

1. define a problem
2. gather your data
3. understand your data
4. prepare and …