Conference Recap: PyCon Asia Pacific (APAC) 2017 in Kuala Lumpur, Malaysia
By Muhammad Najmi bin Ahmad Zabidi
December 2, 2017
I got a chance to attend the annual PyCon APAC 2017 (Python Conference, Asia Pacific) which was hosted in my homeland, Malaysia. In previous years, Python conferences in Malaysia were held at the national level and this year the Malaysia’s PyCon committee worked hard on organizing a broader Asia-level regional conference.
Highlights from Day 1
The first day of the conference began with a keynote delivered by Luis Miguel Sanchez, the founder of SGX Analytics, a New York City-based data science/data strategy advisory firm. Luis shared thoughts about the advancement of artificial intelligence and machine learning in many aspects, including demonstrations of automated music generation. In his talk Luis presented his application which composed a song using his AI algorithm. He also told us a bit on the legal aspect of the music produced by his algorithm.
Luis speaking to the the audience. Photo from PyCon’s Flickr.
Then I attended Amir Othman’s talk which discussed the data mining technique of news in the Malay and German languages (he received his education at a German tertiary institution). His discussion included the verification of the source of the news and the issue of the …
conference python
Logstash: Removing fields with empty values
By Jon Jensen
November 22, 2017
The Elastic stack is a nice toolkit for collecting, transporting, transforming, aggregating, searching, and reporting on log data from many sources. It was formerly known as the ELK stack, after its main components Elasticsearch, Logstash, and Kibana, but with the addition of Beats and other tools, the company now calls it simply the Elastic stack.
We are using it in a common configuration, on a central log server that receives logs via rsyslog over TLS, which are then stored in local files and processed further by Logstash.
When conservation is recommended
When forwarding logs on to SaaS log services such as Logentries, SumoLogic, etc., we have a limited amount of data transfer and storage allotted to us. So we need to either economize on what we send them, pay for a more expensive plan, or retain a shorter period of history.
For some very busy logs (nginx logs in JSON format) we decided to delete fields with empty values from the log event during the filter phase in Logstash. This removes a lot of data from the log message we send to the log service over the wire, and reduces the size of each log event stored in their system.
I expected this to be simple, but that expectation …
hosting logging
From Zero to HTTPS in an afternoon
By Matt Vollrath
November 20, 2017
I’ve been hosting my own humble personal web site since 2012. I had never bothered setting up HTTPS for my domain, but after hearing about the Let’s Encrypt project, I was completely out of excuses.
For the unfamiliar, Let’s Encrypt offers free and fully automatic HTTPS certificates. The web cares about HTTPS now more than ever. Deeply interactive interfaces like geolocation and user media (camera, microphone) are too sensitive to trust an insecure transport. By leveraging the security features present in modern browsers, users can expect a reasonable safety from attacks that would exploit the weaknesses of HTTP.
To take the security mission even further, I decided to completely containerize my server and expose only a couple of ports. Using a Docker composition made it very easy to deploy up-to-date nginx and keep it isolated from the rest of my host shard.
The first mission was to set up certificates with certbot, the EFF’s free certificate tool. certbot has a plugin that writes nginx configuration for you, but in this case I didn’t want nginx installed on my host at all. Instead of following the nginx-specific instructions for my platform, I opted for the webroot plugin to just …
hosting security tls nginx
President of UN General Assembly Thanks End Point
By Jonathan Blessing
November 17, 2017
The President of UN General Assembly, Peter Thomson, thanked End Point for supporting the Ocean Conference, which was held at the United Nations Headquarters this past summer to bring attention and action to saving the world’s oceans.
End Point’s Liquid Galaxy helped bring to life “Reconnecting Humanity to the Sea,” an exhibition meant to showcase the beauty of the ocean and the challenges it faces today. End Point created the presentation’s content and showcased it at the conference.
“We were very pleased to see End Point’s Liquid Galaxy used to promote a hopeful future for the world’s oceans. It’s very satisfying to see our technology used to make an important story that much more compelling.”
Rick Peltzman
CEO, End Point
This UN press release explains more about the conference and its results:
“UN Ocean Conference wraps up with actions to restore ocean health, protect marine life”
company visionport event
Using GitHub for Blog Comments
By Phineas Jensen
November 14, 2017
Last Saturday, November 11, we rolled out a new website that we’ve been working on for a few months. Part of this update was moving from Blogger as our blogging platform to static HTML generated by Middleman. We were more than happy to move away from Blogger for a variety of reasons, including its lack of HTTPS support for custom domains and how difficult it was to keep its templating and styling up to date with our main website. We were also able to move from blog.endpoint.com to www.endpoint.com/blog.
The most obvious thing that is missing from Middleman’s blog extension is the lack of a commenting system. After exploring some options for comments, we settled on using GitHub issues and comments, inspired by Don Williamson’s post about doing the same thing. It’s a bit of an unconventional approach, so this post will explain how to use our commenting system and how we implemented it.
Commenting requires a GitHub account, which is easy to sign up for and free, and the general target audience of our blog will often already have a GitHub account. At the bottom of each post will be a link to a GitHub issue at the top of the list of comments, if there are any. Click on the issue, write …
javascript community api static-site-generator
A Collaborative Timezone Utility
By Joe Marrero
October 30, 2017
Try It Out Yourself
The code for this project is hosted on GitHub and can be cloned from here.
At End Point Corporation, our team is spread out across 10 time zones. This gives us the advantage of being able to work around the clock on projects. When one co-worker leaves for day, another can take over. Consider this scenario. It’s Monday evening and Martin needs to continue installing software on that Linux cluster, but it’s already 6pm and his wife is going to murder him if he’s not ready to go out for their anniversary dinner. Let’s see who can take over… Ah, yes, Sanjay in Bangalore can continue with the maintenance. Tuesday morning, the client wakes up to be surprised that 16 hours of work was completed in a day. With respect to software development, the same efficiencies can be realized by parallelizing tasks across time-zones. Code reviews and further development can be continued after normal business hours.
With all the blessings of a distributed engineering team, collaborating with co-workers can be, occasionally, challenging. Some of these challenges stem from complexities of our system of time. Every co-worker may be …
linux open-source tools c
Hot-deploy Java classes and assets in Wildfly 8/9/10
By Piotr Hankiewicz
October 27, 2017
Introduction
Java development can be really frustrating when you need to re-build your project and restart a server every time you change something. I know about JRebel, but while it’s a good tool, it’s also pretty expensive. You can use the open-source version, but then you need to send project statistics to the JRebel server, which is not a viable option for your more serious projects.
Fortunately, there is an open-source project called HotSwapAgent and it does the same thing as JRebel, for free (thank you, guys!).
I will explain how to combine it with Widlfly in order to hot-deploy Java classes as well as how to hot-deploy other resources (JavaScript, CSS, images).
Wildfly configuration
Let’s assume that we use the standalone-full.xml configuration file.
We need to use exploded deployment instead of deploying WAR or EAR. You can do this in production as well to allow for application changes with zero downtime.
Start by configuring the metaspace size; we had to increase defaults for our application, but it’s possible that it will be just fine in your case. It’s encouraged that you play with these values after completing all steps.
In:
WILDFLY_DIR/bin/standalone.conf
set: …
java intellij-idea
Using tail_n_mail after hours
By Greg Sabino Mullane
October 23, 2017
(Photo of Turtle Island by Edwin Poon)
Someone recently asked me something about
tail_n_mail, a program that watches over your log files, scans for certain patterns,
and sends out an email if matches are found. It is frequently used to watch over
Postgres logs so you can receive an automatic email alert when Bad Things start happening
to your database. The questioner wanted to know if it was possible
for tail_n_mail to change its behavior based on the time of day—would it be able
to do things differently outside of “business hours”? Although tail_n_mail cannot
do so directly, a simple solution is to use alternate configuration files—which
get swapped by cron—and the INHERIT keyword.
To demonstrate the solution, let’s spin up a Postgres 10 instance, route the logs to syslog, setup tail_n_mail, and then create separate configuration files for different times of the week. First, some setup:
$ initdb --version
initdb (PostgreSQL) 10.0
$ initdb --data-checksums data
$ cat >> data/postgresql.conf << EOT
log_line_prefix=''
log_destination='syslog'
EOT
$ echo 'local0.* /var/log/postgres.log' | sudo tee -a /etc/rsyslog.conf > /dev/null
$ sudo …postgres monitoring