Systemd at 3am

A few of systemd features that helps you and your fellow sysadmins.

At 3am, I want to sleep. I do not want SMS with “Service X is down”, and I do not want my systems to wake the on-call personnel, so they can scratch their heads and call me about “Service X is down, and I need help fixing it”.

There are a couple of things you can do to avoid this.

Automatic restarts

Sometimes processes die. Particularly at ... [continue reading]

Feeding the Elastic Stack

This is the last of three posts about Elastic Stack.

By now, we should have a reasonably secure Elastic Stack. It is sadly empty, so we should feed it some logs.

Logstash is a log processor. It can be configured with inputs, filters, and outputs.

  • Inputs are commonly log files, or logs received over the network.
  • Filters are used to accept, drop and modify log events.
  • Outputs are used for storing the filtered logs.

Filebeat is a log ... [continue reading]

Enabling HTTP/2 for a site

When we installed the new frontend nodes for our main site, we wanted make use of some technologies that aren’t yet in broad use by our customers. The intention was both to gain more experience with said technologies, and to show that they are ready for production use. HTTP/2 was one of these technologies.

Why use HTTP/2?

HTTP/2 offers several features that improve the load speed of pages. To quote the HTTP/2 FAQ, the new version of ... [continue reading]

Small-scale honeynet with Raspberry Pi

The Raspberry Pi units are small and don’t use much power. If you have one or two to spare, why not use them to explore the sweet smell of honeypots?

Ye who enter here

First of all, a warning: Even though honeypot software is usually isolated from the underlying operating system, bugs do exist and accidents can happen. You should not run any other services on a system hosting honeypot software.

I would not recommend running an unattended ... [continue reading]

Deduplication of old file systems

Modern file systems, and even storage systems, might have built-in deduplication, but common file systems still do not. So checking for redundant data and do deduplication when possible might save disk space.

Once up on a a time, there was a system, were we had this 6TB spool of binary files on an production ext4 file-system, and the volume was running out of disk space. The owner of the data thought it likely that there were duplicates in the vast ... [continue reading]

JMole monitoring framework

Monitoring Java applications can be a painful operation that often require lots of configuration, with technologies like byte code instrumentation and JMX, you can literally have thousands of Metrics to choose from just from a single Java application. This post will talk about JMole which has good sensible defaults that makes monitoring java applications much less painful.

JMole is an excellent monitoring client framework for gathering data dynamically from various Java MBeans (Java 6 and higher). It consists of a ... [continue reading]

Fun with firewall activity plotting

A firewall activity plot for showing port access. The temptation was just a bit too great to ignore, so I chose to see it as a canvas for artwork. All I should need to do is to convert a PNG image to series of nmap commands, easy right?

Plot size

First off, lets determine the actual size of the canvas – I mean plot. The graph area is 1040x417 pixels, but since the graph is logarithmic, ... [continue reading]

Pros and cons of visualizing firewall activity

For some time now, I’ve been graphing all unsolicited network traffic destined for my network. For instance, it’s quite useful for detecting slow scans, which will show up as the diagonally aligned green scatter points in this plot (click to zoom):

Slow_portscan
Slow portscan, from high ports to low ports.

Other scans and probes often happen faster, when the attacker isn’t much concerned about being detected. These will appear in the ... [continue reading]

Securing the Elastic Stack

This is the second of three posts about Elastic Stack.

The Elastic Stack service is available to anyone who can reach it by default. This allows you to choose your security level and tools to provide it.

A simple search on Shodan for Kibana or Elasticsearch will quickly reveal that many do not secured their logs. I hope this post will encourage you to do so.

One efficient way to increase security is to place Elasticsearch and Kibana ... [continue reading]

Use virt-manager to build disk-images

For cattle purposes, it makes sense to follow a build-once-run-many principle. This is what we prefer for the machines powering our infrastructure. The current build method for deployments uses the tool-chain from the virt-manager project to achieve this.

Build targets

The combination of virt-install(1) and virt-builder(1) provides a layered approach for generating disk-images. Those images can then be used as a base for constructing images for the different environments that we support:

... [continue reading]