Creating Elasticsearch Snapshots

Elasticsearch snapshotsBenjamin Franklin once wrote “…in this world nothing can be said to be certain, except death and taxes”. In this computerized world of ours, I would add having to backup your data to free up disk space to that list of eventualities.

For Elasticsearch users, backups are done using the Elasticsearch snapshot facility. In this article I’ll go through the design of an Elasticsearch backup system that you can use to create snapshots of your cluster’s indices and documents.

Continue reading

Improved OSSEC Log Parsing with Logstash

logstash-ossecThe ELK stack (Elasticsearch-Logstash-Kibana) provides a cost effective alternative to commercial SIEMs for ingesting and managing OSSEC alert logs. Previously I wrote a blog – OSSEC Log Management with Elasticsearch – that discusses the design of an ELK based log system.

Since then some readers have asked for and suggested ways to parse additional fields from the OSSEC alert log stream. For example, the IP addresses of systems that cause certain security events is buried down in the Details field. So I have created a Logstash configuration file that does just that.

Continue reading

Elasticsearch Client Programming (2/2) – Python

Elasticsearch PythonThe first article in this two part series focused on developing Elasticsearch clients with Perl. Elasticsearch also has an excellent Python library which lets you search for and analyze your data with one of the many mathematics and machine learning libraries available for Python.

In this article I’ll cover how to create an Elasticsearch client using Python that has the same capabilities as the Perl client from the part 1 article.

Continue reading

Elasticsearch Client Programming (1/2) – Perl

Elasticsearch PerlSince creating a log management system for the OSSEC HIDS with Elasticsearch, I have been busy applying this useful search technology in other projects. Elasticsearch is a marvelous system for ingesting streaming data that gets indexed on the fly and quickly searching your data.

The Elasticsearch community provides client libraries that expose their search API in several popular languages, including Perl and Python. This article is the first of a two part series where I show you how to write an Elasticsearch search client application in both of these languages, starting with Perl.

Continue reading

TCP/IP Sockets with Time Out Capabilities

Socket Time OutRecently I had a question from one of my readers about how to close connections on a server when there are no requests received after a certain period of time.  The question was asked with regard to the tcpsockets classes I covered in my blog TCP Network Programming Design Patterns in C++, none of which support time out capabilties.

Timing out on both receive and connect operations are common use cases. So in this article I’ll update my tcpsockets classes to provide these capabilties.

Continue reading