The ELK stack (Elasticsearch-Logstash-Kibana) provides a cost effective alternative to commercial SIEMs for ingesting and managing OSSEC alert logs. Previously I wrote a blog – OSSEC Log Management with Elasticsearch – that discusses the design of an ELK based log system.
Since then some readers have asked for and suggested ways to parse additional fields from the OSSEC alert log stream. For example, the IP addresses of systems that cause certain security events is buried down in the Details field. So I have created a Logstash configuration file that does just that.
The first article in this two part series focused on developing Elasticsearch clients with Perl. Elasticsearch also has an excellent Python library which lets you search for and analyze your data with one of the many mathematics and machine learning libraries available for Python.
In this article I’ll cover how to create an Elasticsearch client using Python that has the same capabilities as the Perl client from the part 1 article.
Since creating a log management system for the OSSEC HIDS with Elasticsearch, I have been busy applying this useful search technology in other projects. Elasticsearch is a marvelous system for ingesting streaming data that gets indexed on the fly and quickly searching your data.
The Elasticsearch community provides client libraries that expose their search API in several popular languages, including Perl and Python. This article is the first of a two part series where I show you how to write an Elasticsearch search client application in both of these languages, starting with Perl.
Recently I had a question from one of my readers about how to close connections on a server when there are no requests received after a certain period of time. The question was asked with regard to the tcpsockets classes I covered in my blog TCP Network Programming Design Patterns in C++, none of which support time out capabilties.
Timing out on both receive and connect operations are common use cases. So in this article I’ll update my tcpsockets classes to provide these capabilties.
When I first started playing around with OSSEC I downloaded the agent and server source packages then proceeded to install them by hand. This method was fine when I had a server and 1 or 2 agent systems, but for a large network of systems it is tedious and error prone.
The OSSEC Project offers RPM packages that can be installed with yum on RedHat derived Linux distributions. Using these packages and the ossec-authd system, you can script the installation of OSSEC and automatically register agents with the server.
Among the many useful features of OSSEC is its capability to send alerts to any system that can consume syslog data. This makes it easy to combine OSSEC with a number of 3rd party SIEMs to store, search and visualize security events. Splunk for OSSEC is one such system that works on top of the Splunk platform.
Splunk can be expensive though, particularly if you collect a lot of log data. So I’ve been working on a solution for collecting OSSEC security alerts based on Elasticsearch that provides a cost effective alternative to Splunk.