4 solutions for log processing and security analytics based on the ELK stack

Containing a wealth of information on the different events taking place in a system, logs lie at the heart of almost every Security Analytics solution in the market.

But before this raw material can be used, it needs to be refined. Logs need to be collected, processed, normalized, enhanced and stored. These processes are usually grouped together under the umbrella term “log management” and are a must to enable the subsequent stage of “log analysis” — the process of querying and visualizing the data.

This might explain why the ELK Stack (Elastic Search, Logstash, Kibana) — the world’s most popular open source log management and log analysis system — is being increasingly used as a SIEM solution (Security information and event management), both open source and commercial. Traditional SIEM tools, such as AlienVault, QRadar, and Splunk, provide a more comprehensive set of capabilities, but a slow and gradual move to more simpler and nimble solutions based on ELK can be increasingly identified.

Confused? So are we. That’s why we decided to put together a comparison to try and better understand the part ELK is playing in the different Security Analytics solutions being introduced into the market.

Custom inhouse solution based on ELK

Let’s start with do-it-yourself ELK. A lot of users are already using the stack as a SIEM solution because it provides solid log management and log analysis capabilities. It supports log collection, log processing, scalable storage, querying, and visualization capabilities. But is that enough?

All the other capabilities expected from a proper SIEM solution, such as alerting, correlation, reporting and incident management are missing. This is why the stack is usually used together with other platforms and services and not on its own.

Therefore vanilla ELK is not an out-of-the-box SIEM solution. However, being completely open source it is definitely a solid base from which to build out a more complete system together with other tools.

Logz.io

This is an interesting and relatively new option in the market. Logz.io offers security analytics built on top of a fully managed ELK solution. So all the capabilities outlined above — log collection, log processing, scalable storage, querying and visualization — are provided as a service.

On top of vanilla ELK, Logz.io has added threat detection, correlation, alerts, and built-in dashboards. Logz.io strives to offer what it calls a “unified” solution, meaning teams can use the same log data used for monitoring and troubleshooting production environments, for security analytics as well.

Even though it is built on top of the open source ELK stack, Logz.io is itself a commercial solution. Reporting and incident management capabilities are also lacking compared to other solutions.

Elastic SIEM

Another option is Elastic SIEM by Elastic, the company behind the ELK Stack. Since this is only a beta, the solution still lacks a lot of the core functionality expected from a SIEM but holds promise for the future.

Similar to Logz.io, Elastic SIEM bundles all the existing log management and log analysis capabilities provided by ELK. In addition, Elastic SIEM offers a dedicated UI for analyzing events. These capabilities are offered free of charge but under Elastic’s Basic license, meaning they are not completely open source.

Additional functionality, like alerting, is available for a paid subscription. As of yet, Elastic SIEM does not provide any advanced threat detection and correlation capabilities. Definitely worth following to how this product develops in the future.

Wazuh

The ELK Stack provides the logging backend for Wazuh — an open source security monitoring solution used to collect, analyze and correlate data, with the ability to deliver threat detection, compliance management, and incident response capabilities. It can be deployed on-premises or in hybrid and cloud environments.

Designed for security monitoring and analytics, Wazuh offers a more comprehensive list of security-focused capabilities and features, such as intrusion and vulnerability detection and incident response.

ELK is deployed together with Wazuh for storing and analyzing log data. But one thing to keep in mind is the fact that users are expected to manage and maintain the stack on their own. On the other hand Wazuh is full open source (you can pay a subscription for support).

Summary

ELK is an extremely popular log management and log analysis and can probably be considered as a de-facto industry standard for operational use cases. More and more companies, however, are also using the stack for security analytics and as a SIEM system. As explained in the introduction, since logs are the main data source used for these systems, this practice makes total sense from an architectural perspective.

The ELK stack lacks a lot of capabilities that traditional SIEM systems offer but the market seems to be slowly catching up with solutions like Logz.io, Elastic SIEM and Wazuh offering additional security functionality on top of the stack. SIEM and Security Analytics is one of the most exciting categories in the wider security market, and it’s fascinating to see how open source is beginning to play a bigger part in it.

Here’s a brief summary table comparing the ELK-based SIEM options in the market: