Building your first SIEM with the Elastic Stack
Preface
This process is an exercise in installing and configuring Elastic Stack components for use as a SIEM. Normally you wouldn’t install all of these services on a single host: you should consider the resource consumption of each host and distribute your services appropriately.
After we get the basic software running we’ll do a few example searches and create an example alert. The power of your SIEM will be determined by how many detections you implement and what kind of alerts you use.
These instructions are for a single Ubuntu 18.04 host running all 6 of the SIEM components. This is not guaranteed to work on all Debian distributions immediately; you may need to tweak some of the source repositories for other distributions. If you are running these components on separate hosts, you will need to modify the configurations and commands listed below to work with your distributed architecture.
Screencap
Installation
Install Elastic repositories
We’ll start out by adding the Elastic apt repositories to our sources to install all the packages we’ll need from Elastic:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
This makes it really easy to install the rest of the Elastic Stack. We’ll talk about configuring each of these services in a little bit; let’s get them installed first:
Install Elasticsearch:
sudo apt-get install elasticsearch
Before we install the rest of the Elastic Stack, lets make sure we get Elasticsearch fired up so that it will be ready for us to put data into:
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Install Kibana:
sudo apt-get install kibana
Install Logstash:
sudo apt-get install default-jre -y
sudo apt-get install logstash
Install Filebeat:
sudo apt-get install filebeat
Install Elastalert:
Installing Elastalert isn’t as straightforward, but it isn’t that difficult either. We’ll clone the Elastalert source repository, and (after making sure that python’s pip3 and the setuptools pip package are installed) move the installation directory to /etc to keep things consistent.
sudo apt install python3-pip -y
pip3 install "setuptools>=11.3" PyYAML
git clone https://github.com/Yelp/elastalert.git
cd elastalert && sudo python3 setup.py install
mkdir bin && mkdir rules
sudo mv /usr/local/bin/elastalert* bin/
cd ../ && sudo mv elastalert/ /etc/
Create Elastalert systemd service
Elastalert doesn’t include a systemd service by default, but it’s pretty trivial to make one. Don’t worry if some of the paths not existing yet, we’ll set those up in a minute:
Create the file /etc/systemd/system/elastalert.service with the following contents:
[Unit]
Description=elastalert
After=multi-user.target
[Service]
Type=simple
WorkingDirectory=/etc/elastalert
ExecStart=/etc/elastalert/bin/elastalert --config /etc/elastalert/config.yaml
StandardOutput=file:/var/log/elastalert.log
StandardError=file:/var/log/elastalert.log
Restart=Always
[Install]
WantedBy=multi-user.target
Then enable the service:
sudo systemctl enable elastalert.service
sudo systemctl start elastalert
Install Zeek
The Zeek binaries are made available through the OpenSUSE project, but if they don’t work on your distribution you can always build it from source.
sudo sh -c "echo 'deb http://download.opensuse.org/repositories/security:/zeek/xUbuntu_18.04/ /' > /etc/apt/sources.list.d/security:zeek.list"
wget -nv https://download.opensuse.org/repositories/security:zeek/xUbuntu_18.04/Release.key -O Release.key
sudo apt-key add - < Release.key
rm Release.key && sudo apt-get update
sudo apt-get install zeek -y
The installer may prompt you for a server to send mail through. Go ahead and accept the defaults on this one, which will configure Zeek not to send email.
Configuration and Setup
Configure Zeek
Zeek is a powerful and extensible network monitor. Ideally, you’d want to use a span, tap, or mirrored port to get a copy of all your network traffic and send it to Zeek to be analyzed and to have log files generated. For this exercise we’ll be using Zeek on the main interface of the SIEM.
Zeek requires a network interface that it can listen on to pick up information about the network. With the driver we’ll be using to pick up packets, we need to set that interface in promiscuous mode for Zeek to analyze traffic. On this machine our interface is enp0s3. First we’ll enable promiscuous mode on our network adapter:
sudo ifconfig enp0s3 promisc
Zeek supports clustering with other Zeek sensors or nodes, so the configuration for each Zeek instance is located in /opt/zeek/etc/node.cfg.
Edit /opt/zeek/etc/node.cfg to include your interface name:
[zeek]
type=standalone
host=localhost
interface=enp0s3
Finally, we’ll set Zeek’s JSON logging property to make the logs from Zeek easier to parse.
Add the following line to the bottom of /opt/zeek/share/zeek/site/local.zeek:
# Output to JSON
@load policy/tuning/json-logs.zeek
Zeek is best managed by the command-line management tool zeekctl. This utility will do most of the legwork for us in watching Zeek to make sure that it restarts if it crashes, and rotating the logs when they reach a certain age. This will be great for our Filebeat agent, which will just read new logs as they are created by Zeek.
First, use zeekctl to set up the rest of the required config files for our Zeek deployment:
sudo /opt/zeek/bin/zeekctl deploy
Then start Zeek:
sudo /opt/zeek/bin/zeekctl start
To test and make sure we are getting network logs from Zeek, lets ping yahoo.com and make sure that we see the DNS lookup event in Zeek’s built-in dns.log. It may take a few seconds for the event to get parsed from Zeek and show up in the log:
ping yahoo.com
sudo tail -f /opt/zeek/logs/current/dns.log
Configure filebeat
Our Filebeat setup is going to take the raw, unenriched log files generated by Zeek, our network monitor, as well as from the system’s log files (which will be our endpoint logs) and put them into Logstash to be normalized and enriched. Normally your network sensor might be on a separate host, but in our example our network monitor is also our endpoint we are monitoring.
First, enable the Zeek and system input modules for Filebeat:
sudo filebeat modules enable zeek system
Next we need to modify our filebeat.yml file to include our log file paths and our Logstash output port.
Replace everything in /etc/filebeat/filebeat.yml with just the following:
filebeat.inputs:
filebeat.config.modules:
path: /etc/filebeat/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "localhost:5601"
output.logstash:
hosts: ["localhost:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
Then restart Filebeat:
sudo systemctl enable filebeat
sudo systemctl restart filebeat
One last change we’ll need to make to our Filebeat configuration is to set the search path in the Zeek module. This doesn’t really do much for us by way of normalization, but it does make Filebeat make an effort to tag the events with their type before sending them into Logstash to be enriched.
Replace the contents of /etc/filebeat/modules.d/zeek.yml with the following:
- module: zeek
# All logs
connection:
enabled: true
var.paths:
- /opt/zeek/logs/current/conn.log
dns:
enabled: true
var.paths:
- /opt/zeek/logs/current/dns.log
http:
enabled: true
var.paths:
- /opt/zeek/logs/current/http.log
files:
enabled: true
var.paths:
- /opt/zeek/logs/current/files.log
ssl:
enabled: true
var.paths:
- /opt/zeek/logs/current/ssl.log
notice:
enabled: true
var.paths:
- /opt/zeek/logs/current/capture_loss.log
- /opt/zeek/logs/current/known_services.log
- /opt/zeek/logs/current/loaded_scripts.log
- /opt/zeek/logs/current/packet_filter.log
- /opt/zeek/logs/current/reporter.log
- /opt/zeek/logs/current/stats.log
- /opt/zeek/logs/current/stderr.log
- /opt/zeek/logs/current/stdout.log
- /opt/zeek/logs/current/weird.log
Let’s make sure our Filebeat config file was accepted by checking to make sure that Filebeat started up alright:
sudo service filebeat status
Configure Logstash
Now that we have Filebeat reading the raw host logs and the raw Zeek network monitor logs, we are ready to set up Logstash to enrich the log files to make them more easy to search.
Logstash allows multiple enrichment pipelines that can run different routines on different types of logs. For this example we’ll use a single pipeline for our Zeek and host logs, but an easy optimization to this configuration would be to create separate pipelines for different types of logs.
Let’s define our pipeline by modifying /etc/logstash/pipelines.yml and replacing the config there with the following lines:
- pipeline.id: zeek_and_host_logs
path.config: "/etc/logstash/conf.d/zeek_and_host.conf"
Now we need to define our enrichment pipeline. Logstash pipelines always have 3 parts: an input, a filter, and an output. The inputs and outputs are pretty straightforward. The filters can get a little complex, and can do some pretty amazing things to your log. You can use things like geoIP lookups to attach a location to a log, or a lookup table to match hostnames in your environment to IP addresses. You can drop logs that you don’t want, or put special tags on logs that you want to pay special attention to. Logstash’s filter plugins allow you to do almost anything you need to enrich and normalize the logs you get from any source.
Another quick change to make is to enable Logstash’s live update feature. Modify the appropriate line in /etc/logstash/logstash.yml:
config.reload.automatic: true
Our enrichment pipeline will be simple to start out. For now we’re simply going to assign our Zeek logs to one index in Elasticsearch and our host’s logs to another index.
To demonstrate how we can use logstash to enrich our logs and provide us better threat data, we will have Logstash put a flag in the log if Zeek detects a connection to the Tor network. To do that, we will download a list of all Tor nodes and compare our connection data in the log with that list, and add a new field if the source or destination IP and port are on the Tor network.
First, download a list of Tor nodes that I’ve curated for this example, then create a tables folder in the /etc/logstash/ directory to store it:
wget https://setup.cronocide.com/media/tor_nodes.json -O tor_nodes.json
sudo mkdir /etc/logstash/tables
Then move tor_nodes.json to /etc/logstash/tables:
sudo mv tor_nodes.json /etc/logstash/tables/
Create the file /etc/logstash/conf.d/zeek_and_host.conf with the following content:
input {
beats {
host => "0.0.0.0"
port => 5044
}
}
filter {
# Lookup source and destination addresses against our list of Tor nodes
if ("" in [source][address]) {
translate {
dictionary_path => "/etc/logstash/tables/tor_nodes.json"
field => "[source][address]"
destination => "[@metadata][torsrc]"
}
mutate {convert => { "[@metadata][torsrc]" => "integer" }}
}
if ("" in [destination][address]) {
translate {
dictionary_path => "/etc/logstash/tables/tor_nodes.json"
field => "[destination][address]"
destination => "[@metadata][tordst]"
}
mutate {convert => { "[@metadata][tordst]" => "integer" }}
}
if [@metadata][torsrc] == [source][port] {
mutate {add_field => { "[source][tor]" => "true"}}
mutate {convert => { "[source][tor]" => "boolean" }}
}
if [@metadata][tordst] == [destination][port] {
mutate {add_field => { "[destination][tor]" => "true"}}
mutate {convert => { "[destination][tor]" => "boolean"}}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
ssl => false
ssl_certificate_verification => false
sniffing => false
manage_template => false
index => "%{[event][module]}-%{+yyyy.MM.dd}"
}
}
Restart logstash to apply our changes:
sudo systemctl enable logstash
sudo systemctl restart logstash
Configure Kibana
Now we’re ready to start visualizing our data. First we need to change the binding port and address so that we can see our dashboards outside the box itself.
Change these two lines in /etc/kibana/kibana.yml:
server.port: 5601
server.host: "0.0.0.0"
Restart Kibana:
sudo systemctl enable kibana
sudo systemctl restart kibana
Now it’s time to log in to our dashboard! Navigate to the IP address of the box Kibana is running on on port 5601. Since we’ll be loading this in the web browser on the SIEM, the address would be http://localhost:5601.
Kibana Welcome Page
When prompted to explore your own data or play with the examples, choose to explore your data.
Kibana is a great visualization and search tool for Elasticsearch. Think of it as as not much more than a GUI for Elasticsearch. Elasticsearch controls how the data is stored, searched, and configured. We can modify all of those parameters through Kibana.
Not every log will have the same information, so we group logs with the same information into groups called indices to help make our data easier to search. In Elasticsearch, indices are a collection of logs that have the same fields.
The first thing we will need to do is create two indices to store our two different types of data: our network logs from Zeek and our host or endpoint logs from Filebeat.
By default, Logstash will make new indices in Elasticsearch every day, but there are many ways that this can be optimized, depending on your log type.
Select the Settings gear in the bottom left of Kibana and select Index Patterns under Kibana in the left. We’ll create a new index pattern that describes any indices that start with ‘system’. set the index pattern to be system-* and click next. When prompted for a time filter field, select ‘@Timestamp’. This lets Kibana know how to assign your events to a timeline.
Creating a 'system-*' index pattern
We’ll do the same with our zeek-* index pattern.
Using Kibana
Now let’s explore the data. Select the Discover icon (second from the top on the left).
Using 'Discover' in Kibana
On the left there will be a dropdown to select your index. This lets you choose the type of data you’re searching for. In our case, we can search for system logs or Zeek logs. The logs are sorted historically, and you can adjust the search time to go further back.
Let’s perform an example search. First, select the system-* index on the left. Then perform a sudo command on your host. This will make an entry in auth.log, which will be read by filebeat and pushed into Logstash, then normalized and put into Elasticsearch. This will happen after a few seconds.
To search for this event, let’s switch the search syntax from KQL to Lucene. Select the ‘KQL’ button at the end of the search bar and turn the ‘Kibana Query Language’ toggle off. This will make your search less flexible, but makes the searches compatible with other services like Elastalert.
To see just the sudo command that was run (and not the sudo session authorizations ), make a search like the one below, substituting ‘username’ for your username:
message: "sudo: username"
You should now be seeing only commands that were run in sudo mode for your user.
Let’s see if we can find new Tor sessions detected by Zeek. To generate some of this traffic you could download and install Tor, but that would take a little too much time for this example, so for now let’s pretend we’re connecting to the Tor network.
Use Netcat to connect to a Tor router:
nc 79.137.79.167 9001
Use CTRL + C to disconnect. Zeek should have detected the connection, and Logstash should have recognized the IP address as being a Tor node and tagged it as such. Head back to Kibana and open up the discover tab again, and switch to the zeek-* index pattern on the left.
After a few seconds the logs should be available to search. To search for our tor session, search for the following:
tags: zeek.connection AND _exists_:destination.tor
Clicking on the event will show us the source address, and let us know that it was our box that was connecting to Tor.
Configure elastalert
Now let’s make an alert that lets us know any time that that event happens.
To get Elastalert going we need to copy the example config file and modify it, or just create /etc/elastalert/config.yaml with the configuration below.
sudo cp /etc/elastalert/config.yaml.example /etc/elastalert/config.yaml
Edit the file /etc/elastalert/config.yaml to include the following contents:
rules_folder: rules
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: localhost
es_port: 9200
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
alert_time_limit:
days: 2
Elastalert can also send it’s metrics data into Elasticsearch. With this data you can troubleshoot alerts.
Run the following to configure the Elasticsearch index for Elastalert:
sudo /etc/elastalert/bin/elastalert-create-index --host localhost --port 9200 --no-ssl --no-auth --index 'elastalert_status' --alias 'elastalert_alerts' --old-index '' --url-prefix ''
Now lets make a rule. A rule in Elastalert is a search that is run in Elasticsearch that should return one or more results. Elastalert can alert you on spikes, lulls, the absence of events, new values of events, and unique events. With the rules available in Elastalert you can get your search just about as fine as you want it.
For this rule, we will generate a Slack alert any time we get a connection to Tor. You will need a Slack Webhook URL for this to work. If you don’t want to use slack, there are a ton of alerting options to choose from. Select the way you’d like this rule to alert and set the settings appropriately in the rule file:
Create the new file /etc/elastalert/rules/tor_connections.yaml with the following contents, substituting your slack webhook URL or your custom alert settings:
name: Zeek Tor Destination Detected
type: any
index: "zeek-*"
timeframe:
minutes: 1
filter:
- query:
query_string:
query: "tags: zeek.connection AND _exists_:destination.tor"
query_key: "source.ip"
alert:
- "slack"
alert_text: "Tor node {0} was contacted on port {1} by {2}."
alert_text_type: alert_text_only
alert_text_args: ["destination.address","destination.port","source.address"]
slack:
slack_webhook_url: "https://hooks.slack.com/services/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"
slack_icon_url_override: "https://avatars0.githubusercontent.com/u/1194067?v=3&s=200"
#Can be good, warning, danger:
slack_msg_color: "danger"
slack_username_override: "Elastalert"
slack_title: "Zeek: Tor Node Detection"
slack_title_link: "http://localhost:5601/app/kibana#"
After creating the rule, restart Elastalert to make sure that the new rules are loaded:
sudo service elastalert restart
Now verify that we are getting alerts for new tor sessions by faking a connection to Tor again:
nc 79.137.79.167 9001
After a few seconds, you should receive an alert with the connection info.
Next Steps
To continue to improve you SIEM, consider what logs you can collect that offer you a tactical advantage, and what enrichments you can add to your logs to make them more valuable. Every environment will have special considerations that will need to be taken into account when designing and improving your SIEM. Start with logs that help you catch the most-obvious threats first and give you the greatest visibility into your environment.