Ids on Inliniac

Suricata bits, ints and vars

Tue, 20 Dec 2016 18:37:05 +0000

Since the beginning of the project we’ve spoken about variables on multiple levels. Of course flowbits defined by the Snort language came first, but other flow based variables quickly followed: flowints for basic counting, and vars for extracting data using pcre expressions.

I’ve always thought of the pcre data extraction using substring capture as a potentially powerful feature. However the implementation was lacking. The extracted data couldn’t really be used for much.

Suricata 3.0 is out!

Wed, 27 Jan 2016 15:33:27 +0000

Today, almost 2 years after the release of Suricata 2.0, we released 3.0! This new version of Suricata improves performance, scalability, accuracy and general robustness. Next to this, it brings a lot of new features.

New features are too numerous to mention here, but I’d like to highlight a few:

netmap support: finally a high speed capture method for our FreeBSD friends, IDS and IPS
multi-tenancy: single instance, multiple detection configs
JSON stats: making it much easier to graph the stats in ELK, etc
Much improved Lua support: many more fields/protocols available, output scripts

Check the full list here in the announcement: http://suricata-ids.org/2016/01/27/suricata-3-0-available/

New Suricata release model

Tue, 24 Nov 2015 15:54:35 +0000

As the team is back from a very successful week in Barcelona, I’d like to take a moment on what we discussed and decided on with regards to development.

One thing no one was happy with is how the release schedules are working. Releases were meant to reasonably frequent, but the time between major releases was growing longer and longer. The 2.0 branch for example, is closing in on 2 years as the stable branch. The result is that many people are missing out on many of the improvements we’ve been doing. Currently many people using Suricata actually use a beta version, of even our git master, in production!

Suricata has been added to Debian Backports

Thu, 08 Jan 2015 00:34:50 +0000

Thanks to the hard work of Arturo Borrero Gonzalez, Suricata has just been added to the Debian ‘backports’ repository. This allows users of Debian stable to run up to date versions of Suricata.

The ‘Backports’ repository makes the Suricata and libhtp packages from Debian Testing available to ‘stable’ users. As ’testing’ is currently in a freeze, it may take a bit of time before 2.0.5 and libhtp 0.5.16 appear.

Anyway, here is how to use it.

Profiling Suricata with JEMALLOC

Tue, 23 Dec 2014 15:34:23 +0000

JEMALLOC is a memory allocation library: http://www.canonware.com/jemalloc/

It offers many interesting things for a tool like Suricata. Ken Steele of EZchip (formerly Tilera) made me aware of it. In Ken’s testing it helps performance.

Install

wget http://www.canonware.com/download/jemalloc/jemalloc-3.6.0.tar.bz2
tar xvfj jemalloc-3.6.0.tar.bz2
cd jemalloc-3.6.0
./configure --prefix=/opt/jemalloc/
make
sudo make install

Then use it by preloading it:

LD_PRELOAD=/opt/jemalloc/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

I haven’t benchmarked this, but if you’re running a high performance setup it may certainly be worth a shot.

Crossing the Streams in Suricata

Sat, 20 Dec 2014 23:27:24 +0000

At it’s core, Suricata is a packet processor. It reads packets and pushes them through a configurable pipeline. The 2nd most important processing unit in Suricata is the flow. In Suricata we use the term flow for the bidirectional flows of packets with the same 5 tuple (proto, src ip, dst ip, sp, dp. Vlans can be added as well). In fact, much of Suricata’s threading effort revolves around the flow. In the 2 main runmodes, autofp and workers, flow based load balancing makes sure that a all packets of a single flow always go through the same threading pipeline. In workers this means one single thread, in autofp 2: the capture thread and a stream/detect/output thread.

SMTP file extraction in Suricata

Tue, 11 Nov 2014 10:47:42 +0000

In 2.1beta2 the long awaited SMTP file extraction support for Suricata finally appeared. It has been a long development cycle. Originally started by BAE Systems, it was picked up by Tom Decanio of FireEye Forensics Group (formerly nPulse Technologies) followed by a last round of changes from my side. But it’s here now.

It contains:

a MIME decoder
updates to the SMTP parser to use the MIME decoder for extracting files
SMTP JSON log, integrated with EVE
SMTP message URL extraction and logging

As it uses the Suricata file handling API, it shares almost everything with the existing file handling for HTTP. The rule keyword work and the various logs work automatically with SMTP as well.

Suricata Flow Logging

Mon, 28 Jul 2014 19:09:07 +0000

Pretty much from the start of the project, Suricata has been able to track flows. In Suricata the term ‘flow’ means the bidirectional flow of packets with the same 5 tuple. Or 7 tuple when vlan tags are counted as well.

Such a flow is created when the first packet comes in and is stored in the flow hash. Each new packet does a hash look-up and attaches the flow to the packet. Through the packet’s flow reference we can access all that is stored in the flow: TCP session, flowbits, app layer state data, protocol info, etc.

Video: Suricata 2.0 installation and quick setup

Sat, 29 Mar 2014 22:01:56 +0000

I’ve made a video on installing Suricata 2.0 on Debian Wheezy. The video does the installation, quick setup, ethtool config and shows a simple way to test the IDS.

It’s the first time I’ve made such a video. Feedback is welcome.

More on Suricata lua flowints

Tue, 23 Apr 2013 10:17:52 +0000

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:
    a = ScFlowintIncr(0)
Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Suricata Lua scripting flowint access

Mon, 22 Apr 2013 16:16:30 +0000

A few days ago I wrote about my Emerging Threats sponsored work to support flowvars from Lua scripts in Suricata.

Today, I updated that support. Flowvar ‘sets’ are now real time. This was needed to fix some issues where a script was invoked multiple times in single rule, which can happen with some buffers, like HTTP headers.

Also, I implemented flowint support. Flowints in Suricata are integers stored in the flow context.

Suricata: Handling of multiple different SYN/ACKs

Fri, 19 Apr 2013 07:53:00 +0000

When processing the TCP 3 way handshake (3whs), Suricata’s TCP stream engine will closely follow the setup of a TCP connection to make sure the rest of the session can be tracked and reassembled properly. Retransmissions of SYN/ACKs are silently accepted, unless they are different somehow. If the SEQ or ACK values are different they are considered wrong and events are set. The stream events rules will match on this.

Suricata 1.4 is out

Thu, 13 Dec 2012 17:54:22 +0000

About 5 months after 1.3 came out we’ve released 1.4, and we’ve been quite busy. Eric Leblond’s post here has all the stats and graphs. There are three big new features: unix socket, ip reputation and luajit. For each of these the same is true: it’s usesable now, but it’s the potential that we’re most excited about. Over the next months we’ll be extending each of those to be even more useful. We’re very much interested in ideas and feedback.

IPv6 Evasions, Scanners and the importance of staying current

Tue, 11 Dec 2012 16:13:49 +0000

Lots of activity on the IPv6 front lately. There was a talk on a conference on bypassing IDS using IPv6 tricks. Also a new scan tool (Topera) claimed to scan a host while staying below the radar of an IDS was released. To start with the latter, even though Suricata doesn’t have a dedicated port scan detector, the tool’s traffic lights up like a Christmas tree. The trick it pulls is to pack a lot of duplicate DST OPTS extension headers in the IPv6 packets. These options are just fillers, the only options they use are the “pad” option. In Suricata we’ve had an event for duplicate DST OPTS headers since 1.3 and the padding only headers generate an event in 1.4. Both alerts will be very noisy, so calling this a stealth attack rather dubious.

Closing in on Suricata 1.4

Thu, 29 Nov 2012 16:50:15 +0000

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

IP Reputation in Suricata

Wed, 21 Nov 2012 19:22:01 +0000

Disclaimer: this work was sponsored by Emerging Threats Pro.

One thing we’ve been talking about for many years at OISF is IP Reputation. The basic idea is that many organizations have information about specific IP-addresses. This information may be that a host is infected, acts as a spam relay or many other things. We’ve always thought it might be useful to apply this info to the IDS directly.

In the last weeks I’ve developed code to load IP reputation information into Suricata. This code is now part of the Suricata git master, so it’s available to all.

Important Suricata update

Thu, 01 Nov 2012 18:16:51 +0000

We just released Suricata 1.3.3 which contains some important accuracy fixes. Also, it should be much more robust against out of memory conditions.

For those of you running Suricata in IPS mode, this is important as well. We found that rules that have the drop or reject actions, were not playing well with thresholding.

So upgrading is highly recommended!

Code changes are not too big, largest changes are due to some extra unittests:

Suricata on Myricom capture cards

Tue, 10 Jul 2012 15:22:02 +0000

Myricom and OISF just announced that Myricom joined to OISF consortium to support the development of Suricata. The good folks at Myricom already sent me one of their cards earlier. In this post I’ll describe how you can use these cards already, even though Suricata doesn’t have native Myricom support yet. So in this guide I’ll describe using the Myricom libpcap support.

Getting started

I’m going to assume you installed the card properly, installed the Sniffer driver and made sure that all works. Make sure that in your dmesg you see that the card is in sniffer mode:

Suricata http_user_agent vs http_header

Mon, 09 Jul 2012 18:43:12 +0000

One of the new features in Suricata 1.3 is a new content modifier called http_user_agent. This allows rule writers to match on the User-Agent header in HTTP requests more efficiently. The new keyword is documented in the OISF wiki. In this post, I’ll show it’s efficiency with two examples.

Example 1: rarely matching UA

Consider a signature where the match if on a part of the UA that is very rare, so not part of regular User Agents. In my example “abc”.

Suricata MD5 blacklisting

Sat, 09 Jun 2012 08:52:22 +0000

For a few months Suricata has been able to calculate the MD5 checksum of files it sees in HTTP streams. Regardless of extraction to disk, the MD5 could be calculated and logged. Martin Holste created a set of very cool scripts to use the logged MD5 to look it up at VirusTotal and some other similar services. This is done outside of Suricata. One thing I have been wanting to try is matching against these MD5’s in Suricata itself.

Suricata scaling improvements

Tue, 29 May 2012 15:52:52 +0000

For the Suricata 1.3beta1 release, one of our goals was to improve the scalability of the engine when running on many cores. As the graph below shows, we made a good deal of progress.

The blue line is an older 1.1 version, the yellow line is 1.3dev. It clearly shows that 1.1 peaked at 4 cores, then started to get serious contention issues. 1.3dev scales nicely beyond that, up to 24 cores in this test (four 6core AMD cpu’s). Tilera recently demonstrated Suricata on their many core systems, running a single Suricata process per cpu. Their cpu’s have 36 real cores.

HTTP parsing events in Suricata

Wed, 11 Jan 2012 19:09:17 +0000

With the 1.2rc1 release you will notice no more HTTP errors on the screen. Or SMTP errors. This output has been disabled finally. This was a long time annoyance.

As you may still be interested in the errors they are now available through the rule language. In rules/http-events.rules and rules/smtp-events.rules rules for all possible events/errors can be found.

Example: app-layer-event:http.missing_host_header;

This will match on HTTP/1.1 requests without a Host header.

Suricata 1.1.1 released

Wed, 07 Dec 2011 18:34:50 +0000

A maintenance update for the Suricata 1.1 series was just released. It fixed an important issue. In some cases Suricata could crash on SMTP traffic.

The full announcement for the 1.1.1 release is here.

Naturally, the issue has also been fixed in the 1.2 development branch.

File extraction in Suricata

Tue, 29 Nov 2011 16:27:27 +0000

Today I pushed out a new feature in Suricata I’m very excited about. It has been long in the making and with over 6000 new lines of code it’s a significant effort. It’s available in the current git master. I’d consider it alpha quality, so handle with care.

So what is this all about? Simply put, we can now extract files from HTTP streams in Suricata. Both uploads and downloads. Fully controlled by the rule language. But thats not all. I’ve added a touch of magic. By utilizing libmagic (this powers the “file” command), we know the file type of files as well. Lots of interesting stuff that can be done there.

Suricata 1.1 released, 1.2 on the horizon

Thu, 10 Nov 2011 16:51:52 +0000

Today we released Suricata 1.1. This ends a rather long development cycle of more than a year. And it shows. Performance, accuracy and features were all greatly improved. I think it’s the best Suricata so far. If you’ve been looking at trying Suricata, now might be a good time to jump in.

The long development cycles should be something of the past. At our last brainstorm session, at RAID 2011, we decided to change our release policy. The aim of this policy is to do time based releases, roughly a “stable” every 2 months and a beta every other month. This way we’ll be making it much easier for users to stay current without have to run our “git master”.

RAID 2011 Thoughts

Sat, 24 Sep 2011 16:09:24 +0000

The last few days I’ve been at the Recent Advances in Intrusion Detection (RAID) conference in California. Overall it has been a very pleasant and interesting experience. The nice California weather was certainly helping a lot!

I’ve seen all talks and some were very interesting. However, being a Suricata IDS developer, I was not just interested in research for the hell of it, but I was actively scouting for ideas we could implement into Suricata. In this respect the conference was highly disappointing. Although with some of the talks I thought the idea was applicable in general security, like Erik Bosmans high speed memory tainting detection, I found nothing like that for NIDS.

Listening on multiple interfaces with Suricata

Fri, 24 Dec 2010 13:13:24 +0000

A question I see quite often is, can I listen on multiple interfaces with a single Suricata instance? Until now the answer always was “no”. I’d suggest trying the “any”-pseudo interface (suricata -i any), with an bpf to limit the traffic or using multiple instances of Suricata. That last suggestion was especially painful, as one of the goals of Suricata is to allow a single process to process all packets using all available resources.

Suricata 1.1 beta 1 released

Tue, 21 Dec 2010 17:56:32 +0000

Today we’ve released Suricata 1.1 beta 1, the first beta of the upcoming Suricata 1.1 release. The official release announcement is here on the OISF website.

The main focus of the new release has been to improve performance and to add support to the features the new ET/ETpro ruleset needs. ET and ETpro have rulesets specially tuned and geared for Suricata. We’re still missing some new rule keywords that are used by VRT, so in the 1.1 beta 2 release we’ll address that.

Setting up Suricata 0.9.0 for initial use on Ubuntu Lucid 10.04

Mon, 10 May 2010 14:27:25 +0000

The last few days I blogged about compiling Suricata in IDS and IPS mode. Today I’ll write about how to set it up for first use.

Starting with Suricata 0.9.0 the engine can run as an unprivileged user. For this create a new user called “suricata”.

useradd –no-create-home –shell /bin/false –user-group –comment “Suricata IDP account” suricata

This command will create a user and group called “suricata”. It will be unable to login as the shell is set to /bin/false.

Compiling Suricata 0.9.0 in Ubuntu Lucid 10.04 in IDS mode

Fri, 07 May 2010 08:16:53 +0000

Note: the difference with the 0.8.2 post is that addition of libcap-ng-dev. This allows Suricata to run as an unprivileged user.

Ubuntu Lucid 10.04 LTS looks like a good platform for running an IDS on. It’s up to date and has long term support. Here is how to compile and install Suricata 0.9.0 on it.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev libcap-ng-dev

Suricata 0.9.0 released

Fri, 07 May 2010 08:08:45 +0000

Yesterday we released we first release candidate for our upcoming 1.0 release of Suricata. See the announcement on the OISF site here.

Most notable changes are the following new features:

- Support for the http_headers keyword was added

- libhtp was updated to version 0.2.3

- Privilege dropping using libcap-ng is now supported

- Proper support for “pass” rules was added

- Inline mode for Windows was added

Go get the release here: http://www.openinfosecfoundation.org/download/suricata-0.9.0.tar.gz

Compiling Suricata 0.8.2 in Ubuntu Lucid 10.04 in IDS mode

Fri, 30 Apr 2010 18:35:51 +0000

The newly released Ubuntu Lucid 10.04 LTS looks like a good platform for running an IDS on. It’s up to date and has long term support. Here is how to compile and install Suricata 0.8.2 on it.

Install the following packages needed to build Suricata: libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev.

apt-get install libpcre3-dev libpcap-dev libyaml-dev zlib1g-dev

Download Suricata 0.8.2 here

Extract the suricata-0.8.2.tar.gz file as follows:

tar xzvf suricata-0.8.2.tar.gz

Enter the extracted directory suricata-0.8.2.

Suricata 0.8.2 released

Mon, 19 Apr 2010 20:38:28 +0000

Today the OISF development team released 0.8.2 of the Suricata IDS/IPS engine. I feel this is definitely the best release so far. Read the announcement here. In short, stability was improved, memory footprint reduced, performance improved and new features were added.

One of the tools we used to help improve the engine is a fuzzer created by Will Metcalf, our QA lead. In short, the script takes a pcap file, runs it through editcap (part of wireshark) altering a number of random bytes, then feeds the altered pcap file to Suricata. This resulted in many interesting corner cases. Naturally the script makes sure you don’t forget to enable “ulimit -c unlimited” and such :) More on that script can be found on Will’s blog node5.

Suricata 0.8.1 released

Sat, 20 Feb 2010 16:14:41 +0000

Yesterday the OISF development team released Suricata 0.8.1. This release is much improved from our December 31st release. It is way more stable, performs better and has more features. Thanks to the now included HTP library we have much better HTTP handling. The stream engine has seen massive improvements. Initial experimental CUDA code has been added. Initial Win32 support has been added. We’ve added number of missing rule keywords. Many bugs were fixed.

Quickdraw beta release

Tue, 30 Jun 2009 08:16:18 +0000

Next to creating a new IDS with the OISF project I’ve been busy lately assisting Digital Bond with their Quickdraw project. The purpose of the project is to create a passive network based event logger for SCADA networks. Digital Bond has now released a first beta of the project here. Check it out!

OISF IDS/IPS engine prototype intro

Wed, 07 Jan 2009 11:24:07 +0000

For over a year I’ve been working on a prototype implementation of a new IDS/IPS engine for the Open Infosec Foundation. This is not necessarily going to be the engine we’ll be using in OISF, although it’s likely that at least some of the code will be used. Discussions about features for the engine are still ongoing ( wiki, list), once that settles down we’ll see whats usable and whats not. In the worst case I still think many parts like hashing functions, pattern matcher implementations, protocol decoders, etc can be used.

Available for contract work

Mon, 05 Jan 2009 13:26:06 +0000

This year there will be a lot of work that needs to be done for the Open Infosec Foundation. And like I wrote a few days ago, a lot of work is already being done. However, most of it is unpaid at this time as it will be some months before our funding comes in. So at least until then I’m available and looking for contract work.

For the last two years I’ve been doing work as a contractor in the (open source) security field. My experience is mostly in coding in C and Perl, primarily on Snort and Snort_inline. Recently I created the (Perl language) SidReporter program for Emerging Threats. Areas I worked in: IPv6 IDS/IPS coding, signature writing, Web Application Firewalls, threading, bandwidth accounting, and more…

Looking forward to 2009: Open Infosec Foundation

Mon, 29 Dec 2008 12:15:20 +0000

The year 2008 was an exciting year to me. The biggest thing going on the infosec side was the formation of the Open Infosec Foundation. We’ve been working on it behind the scenes for more than a year now, and it’s cool that we’ve finally announced our plans. Of course, the work is just getting started. Next year, we expect to finalize our foundation setup. We’re working with the Software Freedom Law Center for setting up the foundation charter and consortium rules. While the US government is funding us initially, we hope the consortium will guarantee our long term funding. We are talking to some interesting companies already, both big and small.

Open Infosec Foundation founded!

Fri, 17 Oct 2008 22:07:59 +0000

Last week Matt Jonkman announced the formation of the Open Infosec Foundation. This foundation has been grant funded to create a new open source IDS/IPS engine. Together with Will Metcalf and of course Matt himself, I will be working on this. We want this to be a real community effort where there is a role for everyone in the infosec community. Developers, admins, vendors, goverments, research, education, everyone. There is a lot of work ahead, but that should be great fun and very inspiring. So far things are interesting already. The discussion mailinglist is growing rapidly with many ppl from the community and industry. A #oisf IRC channel was created today on freenode. Join us there to participate in discussion about this project!