Recently I took part in a Cyber Defense Exercise aimed at Critical Infrastructure at the Swedish Defense Research Agency, being a member on one of two designated Blue Teams. Each Blue Team was defending their production environment against a Red Team.
This exercise was carried out over 3 days with presentations and evaluations on top of the actual “hands-on” exercise where we were really beat down by the Red Team. We had a number of things stacked against us, an unknown network which was not patched and running outdated software and operating systems as well as internal staff who had hidden Trojans and other malicious code on our machines. On top of that, our firewalls were not configured very well either. As for our team, we did not know each other which also added to the internal chaos. Even though we came out as the better Blue Team, there a a number of things that I learned that I thought I would share.
1. Incident Detection
This was one of things that we were taught to pay attention to. Detection is very hard even though you have an IDS running, just interpreting what you see can be tough and this requires skill. There a number of IDS/IPS solutions out there, and I do not care which one you use, you need training to be able to utilize it properly. We had Snorby running, which is a web-based GUI for IDS, a screenshot seen below. Even though we had this, it was not easy detecting exactly what was going on.
Another thing was visualization. If you visualize the network is it also a lot easier to actually see what is happening in the network. There are different tools out there that can do this, but a open source one is EtherApe. This tool can present a visual representation of your network if it is able to listen on the network traffic as seen from the screenshot below.
Last but not least is system monitoring, to actually know what is happening on your clients and servers. Central logging with an analysis engine running is not a bad idea, to perhaps catch when a user account is being added locally on a system. Again, there a number of tools for central logging and analysis. This is often referred to as SIEM, Security Information and Event Management, and there a number of tools for this as well. I cant argue with the fact that without central management of what is happening on your network and on your systems, it is harder to detect an incident. The only thing I want to really mark as extremely important is the use of time synchronization, without it, SIEM is not gonna work as expected. Every event captured must have a mutual understanding of what the time is, otherwise correlating events and logs is impossible.
2. Incident Response
When you try to resolve an incident, there a number of things to think about. The point is that when you are stressed and you do not have a plan or organization to work with, incident response falls apart. Planning and organization is key, and this is something you have to address before an incident occurs, and evaluate after each incident. The key here is to assign roles with has duties but also mandate to act, such as a site manager, and IT security officer, perhaps a network engineer, and IDS engineer and so on … it all depends on your internal structures. To have all these roles designated is vital as it allows you to assign the correct people with the correct skills to an incident. Once something happens you usually have very little time to invent things like an incident policy nor an incident organization.
One key factor is the skill of your team. An incident response team needs training, simple as that. At least try and run drills in house using simulations, these can be quite effective. Spending time doing this might prove it’s worth once something really happens. If you are unprepared when an incident occurs, your incident response will simply not be very efficient. There is a reason that firefighters train as much as they do, being unprepared is is simply not an option for them, and you should consider having the same mindset in your organization.
3. Be Prepared
This has already been said, but I cant stress this enough. Once the shit hits the fan so to speak, unless you have prepared to deal with it, it is gonna hurt a lot more than if you did. Put in the real work before hand and you will manage an incident a lot better. Handling an incident is almost never easy but it can quickly become overwhelming if you are unprepared.
4. Learn from it
Last but certainly not least, learn from every incident. What did you do well and perhaps not so well? Did your plan and organization work as expected, or perhaps something needs to change? Be your own worst critic, you are the one who will benefit from it the next time an incident occurs.