My Experience at PRCCDC
Last weekend I participated in my first Pacific Rim Collegiate Cyber Defense Competition. PRCCDC is an annual two-day event held at Highline Community College. Competing teams representing colleges from Washington, Oregon, and Idaho are charged with maintaining a notional company’s network. The winning team from this regional competition moves on the the National CCDC. University of Washington has won the past six years in a row.
While these student “blue teams” are administrating their networks, various other teams are at work. These teams simulate what can generally be thought of as the worst two days work ever for them. White team is made up of the judges and event coordinators. They created the networks, keep score, and track our performance throughout the weekend. This year saw an orange team who make phone calls to blue teams with variously ridiculous commentary and requests. Most notoriously, red team are the “hackers” who spend the weekend trying to compromise the teams’ systems as best they can.
Scoring was based on five main factors: attacks, incident response, injects, phone calls, and uptime. Successful attacks made by the red team deducted points from a team’s score. Submitting incident responses regained some of these points. Throughout the weekend, tasks and deadlines were delivered to teams as injects. Successful completion of injects earned more points. Team responses to orange team’s phone calls were rated and scored. Finally, the percentage of uptime was weighed into scoring.
A mix of students from our Information Technology and Systems program and from the Grey Hat Group made up our team. Though UWT has traditionally participated in PRCCDC, we did not have a team last year. Thus, only one member of our team had ever been to a CCDC event and most of us had little idea of what to expect. Here is a breakdown of each area of the competition and how the team and myself personally succeeded and failed.
From the beginning, we faced difficulties in preparing the team for competition. We didn’t begin organizing a team until January, which was clearly too late. Also, given that the team was divided between the different programs at the Institute of Technology, schedules always conflicted. The best solution we could come up with was to have two separate groups meeting weekly for training. Communication between these groups was pretty sparse and difficult. It created an unnecessary tension in what should have been a united group.
As far as training goes, we did the best we could. Network layouts are completely unknown before the competition’s start. Having little idea of what to expect, we decided on creating a team structure:
- Team Leader: Receives outside communication and delegates tasks.
- Business Analyst: Assists leader responsible for administrative tasks.
- Domain Controller: Responsible for Active Directory and controlling all Windows access.
- Router: Controls and monitors all traffic into and out of our network and maintains the firewall.
- Database: We were anticipating some implementation of SQL.
- Web: Keeps whatever website we had running and secure.
- Mail: Maintains the company mail server.
- Kali: Tests security from the outside.
We didn’t finalize role assignments until the Friday before the competition.
I think the roles we defined worked well. We had to assume some unknowns, but for the most part these were safe assumptions. We expected a Linux based mail server and we got Ubuntu. We even got the Kali box we expected. We anticipated seeing Active Directory and we did, but we were expecting Windows Server 2012 and we got 2008. Routers threw us a curve ball. We expected a physical Cisco router and instead received a virtualized Vyatta system. Also we were not allowed to use the firewall (yeah, really).
In the future, I would like to have roles assigned much earlier to allow team members to better prepare. We were lucky enough to have some talented people who were able to rise to the challenge, but I this isn’t something to count on moving forward. I would also like to begin much earlier in the year. Our team has five members graduating this year, but three of us, including myself, will begin planning for next year’s competition immediately. I also hope to unite the team for training much more frequently in the future. We need more time and more contact with each other.
Attacks & Uptime
I cannot account for every team mate’s experience, but I can describe mine. I was running the mail server.
When we walked in on day one, we assigned workstations, logged in, changed passwords. I was running a 32-bit Windows XP system. This wasn’t the best news for me; I have been exclusively using Unix-like systems for almost ten years. I had an Ubuntu server though, so I felt all right.
I installed PuTTY on my station and SSH tunnelled into my server. I changed passwords, created a new sudoer account for myself and switched to it. I disabled root SSH and locked all the other accounts on the system in
/etc/shadow. My server was running Postfix, Courier and SquirrellMail over Apache. I am admittedly a n00b when it comes to this, but the set up was pretty straightforward. Virtualizing users using MySQL would have added another layer of complexity that I wasn’t ready for. Injects arrived through email, so their continued arrival was reassuring.
I also picked up responsibility for maintaining backups of all our systems. We had eight workstations, eight servers, and the Kali box. We were given access to and responsibility for our own snapshots using vSphere. I took hourly snapshots and ended up reverting many of the machines several times during both days.
Another (much more experienced) team mate ended up doing most of the hardening of my server as well as our other Linux systems. My server’s uptime was very good, but red team did get in and use it to pivot into the network (compromising our router).
On the second day, we were given two more servers without being told and we never found out about them. They were outside of our subnet and we didn’t receive anything from white team alerting us. Red team had a field day with them without us ever even knowing.
We walked away with a firm understanding of the red team’s tempo. Day one was quiet. They were finding easy ways in, planting something, and backing off. Only on day two did they let us know that they were there. Just because we see no activity does not mean there is no activity. Red team would rather play with us than shut us down; they have a whole weekend to fill.
It turns out that most of the attacks we saw could have been prevented with some fundamentals. Red team was committed to using known exploits. They relied on tools like Metasploit, Cobalt Strike, and Armitage. The most critical steps in securing the servers is to: change all passwords, patch software, and know vulnerabilities and fixes beforehand. The red team doesn’t use 0-day exploits or launch obscure attacks; they are looking for the easiest way in. Default passwords, Telnet, SSH, IPv6, and RDP accounted for the majority of our attacks.
The secret introduction of the two servers on Sunday really hurt us. I think the top scoring teams were the ones who noticed them. Do no trust that any information provided is complete or correct. From the beginning we were finding deficiencies in the network layout we were given. Servers were missing. IP addresses were wrong. Passwords were incorrect. Collect and confirm all data within the team; do not take it for granted. Frequent network scans would have revealed the new servers and saved us a ton of points.
Injects & Phone Calls
I am very proud of our team’s performance in this area. We may have been owned, but we kept our cool and delivered on our tasks non-stop. Throughout the competition we received 28 injects and there was only one that we completely dropped.
We were not as strong on phone calls, but we still did well. Calls were either unverifiable requests for access, customer complaints, or complete nonsense. We were unsure about what kind of weight we should be giving the requests we were receiving over the phone, but we tried to keep a friendly attitude.
We succeeded on injects. We earned more points than any other team in this category. Designating a team leader to interface with outside parties and manage injects was a great call. Also, creating a business analyst position paid off. Almost half of our injects were requests for some kind of document, report or policy. Creating a position devoted to these kind of tasks proved wise.
It turns out, none of the phone calls were legitimate requests that required a response on our end. The purpose of the calls was to rate our customer service. All we need to do on the phone is be friendly. This should make things easier in future competitions as we know that our only focus is on being courteous and professional.
This was another weak point for our team. We filed a few reports throughout the competition, but only for major events like losing access to the SCADA server for an hour. Also, near the end of the competition we spend hours wrestling with red team for control of Active Directory. Our DC person was too busy to submit anything; he needed help from other team mates just to stay on top of the attacks.
We filed a few incident responses, but not nearly enough. We ignored too many incidents because we thought they were minor. It is never a bad idea to report an incident. We need to lower the threshold of what qualifies. We also missed a lot of activity completely. We need to monitor log files and identify suspicious activity. Also, a person involved with addressing an issue is simply too busy to stay on top of reporting as well. We need to develop a system for identifying an incident and delegating its report. Integrate incident response reporting into the actual incident response.
On the whole, I think we succeeded at PRCCDC. We placed 7th out of 12 (perhaps 5th or 6th accounting for some scoring errors). We learned a lot about what to expect and how to prepare. I would like to begin establishing a team and a plan for next year immediately. I would also like to investigate getting some coaching. Most of the other teams received help from college faculty or outside organizations. I think establishing checklists for hardening specific types of systems and software as well as for response to specific attacks would be especially helpful. The first thirty minutes are the most important of the whole weekend. The more you can harden before red team begins, the fewer fires you have to put out as you go. If we can make these steps automatic, we will improve dramatically. Overall, we have a lot of work to do, but I’m happy with our progress so far. Not bad for a first go at it.