It was a nice early Sunday afternoon. I was working on some content for the blog. All of a sudden I got a call from Paweł, another Predica’s board member: “Hey, I think our blog got hacked!” Well. It turned out Paweł was right – we’ve got HACKED! There’s no better way to learn something than to experience it on your own. Right? Let’s go through this together.

I bet no one wants to face such an accident, but the truth is – it will happen at least once in every organization. In yours too. “ASSUME BREACHES” – that’s our brand new motto to follow in information security!

It is my responsibility as Predica’s CTO to handle this kind of incidents. It is also my responsibility to step forward and say – WE FAILED. But well, every time you fail, you also learn – so I decided to share this story with you, and perhaps keep you a little bit closer to the safety gate. 

Why are we sharing this?

Before I jump into what happened and how we managed to fix it, I will answer the question that’s probably showing up right now in your head – WHY AM  I SHARING such a case? We could just find out what had happened, fix it and move forward. No one would notice, and those who did will forget about it in a few days time.

Yep. That is probably how it might be done by the rest of the world, but this is not the way we do things at Predica. Well, all in all, sharing is carrying, right? Simple as that. We believe that exchanging knowledge and experience is an important part of our contribution and a huge added value.

We need to figure out a more open approach and information sharing between organizations in the security industry. If more information is shared among the companies about security incidents, we will get more secure as a whole. Even if this case triggers only ONE company to do something to protect itself against the same threat or improve its operations – we will meet the goal.

OK, now let’s jump to the subject as everyone is waiting for it – all the details.

The incident – what happened?

On Sunday, Feb 2nd we’ve noticed that one of our blog posts – an excellent Marek’s piece about Office 365 external sharing WAS DEFACED.

It occurred with a headline and the content itself quite a bit different from the intended. Other parts of the website looked normal, and the content was not modified… However, this actually looked BAD and no one wants to ignore such a mess.

The site affected by the breach was predica.pl/blog – hosted on external infrastructure and not connected in any way to our internal sites and assets. Neither our company’s assets related to our projects has suffered nor any data we store in our systems.

The incident – action plan

Our first action was to perform a quick analysis of the environment and assess potential downfalls. We got our team responsible for the blog site on the call and immediately started to work on assessment and resolution.

• The first step was to take down the affected post. Since it was broadly promoted, and it linked to our social profiles – we preferred to have HTTP/404 instead of getting people to this web page.
• The next step was to assess what was the attack vector and whether we need to take down the entire site to make sure no one got hurt by the content, or it is less severe and well-contained.

Another team was working on finding out what could happen and very quickly we have found out the cause. On January 26’th WordPress (powering our blog) has released new version 4.7.2 which contained security fix for the new vulnerability.

It fixed security vulnerability specific to WordPress 4.7.1 which we were running: Unauthenticated Privilege Escalation Vulnerability in a REST API Endpoint.

Regardless of what was written on the WP blog on the vulnerability, the exploit had been already available and used in the wild (pretty successfully as we were not alone in this case.

This is where we failed…

Simple as that – it was our blog running on the affected version of WordPress.

What were the actions taken to close down this problem?

• WordPress version was updated to 4.7.2, and all the plugins were checked for the latest versions (all were up to date)
• We’ve isolated the point in time when this incident has happened (4 hours before detected), and we have compared site content against the previous backup (using hashes of each file) to get to know if something else was modified ( NO MODIFICATIONS! )
• All user passwords were reset as a precaution step – we have no evidence of passwords being compromised, but… better be safe than sorry.
• The content on the blog was restored from the last backup from before the incident has happened.

Success – the blog was back! What we are doing right now is reviewing logs to check for possible other activities.

Wrap-up

To summarize the case:

• Cause: WordPress engine has not been updated in time when an important security update was released.
• Technical cause: An exploitation of the well-known security vulnerability in WordPress platform.
• Scope: Contained within Predica blog site. Only public website affected.

Lesson learned

There are actually a few lessons learned for us but also for others who maintain systems and have to keep them secure:

• No one is immune, and a breach will happen. You need to act in this mode and have procedures in place to handle incident when detected.
• The time to patch systems is quite short before someone exploits a recognized vulnerability. Take this into consideration especially for services to face public Internet connection. It also applies to the internal infrastructure.

Checked. In very short words – PATCH YOUR WEBSITE REGULARLY! Or if you can – move it to the cloud. If the service was running in SaaS, the patch would be applied by a service provider much faster than we did.

I will do a follow-up to this post to describe also some changes we made in our infrastructure after the whole accident happened. Well, this time it looks like it wasn’t that bad. My word for you – stay secure, patch your systems and always ASSUME BREACH!