My Linode account got compromised!
5 days ago I got an email regarding events from Linode. Usually I tend to ignore these emails because from time to time they describe linode running some maintenance on my image.
Luckily, this time I didn’t ignore this email. To my surprise, I found this:
* ufljsc1r-eu-central - (215220139) System Shutdown - Completed Wed, 30 Sep 2020 20:00:58 GMT * ufljsc1r-eu-central - (215220141) Inactivate Linode - Completed Wed, 30 Sep 2020 20:00:59 GMT * linode2723253-us-west - (215220388) Linode Initial Configuration - Completed Wed, 30 Sep 2020 20:02:48 GMT * linode2723253-us-west - (215220389) Create Disk - linode2723253 - Debian 8 Disk - Completed Wed, 30 Sep 2020 20:02:48 GMT * linode2723253-us-west - (215220390) Thawing Disk - linode2723253 - Debian 8 Disk - Completed Wed, 30 Sep 2020 20:02:49 GMT * linode2723253-us-west - (215220391) Create Swap - Completed Wed, 30 Sep 2020 20:03:33 GMT * linode2723253-us-west - (215220392) System Boot - My linode2723253 - Debian 8 Disk Profile - Completed Wed, 30 Sep 2020 20:03:34 GMT * ufljsc1r-eu-central - (215193391) System Shutdown - Completed Wed, 30 Sep 2020 14:54:25 GMT * ufljsc1r-eu-central - (215193416) Change root password -  Ubuntu 18.04 LTS Disk - Completed Wed, 30 Sep 2020 14:54:50 GMT * ufljsc1r-eu-central - (215193438) System Boot - My Ubuntu 18.04 LTS Profile - Completed Wed, 30 Sep 2020 14:55:17 GMT * ufljsc1r-eu-central - (215193687) System Shutdown - Completed Wed, 30 Sep 2020 14:59:38 GMT
This didn’t seem right! I didn’t remember seeing this kind of message before. I logged into to my account( my password still worked) but then there were a bunch of instances I didn’t recognize the activity. As you can see from the text above a bunch of instances were created and deleted. And in that list the instance that was running this site also was deleted. So for a good few hours my site was down not that it matters :P.
I immediately panicked! And reached out to Linode support because I didn’t have backups turned on! Rather I felt like paying extra money to backup this site’s server wasn’t really money well spent. Some could also argue why pay to host a site in the first place at all but then I like having a server. That gives me freedom to tryout different things (hugo, wordpress) before I ended up making Hakyll the static site generator for this website but I digress!
I also tweeted @Linode account to understand if my account was a random account that got compromised or if Linode’s security was somehow breached and alot of accounts were compromised. Linode responded quickly to confirm that my account was freak show.
my @linode account got compromised or my server instance got deleted. Wondering if this is a one off or Linode as whole has been completely compromised #linodehacked ?— Abhinav G (@abhixec) September 30, 2020
So I quickly changed my password on this account and also added 2FA on this account. The reason I didn’t setup 2FA sooner was because I didn’t really think my account/site was worth anyone’s effort! But lesson learned.
Linode was very prompt in getting me a snapshot of my instance that was running this site. I didn’t quite think that they would oblige since they had a feature that customers had to pay extra to ensure that my instance was constantly being backed up for a situation just like this. But kudos to Linode! They really won me over on this!
awesome job from @linode support to quickly get my server image back up #kudos Thanks a bunch!— Abhinav G (@abhixec) September 30, 2020
At this point I deleted all the unknown instances that was started by the person who gained access to my account. And thought that my account was now secured and damage reverted since I was easily able to boot backup from the saved snapshot given to me by the Customare Support from Linode.
As I was trying to check/analyze the damage of the account breach. A new instance got spun up. This spooked me up a bit. Upon investigating further, found out that the instances were being spun up from stack scripts.
My default instinct prompted me to delete those stack scripts and so I did. But as I deleted them I realized I should have opened those scripts to see what exactly they did. /me feels sad
If you ever find yourself in this situation here are things you should look do to secure yourself.
List of TODOs:
- Change password (should be a default thing to do)
- Set 2FA
- Make sure there aren’t any unauthorized apps or tokens that have access to your account.
- Check to make sure that there aren’t any stack scripts running. 4.1 Bonus point if you have curiosity to look at what the script does.
- Run a scan on all your online accounts on different service and make sure that you change the password if by faint chance they share same username/password.
One lesson that I learned from this incident is to enable 2FA. The flip side of this is if I ever decide to change my phone then I need to remember to move the accounts over to my new phone before I decommision the old one.
I remember couple of my online accounts for which I had 2FA setup on my older phones and now had to go through arduous process to get my account unlocked because I had totally forgotten that I had setup 2FA.
But as 2FA becomes more and more of a necessity I think this will start to become a part of the transfer protocol when I switch smartphones.
Earlier in the year my Etsy account was compromised, more on that in another post!