As if being a Monday was not enough. I got a call from my ex-cofounder that he received a call from Amazon saying that his credit card has pending AWS bills for $8000 (his credit card details was on the card – thankfully $8000 was way beyond the credit limit). Amount was spread over 2 months – February for around $2000 and March for around 6000$ and there has been some crazy amount of activity going on with the account. I logged in to the above said AWS account and confirmed the claim. Till then, I was thinking it must have been a prank a week before the April Fools day. Now I started to worry.
I was wondering why on earth Amazon would tell me all this after this much damage has been done and not before, or at least they should have sent me the bill statement from the first month? And where on earth is all this usage coming from? And what the fuck is this Elastic compute cloud that is activated for 5 servers from my account? Too many open questions.
Well, Amazon did send emails about the bill and the fact that my account might have been compromised and the fact that the key is shared openly on Github, but unfortunately it was to an admin account for our startup that I didn’t check for the last 3 months as there was no admin activity. That was the only account out of the 10 I have on the startup domain that was not configured properly on my email client. Coincidence?!
When we moved our Github repository from private to public (as the co-founder was leaving and I had to share it with some other prospective co-founders), the AWS keys were all in the open for everyone under the sky to play with. Doing some online search, I figured out we were not the only ones. It’s apparently quite a common mistake developers do.
Going through about 8 to 10 blogs, I figured out Amazon is great organization when it comes to ethics, customer support and dispute resolution and I sent them a long email explaining the scenario. I got a call within the next 5 mins (They have an excellent “Call Me” feature). This is when I realize there is a catch 22 situation. For them to be able to start investigating my case and reverse the charges, they need to know how much they have to reverse. For that the usage of my account needs to stop. I immediately delete the keys from my AWS but apparently EC2 instances will need to be stopped separately and for that credit card details need to be provided to be able to access that subsection of the AWS management console. I had about 3-4 calls with customer support to discuss this situation and I politely mentioned I do not want to give my credit card details as that will mean my card will be charged for something I have not done. They tried to convince me that the charges will “most likely” be reversed, but the damage was huge and I was reluctant.
Over a course of next week, they managed to change some backend settings and requested me to stop the instances but it won’t work. Last resort – they suggested if its possible for me to cancel my account. I had some images stored in S3 – backed them up and cancelled the account. Customer support mention they will now investigate and get back to me with within a week’s time. I indeed got an email after 5 days that they have taken care of the charges on my account.
The best way to minimize risk is to set up AWS billing alerts. You can set up an alert to be notified automatically via e-mail when estimated charges reach a threshold that you choose.