Amazon AWS vs Github – “Suspected Unauthorized Activity”

As if being a Monday was not enough. I got a call from my ex-cofounder that he received a call from Amazon saying that his credit card has pending AWS bills for $8000 (his credit card details was on the card – thankfully $8000 was way beyond the credit limit). Amount was spread over 2 months – February for around $2000 and March for around 6000$ and there has been some crazy amount of activity going on with the account. I logged in to the above said AWS account and confirmed the claim. Till then, I was thinking it must have been a prank a week before the April Fools day. Now I started to worry.

I was wondering why on earth Amazon would tell me all this after this much damage has been done and not before, or at least they should have sent me the bill statement from the first month? And where on earth is all this usage coming from? And what the fuck is this Elastic compute cloud that is activated for 5 servers from my account? Too many open questions.

Well, Amazon did send emails about the bill and the fact that my account might have been compromised and the fact that the key is shared openly on Github, but unfortunately it was to an admin account for our startup that I didn’t check for the last 3 months as there was no admin activity. That was the only account out of the 10 I have on the startup domain that was not configured properly on my email client. Coincidence?!

Image

When we moved our Github repository from private to public (as the co-founder was leaving and I had to share it with some other prospective co-founders), the AWS keys were all in the open for everyone under the sky to play with. Doing some online search, I figured out we were not the only ones. It’s apparently quite a common mistake developers do.

Going through about 8 to 10 blogs, I figured out Amazon is great organization when it comes to ethics, customer support and dispute resolution and I sent them a long email explaining the scenario. I got a call within the next 5 mins (They have an excellent “Call Me” feature). This is when I realize there is a catch 22 situation. For them to be able to start investigating my case and reverse the charges, they need to know how much they have to reverse. For that the usage of my account needs to stop. I immediately delete the keys from my AWS but apparently EC2 instances will need to be stopped separately and for that credit card details need to be provided to be able to access that subsection of the AWS management console. I had about 3-4 calls with customer support to discuss this situation and I politely mentioned I do not want to give my credit card details as that will mean my card will be charged for something I have not done. They tried to convince me that the charges will “most likely” be reversed, but the damage was huge and I was reluctant.

Over a course of next week, they managed to change some backend settings and requested me to stop the instances but it won’t work. Last resort – they suggested if its possible for me to cancel my account. I had some images stored in S3 – backed them up and cancelled the account. Customer support mention they will now investigate and get back to me with within a week’s time. I indeed got an email after 5 days that they have taken care of the charges on my account.

Relieved.

The best way to minimize risk is to set up AWS billing alerts. You can set up an alert to be notified automatically via e-mail when estimated charges reach a threshold that you choose.

Advertisements

I don’t like changing passwords!

FrustrationAre you required to change the password of your most frequently visited website once every two weeks and you end up changing it from T1mesSqu@re (your favorite password) to T2mesSqu@re to T3mesSqu@re and so on and by the end of the 8th week you are frustrated because you don’t remember which website is on which number?

How about a system which works something like the SecureID’s two factor authentication (http://en.wikipedia.org/wiki/SecurID) where one part of the authentication is what the user chooses and remembers and the other part is changing.

Let me explain with an example

I am creating an account on http://www.blabla.com. I will be required to choose a password “T_ _mesSqu@re”. Also I will be choosing a method that will define what the two blanks will be. Options for a method can be

  • Sum of DDMMYY up to 2 digits (date can be current date, anniversary date, birth date etc.)
  • Multiplication of date and month and adding the digits to get a 2 digit number
  • .. anything really that is changing  (sum of the digits of the current population of India? 😉

So if I am logging onto blabla.com today (02/08/13), my password (with DDMMYY sum method) will be T14mesSqu@re (Sum is 14). While setting the password we could also have set it like T_mesSqu_@re, in which case the password now will be T1mesSqu4@re.

So now we have a password where even if the one to remember part is compromised, the digits calculated with the method will make sure the total password is still safe (safer?).

Will be happy to get some feedback on the same, or if you have seen anything similar?

JBL flip is a must have gadget for the music buffs

Image

What I needed was a bluetooth speaker that is portable to support my upcoming travels and gives good enough sound and bass levels (I was not expecting to use it to support an Iron Maiden concert). After some online research and couple of hours of walking around the alleys of notorious Nehru Place (largest IT market in India), I shortlisted the JBL Flip. Tagged at INR 5999, you can strike a deal for 15-20% cheaper at Nehru Place.

Pros

  1. Compact and solid built. You can keep it horizontally or vertically.
  2. Recharge and rock up to 5 hours.
  3. Built in microphone
  4. Can be used inside your car (or anywhere) connected to your phone. Has a call pick up button.
  5. Traval handy – one piece and comes with a carrying pouch.

Cons

  1. It cant be USB charged. Comes with an adaptor to recharge the batteries. So if you plan to stay and use it outside for long, Carry another pair of batteries.
  2. Setting it up on a Mac is a bit tricky. The instructions that come with it doesnt work.  This is how you can do it. Click on “Setup Bluetooth device” on your mac and while it searches for available devices, you keep pressing the power button till the Mac detects it (and not for 3s as mentioned in the manual)

Have fun. Rock On.

Application Support – The thin line between you and your customer

After wearing different kinds of Application/ Technical/ Customer support hats in my last three organizations spanning 8 years, I realized I should write about some best practices that are my experiences in this world. They are easy to understand and implement.  They have helped me and my teams in dealing with some of the toughest / craziest customers in the industry – both internal and external to the organization. Time is money so I will keep it short and precise. Most of my experience is from the extremely fast paced Investment banking / Finance domain but all of this should hold true for any other domain / sector.

  1. You should know about a problem before your customers do. Create a list of known problems and “workarounds” that you can give to your customer when they hit the issue and call you for help. For bigger / important / known issues, you should let the customers know in advance. This also means you need to have effective monitoring/alerting systems in place so that your alerts tell you about a situation (e.g. file storage space full, user account on a shared location locked on 3 wrong password attempts) before your customers get to know about it.
  2. Be empathetic and reasonably apologetic. Customers are paying for your product/ services and they hate late/ impersonal replies (even more in the Financial World but really everywhere). Every minute of delay might result in financial/ reputational loss for your customer. It’s obviously not easy to manage expectations but being silent never never never helps. They would like to know they are being attended to and someone in on their case. That gives them assurance and gives you some time to breathe, think and work on their case without being hammered by stinker emails. When something goes wrong, apologize and say sorry. It’s easy and customers like it. The customer may not always be right, but the customer must always win. In most cases talking to your customer on phone helps. Written communication is just 7% of the total communication while vocal tones add 38% to it.
  3. Make sure you repeat your understanding of the problem to the customer (on email and / or phone) and get the customer to confirm that.  You do not want to spend days on a problem only to realize you were solving the wrong one. Most of the times, your language will be different from the customers. You are the expert, so it’s your responsibility to speak in a language that your customer understands. You cannot expect the reverse! Do not make any assumptions and be a good listener concentrating on the tone of voice, words and importantly how they feel. You will learn a lot about actual pain point for the customer.

I remember dealing with a customer from Brazil in my first job with IBM.

Me: “Now that your computer has started, can you see ‘Windows’?”
Customer: “I am sitting next to a window and its beautiful outside”.