Exposed AWS Secret Access Key To GitHub Can Be a Costly Affair - A Personal Experience
A personal experience related to securely storying access keys and billing of AWS cloud account, and how it can be a costly affair.
Join the DZone community and get the full member experience.Join For Free
I would like to share an experience which is related to securely storing access keys and billing of AWS cloud account.
6 years back, I have started using AWS Cloud services for one of our project requirements. It was an excitement to start working on the cloud. We started exploring and using different services. For one of the use case, we need to store some documents which should be secure, easily managed, and will be growing as the project feature will grow. We opted for AWS Simple Storage Service.
We have done with the Proof of Concepts and implemented Simple Storage Service in the project without any issue. Features went live to production, so far no issue.
As a common practice, we use to create help documents if done something new so did I, for future reference. I just created a text file, jot down steps to integrate AWS Simple Storage Service along with Secret Access ID and Secret Access Key which we know is used to access any cloud pragmatically, and saved this file locally in a folder outside the project and forgot.
Such keys are confidential and should not be compromised hence we should never keep them outside the project configuration file or AWS or using the Key Management System.
So far so good. A few months later, my system got crashed. I lost almost all my files which I had not stored on the cloud. It was a huge loss but thanks to the IT team which helped to recover some of the files and which included help documents as well :). Very happy at that moment, to secure these files which are more than 50 documents, I decided to move them to cloud.
Here committed the mistake to opt personal GitHub account to store these files. As it was a huge set of files and the only thing that was in my mind was to secure these files, I committed all files to my personal GitHub account.
Within a day or two after commit, the team started receiving tickets from various production sites that one of our services is not working. After checking we found that it is AWS Simple Storage Service which is down. We started digging into the issue.
In the meantime, we received an abuse report from AWS that the Simple Storage Service service has been stopped. We contacted AWS through Support and then came to know the secret access key of Simple Storage Service has been compromised.
Thanks to AWS smart security services which detected immediately in their security scanning. Digging into the issue came as a shock to me that it's my personal GitHub account where one of the text files contains Secret Access Key of Simple Storage Service.
Fine, the first thing we had to restore service at production by resolving the issue as per AWS guidelines:
- Deactivated the current Access Key from Identity and Access Management User Security Credentials
- Generated new Key, integrated and restored the services
- Deleted the compromised files from GitHub account - Deleted as per GitHub guidelines
- Send mail to AWS support that all suggested steps are taken to close Case of abuse report opened by AWS
- Finally most important, report this activity to the client.
There was a glitch here, we missed to delete the inactive key which was there in Identity and Access Management under Users Security Credential i.e. another mistake. It was a miscommunication or misunderstanding, worthless to discuss. As the case was closed by AWS and services were restored, there was nothing left for us to check or monitor.
A few days back while auditing our AWS account using AWS Trusted Advisor which is another very important and useful service by AWS, provides guidance to increase security which is one of the best services among other important features. Here we found Exposed Access Keys in the Security check of the Security report.
Contacted AWS support again, they explained that we were supposed to take two steps. One removes text files from the public domain which I did and another destroys the Secret Access Key from Identity and Access Management Users Security Credential which was missed.
Anyway, we deleted this key from AWS Identity and Access Management and completed all steps.
But there was another shock i.e. Exposed Access Keys report was showing the additional cost to incur which was showing 346$ per day.
What to do now, we again contacted AWS billing support explained all scenarios to the AWS support team that we missed unknowingly and also there has not been any unauthorized activity or abuse.
AWS support team has gone through the case. Thanks to the AWS Support team for understanding and explaining that there will be no charge for this as the key was made inactive by the date (next to cost column there is another column as Deadline) mentioned in the report of Trusted Advisor. This gives me a huge sigh of relief :).
- Never store any Secret Keys or credentials in a folder which is out of the project directory
- Think twice thrice and even more, I must say, before making any commitment to git hub especially when you are committing your personal stuff
- Proper reporting of such incidents to stakeholders
One may not be lucky to save additional costs, so better to follow the above steps diligently.
Opinions expressed by DZone contributors are their own.