Twilio’s Response to the Recent Codecov Vulnerability

Twilio believes that the security of our products and our customers’ data is of paramount importance and when an incident occurs that might threaten that security, we tell you about it. To that end, we wanted to provide an overview of the impact we experienced from the recently disclosed Codecov vulnerability and how we managed that event.

What happened?

On April 15, 2021, Codecov publicly disclosed a security event where an attacker modified the Bash Uploader component which enabled the attacker to potentially export information stored in continuous integration (CI) environments. Twilio was notified of the event by Codecov and immediately began our security incident response process. We have Codecov tools, including the Bash Uploader component, in use in a small number of our projects and CI pipelines. These projects and CI pipelines are not in the critical path to providing updates or functionality to our communication APIs.

Our subsequent investigation into the impact of this event found that a small number of email addresses had likely been exfiltrated by an unknown attacker as a result of this exposure. We have notified those impacted individuals privately and have remediated the additional potential exposure by thoroughly reviewing and rotating any potentially exposed credentials.

What have we done?

As soon as we became aware of the event, we identified any potentially exposed credentials or secrets and rotated them. This removed any ability the bad actor would have had to access our environment. Additionally, we investigated the scope of those credentials and validated, to the best of our ability, that there hadn’t been any abuses of them.

On April 22, 2021, we received a notification from GitHub.com that suspicious activity had been detected related to the Codecov event and a Twilio user token that had been exposed. GitHub.com had identified a set of GitHub repositories that had been cloned by the attacker in the time before we were notified by Codecov. Our investigation turned from identifying secrets to identifying the content of the repositories that were cloned.

We wanted to be certain that we covered any potential exposures, so we conducted a deep-dive review of our repositories. In one of them, we found a small number of email addresses belonging to Twilio customers. At this time, we have no evidence that any other customer information was accessed and no indication that any of our repositories were modified by the attacker.

Further, we performed automated scanning to detect secrets in our repositories and manual reviews to verify findings. This resulted in rotating all secrets contained in possibly exposed repos.

What are we doing to prevent similar issues in the future?

With respect to protecting Twilio against supply chain risks such as this, we have a robust third party security team that evaluates both new and existing vendors. This process ensures our technology supply chain always meets our standards for security. When we become aware of an incident or vulnerability within that supply chain we move quickly to remediate the issue or remove the software from our environment.

Further, while it would not have prevented this particular issue where the secrets were compromised due to a supply chain attack, to prevent inadvertent leakage of secrets, we have an active internal service, called Deadshot, that scans GitHub pull requests. The service scans pull requests in real time to identify secrets and other common insecure coding practices in code being merged to GitHub. If Deadshot finds insecure code, it notifies the user doing the pull request and notifies our Product Security team when a specific type of secret is found. This allows developers to go back and delete or change their code before merging it to GitHub.

Our Product Security team also maintains tooling within our environment to conduct Static Application Security Testing. This tooling will perform scans for secrets in code, as well as look for security issues, such as unsafe coding practices, and vulnerabilities, such as the OWASP Top 10. In this way, if there is unauthorized access to GitHub repositories containing our code, we contain the further damage that could be done by such an unauthorized actor.

Again, at this time, we have no indication that any customer data, beyond the small number of email addresses, was accessed or is at risk. We also do not have or foresee any issues with the availability or functionality of any Twilio products. The Twilio Security Incident Response Team will post any updates here if that changes. If you have further questions, please reach out to your customer support partner.