Grafana breach caused by missed token rotation after TanStack attack

At a glance:

• A missed GitHub workflow token rotation after the TanStack npm attack led to Grafana's data breach.
• Attackers gained access to private repositories but did not modify code or impact customer production systems.
• Stolen data included business contact information, not customer data from GrafanaCloud.

Breach Overview

Grafana Labs suffered a data breach caused by a missed GitHub workflow token rotation following the TanStack npm supply-chain attack. The company revealed that intruders gained access to private repositories but did not modify any code or compromise customer production systems. This incident highlights vulnerabilities in continuous integration and deployment (CI/CD) pipelines when supply-chain attacks occur.

The breach originated from the Shai-Hulud malware campaign, where malicious TanStack packages were published on npm, infecting developer environments. Grafana’s CI/CD workflow consumed one such package, leading to the exfiltration of a GitHub workflow token that was not rotated in time.

Attack Chain and Vulnerability

The attack chain began when TeamPCP hackers injected credential-stealing code into dozens of TanStack packages on the npm registry. These packages, popular among React developers, were downloaded by various organizations, including Grafana. Once integrated into Grafana’s development environment, the malicious code executed in the GitHub Actions workflow, stealing tokens used for automation.

Grafana detected the malicious activity on May 1 and promptly rotated a significant number of GitHub workflow tokens as part of its incident response. However, a single token was overlooked, allowing attackers to maintain access and infiltrate the company’s private repositories. This oversight underscores the challenges in ensuring complete token revocation after a supply-chain compromise.

Impact Assessment

The attackers downloaded operational information and business contact details, such as names and email addresses exchanged in professional contexts. Grafana clarified that this data was not pulled from or processed through production systems or the GrafanaCloud platform, meaning it did not involve customer-specific information. The company stressed that no customer production data was accessed or exfiltrated.

Furthermore, Grafana confirmed that its codebase remained unaltered during the breach, ensuring that the software distributed to users is safe. Customers are not required to take any action, and the company has not paid any ransom to the attackers. The incident response team continues to investigate the full extent of the breach.

Response and Mitigation Efforts

After detecting the breach on May 1, Grafana immediately deployed its incident response plan, which included rotating GitHub workflow tokens and securing its repositories. A subsequent review revealed that a specific workflow initially thought to be unaffected was indeed compromised due to the missed token. The company has since enhanced its monitoring and token management processes.

Grafana Labs has committed to notifying impacted customers directly if the evaluation changes based on new evidence from the ongoing investigation. The swift containment measures prevented further unauthorized access and minimized potential damage to both the company and its users.

Lessons and Industry Implications

This breach serves as a critical reminder of the importance of rigorous token management and supply-chain security in CI/CD pipelines. Organizations must implement automated rotation mechanisms and continuous monitoring to detect and respond to anomalies promptly. The TanStack incident illustrates how a single missed step can lead to significant security lapses.

In the broader context, supply-chain attacks are becoming increasingly common, targeting trusted software components to infiltrate multiple victims. Companies should adopt a defense-in-depth approach, including regular security audits, multi-factor authentication, and employee training, to mitigate such risks. The Grafana breach underscores the need for proactive security measures in an interconnected development ecosystem.