JetBrains, the maker of a continuous integration and delivery (CI/CD) server platform called TeamCity, and cyber security firm Rapid7 are trading blows over the handling of two serious vulnerabilities in the service as customers rush to patch in the face of confirmed exploitation.
The two issues in question are tracked as CVE-2024-27198 and CVE-2024-27199. The first is an authentication bypass flaw in TeamCity’s web component via an alternative pass issue, with a CVSS base score of 9.8, meaning it is a critical issue. The second has the same effect, but has a CVSS base score of 7.3.
In a blog posting detailing the issues, Rapid7 principal researcher Stephen Fewer, who discovered the vulnerabilities, wrote: “Compromising a TeamCity server allows an attacker full control over all TeamCity projects, builds, agents and artefacts, and as such is a suitable vector to position an attacker to perform a supply chain attack.”
At the core of the disagreement lies a difference in approaches to vulnerability disclosure and patching.
The vulnerabilities were disclosed to JetBrains via its coordinated disclosure policy on Thursday 15 February 2024. JetBrains acknowledged this on Monday 19 February and reproduced the issues on Tuesday 20 February after being provided with technical analysis by Rapid7.
In Rapid7’s version of the narrative, JetBrains then suggested releasing patches privately before a public disclosure. It responded by emphasising the importance of coordinated disclosure, and outlined its stance against so-called silent patching.
Things then went quiet for several days until Friday 1 March, when Rapid7 went back to JetBrains and restated a request for more information about affected versions of TeamCity and vendor mitigation guidance. It was advised of the assigned CVE numbers, but otherwise told the issue was still under investigation.
Then, on Monday 4 March, with no communication to Rapid7, JetBrains published a blog announcing the release of the new version of TeamCity, which patched the vulnerabilities. Rapid7 said it expressed its concern that the patch was released without notification or coordination, and with no published advisories.
For TeamCity on-premise users, the botched disclosure means the ability to assess your risk factors has been taken away, and the only solution is to patch immediately
Under its own vulnerability disclosure policy, if Rapid7 becomes aware a silent patch was issued, it will “aim to publish” details of the vulnerability within 24 hours, which it has now done.
JetBrains has since published a blog on the issue, and an advisory, and stated that the CVEs were included in the release notes for the new version of TeamCity, but it has not directly responded to Rapid7’s concerns about the uncoordinated disclosure.
In JetBrains’ version of the story, it does not dispute it proposed what Rapid7 would term a silent patch, but maintained that this disclosure method followed its coordinated disclosure policy.
It said it wanted to follow this path to enable customers to make an informed decision about the risk level, to give them time to upgrade, and to stop less skilled attackers from exploiting the flaws in the interim.
JetBrains said it made a decision not to make a coordinated disclosure after learning that this would mean Rapid7 would publish full technical details of the vulnerabilities at the same time the patches dropped.
“To reiterate, we never had any intention to release a fix silently without making the full details public. As a CVE Numbering Authority (CNA), we assigned CVE IDs for both issues a day after receiving the report,” wrote Daniel Gallo, TeamCity solutions engineer at JetBrains.
“We suggested disclosing the details of the vulnerabilities in the same way we have followed in the past, with a time delay between releasing a fix and making a full disclosure, which allows our customers to upgrade their TeamCity instances.
“This suggestion was rejected by the Rapid7 team who published full details of the vulnerabilities and how to exploit them a few hours after we had released a fix to TeamCity customers.”
Silent patching: a bad idea
The approach to coordinated disclosure taken by Rapid7 is widely accepted and entirely normal within the cyber security world,
While JetBrains has not explicitly stated why it rejects this approach, writing in 2022, Rapid7’s principal security research manager, Tod Beardsley, offered a possible explanation when he said that taken at face value, silent patching might seem appropriate to some because it seems to limit the pool of people who understand the issue and how to take advantage of it.
“Silent patching is tantamount to full disclosure to a very small audience who mostly want to hurt you and your users” Tod Beardsley, Rapid7
Outlining why this is not the case, Beardsley wrote: “When a software company release a patch … at some point it’s got to modify the code on the running software, which means it’s all available to anyone who has a running instance of the patched software and knows how to use a debugger and a disassembler. And who uses debuggers to inspect the effects of patches? Exploit developers, almost exclusively.”
With this in mind, said Beardsley, silent patching in fact limits knowledge of the patched vulnerability to skilled exploit developers, that is to say threat actors, so while it is true that silent patching cuts out casual, low-skilled hackers and script kiddies, it also excludes the good guys, legitimate pen testers, the developers of defensive technologies, incident responders, and the entire cyber community who might benefit from being able to understand the issue better and communicate it effectively.
“Most significantly, you’re excluding the most important audience for your patch: the regular IT administrators and managers who need to sort out the incoming flow of patches based on some risk and severity criteria and make the call for downtime and update scheduling based on that criteria. Not all vulnerabilities are equal, and while protectors want to get around to all of them, they need to figure out which ones to apply today and which ones can wait for the next maintenance cycle,” he wrote.
Summing up Rapid7’s argument, Beardsley said silent patching communicates vulnerability details “exclusively” to the skilled cyber criminal attackers and nation-state actors who are already targeting the vulnerable product.
“All this is to say, silent patching is tantamount to full disclosure to a very small audience who mostly want to hurt you and your users,” he concluded.
In the case of the new TeamCity vulnerabilities, the importance of coordinated disclosure takes on additional importance, given that previous issues in the service have been heavily exploited by none other than APT29, aka Cozy Bear, the cyber unit of the Russian foreign intelligence service (SVR).
For TeamCity on-premise users caught in the crossfire – cloud versions are fully patched – the guidance is simple: the botched disclosure means the ability to assess your risk factors has been taken away, and the only solution is to patch immediately.