GitHub Dependabot is a 'noise machine', and should be turned off, says Go library maintainer

A Go library maintainer has recommended turning off GitHub’s Dependabot, a tool for keeping library dependencies in a repository up to date, because of false positives which "reduce security by causing alert fatigue."

Filippo Valsorda was formerly in charge of the Go security team at Google, and now maintains the cryptography packages in the Go standard library. Last week he published a security fix for one of the libraries he maintains, fillipio/edwards25519, which is used for EdDSA (Edwards-curve digital signature algorithm) cryptography. As a result, Dependabot "opened thousands of PRs [pull requests] against unaffected repositories," said Valsorda The automated process also generated what Valsorda called "a nonsensical made up CVSS [common vulnerability scoring system] v4 score" and warned developers of a 73 percent compatibility score, implying a 27 percent chance of breaking code, even though the fix was "one line in the method no one uses." 

The most common reason for depending on this library is that a Go MySQL database driver uses it, but since this does not call the amended function (MultiScalarMult) it is not affected. Valsorda called the GitHub tool a "noise machine" and recommended turning it off.

Dependabot originated as a third-party tool and was acquired by GitHub in May 2019. It uses data from the GitHub Advisory Database to automate pull requests and security alerts in repositories that include dependencies on the listed vulnerabilities. 

Valsorda said that Dependabot was both too noisy and insufficient for its purpose. It is too noisy because it appears to check only that the dependency exists, not whether the impacted package is actually used. "Any decent vulnerability scanner will at the very least filter based on the package," he said. Further, he recommended using a static analysis tool like govulncheck (in the case of Go vulnerabilities) that will identify the reachability of the flaw.

Despite its noise, Dependabot is also insufficient, Valsorda added, because a real vulnerability "should be assessed for its impact: production might need to be updated, secrets rotated, users notified." If developers rely on Dependabot to manage dependency vulnerabilities, they are not doing enough.

If that is not enough, Valsorda also dislikes another feature of Dependabot, which is keeping dependencies up to date with the latest versions. He argues that dependencies should be updated according to the project’s development cycle, not whenever a new version of a package appears. Updating quickly also carries some risk, if malicious code has been added to a package. He recommends testing updated packages in a sandboxed CI (continuous integration) process to discover any problems without updating production code.

A Hacker News discussion on the subject, found widespread agreement with Valsorda’s points, though customers may not understand the nuances. "Customers will scan your code with these tools and they won't accept ‘we never call that function’ as an answer … this is where actual security starts to really diverge from the practices we've developed in the name of security,” said one comment.

There is also a view that the value of Dependabot varies according to what would get put in its place. The Go ecosystem is well set up for dependency checking, and Valsorda describes other tools and processes that developers can use. In other cases, when resources are limited and there is no obvious alternative, Dependabot is likely better than ignoring the problem it addresses.