🌱 New check: Do maintainers respond to public bug and security issues?

[AdamKorcz]

What kind of change does this PR introduce?

New check

• PR title follows the guidelines defined in our pull request documentation

This PR adds a new check for maintainer inactivity on public issues with "bug" and "security" related labels. In essence, the core of this new check is to assess how many of a project's issues labelled "bug" or "security" receive no maintainer response within the first 180 days after the label has been added.

This is a form of inactivity check of a project and provides a signal about a project's practices with public "bug" and "security" issues. The check aims to make it easy for projects to perform well in this check and as it is now can be useful in assessing whether a project is very poor in responding to public issues. As such, the goal is to identify obvious bad behaviours instead of differentiating between fine-grained limits. Said otherwise, the check sets a high bar for scoring low. If a project scores low, it can be a signal of poor maintenance and with a probe and a check, Scorecard users can both take programmatic action or manually assess a project. In my opinion it is a useful signal in and of itself as well as an indicator of potential underlying issues.

At a more technical level, the check supports GitHub and Gitlab-hosted projects. The check uses the existing issuesHandlers and fetches the issues of a repository, sorts the ones that are relevant for the check - the ones that have had a "bug" and "security" label at some point - and then checks the history of each issue. Finally, the check will do a relative scoring between all the issues that were labelled with a "bug" or "security" label at any point and the ones that have not had any maintainer activity within 180 days of the issue being labelled "bug" or "security". If 0-20% of issues have not had any maintainer activity whatsoever, the project scores 10. If 21-40% of such issues have had a maintainer response, the project scores 5. If 41% or more of such issues have not had any maintainer activity, the project scores 0. Under the hood, the check also fetches information about which users have elevated privileges in the repository.

Some example scenarios are:

1. A project has 10 issues with "bug" and "security" labels.
2. A maintainer has responded with comments to 4 of these within 50 days.
3. A maintainer has closed 4 of these within 100 days.
4. The remaining 2 issues have had the "bug" label for 5 years.

The project will score 10 because the maintainers have responded to 80% of the issues.

Another scenario:

1. A project has 10 issues with "bug" and "security" labels.
2. A maintainer has responded with comments on 4 of these within 100 days.
3. A maintainer has removed the "bug" from 2 of these within 100 days.
4. A maintainer has added a "triage" label to 1 of these within 100 days.
5. The remaining 3 issues have had the "security" label for 5 years.

The project will score 5 because maintainers have responded to 70% of these within 180 days.

In its current state, the check does account for some edge cases such as:

1. If the issue gets closed and reopened - the check will still check for any maintainer response and will only count the time it was open.
2. The check shouldn't consider issues that have not been open for more than 180 days. In other words, it should count the time issues have had a "bug" or "security" label in an open state.
3. Different activity types. The check should consider every possible maintainer activity.

That being said, there will likely be further edge cases that can be explored through usage.

In terms of the boundaries (20% and 40%), the scores (0/5/10), the output in the terminal when running this as well as the impact scoring of the check, I am more than open to modifying this.

Special notes for your reviewer

I would recommend doing this before reviewing the code:

1. Read the docs/checks/internal/checks.yaml.
2. Test it out from the command-line first: go run main.go --repo=https://github.com/owner/repo--checks=Maintainer-Response-BugSecurity

Does this PR introduce a user-facing change?

yes

Add new Check: Maintainer-Response-BugSecurity

[codecov]

Codecov Report

❌ Patch coverage is 66.23877% with 263 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.46%. Comparing base (353ed60) to head (7502756).
⚠️ Report is 285 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4867      +/-   ##
==========================================
+ Coverage   66.80%   69.46%   +2.66%     
==========================================
  Files         230      256      +26     
  Lines       16602    16400     -202     
==========================================
+ Hits        11091    11393     +302     
+ Misses       4808     4116     -692     
- Partials      703      891     +188     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spencerschrock]

This PR adds a new check for maintainer inactivity on public issues with "bug" and "security" related labels

This seems like a small tweak based on the existing issueActivityByProjectMember probe, where the key difference is the presence of a label, and perhaps a larger time frame.

Is this significant enough to warrant its own check vs. evaluating tweaking the existing Maintained?