Don’t judge progress by the number of open issues

January 9, 2022

Topics: Accessibility, WordPress.

Recently, I’ve seen more than one person in the accessibility community sharing Tweets about the number of accessibility issues in the Gutenberg repository. The increase in issues over the last few years is presented to show that WordPress is not considering accessibility. Specifically, I’m referring to comments by Adrian Roselli & Dennis Lembree:

https://twitter.com/aardrian/status/1474088282589962241

There are plenty of legitimate reasons to criticize the accessibility of Gutenberg. Progress on accessibility has been inconsistent at best. This is heavily influenced by the lack of sponsored contributors to WordPress who are specialists focusing on accessibility. (Current count, to my knowledge, is zero.)

But the idea that a change in absolute numbers represents any change in accessibility is a shockingly disingenuous claim from people who are otherwise reliable sources of good information. Even if the intent was to represent this with humor, it’s grossly invalid way to measure progress.

A simple example

  • A project has 50 accessibility issues on January 1st.
  • On January 8th, somebody opens 50 additional issues, then fixes and closes 25 of them by January 15th.
  • The project now has 75 open accessibility issues.

Did the accessibility of that project improve during the period of measurement? If we assume no other changes, yes, it absolutely did. This is true even though the number of open isues has increased. If we assume that a grossly inaccessible feature was committed on January 7th, then the answer is more ambiguous.

More importantly, the absolute number of issues does not provide any useful information about change in the project. We can’t answer that question without better context.

Within the Gutenberg project

The observation is that the number of issues changed from 81 to 175 between August 6th, 2018 and December 23, 2021. During the intervening three years, the Gutenberg project was expanded to include a replacement for the WordPress widgets and the entire Full Site Editing suite. The overall complexity of the project increased enormously during that time – and the number of accessibility issues reflects that.

How many accessibility issues were closed in the same time period? 485 issues.

That’s sounding more like progress is happening, isn’t it? However, that’s still not really sufficient context: “closed” does not mean “fixed”. That would require a considerably more extensive study.

Real Accessibility Issues in Gutenberg

There are some real and serious accessibility problems in Gutenberg. The widget interface is, frankly, largely unusable from the keyboard. I have not yet been able to test the full site editor because my one attempt to do so gave me a panic attack.

I can’t honestly say why. However, when I tried it, I was instantly so overwhelmed that I raced to find a way to exit the editing environment. My heart was racing, and I felt unable to actually function within that framework.

These real accessibility problems need concerted attention and time from accessibility professionals; implying a connection between the number of accessibility issues and the accessibility problems merely discourages people from getting involved. It doesn’t help solve the problem.

Have something to contribute?




« Read my Comment Policy

2 Comments on “Don’t judge progress by the number of open issues”

  1. Thanks for commenting, Adrian. There is accessibility progress, it just doesn’t follow a good process, and has minimal expert review. I have a post brewing on the devaluation of subject matter expertise in the WordPress community…

  2. As you say, I know better. I know error density, changes to code over time, plus overall complexity increases, all factor into progress that raw numbers do not show. Just as you note closing an issue does not mean fixing it.

    My attempt at a snarky joke was that Gutenberg was a more accessible product when it did not exist, as supported by my spurious math suggesting time flows backward.

    But it was also a shot at Automattic’s failure to support (in dollars and developers) the accessibility of the tool it forced through the process. Where the accessibility progress it makes is on the backs of OSS developers.