Somebody recently contacted me with a fundamental question: they were undertaking an accessibility audit, and didn’t know how to structure the process. They knew web accessibility well enough — but from the perspective of setting out to perform an audit, they weren’t sure where to go.
As a result, I’m putting together this article to talk a little about how to structure an accessibility review, in all the practical ways — how you address coming up with a quote or estimate, ways to structure your research and site inspection process, and dealing with long-term follow-up.
- Specify the scope of the audit
- Identify “show-stoppers”
- Estimate or quote the project
- Plan your audit process
- Schedule a follow-up
Figuring out scope
Although the ideal is to perform an accessibility review which identifies every problem on a site and specifies solutions, that’s not always practical. In fact, it’s sometimes utterly pointless — it depends on the ultimate goal of the review. The review process for a site which is considering a redesign is radically different than for a set of templates intended for a web site still in development.
Setting goals at the outset is key. Are we looking to identify key areas for improvement? What are the resources available for fixing problems? If an aspect of the site is rife with major problems, is it best to identify every problem within that area, or simply describe the general issues and recommend finding a new solution?
An initial review does not need to identify every problem. One route I’ve frequently used is to specify a multi-stage review process: an initial review, in which I note major issues and provide guidance on fundamental principles of web accessibility, and a follow-up in which I re-check the site and, if desired, provide further guidance and detail for continuing development.
Identifying “show-stoppers”
What’s a “show-stopper”? Essentially, that’s a function of the site which is so completely inaccessible that there is no value to identifying the problems — instead, you should just describe what would constitute a good solution and recommend replacement.
It’s not uncommon for this to arise at the very earliest stages of the review, when discussing scope. If a site is using a CMS (Content Management System) or a framework which is fundamentally rendering it inaccessible, you may want to begin by recommending redesign of the site. However, due to the common usage of external resources to provide video, interactive widgets, or social media (to provide a few examples), it may be that there are elements in use which won’t stand out right from the beginning.
Rather than performing a detailed exhumation of a fundamentally broken aspect of the site, it may be in the best interest of all parties to simply flag it for replacement and discuss that possibility with the client.
Why is this section part of the business structure? Because spending hours reviewing replaceable services is a poor use of your time and your client’s money. You should be looking for ways to improve your client’s site without costing them an arm and a leg!
Coming up with a fair estimate
It’s difficult to provide a good estimate on what a large-scale accessibility review is going to require, in terms of hours or dollars. The larger the project, the harder it is to quote. Keep in mind, however, that what you’re quoting is not generally going to be based on the number of documents on the site — rather, it will be based on the number of unique templates, forms, and navigation structures found on the site.
It is entirely possible that the site you’re reviewing will have 12,000 pages, most with images containing improper alt attributes. However, as an accessibility consultant your time is better spent identifying one or two representative examples and explaining how to properly use the alt attribute than it is painstakingly identifying every single inappropriate attribute throughout the site.
For this reason, you should be looking at the site as a process, not as a collection of pages. The ideal process of the site can be described, greatly simplified, in four steps:
- The visitor arrives at the web site, which can happen at any place in the site.
- The visitor will attempt to move to another point on the site, which may be another part of the same document
- The visitor will begin a goal, which may be a purchase, a form submission, or the acquisition of information
- The visitor will successfully complete that goal, and be notified of the results.
This general outline describes any visitor to a web site — as a consultant, your job is to identify each barrier they may encounter while completing the process. You don’t need to look at every single page of the site in order to see the shape of the problems — if you can’t navigate using a keyboard on one page, you probably won’t be able to on any other page, either!
With this knowledge, it becomes readily apparent that a web site which has 12,000 pages but uses only one navigation structure, one template, and has only a single form will be much more quickly reviewed than a site with 120 pages which uses 5 templates, provides ecommerce, and has different navigation structures depending on what area of the site is being used.
A 100% complete audit must allow for the possibility that each page may exhibit different problems. After all, if you haven’t looked at a page, you have no way of knowing how different it may be from what you’ve already seen. However, in all probability, pages of a site will be at a fairly consistent level of accessibility. The pragmatic approach — mindful to your client’s budget needs — is going to be a selective audit, not a complete audit.
Plan your Accessibility Audit Process
Now, once you’ve done this a few times, you’ll probably have the basic approach down cold. Every web site is different, however, so that isn’t going to completely free you from doing some planning. You still need to decide what your approach will be. Two example starting approaches could be process-based testing or issue-based testing.
If you’re going with a process-based testing procedure, you’ll start by selecting a process – any process. The path to make a purchase; getting to the Privacy link in the footer; sending a contact message. Follow that process all the way through in painstaking detail, isolating the accessibility issues encountered along the way.
With issues-based testing, you’ll instead pick an issue – such as keyboard accessibility. Work your way through the entire site, noting keyboard-relevant issues as you go. Then move on to the next issue and start your hunt over.
It may seem like I’m describing a very pedantic way to thinking about conducting an audit — you know that you will ultimately do both of these in their entirety. However, the reason for having a plan is simple: you need to avoid the trap of the scatter-plot approach. If you don’t have a system, you’re far more likely to end up missing problems – either because you failed to consider keyboard accessibility over here; or because you forgot to check the contact form when you were reviewing contrast. Having a plan and a process will help you avoid these gaffs.
Schedule a Follow-up
It would be nice to believe that when you’ve written and sent your painstakingly thorough accessibility audit to the client you are done with the project. Sadly, we don’t always live in that pleasant fantasy world. (Although, to be honest, in my fantasy world an accessibility audit would consist of nothing more than a smiley sticker and the phrase “Good Job!”)
Unfortunately, no matter how detailed you made your report — the report doesn’t fix the web site. The client’s developers need to do that. Or, as is sometimes the case, the client’s developers will fail to absorb the principles of the document…and while fixing the problems you described, will create new accessibility issues. Or, they’ll implement a fix which shunts the problem somewhere else, rather than resolving it entirely.
This isn’t a guarantee – there are developers working out there who don’t know accessibility, but immediately catch on to the concept once the issues are presented to them. But there are also developers who…don’t.
Planning at least one follow-up review is important. You’ll also need to make yourself available to answer questions from the development team while they work through your suggestions.
The follow-up review should be a quicker task than the initial review. You can spot-check to see how the developers worked with your suggestions. If you like what you see, you can probably be fairly satisfied. If you see problems, you may need to keep going. If you see a lot of problems, you may need to give the client a call before continuing.
In Summary
Performing an accessibility audit requires many skills: an eye for detail, a strong sense for when an accessibility issue requires fixing, and when it requires replacement, and the ability to describe accessibility concepts in language developers can respond to. Join those skills with sound business planning and a personal investment in your clients’ success, and you’re ready to go!
Joe Dolson
Thanks, Sam – as a note, you might want to take a look at the copy on your Audit Phases page. The first paragraphs seems to be missing some copy.
Sam
At SSB we use a unified audit methodology which allows us to deliver repeatable audits while leveraging our time effectively: https://www.ssbbartgroup.com/reference/index.php/Unified_Audit_Methodology_Home
Pooja Nahata
Always enlightening to read such blogs! I so agree with your point of reusing content/sample code for reports – that brings about standardization and saves time as we don’t need to re invent the wheel. One point that many people struggle to understand is the type of testings to be conducted – often we see emphasis from clients to do automated over manual as they dont have budgets or the time. Thus its vital when we plan our audit process to ensure that manual testing with assistive technology is considered along with automated.
Joe Dolson
That’s a good point, Richard — it’s definitely frustrating to give instructions, and have the client tell you that they can’t find what you’re talking about on the site…because it’s not there anymore.
Richard - accessibleweb
Good advice.
On thing I would add, if a site is regularly changing content, or is still being developed then it is important to get a snapshot of each page at the time of testing, both visual and code. And no matter how many times you tell developers to freeze changes during testing, do they take much notice?
Joe Dolson
Thank you, Catherine! I’m glad you found it useful. It’s a good point that people looking to hire somebody to do an audit also will find this information valuable!
Catherine Roy
Thank you for writing up this very useful article. Not only is this useful for those who need to propose and perform reviews, but it is also very helpful for those who need to shop around for an audit and who may have limited knowledge in what they should be looking for.
Definitely bookmarked as a valuable resource 🙂
Joe Dolson
Thanks, Jared – that’s definitely something that I find very difficult. What I primarily work from is a small library of basic principles statements, which I can use as a starting point for given problems and a small library of simplified code examples for common problems. Then I can use that material to branch from and illustrate the principle of the issue effectively. Essentially, I begin any section with either a statement of principle or an illustrative sample (or both). It allows me to avoid having to re-write common themes over and over again.
It’s an indirect route to re-using information, since I still need to deal with the specifics of their scenario – but at least I don’t need to re-frame the problem every time.
That’s the best I’ve come up with!
Jared Smith
Excellent article. For what it’s worth, this is very similar to WebAIM’s approach to structuring accessibility reviews. Like you, our report structure is typically either task-based, page-based, or issue-based. They are rarely guideline based (though we provide brief compliance overviews). I agree that it’s usually pointless to do in-depth reviews on inaccessible content and document the same issues over and over again (though I see many reports that do this).
One thing I struggle with is templating reports and documentation in a way that allows reuse of content. Despite several efforts, we find each client and each report is so unique that it’s hard to re-use content from one report to another. It’s relatively easy to document how to build an accessible widget, but detailing how to fix their own specific widget is more valuable and educational to them – and this requires customized (and thus more time-intensive) documentation.
I’d be happy to know if you’ve found a way to address this.