Website Evaluation

There are many approaches to website evaluation, and they can seem daunting, especially if you are not an accessibility professional. Thankfully, there are some excellent tools that can assist you in your evaluation process, but such tools are only useful as a step in the evaluation process. This page will cover some key principles to keep in mind as you perform your accessibility evaluations.

Beyond Conformance

A common mistake in accessibility evaluation is to consider conformance to accessibility laws and guidelines as the end goal. In reality, it is entirely possible to create a web page that is 100% conformant but is unusable by disabled audiences. In the end, what is most important is functional accessibility, or how usable a page is by diverse groups of users. It is far more useful to consider accessibility guidelines and standards in that context.

To help keep the goal of functional accessibility in mind, consider the following from a user's perspective as you perform your evaluations:

  1. Purpose? - Is this the page I'm looking for? Can I easily determine what the page is for?
  2. Structure? - What is the page layout? Can I find where everything is on the page and how it all fits together?
  3. Interaction? - Can I do what I came to do (e.g. successfully fill out a form or watch a video)?
  4. Navigation? - Can I get to everything on the page? Where can I go from here? Can I get back here if I need to?

Keeping these four principle areas in mind will help you glean the purpose of a given accessibility requirement and will ensure that pages that pass your evaluations are usable by diverse user groups.

Progressive Evaluation

Accessibility evaluation is somewhat subjective, with a multitude of diverse requirements that can sometimes seem at odds. To simplify the evaluation process, consider a progressive approach, performing the easiest checks first and moving on to more in-depth checks from there.  Progressive evaluation begins with checking keyboard navigation support, followed by using evaluation tools, and then proceeding to a check of the source code.  Following these checks, create a prioritized list of issues for remediation.  Evaluating in this order can eliminate the need for time-consuming evaluations when the simplest, most important accessibility support features are not present.

Step 1: Keyboard Navigation

Blind users and some low-mobility users cannot use a mouse to interact with web pages. Because of this, a web site with no keyboard support can not be considered accessible, no matter what other accessibility enhancements are present. Ask yourself the following questions as you perform this step of your evaluation:

  1. How easily can you interact with the page using just one finger? Most blind and low-mobility users navigate through websites using the tab and arrow keys, so the easiest check you can perform on a webpage is to navigate through it using the tab key.
  2. Can you tell where you are on the page? This is referred to as a visual focus indicator (or visual focus for short). Visual focus is essential for low-mobility users who cannot use a mouse, and it is helpful for all users. The visual focus indicators should not rely solely on color to indicate that an element has focus; otherwise, color-blind users may not be able to perceive the indicator.
  3. Can you operate everything on the page? It is common for drop-down menus and custom select controls to only be operable via the mouse or they have all of their contents visible and positioned off-screen but do not update the screen to make them visible on focus (an outdated technique for blind users). To be accessible, all controls must operate via the keyboard in a similar fashion to the way a user would interact with the mouse.
  4. Does the navigation make sense? As you tab through the page, does the focus move where you expect it to go, or does it surprise you? Keyboard navigation in a page must be logical and based on the linear order (referred to as the reading order) of the html source.  CSS positioning of page elements can cause the visual presentation of a page and it's logical flow to become out of sync. That can cause disorientation and confusion for blind, low-sighted, and low-mobility users, and could ultimately make a page unusable.

If the answer to any of the above questions is 'no' or answerable with difficulty, more work is needed to make the page accessible. If no keyboard support is present, you could arguably conclude your evaluation at this step, as the page is unusable by users who must interact with a keyboard. You might consider moving on to the next steps if you are compiling a report for a remediation plan.

Step 2: Use Evaluation Tools

You can determine a lot about a page from the first step, and once you have determined that some of the basic accessibility support is present it is time to dive into the details of a page. this is where using evaluation tools can help you.  These tools help structure an evaluation and can provide insight into issues that are not easily recognizable.

No single tool does everything, and it is a good idea to become familiar with a few of the available tools. There are two primary types of evaluation tool:

  1. High-level, standards based tools, such as FAE and WAVE.
  2. Low-level tools to look at the source code, such as Firebug and the W3C Validator.

Another thing to keep in mind about using evaluation tools is that most pages will not evaluate 100% cleanly for accessibility. An example of this is the use of ARIA in pages with an XHTML doctype.  Using ARIA in that context does not cause problems, but the W3C validator will sometimes flag the additional attributes that ARIA specifies. It is important, then to determine what errors can be safely ignored and what errors should be corrected to ensure robustness and accessibility.

The order in which you run evaluation tools depends upon preference. Some evaluatators run the W3C validator first, to determine if the HTML source is standards-based. Following that, they will run FAE or another similar tool. Finally, some use tools to turn off CSS to see if the reading order of the page is still logical and turn off background images to see get an idea of what the page would look like in a high-contrast mode in MS Windows.

More discussion on available evaluation tools can be found on the Evaluation Tools page.

Step 3: Review the Page Source

Assuming that you found issues during the previous step, it is time to take a look at the source code.  This step requires a working knowledge of HTML, but there are some basic checks that you can do, even if your knowledge is limited. Using a tool like Firebug and the noted issues as a guide, you can examine issues pertaining to header structure, ALT text for images, labels for form control, and visual focus indicators. There are many more details to look for, but these four issues account for a majority of the issues commonly seen with web accessibility (lack of keyboard support is the most common issue).

If ARIA is used, it is important to check that any ARIA attributes and roles have valid properties and states. Check that aria widget roles are not used on standard HTML elements. For example, role="checkbox" should not be used on an input control of type checkbox.

Step 4: Setting Priorities for Remediation

Once you have identified the accessibility issues in a page, it is time to compile a report for remediation. Accessibility reports can contain a lot of detail concerning many different issues, and this can be overwhelming -- especially for vendors who are relatively knew to providing support for accessibility in their products. It can be very helpful to organize and prioritize the issues in an accessibility report by their level of impact.

Generally speaking, the range of impact for accessibility issues can be considered to go from issues that will prevent use of the system (most important) to issues that, while annoying, do not prevent users from successfully using the application (least important). We use three priority levels in the usability and accessibility reports that we compile:

  1. Priority 1 - Issues that prevent some individuals from using an application. These issues are included first in the report and must be fixed prior to releasing the site.
  2. Priority 2 - Issues that make using an application difficult and/or disorienting.
  3. Priority 3 - Issues that are annoying or enhancements that could be made to the application. These are usually included last.

Whatever prioritization scheme you use, it is best to group and label issues in the report by priority. It can also be a good idea to group site-wide issues to reduce report size. Recommend that the highest priority items be corrected first, and if you are able, offer strategies for remediation of particular issues. In our experience, it is always better to adopt a posture of working with a vendor or developer -- especially those new to accessibility -- than it is to simply tell them they are doing something wrong.

Skills For Effective Evaluation

Accessibility evaluation is somewhat subjective, with a multitude of diverse requirements that can sometimes seem at odds. Evaluation is iterative by nature, with no single, comprehensive step. In order to perform effective evaluations, it is best if you have a working knowledge in the following areas:

  1. Web Content Accessibility Guidelines (WCAG) - In order to know when something is non-compliant with accessibility regulations, you have to have at least a passing knowledge of what those regulations are, how to pass them and how to fail them. With somewhere around 400 pages of supporting documentation, understanding WCAG 2.0 is not a simple matter but neither is it impossible. Start with the four priniciples and then work through the Success Criteria. This will give you a context for understanding the finer points as you work to master WCAG 2.0. 

  2. HTML - Be aware of current, standards-based coding practices and have working knowledge of how to use structural markup, headings, etc., and semantic markup, such as form labels and lists.

  3. CSS - Have a working understanding of CSS. Know how to use pseudo-selectors, such as :hover and :focus. Be familiar with how CSS positioning and display attributes interact with html source order.

  4. JavaScript - Basic understanding of the browser event model (i.e. click, focus, keydown, etc.) and how scripts can manipulate the css and html of a page is necessary for evaluating more dynamic web pages.

  5. Evaluation Tools - Have a working knowledge of at least one evaluation tool, such as FAE or WAVE. Know how to interpret their results meaningfully.

  6. W3C Validator - Standards-based coding practices are essential for ensuring the robustness and compatibility of web code. Know how to use the W3C validators and interpret their results.

  7. Firebug - Firebug for FireFox is very examining the source code of a web page. There are other, similar tools, but Firebug seems to be best at showing how scripts interact with the page source.