Software Evaluation

Evaluating software for accessibility is similar to website evaluation. The principles of functional accessibility apply, as do the steps of progressive evaluation - though with different evaluation tools at your disposal. This section will discuss techniques for evaluating software on various platforms. For now, Mac OS X and Windows XP/VISTA/7 will be covered.

Beyond Conformance

A common mistake in accessibility evaluation is to consider conformance to accessibility laws and guidelines as the end goal. In reality, it is entirely possible to create an application that is 100% conformant but is unusable by disabled audiences. In the end, what is most important is functional accessibility, or how usable an application is by diverse groups of users. It is far more useful to consider accessibility guidelines and standards in that context.

To help keep the goal of functional accessibility in mind, consider the following from a user's perspective as you perform your evaluations:

  1. Purpose? - Is this the application I intended to run? Can I easily determine what the application is for?
  2. Structure? - What is the interface layout? Can I find where everything is in the application window and how it all fits together?
  3. Interaction? - Can I do what I ran the application to do (e.g. successfully type a document or configure a system setting)?
  4. Navigation? - Was I made aware that the application launched? Can I get to everything in the interface? Can I get back to each application screen if I need to?

Keeping these four principle areas in mind will help you glean the purpose of a given accessibility requirement and will ensure that applications that pass your evaluations are usable by diverse user groups.

Progressive Evaluation

Accessibility evaluation is somewhat subjective, with a multitude of diverse requirements that can sometimes seem at odds. To simplify the evaluation process, consider a progressive approach, performing the easiest checks first and moving on to more in-depth checks from there.  Progressive evaluation for software begins with checking keyboard navigation support, checking the interface in high-contrast mode, followed by using a screen reader to check interface labels.  Unlike progressive evaluation for the web, it is often not possible or helpful to look at the source code of an application. Following these checks, create a prioritized list of issues for remediation.  Evaluating in this order can eliminate the need for time-consuming evaluations when the simplest, most important accessibility support features are not present.

Step 1: Keyboard Navigation

Blind users and some low-mobility users cannot use a mouse to interact with software applications. Because of this, an application with no keyboard support can not be considered accessible, no matter what other accessibility enhancements are present. Ask yourself the following questions as you perform this step of your evaluation:

  1. How easily can you interact with the page using just one finger? Most blind and low-mobility users navigate through applications using the tab and arrow keys, so the easiest check you can perform on an application interface is to navigate through it using the tab key.
  2. Can you tell where you are in the application window? This is referred to as a visual focus indicator (or visual focus for short). Visual focus is essential for low-mobility users who cannot use a mouse, and it is helpful for all users. The visual focus indicators should not rely solely on color to indicate that an element has focus; otherwise, color-blind users may not be able to perceive the indicator.
  3. Can you operate everything in the application? It is common for application menus and other controls to only be reachable via the mouse. To be accessible, all controls must operate via the keyboard in a similar fashion to the way a user would interact with the mouse.
  4. Does the navigation make sense? As you tab through the interface, does the focus move where you expect it to go, or does it surprise you? Keyboard navigation in an application must be logical. If focus moves in a surprising manner, this can cause confusion for blind, low-vision, or cognitively impaired users.

If the answer to any of the above questions is 'no' or answerable with difficulty, more work is needed to make the page accessible. If no keyboard support is present, you could arguably conclude your evaluation at this step, as the page is unusable by users who must interact with a keyboard. You might consider moving on to the next steps if you are compiling a report for a remediation plan.

Step 2: High-Contrast Mode

An additional step that is very important for software applications is checking the interface in high-contrast mode. Many software applications use images for the interface controls. On Widnows, high-contrast modes disable some images, and this can render application interfaces unusable for low-vision users.

Step 3: Use A Screen Reader

You can determine a lot about an application from the first two steps, and once you have determined that some of the basic accessibility support is present it is time to see if the application supports the accessibility layer of the operating system. To do this, you will need a screen reader.

On Mac OS and iOS (4+), you should use VoiceOver. VoiceOver is built into the operating system. For windows XP and newer, we suggest using NVDA. JAWS, from Freedom Scientific, is an excellent screen reader, but it tries to guess the label for controls, which can cause you to miss accessibility issues. JAWS requires a purchased license, but it will run in a demo mode for 40 minutes. NDVA is free; though, setting up a nice voice for it requires some additional configuration.  Android devices have a few options, most of which are third-party apps that must be installed. TalkBack, from Google, comes pre-installed on later devices, and it can be configured for touch operation of the device. Details on how to configure accessibility options for your Android device will vary by manufacturer and distribution.

When evaluating software with a screen reader you should look for the following things:

  1. Is the application announced when it starts? If appropriate, does the application take focus?
  2. Are the application's window titles announced properly?
  3. Do the controls and images in the application have labels?
  4. Do keyboard shortcuts defined for the application interfere with the operation of the screen reader? This may require downloading a key reference for popular screen readers (Here are some helpful links: VoiceOver Downloads, WebAIM list of NDVA shortcuts, JAWS Keystrokes).

Note: Some screen reader software can emulate mouse clicks and mouse movement, but not all users know how to enter this mode. Contrals that require mouse emulation should not be considered accessible in most cases.

Step 4: Setting Priorities for Remediation

Once you have identified the accessibility issues in an application, it is time to compile a report for remediation. Accessibility reports can contain a lot of detail concerning many different issues, and this can be overwhelming -- especially for vendors and developers who are relatively knew to providing support for accessibilty in their products. It can be very helpful to organize and prioritize the issues in an accessibility report by their level of impact.

Generally speaking, the range of impact for accessibility issues can be considered to go from issues that will prevent use of the application (most important) to issues that, while annoying, do not prevent users from successfully using the application (least imporant). We use three priority levels in the usability and accessibility reports that we compile:

  1. Priority 1 - Issues that prevent some individuals from using an application. These issues are included first in the report.
  2. Priority 2 - Issues that make using an application difficult and/or disorienting.
  3. Priority 3 - Issues that are annoying or enhancements that could be made to the application. These are usually included last.

Whatever prioritization scheme you use, it is best to group and label issues in the report by priority. It can also be a good idea to group application-wide issues (i.e. problems fouond on every application window) to reduce report size. Recommend that the highest priority items be corrected first, and if you are able, offer strategies for remediation of particular issues. In our experience, it is always better to adopt a posture of working with a vendor or developer -- especially those new to accessibility -- than it is to simply tell them they are doing something wrong.