UA/TestingPlan

From CodeTalks

1 Develop test suite
2 Add new testcases to codetalks.org
3 Test manually against test suite
4 Automated testing tool development
5 ARIA testing for authors

Develop test suite

Develop Test suite, being developed by grantees and volunteers in free-aria community.

Working group includes Jon Gunderson, Steve Faulkner, Stefan Schnabel, Aaron Leventhal and David Bolter.

Add new testcases to codetalks.org

Get SVN server on codetalks so we can collaborate testcases (David Bolter).
Move UIUC test cases to codetalks SVN (Jon Gunderson)
Move mozilla.org test cases to codetalks SVN (Aaron Leventhal)

Test manually against test suite

To save time, it is not necessary to test browser-screen reader combinations that already are known not to work (see below).

Details for Browser Testing

Firefox 3 and Firefox 3.1 nightly builds (file bugs at bugzilla.mozilla.org)
IE 8 beta 2 -- do not test with NVDA right now
~~Opera 9.52~~ ... Please hold off on testing this for now, as it appears that there is an MSAA hierarchy but no focus events are fired. (file bugs at bugs.opera.com/wizard)
~~Webkit/Safari nightly builds~~ ... Please hold off on testing this for now, as it appears that there is an MSAA hierarchy but no focus events are fired. (file bugs at bugs.webkit.org)

Details for Screen Reader Testing

Please test screen readers with WAI-ARIA support with *all* relevant modes (virtual-buffer/browse mode, forms mode, etc.)

JAWS 9 (widgets testcases only) and JAWS 10 beta 2 (use all testcases except for aria-relevant and aria-channel)
Window-Eyes 7 (widgets testcases only)
NVDA 0.6p2 (all testcasess) -- do not test with IE right now, the support is not really there yet, according to NVDA developers
Orca 2.22.1 (all testcases)
- note: Orca is part of the GNOME platform and Orca's releases are coupled with the releases of the GNOME platform. The GNOME platform releases on a 6 month cycle, with stable releases typically around the March and October timeframes. During the 6 month cycle, there are a number of "odd" releases, such as GNOME 2.21.1, GNOME 2.21.2, etc. The stable releases that follow are "even" releases, such as GNOME 2.22.0, GNOME 2.22.1, etc.
ZoomText 9.1 Magnifier/Reader -- test with speech synthesis turned on.

Who's Testing What?

JAWS 9 & 10 with Firefox 3: Frank DiPalermno and Marco Zehe
JAWS 9 & 10 with IE beta 2: Aaron Leventhal, Stefan Schnabel and David Bolter
NVDA 0.6p2 with Firefox 3: Marco Zehe and Thomas Logan
Orca with Firefox 3: Marco Zehe
Window-Eyes 7 with IE 8 beta 2 and Firefox 3: Jon Gunderson
ZoomText 9.1 with Firefox 3 and IE 8 beta 2: Steve Faulkner
Dragon 10 with Firefox 3 and IE 8 beta 2: Thomas Logan

Crucial! Log all bugs!

Don't add a bug to the table without logging it! If it's added, then everyone will assume the appropriate development team already has the report.

Before logging a bug review test cases first to determine if the bug found is a known issue. If an issue is documented on this page it means that a bug has been logged. Links to filed bugs in databases are provided whenever possible.

Details about how to log a bug can be found here

Automated testing tool development

Develop Speclenium tool for testing implemenations(Eitan Isaacson)
- Cross-browser comparisons for hierarchy and events
- Write specific platform API tests against Implementor's Guide
- Use test suite to test browsers with Speclenium