The Automation Gap: What Accessibility Tests Still Miss

Automated Accessibility Testing: What Tools Still Miss

Accessibility testing means testing websites and applications to confirm that all users can use them without difficulty.

This covers users with audio, visual, physical, neurological, and cognitive conditions.

The aim is to provide an equivalent experience for users of all abilities across all software products.

Automated accessibility testing is able to quickly identify rule-based problems and simple ARIA issues, but it cannot completely assess those areas which rely on context and human judgment.

The purpose of this article is to explore what automated accessibility testing can detect, what it cannot detect, why a manual review is still necessary, and how teams can combine both methods.

What Can Automated Accessibility Testing Detect?

Automation works well for deterministic, rule-based WCAG issues.

Color contrast problems, missing alt text, form-label associations, and basic ARIA landmark errors are examples of issues that automated tools can detect consistently.

These issues share one trait- they can be checked against a defined condition.

A contrast ratio can be measured against a required threshold.

An image can be checked for the presence of an alt attribute, and a form input can be checked for an associated label.

Because each of these checks has a clear pass or fail condition, automation catches them at scale without variation between runs.

The following are some accessibility issues that automated testing can reliably detect:

• Color Contrast (WCAG 1.4.3): Automation measures the contrast ratio between text and its background and flags values below the Level AA threshold of 4.5:1 for normal text, giving teams an exact number to fix against rather than a subjective judgment call.
• Missing Alt Text (WCAG 1.1.1): Automation identifies images without an alt attribute and flags them for review, which catches the structural absence of a description even though it cannot judge the quality of that description.
• Missing Form Labels and Name, Role, Value (WCAG 4.1.2): Automation is used to detect inputs, buttons, and other controls which do not have accessible names or programmatic labels, since it is important that assistive technology relies on this information to announce what a control does.
• Missing Lang Attribute (WCAG 3.1.1): Automation checks whether a page declares its primary language and flags pages where the lang attribute is missing, which affects how a screen reader chooses pronunciation rules.
• Empty Links and Buttons: Automation identifies interactive elements without accessible names, since a link or button with no visible text and no label reads as blank to a screen reader user even if it is visually clear from surrounding context.
• Basic ARIA Landmarks: Automation can identify certain landmark structure issues and flag incorrect or missing landmark roles, which supports users who navigate a page by jumping between regions instead of reading it top to bottom.

What Automated Accessibility Testing Can’t Detect?

Automated accessibility testing cannot fully evaluate complex interactions, content meaning, or the overall experience of a person using assistive technology.

These areas require human judgment because they depend on context, intent, and how someone actually moves through a page, none of which reduces to a rule a scanner can apply.

An automated scanner reads the DOM and the accessibility tree, and it can confirm that certain attributes and structures exist.

It has no way to interpret whether those attributes serve their purpose for a real task, and that distinction is where most of the remaining WCAG success criteria live.

Here are the areas where automated accessibility testing cannot fully assess the experience:

• Meaningful Alt Text: Automation can confirm that an alt attribute exists, but it cannot reliably determine whether the description explains the image’s purpose or meaning in context, so a nonsensical or irrelevant description passes the same check as an accurate one.
• Complex and Custom UI Components: Custom dropdowns, modals, date pickers, and drag-and-drop interfaces often require manual testing to confirm that they work correctly with assistive technologies, since these components rely on scripted behavior that a static scan does not exercise.
• Keyboard Operability of Custom Controls (WCAG 2.1.1): Automated tools can detect some keyboard-related issues, such as missing tabindex handling, but they cannot fully confirm that a custom component can be operated entirely through the keyboard from start to finish.
• Logical Reading Order: Automation can inspect the order of elements in the DOM, but it cannot reliably determine whether a screen reader presents that content in a sequence that makes sense to a person trying to complete a task.
• Error Messages and Suggestions (WCAG 3.3): Automation can identify missing or poorly structured error information, but it cannot judge whether an error message clearly explains the problem or tells the user how to correct it.
• Content Clarity: Whether content reads as clear and understandable depends heavily on the audience and the situation, which places it outside what a rule-based check can measure and squarely in the territory of human evaluation.

Why Does Accessibility Testing Still Require Human Review?

Accessibility testing still requires human review because many accessibility issues depend on meaning, context, and how people interact with a product.

Automated tools can identify many rule-based problems, but they cannot fully judge whether an experience works well for people with disabilities or whether the fixes address the underlying issue.

The following are some areas where human expertise remains necessary:

• Structural Code Changes: Accessibility begins with the underlying HTML, CSS, and JavaScript. Automated tools can identify some problems, but developers need to review semantic structure, keyboard navigation, focus management, custom components, and ARIA usage to make lasting fixes.
• Documents and Media: PDFs, videos, and other non-HTML content cannot be made accessible through automation alone, since a PDF needs a logical tag structure, a clear reading order, and accurate descriptions for images and charts, while video needs captions, transcripts, and sometimes audio descriptions. Producing any of this requires a person to understand what the content is actually communicating.
• Compliance Documentation Such as VPATs: A Voluntary Product Accessibility Template documents how a product meets standards such as Section 508, WCAG, or EN 301 549. Completing this documentation requires trained reviewers to assess the product against the relevant criteria and accurately record the findings.
• Comprehensive Auditing: When it comes to auditing, although automated scans are able to detect a large number of technical problems, a full audit must have experts look at actual interactions, the structure of the pages, the way the keyboard is used, and the compatibility with assistive technologies. Human reviewers are also in a position to understand accessibility requirements in the context in which they apply and to spot problems that automated checks might fail to identify.

What Are the Limitations of Automation-Only Accessibility Testing?

Automation-only accessibility testing can identify many technical issues quickly, but it cannot determine whether a website or application is fully accessible.

Automated tools check specific rules and patterns.

They do not understand user context or the complete experience of people with disabilities.

Common limitations include:

• Limited WCAG Coverage: Automated tools cannot test every aspect of WCAG. Many success criteria require human judgment.
• Contextual Issues: Tools can detect whether alternative text exists but cannot reliably determine whether the text accurately describes the image’s purpose.
• Keyboard Usability: Automation can identify some keyboard issues but cannot fully judge whether a user can complete an entire task using only a keyboard.
• Screen Reader Experience: Automated checks cannot fully evaluate how content is announced or whether the overall experience makes sense when using assistive technology.
• False Positives and False Negatives: Automated results may report issues that are not actual violations or miss problems that require human interpretation.
• Usability and Task Completion: A product can meet technical accessibility requirements and still be difficult for people with disabilities to use.
• Dynamic Interactions: Complex interactions such as modals, focus changes, and live regions may require manual checks to determine whether they work correctly with assistive technologies.

How Can Teams Combine Automated and Manual Accessibility Testing?

Teams can integrate both automated and manual accessibility testing by assigning each method a definite role within the testing process.

Automated testing is useful for identifying repeatable technical problems across different pages and builds, while manual testing is necessary for accessibility issues that depend on user context and involve actual interaction with assistive technologies.

When both approaches are used, the team achieves a wider coverage than if it relied on just one of the methods.

Automated accessibility testing can check common issues such as missing alternative text, low color contrast, incorrect ARIA attributes, form labeling problems, and heading structure errors.

These checks can be added to development and CI/CD workflows so accessibility issues are identified as part of regular testing.

Manual testing can then be used to check keyboard navigation, focus movement, reading order, screen reader announcements, touch interactions, and other areas where automated rules cannot provide a complete assessment.

TestMu AI’s accessibility testing suite supports both automated and manual accessibility testing across web and mobile applications.

Its automated accessibility testing tool uses the axe-core rules engine and provides 39 WCAG-aligned automated checkpoints for web applications.

It also provides 15 rules for iOS and 18 rules for Android.

The suite supports WCAG 2.0, 2.1, and 2.2 across Levels A, AA, and AAA along with standards and regulations such as ADA, Section 508, AODA, and the European Accessibility Act.

Let’s look at why you should choose TestMu AI (formerly LambdaTest) for accessibility automation:

• Comprehensive Scanning Options: Offers a range of scanning options, including full-page, partial-page, multi-page, and workflow scans, allowing us to scan individual components, entire websites, and dynamic application workflows.
• Accessibility DevTools: Delivers browser-based scanning powered by axe-core, along with detailed issue reports, full-page analysis, and debugging support for developers.
• Accessibility Automation: Connects accessibility testing with CI/CD pipelines using Selenium, Playwright, and Cypress to run regular automated checks.
• Accessibility Test Scheduling: Supports recurring and one-time scans, with a sitemap extractor that pulls URLs automatically from the website’s sitemap so teams don’t have to update URLs manually.
• Native App Accessibility Testing: Run manual and automated accessibility tests on real Android and iOS devices, with detailed insights and actionable reports across the mobile app surface.
• Screen Reader Verification: Verifies the user experience using native screen readers – NVDA on Windows, VoiceOver on macOS and iOS, and TalkBack on Android – across major platforms.
• Accessibility MCP Server: Integrates seamlessly with your testing infrastructure to deliver AI-native audits using the Model Context Protocol (MCP), so AI assistants can detect, explain, and fix accessibility issues directly in your IDE for both publicly hosted websites and local front-end React applications.

Conclusion: Better Accessibility Starts Where Automation Stops

Automated accessibility testing helps development teams detect common issues faster, reduce repetitive work, and catch problems before they reach users.

However, a passing test report cannot tell you everything about how accessible your product actually is.

To get the most value from accessibility testing, teams should focus on three things:

• Automate what can be measured. Use automated testing to catch color contrast failures, missing labels, and structural errors consistently across your application. This saves time and helps developers fix common issues earlier.
• Manually test what requires human judgment. Check keyboard navigation, screen reader experiences, error messages, and critical user journeys to understand whether people can actually complete tasks.
• Make accessibility part of development. Run automated checks throughout development and schedule manual reviews when introducing new components or changing important workflows. This helps prevent accessibility problems from accumulating.

✨ Does passing every automated test mean your website is accessible? No, because automated tools cannot fully evaluate context, meaning, or real-world usability.

✨ Do you need to manually inspect every page after every code change? Not necessarily. Automation can handle repeatable checks, while manual testing can focus on new features, complex interactions, and high-impact user journeys.

✨ Can combining both methods guarantee WCAG compliance? No testing approach guarantees compliance on its own, but combining methods provides stronger evidence and helps teams identify issues that either approach might miss.

I was recently thinking about how much our decisions depend on what we can measure, especially while reading about Nobel Prize-winning economic research. Economists use models to understand complex behavior, but those models still depend on assumptions and interpretation.

I see a similar lesson in accessibility testing: automated results provide valuable information, but we still need to understand what those results mean for real people.

The next step is to identify which checks your team can automate today and where manual testing will add the most value.

If you’re building or improving your testing process, explore our guide to automated accessibility testing to learn how to introduce repeatable accessibility checks into development and make better use of your team’s testing time.

Start with the issues automation can catch, then use human testing to understand the experience it cannot measure.