Short answer
This AI QA testing glossary defines the terms you will meet when you evaluate or use an AI testing tool, in plain English and in one or two sentences each. Every term has its own link, so you can share a single definition.
- Terms are grouped by stage: the agent, the tests, the results, and where tests run.
- Definitions describe the general idea first, then how TestOptim AI handles it where that helps.
On this page
AI QA testing glossary: the agent
- AI QA testing
- Quality assurance in which an AI agent explores an app, writes tests, runs them and reports bugs, instead of people writing and maintaining test scripts. See the AI QA testing guide.
- Autonomous QA agent
- An AI system that tests an application without being told each step. It decides what to try, observes the result and records failures.
- Exploratory testing
- Testing without a fixed script: the tester learns the app while testing it and follows whatever looks likely to break. AI agents automate this by trying unusual inputs and sequences.
- Knowledge base
- In TestOptim AI, the map of pages, forms, entities and flows the agent records while exploring. Test cases are generated from it, and you can review and edit it.
- Test account
- A login created only for testing, with the access of a normal user of one role. Giving an agent a real person's account or an admin account risks real data.
- Credential profile
- A labelled set of sign-in details stored on a TestOptim project, such as Customer or Admin, so the agent can explore as each role. See add login credentials.
- One-time code (OTP)
- A short code sent by SMS or email at sign-in. Automated tests either use a fixed test code the app accepts, or pause for a person to enter the live one.
Test design terms
- End-to-end (E2E) test
- A test that drives the real app through a browser the way a user would, from the first page to the final result, across frontend, backend and database.
- Test case
- One scenario to check: a starting point, numbered steps and an expected result.
- Test suite
- All the test cases for an app or feature, run together.
- Happy path
- The scenario where the user does everything right, such as a valid sign-up. Necessary, but rarely where bugs hide.
- Negative test
- A test that does something wrong on purpose, such as submitting an empty form, to check that the app refuses it clearly.
- Edge case
- An input or sequence at the limits of what the app expects: the longest name, a date in the past, a refresh mid-checkout.
- Regression testing
- Re-running existing tests after a change to confirm that what worked before still works.
- Smoke test
- A short set of checks that the most important flows work at all, run before deeper testing or right after a deploy.
- Self-healing test
- A test that finds the element it needs even after the page's markup changed, instead of failing on a renamed selector. See self-healing tests.
- Selector (locator)
- The rule a script uses to find an element, such as a CSS class or XPath. Brittle selectors are the most common cause of broken scripted tests.
Results and bug report terms
- Test run
- One execution of some or all test cases, with a result for each. See run your tests.
- Flaky test
- A test that passes and fails on the same code, usually because of timing, shared test data or a slow network rather than a real bug.
- False positive
- A reported bug that is not a bug, for example a deliberate design choice the agent misread. Good evidence on each report makes false positives quick to dismiss.
- Severity
- How bad a bug is. TestOptim uses critical (for example a wrong total or charge, or a key flow fully blocked), high, medium and low (cosmetic).
- Steps to reproduce
- The numbered actions that make a bug happen again. Without them, a developer has to rediscover the bug before fixing it.
- Expected vs actual result
- What should have happened next to what did happen. Together with the steps, it turns a complaint into a fixable bug.
- Trace
- A recording of a browser test's actions, network requests, console output and page snapshots that you can step through after the run. TestOptim runs offer a trace download.
- Fix verification
- Re-running the test behind a bug after it is marked fixed, so the fix is proven rather than assumed. See verify a fix.
Where tests run
- Staging environment
- A copy of the app with its own data, used to test before release. The safe place to point an AI agent.
- Preview deployment
- A temporary copy of the app built from one branch or change, with its own URL. Vercel creates one for every branch; some are behind a login.
- CI/CD
- Continuous integration and delivery: the automated pipeline that builds, tests and ships each change. AI QA testing can run as one step of it.
- Visual regression testing
- Comparing screenshots of each build with an approved baseline to catch visual changes. It is a separate kind of tool from functional AI QA testing.
- Vibe coding
- Building software by describing it to an AI app builder or coding agent instead of writing the code yourself. See testing vibe-coded apps.
- Row level security (RLS)
- Database rules that decide which rows each user may read or change. Missing RLS is a common data leak in apps built on Supabase.
Frequently asked questions
What is the difference between a test case and a test run?
A test case is one scenario to check, with steps and an expected result. A test run is one execution of a set of test cases at a point in time, with a pass or fail for each. The same test case appears in many runs.
What is the difference between a flaky test and a false positive?
A flaky test gives different results on the same code, usually because of timing or shared data. A false positive is a reported bug that is not a real bug. A flaky test can produce false positives, but a stable test can too if it checks the wrong expectation.
Is exploratory testing the same as AI QA testing?
No. Exploratory testing is a way of testing without a script, done by people long before AI. AI QA testing often automates exploratory testing, and then also turns what it finds into repeatable test cases and runs them.
What is the difference between a negative test and an edge case?
A negative test feeds the app something invalid, such as an empty required field, and checks it is rejected. An edge case uses a valid but extreme input or sequence, such as the maximum quantity or a refresh mid-flow, and checks it still works.
Do self-healing tests hide real bugs?
They should not. Self-healing re-locates an element that moved or was renamed; it does not change what the test expects to happen. If the button is gone or the result is wrong, the test still fails and reports it.
What is a Playwright trace used for?
A trace lets you replay a browser test step by step after it ran, with the page snapshot, network requests and console output at each action. Developers use it to see exactly why a test failed without re-running it.
More AI QA testing guides
- AI QA testing: what it is, how it works, and when to use itAI QA testing explained: how autonomous QA agents explore a web app, run end-to-end tests and report bugs, what they miss, and how to choose a tool.
- E-commerce testing with AI: protect checkout, cart and searchE-commerce testing with an AI agent: what to test in cart, checkout, search and accounts, the bugs that cost stores money, and how to run it on staging safely.
- SaaS testing with AI: roles, sign-in and core workflowsSaaS testing with an AI agent: cover sign-up, roles and permissions, billing and core workflows on every release, without a test framework to maintain.
- QA for startups without a QA teamQA for startups without a QA engineer: what to test first, how AI QA testing covers regression on every release, and what it costs compared with a hire.