How to choose a test automation vendor
Updated
To choose a test automation vendor, write the scope first, then send an RFP that asks for evidence rather than promises: automation cases with metrics, a code ownership clause, a sample CI configuration and certificates you can check at the issuer. Score the answers against a fixed checklist, call references, and run a paid pilot on one critical flow with acceptance criteria agreed in advance. After signing, track flaky rate, time to fix broken tests, escaped defects, coverage of critical flows and suite runtime every month.
This guide expands the six steps on the ranking page into a full procedure. For candidate names, start with the ranking and the shortlist.
The RFP process, stage by stage
Each stage removes vendors that cannot produce a specific artifact, so the expensive stages (references and the pilot) are spent on two or three candidates.
The durations below are planning ranges, not measured data. Add the vendor's own time to start. Among the twelve ranked companies, the shortest stated start is 3 days (QA Madness). The longest is 30 days (Zoolatech).
| Stage | What to ask | Evidence to request | Who decides |
|---|---|---|---|
| 1. Internal brief (typical range: 1-2 weeks) | Internal only: platforms, critical flows, frameworks, release cadence, CI tool, data constraints | Your list of critical flows, ranked by business risk | QA lead and engineering manager |
| 2. Long list and RFI (typical range: about 1 week) | Frameworks used on products like yours, engagement models, time to start, minimum engagement | Links to public cases, review profiles and a certifications page | QA lead |
| 3. RFP responses (typical range: 2-3 weeks for vendors to answer) | Approach for your top flows, maintenance model, repair time, pricing basis | Cases with metrics, draft ownership clause, sample pipeline file, named team lead | QA lead, engineering manager, procurement |
| 4. Evidence check and references (typical range: 1-2 weeks) | Whether past clients confirm case numbers and certificates match the contracting entity | Reference calls, certificate numbers and issuing bodies, audit report or trust page | QA lead, security or compliance owner |
| 5. Paid pilot (typical range: 2-4 weeks) | Automate one or two critical flows in your repository and CI | Merged code, run history, repair log, handover notes | Engineering manager, with QA lead scoring |
| 6. Contract and security review (runs in parallel with the pilot) | Ownership, exit terms, KPIs, data processing terms | Signed DPA or BAA where required, KPI annex, exit plan | Procurement, legal, security, budget owner |
Send every vendor the same brief and scoring sheet, and fix the weights before the first response arrives. Weights chosen after reading proposals drift toward the vendor the team already likes. Turn the questions before hiring into required attachments: an answer given only in a sales deck counts as missing.
Evaluation checklist mapped to eight criteria
The checklist follows the eight dimensions and weights of this site's methodology. Your weights can differ: a regulated product may raise security, a team with a mature framework may raise maintenance. Score each row 0, 25, 50, 75 or 100 and write one line of evidence per score.
| Criterion (methodology weight) | What to check | A strong answer | A warning sign |
|---|---|---|---|
| Framework depth and documented cases (20) | Cases close to your stack and product type | Three or more cases with a starting point and a measured change, on the frameworks you use | Framework logos only, or cases with no numbers |
| Test code ownership and handover (16) | Where the code lives and what happens at exit | Written clause: code in your repository from the first commit, open-source frameworks, documented handover | Tests that run only inside the vendor's platform with no export path |
| CI/CD integration and maintenance (14) | Pipeline practice, flaky-test handling, who repairs failures | Named CI tools, a described flaky-test process, a repair time in hours | "We maintain tests as needed" with no owner or time frame |
| Client review quality and volume (12) | Reviews on Clutch, G2 or GoodFirms that describe automation work | Recent reviews from clients of your size that mention automation | Few reviews, or reviews that describe unrelated services |
| Named clients and published outcomes (12) | Cases with a named client and a result | Named clients who agree to a reference call | Anonymised cases only, with no reference offered |
| Security and delivery governance (10) | Certificates, data handling, environment access | Certificate verified at the issuer, a written test-data policy | Badges on the website with no certificate number or scope |
| Engagement flexibility and onboarding (9) | Models offered, time to start, minimum contract | Two or more models, a stated start time and a pilot option | One model only, long minimum term |
| Pricing transparency (7) | What the price covers and how it grows with the suite | Published rates or a pricing basis tied to scope, maintenance included or priced separately | Price only after several calls, maintenance undefined |
Read the lowest row as well as the total: a vendor near zero on code ownership can leave you with a suite you cannot move.
Guidance for regulated products
Healthcare and fintech products add three questions: what data the vendor touches, which certificates cover the work, and which contract terms govern the data. Test automation should not need real personal or payment data at all.
Verify certifications at the issuer
Ask for the certificate number, the certification body and the scope statement, then look the certificate up in the issuer's registry. Check three details: the legal entity named on the certificate, the sites or services in scope, and the expiry date. Verify the current audit scope during procurement, because scopes and dates change between renewals.
QASource states that its ISO 27001 is held through its delivery partner, bebo Technologies (security protocols). TestFort states CMMI Level 3 on its home page, and this ranking did not verify it in the CMMI Institute appraisal registry (TestFort). Neither is disqualifying on its own, but each should be settled in writing before signing.
For medical device software, ask whether the vendor holds ISO 13485, the quality management standard for medical devices. BetterQA lists ISO 13485 on its about page (BetterQA). ScienceSoft also lists it (ScienceSoft). Both are vendor-stated and should be checked at the issuer.
Audit periods
A SOC 2 Type 2 report covers a defined observation period, so ask for the period dates and read whether the report is recent enough for your own auditors. If the period ended months ago, ask for a bridge letter covering the gap. ISO certificates run on a multi-year cycle with periodic surveillance audits; ask for the date of the last one.
Data handling, BAAs and DPAs
Ask for a written test-data policy. Kualitatem's FAQ states that its standard practice is anonymized, masked or synthetic test data (FAQ). Ask as well where engineers sit and where data is stored: QualityLogic states that its team is US-based and that no data leaves the country (QualityLogic).
If the vendor could see protected health information, a US healthcare buyer needs a Business Associate Agreement before access starts. If the vendor processes personal data of people in the EU, a Data Processing Agreement is required. Some vendors publish these documents on a trust page. QA Wolf's Trust Center, for example, lists a Data Processing Agreement alongside its security reports (Trust Center). For fintech, add environment rules to the contract: no production credentials, masked account data, and access through accounts you control and can revoke. The ranking's trust and compliance section shows how to read a vendor's certifications.
Measuring performance after signing
Put five KPIs in a contract annex, agree how each is measured, and review them monthly from your own CI data rather than the vendor's report.
| KPI | How to measure | Where the data comes from |
|---|---|---|
| Flaky rate | Share of test runs that fail and then pass on a rerun with no code change, over a rolling 30 days | CI run history, with reruns logged |
| Time to fix a broken test | Median hours from the first red build caused by a test to the merged fix | CI timestamps and pull request merge times |
| Escaped defects | Production defects per release in flows the suite is meant to cover | Issue tracker, tagged by flow |
| Coverage of critical flows | Share of the agreed critical flows with at least one automated test passing in CI | The flow list from stage 1, mapped to test files |
| Suite runtime | Wall-clock time of the full regression run, median and slowest of the month | CI job durations |
Set targets from the pilot's own numbers, not from marketing figures. a1qa lists "CI signal quality" as a KPI of its AI accelerator, tracking flaky tests, false failures, reruns and execution stability (a1qa). One a1qa case reports flaky tests cut from 18% to 4.6%, a vendor-published figure. Ask any vendor quoting such numbers for the raw run data.
Review the critical-flow list every quarter, so coverage and escaped defects are measured against the product as it is now.
Designing the pilot
A pilot shows how the vendor works inside your repository, CI and release rhythm. Design it so the result is a set of numbers, not an impression.
- Scope. One or two critical flows, chosen from the stage 1 list, with at least one flow that touches a third-party integration or a tricky state such as payments or login with multi-factor authentication.
- Same brief for each finalist. If two vendors run pilots, give them the same flows, the same environment and the same deadline.
- Setup rules. Code goes into your repository through pull requests reviewed by your engineers, and the suite runs in your CI from the first week.
- A planned break. In the second half of the pilot, change a selector or a page layout on purpose and time how long the vendor takes to repair the tests.
- Repeated runs. Run the suite at least daily and record the flaky rate over the whole pilot.
- Exit criteria written before day one. Tests delivered, pass rate over repeated runs, repair time after the planned break, documentation good enough for your team to run the suite alone.
Score the pilot with the RFP checklist and compare it with the paper score; a large gap is itself a finding. Pilot budgets are covered in the guide to test automation outsourcing cost. The guides to outsourcing vs in-house and test automation as a service help if the pilot raises the question of who should own the work.
FAQ
What should a test automation RFP ask for?
A test automation RFP should ask for evidence as attachments, not answers in prose. Request two or three automation cases with a starting point and a measured result, a draft clause stating that test code lives in your repository, and a sample CI pipeline file. Ask for the maintenance model with a repair time in hours, certificate numbers with the issuing body, and the pricing basis. Include your critical flows and ask how the vendor would automate the top three.
How long does a test automation RFP take?
Under the planning ranges in this guide, an RFP from internal brief to signed contract takes roughly 7 to 12 weeks, with the contract and security review running in parallel with the pilot. The brief takes one to two weeks, vendor responses two to three, reference checks one to two, and the pilot two to four. These are planning estimates, not measured data. The chosen vendor's own time to start comes on top.
How many vendors should reach the pilot stage?
Two vendors in the pilot is a practical ceiling for most teams. Each pilot needs your engineers to review pull requests and provide an environment, and a third pilot rarely changes the decision. A common funnel runs from six to eight names at the RFI, to three or four scored RFP responses, to two finalists after reference calls. The ranking's shortlist section suggests a starting trio. If one finalist clearly leads after references, a single pilot is enough.
What belongs on a QA vendor evaluation checklist?
A QA vendor evaluation checklist should cover eight areas: documented automation cases, code ownership and handover, CI integration and maintenance, client reviews, named clients with outcomes, security and data handling, engagement models, and pricing transparency. Before responses arrive, write what a strong answer and a warning sign look like for each row. Score every row on a fixed scale, such as 0 to 100 in steps of 25, and record one line of evidence per score.
Which evaluation criteria should carry the most weight?
Documented cases and code ownership deserve the most weight for most buyers, because they are the hardest to fix after signing. In this site's methodology, framework depth and documented cases carry 20 points of 100. Code ownership and handover carry 16, and CI integration and maintenance 14. Reviews and named clients carry 12 each, security 10, engagement 9 and pricing 7. A regulated product may justify raising security. Fix the weights before reading proposals.
How do you measure a test automation vendor's performance?
Measure performance with five KPIs taken from your own CI and issue tracker. Flaky rate is the share of runs that fail and then pass on rerun with no code change. Time to fix is the median hours from the first red build to the merged fix. Escaped defects are production bugs in flows the suite should cover. Critical-flow coverage and full regression runtime complete the set. Put all five in a contract annex and review them monthly.
How do I choose a test automation company for a regulated product?
Start from the data. Design the engagement so the vendor never needs real patient or payment data, and ask for a written policy based on masked or synthetic test data. Verify each certificate at the issuing body, checking the legal entity, scope and expiry date. For SOC 2 Type 2, ask for the audit period dates. Sign a BAA if protected health information could be reached, and a DPA for EU personal data. The ranking's scenarios name vendors for healthcare and fintech.
Can I replace a test automation vendor that underperforms mid-contract?
Yes, if the contract prepared for it. A switch is simple when the test code already lives in your repository, runs in your CI and uses an open-source framework. It is hard when the suite runs only on the vendor's platform. Write an exit clause covering handover of code, test data, pipeline configuration and documentation, with a transition period. Tie the right to exit to KPI targets missed over two or three consecutive months.
What is the shortest sound process for choosing a vendor?
The shortest process that still protects the buyer has four steps. Write a one-page brief with your critical flows and stack. Ask three vendors for cases with metrics, an ownership clause and a sample pipeline, and score them on one checklist. Call one reference per finalist. Then run a paid pilot of two to four weeks with exit criteria set in advance. Build the list from the ranking and see how each criterion is scored in the methodology.