The fastest way to evaluate an RLHF data vendor: run a paid pilot with known-answer items, and judge them on measured accuracy — not on their sales deck. Everything else on this checklist exists to tell you whether a vendor will survive that test before you spend the money.
Use these ten questions in every evaluation call. Vendors with good answers give them quickly; vendors without them pivot to case studies.
The checklist
1. Is pricing public, or does it require a sales process?
Opaque pricing usually means enterprise minimums. If you cannot estimate a pilot's cost from the website, expect a six-figure conversation. (For reference points, see our RLHF data pricing breakdown.)
2. Can you embed golden questions?
Golden questions — items with known correct answers mixed into production work — are the only quality signal you fully control. If a vendor cannot tell you the accuracy of workers on known answers, you are trusting self-reported quality. Ask: "Can I add my own known-answer items, and where do I see the accuracy per worker?" On trAIn, goldens are a first-class campaign feature with accuracy tracked per submission.
3. Do they support consensus labeling?
Single-label data bakes in individual annotator error. Ask whether items can be labeled by multiple annotators and how agreement is measured and reported to you. Three-label consensus with measured agreement is the difference between "we have labels" and "we can prove the labels are right."
4. What do you get back, exactly?
Request a sample export before signing anything. The deliverable should be machine-readable (JSONL/CSV/JSON) in a documented schema — one record per submission, with the label and the metadata you actually need — not a PDF summary. On trAIn, Managed-plan campaigns export JSONL directly and via API.
5. Who are the annotators and how are they gated?
"Global workforce" is not an answer. Ask how annotators are tested: is there an exam, a certification per task type, ongoing accuracy monitoring? A vendor that certifies annotators per skill (and can show pass rates) controls quality at the input, not just the output.
6. Can you provide guidelines, examples, and edge cases?
RLHF work fails on ambiguous instructions. The vendor's task setup should accept structured guidelines: worked examples, counter-examples, edge-case lists, and reference attachments. If the intake form is a free-text box and nothing else, expert-level tasks will come back wrong.
7. What are the minimums and the pilot shape?
A confident vendor lets you spend a few hundred dollars to evaluate them. Ask for the smallest paid pilot they support and whether pilot pricing matches production pricing. A pilot priced differently from production is a marketing exercise, not an evaluation.
8. Who owns the data, and how is it handled?
Get explicit answers in writing: you own the deliverables; annotators are under confidentiality terms; your data is not used to train anyone else's models; deletion on request is supported. This should be in the standard terms, not negotiated.
9. What happens when quality misses?
Every vendor has rework. The difference is whether misses are detected by their QA (goldens, consensus, review queues) or by your team after delivery. Ask what fraction of items get reviewed and what the rejection/rework flow looks like from your dashboard.
10. Can you talk to production data before scaling?
Before a large commitment, export a slice of completed pilot items and audit them yourself. Ten minutes of spot-checking against your rubric tells you more than any reference call.
Scoring the answers
Give each question 0 (no), 1 (partial), or 2 (yes, with evidence). A vendor scoring under 14/20 will cost you more in rework than their quote saves. A vendor scoring 18+ with transparent pilot pricing is worth a real test.
Try it on us
This checklist is how we would want to be evaluated, because we built trAIn to answer it: public pricing, golden questions and optional 3-label consensus, structured guidelines on every campaign, certified annotators, and JSONL exports. Download a sample deliverable and start a pilot from our companies page — then score us against the ten points above.