How to evaluate the Nyne enrichment APIs

The dimensions that matter

"Is it any good?" is not measurable. These four dimensions are. Track each one separately - a single blended score hides the tradeoffs you are trying to see.

On latency, measure the whole round trip. Real endpoints are asynchronous: you submit, get a request_id, and poll until the result is ready. Server compute time is only part of what your users feel - the submit-then-poll loop is the number that matters. See Async requests and Request depth for how lookup_mode and search type move both coverage and time.

On cost, remember there are two shapes. Enrichment is largely per-call; search and employees are per-result, so budget for roughly credits x limit. Feature toggles like show_emails, show_phone_numbers, and insights change the bill, so record cost per run alongside accuracy. The full rules are in Credits.

Build fair ground truth

A benchmark is only as honest as the answer key behind it. Use a labeled test set of known-correct contacts you already trust - records verified independently of the API you are grading. Measure the API's output against that fixed set, and keep the set unchanged across runs so every result is comparable.

The output of a run is a scorecard you compute, not something the API hands back. Keep per-field hit rate, recall, precision, latency, and cost together so no single number gets read in isolation:

{
  "depth": "fast",
  "sample_size": 500,
  "field_hit_rate": {
    "best_business_email": 0.91,
    "fullphone": 0.62,
    "linkedin": 0.88,
    "title": 0.79,
    "location": 0.83
  },
  "precision": 0.94,
  "recall": 0.71,
  "median_latency_ms": 840,
  "credits_per_match": 1
}

A repeatable workflow

Run these six steps in order. Step 5 loops back through the middle: change one knob, re-read the metrics, record the cost and latency, repeat.

Anti-patterns to avoid

Most bad benchmarks fail the same handful of ways. Each of these quietly inflates or muddies your numbers:

• Counting ambiguous results as matches. A low-confidence guess is not a hit. Set a probability or confidence bar and score anything below it as a miss. The scoring field is documented in Responses.
• Comparing depths at different settings. If deep has more toggles on than fast, you are comparing two products, not two depths. Change one thing per run.
• Treating cached responses as fresh trials. Re-running the exact same input may be served from cache, so it is not an independent sample. Vary the inputs across your test set instead of hammering one record.
• Conflating the two usage ledgers. Per-call and per-result charges are metered differently; tallying them as one number will misstate cost. See Credits for how each is counted.

Production-readiness checklist

Once the numbers hold up, lock in the decisions before you ship:

• Pick a depth and toggle set - the lookup_mode / search type and feature flags your metrics justify.
• Set a confidence bar and enforce it in code, so low-probability results never reach downstream systems as facts.
• Wire usage-analytics monitoring so per-field hit rate and cost stay visible in production, not just in the benchmark.
• Watch failure and timeout rates, and alert when they drift from your baseline.
• Set a max-wait and backoff for polling, so a slow request degrades gracefully instead of hanging your pipeline.

Enforcing the confidence bar in code is a one-liner - do it at the seam where results enter your system:

# Only count matches at or above your confidence bar.
# A low-probability result is a "maybe", not a hit.
match = result.get("probability", 0) >= 0.85

What's next

• Request depth - how lookup_mode and search type trade coverage against latency and cost.
• Credits - per-call vs per-result metering and how feature toggles change the bill.
• Responses - the response shape, the scoring field, and how to read a confidence bar.
• Effective requests - shaping inputs and toggles to get the coverage you measured for.