# How to evaluate the Nyne enrichment APIs

> A practical, honest-tradeoffs guide to benchmarking the Nyne.ai person and company enrichment, lookup-fields, and search APIs before you trust them in production: fair ground truth, the dimensions that matter (accuracy, coverage, latency, freshness, cost), and a repeatable workflow that changes one knob at a time.

HTML version: https://api.nyne.ai/documentation/evaluating

## Start simple, stay fair

The golden rule of evaluation: **baseline one endpoint at one depth against a fixed dataset before you touch a single feature toggle.** Get a clean, boring number first. Then, and only then, start changing things - one knob per run.

The most common way to get a misleading result is to compare depths on unequal footing: pitting `fast` latency against `deep` coverage, for example. Those are two different questions. Hold everything else constant and change one variable at a time, or you will not be able to say which change caused which shift in the numbers.

```bash
curl -X POST https://api.nyne.ai/person/enrichment \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "X-API-Secret: YOUR_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{ "email": "jane.doe@acme.com", "lookup_mode": "fast" }'
```

**One knob at a time.** A run is only comparable to the run before it if exactly one thing changed. Move `lookup_mode` from `fast` to `balanced`, or flip `high_freshness` on - but never both in the same step, and never against a different dataset.

## The dimensions that matter

"Is it any good?" is not measurable. These four dimensions are. Track each one separately - a single blended score hides the tradeoffs you are trying to see.

| Dimension | What to measure | Nyne specifics |
| --- | --- | --- |
| Accuracy & coverage | Per-field hit rate (email, phone, LinkedIn/social, title, company, location); recall vs precision. | Recall and precision shift with depth: deeper modes surface more fields but can admit lower-confidence ones. Set a confidence bar per field. |
| Latency | End-to-end time, including the async submit and poll round trip - not just server time. | Varies by lookup_mode (fast / balanced / deep) and search type (light / medium / premium). Measure the whole submit-then-poll loop. |
| Freshness | How recent the returned data is, and where coverage thins out. | The search high_freshness toggle trades speed for recency. Report coverage gaps honestly rather than averaging them away. |
| Cost | Credits per call: per-call fixed vs per-result, and how toggles move the bill. | Search and employees bill roughly credits x limit; show_emails, show_phone_numbers, and insights each add to the charge. |

On **latency**, measure the whole round trip. Real endpoints are asynchronous: you submit, get a `request_id`, and poll until the result is ready. Server compute time is only part of what your users feel - the submit-then-poll loop is the number that matters. See [Async requests](https://api.nyne.ai/documentation/async.md) and [Request depth](https://api.nyne.ai/documentation/request-depth.md) for how `lookup_mode` and search `type` move both coverage and time.

On **cost**, remember there are two shapes. Enrichment is largely per-call; search and employees are per-result, so budget for roughly `credits x limit`. Feature toggles like `show_emails`, `show_phone_numbers`, and `insights` change the bill, so record cost per run alongside accuracy. The full rules are in [Credits](https://api.nyne.ai/documentation/credits.md).

## Build fair ground truth

A benchmark is only as honest as the answer key behind it. Use a **labeled test set** of known-correct contacts you already trust - records verified independently of the API you are grading. Measure the API's output against that fixed set, and keep the set unchanged across runs so every result is comparable.

**Do not grade the API against its own output.** The classic trap: you enrich a list, then treat those results as "truth" and re-score the same API against them. That measures consistency, not correctness - the API will always agree with itself. Ground truth has to come from outside the system under test, and the test set must not be one the API was seeded from.

The output of a run is a **scorecard you compute**, not something the API hands back. Keep per-field hit rate, recall, precision, latency, and cost together so no single number gets read in isolation:

```json
{
  "depth": "fast",
  "sample_size": 500,
  "field_hit_rate": {
    "best_business_email": 0.91,
    "fullphone": 0.62,
    "linkedin": 0.88,
    "title": 0.79,
    "location": 0.83
  },
  "precision": 0.94,
  "recall": 0.71,
  "median_latency_ms": 840,
  "credits_per_match": 1
}
```

**Reuse the methodology, not a harness.** Nyne runs its own internal accuracy tooling against its search backend. You cannot plug into that harness, and you should not want to - the point is to borrow the method: a fair labeled set the system was not trained on, measured one knob at a time. That is what makes any number you report defensible.

## A repeatable workflow

Run these six steps in order. Step 5 loops back through the middle: change one knob, re-read the metrics, record the cost and latency, repeat.

1. **Define scope** - decide what good means for your use case and which fields matter.
2. **Fix the dataset** - pick a labeled test set the API was not seeded from.
3. **Baseline one depth** - run ONE endpoint at ONE depth, toggles off.
4. **Read per-field metrics** - hit rate, recall, precision, by field, not one blended score.
5. **Iterate one knob** - change a single knob, hold everything else constant.
6. **Record all three** - log cost and latency next to accuracy, every run.

## Anti-patterns to avoid

Most bad benchmarks fail the same handful of ways. Each of these quietly inflates or muddies your numbers:

- **Counting ambiguous results as matches.** A low-confidence guess is not a hit. Set a `probability` or confidence bar and score anything below it as a miss. The scoring field is documented in [Responses](https://api.nyne.ai/documentation/responses.md).
- **Comparing depths at different settings.** If `deep` has more toggles on than `fast`, you are comparing two products, not two depths. Change one thing per run.
- **Treating cached responses as fresh trials.** Re-running the exact same input may be served from cache, so it is not an independent sample. Vary the inputs across your test set instead of hammering one record.
- **Conflating the two usage ledgers.** Per-call and per-result charges are metered differently; tallying them as one number will misstate cost. See [Credits](https://api.nyne.ai/documentation/credits.md) for how each is counted.

**A flattering benchmark is a broken one.** If your evaluation always makes the API look great, suspect the evaluation. Fair ground truth and a confidence bar exist precisely to let real gaps show up in the numbers.

## Production-readiness checklist

Once the numbers hold up, lock in the decisions before you ship:

- [ ] Pick a **depth and toggle set** - the `lookup_mode` / search `type` and feature flags your metrics justify.
- [ ] Set a **confidence bar** and enforce it in code, so low-probability results never reach downstream systems as facts.
- [ ] Wire **usage-analytics monitoring** so per-field hit rate and cost stay visible in production, not just in the benchmark.
- [ ] Watch **failure and timeout rates**, and alert when they drift from your baseline.
- [ ] Set a **max-wait and backoff** for polling, so a slow request degrades gracefully instead of hanging your pipeline.

Enforcing the confidence bar in code is a one-liner - do it at the seam where results enter your system:

```python
# Only count matches at or above your confidence bar.
# A low-probability result is a "maybe", not a hit.
match = result.get("probability", 0) >= 0.85
```

## What's next

- [Request depth](https://api.nyne.ai/documentation/request-depth.md) - how `lookup_mode` and search `type` trade coverage against latency and cost.
- [Credits](https://api.nyne.ai/documentation/credits.md) - per-call vs per-result metering and how feature toggles change the bill.
- [Responses](https://api.nyne.ai/documentation/responses.md) - the response shape, the scoring field, and how to read a confidence bar.
- [Effective requests](https://api.nyne.ai/documentation/effective-requests.md) - shaping inputs and toggles to get the coverage you measured for.