Apify dataset quality: four checks before import

Your scraper can finish while its output no longer meets your downstream requirements. Check the records before deciding whether to import them.

Your recurring scraper finishes. A dataset appears, and the next step imports it into a database. Later, you discover missing prices, repeated identifiers or a required field that disappeared.

Apify’s run documentation describes SUCCEEDED as a terminal execution status. Whether the resulting records satisfy your business requirements is a separate question. Some Actors already validate their output; inspect their documented checks before adding another tool.

1. Check required fields in each record

Start with the fields the next step actually needs. For a product workflow, these might be product_id, source_url, price and currency.

Field existence and valid content are different checks. A record can contain price with a null value. Define whether each field must exist, which type it needs and whether null or blank values are acceptable. Do not reject an optional description simply because it is absent.

{
  "required": [
    "product_id",
    "price",
    "currency"
  ],
  "unique_key": [
    "product_id",
    "variant_id"
  ],
  "max_empty_price_percent": 5
}

This is an illustrative checklist configuration, not an executable schema or an available product input. The 5% limit is an example, not a recommended universal threshold.

2. Measure empty values consistently

A field can remain present while its useful values disappear. Suppose empty prices rise from 2% of records to 31%. That change is 29 percentage points and warrants investigation if the runs cover equivalent inputs.

Decide in advance how to count missing fields, nulls, empty strings and whitespace-only strings. Do not treat valid values such as numeric zero or Boolean false as empty. Report the denominator and affected records so another person can reproduce the percentage.

3. Verify the right identifier is unique

Repeated identifiers can cause duplicate entries or overwrite records downstream. But a repeated product ID might be correct if each row represents a different variant.

Choose a single or composite key that matches the record’s intended granularity. Preserve identifiers as text when leading zeros matter. Count missing identifiers separately: a record without a key cannot be confidently matched between runs.

Also define the duplicate metric. “47 duplicate occurrences” means 47 rows beyond the first occurrence of their respective keys. It does not necessarily mean 47 distinct duplicate keys.

4. Compare record counts within the same scope

A drop from 1,000 records to 620 is a 38% decline. It deserves attention, but does not by itself establish a scraping failure. Listings may have expired, filters may have changed, or the latest run may cover different inputs.

Check the input URLs, filters, run limits, Actor build, pagination and collection time before interpreting the change. A comparison against yesterday’s incomplete run is not a reliable baseline either.

Define the outcome of your Apify dataset quality checks

A useful result distinguishes three outcomes:

• Rule failed: an explicit requirement was violated.
• Needs review: a change occurred, but its cause is unknown.
• Not assessed: the available data cannot support the check.

A single score hides these differences. Keep the rule, observed value and affected records alongside each finding. Decide whether a failed requirement should block an import; avoid automatically deleting or “repairing” records based only on a suspicious change.

Start with the tools you already have

Apify datasets support schemas and output validation, and can be exported for inspection. If you only need field and type checks, start with those capabilities or a validation library in your existing workflow.

Cross-run comparison may be useful when your problem is change over time. It still needs explicit rules and equivalent collection scope; it does not establish the accuracy of scraped facts or certify the source.

Sources and related pages

• Apify: Actor runs, builds and lifecycle statuses
• Apify: datasets, schemas, validation and exports
• Dataset Watch: proposed scope and example report

Independent educational material by Nuvetra, hosted on ClauseDock. No affiliation with Apify is implied. Dataset Watch is separate from ClauseDock’s invoice review product.