Building a web scraping pipeline with orchestration: what developers actually need to think about

Orchestration is not the hard part of a scraping pipeline. The hard part is making the extraction itself reliable enough to be worth orchestrating.

This distinction matters a lot when you are designing a recurring data collection system. Tools like Dagster, Airflow, or Prefect are genuinely useful for scheduling jobs, managing dependencies between pipeline steps, and monitoring runs. But they are orchestration layers. They do not fetch pages, handle JavaScript rendering, rotate proxies, or parse structured data out of raw HTML. That work still needs to happen somewhere, and it is where most pipelines accumulate hidden complexity over time.

What orchestration actually handles

An orchestration tool gives you a way to define when a job runs, what order steps execute in, and what happens when something fails. In a scraping pipeline, this typically means triggering a scraper on a schedule, moving output files to storage, and kicking off downstream transformation steps.

What it does not give you is a scraper. The extraction logic, the infrastructure that fetches and renders pages, the handling of blocked requests, the parsing of structured fields from unstructured HTML — all of that lives outside the orchestrator. You either build it yourself or you connect to something that handles it.

This is a meaningful architectural choice. If you are managing your own scraper stack, you are also managing JavaScript rendering, proxy rotation, retry logic, and selector maintenance. Each of those components adds surface area that can break independently, and the maintenance cost compounds as you add more target sites or increase volume.

Where the real complexity lives

A scraping pipeline that runs against a handful of static pages is straightforward. The complexity grows quickly when you are dealing with dynamic, JavaScript-heavy sites, pages behind anti-bot systems, geo-targeted content, or layouts that change periodically without notice.

Handling JavaScript rendering alone requires running a headless browser or connecting to a rendering service. Proxy rotation requires managing IP pools, detecting failures, and switching providers when blocks occur. Anti-bot bypass requires staying current with detection techniques that evolve continuously. And selector maintenance means revisiting your extraction logic every time a target site updates its HTML structure.

None of this is handled by your orchestrator. It all lives in the extraction layer, and it all requires ongoing engineering attention.

A cleaner separation: orchestration vs. extraction

One approach that simplifies this architecture is treating extraction as a managed API call rather than a self-maintained stack. Instead of your orchestrator triggering a custom scraper that handles rendering, proxies, and parsing internally, it calls an extraction endpoint with a list of URLs and receives structured JSON back.

This is what the Minexa API does. It combines crawling, JavaScript rendering, anti-bot handling, and structured data extraction into a single POST request. Your orchestration layer submits URLs, and the API returns clean, structured output. The infrastructure complexity stays on the API side.

The extraction model works through a trained scraper. You use the Minexa Chrome extension to select the HTML container on a target page, and Minexa generates a reusable scraper automatically, identifying all relevant data fields within that container. This takes a few minutes. The resulting scraper is assigned a scraper_id that you reference in every subsequent API call.

Install the Minexa Chrome extension

How the API request is structured

A standard extraction request looks like this:

POST https://api.minexa.ai/data/

{
  "batches": [
    {
      "scraper_id": 4721,
      "columns": ["top_30"],
      "urls": ["https://example.com/listing/1"],
      "scraping": {
        "js_render": true,
        "timeout": 30,
        "js_code": [
          { "wait_time": 2 },
          { "page_init": true },
          { "wait_time": 4 }
        ],
        "proxy": "verified",
        "retry": 3
      }
    }
  ],
  "threads": 5
}

The scraper_id identifies which trained scraper to use. The columns parameter controls which fields are returned. Using "top_30" returns the thirty most relevant fields ranked by Minexa's internal algorithm, which is useful during exploration. For production pipelines, you can replace this with an explicit list of named columns like ["price", "location", "availability"]. Both approaches cost the same.

The scraping object controls how pages are fetched. js_render enables JavaScript execution for dynamic pages. proxy sets the IP type. retry handles automatic re-attempts on failure. For sites with stronger bot protection, you can switch the provider field to a more capable engine, though this increases credit consumption per page.

The threads parameter sets how many URLs are processed in parallel. Higher values increase throughput up to the limit of your plan.

Using pre-stored HTML to reduce costs

If you already have HTML files stored from a previous crawl, for example in S3 or on a CDN, you can pass those directly to the API using the file_urls parameter. Minexa reads the HTML from those URLs instead of re-fetching the original pages. This is the cheapest scraping configuration since no live crawling or JavaScript rendering is needed.

{
  "scraping": { "js_render": false, "proxy": "verified" },
  "file_urls": [
    "https://your-cdn.cloudfront.net/page-1.html",
    "https://your-cdn.cloudfront.net/page-2.html"
  ],
  "urls": [
    "https://original-site.com/page-1",
    "https://original-site.com/page-2"
  ]
}

The urls field here contains the original source URLs so extracted data maps back to the correct page. This pattern fits well in pipelines where crawling and extraction are separate scheduled steps.

Scheduling and cron jobs

The Minexa API does not manage your schedule. You control when jobs run, which fits naturally into an orchestration setup. Your orchestrator or cron job assembles the URL list for a given run and calls the API. The API processes the batch and returns structured output. Your pipeline then moves that output to storage or triggers the next step.

This separation is intentional. For pipelines with many different URLs across different sites, managing the schedule externally and calling the API per batch gives you full control over timing, retry behavior, and how results flow into downstream systems.

What deterministic extraction means for pipeline reliability

Minexa's extraction is DOM-based and deterministic. Running the same scraper on the same page always produces identical JSON output as long as the underlying HTML has not changed. There is no variance between runs, no fabricated values when a field is missing, and no silent failures. If a page structure changes and the scraper no longer matches, affected fields return null or an explicit error, not a plausible-looking wrong value.

This matters for orchestrated pipelines specifically because silent data quality issues are hard to catch downstream. A system that fails loudly when something breaks is easier to monitor and debug than one that continues producing output that looks correct but is not.

Read the Minexa API docs

FAQ

Does Minexa handle pagination automatically when using the API? Pagination is not automatic in the API. If you need to navigate through multiple pages, you define the required JavaScript interactions in the js_code array within the scraping configuration. Automatic pagination handling is available when using the Chrome extension directly.

How does retraining work when a site changes its layout? When a site redesigns and the existing scraper starts returning errors or null values, you open the updated page in the browser extension, select the new container, and create a new scraper. This generates a new scraper_id. The only required code change is updating that ID in your API request body.

Can I submit large batches of URLs in one request? Yes. The API accepts thousands of URLs in a single batch request, which makes it practical to trigger large extraction jobs from a single orchestration step.

For a deeper look at how the complete scraping process breaks down stage by stage, this post covers each step in detail: The complete web scraping process: what each stage actually involves.