How to scrape logistics and supply chain data from Shiprocket using the Minexa API

Logistics data sits on public-facing pages in plain sight, but getting it into a structured format for analysis is rarely straightforward. Shiprocket, one of India's widely used shipping and logistics platforms, publishes a range of supply chain-relevant content across its site. Extracting that content consistently, at scale, and without writing custom scraper code is exactly the kind of problem the Minexa API is built to solve.

This guide walks through the full workflow: training a scraper on Shiprocket using the Minexa Chrome extension, then calling the Minexa API programmatically to run extractions across as many URLs as needed.

Watch the full tutorial first

Before going through the step-by-step breakdown, the video below covers the complete extraction workflow from start to finish. It shows the extension in action on Shiprocket and walks through the API request generation.

https://www.youtube.com/watch?v=3_NWEvewIVc

The problem with scraping logistics platforms manually

Logistics and supply chain pages often contain layered content: section headers, descriptive blocks, internal navigation anchors, data overviews, and update schedules all coexist on the same page. Manually copying this into a spreadsheet is slow and error-prone. Writing a custom scraper means dealing with DOM structure changes every time the site updates. Neither approach scales.

The Minexa API addresses this by separating the training step from the extraction step. You train a scraper once using the browser extension, and then reuse that scraper across any number of structurally similar pages via API calls. No selectors to write, no HTML parsing libraries to configure.

Step 1: Navigate to Shiprocket and open the extension

Start by opening the target page in Chrome. For this walkthrough, the starting URL is https://www.shiprocket.in/?s=shipping. Once the page has loaded, open the Minexa Chrome extension.

The extension detects the page and presents the option to confirm you are on the right page before proceeding.

Step 2: Confirm pagination and choose your scraping mode

After confirming the page, the extension analyses the page structure and surfaces any pagination it detects, such as a next-page button or load-more trigger. You review this and click Continue to proceed.

Next, you choose whether to scrape a single list page or to follow links into detail pages. For most logistics data collection workflows, the list mode is sufficient to capture the structured content available on each search result page.

Step 3: Select the data container and create the scraper

Once you have chosen your mode, the extension prompts you to select the HTML container holding the data you want. You hover over the block that wraps all the relevant content and click to confirm. Minexa then analyses the structure and automatically identifies all data points within that container.

After the scraper is created, all extracted columns are visible in the extension panel. You can navigate through them to verify the fields before proceeding.

What the extracted data looks like

Here is a sample of what Minexa returns from the Shiprocket page. Fields with empty values are included in the output as empty strings, which makes downstream processing predictable since the schema stays consistent across every row.

[
  {
    "market_survey_description": "毎日の卸売価格グラフ",
    "daily_info_description": "",
    "daily_info_link": "",
    "data_overview": "",
    "element_id": "",
    "link_id": "",
    "link_reference": ""
  },
  {
    "market_survey_description": "概要",
    "daily_info_description": "",
    "daily_info_link": "",
    "data_overview": "",
    "element_id": "",
    "link_id": "#a01",
    "link_id_2": "#a02",
    "link_id_3": "#a03",
    "link_reference": "#a05",
    "link_reference_2": "#a06"
  },
  {
    "market_survey_description": "",
    "daily_info_description": "",
    "element_id": "a01",
    "element_id_2": "",
    "link_id": ""
  }
]

A few fields worth noting. The link_id, link_id_2, link_id_3, and link_id_4 fields each encode an anchor-based internal navigation identifier. These hash-prefixed strings map to specific content sections within the page DOM, which is useful when you need to reconstruct the internal navigation structure of a page programmatically. The element_id and element_id_2 fields surface the raw DOM anchor values used as section targets, appearing without the hash prefix. Together, the link_id and element_id fields give you both sides of the anchor link relationship in a single row. The market_survey_description field returns the primary descriptive text per row, covering everything from category labels and section headings to longer content blocks depending on the row type.

Step 4: Get the API request and run extractions at scale

Once the scraper is created, click 'API Request' in the extension to view the pre-generated Python code. This code includes your scraper ID and a ready-to-run request body.

Below is the full Python script you can use to run the extraction. Replace the scraper ID and URLs with your own values. The script saves a checkpoint file after each API response, so you never lose progress on large batches.

import pandas as pd
import json, os, requests
from datetime import datetime

url = "https://api.minexa.ai/data/"
api_key = "YOUR_API_KEY"

data = {
  "batches": [{
    "scraper_id": 6214,
    "columns": ["top_40"],
    "urls": ["https://www.shiprocket.in/?s=shipping"],
    "scraping": {
      "js_render": True,
      "timeout": 30,
      "js_code": [{"wait_time": 2},{"page_init": True},{"wait_time": 4}],
      "proxy": "verified",
      "retry": 3
    }
  }],
  "threads": 5
}

headers = {"Content-Type": "application/json", "api-key": api_key}
next_set, started = None, False
iterated_data, extracted_data = [], []
file_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
  f"extraction_{datetime.now():%Y_%m_%d_%H_%M}")

while not started or next_set:
  if next_set:
    data["next"] = next_set
  elif "next" in data:
    del data["next"]
  response = requests.post(url, json=data, headers=headers)
  if response.status_code != 200:
    print(f"Request failed: {response.status_code}", response.content)
    break
  json_content = response.json()
  started = True
  for extraction in json_content["response"]:
    for rows in extraction["results"]:
      if rows.get("error"): continue
      iterated_data.append({k: v if isinstance(v, str) else
        [x["value"] for x in v] for k, v in rows.items()})
  extracted_data += json_content["response"]
  with open(f"{file_path}.json", "w", encoding="utf-8") as f:
    json.dump(extracted_data, f, ensure_ascii=False, indent=4)
  df = pd.DataFrame(iterated_data)
  df.to_excel(f"{file_path}.xlsx", index=False)
  next_set = json_content.get("meta", {}).get("next")
  print("finished" if not next_set else f"run {next_set}")
  if not next_set: break

The columns parameter accepts either a top_N shorthand or an explicit list of field names. Using top_40 returns the 40 highest-ranked columns as determined by Minexa's relevance algorithm. The ranking is deterministic, so the same value consistently maps to the same set of columns across runs. If you need only specific fields, replace top_40 with a named list such as ["market_survey_description", "link_id", "element_id"].

For Shiprocket pages that rely on JavaScript rendering, the scraping configuration above uses js_render: true with a proxy setting. If you find that certain pages are not returning complete data, try switching the provider to service3 or increasing the timeout. The extension's API Request dropdown includes pre-built scenario configs you can copy directly rather than tuning parameters manually.

If you are running extractions across a large set of URLs, set up your own cron job to call the API on a schedule and pass batches of URLs at each run. This gives you full control over timing and frequency without relying on any built-in scheduling interface.

Once the job completes, the full dataset is available for export as Excel or JSON directly from the Minexa interface, or you can work with the checkpoint files the script saves at each iteration.

The scraper you trained on Shiprocket can be reused across any structurally similar page without modification. If Shiprocket updates its page layout, the scraper will return null values or an explicit mismatch error rather than silently extracting wrong data. Retraining takes the same few minutes as the original setup and produces a new scraper ID to update in your request body.

To get started, install the Minexa Chrome extension, navigate to your target Shiprocket page, and follow the steps above. The API documentation is available at minexa.stoplight.io if you need to reference specific request parameters.

For a related walkthrough covering a similar API-based extraction workflow, see: Scraping tax and accounting data from IDX using the Minexa API.