How to scrape clinical trials data from Fred Hutch using the Minexa API

Clinical trials data is one of the most structurally rich and time-sensitive categories of public research information available on the web. Fred Hutch Cancer Center publishes a continuously updated directory of active trials at fredhutch.org, covering oncology, hematology, behavioral interventions, and surgical comparisons across phase I through phase III studies. Extracting that data programmatically opens up a range of research intelligence workflows that are difficult to maintain manually.

This guide walks through how to pull structured clinical trial listings from Fred Hutch using the Minexa API. The workflow has two phases: train a scraper once using the Minexa Chrome extension, then call the API to extract data at scale across as many pages as needed.

https://www.youtube.com/watch?v=ILaRpojIz3o

Phase 1: Training the scraper in the extension

Open the Fred Hutch clinical trials page in Chrome with the Minexa extension active. The extension detects the repeating trial card structure automatically.

Once the page loads, open the extension popup and confirm you are on the correct page. This tells Minexa to begin analyzing the page structure and identifying the pagination pattern used by the trial directory.

The extension surfaces the pagination logic it detected. Fred Hutch uses a URL parameter-based pagination system. Review the detected pages and click Continue to proceed.

After validating pagination, you will see options for how to approach the scraping job. You can choose to extract list-level data only, or instruct Minexa to follow each trial link and pull detail page content as well.

Select simple or advanced scraping mode depending on how granular you need the field extraction to be. For most clinical trial pipeline use cases, simple mode captures everything needed.

Click on the trial card container to highlight it. Minexa locks onto that DOM element and extracts all data points it finds within the repeating structure.

All extracted fields appear in the review panel. You can inspect each column, verify the values, and confirm the scraper configuration before moving to the API step.

Click the API request button to generate your scraper configuration. This screen provides the scraper ID you will use in every subsequent API call, along with ready-to-run Python code.

Get started with Minexa to train your first scraper

Phase 2: Calling the Minexa API

Once the scraper is trained, you call the Minexa API with your scraper ID and the list of URLs you want to process. Here is a working Python example:

import requests

url = "https://api.minexa.ai/data"
headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer YOUR_API_KEY"
}
payload = {
    "scraper_id": 6231,
    "urls": [
        "https://www.fredhutch.org/en/research/clinical-trials.html"
    ],
    "columns": "top_20"
}
response = requests.post(url, json=payload, headers=headers)
print(response.json())

The columns parameter accepts either a top_N shorthand to return the highest-ranked fields automatically, or a named list of specific columns if you only need a subset. For pagination across multiple pages, pass each paginated URL explicitly in the urls array and set up your own scheduling logic or cron job to run this at whatever cadence your pipeline requires.

What the extracted data looks like

Each trial listing returns a structured record. Below are two example records from the Fred Hutch clinical trials directory:

[
  {
    "study_title": "Group Retreat Psilocybin Therapy for the Treatment of Anxiety and Depression in Patients With Metastatic Solid Tumors or Incurable Hematologic Malignancies",
    "study_type": "Clinical Trial",
    "study_description": "A Phase 2b Study of Group Retreat Psilocybin Therapy for Cancer-Related Anxiety and Depression",
    "investigator_info": "Trial phase: II",
    "additional_investigators": "Investigator: Anthony Back",
    "trial_link": "https://www.fredhutch.org/en/research/clinical-trials/trial-details.fh_trial_id_15686..."
  },
  {
    "study_title": "A Study of IDRX-42 (GSK6042981) Versus Sunitinib in Participants With Gastrointestinal Stromal Tumors After Imatinib Therapy",
    "study_type": "Clinical Trial",
    "study_description": "A Phase 3, Randomized, Multicenter, Open-Label Study... (StrateGIST 3)",
    "investigator_info": "Trial phase: III",
    "additional_investigators": "Investigator: Min Park",
    "trial_link": "https://www.fredhutch.org/en/research/clinical-trials/trial-details.fh_trial_id_15678..."
  }
]

The clinical_trial_details field deserves particular attention. It returns a structured array of typed objects per listing card. Each object carries a tag name, an attribute type, and a value. This means a single field encodes the trial type label, the full display title, the detail page href as an attribute value, the complete protocol description, the phase string, the lead investigator name, and the action button text. You can traverse this array to reconstruct the full trial card without making any additional requests.

The trial_phase_info field surfaces a CSS class string rather than a plain text value. Listings where the phase is rendered as a standalone labeled element carry a different class than listings where the phase appears inline within the title block. This distinction is useful for post-processing: it lets you identify which records have an explicitly labeled phase versus those where the phase is embedded in the protocol description string.

Read the full Minexa API documentation

Scenario: building a phase-segmented trial tracker

A common use case for this dataset is building a tracker that segments active Fred Hutch trials by phase. Using the investigator_info field when it contains a phase value, combined with the trial_phase_info CSS class to identify which rendering pattern applies, you can filter records into phase I, II, III, and N/A buckets programmatically. Pairing this with the trial_link field gives you a direct URL per record to pull detail page content in a follow-up extraction run. Trials with no phase designation, such as expanded access or retrospective studies, surface with a phase value of N/A and can be routed to a separate pipeline branch.

The additional_investigators field adds another dimension: it captures co-investigator names as a plain string per listing when a second investigator is listed. This field is empty on single-investigator trials, making it straightforward to filter multi-investigator studies for collaboration network analysis.

Completing the job and exporting results

Once the API returns results, data is available as JSON. If you prefer a tabular format, the Minexa dashboard also supports export to Excel and Google Sheets directly from the job results view. For ongoing monitoring of the Fred Hutch trial directory, set up a cron job to call the API on whatever schedule fits your update requirements and store each run incrementally.

View Minexa plans and get API access

For a related extraction walkthrough covering another academic research data source, see: Scraping tax and accounting data from IDX using the Minexa API.