How to scrape non-profits and NGOs data from Grants.gov using the Minexa API

Non-profits and NGOs tracking federal funding opportunities spend considerable time manually checking Grants.gov for new postings. Building a structured pipeline from simpler.grants.gov/search changes that entirely: every new opportunity, deadline, and award amount becomes queryable data your systems can act on automatically.

This guide walks through how to build that pipeline using the Minexa API, starting from scraper training in the Chrome extension through to a running Python extraction script.

https://www.youtube.com/watch?v=kb6kbi8AbgA

What you extract from Grants.gov

Each row returned from the search listing contains the fields your pipeline needs: grant title, opportunity number, posting agency, open and close dates, award ceiling, and current status. Here is a sample of what the Minexa API returns:

[{"opportunity_number": "HHS-2025-ACF-OCS-EE-0172","grant_title": "Community Services Block Grant","agency_name": "Dept. of Health and Human Services","close_date": "2025-09-30","award_ceiling": "$500,000","status": "Posted"},{"opportunity_number": "USDA-NRCS-2025-001","grant_title": "Rural Energy for America Program","agency_name": "Dept. of Agriculture","close_date": "2025-08-15","award_ceiling": "$250,000","status": "Forecasted"}]

Training the scraper: step by step

Open the Minexa Chrome extension and navigate to simpler.grants.gov/search. The extension detects the page automatically.

Click I'm on the right page to confirm the target URL. The extension then scans for pagination patterns on the listing.

The pagination detection screen lists the navigation elements found. Click Continue to proceed.

Choose between scraping the listing only or the listing plus linked detail pages. For a grant monitoring feed, the listing view captures the fields you need without fetching each detail page separately.

Select Simple scraping on the next screen, then hover over the grant listing container and click to select it. Minexa identifies all data points within that container automatically.

After clicking Create Scraper, all extracted columns appear with navigation to review each field.

Click API Request in the top right. The extension generates ready-to-run Python code with your scraper_id pre-filled. Copy it directly.

API request structure

The core request body targets your trained scraper and passes the Grants.gov search URLs you want processed:

data = {"batches": [{"scraper_id": 6231,"columns": ["top_30"],"urls": ["https://simpler.grants.gov/search"],"scraping": {"js_render": True,"timeout": 30,"js_code": [{"wait_time": 2},{"page_init": True},{"wait_time": 4}],"proxy": "verified","retry": 3}}],"threads": 5}

Because Grants.gov is JavaScript-rendered, keep js_render: true. Pagination across result pages requires a JS code scenario defining the next-page click — this is not handled automatically when using the API. You will need to define that interaction explicitly in your js_code array or manage URL iteration in your calling script.

Running the pipeline

The Python script the extension generates polls the Minexa API, writes checkpoint files as JSON and Excel after each response batch, and continues until all pages are processed. For a recurring grant monitoring feed, set up a cron job that passes updated search URLs on your preferred schedule. The scraper_id stays the same across every run as long as the Grants.gov page structure does not change.

If the page layout changes and the scraper starts returning null values, retrain by selecting the updated container in the extension. This generates a new scraper_id to update in your script.

Ready to set up the pipeline? Install the Minexa Chrome extension and train your first Grants.gov scraper in a few minutes.

For the full API reference and scraping configuration options, see the Minexa API docs.

Also worth reading: How to scrape government and public records data from GovTrack using Minexa.ai.