How to scrape environmental data from AQICN using the Minexa API
- Minexa.ai
- 22 hours ago
- 3 min read
Air quality data is publicly available on AQICN, but getting it into a structured format that a pipeline can actually use is a different problem. The page renders pollutant readings dynamically, attribution strings span multiple lines, and each row carries embedded chart images encoded as Base64. None of that is accessible through a simple HTTP request.
This guide walks through how to extract that data programmatically using the Minexa API, covering the full workflow from scraper training to a running Python script.
The problem: dynamic rendering blocks standard extraction
AQICN city pages like aqicn.org/city/london/ load pollutant data through JavaScript. Fields like PM2.5, PM10, O3, NO2, SO2, CO, temperature, pressure, and humidity only appear after the page has fully rendered. Any extraction approach that fetches raw HTML without JavaScript execution returns an empty or incomplete dataset.
The Minexa API handles JavaScript rendering natively. By setting js_render: true in the scraping configuration, the page is fully executed before extraction begins, so all pollutant rows are present in the DOM when the scraper runs.
Step one: train the scraper using the Chrome extension
Before calling the API, a scraper needs to be trained once. Watch the full walkthrough below before going through the steps.
1. Open the Minexa Chrome extension on the AQICN London page.
2. Confirm you are on the right page and proceed through the extension flow.
3. Select the container holding the full pollutant data table. Minexa identifies all columns automatically.
4. Review the extracted columns and click through to generate the API request.
The problem: attribution and source data are buried in multi-line strings
Each pollutant row on AQICN carries two distinct attribution fields. The air_quality_measurement_info field contains a formatted multi-line string naming the monitoring agencies, such as UK-AIR (Defra) and the London Air Quality Network, along with a note that values are converted to the US EPA AQI standard. The air_quality_measurement_source field contains a separate attribution string referencing the original measurement platform and the unit conversion method applied.
These are not duplicates. They encode different provenance layers and are useful for data lineage tracking in environmental monitoring pipelines. Minexa extracts both as separate columns, preserving the full text without truncation.
Sample extracted data
Below is a representative sample from the extraction output, with internal prefixes removed for clarity:
[
{
"pollutant_id": "tr_pm25",
"numeric_data": "42",
"value_numeric": "59",
"value_short": "91",
"air_quality_measurement_info": "London PM25 measured by UK-AIR (Defra) and London Air Quality Network. Values converted to US EPA AQI standard.",
"current_data_id": "cur_pm25",
"max_parameter": "max_pm25"
},
{
"pollutant_id": "tr_o3",
"numeric_data": "18",
"value_numeric": "67",
"value_short": "67",
"air_quality_measurement_info": "London O3 (ozone) measured by UK-AIR (Defra) and London Air Quality Network.",
"current_data_id": "cur_o3",
"max_parameter": "max_o3"
}
]
The problem: multiple numeric values per row with different meanings
Each pollutant row exposes three distinct numeric fields: numeric_data holds the current AQI reading, value_numeric holds a second numeric value representing a different aggregation window, and value_short holds a third figure that reflects yet another time horizon. Without understanding which field maps to which measurement window, pipelines risk mixing current readings with historical aggregates.
Minexa extracts all three as separate named columns, so downstream filtering and time-series alignment can be applied correctly without guessing which value is which.
Running the extraction via the API
Once the scraper is trained and you have the scraper_id, the extraction runs through a standard POST request to https://api.minexa.ai/data/.
data = {
"batches": [{
"scraper_id": 6271,
"columns": ["top_40"],
"urls": ["https://aqicn.org/city/london/"],
"scraping": {
"js_render": True,
"timeout": 30,
"js_code": [
{"wait_time": 2},
{"page_init": True},
{"wait_time": 4}
],
"proxy": "verified",
"retry": 3
}
}],
"threads": 3
}
The columns: ["top_40"] parameter returns the 40 highest-ranked fields Minexa identified during training. You can replace this with an explicit list of column names if you only need specific fields like numeric_data, pollutant_id, or air_quality_measurement_info.
The ready-to-run Python script from the extension handles pagination, writes checkpoint files as JSON and Excel after each iteration, and exits cleanly when the job finishes. Copy it directly from the extension after clicking 'API Request', update the scraper_id and URL list, and run it.
The same scraper works across any structurally similar AQICN city page. Swap the city slug in the URL and the extraction produces the same column structure, making it straightforward to build a multi-city air quality dataset from a single trained scraper.
For more on building structured data pipelines from environmental and public data sources, see: Scraping tax and accounting data from IDX using the Minexa API.
Comments