HDX pipeline recommendations

Read more for HDX data standards and naming conventions

When you build a pipeline that publishes data to HDX, a few shared recommendations keep datasets consistent and discoverable. This page summarizes transformations best practices, and ideal conventions to name and describe datasets and resources.

These conventions apply to new pipelines and to existing ones that are actively being updated. These are recommendations for any developer or data contributor adding data programmatically through the HDX Python API or even manual uploads through the UI, and they're written with the assumption that contributors will use the HDX Python API wrapper.

Preparing source data

Before applying any transformations, normalize raw source data to a consistent structure:

• Remove blank rows before processing.
• Remove merged cells from Excel sources before reading them. Merged cells cause misaligned data when parsed.
• Make sure column names are in the first row. If a source file has metadata rows, titles, or blank rows above the header, strip them before parsing.

We deprecated the Humanitarian Exchange Language (HXL) standard in early 2026. Read our blog here for more information.

Standard transformations

The transformations below apply broadly across HDX data. Other transformations like filtering, merging, deduplication, aggregation, and type casting are applied selectively depending on the source format and the output you need.

Dates and times

• Store date-only values in YYYY-MM-DD format.
• For datetime values, include the time in UTC. Use a time component for event-based data; a date is enough for data that updates daily or less often.
• Store year-only and year-month values as full date ranges, not single dates. For example, store year-only 2026 as 2026-01-01 to 2026-12-31, and year-month July 2026 as 2026-07-01 to 2026-07-31. Use parse_date_range() from hdx.utilities.dateparse to convert partial dates into these start/end pairs.
• Always store start and end dates in separate columns. Use a start_date/end_date pair when period boundaries matter, and a single date column when one reference point is enough.

Locations and administrative areas

• Normalize country identifiers to ISO3. Use the Country class from HDX Python Country for country code lookup, validation, and name-to-ISO3 resolution. It draws on the Countries and Territories dataset on HDX, which covers name variants, aliases, and non-English names.
• For subnational data, use p-codes (admin1_pcode, admin2_pcode, and so on) as administrative identifiers, validated against the Global P-Codes dataset on HDX. Keep source admin names alongside the p-codes, but don't use the names as the primary key.
• Where p-codes are added, note the p-code version in the dataset's Caveats or Methodology metadata field.

String cleaning

• Strip leading and trailing whitespace from all string fields before writing output.
• Represent missing values as empty cells in CSV output (nothing between the commas). Frictionless, which underlies the tabular reading in HDX Python Utilities, returns empty cells as None in Python regardless of field type. Don't write None, null, or empty strings explicitly.
• For categorical fields such as country names or indicator codes, normalize to a consistent case before lookup or comparison, matching the convention of the target field. For example, ISO3 codes are uppercase.
• Leave non-ASCII characters as they are. Encode CSV output as UTF-8-sig.

Output formats

Choose the format that best matches your data type.

Format	When to use
CSV	Most common for tabular data.
GeoJSON / Shapefile	Geospatial data with geometry. GeoJSON for interoperability; Shapefile for compatibility with legacy GIS tools.
GeoPackage	Best performance when using QGIS.
File Geodatabase	Best performance when using ArcGIS. For large datasets, produce zipped files roughly 2–3x smaller than the next alternative.
Parquet / GeoParquet	Large tabular datasets where file size and query performance matter. Use the geoparquet-io library with a v2.0 profile.
JSON	Nested data that doesn't fit a flat CSV structure. Document the schema.
Excel	Only when explicitly required. Output plain tabular data meaning no merged cells, colors, or formatting.

File size

The file size limit per resource is 300 MB. If a dataset needs a file larger than this, contact the HDX team at [e-mail].

Linking to external resources

Uploading files to HDX is the most common approach, but some organizations prefer to link directly to data hosted elsewhere. To do this, create the resource with a url field pointing to the external location instead of uploading a file. The HDX Python API sets url_type and resource_type automatically.

The minimum fields for an external resource are:

resource = {
    "name": "my_file.tif",
    "description": "Brief description of the resource",
    "url": "https://example.com/data.tif",
    "format": "GeoTIFF",
}
dataset.add_update_resources([resource])

Naming conventions

Consistent naming improves discoverability, keeps the platform predictable, and helps datasets work together. Follow these conventions when naming and describing datasets, resources, and files.

Dataset title

The dataset title is the human-readable name shown on the dataset page, in search results, in HDX listings, and in external search engines.

• Start country-specific titles with the country name followed by a hyphen. For datasets that aren't country-specific, begin with the broadest meaningful scope. For example "Global" or the relevant region name.
• Use title case.
• Don't include dates or version numbers.
• Avoid organization names in titles; the data partner is already identified by its logo and metadata on the page. Exception: if the organization name is part of a recognized product or series (for example "WFP HungerMap" or "ACLED Conflict Data"), keep it as it identifies the product, not just the contributor.
• Avoid the word "data" as it's implicit.

Pattern	Example
[Country] - [Description]	Somalia - Acute Food Insecurity
[Region] - [Description]	Sahel - Humanitarian Needs Overview
Global - [Description]	Global - Requirements and Funding

Dataset name

The dataset name is the HDX URL slug for the dataset page. Use lowercase with hyphens as separators. Where the name includes a location, use the ISO3 code rather than the full name. This keeps slugs short and avoids ambiguity between name variants.

Dataset title	Dataset name
Somalia - Acute Food Insecurity	som-acute-food-insecurity

Dataset description

The dataset description briefly summarizes what the dataset contains.

• Keep it concise in one to two paragraphs.
• Use plain language. Describe the content; leave out methodology.
• Leave the time period and geographic area to the metadata fields.
• Include useful links where relevant, such as a partner survey.

Example: "This dataset contains data obtained from the HDX Humanitarian API (HDX HAPI), which provides standardized humanitarian indicators designed for seamless interoperability from multiple sources. The data facilitates automated workflows and visualizations to support humanitarian decision making. For more information, please see the HDX HAPI landing page and documentation."

Dataset metadata

• Time Period: Set the start and end dates to match the actual date range of the data.
• Update Frequency: Reflect how often the pipeline refreshes the data. Avoid "As needed" and "Live"; use a specific frequency such as "Every day", "Every week", or "Every month" where possible.

Resource name and filename

Use the same value for both the resource name and the filename.

• Use lowercase with underscores as separators to be machine-readable.
• For country-specific files, begin with the ISO3 code.
• For organization-specific files, begin with the organisation acronym.
• Use global in place of an ISO3 code for files covering all countries.
• Don't include dates or version numbers, unless the pipeline maintains a versioned archive, or the date is part of the identity of a specific event. For event-based data where multiple instances of the same event type can occur in the same place (for example, disaster alerts), a date suffix may be needed to avoid filename conflicts.
• Avoid the word "data" as it's implicit.

Pattern	Example
[iso3]_[org]_[description].[ext]	yem_wfp_hungermap.csv
global_[org]_[description].[ext]	global_fts_requirements_funding.csv
[org]_[description].[ext]	wfp_hungermap.csv
[iso3]_[org]_[description]_[YYYYMMDD].[ext]	yem_wfp_adam_20230206.csv

Resource descriptions

Write one to three sentences describing what the file contains and any qualifiers that set it apart from other resources in the same dataset. Include the reference period and the geographic or administrative level when relevant.

• If a dataset has multiple resources, make the differences between them clear in each description.
• Spell out acronyms on first use, with the abbreviation in parentheses.

Example: "United States Virgin Islands carbon dioxide equivalent (CO2e) emissions on a 20-year time horizon, over the past two years, at the source level."

Need help?

If you have questions about building a pipeline for HDX or applying these standards, contact the HDX team at [e-mail].