> For the complete documentation index, see [llms.txt](https://docs.humdata.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.humdata.org/build/hdx-pipeline-recommendations.md).

# HDX pipeline recommendations

When you build a pipeline that publishes data to HDX, a few shared recommendations keep datasets consistent and discoverable. This page summarizes transformations best practices, and ideal conventions to name and describe datasets and resources.

These conventions apply to new pipelines and to existing ones that are actively being updated. These are recommendations for any developer or data contributor adding data programmatically through the [HDX Python API](https://hdx-python-api.readthedocs.io/en/latest/) or even manual uploads through the UI, and they're written with the assumption that contributors will use the HDX Python API wrapper.

### Preparing source data

Before applying any transformations, normalize raw source data to a consistent structure:

* Remove blank rows before processing.
* Remove merged cells from Excel sources before reading them. Merged cells cause misaligned data when parsed.
* Make sure column names are in the first row. If a source file has metadata rows, titles, or blank rows above the header, strip them before parsing.

*We deprecated the Humanitarian Exchange Language (HXL) standard in early 2026. Read our blog* [*here*](https://centre.humdata.org/retiring-hxl-services/) *for more information.*

### Standard transformations

The transformations below apply broadly across HDX data. Other transformations like filtering, merging, deduplication, aggregation, and type casting are applied selectively depending on the source format and the output you need.

#### Dates and times

* Store date-only values in `YYYY-MM-DD` format.
* For datetime values, include the time in UTC. Use a time component for event-based data; a date is enough for data that updates daily or less often.
* Store year-only and year-month values as full date ranges, not single dates. For example, store year-only `2026` as `2026-01-01` to `2026-12-31`, and year-month `July 2026` as `2026-07-01` to `2026-07-31`. Use `parse_date_range()` from[ hdx.utilities.dateparse](https://hdx-python-utilities.readthedocs.io/en/latest/src/hdx/utilities/dateparse/) to convert partial dates into these start/end pairs.
* Always store start and end dates in separate columns. Use a start\_date/end\_date pair when period boundaries matter, and a single date column when one reference point is enough.

#### Locations and administrative areas

* Normalize country identifiers to ISO3. Use the `Country` class from[ HDX Python Country](https://hdx-python-country.readthedocs.io/en/latest/src/hdx/location/country/#hdx.location.country.Country) for country code lookup, validation, and name-to-ISO3 resolution. It draws on the[ Countries and Territories dataset](https://data.humdata.org/dataset/countries-and-territories) on HDX, which covers name variants, aliases, and non-English names.
* For subnational data, use p-codes (`admin1_pcode`, `admin2_pcode`, and so on) as administrative identifiers, validated against the[ Global P-Codes dataset](https://data.humdata.org/dataset/global-pcodes) on HDX. Keep source admin names alongside the p-codes, but don't use the names as the primary key.
* Where p-codes are added, note the p-code version in the dataset's *Caveats* or *Methodology* metadata field.

#### String cleaning

* Strip leading and trailing whitespace from all string fields before writing output.
* Represent missing values as empty cells in CSV output (nothing between the commas).[ Frictionless](https://specs.frictionlessdata.io/), which underlies the tabular reading in HDX Python Utilities, returns empty cells as `None` in Python regardless of field type. Don't write `None`, `null`, or empty strings explicitly.
* For categorical fields such as country names or indicator codes, normalize to a consistent case before lookup or comparison, matching the convention of the target field. For example, ISO3 codes are uppercase.
* Leave non-ASCII characters as they are. Encode CSV output as `UTF-8-sig`.

### Output formats

Choose the format that best matches your data type.

| **Format**           | **When to use**                                                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| CSV                  | Most common for tabular data.                                                                                                                     |
| GeoJSON / Shapefile  | Geospatial data with geometry. GeoJSON for interoperability; Shapefile for compatibility with legacy GIS tools.                                   |
| GeoPackage           | Best performance when using QGIS.                                                                                                                 |
| File Geodatabase     | Best performance when using ArcGIS. For large datasets, produce zipped files roughly 2–3x smaller than the next alternative.                      |
| Parquet / GeoParquet | Large tabular datasets where file size and query performance matter. Use the[ geoparquet-io](https://geoparquet.io/) library with a v2.0 profile. |
| JSON                 | Nested data that doesn't fit a flat CSV structure. Document the schema.                                                                           |
| Excel                | Only when explicitly required. Output plain tabular data meaning no merged cells, colors, or formatting.                                          |

#### File size

The file size limit per resource is 300 MB. If a dataset needs a file larger than this, contact the HDX team at <hdx@un.org>.

#### Linking to external resources

Uploading files to HDX is the most common approach, but some organizations prefer to link directly to data hosted elsewhere. To do this, create the resource with a `url` field pointing to the external location instead of uploading a file. The HDX Python API sets `url_type` and `resource_type` automatically.

The minimum fields for an external resource are:

```
resource = {
    "name": "my_file.tif",
    "description": "Brief description of the resource",
    "url": "https://example.com/data.tif",
    "format": "GeoTIFF",
}
dataset.add_update_resources([resource])
```

### Naming conventions

Consistent naming improves discoverability, keeps the platform predictable, and helps datasets work together. Follow these conventions when naming and describing datasets, resources, and files.

#### Dataset title

The dataset title is the human-readable name shown on the dataset page, in search results, in HDX listings, and in external search engines.

* Start country-specific titles with the country name followed by a hyphen. For datasets that aren't country-specific, begin with the broadest meaningful scope. For example "Global" or the relevant region name.
* Use title case.
* Don't include dates or version numbers.
* Avoid organization names in titles; the data partner is already identified by its logo and metadata on the page. **Exception**: if the organization name is part of a recognized product or series (for example "WFP HungerMap" or "ACLED Conflict Data"), keep it as it identifies the product, not just the contributor.
* Avoid the word "data" as it's implicit.

| **Pattern**                 | **Example**                         |
| --------------------------- | ----------------------------------- |
| `[Country] - [Description]` | Somalia - Acute Food Insecurity     |
| `[Region] - [Description]`  | Sahel - Humanitarian Needs Overview |
| `Global - [Description]`    | Global - Requirements and Funding   |

#### Dataset name

The dataset name is the HDX URL slug for the dataset page. Use lowercase with hyphens as separators. Where the name includes a location, use the ISO3 code rather than the full name. This keeps slugs short and avoids ambiguity between name variants.

| **Dataset title**               | **Dataset name**          |
| ------------------------------- | ------------------------- |
| Somalia - Acute Food Insecurity | som-acute-food-insecurity |

#### Dataset description

The dataset description briefly summarizes what the dataset contains.

* Keep it concise in one to two paragraphs.
* Use plain language. Describe the content; leave out methodology.
* Leave the time period and geographic area to the metadata fields.
* Include useful links where relevant, such as a partner survey.

**Example:** "This dataset contains data obtained from the [HDX Humanitarian API](https://hapi.humdata.org/) (HDX HAPI), which provides standardized humanitarian indicators designed for seamless interoperability from multiple sources. The data facilitates automated workflows and visualizations to support humanitarian decision making. For more information, please see the HDX HAPI [landing page](https://data.humdata.org/hapi) and [documentation](https://hdx-hapi.readthedocs.io/en/latest/)."

#### Dataset metadata

* **Time Period:** Set the start and end dates to match the actual date range of the data.
* **Update Frequency:** Reflect how often the pipeline refreshes the data. Avoid "As needed" and "Live"; use a specific frequency such as "Every day", "Every week", or "Every month" where possible.

#### Resource name and filename

Use the same value for both the resource name and the filename.

* Use lowercase with underscores as separators to be machine-readable.
* For country-specific files, begin with the ISO3 code.
* For organization-specific files, begin with the organisation acronym.
* Use `global` in place of an ISO3 code for files covering all countries.
* Don't include dates or version numbers, unless the pipeline maintains a versioned archive, or the date is part of the identity of a specific event. For event-based data where multiple instances of the same event type can occur in the same place (for example, disaster alerts), a date suffix may be needed to avoid filename conflicts.
* Avoid the word "data" as it's implicit.

| **Pattern**                                   | **Example**                            |
| --------------------------------------------- | -------------------------------------- |
| `[iso3]_[org]_[description].[ext]`            | yem\_wfp\_hungermap.csv                |
| `global_[org]_[description].[ext]`            | global\_fts\_requirements\_funding.csv |
| `[org]_[description].[ext]`                   | wfp\_hungermap.csv                     |
| `[iso3]_[org]_[description]_[YYYYMMDD].[ext]` | yem\_wfp\_adam\_20230206.csv           |

#### Resource descriptions

Write one to three sentences describing what the file contains and any qualifiers that set it apart from other resources in the same dataset. Include the reference period and the geographic or administrative level when relevant.

* If a dataset has multiple resources, make the differences between them clear in each description.
* Spell out acronyms on first use, with the abbreviation in parentheses.

**Example:** "United States Virgin Islands carbon dioxide equivalent (CO2e) emissions on a 20-year time horizon, over the past two years, at the source level."

### Need help?

If you have questions about building a pipeline for HDX or applying these standards, contact the HDX team at <hdx@un.org>.

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.humdata.org/build/hdx-pipeline-recommendations.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.