WBData.jl

A Julia client for the World Bank Open Data API: paginated queries as tidy DataFrames, plus cached metadata and local search.

View on GitHub

Install

] add https://github.com/angrahed/WBData.jl

Usage

using WBData


# Real countries (default), 1960 to 2023, joined with country metadata.
df = WBData.data("SP.POP.TOTL"; date = 1960:2023)

README

Source on GitHub

A Julia client for the World Bank Open Data API. It does for Julia roughly what R's wbstats does, and it's meant to replace WorldBankData.jl: full coverage of the API, tidy DataFrame returns, no hidden row caps, and a test suite that runs without touching the network.

Why not WorldBankData.jl?

WorldBankData.jl doesn't paginate. It sets per_page once and throws away anything that doesn't fit in a single response, which caps a query at about 25,000 rows. WBData.jl reads the page count from the response metadata and fetches the remaining pages at the same time. Ask for population across every country and aggregate from 1960 to today (around 16,000 rows) and all of it comes back, with nothing quietly dropped.

Installation

] add https://github.com/angrahed/WBData.jl

Quick start

using WBData

# Real countries (default), 1960 to 2023, joined with country metadata.
df = WBData.data("SP.POP.TOTL"; date = 1960:2023)

# Include aggregates (WLD, EUU, HIC, …):
df = WBData.data("SP.POP.TOTL"; country = :all, date = 1960:2023)

# Multiple indicators are fetched concurrently.
df = WBData.data(["SP.POP.TOTL", "NY.GDP.MKTP.CD"];
                 country = ["USA", "GBR", "FRA"],
                 date = 2000:2023)

# Most recent 5 values per series.
df = WBData.data("NY.GDP.MKTP.CD"; mrv = 5)

# Quarterly data: freq sets both the date format and the URL.
df = WBData.data("DPANUSSPB"; country = "USA", date = 2010:2020, freq = :quarterly)

# Wide layout: one column per indicator id, indexed by (iso3c, iso2c, country, date).
dfw = WBData.data(["SP.POP.TOTL", "NY.GDP.MKTP.CD"];
                  country = "USA", date = 2010:2023, wide = true)

The default long-format return has columns (iso3c, iso2c, country, indicator_id, indicator, date, value, unit, obs_status, last_updated) (plus the enrichment columns region, income_level, lending_type, and capital_city when enrich=true, and country_id, which carries the World Bank internal code that sometimes differs from the real ISO-2).

Metadata

WBData.indicators()      # ~17K rows, cached for 7 days
WBData.countries()
WBData.topics()
WBData.sources()
WBData.regions()
WBData.income_levels()
WBData.lending_types()
WBData.languages()

Every metadata call caches to disk through Scratch.jl. The default TTL is 7 days; change it with WBData.set_cache_ttl!(Day(1)).

WBData.search_indicators("gdp per capita")
WBData.search_indicators(r"^SP\.POP")           # regex
WBData.search_indicators("school"; topic = 4)   # filter by topic id

WBData.search_countries("franc")

Search runs locally over the cached metadata, so it makes no extra API calls.

Configuration

WBData.set_default_language!("es")
WBData.set_user_agent!("MyProject/1.0 (me@example.com)")
WBData.set_timeout!(60)

These persist across sessions through Preferences.jl.

Cache management

WBData.cache_info()                # what is cached, plus age and size
WBData.clear_cache!()              # delete all metadata caches
WBData.clear_cache!("indicators_en")
WBData.set_cache_ttl!(Day(14))

Only metadata is cached; observations never are. Indicator and country listings rarely change, so a TTL of a few days is usually fine.

Errors

Errors are typed, and the API's habit of returning HTTP 200 with the error buried in the JSON body never reaches you as a surprise:

  • WBData.WBAPIError when the API returns an error payload.
  • WBData.WBNotFound when a metadata lookup matches nothing.
  • WBData.WBNetworkError when the retries run out (3 attempts, exponential backoff).
  • WBData.WBParseError when the response isn't the JSON we expected.

Migrating from WorldBankData.jl

WorldBankData.jl WBData.jl
wdi("SP.POP.TOTL", "US") WBData.data("SP.POP.TOTL"; country = "USA")
wdi(["A","B"], "all", 2000, 2020) WBData.data(["A","B"]; date = 2000:2020)
search_wdi("indicators", "name", r"...") WBData.search_indicators(r"...")
(no equivalent) WBData.data(...; mrv = 5)
(no equivalent) WBData.data(...; gapfill = true)
(no equivalent) WBData.data(...; freq = :quarterly)

Development

julia --project=. -e 'using Pkg; Pkg.instantiate()'
julia --project=. -e 'using Pkg; Pkg.test()'

Tests are mocked end to end, so they need no network. Set ENV["WBDATA_INTEGRATION"] = "1" to also run the optional live-API checks.

License

MIT.

Esc
↑↓ move ↵ open