Building a dataset provider plugin#
So far, we’ve been eagerly loading datasets for Xpublish to serve, but this tends not to scale well between memory needs and slow startup. Xpublish plugins can also be Dataset Providers and handle loading of datasets on request.
This also allows organizations to quickly be able to adapt Xpublish to work in their own environment, rather than needing Xpublish to explicitly support it.
import xarray as xr
from requests import HTTPError
from xpublish import Plugin, Rest, hookimpl
class TutorialDataset(Plugin):
name: str = 'xarray-tutorial-dataset'
@hookimpl
def get_datasets(self):
return list(xr.tutorial.file_formats)
@hookimpl
def get_datatree(self, dataset_id: str, group: str):
# The xarray tutorial datasets are flat, so we only serve the root.
# Note: ``group`` must be a positional parameter (no default) — pluggy
# will not forward arguments that have defaults to the hookimpl.
if group:
return None
try:
ds = xr.tutorial.open_dataset(dataset_id)
except HTTPError:
return None
return xr.DataTree(dataset=ds)
rest = Rest({})
rest.register_plugin(TutorialDataset())
rest.serve()
With this plugin, Xpublish can serve the same datasets as we explictly defined and loaded in serving multiple datasets, as well as any others supported by xr.tutorial
The plugin implements xpublish.plugins.hooks.PluginSpec.get_datatree(),
which receives both dataset_id and a group path so it can serve hierarchical
data. The simpler get_dataset() hook
is also first-class — pick it for providers that only ever serve flat datasets.
See the DataTrees tutorial for the lazy-by-group pattern used
by Zarr/Icechunk-backed providers.
A provider can set the id Xpublish serves its datasets under, which is what plugins build their cache keys from. Versioning that id (e.g. when a store is re-written or a new model run lands) turns over the cached responses for the dataset and all of its groups at once. See dataset ids and cache keys.
Note
For more details on building dataset provider plugins, please see the plugin user guide