- class eradiate.data.SafeOnlineDataStore(base_url, path, registry_fname='registry.txt')[source]#
Serve files located online, with integrity check.
pooch.Pooch) – The Pooch instance used to manage downloaded content.
Path) – Path to the registry file, relative to path.
This class basically wraps a
- fetch(filename, downloader=None)[source]#
Fetch a file from the data store. This method wraps
pooch.Pooch.fetch()and automatically selects compressed files when they are available.
filename (path-like) – File name to fetch from the local storage, relative to the storage root.
callable(), optional) – A callable that will be called to download a given URL to a provided local file name. This is mostly useful to display progress bars during download.
Path) – Absolute path where the retrieved resource is located.
If a compressed resource exists, it will be served automatically. For instance, if
"foo.nc"is requested and
foo.nc.gzis registered, the latter will be downloaded, decompressed and served as
- is_registered(filename, allow_compressed=True)[source]#
Check if a file is registered, with an option to look for compressed data.
Path) – The file name which matched filename.
ValueError – If filename could not be matched with any entry in the registry.
Purge local storage location. The default behaviour is very aggressive and will wipe out the entire directory contents.
str, optional) – If set to
"registered", files in the registry, as well as the registry file itself, will not be deleted. Finer control is possible by passing a list of exclusion rules (paths relative to the store’s local storage root, shell wildcards allowed).
keep="registered"keeps registered files to minimize the amount of data to be downloaded upon future queries the the data store. This means, for instance, that if data is registered and downloaded as a compressed file, then served decompressed, the compressed file will be kept, while the decompressed file will be deleted.
This is a destructive operation, make sure you know what you’re doing!
Get the absolute path to the registry file and make sure that it is written to the local cache.
Get a list of registered files.
callable(), optional) – A filter function taking a file path as a single string argument and returning a Boolean. Filenames for which the filter returns
Truewill be returned.
Reload the registry file from the local cache.
bool, optional) – If
True, the existing registry file will be deleted and downloaded again.
- property path#
Absolute path to the local data storage folder.
- property registry_path#
Absolute path to the registry file.