You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
kweekly edited this page Jul 27, 2012
·
2 revisions
Synopsis
A source describes to the system a repository of data streams from which it can retrieve data. Sources are distinguished by their source name, which is mapped to a corresponding .src file in the sources/ directory of the python repository. In general, all a source needs to define is a type, such as TIMESERIES, and a driver which knows how to retrieve data from that source.
Each specific driver may require additional fields to tell it where to get data, for instance, the SMAP driver needs to know the base url from which to retrieve the data.
A source ID is a string distinguishing a single data stream to the driver.
Known Sources
Below are a list of the provided sources (hopefully updated with changes). These can also be obtained by running fetcher.py list.
csv
Type
Time Series
source ID meaning
Filename of CSV file to read from. This file must contain the time points that are requested, i.e. by `--to` and `--from`, otherwise it will be truncated to the data available.
This source is mainly for testing, i.e. if you already have a CSV file, or you want to run a partial flow on an intermediate result.
smapcsv
This is the same as above, but it assumes that time and value units are in millis-. Use this source if pulling from raw-data CSVs from SMAP.
cosm
Type
Time Series
source ID meaning
The last part of the url to retrieve data via the curl API. e.g. "504/datastreams/energy"