TensorFlow Datasets

TensorFlow Datasets provides many public datasets as tf.data.Datasets.

• List of datasets
• Try it in Colab
• API docs
• Guides
  • Overview
  • Datasets versioning
  • Using splits and slicing API
  • Add a dataset
  • Add a huge dataset (>>100GiB)

Table of Contents

• Installation
• Usage
• DatasetBuilder
• NumPy usage
• Want a certain dataset?
• Disclaimers

Installation

pip install tensorflow-datasets
# Requires TF 1.15+ to be installed.# Some datasets require additional libraries; see setup.py extras_require
pip install tensorflow
# or:
pip install tensorflow-gpu

Join our Google group to receive updates on the project.

Usage

importtensorflow_datasetsastfdsimporttensorflowastf# tfds works in both Eager and Graph modestf.compat.v1.enable_eager_execution()
# See available datasetsprint(tfds.list_builders())
# Construct a tf.data.Datasetds_train, ds_test=tfds.load(name="mnist", split=["train", "test"])
# Build your input pipelineds_train=ds_train.shuffle(1000).batch(128).prefetch(10)
forfeaturesinds_train.take(1):
image, label=features["image"], features["label"]

Try it interactively in a Colab notebook.

DatasetBuilder

All datasets are implemented as subclasses of DatasetBuilder and tfds.load is a thin convenience wrapper. DatasetInfo documents the dataset.

importtensorflow_datasetsastfds# The following is the equivalent of the `load` call above.# You can fetch the DatasetBuilder class by stringmnist_builder=tfds.builder("mnist")
# Download the datasetmnist_builder.download_and_prepare()
# Construct a tf.data.Datasetds=mnist_builder.as_dataset(split=tfds.Split.TRAIN)
# Get the `DatasetInfo` object, which contains useful information about the# dataset and its featuresinfo=mnist_builder.infoprint(info)
tfds.core.DatasetInfo(
name='mnist',
version=1.0.0,
description='The MNIST database of handwritten digits.',
urls=[u'http://yann.lecun.com/exdb/mnist/'],
features=FeaturesDict({
'image': Image(shape=(28, 28, 1), dtype=tf.uint8),
'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10)
},
total_num_examples=70000,
splits={
u'test': <tfds.core.SplitInfonum_examples=10000>,
u'train': <tfds.core.SplitInfonum_examples=60000>
},
supervised_keys=(u'image', u'label'),
citation='""" @article{lecun2010mnist, title={MNIST handwritten digit database}, author={LeCun, Yann and Cortes, Corinna and Burges, CJ}, journal={ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist}, volume={2}, year={2010} } """',
)

You can also get details about the classes (number of classes and their names).

info=tfds.builder('cats_vs_dogs').infoinfo.features['label'].num_classes# 2info.features['label'].names# ['cat', 'dog']info.features['label'].int2str(1) # "dog"info.features['label'].str2int('cat') # 0

NumPy Usage with tfds.as_numpy

As a convenience for users that want simple NumPy arrays in their programs, you can use tfds.as_numpy to return a generator that yields NumPy array records out of a tf.data.Dataset. This allows you to build high-performance input pipelines with tf.data but use whatever you'd like for your model components.

train_ds=tfds.load("mnist", split=tfds.Split.TRAIN)
train_ds=train_ds.shuffle(1024).batch(128).repeat(5).prefetch(10)
forexampleintfds.as_numpy(train_ds):
numpy_images, numpy_labels=example["image"], example["label"]

You can also use tfds.as_numpy in conjunction with batch_size=-1 to get the full dataset in NumPy arrays from the returned tf.Tensor object:

train_ds=tfds.load("mnist", split=tfds.Split.TRAIN, batch_size=-1)
numpy_ds=tfds.as_numpy(train_ds)
numpy_images, numpy_labels=numpy_ds["image"], numpy_ds["label"]

Note that the library still requires tensorflow as an internal dependency.

Want a certain dataset?

Adding a dataset is really straightforward by following our guide.

Request a dataset by opening a Dataset request GitHub issue.

And vote on the current set of requests by adding a thumbs-up reaction to the issue.

Disclaimers

This is a utility library that downloads and prepares public datasets. We donot host or distribute these datasets, vouch for their quality or fairness, orclaim that you have license to use the dataset. It is your responsibility todetermine whether you have permission to use the dataset under the dataset'slicense.

If you're a dataset owner and wish to update any part of it (description,citation, etc.), or do not want your dataset to be included in thislibrary, please get in touch through a GitHub issue. Thanks for yourcontribution to the ML community!

If you're interested in learning more about responsible AI practices, includingfairness, please see Google AI's Responsible AI Practices.

tensorflow/datasets is Apache 2.0 licensed. See the LICENSE file.