Skip to content
BranchesTags

Repository files navigation

TensorFlow Data Validation

PythonPyPIDocumentation

TensorFlow Data Validation (TFDV) is a library for exploring and validating machine learning data. It is designed to be highly scalable and to work well with TensorFlow and TensorFlow Extended (TFX).

TF Data Validation includes:

  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well as faceted comparison of pairs of features (Facets)
  • Automated data-schema generation to describe expectations about data like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies, such as missing features, out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and learn more in order to correct them.

For instructions on using TFDV, see the get started guide and try out the example notebook. Some of the techniques implemented in TFDV are described in a technical paper published in SysML'19.

Installing from PyPI

The recommended way to install TFDV is using the PyPI package:

pip install tensorflow-data-validation

Nightly Packages

TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on Google Cloud. To install the latest nightly package, please use the following command:

pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation

This will install the nightly packages for the major dependencies of TFDV such as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).

Build with Docker

This is the recommended way to build TFDV under Linux, and is continuously tested at Google.

1. Install Docker

Please first install docker and docker-compose by following the directions: docker; docker-compose.

2. Clone the TFDV repository

git clone https://github.com/tensorflow/data-validation
cd data-validation

Note that these instructions will install the latest master branch of TensorFlow Data Validation. If you want to install a specific branch (such as a release branch), pass -b <branchname> to the git clone command.

3. Build the pip package

Then, run the following at the project root:

sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010

where PYTHON_VERSION is one of {36, 37, 38}.

A wheel will be produced under dist/.

4. Install the pip package

pip install dist/*.whl

Build from source

1. Prerequisites

To compile and use TFDV, you need to set up some prerequisites.

Install NumPy

If NumPy is not installed on your system, install it now by following these directions.

Install Bazel

If Bazel is not installed on your system, install it now by following these directions.

2. Clone the TFDV repository

git clone https://github.com/tensorflow/data-validation
cd data-validation

Note that these instructions will install the latest master branch of TensorFlow Data Validation. If you want to install a specific branch (such as a release branch), pass -b <branchname> to the git clone command.

3. Build the pip package

TFDV wheel is Python version dependent -- to build the pip package that works for a specific Python version, use that Python binary to run:

python setup.py bdist_wheel

You can find the generated .whl file in the dist subdirectory.

4. Install the pip package

pip install dist/*.whl

Supported platforms

TFDV is tested on the following 64-bit operating systems:

  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.

Notable Dependencies

TensorFlow is required.

Apache Beam is required; it's the way that efficient distributed computation is supported. By default, Apache Beam runs in local mode but can also run in distributed mode using Google Cloud Dataflow and other Apache Beam runners.

Apache Arrow is also required. TFDV uses Arrow to represent data internally in order to make use of vectorized numpy functions.

Compatible versions

The following table shows the package versions that are compatible with each other. This is determined by our testing framework, but other untested combinations may also work.

tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a

Questions

Please direct any questions about working with TF Data Validation to Stack Overflow using the tensorflow-data-validation tag.

Links

About

Library for exploring and validating machine learning data

Resources

Readme
Apache-2.0 license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - jay90099/data-validation: Library for exploring and validating machine learning data · GitHub
Skip to content
BranchesTags
Repository files navigation
TensorFlow Data Validation
PythonPyPIDocumentation
TensorFlow Data Validation (TFDV) is a library for exploring and validating
machine learning data. It is designed to be highly scalable
and to work well with TensorFlow and TensorFlow Extended (TFX).
TF Data Validation includes:
  • Scalable calculation of summary statistics of training and test data.
  • Integration with a viewer for data distributions and statistics, as well
as faceted comparison of pairs of features (Facets)
  • Automated data-schema
generation to describe expectations about data
like required values, ranges, and vocabularies
  • A schema viewer to help you inspect the schema.
  • Anomaly detection to identify anomalies,
such as missing features,
out-of-range values, or wrong feature types, to name a few.
  • An anomalies viewer so that you can see what features have anomalies and
learn more in order to correct them.
For instructions on using TFDV, see the get started guide
and try out the example notebook.
Some of the techniques implemented in TFDV are described in a
technical paper published in SysML'19.
Installing from PyPI
The recommended way to install TFDV is using the
PyPI package:
pip install tensorflow-data-validation
Nightly Packages
TFDV also hosts nightly packages at https://pypi-nightly.tensorflow.org on
Google Cloud. To install the latest nightly package, please use the following
command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation
This will install the nightly packages for the major dependencies of TFDV such
as TFX Basic Shared Libraries (TFX-BSL) and TensorFlow Metadata (TFMD).
Build with Docker
This is the recommended way to build TFDV under Linux, and is continuously
tested at Google.
1. Install Docker
Please first install docker and docker-compose by following the directions:
docker;
docker-compose.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
Then, run the following at the project root:
sudo docker-compose build manylinux2010
sudo docker-compose run -e PYTHON_VERSION=${PYTHON_VERSION} manylinux2010
where PYTHON_VERSION is one of {36, 37, 38}.
A wheel will be produced under dist/.
4. Install the pip package
pip install dist/*.whl
Build from source
1. Prerequisites
To compile and use TFDV, you need to set up some prerequisites.
Install NumPy
If NumPy is not installed on your system, install it now by following these
directions.
Install Bazel
If Bazel is not installed on your system, install it now by following these
directions.
2. Clone the TFDV repository
git clone https://github.com/tensorflow/data-validation
cd data-validation
Note that these instructions will install the latest master branch of TensorFlow
Data Validation. If you want to install a specific branch (such as a release
branch), pass -b <branchname> to the git clone command.
3. Build the pip package
TFDV wheel is Python version dependent -- to build the pip package that
works for a specific Python version, use that Python binary to run:
python setup.py bdist_wheel
You can find the generated .whl file in the dist subdirectory.
4. Install the pip package
pip install dist/*.whl
Supported platforms
TFDV is tested on the following 64-bit operating systems:
  • macOS 10.14.6 (Mojave) or later.
  • Ubuntu 16.04 or later.
  • Windows 7 or later.
Notable Dependencies
TensorFlow is required.
Apache Beam is required; it's the way that efficient
distributed computation is supported. By default, Apache Beam runs in local
mode but can also run in distributed mode using
Google Cloud Dataflow and other Apache
Beam
runners.
Apache Arrow is also required. TFDV uses Arrow to
represent data internally in order to make use of vectorized numpy functions.
Compatible versions
The following table shows the package versions that are
compatible with each other. This is determined by our testing framework, but
other untested combinations may also work.
tensorflow-data-validationapache-beam[gcp]pyarrowtensorflowtensorflow-metadatatensorflow-transformtfx-bsl
GitHub master2.31.02.0.0nightly (1.x/2.x)1.2.0n/a1.2.0
1.2.02.31.02.0.01.15 / 2.51.2.0n/a1.2.0
1.1.12.29.02.0.01.15 / 2.51.1.0n/a1.1.1
1.1.02.29.02.0.01.15 / 2.51.1.0n/a1.1.0
1.0.02.29.02.0.01.15 / 2.51.0.0n/a1.0.0
0.30.02.28.02.0.01.15 / 2.40.30.0n/a0.30.0
0.29.02.28.02.0.01.15 / 2.40.29.0n/a0.29.0
0.28.02.28.02.0.01.15 / 2.40.28.0n/a0.28.1
0.27.02.27.02.0.01.15 / 2.40.27.0n/a0.27.0
0.26.12.28.00.17.01.15 / 2.30.26.00.26.00.26.0
0.26.02.25.00.17.01.15 / 2.30.26.00.26.00.26.0
0.25.02.25.00.17.01.15 / 2.30.25.00.25.00.25.0
0.24.12.24.00.17.01.15 / 2.30.24.00.24.10.24.1
0.24.02.23.00.17.01.15 / 2.30.24.00.24.00.24.0
0.23.12.24.00.17.01.15 / 2.30.23.00.23.00.23.0
0.23.02.23.00.17.01.15 / 2.30.23.00.23.00.23.0
0.22.22.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.12.20.00.16.01.15 / 2.20.22.00.22.00.22.1
0.22.02.20.00.16.01.15 / 2.20.22.00.22.00.22.0
0.21.52.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.42.17.00.15.01.15 / 2.10.21.00.21.10.21.3
0.21.22.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.12.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.21.02.17.00.15.01.15 / 2.10.21.00.21.00.21.0
0.15.02.16.00.14.01.15 / 2.00.15.00.15.00.15.0
0.14.12.14.00.14.01.140.14.00.14.0n/a
0.14.02.14.00.14.01.140.14.00.14.0n/a
0.13.12.11.0n/a1.130.12.10.13.0n/a
0.13.02.11.0n/a1.130.12.10.13.0n/a
0.12.02.10.0n/a1.120.12.10.12.0n/a
0.11.02.8.0n/a1.110.9.00.11.0n/a
0.9.02.6.0n/a1.9n/an/an/a
Questions
Please direct any questions about working with TF Data Validation to
Stack Overflow using the
tensorflow-data-validation
tag.
Links
About
Library for exploring and validating machine learning data
Resources
Readme
Apache-2.0 license
Contributing
Contributing
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages