BranchesTags

Repository files navigation

DataTracker

About

DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.

DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14. You can get a copy of the paper from VU Digital Archive Repository (VU-DARE). We also have a demo on YouTube. Presentation slides available upon request.

Requirements

DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:

  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.

Installation

After cloning DataTracker, follow these steps to compile it.

Multiarch setup (intel64 only): On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32. For this, you will need a working multiarch setup. Google and serverfault are your friends for this.

Build environment: On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.

Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:

make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:

export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:

make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.

Runnning

Capturing raw provenance

To capture provenance from a program, launch it from the unix shell using something like this:

./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:

  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.

Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:

./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.

Converting to PROV

The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:

python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance

For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.

Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax. If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.

Sample programs

In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:

make -C samples

About

DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.

Resources

Readme
License
Activity

Stars

95 stars

Watchers

9 watching

Forks

20 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
DataTracker
About
DataTracker is a tool for collecting high-fidelity data provenance from unmodified Linux programs. It is based on Intel PinDynamic Binary Instrumentation framework and libdftDynamic Taint Analysis library. The taint marks supported by the original libdft are of limited size and cannot provide adequate fidelity for use in provenance tracking. For this, DataTracker uses a modified version of the library developed at VU University Amsterdam.
DataTracker was developed at VU University Amsterdam by Manolis Stamatogiannakis and presented at IPAW14.
You can get a copy of the paper from VU Digital Archive Repository (VU-DARE).
We also have a demo on YouTube.
Presentation slides available upon request.
Requirements
DataTracker can work with 32bit Linux programs. This limitation is imposed by the current version of libdft. However, the methods of both software are not platform-specific. So, in principle, they can be ported on any platform supported by Intel Pin. The requirements for running DataTracker are:
  • A C++11 compiler and unix build utilities (e.g. GNU Make).
  • A recent (>=2.13) version of Intel Pin. The framework must be present in directory pin inside the DataTracker top directory.
  • A suitable version of the modified libdft - typically the latest available. This must be placed in directory support/libdft.
  • Python 2.7 for converting raw provenance to PROV format in Turtle syntax.
Installation
After cloning DataTracker, follow these steps to compile it.
Multiarch setup (intel64 only):
On intel64 (a.k.a. x86_64) hosts, DataTracker and libdft need to be cross-compiled to ia32.
For this, you will need a working multiarch setup.
Google and serverfault are your friends for this.
Build environment:
On Debian/Ubuntu systems, you should install build-essential meta-package which will provide a C++ compiler and GNU Make. On other systems, you should either install some equivalent meta-package or install the tools one by one using trial and error.
Intel Pin: You can manually download a suitable Pin version and extract it in pin directory. For convenience, a makefile is provided which takes care of this. I.e. it downloads and extracts a suitable Pin version. Invoke it using:
make -C support -f makefile.pin

libdft: The modified libdft is packed as a submodule of DataTracker. You need to disable Git's certificate checking to successfully retrieve it. Because libdft does not use Pin's makefile infrastructure you need to set PIN_ROOT environment variable before compiling it. E.g.:
export PIN_ROOT=$(pwd)/pin
GIT_SSL_NO_VERIFY=true git submodule update --init
make support-libdft

dtracker pin tool: Finaly compile the pin tool of DataTracker using:
make

If all above steps were successfull, obj-ia32/dtracker.so will be created. This is Pin tool containing all the instrumentation required to capture provenance.
Runnning
Capturing raw provenance
To capture provenance from a program, launch it from the unix shell using something like this:
./pin/pin.sh -follow_execv -t ./obj-ia32/dtracker.so <knobs> -- <program> <args>

The command runs the program under Pin
In addition to the standard Pin knobs, DataTracker additionally supports these tool-specific knobs:
  • -stdin [1|0]: Turns tracking of data read from the standard input on or off. Default if off.
  • -stdout [1|0]: Turns logging of provenance of data written to standard output on or off. Default if on.
  • -stderr [1|0]: Turns logging of provenance of data written to standard error on or off. Default if off.
Note that launching large programs using the method above takes a lot of time. For such programs, it is suggested to first launch the program and then attach DataTracker to the running process like this:
./pin/pin.sh -follow_execv -pid <pid> -t ./obj-ia32/dtracker.so <knobs>

The raw provenance generated by DataTracker is contained in file rawprov.out. Any additional debugging information are written in file pintool.log.
Converting to PROV
The raw2ttl.py script converts the raw provenance generated by DataTracker to PROV format in Turtle syntax. The converter works as a filter. So, a conversion would look like this:
python raw2ttl.py < rawprov.out > prov.ttl

Visualizing provenance
For visualization of the generated provenance, we suggest using provconvert from Luc Moreau's ProvToolbox. It is suggested to use the binary release.
Of course any other PROV-compatible tool can be used, either directly, or via conversion of the Turtle file to a supported syntax.
If you were able to produce any good-looking provenance graph, we'd love to incorporate them in these pages.
Sample programs
In this repository also include a few sample programs we used for evaluating the effectiveness of DataTracker. You can find these programs in the samples directory. To build them, use:
make -C samples

About
DataTracker: A Pin tool for collecting high-fidelity data provenance from unmodified programs.
Resources
Readme
License
Activity
Stars
95 stars
Watchers
9 watching
Forks
20 forks
Report repository
Releases
Packages
Used by
Contributors
Languages