BranchesTags

Repository files navigation

XText

 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...

Tika provides all sorts of solid content conversion and parsing capabilities. But it is a developer's tool -- for those who want to get close to all the gory details. There is some amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.

XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform ConvertedDocument API class that provides the following benefits:

  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title"). Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem, contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON property sheet for the conversion. In this way you can easily carry the complete output of converting an individual file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not interfere with any NLP operations on the text.

The converter execution in the XText API class facilitates conversion tasks in a few ways:

  • Provides options for caching conversions to the filesystem (or not). In the filesystem, conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g., when a document is collected, record it; when a document is converted to text process it. Such extensions are useful for application streaming where you do not want a lot of File I/O.

Supported Document Conversions

Major file conversions supported include:

  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction

Major features added beyond Tika

Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure Metadata preservation: metadata about original and the conversion process are persisted with conversions XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the bare Tika library. Such features include:

  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document

Supported customizations:

  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.

Documentation

See Java API for details on metadata fields and API usage.

Usage

Running it, from a release try the example below.

 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build

 // Build, then make a distribution that is more easily distributed.
ant build
ant dist

Publish via Maven:

 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release

RELEASE NOTES

3.8 NOVA

  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases

v3.7 STEADY

  • Tika 2.9.x
  • Security patches

v3.6 TIKA MASALA

  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath, but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here

v3.5 NEW BEGINNING

  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates

v3.4 VAMP

  • Xponents Core API 3.4 updated.

v3.3 HOLLY JOLLY

  • Xponents Core API 3.3 updated.

v3.2 DEAD HEAT

  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.

v3.1.0 SUMMER SOLSTICE 2019

  • Tika v1.21
  • Xponents 3.1.0

v3.0.6 SUPERBOWL 2019

  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API

v3.0 INDEPENDENCE DAY 2018

  • OpenSextant Xponents 3.0 release

v2.10.7

  • Moved XText out of Xponents folder.

v2.6 through v2.10

  • Just keeping pace with the rest of Xponents releases

v2.5.1 SUMMER, 2014

  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.

v1.4 ST PATRICK's DAY, 2014

  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and date/time as formal metadata that can be retrieved later.

v1.0 ST PATRICK's DAY, 2013

  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.

About

Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.

Resources

Readme
Apache-2.0 license

Code of conduct

Code of conduct

Contributing

Contributing
Activity
Custom properties

Stars

4 stars

Watchers

7 watching

Forks

1 fork
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
XText
 Author: Marc. C. Ubaldino, MITRE Corporation
Date: 2013-March
Updated: 2021-April, 2022-Oct
Copyright MITRE Corporation, 2012-2021

Apache Tika is awesome, but...
Tika provides all sorts of solid content conversion and parsing capabilities. But it is a
developer's tool -- for those who want to get close to all the gory details. There is some
amount of simplification and readiness for operationalizing things like Tika for more direct use in pipelines.
XText wraps around Tika APIs and other parser APIs with the objective of providing a uniform
ConvertedDocument API class that provides the following benefits:
  • streamlined meta-data model, using a simple Properties map and fewer fields, e.g., getProperty("title").
Other stock fields are pub_date, filepath, filesize, author, conversion_date, url
  • simple and obvious "text" buffer
  • serialization of "plain text" + "metadata" output in a single text file. An XText output saved to the filesystem,
contains the textual content followed by an XTEXT:<payload> footer where the payload is an encoded JSON
property sheet for the conversion. In this way you can easily carry the complete output of converting an individual
file in one-to-one fashion for most files. The XTEXT footer remains at the bottom of the file and should not
interfere with any NLP operations on the text.
The converter execution in the XText API class facilitates conversion tasks in a few ways:
  • Provides options for caching conversions to the filesystem (or not). In the filesystem,
conversions can be archived alongside archive or maintained in a parallel folder hierarchy
  • Provides a solution for managing unpacking and converting compound file types such as ZIP, TAR, Email and
other containers.
  • Provides ConversionListener and CollectionListener classes to extend XText for pipeline integration, e.g.,
when a document is collected, record it; when a document is converted to text process it. Such extensions
are useful for application streaming where you do not want a lot of File I/O.
Supported Document Conversions
Major file conversions supported include:
  • Anything Tika can do. By default the Tika AutoDetectParser is employed.
  • Email message archiving (traverse RFC822 MIME message, saving attachments, etc) and traversal, conversion, etc.
  • Limited web crawl and archiving crawls
  • JPEG EXIF parsing (saves full EXIF header as text; EXIF location & time as metadata)
  • Support for Embedded Object extraction
Major features added beyond Tika
Conversion caching/archiving: conversions can be maintained close to originals or in parallel structure
Metadata preservation: metadata about original and the conversion process are persisted with conversions
XText adds some typical conventions for integrators who wish to use a document conversion tool rather than the
bare Tika library. Such features include:
  • document file type filters
  • logging and metrics
  • input/output options for saving converted documents and related metadata
  • lightweight listener design so you can unpack, convert and process all in the same loop
  • formalizing the document meta-data practices: that is, what metadata is really important and how do we store it with the converted document
Supported customizations:
  • PDF metadata harvesting (from Tika/PDFBox); Detecting of encrypted PDFs
  • Web content scrapping; Default HTML parser is Tika's, but for web articles, Boilerplate parser is better.
  • Decomposing and extracting text from compound documents
  • Content is normalized to UTF-8 with unix line endings ('\n') only.
  • Java Documentation contains what you need to know for development.
Documentation
See Java API for details on metadata fields and API usage.
Usage
Running it, from a release try the example below.
 ./script/README_convert.txt
./script/convert.sh or convert.bat script
USAGE: ./script/convert.sh -input FOLDER -output FOLDER [ other options ]
ANT: ant -f ./script/xtext-ant.xml -Dinputfile=./test/somestuff/ convert
An example:
mkdir /tmp/conversions/
cd ./XText-3.8/
./script/convert.sh --input ./test --output /tmp/conversions

Build
 // Build, then make a distribution that is more easily distributed.
ant build
ant dist 
Publish via Maven:
 // // Fix all versions to be release versions.
// Ensure GPG key is known...
// and OSSRH login is setin settings.xml
mvn clean deploy -P release
RELEASE NOTES
3.8 NOVA
  • Tika 3.2 -- did not break much.
  • Apache Commons libraries and Maven plugins patched; Some security issues noted -- mainly cited by Tika releases
v3.7 STEADY
  • Tika 2.9.x
  • Security patches
v3.6 TIKA MASALA
  • Tika 2.5+ upgrade: Deprecated API uses removed; Tika dependencies and namespace resolved
  • Log4J usage: Certain packages rely on Log4J for compiled code (POI, Tika, etc). So keep Log4J API/Core in classpath,
but in separate classpath such that SLF4J finds the logger implementation desired. Logback is used here
v3.5 NEW BEGINNING
  • Xponents v3.5.4 series released for major updates on versions and Log4J resolution
  • Tika 1.28 and PDFBox 2.0.25 are upgraded here, in part due to Log4J updates
v3.4 VAMP
  • Xponents Core API 3.4 updated.
v3.3 HOLLY JOLLY
  • Xponents Core API 3.3 updated.
v3.2 DEAD HEAT
  • Xponents core API 3.2 streamlines dependencies -- No solr needed here.
v3.1.0 SUMMER SOLSTICE 2019
  • Tika v1.21
  • Xponents 3.1.0
v3.0.6 SUPERBOWL 2019
  • Tika 1.19+
  • Jodd JSON library 5.x
  • bug fix in ConvertedDocument properties API
v3.0 INDEPENDENCE DAY 2018
  • OpenSextant Xponents 3.0 release
v2.10.7
  • Moved XText out of Xponents folder.
v2.6 through v2.10
  • Just keeping pace with the rest of Xponents releases
v2.5.1 SUMMER, 2014
  • PDFBox updated
  • JavaDoc improvements, looking to Java 8 stringent javadoc checking
  • Added Outlook PST support (initial). via java-libpst. This support is planned for Tika 1.6.
  • PathManager construct added to offload complexities of dealing with caching, crawling, collecting.
v1.4 ST PATRICK's DAY, 2014
  • Added Tika 1.5 as primary conversion tool
  • Introduced content collectors: Email, web, Sharepoint
  • Added MessageConverter for email traversal, conversion and archiving.
  • Added OLEConverter to support MS object conversion, e.g. Outlook message files (untested)
  • Added ImageConverter which saves full EXIF header as text and preserves interesting GPS location and
date/time as formal metadata that can be retrieved later.
v1.0 ST PATRICK's DAY, 2013
  • initial design
  • added Testing archive -- not released; UBL Letters from SOCOM where released Fall 2012.
    
They are PDFs and Word docs in English and Arabic. They offer a good test opportunity.
About
Textual Content extraction from multimedia featuring modes for crawling folders, websites and Sharepoint. This is Tika-based, but aims to simplify this for pipelining in extraction applications.
Resources
Readme
Apache-2.0 license
Code of conduct
Code of conduct
Contributing
Contributing
Activity
Custom properties
Stars
4 stars
Watchers
7 watching
Forks
1 fork
Report repository
Releases
Packages
Used by
Contributors
Languages