fileconversion

This is a Go library to convert various file formats into plaintext and provide related useful functions.

This library is used for https://intelx.io and was successfully tested over 184 million individual files. It is partly written from scratch, partly forked from open source and partly a rewrite of existing code. Many existing libraries lack stability and functionality and this libraries solves that.

We welcome any contributions - please open issues for any feature requests, bugs, and other related issues.

It supports following file formats for plaintext conversion:

• Word: DOC, DOCX, RTF, ODT
• Excel: XLS, XLSX, ODS
• PowerPoint: PPTX
• PDF
• Ebook: EPUB, MOBI
• Website: HTML

Functions for compressed and container files:

• Decompress files: GZ, BZ, BZ2, XZ
• Extract files from containers: ZIP, RAR, 7Z, TAR

Picture related functions:

• Check if pictures are excessively large
• Compress (and convert) pictures to JPEG: GIF, JPEG, PNG, BMP, TIFF
• Resize and compress pictures
• Extract pictures from PDF files

To download this library:

go get -u github.com/IntelligenceX/fileconversion

And then use it like:

package main
import (
"bytes""fmt""os""github.com/IntelligenceX/fileconversion"
)
constsizeLimit=2*1024*1024// 2 MBfuncmain() {
// extract text from an XLSX filefile, err:=os.Open("Test.xlsx")
iferr!=nil {
fmt.Printf("Error opening file: %s\n", err)
return
}
deferfile.Close()
stat, _:=file.Stat()
buffer:=bytes.NewBuffer(make([]byte, 0, sizeLimit))
fileconversion.XLSX2Text(file, stat.Size(), buffer, sizeLimit, -1)
fmt.Println(buffer.String())
}

Functions

The package exports the following functions:

XLSX2Text(fileio.ReaderAt, sizeint64, writerio.Writer, limitint64, rowLimitint) (writtenint64, errerror)
DOCX2Text(fileio.ReaderAt, sizeint64) (string, error)
EPUB2Text(fileio.ReaderAt, sizeint64, limitint64) (string, error)
HTML2Text(readerio.Reader) (pageTextstring, errerror)
HTML2TextAndLinks(readerio.Reader, baseURLstring) (pageTextstring, links []string, errerror)
Mobi2Text(file io.ReadSeeker) (string, error)
ODS2Text(file io.ReaderAt, sizeint64, writer io.Writer, limitint64) (writtenint64, errerror)
ODT2Text(file io.ReaderAt, sizeint64, writer io.Writer, limitint64) (writtenint64, errerror)
PDFListContentStreams(f io.ReadSeeker, w io.Writer, sizeint64) (writtenint64, errerror)
PPTX2Text(file io.ReaderAt, sizeint64) (string, error)
RTF2Text(inputRtfstring) stringXLS2Text(reader io.ReadSeeker, writer io.Writer, sizeint64) (writtenint64, errerror)
XLSX2Text(file io.ReaderAt, sizeint64, writer io.Writer, limitint64, rowLimitint) (writtenint64, errerror)

Picture functions:

IsExcessiveLargePicture(Picture []byte) (excessivebool, errerror)
CompressJPEG(Picture []byte, qualityint) (compressed []byte)
ResizeCompressPicture(Picture []byte, Qualityint, MaxWidth, MaxHeightuint) PDFExtractImages(input io.ReadSeeker) (images []ImageResult, errerror)

Compression and container file functions:

DecompressFile(data []byte) (decompressed []byte, validbool)
ContainerExtractFiles(data []byte, callbackfunc(namestring, sizeint64, date time.Time, data []byte))

Dependencies

This library uses other go packages. Run the following command to download them:

go get -u github.com/nwaples/rardecode
go get -u github.com/saracen/go7z
go get -u github.com/ulikunitz/xz
go get -u github.com/mattetti/filebuffer
go get -u github.com/richardlehane/mscfb
go get -u github.com/taylorskalyo/goreader/epub
go get -u github.com/PuerkitoBio/goquery
go get -u github.com/ssor/bom
go get -u github.com/levigross/exp-html
go get -u github.com/neofight/mobi/convert
go get -u github.com/neofight/mobi/headers
go get -u github.com/unidoc/unipdf
go get -u github.com/nfnt/resize
go get -u github.com/tealeg/xlsx
go get -u gopkg.in/xmlpath.v2

Tests

There are no functional tests. The only test functions are used manually for debugging.

Forks

Other packages were tested and either found insufficient, or unstable. Many of the below listed packages were found to be unstable, cause crashes, as well as exhaust memory due to bad programming, bad input sanitizing and bad memory management.

• html2text is forked from https://github.com/jaytaylor/html2text
• odf is forked from https://github.com/knieriem/odf
• ole2 is forked and partly rewritten from https://github.com/extrame/ole2
• xls is forked from https://github.com/sergeilem/xls which is a fork from https://github.com/extrame/xls
• doc is forked from https://github.com/EndFirstCorp/doc2txt
• docx is forked from https://github.com/guylaor/goword
• mobi is forked from https://github.com/neofight/mobi
• odt is forked from https://github.com/lu4p/cat
• pptx is forked from https://github.com/mr-tim/rol-o-decks
• rtf is forked from https://github.com/J45k4/rtf-go

License

This is free and unencumbered software released into the public domain.

Note that this package includes, or consists partly of forks or rewrite of existing open source code. Use at your own risk. Intelligence X does not provide any warranty for this library or any parts of it.