Flat Data: Postprocessing Library and Examples

A collection of postprocessing helper functions and examples for Flat Data.

These examples and functions are written in Deno, a new language created by the same founders of Node.js and meant to improve on many aspects of Node.

Note: If you're noticing your scripts failing recently, try updating to 0.0.15. More info here

Usage

When writing a Flat Data Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further.

- name: Fetch data uses: githubocto/flat@v2with:
http_url: http://api.coindesk.com/v2/bpi/currentprice.json # The endpoint to fetchdownloaded_filename: btc-price.json # The http_url gets saved and renamed in our repository as btc-price.jsonpostprocess: postprocess.ts # A postprocessing javascript or typescript file written in Deno

This is an example of a postprocessing script. Notice the use of Deno.args[0] to pass in the path of the downloaded_filename.

// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts// Replace 'x' with latest library versionimport{readJSON,writeJSON}from'https://deno.land/x/flat@0.0.x/mod.ts'constfilename=Deno.args[0]// equivalent to writing `const filename = 'btc-price.json'`constdata=awaitreadJSON(filename)// pluck a specific key off and write it out to a new fileconstnewfile=`postprocessed_${filename}`awaitwriteJSON(newfile,data.path.to.something)

Examples

Can be found in the examples folder. Once you install Deno you can run these examples with:

• deno run -A examples/csv/csv-example.ts
• deno run -A examples/csv/arquero-example.ts
• deno run -A --unstable examples/image/image-example.ts
• deno run -A examples/json/json-example.ts
• deno run -A examples/sheets/sheets-example.ts
• deno run -A examples/xlsx/xlsx-example.ts
• deno run -A --unstable examples/zip/zip-example.ts

Deno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:

deno run -A examples/csv/csv-example.js

Using Python

While our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example: https://github.com/pierrotsmnrd/flat_data_py_example. Thank you @pierrotsmnrd!

Using bash

You can also use bash as specified in this example: https://github.com/aborruso/flat_data_bash_example. By @aborruso

Postprocessing Library

The Flat Data postprocessing library can be found at: https://deno.land/x/flat/mod.ts

You can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.

CSV

readCSV

readCSV(path: string,options?: ParseOptions): Promise<Record<string,unknown>[]>

Args:

• path: path to a local CSV file
• options:options for parsing the CSV file

Usage:

const csv = await readCSV('./path/to/file.csv')

writeCSV

writeCSV(path: string,data: Record<string,unknown>[]|string,options?: Deno.WriteFileOptions)

Args:

• path: path to a local CSV file
• data: string or object array to store
• options: options for writing the CSV file

Usage:

constdata=[{age: 70,name: 'Rick'},{age: 14,name: 'Smith'}]awaitwriteCSV('./path/to/file.csv',data)

TXT

readTXT

readTXT(path: string): string

Args:

• path: path to a local TXT file

Usage:

consttext=awaitreadTXT('./path/to/file.txt')

writeTXT

writeTXT(path: string,text: string,options?: Deno.WriteFileOptions): void

Args:

• path: path to a local TXT file
• text: text to write to file
• options: options for writing the TXT file

Usage:

awaitwriteTXT('./path/to/file.txt','Content for the file')

JSON

readJSON

readJSON(path: string): JSON

Args:

• path: path to a local JSON file

Usage:

constjson=awaitreadJSON('./path/to/file.json')

readJSONFromURL

readJSONFromURL(url: string): JSON

Args:

• url: URL to a json file

Usage:

constjson=awaitreadJSON('www.url.com/file.json')

writeJSON

writeJSON(path: string,data: any,replacer?: any,space?: string|number): void

Args:

• path: path to a local JSON file
• data: data to store as JSON
• replacer: replacer function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified
• space: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.

Usage:

constdata={age: 40}awaitwriteJSON('./path/to/file.json',data)awaitwriteJSON('./path/to/file-with-indentation',data,null,2)

XLSX

Our library relies on SheetJS, a library for parsing various spreadsheet formats. In addition to a simple readXLSX function you can access the core xlsx module by importing it directly.

import{xlsx,readXLSX}from'https://deno.land/x/flat/mod.ts'

xlsx provides many more utility functions.

readXLSX

readXLSX(path: string): XLSX.WorkBook

Args:

• path: path to a local XLSX file

Usage:

constworkbook=awaitreadXLSX('./path/to/file.xlsx')constsheetData=workbook.Sheets[workbook.SheetNames[0]]constcsvString=awaitxlsx.utils.sheet_to_csv(sheetData)

Image

We recommend using a library like imagescript for more advanced image manipulation. See an example here.

readImageFromFile

readImageFromFile(path: string): Promise<Uint8Array>

Args:

• path: path to a local image file

Usage:

constbytes=awaitreadImageFromFile('./path/to/image.jpeg')

readImageFromURL

readImageFromURL(url: string): Promise<{bytes: Uint8Array;name: string;}>

Args:

• url: url string to an image

Usage:

constimage=awaitreadImageFromURL('www.url.com/image.jpg')constbytes=image.bytesconstname=image.name

writeImage

writeImage(imageBytes: Uint8Array,path: string): void

Args:

• imageBytes: a byte array
• path: path and name to write the image file

Usage:

awaitwriteImage(bytes,'./path/to/image.jpeg')

Zip

unZipFromFile

unZipFromFile(filePath: string,destinationPath: string |null="./",options: any={},): Promise<string|false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

constresult=awaitunZipFromFile('./path/to/folder.zip','./unzip/path')constoutput=result ? 'File unzipped successfully' : 'Error unzipping'

unZipFromURL

unZipFromURL(fileURL: string,destinationPath: string |null="./",options: any={},): Promise<string|false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

constresult=awaitunZipFromURL('www.url.com/file.zip','./unzip/path')constoutput=result ? 'File unzipped successfully' : 'Error unzipping'

Remove

removeFile

removeFile(path: string): void

Args:

• path: path to a local file to delete

Usage:

awaitremoveFile('/path/to/file.x')

Testing

Run all the tests:

deno test -A --unstable tests/*

Run separate tests

deno test -A --unstable tests/csv-test.ts

License

MIT