The Project

To help with the process of importing CSVs to GraphCMS, we've created some simple scripts that can assist in the process. These scripts are not robust, they lack in-depth error checking, and probably lack some general code elegance, but you can view them as an import workspace to quick-start your importing needs.

To follow along you'll need to signup for an account at GraphCMS, create a new project and implement the schema as defined above. For more details about creating a project with GraphCMS, check our getting started guide.

The project structure is as follows.

data/
src/
index.js # for compiling with babel
app.js
inspectData.js
utils/
fetch.js
inspect.js
paginate.js
pickQuery.js
readCSV.js
transforms/
coffeetype.js
queries/ # not used
mutations/

Data Inspection

The file inspectData.js includes a series of helpers that let us check for uniqueness of various keys across our data set, report duplicates, check for total supported character set and more.

The helpers are listed in utils/inspect.js, and the utilities have been abstracted to general purpose usage which should allow them to be used in any project.

/*Simple helper to pick the value off thefirst key of a passed in object*/constfirstKey=obj=>Object.keys(obj)[0];/*Helper to reduce the data to an objectwith a shape of a single key,provided as the only parameter.*/exportconstpickKey=key=>data=>data.map(obj=>{constnewObj={};newObj[key]=obj[key];returnnewObj;});/*Returns a list of all the unique values.*/exportconstuniqueValues=asyncdata=>{return[...newSet(data.map(obj=>obj[firstKey(obj)]))];};/*Returns all the chars that need to besupported by a field. Helpful for validatingif a string field is needed or if integeris allowed.*/exportconstuniqueChars=asyncdata=>{return[
...newSet(data.reduce((prev,curr)=>[...prev, ...curr[firstKey(curr)].split('')],[])),].join();};/*Checks to see if the provided keys have data.*/exportconstrequiredKeys=(keys,identityKey)=>asyncdata=>{return[
...newSet(data.reduce((prev,curr)=>{for(letkeyofkeys){if(!curr[key].length)prev.push(curr[identityKey]);}returnprev;},[])),];};/*Checks to see if there are duplicates, boolean response.*/exportconstuniqueKey=key=>asyncdata=>{constnewLength=[
...newSet(pickKey(key)(data).map(obj=>obj[firstKey(obj)])),];console.log(data.length);console.log(newLength.length);if(data.length!==newLength.length)returnfalse;returntrue;};/*Reports all duplicates with an optional diagnostic keyas input.*/exportconstfindDuplicates=key=>async(data,diagnosticKey)=>{returndata.sort((a,b)=>{if(a[key]<b[key])return-1;if(a[key]>b[key])return1;return0;}).filter((el,index,arr)=>{if(index===0)returnfalse;returnel[key]===arr[index-1][key];}).map(el=>el[diagnosticKey||key]);};

You can implement them like the following.

import{pickKey,uniqueValues,uniqueChars,uniqueKey,requiredKeys,findDuplicates,}from'./utils/inspect';import{readFile}from'./utils/readCSV';exportconstinspectData=async()=>{constdata=awaitreadFile('./data/all_starbucks_locations_in_the_world.csv');constcollection={};// Check for Brandsconstbrands=pickKey('Brand');collection.brands=awaituniqueValues(brands(data));// Get List of Ownership Type EnumconstownershipType=pickKey('Ownership Type');collection.ownershipEnum=awaituniqueValues(ownershipType(data));// Get List of unique phone charsconstphone=pickKey('Phone Number');collection.phoneChars=awaituniqueChars(phone(data));// Check for missing keysconstcheckEmptyLocation=requiredKeys(['Latitude','Longitude'],'Name');collection.missingLocationData=awaitcheckEmptyLocation(data);// Check for uniqueness of names.constuniqueName=uniqueKey('Name');collection.uniqueNames=awaituniqueName(data);// Data cleansing revealed I had duplicates, this helps find them.constfindCommonName=findDuplicates('Name');constcommonNames=awaitfindCommonName(data);collection.commonNamesFlat=[...newSet(commonNames)];// Looking for Duplicate Lat/Long valuesconstuniqueLatitude=uniqueKey('Latitude');collection.uniqueLatitudes=awaituniqueLatitude(data);constuniqueLongitude=uniqueKey('Longitude');collection.uniqueLongitudes=awaituniqueLongitude(data);console.log(collection);};

Data Cleansing and Transformation

The transformation methods are in /transforms, I recommend using a similar pattern. The transforms will be applied in the next section.

constownership={LS: 'Licensed',CO: 'CompanyOwned',JV: 'JointVenture',FR: 'Franchise',};constduplicates=['CentrO Ground Floor','Division del Norte','Gouda Station','Lemessos Enaerios','Mabohai Shopping Mall','Magnolia','Plaza America','SPA','Starbucks','مركز أوتاد',];exportconsttransform=arr=>{returnarr.map(obj=>{constdata={};data.status='PUBLISHED';data.number=obj['Store Number'];data.name=duplicates.includes(obj.Name)
? `${obj.Name}${obj['Store ID']}`
: obj.Name;data.ownership=ownership[obj['Ownership Type']];data.city=obj.City;data.country=obj.Country;data.postcode=obj['Postal Code'];data.phoneNumber=obj['Phone Number'];data.location={latitude: obj.Latitude/1,longitude: obj.Longitude/1,};data.storeID=obj['Store ID']/1;data.olsonTimezone=obj['Olson Timezone'];returndata;});};

Import

The final step is to put the pieces together, transform the content and import the content.

You'll first need to write a mutation query and save that in the mutation folder. Mine looks like the following:

mutationUpdateCoffeeShop(
$brand: String$data: [CoffeeShopCreateWithoutBrandInput!]
) {
updateBrand(
where: { name: $brand }
data: { coffeeShops: { create: $data } }
) {
id
}
}

I've provided some helper utilities using Axios, you will need to provide your own Project API ID and create a secure token for importing the data. For more information on creating a token, check the docs on working with Permanent Auth Tokens.

Fill out the following values in your `.env file.

MUTATION_ENABLED_PAT=eyJ0eXAiOOiJIUz…
PROJECT_ID=ck2lzdpl52f…

When you put all the parts together, you will end up with a file like this:

import{uploadMutation}from'./utils/fetch.js';import{readFile}from'./utils/readCSV';import{inspectData}from'./inspectData';import{paginate}from'./utils/paginate';import{transform}from'./transforms/coffetype';importmutationfrom'./mutations/batchImportCoffeeShop.graphql';construn=async()=>{constdata=awaitreadFile('./data/all_starbucks_locations_in_the_world.csv');constbrand='Coffee House Holdings';constfilteredData=data.filter(el=>el.Brand===brand);consttransformedData=transform(filteredData);paginate(transformedData,uploadMutation(mutation,{brand: brand},'name'));};try{console.log('Get Ready!');// inspectData()run();}catch(e){console.log('Server Error!',e);}

The pagination function implements a two-second delay and a default page size of 50 records. You can pass in a custom page size as a third parameter to the paginate function.