🕷️ Xcrap API Client

Xcrap API Client is a package for the Xcrap Cloud service, designed for communicating with the API and handling authentication, management, and execution of scrapers and clients.

📦 Installation

Installation is straightforward. Just use your preferred dependency manager. Here is an example using NPM:

npm i @xcrap/api

🚀 Usage

If you're using a different server based on Xcrap Cloud (yes, we're open source), change the baseUrl in the Xcrap constructor.

Authentication

To access the routes, you must first authenticate. There are two ways to do this: using an email and password to get access tokens or using API Keys.

Using email and password:

import{Xcrap,AuthService}from"@xcrap/api";(async()=>{constauth=newAuthService()consttokens=awaitauth.signIn({email: "YOUR_EMAIL",password: "YOUR_PASSWORD"})constxcrap=newXcrap(tokens)})();

Using API Keys:

import{Xcrap}from"@xcrap/api"constxcrap=newXcrap({apiKey: "YOUR_API_KEY"})

---

Clients

Creating a Client

import{Xcrap,ClientType}from"@xcrap/api";(async()=>{// Your way of instantiating Xcrapconstclient=awaitxcrap.clients.create({name: "A Cool Client",description: "A description for my Cool Client.",// optionaltype: ClientType.Axios// Check if the type is actually available for the pointed server})console.log(client)// Displays a class called Client that has the following methods: update, delete})();

Finding a Client

constclient=awaitxcrap.clients.findOne("YOUR_CLIENT_ID")

Finding Multiple Clients

constclients=awaitxcrap.clients.findMany()

You can also paginate. The default and maximum amount of results will depend on the targeted server, but you can define it explicitly:

constclients1=awaitxcrap.clients.findMany({page: 1,perPage: 20})constclients2=awaitxcrap.clients.findMany({page: 2,perPage: 20})

Updating a Client

constupdatedClient=awaitxcrap.clients.update("YOUR_CLIENT_ID",{name: "Updated Cool Client",description: "A updated description for my Cool Client.",type: "got_scraping"// Again, check if the type is available for the pointed server})

Using a Client object:

awaitclient.update({name: "Updated Cool Client",description: "A updated description for my Cool Client.",type: "got_scraping"// Again, check if the type is available for the pointed server})

Deleting a Client

awaitxcrap.clients.delete("YOUR_CLIENT_ID")

Using a Client object:

awaitclient.delete()

---

Scrapers

Creating a Scraper

import{Xcrap,ParsingModelFieldType}from"@xcrap/api";(async()=>{// Your way of instantiating Xcrapconstscraper=awaitxcrap.scrapers.create({name: "A Cool Scraper",description: "A description for my Cool Scraper",// optionaldefaultUrl: "http://example.com",// optionalclientId: "YOUR_CLIENT_ID",parsingModel: {type: ParsingModelFieldType.Html,// Check if the type is available for the pointed servermodel: {title: {query: "title",extractor: "text"// Check if the extractor is available for the pointed server},title: {query: "title",extractor: "attribute:content"// Again, check if the extractor is available for the pointed server}}}})console.log(scraper)// Displays a class called Scraper that has the following methods: update, delete, execute})();

Finding a Scraper

constscraper=awaitxcrap.scrapers.findOne("YOUR_SCRAPER_ID")

Executing a Scraper

const{ data, metadata }=awaitxcrap.scrapers.executeOne("YOUR_SCRAPER_ID")

Using a Scraper object:

const{ data, metadata }=awaitscraper.execute()

Executing a Dynamic Scraper

To be clear, a dynamic scraper is a scraper that isn't registered in the database; it's created dynamically in real-time. You can also create a Client with it or just pass the ID of an existing one.

Using an existing Client:

const{ data, metadata }=awaitxcrap.scrapers.executeOneDynamic({url: "http://example.com",clientId: "YOUR_CLIENT_ID",parsingModel: {type: ParsingModelFieldType.Html,// Check if the type is available for the pointed servermodel: {title: {query: "title",extractor: "text"// Check if the extractor is available for the pointed server},title: {query: "title",extractor: "attribute:content"// Again, check if the extractor is available for the pointed server}}}})

Dynamically creating a Client:

const{ data, metadata }=awaitxcrap.scrapers.executeOneDynamic({url: "http://example.com",client: {type: "puppeteer_real_browser"// Check if the type is actually available for the pointed server},parsingModel: {type: ParsingModelFieldType.Html,// Check if the type is available for the pointed servermodel: {title: {query: "title",extractor: "text"// Check if the extractor is available for the pointed server},title: {query: "title",extractor: "attribute:content"// Again, check if the extractor is available for the pointed server}}}})

Finding Multiple Scrapers

constscrapers=awaitxcrap.scrapers.findMany()

You can also paginate. The default and maximum amount of results will depend on the targeted server, but you can define it explicitly:

constscrapers1=awaitxcrap.scrapers.findMany({page: 1,perPage: 20})constscrapers2=awaitxcrap.scrapers.findMany({page: 2,perPage: 20})

Updating a Client

constupdatedScraper=awaitxcrap.SCRAPERS.update("YOUR_SCAPER_ID",{name: "Updated Cool Scraper",description: "A updated description for my Cool Scraper.",model: {title: {query: "title",extractor: "text"// Check if the extractor is available for the pointed server},title: {query: "title",extractor: "attribute:content"// Again, check if the extractor is available for the pointed server},heading: {query: "h1",extractor: "text"}}})

Using a Scraper object:

awaitscraper.update({name: "Updated Cool Scraper",description: "A updated description for my Cool Scraper.",model: {title: {query: "title",extractor: "text"// Check if the extractor is available for the pointed server},title: {query: "title",extractor: "attribute:content"// Again, check if the extractor is available for the pointed server},heading: {query: "h1",extractor: "text"}}})

Deleting a Scraper

awaitxcrap.scrapers.delete("YOUR_CLIENT_ID")

Using a Client object:

awaitscraper.delete()

🧪 Tests

Automated tests are in __tests__. To run them:

npm run test

🤝 Contributing

• Want to contribute? Follow these steps:
• Fork the repository.
• Create a new branch (git checkout -b feature-new).
• Commit your changes (git commit -m 'Add new feature').
• Push to the branch (git push origin feature-new).
• Open a Pull Request.

📝 License

This project is licensed under the MIT License.