🕷️ Xcrap Impit Client

Xcrap Impit Client is a package within the Xcrap framework that implements an HTTP client using the Impit library.

---

📦 Installation

Installation is straightforward; just use your preferred dependency manager. Here's an example using NPM:

npm i @xcrap/impit-client @xcrap/core @xcrap/extractor

You also need to install @xcrap/extractor and @xcrap/core because they are listed as peerDependencies. This means the package requires @xcrap/extractor and @xcrap/core, but it will use the versions that the user has installed in their project.

---

🚀 Usage

Like all HTTP clients, ImpitClient has two methods: fetch() to make a request to a specific URL and fetchMany() to make requests to multiple URLs simultaneously, with control over concurrency and delays between requests.

Usage Example

import{ImpitClient}from"@xcrap/impit-client"import{extract,css}from"@xcrap/extractor";(async()=>{constclient=newImpitClient()consturl="https://example.com"constresponse=awaitclient.fetch({url: url})constparser=response.asHtmlParser()constpageTitle=awaitparser.extractValue({query: css("title"),extractor: extract("innerText")})console.log("Page Title:",pageTitle)})();

Adding a Proxy

In an HTTP client that extends BaseClient, you can add a proxy in the constructor as shown in the following examples:

1. Providing a proxy string:

   constclient=newImpitClient({proxy: "http://47.251.122.81:8888"})

2. Providing a function that generates a proxy:

   functionrandomProxy(){constproxies=["http://47.251.122.81:8888","http://159.203.61.169:3128"]constrandomIndex=Math.floor(Math.random()*proxies.length)returnproxies[randomIndex]}constclient=newImpitClient({proxy: randomProxy})

Using a Custom User Agent

In a client that extends BaseClient, you can also customize the User-Agent for requests. You can do this in two ways:

1. Providing a userAgent string:

   constclient=newImpitClient({userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"})

2. Providing a function that generates a userAgent:

   functionrandomUserAgent(){constuserAgents=["Mozilla/5.0 (iPhone; CPU iPhone OS 9_8_4; like Mac OS X) AppleWebKit/603.37 (KHTML, like Gecko) Chrome/54.0.1244.188 Mobile Safari/601.5","Mozilla/5.0 (Windows NT 10.3;; en-US) AppleWebKit/537.35 (KHTML, like Gecko) Chrome/47.0.1707.185 Safari/601"]constrandomIndex=Math.floor(Math.random()*userAgents.length)returnuserAgents[randomIndex]}constclient=newImpitClient({userAgent: randomUserAgent})

Using a Custom Proxy URL

In a client that extends BaseClient, you can use proxy URLs. While the exact explanation of how they work might be complex, I encountered this type of proxy when trying to solve CORS issues by making client-side requests, and that's how I learned about CORS Proxy. Here's a template for a CloudFlare Workers implementation if you want to set up your own.

You can configure this in the same way we configured userAgent:

1. Providing a proxyUrl string:

   constclient=newImpitClient({proxyUrl: "https://my-proxy-app.my-username.workers.dev"})

2. Providing a function that generates a proxyUrl:

   functionrandomProxyUrl(){constproxyUrls=["https://my-proxy-app.my-username-1.workers.dev","https://my-proxy-app.my-username-2.workers.dev"]constrandomIndex=Math.floor(Math.random()*proxyUrls.length)returnproxyUrls[randomIndex]}constclient=newImpitClient({proxyUrl: randomProxyUrl})

---

🧪 Tests

Automated tests are located in __tests__. To run them:

npm run test

---

🤝 Contributing

Want to contribute? Follow these steps:

• Fork the repository.
• Create a new branch (git checkout -b feature-new).
• Commit your changes (git commit -m 'Add new feature').
• Push to the branch (git push origin feature-new).
• Open a Pull Request.

---

📝 License

This project is licensed under the MIT License.