CycleTLS

Currently a WIP and in Active development. See the Tab for more info

If you have a API change or feature request feel free to open an Issue

🚀 Features

High-performance Built-in goroutine pool used for handling asynchronous requests
Custom header ordering via fhttp
Proxy support
Ja3 Token configuration

Usage

Example CycleTLS Request for Typescript and Javascript

You can run this test in tests/simple.test.ts

constinitCycleTLS=require('cycletls');// Typescript: import initCycleTLS from 'cycletls';(async()=>{// Initiate CycleTLSconstcycleTLS=awaitinitCycleTLS();// Send requestconstresponse=awaitcycleTLS('https://ja3er.com/json',{body: '',ja3: '771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0',userAgent: 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0',proxy: 'http://username:password@hostname.com:443'},'get');console.log(response);// Cleanly exit CycleTLScycleTLS.exit();})();

Example CycleTLS Request for Golang

package main
import (
"log""github.com/Danny-Dasilva/CycleTLS/cycletls"
)
funcmain() {
client:=cycletls.Init()
response, err:=client.Do("https://ja3er.com/json", cycletls.Options{
Body : "",
Ja3: "771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0",
UserAgent: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}, "GET");
iferr!=nil {
log.Print("Request Failed: "+err.Error())
}
log.Println(response)
}

Creating an instance

In order to create a cycleTLS instance, you can run the following:

JavaScript

// The initCycleTLS function spawns a Golang process that handles all requests concurrently via goroutine loops. constinitCycleTLS=require('cycletls');// import initCycleTLS from 'cycletls';// Async/Await methodconstcycleTLS=awaitinitCycleTLS();// .then methodinitCycleTLS().then((cycleTLS)=>{});

Golang

import (
"github.com/Danny-Dasilva/CycleTLS/cycletls"
)
//The `Init` function initializes golang channels to process requests. client:=cycletls.Init()

CycleTLS Alias Methods

The following methods exist in CycleTLS

cycleTLS(url, [config])

cycleTLS.get(url, [config])

cycleTLS.delete(url, [config])

cycleTLS.head(url, [config])

cycleTLS.options(url, [config])

cycleTLS.post(url, [config])

cycleTLS.put(url, config)

cycleTLS.patch(url, [config])

Url is not optional, config is optional

CycleTLS Request Config

{// URL for the request (required if not specified as an argument)
url: "https://example.com"// Method for the request ("head" | "get" | "post" | "put" | "delete" | "trace" | "options" | "connect" | "patch")
method: "get"// Default method// Custom headers to send
headers: {"Authorization": "Bearer someexampletoken"}// Custom cookies to send
Cookies: [{"name": "key","value": "val","path": "/docs","domain": "google.com","expires": "Mon, 02-Jan-2022 15:04:05 EST""maxAge": 90,"secure": false,"httpOnly": true,"sameSite": "Lax"}],// Body to send with request (must be a string - cannot pass an object)body: '',// JA3 token to send with requestja3: '771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0',// User agent for requestuserAgent: 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0',// Proxy to send request through (must be in the same format)proxy: 'http://username:password@hostname.com:443',// Amount of seconds before request timeout (default: 7)timeout: 2,// Toggle if CycleTLS should follow redirectsdisableRedirect: true// Custom header order to send with request (This value will overwrite default header order)
headerOrder: ["cache-control","connection","host"]});

CycleTLS Response Schema

{// Status code returned from server (Number)status: 200,// Body returned from the server (String)body: "",// Headers returned from the server (Object)headers: {"some": "header",
...}});

Multiple Requests Example for Typescript and Javascript

If CycleTLS is being used by in a JavaScript environment, CycleTLS will spawn a Golang process to handle requests. This Golang process handles requests concurrently in a worker pool. Due to this, CycleTLS returns response objects as soon as they are made available (in other terms, CycleTLS processes requests as they are received, but responses are returned asynchronously so they will NOT be returned in the order requested)

If you are using CycleTLS in JavaScript, it is necessary to exit out of the instance to prevent zombie processes. The example below shows one way to approach cleanly exiting CycleTLS if you need to process multiple requests (note: keep in mind that calling the exit() function will kill any requests in progress). If your workflow requires requests running the entire time the process runs, modules such as exit-hook could serve as an alternative solution to cleanly exiting CycleTLS.

constinitCycleTLS=require("cycletls");// Typescript: import initCycleTLS from 'cycletls';// Defining custom JA3 token and user agenton multiple requests,"https://httpbin.org/user-agent": {ja3: ja3,userAgent: userAgent,},"http://httpbin.org/post": {body: '{"field":"POST-VAL"}',method: "POST",},"http://httpbin.org/cookies": {cookies: [{name: "example1",value: "aaaaaaa",expires: "Mon, 02-Jan-2022 15:04:05 EST",},],},};// Promises array of requestsconstpromises=[];// Anonymous async function(async()=>{// Initiate CycleTLSconstcycleTLS=awaitinitCycleTLS();// Loop through requestDict (Object) defined abovefor(consturlinrequestDict){// Fetch configs from requestDict (Object)constparams=requestDict[url];// Send request (note: no waiting)constresponse=cycleTLS(url,{body: params.body??"",//?? is just setting defaults in this caseja3: params.ja3??ja3,userAgent: params.userAgent??userAgent,headers: params.headers,cookies: params.cookies,},params.method??"GET");// console.log the response objectresponse.then((out)=>{console.log(url,out);});// Push request to promise arraypromises.push(response);}// Wait for all requests to execute successfullyPromise.all(promises).then(()=>{// Cleanly exit CycleTLS one all requests have been receivedcycleTLS.exit();});})();

Dev Setup

If you would like to compile CycleTLS on your own, use the following commands:

install npm packages

npm install

To recompile index.ts in the src folder

npm run build

To recompile Golang files in the golang folder

Windows

npm run build:windows

Linux

npm run build:linux

Mac

npm run build:mac:

Questions

How do I set Cookies

Details

There are two simple ways to interface with cookies

Javascript Simple Cookie Configuration

constinitCycleTLS=require("cycletls");(async()=>{// Initiate cycleTLSconstcycleTLS=awaitinitCycleTLS();constresponse=awaitcycleTLS("https://httpbin.org/cookies",{cookies: {cookie1: "value1",cookie2: "value2",},});console.log(response.body);/* Expected { "cookies": { "cookie1": "value1", "cookie2": "value2" } } */cycleTLS.exit();})();

In this simple example you can set the cookie name and value within an object

Javascript Complex Cookie Configuration

If you wish to have more fine grained control over cookie parameters you have access to the full underlying Go struct

here are the following values you can set

exportinterfaceCookie{name: string;value: string;path?: string;domain?: string;expires?: string;rawExpires?: string;maxAge?: number;secure?: boolean;httpOnly?: boolean;sameSite?: string;unparsed?: string;}

you can use them in a request as follows

constinitCycleTLS=require("cycletls");(async()=>{// Initiate cycleTLSconstcycleTLS=awaitinitCycleTLS();constcomplexCookies=[{name: "cookie1",value: "value1",domain: "httpbin.org",},{name: "cookie2",value: "value2",domain: "httpbin.org",},];constresponse=awaitcycleTLS("https://httpbin.org/cookies",{cookies: complexCookies,});console.log(response.body);/* Expected { "cookies": { "cookie1": "value1", "cookie2": "value2" } } */cycleTLS.exit();})();

Golang Configure Cookies

package main
import (
"github.com/Danny-Dasilva/CycleTLS/cycletls"
)
funcmain() {
resp, err:=client.Do("https://httpbin.org/cookies", cycletls.Options{
Body: "",
Ja3: "771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0",
UserAgent: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
Cookies: []cycletls.Cookie{{Name: "cookie1", Value: "value1"},
{Name: "cookie2", Value: "value2"}},
}, "GET")
iferr!=nil {
log.Print("Request Failed: "+err.Error())
}
log.Println(resp.Body)
/* Expected { "cookies": { "cookie1": "value1",  "cookie2": "value2" } } *///Altenatively if you want access to values within a maplog.Println(resp.JSONBody())
/* Expected map[cookies:map[cookie1:value1 cookie2:value2]] */
}

Feel free to open an Issue with a feature request for specific file type support.

How do I use CookieJar in CycleTLS?

Details

constinitCycleTLS=require("cycletls");consttough=require("tough-cookie");constCookie=tough.Cookie;(async()=>{// Initiate cycleTLS and CookieJarconstcycleTLS=awaitinitCycleTLS();constcookieJar=newtough.CookieJar();// Capture a set cookieconstfirstResponse=awaitcycleTLS.get("https://httpbin.org/cookies/set?freeform=test",{disableRedirect: true,});// Now use the processCookies function to add the cookies from the response headers to the cookie jarawaitprocessCookies(firstResponse,"https://httpbin.org/cookies/set?freeform=test",cookieJar);// Now send a second to verify we have our cookiesconstsecondResponse=awaitcycleTLS.get("https://httpbin.org/cookies",{headers: {cookie: awaitcookieJar.getCookieString("https://httpbin.org/cookies"),},});//verify cookies were setconsole.log(secondResponse.body)/* Expected { "cookies": { "freeform": "test" } } */cycleTLS.exit();})();asyncfunctionprocessCookies(response,url,cookieJar){if(response.headers["Set-Cookie"]instanceofArray){response.headers["Set-Cookie"].map(async(cookieString)=>awaitcookieJar.setCookie(cookieString,url));}else{awaitcookieJar.setCookie(response.headers["Set-Cookie"],url);}}

Golang example coming soon

How do I download images?

Details

Images with a Content-Type header of the following types are base 64 encoded.

Supported Image Types

image/svg+xml
image/webp
image/jpeg
image/png
application/pdf

To write them to a file you can use the below methods

Javascript Image Write to File

constinitCycleTLS=require("cycletls");varfs=require("fs");//Function to write image to a fileconstwriteImage=(filename,data)=>{letwriteStream=fs.createWriteStream(filename);// write some data with a base64 encodingwriteStream.write(data,"base64");writeStream.on("finish",()=>{console.log(`wrote to file ${filename}`);});// close the streamwriteStream.end();};(async()=>{constcycleTLS=awaitinitCycleTLS();// try {constjpegImage=awaitcycleTLS("http://httpbin.org/image/jpeg",{ja3: "771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0",userAgent:
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",});//Write ImagewriteImage("test.jpeg",jpegImage.body);cycleTLS.exit();})();

Golang Image Write to File

package main
import (
"encoding/base64""os""github.com/Danny-Dasilva/CycleTLS/cycletls"
)
funcmain() {
client:=cycletls.Init()
response, err:=client.Do("http://httpbin.org/image/jpeg", cycletls.Options{
Body: "",
Ja3: "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0",
UserAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36",
}, "GET")
// Decode Base64dec, err:=base64.StdEncoding.DecodeString(response.Body)
iferr!=nil {
panic(err)
}
//create file to writef, err:=os.Create("test.jpeg")
iferr!=nil {
panic(err)
}
deferf.Close()
//write b64 to fileif_, err:=f.Write(dec); err!=nil {
panic(err)
}
iferr:=f.Sync(); err!=nil {
panic(err)
}
}

Additional file type support is planned.

Feel free to open an Issue with a feature request for specific file type support.

Cross Compiling for other platforms

Details

Natively the 4 Operating System types linux, darwin, windows and freebsd should cover most use cases.

You can use the built in Golang cross compiling commands go build to compile for another operating system.

As an example for linux arm you need to pass in the GOOS and GOARCH arguments

$ GOOS=linux GOARCH=arm go build -o ./dist/index ./golang && chmod +x ./dist/index

With the above command you can simply run ./index and CycleTLS should function as intended.

Use this gist for different Operating Systems that support cross-compilation and feel free to open an Issue with a feature request for your specific operating system use case.

LICENSE

GPL3 LICENSE SYNOPSIS

TL;DR* Here's what the GPL3 license entails:

1. Anyone can copy, modify and distribute this software.
2. You have to include the license and copyright notice with each and every distribution.
3. You can use this software privately.
4. You can use this software for commercial purposes.
5. Source code MUST be made available when the software is distributed.
6. Any modifications of this code base MUST be distributed with the same license, GPLv3.
7. This software is provided without warranty.
8. The software author or license can not be held liable for any damages inflicted by the software.

More information on about the LICENSE can be found here

Name		Name	Last commit message	Last commit date
.github		.github
cycletls		cycletls
docs		docs
examples		examples
src		src
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
jest.config.js		jest.config.js
main.go		main.go
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Navigation Menu

Repository files navigation

CycleTLS

🚀 Features

Table of contents

Dependencies

Installation

Usage

Example CycleTLS Request for Typescript and Javascript

Example CycleTLS Request for Golang

Creating an instance

JavaScript

Golang

CycleTLS Alias Methods

CycleTLS Request Config

CycleTLS Response Schema

Multiple Requests Example for Typescript and Javascript

Dev Setup

To recompile index.ts in the src folder

To recompile Golang files in the golang folder

Questions

How do I set Cookies

Javascript Simple Cookie Configuration

Javascript Complex Cookie Configuration

Golang Configure Cookies

How do I use CookieJar in CycleTLS?

How do I download images?

Javascript Image Write to File

Golang Image Write to File

Cross Compiling for other platforms

LICENSE

GPL3 LICENSE SYNOPSIS

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

CycleTLS

🚀 Features

Table of contents

Dependencies

Installation

Usage

Example CycleTLS Request for Typescript and Javascript

Example CycleTLS Request for Golang

Creating an instance

JavaScript

Golang

CycleTLS Alias Methods

CycleTLS Request Config

CycleTLS Response Schema

Multiple Requests Example for Typescript and Javascript

Dev Setup

To recompile index.ts in the src folder

To recompile Golang files in the golang folder

Questions

How do I set Cookies

Javascript Simple Cookie Configuration

Javascript Complex Cookie Configuration

Golang Configure Cookies

How do I use CookieJar in CycleTLS?

How do I download images?

Javascript Image Write to File

Golang Image Write to File

Cross Compiling for other platforms

LICENSE

GPL3 LICENSE SYNOPSIS

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages