AngryCurl

• used for parsing information from remote resourse using user-predefined amount of simultaneous connections over proxies-list.

Basic information

Depencies:

• PHP 5 >= 5.1.0
• RollingCurl
• cURL

Use cases:

• multi-threaded parsing over proxy
• overcoming simple parsing protection by using User-Agent header and proxy-lists
• proxy list checking
• validating proxies' response

Main features

• loading proxy-list from file or array
• removing duplicates
• filtering alive proxies
• checking if proxy given response content is correct
• loading useragent-list from file or array
• changing proxy/useragent "on the fly"
• preventing direct connections without any proxy/useragent if such options are set
• multi-thread connections
• callback functions
• working with chains of requests
• web-console mode
• logging

Documentation

Preferred environment configuration

• PHP as Apache module
• safe_mode Off
• open_basedir is NOT set
• PHP cURL installed
• gzip Off

Basic usage

require("RollingCurl.class.php");
require("AngryCurl.class.php");
functionmy_callback($response, $info, $request)
{
// callback function here
}
// sending callback function name as param$AC = newAngryCurl('my_callback');
// initializing console-style output$AC->init_console();
// Importing proxy and useragent lists, setting regexp, proxy type and target url for proxy check// You may also import proxy from an array as simple as $AC->load_proxy_list($proxy array);$AC->load_proxy_list(
// path to proxy-list file'proxy_list.txt',
// optional: number of threads200,
// optional: proxy type'http',
// optional: target url to check'http://google.com',
// optional: target regexp to check'title>G[o]{2}gle'
);
// You may also import useragents from an array as simple as $AC->load_useragent_list($proxy array);$AC->load_useragent_list('useragent_list.txt');
while(/* */)
{
$url = /**/;
// adding URL to queue$AC->get($url);
// you may also use // $AC->post($url, $post_data = null, $headers = null, $options = null);// $AC->get($url, $headers = null, $options = null);// $AC->request($url, $method = "GET", $post_data = null, $headers = null, $options = null);// as well
}
// setting amount of threads and starting connections$AC->execute(200);
// if console_mode is off//AngryCurl::print_debug(); 
unset($AC);

cURL options

You may also pass cURL options for each url before adding to queue like here:

// Define HTTP headers (CURLOPT_HTTPHEADER) if needed, or just set to NULL$headers = array('Content-type: text/plain', 'Content-length: 100');
// Define cURL options (will be passed through curl_setopt_array) if needed, or just set to NULL$options = array(CURLOPT_HEADER => true, CURLOPT_NOBODY => true);
// Define post-data array to send in case of POST method, or just set to NULL$post_data = array('param' => 'value');
// Add request$AC->get($url, $headers, $options);
// or$AC->post($url, $post_data, $headers, $options) ;
// or$AC->request($url, $method = "GET", $post_data, $headers, $options);
// ATTENTION: temporary "on-the-fly" proxy/useragents lists are not// working with AngryCurlRequest. Keep it in mind if you will use code below// as alternative to written above.$request = newAngryCurlRequest($url);
// $url, $method, $post_data, $headers, $options - public properties of AngryCurlRequest$request->options = array(CURLOPT_HEADER => true, CURLOPT_NOBODY => true);
$AC->add($request);

Because this class is kind of extension of RollingCurl class you may use any constructions RollingCurl has. For other information read here: http://code.google.com/p/rolling-curl/source/browse/trunk/

TODO

• chains of requests
• stop on error_limit exceed
• better documentation and examples

Credits

You may join this class discussion here: http://stupid.su/php-curl_multi/ Any questions, change requests and other things you may send to my email written in class comments.

Thank you for reading.

• naive