wee

Wee is a powerful and flexible web crawling library based on go-rod, providing a clean interface for writing web crawlers in Go. It offers advanced browser control and customization options.

Features

• Easy-to-use API for web crawling tasks
• Support for headless and headed browser modes
• User mode for connecting to system Chrome browser
• Cookie management and persistence
• Customizable timeouts and error handling
• Humanized and stealth mode options
• Popover handling
• Advanced browser configuration options
• Support for Chrome extensions
• Proxy configuration

Installation

go get github.com/coghost/wee

Usage

Here's a basic example of using wee to crawl search results from Baidu:

package main
import (
"fmt""github.com/coghost/wee"
)
funcmain() {
bot:=wee.NewBotDefault()
deferbot.BlockInCleanUp()
const (
url=`https://www.baidu.com`input=`input[id="kw"]`keyword=`golang`items=`div.result[id] h3>a`noItems=`div.this_should_not_existed`
)
bot.MustOpen(url)
bot.MustInput(input, keyword, wee.WithSubmit(true))
got:=bot.MustAnyElem([]string{items, noItems})
ifgot==noItems {
fmt.Printf("%s has no results\n", keyword)
return
}
elems, err:=bot.Elems(items)
iferr!=nil {
fmt.Printf("get items failed: %v\n", err)
return
}
for_, elem:=rangeelems {
fmt.Printf("%s - %s\n", elem.MustText(), *elem.MustAttribute("href"))
}
}

Advanced Usage

Creating a Bot

Wee provides several ways to create a bot:

// Default botbot:=wee.NewBotDefault()
// Headless botbot:=wee.NewBotHeadless()
// Bot for debuggingbot:=wee.NewBotForDebug()
// User mode bot (connects to system Chrome)bot:=wee.NewBotUserMode()

Customizing Bot Behavior

You can customize bot behavior using options:

bot:=wee.NewBot(
wee.UserAgent("Custom User Agent"),
wee.AcceptLanguage("en-US"),
wee.WithCookies(true),
wee.Humanized(true),
wee.StealthMode(true),
wee.WithPopovers("popover-selector"),
)

Cookie Management

Wee supports various ways to manage cookies:

// Use cookies from default locationwee.WithCookies(true)
// Specify cookie folderwee.WithCookieFolder("/path/to/cookies")
// Use specific cookie filewee.WithCookieFile("/path/to/cookiefile.json")
// Use cookies from clipboard (Copy as cURL)wee.CopyAsCURLCookies(clipboardContent)

Error Handling

You can customize error handling behavior:

bot.SetPanicWith(wee.PanicByLogError)

Advanced Browser Configuration

Wee provides powerful options to configure the browser:

Creating a Browser

launcher, browser:=wee.NewBrowser(
wee.BrowserHeadless(false),
wee.BrowserSlowMotion(1000),
wee.BrowserUserDataDir("/path/to/user/data"),
wee.BrowserPaintRects(true),
wee.BrowserProxy("http://proxy.example.com:8080"),
)

User Mode Browser

launcher, browser:=wee.NewUserMode(
wee.BrowserUserDataDir("/path/to/user/data"),
wee.LaunchLeakless(true),
)

Customizing Launcher

launcher:=wee.NewLauncher(
wee.BrowserHeadless(true),
wee.BrowserExtensions("/path/to/extension1", "/path/to/extension2"),
wee.BrowserFlags("--disable-gpu", "--no-sandbox"),
)

Browser Options

Wee supports various browser options:

• BrowserHeadless(bool): Run browser in headless mode
• BrowserSlowMotion(int): Set slow motion delay in milliseconds
• BrowserUserDataDir(string): Set user data directory
• BrowserPaintRects(bool): Show paint rectangles (useful for debugging)
• BrowserProxy(string): Set proxy server
• BrowserExtensions(...string): Load Chrome extensions
• BrowserFlags(...string): Set additional Chrome flags
• LaunchLeakless(bool): Use leakless mode when launching browser
• BrowserIncognito(bool): Launch browser in incognito mode
• BrowserIgnoreCertErrors(bool): Ignore certificate errors

More Examples

Check out the examples folder for more detailed examples.

Common Issues

panic: context canceled

The "context canceled" issue usually occurs when an element is bound with a timeout on bot.Elem(xxx). To resolve this, you can use:

elem.CancelTimeout().Timeout(xxx)

This will rewrite the previous timeout value.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License