@blockingmachine/core

Core functionality for Blockingmachine, providing robust filter list processing and rule management for AdGuard Home and similar applications.

Related Projects

• Blockingmachine Desktop - Desktop application
• Blockingmachine CLI - Command line interface
• Blockingmachine Database - Filter list repository

Features

• 🚀 Fast filter list processing
• 🔄 Rule deduplication
• 📥 Remote list fetching with retry logic
• ✨ Clean rule formatting
• 🛡️ AdGuard Home compatibility
• 💪 TypeScript support
• 🔁 Automatic retries for failed downloads
• 🎯 Efficient memory usage
• ⚡ Async processing support

Installation

# Using npm
npm install @blockingmachine/core
# Using yarn
yarn add @blockingmachine/core
# Using pnpm
pnpm add @blockingmachine/core

Quick Start

import{RuleDeduplicator,parseFilterList,fetchContent,}from"@blockingmachine/core";// Basic usageconstrules=awaitparseFilterList("||example.com^");constdeduplicator=newRuleDeduplicator();constuniqueRules=deduplicator.process(rules);// Advanced usage with remote listsasyncfunctionprocessRemoteLists(urls: string[]){constdeduplicator=newRuleDeduplicator();letallRules: string[]=[];for(consturlofurls){constcontent=awaitfetchContent(url);if(content){construles=awaitparseFilterList(content);allRules=[...allRules, ...rules];}}returndeduplicator.process(allRules);}

API Reference

RuleDeduplicator

Process and deduplicate filtering rules.

classRuleDeduplicator{constructor(options?: {caseSensitive?: boolean;keepComments?: boolean});process(rules: string[]): string[];addRule(rule: string): void;clear(): void;}

Options

• caseSensitive (boolean, default: true): Preserve case when comparing rules
• keepComments (boolean, default: false): Retain comment lines in output

parseFilterList(content: string, options?: ParseOptions): Promise<string[]>

Parse raw filter list content into individual rules.

interfaceParseOptions{skipComments?: boolean;skipEmpty?: boolean;trim?: boolean;}// Example usageconstrules=awaitparseFilterList("||example.com^\n||example.org^",{skipComments: true,skipEmpty: true,trim: true,});

fetchContent(url: string, options?: FetchOptions): Promise<string | null>

Fetch remote filter lists with built-in retry logic.

interfaceFetchOptions{timeout?: number;retries?: number;retryDelay?: number;}// Example with optionsconstcontent=awaitfetchContent("https://example.com/filterlist.txt",{timeout: 5000,// 5 secondsretries: 3,// Try 3 timesretryDelay: 1000,// Wait 1 second between retries});

Error Handling

try{constcontent=awaitfetchContent("https://example.com/filterlist.txt");if(!content){console.error("Failed to fetch content");return;}construles=awaitparseFilterList(content);}catch(error){console.error("Error processing rules:",error);}

Best Practices

1. Memory Management

// Process large lists in chunksconstdeduplicator=newRuleDeduplicator();for(constchunkofchunks){construles=awaitparseFilterList(chunk);deduplicator.process(rules);}

2. Error Recovery

// Implement retry logic for failed fetchesconstcontent=awaitfetchContent(url,{retries: 5,retryDelay: 2000,});

Contributing

We welcome contributions! Please see our contributing guidelines for details.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the BSD 3-Clause License - see the LICENSE file for details.

Summary of BSD 3-Clause License

You are free to:

• Use the software commercially
• Modify the software
• Distribute the software
• Place warranty on the software

Under the following conditions:

• License and copyright notice must be included with the software
• Neither the names of the copyright holder nor contributors may be used to promote derived products
• Source code must retain copyright notice, list of conditions, and disclaimer

Read full license text

Related Projects

• Blockingmachine Desktop - Desktop application
• Blockingmachine CLI - Command line interface

Support

• 📝 Documentation
• 🐛 Issue Tracker
• 💬 Discussions

Acknowledgments

• AdGuard for their excellent filter syntax documentation
• All our contributors and users

Advanced Usage

Processing Multiple Filter Lists

constsources=["https://example.com/list1.txt","https://example.com/list2.txt",];constprocessAllLists=async()=>{constdeduplicator=newRuleDeduplicator();constresults=awaitPromise.allSettled(sources.map((url)=>fetchContent(url)),);for(constresultofresults){if(result.status==="fulfilled"&&result.value){construles=awaitparseFilterList(result.value);deduplicator.process(rules);}}};

Custom Rule Processing

constcustomProcessor=async(content: string)=>{construles=awaitparseFilterList(content,{skipComments: true,skipEmpty: true,trim: true,});// Custom processing logicreturnrules.filter((rule)=>{// Filter out rules containing specific patternsreturn!rule.includes("specific-pattern");});};

Performance Tips

Memory Optimization

• Process large lists in chunks
• Clear the deduplicator cache periodically
• Use streaming for very large files

constprocessLargeFile=async(filePath: string)=>{constdeduplicator=newRuleDeduplicator();constCHUNK_SIZE=1000;letrules: string[]=[];// Read file in chunksforawait(constchunkofreadFileInChunks(filePath)){constparsedRules=awaitparseFilterList(chunk);rules=[...rules, ...deduplicator.process(parsedRules)];if(rules.length>CHUNK_SIZE){// Process chunk and clear cacheawaitprocessRules(rules);deduplicator.clear();rules=[];}}};

Concurrent Processing

constconcurrentProcessing=async(urls: string[])=>{constBATCH_SIZE=5;// Process 5 URLs at a timeconstresults: string[]=[];for(leti=0;i<urls.length;i+=BATCH_SIZE){constbatch=urls.slice(i,i+BATCH_SIZE);constbatchResults=awaitPromise.all(batch.map((url)=>fetchContent(url)),);results.push(...batchResults.filter(Boolean));}returnresults;};

Debugging

Enable debug logging by setting the environment variable:

# macOS/Linuxexport DEBUG=blockingmachine:*# In your code
const debug = require('debug')('blockingmachine:core');
debug('Processing rules:', rules.length);

Common Issues and Solutions

Timeout Issues

// Increase timeout for slow connectionsconstcontent=awaitfetchContent(url,{timeout: 10000,// 10 secondsretries: 5,});

Memory Issues

// Use streaming API for large filesconstdeduplicator=newRuleDeduplicator({useStreaming: true,chunkSize: 1000,});

Rate Limiting

// Add delays between requestsconstdelay=(ms: number)=>newPromise((resolve)=>setTimeout(resolve,ms));constfetchWithRateLimit=async(urls: string[])=>{for(consturlofurls){awaitfetchContent(url);awaitdelay(1000);// Wait 1 second between requests}};

FAQ

General Questions

Q: What types of filter lists are supported? A: We support AdGuard-style filter lists, including:

• Domain-based rules (||example.com^)
• Basic pattern rules (/ads/)
• Comment lines (! This is a comment)
• AdGuard Home specific syntax

Q: How large of a filter list can this handle? A: The library is optimized for large lists and can handle millions of rules when used with proper memory management practices (see Performance Tips section).

Performance

Q: Why is processing taking longer than expected? A: Several factors can affect processing speed:

• Large number of rules
• Complex pattern matching
• Network latency when fetching remote lists
• System memory constraints

Solution: Use the chunking and streaming options described in the Performance Tips section.

Common Use Cases

Q: How do I combine multiple filter lists?

constcombineLists=async(urls: string[])=>{constdeduplicator=newRuleDeduplicator();for(consturlofurls){constcontent=awaitfetchContent(url);if(content){construles=awaitparseFilterList(content);deduplicator.process(rules);}}returndeduplicator.process([]);};

Q: How can I exclude certain domains from being blocked?

constexcludeDomains=(rules: string[],excludeList: string[])=>{returnrules.filter((rule)=>{return!excludeList.some((domain)=>rule.includes(domain));});};

Troubleshooting

Q: Why am I getting timeout errors? A: Remote lists might be slow to respond. Try:

constcontent=awaitfetchContent(url,{timeout: 30000,// 30 secondsretries: 5,// 5 attemptsretryDelay: 2000,// 2 seconds between retries});

Q: How do I handle invalid rules? A: Use the parsing options to skip problematic rules:

construles=awaitparseFilterList(content,{skipInvalid: true,onError: (error,rule)=>{console.warn(`Skipping invalid rule: ${rule}`);},});

Integration

Q: Can I use this with Express.js? A: Yes, here's a basic example:

importexpressfrom"express";import{RuleDeduplicator,parseFilterList}from"@/core";constapp=express();app.post("/process-rules",async(req,res)=>{try{construles=awaitparseFilterList(req.body.content);constdeduplicator=newRuleDeduplicator();constprocessed=deduplicator.process(rules);res.json({rules: processed});}catch(error){res.status(500).json({error: error.message});}});

Q: How do I save processed rules to a file? A: Use the built-in file system functions:

import{promisesasfs}from"fs";constsaveRules=async(rules: string[],filepath: string)=>{awaitfs.writeFile(filepath,rules.join("\n"),"utf8");};

Updates and Maintenance

Q: How often should I update my filter lists? A: Best practices suggest:

• Daily updates for actively maintained lists
• Weekly updates for stable lists
• Implement rate limiting when fetching multiple lists
• Use the If-Modified-Since header (supported by fetchContent)

Q: How do I handle updates efficiently? A: Use the incremental update feature:

constupdateRules=async(existingRules: string[],newContent: string)=>{constnewRules=awaitparseFilterList(newContent);constdeduplicator=newRuleDeduplicator();returndeduplicator.process([...existingRules, ...newRules]);};

Timeline

Current Release (v1.0.0-beta.3)

• 🎉 Initial public release
• 🚀 Core functionality implementation
• 💪 TypeScript support
• 📥 Remote list fetching
• 🔄 Rule deduplication
• ⚡ Async processing
• 🖥️ Works with Blockingmachine Desktop
• ✅ Improved compatibility with Blockingmachine CLI
• 🐞 Bug fixes and performance improvements

Upcoming Features (v1.0.0)

• 📊 Rule statistics and analytics
• 🔍 Enhanced pattern matching
• 📋 Support for additional filter list formats
• 🌐 Better network resilience
• 🎯 Rule optimization algorithms
• 📦 Reduced bundle size
• 🧪 Extended test coverage

Future Roadmap (v1.x+)

• 🔄 Streaming API for large files
• 🌍 Internationalization support
• 🔒 Enhanced security features
• 📈 Performance improvements
• 🧩 Plugin system
• 🤝 Third-party integrations
• 🛠️ Improved error handling

Version History

1.0.0-beta.3 (Current)

• Improved CLI compatibility
• Fixed several critical bugs
• Performance optimizations
• Documentation updates
• Added more comprehensive examples
• Fixed issues with NPM package integration

1.0.0-beta.2

• Added support for additional filter formats
• Enhanced error handling
• Improved documentation
• Fixed dependency issues
• Fixed CLI integration bugs

1.0.0-beta.1

• Initial public release
• Core functionality stable
• Basic documentation
• Essential features implemented

0.9.0 (Internal)

• Feature complete
• Internal testing
• Performance optimization
• Documentation drafting

0.5.0 (Development)

• Core architecture
• Basic feature implementation
• Initial testing setup