Skip to content
BranchesTags

Repository files navigation

Paper Crawler for Top AI Conferences

This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:

conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:

  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.

Change Log

  • 4-Dec-2022
    • This project was rewritten by @hellock.
  • 28-OCT-2022
    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022
    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022
    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022
    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022
    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.

Install

pip install scrapy git+https://github.com/hellock/paperCrawler.git

Usage

It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/ then call the spider. Some examples are provided below.

For single conference

scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]

  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.

  • year: Four-digit numbers, use comma to separate.

  • keys: The abstract must contain at least one of the keywords. Use comma to separate.

  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same output filename.

  • checkpoint_folder: The folder to store the spider state.

Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv

The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion" appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the checkpoint. It scrapes really fast.

Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1

The command above will save the scraping checkpoint in a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same command so that the scraping can continue.

For multiple conferences

python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal

The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.

About

This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - ly015/paperCrawler: This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc... · GitHub
Skip to content
BranchesTags
Repository files navigation
Paper Crawler for Top AI Conferences
This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
conference, year, title, authors, abstract, code_url, pdf_url, matched_queries

The crawler scrapes accepted papers from top AI conferences, including:
  • CVPR and ICCV since 2013.
  • ECCV since 2018.
  • AAAI since 1980.
  • IJCAI since 2017.
  • NIPS since 1987.
  • ICML since 2017.
  • ICLR 2018, 2019, 2021 and 2022.
Change Log
  • 4-Dec-2022

    • This project was rewritten by @hellock.
  • 28-OCT-2022

    • Added a feature in which the target conferences can be specified in main.py. See Example 4.
  • 27-OCT-2022

    • Added the crawler for ACM Multimedia.
  • 20-OCT-2022

    • Fixed a bug that falsely locates the paper pdf url for NIPS.
  • 7-OCT-2022

    • Rewrote main.py so that the crawler can run over all the conferences!
  • 6-OCT-2022

    • Removed the use of PorterStemmer() from nltk as it involves false negative when querying.
Install
pip install scrapy git+https://github.com/hellock/paperCrawler.git
Usage
It's a Scrapy project. Simply cd to PaperCrawler/conf_crawler/
then call the spider. Some examples are provided below.
For single conference
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a queries=[key1,key2,...,keyn] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
  • conference name: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr. Must be lowercase.
  • year: Four-digit numbers, use comma to separate.
  • keys: The abstract must contain at least one of the keywords. Use comma to separate.
  • output_filename: The output csv filename. The outputs will attach to previous file if two commands share the same
output filename.
  • checkpoint_folder: The folder to store the spider state.
Example 1
scrapy crawl iccv -a years=2021,2019,2017 -a queries=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv. It won't count the citations and save the
checkpoint. It scrapes really fast.
Example 2
scrapy crawl ijcai -a years=2021,2020 -a query=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1. If the scraping process is interrupted by CTRL+C or other incidents, simply execute the same
command so that the scraping can continue.
For multiple conferences
python main.py [conf1,conf2,...,confn] [year1,year2,...,yearn] -query [key1,key2,...,keyn]
Example 3
python main cvpr,iccv,eccv 2018,2019,2020,2021,2022 -query emotion,multimodal,multi-modal
The command would scrape the papers, whose abstracts contain at least one query, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv, which is defined in settings.py.
About
This is a Scrapy-based web-spider. It scrapes papers from TOP conferences, including CVPR, NIPS, ICLR, etc...
Resources
Readme
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages