Skip to content
BranchesTags

Latest commit

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md

Repository files navigation

Remote sensing: auto-updating dataset information

Goal: automatically update a list of datasets used for machine learning in remote sensing.

This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.

Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.

Starting point: the list of datasets composed by (Schmitt et al., 2023).

Tasks

  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list

Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.

Possible prompt: "In this excel, can you fill the last 3 columns for the first 10 items? These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."

With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph). Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.

Set-up

Installation:

conda env create -f environment.yml
playwright install
playwright install-deps

Additionally, it is necessary to get the following API keys:

  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure

How to run

python main.py

About

Automatically maintain a list of datasets for remote sensing

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - JorisCod/auto-datasets: Automatically maintain a list of datasets for remote sensing · GitHub
Skip to content
BranchesTags
Latest commit
History
5 Commits
Folders and files
NameName
Last commit message
Last commit date
.gitignore
.gitignore
LICENSE
LICENSE
environment.yml
environment.yml
main.py
main.py
readme.md
readme.md
Repository files navigation
Remote sensing: auto-updating dataset information
Goal: automatically update a list of datasets used for machine learning in remote sensing.
This will be done leveraging a combination of webscraping and an LLM to process the information into a structured format. Basically, this will create an AWESOME list that is automatically maintained.
Note that is not possible with GPT-4 out of the box, the link reader/search pluging and the advanced data analysis (allowing to load an excel file) cannot be combined.
Starting point: the list of datasets composed by (Schmitt et al., 2023).
Tasks
  1. Convert pdf with list of datasets to a more readable format
  2. Get Langchain's example web scraping working
  3. Fill extra columns of the list: application, license, reference & link to Github/Zenodo
  4. Add new datasets to the list
  5. Write out new list
Task 1. was done by using Adobe's pdf to Excel tool. However, it might be more interesting to further convert it to a (sql) database format. This will be done in task 5. if necessary.
Possible prompt:
"In this excel, can you fill the last 3 columns for the first 10 items?
These are datasets for remote sensing, ie for training and validating deep learning methods. You will need to perform search to fill the last 3 columns, using the remote sensing as search criteria. The application column is the application domain, such as flooding, urban or land cover. The license column is the license of the data, often on github or zenodo. The last column is a link to the dataset or dataset paper; just paste any of the links you used to extract the data for previous two columns here."
With some extension, this application would be able to read a tender through https://python.langchain.com/docs/use_cases/question_answering/ and then generate questions on it, answering those with the web scraping. The references found could be checked against an existing reference database (either sql or graph).
Test: Should be able to find GEO-bench through search (eg remote sensing foundation model benchmark) based on ITT documents.
Set-up
Installation:
conda env create -f environment.yml
playwright install
playwright install-deps
Additionally, it is necessary to get the following API keys:
  1. OpenAI API key
  2. Google API key: https://developers.google.com/webmaster-tools/search-console-api/v1/configure
How to run
python main.py
About
Automatically maintain a list of datasets for remote sensing
Resources
Readme
Apache-2.0 license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages