BranchesTags

Latest commit

History

56 Commits

Folders and files

NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md

Repository files navigation

GDELT-Diff

Codacy gradeGitHub

Abstract

This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.

What is the GDELT?

The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.

To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.

Install Instructions

NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.

  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash
  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config
  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d
  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer
  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:

This will NOT remove the files you have downloaded

sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool

When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:

sudo gdelt-diff --diff

To sync only one stream use:

sudo gdelt-diff --diff_english

OR

sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:

sudo gdelt-diff --retry

To refresh the database of synced files:

sudo gdelt-diff --refresh_database

To see all options and flags:

sudo gdelt-diff -help

About

An Automated File Manager for Maintaining a Local Copy of GDELT Source Files

Topics

gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph

Resources

Readme
MIT license
Activity

Stars

3 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Latest commit
History
56 Commits
Folders and files
NameName
Last commit message
Last commit date
build
build
core
core
.gitignore
.gitignore
.gitmodules
.gitmodules
LICENSE
LICENSE
README.md
README.md
Repository files navigation
GDELT-Diff
Codacy gradeGitHub
Abstract
This small tool is designed to automate the download, orginization, and storage of GDELT source files. GDELT-Diff includes a daemon that runs every 60 mins fetching any new or missing files and sorts them into folders for easy storage. Additionally, an extremely lightweight tool is provided to maintain a copy of only the streams most recent files in /tmp/gdelt-live. This is for anyone doing real-time analysis of the GDELT and doesn't require a full copy of the source files.
What is the GDELT?
The GDELT Project is the largest, most comprehensive, and highest resolution open database of human society ever created. Just the 2015 data alone records nearly three quarters of a trillion emotional snapshots and more than 1.5 billion location references, while its total archives span more than 215 years, making it one of the largest open-access spatio-temporal datasets in existance and pushing the boundaries of "big data" study of global human society. Advanced users and those with unique use cases can download the entire underlying event and graph datasets in CSV format. Deep technical knowledge and extensive experience working with large datasets is required to make use of these datasets, with the GKG alone requiring more than 2.5TB of storage compressed.
To learn more about the GDELT and the records that make up its database, check out the offical documentaion page.
Install Instructions
NOTE: This utlity is designed for large servers with a MINIMUM +100GB OS Drive, +10TB of storage, and +32GB of RAM. Also please consider how many files you need to sync before running.
  1. If you have a pre-existing directory of GDELT files, YOU MUST ensure that files are organized into folders by stream, year and month(/path/stream/2015/05/)
  2. Install GDELT-Diff:
curl https://raw.githubusercontent.com/JustinTimperio/gdelt-diff/master/build/install.sh | bash

  1. Edit Your User Config File With The Paths You Wish to Use:
sudo vi /etc/gdelt-diff/config

  1. Manually Run GDELT-Diff to Ensure Everything is Setup:
sudo gdelt-diff -d

  1. Enable Automatic Downloads With:
sudo systemctl enable gdelt-diff.timer

  1. Enable Automatic Live Downloads With:
sudo systemctl enable gdelt-live.timer

Uninstall GDELT-Diff:
This will NOT remove the files you have downloaded
sudo /opt/gdelt-diff/build/remove.sh

CLI-Tool
When using the utlity manually simply stop the systemd.timers and call gdelt-diff manually:
sudo gdelt-diff --diff

To sync only one stream use:
sudo gdelt-diff --diff_english

OR
sudo gdelt-diff --diff_translation

To force a fetch of 404'd URLs use:
sudo gdelt-diff --retry

To refresh the database of synced files:
sudo gdelt-diff --refresh_database

To see all options and flags:
sudo gdelt-diff -help

About
An Automated File Manager for Maintaining a Local Copy of GDELT Source Files
Topics
gdeltgdelt-datagdelt-eventsgdelt-filesgdelt-knowledge-graph
Resources
Readme
MIT license
Activity
Stars
3 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages