Stack Brainstorming

Jump to bottom
Daniel Reeves edited this page Jun 7, 2020 · 4 revisions

Task queue

What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.

Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).

Comparison of task queue frameworks:

CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)

Object Relational Mapping ("ORM")

What it is: Translates SQL objects into Python objects and vice-versa.

Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.

Comparison of ORM frameworks:

SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow

SQL Database

The choices are:

  • SQLite
  • Postgres

Clone this wiki locally

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Stack Brainstorming
Jump to bottom

Daniel Reeves edited this page Jun 7, 2020
·
4 revisions
Task queue
What it is: Runs tasks called from other places asynchronously of what called it. For example, refreshing a page can call an asynchronous task.
Why we need it: In the very long run, we want our database to do either one of two things (1) run once an hour and update the model, (2) run up to once an hour, updating whenever someone refreshes the web page. Both of these designs, but especially the first, would benefit from a task queue. The first design benefits from a task queue because it lets us set cron jobs; the second task benefits because it handles what we should do when someone refreshes the page multiple times before results are cached (i.e. only run the task once).
Comparison of task queue frameworks:
CeleryHueyRQDramatiq
Windows SupportOKGoodBadOK to Good
Unix SupportGoodGoodGoodGood
APIDecoratorsDecoratorsProcedural callsDecorators
Resource intensivenessMediumLowLowLow
Github stars14.9k2.9k6.9k2.0k
Native cron schedulingYesYesYesNo
I/O manager (broker)Redis, RabbitMQ, Amazon SQSRedis, SQLite, memoryRedisRedis, RabbitMQ, memory
ComplexityHighLowMediumLow
Monitoring UIYes (Flower)NoYes (rq-dashboard)Yes, but not well-maintained (dramatiq_dashboard)
Object Relational Mapping ("ORM")
What it is: Translates SQL objects into Python objects and vice-versa.
Why we need it: ORMs handle the logic of connecting your database to your Python code. ORM's let you write code in Python's idiom, as oppossed to your database's idiom. That said, we probably don't need to rely too heavily on an ORM. In a sense, Pandas is kind of our "ORM." By that I mean: it is very likely we will be passing the database into Pandas DataFrames in most cases, and interact with the data that way. There are a few cases where this might not be true, and in any case it doesn't hurt to pass an ORM's database connection through pd.read_sql, hence why we still want to decide on an ORM.
Comparison of ORM frameworks:
SQL-AlchemyPeewee
Github stars3.1k (depends if you count other repos)7.5k
ComplexityMediumLow
SQL Database
The choices are:
  • SQLite
  • Postgres
Clone this wiki locally