This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
This repository was archived by the owner on Aug 29, 2025. It is now read-only.

CSV connector - #375

Merged
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector
Feb 21, 2018
Merged

CSV connector#375
n-riesco merged 2 commits into
plotly:masterfrom
n-riesco:csv-connector

Conversation

@n-riesco

Copy link
Copy Markdown
Contributor

const connectionData = {};

// define type error thrown by this connector
export function CSVError(url, errors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You created your own CSV Error with specific information. I am just trying to clarify why you needed your own error type. How will this be handled on the client?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco I did a review this and everything looks good with the exception of creating your own error. I just wanted to better understand this and how these errors will be displayed to the client.

@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal At the moment, the frontend only shows error.message.

The reason I defined a new error type was because papaparse returns a list of errors and line numbers (and I wanted to keep track of them). At the moment, I'm setting error.message to errors[0].message, but this may change in the future.

I don't know yet if I'm going to stick to papaparse, because it fails to parse many CSV files I've tested (I'm still fiddling with the settings).

The PR is still work in progress. It works, but it still has a few rough corners.

* This connector takes a URL as input, downloads the URL and parses it
as a CSV file into a Javascript object that can be queried using
alaSQL.
* Added test specs and the script `test-unit-csv` to run them.
* Updated `Setting.react.js` so that future connectors don't need to
update it (updated `NEW_CONNECTION.md` accordingly).
* TODO (in future PRs):
- `papaparse` parses CSV data only into numbers and strings. Add
option to let the user set a date format for parsing.
- let the user assign a table name
- let the user upload multiple CSV files
Closesplotly#349
@n-riescon-riesco changed the title [WIP] CSV connectorCSV connectorFeb 16, 2018
@n-riesco

Copy link
Copy Markdown
ContributorAuthor

@shannonlal I've just fixed the last few rough corners with this PR. I think the PR is ready for review. Would you like to review it?

@shannonlal

Copy link
Copy Markdown
Contributor

@n-riesco Yes I will have time to look this over on the weekend

@shannonlalshannonlal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the comments I added are small and minor. Everything looks good it was just some minor suggestions. I think it is ready for a dancer. Just let me know if you think my changes make sense or if you want to merge the changes as is.

if (cell) return type(cell);
}

return 'String';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just add a quick comment to explain that default is string. Or add to JSDocs to function

CSVError.prototype.constructor = CSVError;

// the data parsed from CSV files is stored here
const connectionData = {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #362 One of the comments was to start using JSDocs style for our code. Should we be doing this on the backend as well?

connectionData[connection.database] = data;
}

export function connect(connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the url is bad, how will this be presented to the user on the Settings UI? will fetch throw an error and this will be gracefully passed to the UI on its own?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same behaviour for all the connectors, i.e.: fetch throws, routes.js catches the exception and replies to the connection request with a 500 status and the body {error: message: error.message}}.

Comment threadbackend/persistent/datastores/csv.js Outdated

export function tables() {
// To take advantage of alaSQL's parser, the table is named '?'
return Promise.resolve(['?']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using '?' or a const CSV_TABLE_NAME = '?'. This way if it changes in the future you don't have to string replace to make sure we find all changes

@n-riesco
n-riesco merged commit bb661bc into plotly:masterFeb 21, 2018
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@n-riesco@shannonlal