This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector - #352

Merged
shannonlal merged 101 commits into
plotly:masterfrom
shannonlal:master
Apr 10, 2018
Merged

Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom
shannonlal:master

Conversation

@shannonlal

@shannonlalshannonlal commented Jan 23, 2018

Copy link
Copy Markdown
Contributor

This is the first pull request for the Athena Database Connector #261. A couple of notes:

  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.

  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.

  3. This is my first major PR on this project so I welcome any and all comments or suggestions.

@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal

shannonlal commented Feb 14, 2018

Copy link
Copy Markdown
ContributorAuthor

@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.

@n-riesco

n-riesco commented Feb 14, 2018

Copy link
Copy Markdown
Contributor

@shannonlal I'm forwarding your message. Again, sorry about the delay.

@jackparmer

jackparmer commented Feb 16, 2018

Copy link
Copy Markdown
Contributor

Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!

@n-riescon-riesco mentioned this pull request Mar 13, 2018
Open
10 tasks
@shannonlal

shannonlal commented Mar 30, 2018

Copy link
Copy Markdown
ContributorAuthor

@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning

@n-riesco

n-riesco commented Mar 30, 2018

Copy link
Copy Markdown
Contributor

@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.

@shannonlal

shannonlal commented Mar 30, 2018•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:

  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done

  2. create AWS.Athena inside connect - Done

  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done

  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).

  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query

Does this make sense?

@n-riesco

n-riesco commented Mar 30, 2018

Copy link
Copy Markdown
Contributor

@shannonlal 👍

shannonlaland others added 9 commits March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco

n-riesco commented Apr 9, 2018

Copy link
Copy Markdown
Contributor

💃

@shannonlal
shannonlal merged commit a2fe8bd into plotly:masterApr 10, 2018
@n-riescon-riesco mentioned this pull request Apr 10, 2018
Closed
@imsrgadich

imsrgadich commented Feb 2, 2020

Copy link
Copy Markdown

I'm getting the following error.

HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)

What is the solution for this?

@shannonlal

shannonlal commented Feb 2, 2020 via email

Copy link
Copy Markdown
ContributorAuthor

@imsrgadich

imsrgadich commented Feb 2, 2020 via email •
edited
Loading

Copy link
Copy Markdown

@theclash12072002

theclash12072002 commented Jun 12, 2020

Copy link
Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)

from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:
HIVE_METASTORE_ERROR: unable to create input format

so i changed the data type varchar into the crawler but i can't specify the length.

There are any news?

@theclash12072002

theclash12072002 commented Jun 12, 2020

Copy link
Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:
HIVE_UNKNOWN_ERROR: Unable to create input format

running the crawler it found the new table and from Athena it appear running show tables.
Maybe the problem can be in the empty "Edit table details"??

Thanks

@theclash12072002

theclash12072002 commented Jun 14, 2020

Copy link
Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content

This repository was archived by the owner on Aug 29, 2025. It is now read-only.

Athena Database Connector  - #352
Merged
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Apr 10, 2018
Merged
Athena Database Connector #352
shannonlal merged 101 commits into
plotly:masterfrom 
shannonlal:master
Conversation
@shannonlal
@shannonlalshannonlal
commented
Jan 23, 2018

Copy link


Copy Markdown

Contributor
This is the first pull request for the Athena Database Connector #261. A couple of notes:
  1. I broke the Athena database connector into two files. The first is the Athena Connector similar to the other Falcon DB Connectors which defines the standard interfaces (connect, query, tables, schema, etc). The second file just handles the connection to Athena using the AWS SDK. I broke this into a separate file as it made unit testing easier and also if at some point a standard NodeJS library for Athena becomes available it would be easier to replaces this.
  2. I was not able to create an integration test for this connector as I don't have access to a Plot.ly instance of Athena.
  3. This is my first major PR on this project so I welcome any and all comments or suggestions.
@shannonlal
Added new Datasource install file
0500b9c
@shannonlal
Check in latest instructions
a62260a
@shannonlal
Check in new file based on comments
f6aa99c
@shannonlal
Add icons for Athena
605e306
@shannonlal
Updated Instructions
9324b38
@shannonlal
Check in constants for Athena
4d9ee13
@shannonlal
Got UI to detect Athena
273e522
@shannonlal
Check in latest version
632f9da
@shannonlal
Check in first version of schemas
c1f37ba
@shannonlal
Check in schema definition
ca49c63
@shannonlal
Check in more documentation for this
0d389f7
@shannonlal
Check in updated new connection
2c943a8
@shannonlal
Check in draft of new connection:
3577b06
@shannonlal
Check in fixes based on Nicolas comment
d72d5b7
@shannonlal
Updated Datastores.js instructions
837b95a
@shannonlal
Check in small fix
f879c18
@shannonlal
Check in settings to get SQL
d2d7975
@shannonlal
Check in small update for Settings.js
06c74fc
@shannonlal
Check in first draft of unit tests
6b443bc
@shannonlal
Updating unit tests
3ffd87c
@shannonlal
Check in unit tests for connection
463b4c6
@shannonlal
Check in latest tets for database
aaef90f
@shannonlal
Update to do for test cases
154a6b7
@shannonlal
Check in git ignore update
8f14f92
@shannonlal
Check in first function for testing
cc09ab5
@shannonlal
Check in basic test to start query
938cece
@shannonlal
Check in version for getting connection status
765474c
@shannonlal
First stab at execution method
6094c21
@shannonlal
Fixing Query Results
1db2667
@shannonlal
Check in latest version
1a1fcf7
@shannonlal
shannonlal
commented
Feb 14, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco Just as an FYI I am based out of Montreal. So if the Plot.ly folks want I can swing by their office in the mile end and walk them through creating the table in Athena if this will help move this along. Just a thought.
@n-riesco
n-riesco
commented
Feb 14, 2018

Copy link


Copy Markdown

Contributor
@shannonlal I'm forwarding your message. Again, sorry about the delay.
@jackparmer
jackparmer
commented
Feb 16, 2018

Copy link


Copy Markdown

Contributor
Hey @shannonlal - thanks for the offer. I just invited the hotmail address in your bio to Plotly's slack if you want to coordinate there. You're welcome by the office anytime!
@n-riescon-riesco
mentioned this pull request
Mar 13, 2018

Open

10 tasks
@shannonlal
shannonlal
commented
Mar 30, 2018

Copy link


Copy Markdown

ContributorAuthor
@n-riesco I just pulled in the latest changes from the master branch into this PR. I am looking at the comments we discussed about this morning
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal OK, I'll be online here and on slack. Please, ping me, if anything needs clarification.
@shannonlal
shannonlal
commented
Mar 30, 2018•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@n-riesco . I just wanted to summarize your PR comments into one location. Can you let me know if the following changes to this PR make sense:
  1. get rid of createAthenaClient: https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/drivers/athena.js#L15-L30 - Done
  2. create AWS.Athena inside connect - Done
  3. things would be easier if the connection object and the config used in new AWS.Athena(config) were the same - Done
  4. get rid of these checks https://github.com/shannonlal/falcon-sql-client/blob/90842418006c35bd9b28f5c6ea2d84637a9887bf/backend/persistent/datastores/athena.js#L29-L51 (AWS.Athena(config) already checks for those missing properties).
  5. For the datastore.athena.test.js add unit tests for the following methods:connect, tables, schemas, and query
Does this make sense?
@n-riesco
n-riesco
commented
Mar 30, 2018

Copy link


Copy Markdown

Contributor
@shannonlal 👍
shannonlaland others
added 9 commits
March 30, 2018 14:44
@shannonlal
Code Cleanup
2a3aea9
@shannonlal
Cleanup athena driver for nock unit tests
bf61dff
@shannonlal
Fixes for UI Constants
7113a2b
@shannonlal
Debugging connect
df7ccdf
@n-riesco
athena: fix tests
466ff00
* All tests share the Athena connection.
* Added connect() test and ensured it runs first, so that
connection.athenaClient is defined for the remaining tests.
* Defined helper function mockAthenaResponses(query, columnNames, rows).
* Fixed tables() test (tables() shouldn't return the column name
`table_names`).
@n-riesco
athena: fix sample credential
d9cf781
@n-riesco
athena: fix tables and remove unnecessary promises
fecd7ab
* Fixed tables() so that it doesn't return the column name table_names.
* Avoided the creation of unnecessary promises.
* Used slice() and map() to generate the response rows.
@shannonlal
Merge pull request#1from n-riesco/athena/pr-352-debugging
de4489b
Athena/pr 352 debugging
@shannonlal
Code clean up for Athena Unit Tests
82d1243
@n-riesco
n-riesco
commented
Apr 9, 2018

Copy link


Copy Markdown

Contributor
💃
@shannonlal
shannonlal
merged commit a2fe8bd
into
plotly:masterApr 10, 2018
@n-riescon-riesco
mentioned this pull request
Apr 10, 2018

Closed

@imsrgadich
imsrgadich
commented
Feb 2, 2020

Copy link


Copy Markdown

I'm getting the following error.
HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
What is the solution for this?
@shannonlal
shannonlal
commented
Feb 2, 2020
via email 

Copy link


Copy Markdown

ContributorAuthor
@imsrgadich
imsrgadich
commented
Feb 2, 2020
via email •

edited

Loading

Copy link


Copy Markdown

@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

Hi, i have the same problem: HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: varchar type is specified without length: varchar (Service: null; Status Code: 0; Error Code: null; Request ID: null)
from glue i created a crawler for a table in Aurora MySQL with a column varchar(50), into the crawler has been set like string and an error appear running an athena query:

HIVE_METASTORE_ERROR: unable to create input format
so i changed the data type varchar into the crawler but i can't specify the length.
There are any news?
@theclash12072002
theclash12072002
commented
Jun 12, 2020

Copy link


Copy Markdown

I tried with a simple Aurora table within a single field of type int, the error is the previous first:

HIVE_UNKNOWN_ERROR: Unable to create input format
running the crawler it found the new table and from Athena it appear running show tables.

Maybe the problem can be in the empty "Edit table details"??
Thanks
@theclash12072002
theclash12072002
commented
Jun 14, 2020

Copy link


Copy Markdown

link to resolve the error "unable to create input format" in Athena: https://aws.amazon.com/premiumsupport/knowledge-center/athena-unable-to-create-input-format/
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


1 more reviewer

@n-riescon-riescon-riesco left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@shannonlal@n-riesco@jackparmer@imsrgadich@theclash12072002