[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

[5.x] Ability to log fake queries - #9695

Merged
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support
Apr 1, 2024
Merged

[5.x] Ability to log fake queries#9695
jasonvarga merged 34 commits into
statamic:masterfrom
JohnathonKoster:adds-stache-query-logging-support

Conversation

@JohnathonKoster

@JohnathonKosterJohnathonKoster commented Mar 10, 2024

Copy link
Copy Markdown
Contributor

Edit by Jason:

It's enabled by default if you have debug enabled. But you can override it and disable it if you want in config/statamic/system.php like this now:

-'fake_sql_queries' => config('app.debug'),+'fake_sql_queries' => false,

Queries will show up in Ray or Debugbar just like a regular database query. There's no need to opt into dumping values.

SQLite is now required (only for this fake query feature).


This PR adds the ability to log Stache "queries" like regular database queries. This is disabled by default. It can be configured by setting new configuration values within config/statamic/stache.php configuration file:

return [
// ..../* |-------------------------------------------------------------------------- | Query Logging |-------------------------------------------------------------------------- | | When enabled, the Stache query builders will log queries | like normal SQL Queries. To log actual query values, | set the dump_values configuration option to true. | */'query_logging' => [
'enabled' => false,
'dump_values' => false,
],
];
  • query_logging.enabled: Determines whether the query logger is enabled
  • query_logging.dump_values: Determines whether the query logger will dump "real" values or placeholders

When enabled, the query logger will emit Laravel's Illuminate\Database\Events\QueryExecuted event. This event will receive a database connection whose closure resolves to null, and no bindings.

By emitting this event existing tools, like Ray, DebugBar or Telescope will be able to pick up on the executed flat-file queries.

Example with dump_values set to false:

image

Example with dump_values set to true:

image

Important Note: The goal of this feature was not to output perfectly valid SQL statements, but rather to be able to observe what the Stache builders are doing in a familiar way with existing tools.

@duncanmcclean

Copy link
Copy Markdown
Member

It might be nice if calling ->dd() on a query gave you the same thing?

Uses a real real in-memory sqlite connection. Small price to pay to make it simpler.
Remove config option as we'll now always put values into bindings.
Now that the QueryExecuted event is populated with more realistic data, bindings are automatically populated within the SQL when using Ray.
- rename trait to fakesqueries
- rework to a closure based method that does the timing etc
- add to iterator builder
- pass table name rather than store, and add methods for figuring out the names
- add to iterator builder (e.g. search)
- pass the query into the dumper class instead of a bunch of arguments
- use a method on the trait to get all the appropriate wheres etc for the dumper
- using that new method, provide the additional where clauses for the entries query that way instead of a "extra from statement"
- remove dump trait method in favor of creating a new dumper
… filtering pages etc. move it to just the search query.
…e builders. removed the condition entirely. not sure when there would be no builder.
this allows to avoid the `$this->columns ?? $columns` situation. `$this->columns` will always be what you need.
also fixes `$query->get(['col1', 'col2'])` not showing proper `select`ed columns within our fake queries
@jasonvarga

jasonvarga commented Apr 1, 2024

Copy link
Copy Markdown
Member

This was really cool. I couldn't help but make a bunch of tweaks. Now it looks more consistent with what happens with regular Laravel DB/Eloquent queries.

CleanShot 2024-04-01 at 15 50 00

CleanShot 2024-04-01 at 15 49 29

  • The event contains actual bindings so now we just always track them and they'll get replaced in the output where appropriate. There's no need for a separate opt-in setting for seeing values.
  • Moved things around quite a bit. Made it more generic. We have non-Stache query builders (like search).
  • I was able to clean things up significantly by using a real DB connection to an in-memory SQL database. The small downside it SQLite is required for this. Only for this feature though, not the whole CMS. Worth the trade off IMO.

@jasonvargajasonvarga changed the title [5.x] Adds support for logging Stache queries[5.x] Adds support for logging fake queriesApr 1, 2024
@jasonvargajasonvarga changed the title [5.x] Adds support for logging fake queries[5.x] Ability to log fake queriesApr 1, 2024
@jasonvarga
jasonvarga merged commit 6b626b8 into statamic:masterApr 1, 2024
@jasonvarga
jasonvarga deleted the adds-stache-query-logging-support branch April 1, 2024 20:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants

@JohnathonKoster@duncanmcclean@jasonvarga