Support custom test commands

[clinejj]

I have checked that this feature is not already implemented

• This feature does not exist

Use case

The current LSP attempts to automatically determine the command to run tests using the Test Explorer, however this doesn't work for all scenarios. For example, Rails applications use bin/rails test out of the box.

The current commands generated by the LSP do not work for all codebases, so allowing developers to customize the test command would make the Test Explorer functionality more useful for all codebases.

Description

I would like to be able to customize the command used for the test explorer to run tests, for example bin/rails test

Implementation

In the Code Lens command creation, prefer to read from the extension configuration instead of the base command: https://github.com/Shopify/ruby-lsp/blob/main/lib/ruby_lsp/listeners/code_lens.rb#L210-L225

[bdewater-thatch]

This used to work, but some time in the past couple of months(?) it stopped working. I ran into this myself too now (@clinejj and I work together) and after some poking around I realized the Rails add-on provides this: https://github.com/Shopify/ruby-lsp-rails/blob/74b324d3faf10b0ebf91121f1ab0d51b44296b2c/lib/ruby_lsp/ruby_lsp_rails/code_lens.rb#L235-L238

FWIW, running Cursor version: 2.1.39 (VSCode version: 1.105.1), Ruby LSP v0.26.4 and Ruby LSP Rails add-on v0.4.8 according to the diagnostics.

[vinistock]

Thanks for the report. Yeah, we do provide the ability to run Rails tests through the add-on, so this should be working.

Have you turned off the fullTestDiscovery feature flag? This should be turned on as we're migrating away from the previous way of discovering tests.

If it's not turned off, then it would be helpful to have a reproduction scenario or the symptoms. Does it still discover the tests? Is it just the command that's incorrect? What test framework is being used?

[bdewater-thatch]

The fullTestDiscovery feature flag is not turned off. Tests are discovered, the lens shows above the tests, icons in the gutter. It's just the command that's not correct (bundle exec ruby ... instead of bin/rails). We use minitest, spec style to be exact. Because of the wrong command, we get this error:

bundle exec ruby -r/Users/bdewater/.rbenv/versions/3.4.7/lib/ruby/gems/3.4.0/gems/ruby-lsp-0.26.4/lib/ruby_lsp/test_reporters/minitest_reporter.rb -r/Users/bdewater/.rbenv/versions/3.4.7/lib/ruby
/gems/3.4.0/gems/ruby-lsp-0.26.4/lib/ruby_lsp/test_reporters/test_unit_reporter.rb -Itest /Users/bdewater/src/github.com/thatch-health/thatch/test/models/user_test.rb --name "/^field\ normalization#test_\d{4}_normalizes\ email\$/"
/Users/bdewater/src/github.com/thatch-health/thatch/test/test_helper.rb:61:in '<top (required)>': uninitialized constant ActiveSupport (NameError)

ActiveSupport::TestCase.parallelize_teardown do |worker|
^^^^^^^^^^^^^
 from /Users/bdewater/.rbenv/versions/3.4.7/lib/ruby/3.4.0/bundled_gems.rb:82:in 'Kernel.require'
 from /Users/bdewater/.rbenv/versions/3.4.7/lib/ruby/3.4.0/bundled_gems.rb:82:in 'block (2 levels) in Kernel#replace_require'
 from /Users/bdewater/src/github.com/thatch-health/thatch/test/models/user_test.rb:4:in '<main>'

Here's the full diagnostic output:

Ruby LSP Information

VS Code Version

1.105.1

Ruby LSP Extension Version

0.9.32

Ruby LSP Server Version

0.26.4

Ruby LSP Add-ons

• Tapioca (0.1.3)
• Ruby LSP Rails (0.4.8)
• RuboCop (1.81.7)

Ruby Version

3.4.7

Ruby Version Manager

rbenv

Installed Extensions

Click to expand

• EditorConfig (0.17.4)
• azure-repos (0.40.0)
• docker (0.18.0)
• gitlens (17.7.1)
• mermaid-markdown-syntax-highlighting (1.7.4)
• op-vscode (1.0.5)
• prettier-vscode (11.0.0)
• rainbow-csv (3.19.0)
• remote-containers (0.394.0)
• remote-repositories (0.42.0)
• remotehub (0.64.0)
• ruby-extensions-pack (0.1.13)
• ruby-lsp (0.9.32)
• sorbet-vscode-extension (0.3.46)
• terraform (2.37.6)
• theme-dracula (2.25.1)
• typos-vscode (0.1.45)
• vscode-coverage-gutters (2.14.0)
• vscode-docker (2.0.0)
• vscode-erb-beautify (0.5.0)
• vscode-eslint (3.0.16)
• vscode-graphql (0.13.2)
• vscode-graphql-execution (0.3.2)
• vscode-graphql-syntax (1.3.8)
• vscode-icons (12.15.0)
• vscode-open-in-github (1.4.1)
• vscode-pull-request-github (0.102.0)
• vscode-remote-extensionpack (0.26.0)
• vscode-socket-security (2.0.3)
• vscode-tailwindcss (0.14.28)
• vscode-yaml (1.19.1)

Ruby LSP Settings

Click to expand

Workspace

{}

User

{
 "enabledFeatures": {
 "codeActions": true,
 "diagnostics": true,
 "documentHighlights": true,
 "documentLink": true,
 "documentSymbols": true,
 "foldingRanges": true,
 "formatting": true,
 "hover": true,
 "inlayHint": true,
 "onTypeFormatting": true,
 "selectionRanges": true,
 "semanticHighlighting": true,
 "completion": true,
 "codeLens": true,
 "definition": true,
 "workspaceSymbol": true,
 "signatureHelp": true,
 "typeHierarchy": true
 },
 "featuresConfiguration": {},
 "addonSettings": {},
 "rubyVersionManager": {
 "identifier": "auto"
 },
 "customRubyCommand": "",
 "formatter": "auto",
 "linters": null,
 "bundleGemfile": "",
 "testTimeout": 30,
 "branch": "",
 "pullDiagnosticsOn": "both",
 "useBundlerCompose": false,
 "bypassTypechecker": false,
 "rubyExecutablePath": "",
 "indexing": {},
 "erbSupport": true,
 "featureFlags": {},
 "sigOpacityLevel": "1"
}

[github-actions]

This issue is being marked as stale because there was no activity in the last 2 months

[gstokkink]

We use minitest-spec-rails and probably because of this, the bundle exec rails test ... --name="foo" approach to running our tests does not work. Instead, we run our tests with bundle exec rails test ... test/foo_test:21. If it was possible to have custom test commands, we'd be able to use the Ruby LSP code lens actions to run our tests directly from VS Code 😄 For now, we have to rely on a different third party plugin (Ruby Test Runner) to provide the correct code lens actions. Thanks for considering!

[willmakley]

I have a project that prefers certain environment variables to be set when running tests, so I have a custom test script I would like to be able to use.