copilot-bluespec: Tests failing on hackage

[ivanperez-keera]

The tests of copilot-bluespec fail on hackage: https://hackage.haskell.org/package/copilot-bluespec-4.5/reports/1

Do we know why? Is this due to a dependency on a tool not available on hackage?

[ivanperez-keera]

Tagging @RyanGlScott for awareness.

[RyanGlScott]

The copilot-bluespec tests require bsc (the Bluespec compiler) to be available so that it can compile and simulate the generated Bluespec code.

[ivanperez-keera]

Is this something we can add as a tool dependency in the cabal package? Or would we need to request that it be on hackage for these tests to pass?

[RyanGlScott]

No, not really—cabal has no knowledge of the bsc tool.

If having the tests fail on Hackage is a concern, then we could mark the test suite as non-buildable by default.

[ivanperez-keera]

Isn't bluespec written in Haskell? If we asked the Bluespec team to put it on hackage, would we be able to depend on it?

[RyanGlScott]

bsc is indeed written in Haskell, but my understanding is that the bsc build system is somewhat non-trivial. For starters, they invoke ghc directly on .hs files instead of using .cabal. I think getting things into a state where it could be put into a .cabal file and uploaded to Hackage would take some effort. (See also B-Lang-org/bsc#546, which requests that specific parts of bsc that language-bluespec makes use of be split into individual libraries.)

[ivanperez-keera]

Could and should we ask the hackage team to make bsc available on hackage?

[RyanGlScott]

We could certainly request this, per the advice in haskell/hackage-server#1414 (comment). I held off on doing so up until this point since I was awaiting an answer to my question here, but I never received a response. At this point, it would likely be more direct just to open an issue on https://github.com/haskell-infra/hackage-doc-builder-config and see what happens.

[RyanGlScott]

I've converted this discussion into an issue at #673.

[chathhorn-galois]

@ivanperez-keera -- After further research, it looks like one can disable running tests on Hackage if you're authenticated (presumably via the build report page, the API endpoint seems to be https://hackage.haskell.org/package/copilot-bluespec-4.6/reports/testsEnabled/ -- see: the hackage-server source
).

This might be a better solution than disabling the tests via a cabal build flag (like I proposed here
). The issue with entirely disabling the test suite by marking it as unbuildable is that cabal test will still report a failure:

Error: [Cabal-7043]
Cannot test the package copilot-bluespec-4.6 because none of the components are available to build: the test suite 'tests' is marked as 'buildable: False'

Another possible solution would be to do a check for bsc at runtime in the test suite itself and still "passing" if bsc isn't found.

[ivanperez-keera]

I believe if we disable them via cabal flags, hackage will still report "no tests" rather than "broken tests".

See, for example: https://hackage.haskell.org/package/dunai-0.14.1/reports/1 and https://hackage.haskell.org/package/dunai-0.14.1.

I don't want to experiment with a package under the copilot framework, or wait until the release to find out.

Can you try it with a different package to see what happens on Hackage itself, or with a dummy package?

[RyanGlScott]

The issue with entirely disabling the test suite by marking it as unbuildable is that cabal test will still report a failure:

I don't think hackage-server will just try to run cabal test unconditionally. Rather, the logic that hackage-server uses to build packages on the server is smart enough to check whether tests are enabled or not, and if they're not enabled, then it won't even attempt to run cabal test in the first place.

For an example of this in action, see the test suites in the dunai package. They're both marked as buildable: False using a similar approach, and the build status page for dunai shows "Tests NotTried", as one would expect.

After further research, it looks like one can disable running tests on Hackage if you're authenticated

Yes, you can go into the "Test settings" page of a Hackage package and manually disable the tests that way. The downside to this approach is that it's manual—it's something that would have to be done every time a new version of copilot-bluespec is uploaded in the future. This really isn't ideal, as we'd prefer to have a solution that doesn't require extra manual effort.

Another possible solution would be to do a check for bsc at runtime in the test suite itself and still "passing" if bsc isn't found.

This doesn't sound like a very appealing solution, as it would be far too easy to accidentally misconfigure your PATH to not have bsc on it, at which point the test suite would pass without actually running any tests. This would be an especially easy mistake to make given that cabal test often omits the output of the test suite if it passes, which means that users might not have any idea that they're inadvertently skipping the tests!

[RyanGlScott]

Hah, @ivanperez-keera posted nearly the same comment moments before I did. Oops!

[chathhorn-galois]

Sounds good, thanks! I was trying to figure out if hackage would report NotTried or not for this and that dunai example looks pretty conclusive.