Add allow-head-failure-until input

[MSP-Greg]

Given the issues that occur with additions to bundled gems in ruby head builds, and (I hope) that many CI workflows are using Ruby head builds...

Add an input like allow-head-failure-until: <some easily parsed date format>, which setup-ruby could use to set an environment variable to be used in logic to allow head builds to fail, but only before the date?

IOW, an attempt to find a way to disable failures due to Ruby head builds, but only for a specified date period? Or, maintainers disable them to stop CI failures, but it's easy to forget to reenable them...

This is a bit messy regarding TruffleRuby & JRuby, which probably don't have the issues with bundled gems (I think). Maybe for now, don't include them in the 'date' calc?

[eregon]

Do such failures during the setup-ruby action?
Do you have an example log of such an issue?

If not during the setup-ruby action, of course we cannot influence other steps so that wouldn't work.

My other concern with this is there is no way to know when something will be fixed but the date would be mostly a guess and it's not unlikely it would fail again after that date.

[MSP-Greg]

Do such failures during the setup-ruby action?

No.

The failures I'm speaking of are failures like the one you looked into in Puma (thank you). Or, a std-lib item changing from default to bundled, causing build failures. There may also be times when ruby head causes unforeseen issues. Errors like

warning: base64 was loaded from the standard library, but is not part of the default gems since Ruby 3.4.0.
Add base64 to your Gemfile or gemspec.
warning: bigdecimal was loaded from the standard library, but is not part of the default gems since Ruby 3.4.0.
Add bigdecimal to your Gemfile or gemspec.
warning: mutex_m was loaded from the standard library, but is not part of the default gems since Ruby 3.4.0.
Add mutex_m to your Gemfile or gemspec.

Two assumptions:

1. We want people to test with Ruby head, and we prefer they don't set CI to 'allow-failure'.
2. Assume that failures are occurring. Currently, removing the job is the main option for getting CI passing. It's easy to forget to re-enable the job at a later time.

I thought of this when I noticed Minitest's #skip_until method.

Using that syntax in Actions, re the Puma error, we could set allow-head-failure-until: [2024, 2, 1], and the failing jobs would be ignored until 01-Feb-2024. Hopefully that would be enough time for the dependencies to be sorted out...

[MSP-Greg]

No, sorry, hopping between a few things.

All this would do is set an ENV variable like 'ALLOW_HEAD_FAILURE' so that later steps could determine whether to allow the CI job to fail, based on the date.

[eregon]

Ah I see, it'd be nice if https://docs.github.com/en/actions/learn-github-actions/expressions had a now() or something to do this directly in a step's if: or so.
Using allow-head-failure-until + using ALLOW_HEAD_FAILURE correctly seems not so easy.

[MSP-Greg]

Even with a now() or something, it wouldn't be aware of head builds.

For instance, with 3.2.3 released today, if it causes CI failures, I suspect most gems/repos would fix the problem quickly. Not sure if the response would be the same for head builds.

Using allow-head-failure-until + using ALLOW_HEAD_FAILURE correctly seems not so easy.

Let me look into it. Can't recall using any js date functions recently, but I'll see if I can get it to work. I can use the Puma workflow to test it...

[dentarg]

Sorry if off-topic, but re:

For step-level continue-on-error: nothing is shown and one needs to expand the log to see if tests passed or not, that's bad.

failed jobs are listed in the "Summary" view, that combined with logging the outcome of steps mitigates hunting for the error. I like this approach because the commits view will then be ✅ as that looks at individual jobs.

[eregon]

@dentarg Yes that's a useful trick. I think it still has the main downside though that if head builds are passing and a PR breaks them probably no one would notice (or much later) because one would need to open that job as it's not reflected at all on the job itself.

[dentarg]

I'm not sure this feature makes much sense / is worth it. I think all bets are off with ruby-head builds, here's an example where ruby-head was passing and then started to not pass: sinatra/sinatra#1988

I get that we want exposure on ruby-head failures but to me it is not worth having the front page of your project show red failing builds. IMHO maintainers and interested contributors can check the Summary page from time to time to find out if head are causing problems (#564 (reply in thread)).

[eregon]

I'm not sure this feature makes much sense / is worth it. I think all bets are off with ruby-head builds, here's an example where ruby-head was passing and then started to not pass: sinatra/sinatra#1988

Yeah, I kinda agree, failure is somewhat expected with head builds, at least some part of the time for ruby-head, given incompatible changes for the next version.

I get that we want exposure on ruby-head failures but to me it is not worth having the front page of your project show red failing builds.

Mmh, I see, and I guess moving head builds to daily doesn't actually solve that.
IMO it's fine to have head builds shown as red on the latest master commit and it's more annoying to show it as red on PRs as people might believe they broke something when it's not so likely.