Add support for multiple output processing modes

[csanders-git]

In some cases we'd like to use an OR instead of an AND.

So for instance if Apache blocks something we'd like to support something like status 400 OR the log_Contains this.

right now something like the following

            output:
                  status: [200,400]
                  log_contains: "id \"920274\""

will always result in checking if it's 200 or 400 AND log_contains.

[fgsch]

What about:

output:
  any: 
    - status: [200, 400]
    - log_contains: id "1234"

This will open the format to also do:

output:
  all:
    - status: [200, 400]
    - log_contains: id "1234"

and:

output:
  none:
    - status: [200, 400]
    - log_contains: id "1234"

If you let me know I could open a PR.

[zmallen]

Seems like a good idea. Do we foresee any other logic in place where multiple of the any/all/none will be needed?

[fgsch]

I don't foresee the need for multiple entries but one of the ideas I entertained is to be able to specify whether a condition is limited to a particular server (e.g. apache). Leaving aside how we can detect or pass this information down to the tests, we could extend the format above to something like this:

output:
  any:
    - status: [200, 400]
    - log_contains: id "1234"
      if: server = "apache"

[zmallen]

I personally think we should avoid running conditionals like that. If its an apache test, perhaps we can just have a separate apache yml file?

I only say this as it could be untenable supporting regressions in logic that fingerprints a server.

Love your idea of any/all/none, though!

[csanders-git]

@zmallen based on the amount of tests we have, splitting out files per WAF would be pretty intense.

[fgsch]

IMO there is value in having a single test covering multiple implementations.

I mentioned "leaving aside how we can detect or pass this information down" and I don't expect ftw to fingerprint the server. I was thinking more around the lines of invoking ftw and passing a particular server to test. For CRS this would translate roughly to py.test --server apache2 .. and if a condition is present in the output check if it matches before evaluating it.

[theseion]

No longer necessary with platform overrides.