Binary articles subjected to expensive EMP Body checks

[jrehmer]

Correctly placed binary articles are going through the EMP Body filter. Oddly, two machines both running the same pyClean code produce different results, where one occasionally rejects binaries due to the EMP Body filter, and one that does not. A few headers below for reference, should there be something particular about these articles causing the issue.

I cannot explain why two machines being fed the same articles produce different results, but regardless it seems unnecessary to further check binary articles. I see I can add patterns to 'emp_exclude' to prevent this behavior, but thought I would bring it up as doing additional checks on binary articles seems expensive, especially on servers seeing high volume.

2022-06-21 19:58:27 INFO reject: mid=part74of78.Iia$a1FHo5f0nTlMXk&1@bloodnut.local, reason=EMP Body Reject
2022-06-21 19:58:36 INFO reject: mid=part78of78.Iia$a1FHo5f0nTlMXk&1@bloodnut.local, reason=EMP Body Reject

[Julien-Elie]

Maybe the moment when your servers trim the EMP list is not the same, with a few minutes of difference?
Anyway, by-passing the EMP filter for correctly placed binaries is of course possible (by default or with a new parameter).

[jrehmer]

This was more a suggestion at including the bin_allowed patterns to emp_exclude in the stock/default configuration. I figured it may have been an oversight not to have them excluded from the EMP filter by default.

[Julien-Elie]

I've had a look at what Cleanfeed does, and I am under the impression it runs the EMP filter even for articles in binary newsgroups, so it may not have been an oversight.
Cleanfeed skips the check when the number of lines is superior to md5_max_length (2000 by default).

I have no real opinion about that. Is PyClean noticely slowing down your server when running the EMP filter on binaries?

[jrehmer]

I'm carrying 5 binary groups of low frequency traffic, so I don't have a good measure for CPU consumption. In my case it is not very noticeable, but I could see where larger volumes of traffic would consume considerable CPU time unnecessarily.

What I observed when the EMP is filtering binary articles is that unique files after being encoding and split into many parts, some parts or articles in this case have an identical encoded body.

Probably unrelated, but another reason I wanted to exclude binaries from as many checks as possible is that I observed under heavy load (feeding another server's spool into a new one, for example), either INN is failing to pass all articles through the python filter or pyClean is dropping articles without checking as some are passing through that should not.

I've been able to determine this was happening by checking for the existence of articles I know should be filtered out (bad_subject and bad_from matches, for example). During the events of high load a large volume of binary articles were involved. Since adding the binary patterns to emp_exclude I haven't noticed anything getting past the filter that shouldn't.

[Julien-Elie]

The fact that the server is under heavy load should not alter how filtering is done. If INN fails to pass all articles through filters or if filters fail to check articles, it is not normal. Filtering would just be slower, and work as expected. There may be a bug there. Also, please note that if the Python filter raises an uncaught exception and dies (bug), the article will be accepted as well as subsequent articles until the filter is reloaded. It may be what you faced with.

It could be interesting to have a look in your pyclean/log/traceback and close_traceback files to see whether something is logged during high load.

[jrehmer]

I discovered today that, at least on my server, emp_exclude is not being honored at all on my server. While using suck to pull in news.answers from another server almost all of the messages are rejected even though '^news.answers' appears in the emp_exclude definition.