Add warning for directory permission to in_tail.rb

[moatom]

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
This PR provides a warning for typical configuration mistakes that prevent log transfer from starting.

When using the tail type as a source, it is common for directories with inappropriate partitions to be mistakenly included in the source path.

In my opinion, it would be preferable to issue a warning in such cases to help users avoid these mistakes.

Steps to Reproduce:

$ sudo mkdir /var/log/noaccess
$ sudo mkdir /var/log/noaccess/noaccess2
$ sudo touch /var/log/noaccess/noaccess2/fluentd.log
$ sudo chmod 0000 /var/log/noaccess
$ sed -Ei 's|^([[:space:]]*)path .*|\1path /var/log/noaccess/noaccess2/fluentd.log|' example/in_tail.conf
$ bundle exec fluentd -c example/in_tail.conf
{no warning...}

Docs Changes:
No documentation changes required.

Release Note:
in_tail: add warning for directory permission.

[moatom]

Test / Ruby 3.4 on macos-latest (pull_request)Failing after 23m

https://github.com/fluent/fluentd/actions/runs/13865592763/job/38858881719?pr=4865#step:6:13416

4) Failure: test_unwatched_files_should_be_removed(TailInputTest::path)
/Users/runner/work/fluentd/fluentd/test/plugin/test_in_tail.rb:1757:in 'test_unwatched_files_should_be_removed'
1754: waiting(20) { sleep 0.1 until Dir.glob("#{@tmp_dir}/*.txt").size == 0 } # Ensure file is deleted on Windows
1755: waiting(5) { sleep 0.1 until d.instance.instance_variable_get(:@tails).keys.size <= 0 }
1756: => 1757: assert_equal(
1758: {
1759: files: [],
1760: tails: []
<{:files=>[], :tails=>[]}> expected but was
<{:files=>[],
:tails=>
["/Users/runner/work/fluentd/fluentd/test/plugin/../tmp/tail/6085abb32771a48075e1/tail.txt"]}>
diff:
? {:files=>[], :tails=>[]}
+ :tails=>
+ ["/Users/runner/work/fluentd/fluentd/test/plugin/../tmp/tail/6085abb32771a48075e1/tail.txt"]}
Error: <{:files=>[], :tails=>[]}> expected but was
<{:files=>[],
:tails=>
["/Users/runner/work/fluentd/fluentd/test/plugin/../tmp/tail/6085abb32771a48075e1/tail.txt"]}>.
diff:
- {:files=>[], :tails=>[]}
+ {:files=>[],
+ :tails=>
+ ["/Users/runner/work/fluentd/fluentd/test/plugin/../tmp/tail/6085abb32771a48075e1/tail.txt"]}

👀

[daipom]

Test / Ruby 3.4 on macos-latest (pull_request)Failing after 23m

https://github.com/fluent/fluentd/actions/runs/13865592763/job/38858881719?pr=4865#step:6:13416

👀

@moatom Thanks for this PR!
I will review this soon!

This test failure is a known test issue, so we don't need to care about it in this PR.

[daipom]

This PR provides a warning for typical configuration mistakes that prevent log transfer from starting.
When using the tail type as a source, it is common for directories with inappropriate partitions to be mistakenly included in the source path.

I see.

In some cases, Permission denied warning logs appear.
In other cases, they do not.

If the directory does not have the x permission, then warning logs do not appear.
This PR will solve that problem.

[daipom]

@moatom
Thanks for this PR!
The direction of checking the readability of the parent directory seems to have some effect.
However, I have some concerns.

• A: It does not seem to be able to support the case of using glob (such as *).
• B: We do not want to warn for paths that do not yet exist.

About A, it would be possible that this feature supports only static paths that do not use wildcards, etc.

About B, I feel the essential difficulty of this issue.
There could be cases where the mid-level parent directory, rather than the direct parent directory, does not have x permissions.
As long as there is no x permission, it would be difficult to check whether the path does not exist yet or whether the permission is inappropriate.
If we do this strictly, it will be necessary to find the directories whose existence can be verified and check their permissions.

I don't think it necessarily needs to be strictly confirmed.
However, it would be better to take these considerations into account and be clear about which cases are to be supported.

For example, I want to avoid false positives, such as generating warning logs for paths that just do not yet exist.

[daipom]

However, it would be better to take these considerations into account and be clear about which cases are to be supported.

For example, I want to avoid false positives, such as generating warning logs for paths that just do not yet exist.

To avoid false positives, we can consider the following direction for example.

• Instead of checking directory permissions, generate a warning log in the next process when the path does not exist.

fluentd/lib/fluent/plugin/in_tail.rb

Lines 372 to 374 in e3dcebc

	(paths - excluded).select{ |path|
	FileTest.exist?(path)
	}.each{ |path|

Log message:

... [warn] ... Unable to confirm the existence of #{path}.
You can ignore this warning log if the file is created later.
If the file already exists, please check the configuration and directory permissions.

(This supports only static paths that do not use wildcards, etc.)

[moatom]

@daipom Thanks for the review and I'm sorry for my late response🙇

If the directory does not have the x permission, then warning logs do not appear.

Oops. I thought the required permission was r. FileTest.readable?(dir) in the commit might not be appropriate...

(This supports only static paths that do not use wildcards, etc.)

The following part seems to expand the glob, so I believe this issue shouldn't occur (if my understanding is correct):

https://github.com/fluent/fluentd/pull/4865/files#diff-456fdeb51bc472beb48891caac0d063e0073655dba7ac2b72e6fdc67dc6ac802L333-R346

I don't think it necessarily needs to be strictly confirmed.
However, it would be better to take these considerations into account and be clear about which cases are to be supported.

For example, I want to avoid false positives, such as generating warning logs for paths that just do not yet exist.

I am polishing my idea as follows:

---

Experienced users are the ones who primarily use complex paths, and they are likely to manage to notice issues on their own. Therefore, in this PR, I would like to focus on adding a warning for the following simple case (if glob expansion does not occur, I want to ignore such cases):

​ /p_1/p_2/.../p_t/.../p_n

where

• p_i is a directory for 0<=i<=n-1
  • p_0 := / (the root directory)
• p_n is the log file
• p_t is the shallowest directory (resp. file, if t = n) that lacks the x (resp. r) permission in the path.

For this specific case, the ideal conditions for generating warnings can be summarized in the following table:

↓ All Permissions in the Path \ → File (p_n) Actually Exist	Exist	Not Exist
Granted (p_t doesn't exist)	○ (No Warning)	○ (No Warning)
Not Granted (p_t exists)	☓ (Warning about p_t)	☓ (Warning about p_t)

We are concerned about false positives for Granted×Not Exist (We need this test case).
Can we avoid it by not emitting warnings when we cannot detect p_t from p_0 .. p_{n-1}?

---

To avoid false positives, we can consider the following direction for example.

Log message:

... [warn] ... Unable to confirm the existence of #{path}.
You can ignore this warning log if the file is created later.
If the file already exists, please check the configuration and directory permissions.

I agree with this direction if the above approach is not effective!

[moatom]

(This supports only static paths that do not use wildcards, etc.)

The following part seems to expand the glob, so I believe this issue shouldn't occur (if my understanding is correct):

https://github.com/fluent/fluentd/pull/4865/files#diff-456fdeb51bc472beb48891caac0d063e0073655dba7ac2b72e6fdc67dc6ac802L333-R346

I might have missed cases where logs are emitted only at the debug level:
https://github.com/fluent/fluentd/pull/4865/files#diff-456fdeb51bc472beb48891caac0d063e0073655dba7ac2b72e6fdc67dc6ac802R352-R355

[daipom]

@moatom
Thanks for the update! I will see it soon.

(This supports only static paths that do not use wildcards, etc.)

The following part seems to expand the glob, so I believe this issue shouldn't occur (if my understanding is correct):

https://github.com/fluent/fluentd/pull/4865/files#diff-456fdeb51bc472beb48891caac0d063e0073655dba7ac2b72e6fdc67dc6ac802L333-R346

If there are not enough permissions for the directory, Dir.glob() does not return the files under that directory.
So, it would be very difficult to tell if the files do not exist or if there are not enough permissions.

[daipom]

Experienced users are the ones who primarily use complex paths, and they are likely to manage to notice issues on their own. Therefore, in this PR, I would like to focus on adding a warning for the following simple case (if glob expansion does not occur, I want to ignore such cases):

This would be no problem!

We are concerned about false positives for Granted×Not Exist (We need this test case).
Can we avoid it by not emitting warnings when we cannot detect p_t from p_0 .. p_{n-1}?

Yes. False positives are my concern.
break true unless FileTest.executable?(current_path) breaks when the current_path does not exist.
So, it causes false positives.

To avoid false positives, we can consider the following direction for example.

Log message:

... [warn] ... Unable to confirm the existence of #{path}.
You can ignore this warning log if the file is created later.
If the file already exists, please check the configuration and directory permissions.

I agree with this direction if the above approach is not effective!

If it is difficult to resolve this false positive, let's go in this direction.

I explain specific examples of the false positive below.

Case

For example, consider the following case.

$ sudo ls -lR /tmp/2/tmp/2:合計 12drwxrwx--- 2 root root 4096 3月 28 13:47 d---drwxrwx--x 2 root root 4096 3月 28 13:47 d--xdrwxrwxr-x 2 root root 4096 3月 28 13:46 dr-x/tmp/2/d---:合計 8-rw-r----- 1 root root 6 3月 28 13:47 file_---.log-rw-r--r-- 1 root root 5 3月 28 13:47 file_r--.log/tmp/2/d--x:合計 8-rw-r----- 1 root root 6 3月 28 13:47 file_---.log-rw-r--r-- 1 root root 5 3月 28 13:47 file_r--.log/tmp/2/dr-x:合計 8-rw-rw---- 1 root root 6 3月 28 13:46 file_---.log-rw-rw-r-- 1 root root 5 3月 28 13:37 file_r--.log

Let in_tail collect these 6 patterns and non-existent path (totally 7 paths).

<source>
@type tail
tag test
path /tmp/2/dr-x/file_r--.log, /tmp/2/dr-x/file_---.log, /tmp/2/d--x/file_r--.log, /tmp/2/d--x/file_---.log, /tmp/2/d---/file_r--.log, /tmp/2/d---/file_---.log, /tmp/2/not_exist/test.log
read_from_head true
refresh_interval 5s
<parse>
@type none
</parse>
</source>
<match test.**>
@type stdout
</match>

Result: master

path	read	warning
dr-x/file_r--.log	y	-
dr-x/file_---.log	n	y
d--x/file_r--.log	y	-
d--x/file_---.log	n	y
d---/file_r--.log	n	n
d---/file_---.log	n	n
not_exist/test.log	-	n

[info]: #0 following tail of /tmp/2/dr-x/file_r--.log
{datetime} test: {"message":"test"}
[info]: #0 following tail of /tmp/2/dr-x/file_---.log
[warn]: #0 Permission denied @ rb_sysopen - /tmp/2/dr-x/file_---.log
[info]: #0 following tail of /tmp/2/d--x/file_r--.log
{datetime} test: {"message":"test"}
[info]: #0 following tail of /tmp/2/d--x/file_---.log
[warn]: #0 Permission denied @ rb_sysopen - /tmp/2/d--x/file_---.log

Result: PR

path	read	warning
dr-x/file_r--.log	y	-
dr-x/file_---.log	n	y
d--x/file_r--.log	y	-
d--x/file_---.log	n	y
d---/file_r--.log	n	y
d---/file_---.log	n	y
not_exist/test.log	-	y (FP)

[warn]: #0 Skip /tmp/2/d---/file_r--.log because a directory in the path lacks execute permission.
[warn]: #0 Skip /tmp/2/d---/file_---.log because a directory in the path lacks execute permission.
[warn]: #0 Skip /tmp/2/not_exist/test.log because a directory in the path lacks execute permission.
[info]: #0 following tail of /tmp/2/dr-x/file_r--.log
{datetime} test: {"message":"test"}
[info]: #0 following tail of /tmp/2/dr-x/file_---.log
[warn]: #0 Permission denied @ rb_sysopen - /tmp/2/dr-x/file_---.log
[info]: #0 following tail of /tmp/2/d--x/file_r--.log
{datetime} test: {"message":"test"}
[info]: #0 following tail of /tmp/2/d--x/file_---.log
[warn]: #0 Permission denied @ rb_sysopen - /tmp/2/d--x/file_---.log

[moatom]

If it is difficult to resolve this false positive, let's go in this direction.

Thanks for explaining the case!
I'll try to investigate this when I have time.

I'm thinking it might work if I switch the traversal direction in the code above to top-down instead.
#4865 (comment)

[moatom]

@daipom
I have fixed this PR to work for all cases!🙏
If it looks good, should I squash the commits and add all 7 test cases to test/plugin/test_in_tail.rb?

P.S. I'm not sure, but it might be a false negative when even / lacks the required permissions in this approach. (Should I mention this in a comment?)

Result: Latest PR

<source>
@type tail
tag test
path /tmp/2/dr-x/file_r--.log, /tmp/2/dr-x/file_---.log, /tmp/2/d--x/file_r--.log, /tmp/2/d--x/file_---.log, /tmp/2/d---/file_r--.log, /tmp/2/d---/file_---.log, /tmp/2/not_exist/test.log
pos_file /tmp/fluentd-test.pos
read_from_head true
refresh_interval 5s
<parse>
@type none
</parse>
</source>
<match test.**>
@type stdout
</match>

path	read	warning
dr-x/file_r--.log	y	-
dr-x/file_---.log	n	y
d--x/file_r--.log	y	-
d--x/file_---.log	n	y
d---/file_r--.log	n	y
d---/file_---.log	n	y
not_exist/test.log	-	n

$ sudo ls -lR /tmp/2
/tmp/2:
total 12
drwxrwx--- 2 root root 4096 Apr 11 14:09 d---
drwxrwx--x 2 root root 4096 Apr 11 14:09 d--x
drwxrwxr-x 2 root root 4096 Apr 11 14:09 dr-x
/tmp/2/d---:
total 8
-rw-r----- 1 root root 5 Apr 11 14:11 file_---.log
-rw-r--r-- 1 root root 5 Apr 11 14:11 file_r--.log
/tmp/2/d--x:
total 8
-rw-r----- 1 root root 5 Apr 11 14:11 file_---.log
-rw-r--r-- 1 root root 5 Apr 11 14:11 file_r--.log
/tmp/2/dr-x:
total 8
-rw-rw---- 1 root root 5 Apr 11 14:11 file_---.log
-rw-rw-r-- 1 root root 5 Apr 11 14:11 file_r--.log
$ bundle exec fluentd -c in_tail.conf
025-04-11 15:44:22 +0000 [warn]: #0 Skip /tmp/2/d---/file_r--.log because '/tmp/2/d---' lacks execute permission.
2025-04-11 15:44:22 +0000 [warn]: #0 Skip /tmp/2/d---/file_---.log because '/tmp/2/d---' lacks execute permission.
2025-04-11 15:44:22 +0000 [info]: #0 following tail of /tmp/2/dr-x/file_r--.log
2025-04-11 15:44:22.441945749 +0000 test: {"message":"test"}
2025-04-11 15:44:22 +0000 [info]: #0 following tail of /tmp/2/dr-x/file_---.log
2025-04-11 15:44:22 +0000 [warn]: #0 Permission denied @ rb_sysopen - /tmp/2/dr-x/file_---.log
2025-04-11 15:44:22 +0000 [info]: #0 following tail of /tmp/2/d--x/file_r--.log
2025-04-11 15:44:22.442540302 +0000 test: {"message":"test"}
2025-04-11 15:44:22 +0000 [info]: #0 following tail of /tmp/2/d--x/file_---.log
2025-04-11 15:44:22 +0000 [warn]: #0 Permission denied @ rb_sysopen - /tmp/2/d--x/file_---.log

[daipom]

@moatom Thanks! Excellent! This direction looks good to me!
I will comment on some minor code improvements later!
We may need to consider how it works on Windows.

If it looks good, should I squash the commits and add all 7 test cases to test/plugin/test_in_tail.rb

Sqush would be good!
For test cases, the current two would be sufficient.

P.S. I'm not sure, but it might be a false negative when even / lacks the required permissions in this approach. (Should I mention this in a comment?)

Do you mean the case where we just specify / as path?

[daipom]

I have commented on some minor points.
Please check them!

[moatom]

Do you mean the case where we just specify / as path?

I was referring to the case where the path is /not_exist_file.log, and the root directory / lacks the permissions.

In my PR, the logic relies on finding the first directory in the path hierarchy for which !p.executable? holds. However, in the situation where / itself lacks the permissions, it is uncertain whether p.directory? can reliably determine if / is a directory in the first place.

This scenario is rare and it’s unclear whether such a case should be officially supported.

[moatom]

@daipom
Hi! I've fixed all the points.
If everything looks good to you, I'll squash the commits.

[daipom]

Thanks! Sorry for the delay, I am seeing this. Please wait some more.

[daipom]

The Windows CI failure is unrelated.

[daipom]

@moatom
Sorry for the repeated reviews.
After we fix this, I think we can merge this!

Review points:

• The log accessor is provided for plugin class, so we should use it in plugin implementations.
  • We should not use the global logger in plugins. Some existing codes are wrong... 😢
  • Sorry I didn't notice this before.
• In tests, we can use Driver#logs to get the logs.
  • If the plugin uses the global logger, this cannot be used.
  • We want to avoid using the global logger for tests as much as possible.
• We don't need to call Driver#instance_shutdown manually unless we call Driver#run with shutdown: false.

[moatom]

@daipom

Sorry for the repeated reviews.

No problem at all. I really appreciate your prompt responses!

After we fix this, I think we can merge this!

I've fixed the points. Could you confirm at your convenience?

[daipom]

LGTM! Thanks so much for this improvement!