FIle count limit?

[gvarela]

Describe the bug
I have a very large codebase that I was trying to vectorise. I get a OSError: [Errno 24] Too many open files: error (plus a lot of follow on errors) . Is there a limit or a recommendation on how to handle larger legacy codebases?

[Davidyz]

Hi, do you know how many files are in the repo? If it's open source could you provide an URL? Also, is there a traceback for the error that shows where the error came from? I'm currently developing on a very weak laptop and querying gets intolerably slow when I use this on a large codebase so I've never run into this myself, but with more details I can investigate.

[gvarela]

error.log

It's a legacy rails app, 2650 files in our models directory alone. May not be the right use case.

[Davidyz]

Hi, I believe this is caused by a limit imposed by the OS on how many files can be opened at the same time. I used async IO to avoid overhead, but apparently I should've implemented some sort of task pool to restrict the number of opened files. I'll fix this asap.

[Davidyz]

Please try again with the latest release(0.4.3), and feel free to re-open this issue if the error persists. Also if it works, I'd love to know how long it takes to complete a query (and your hardware specs too, if you don't mind).

[gvarela]

Thanks it's fixed! I indexed our app directory just the *.rb files. Took a few minutes to index and the queries don't feel too slow.

[gvarela]

I should mention I just queried in the terminal. Still working on getting it wired up correctly with CodeCompanion.

[gvarela]

Ok, getting retrieval failed errors in my nvim config on the big repos and it works fine on a smaller one. I will probably have to be more specific with indexing what I need to work on.