[BUG]: Non-Latin words are case-sensitive

[winterwolf]

Issue Description

Non-Latin words are case-sensitive

Operating System

Linux (Arch)

Editor

VSCode

Codebook Version

0.3.37

Configuration

dictionaries = ["en_us", "en_gb", "ru"]
words = [
    "cachy",
    "cwebp",
    "konsave",
    "magick",
    "numpad",
    "nvidia",
    "pacman",
    "ГБ",
    "ИИ",
    "Линкуем",
    "апгрейдить",         # Апгрейдить апгрейдить
    "виндовом",           # Виндовом виндовом 
    "дефолтный",
    "дотфайлов",
    "дотфайлы",
    "коммиты",
    "конфига",
    "кроссплатформенный",
    "лаунчеры",
    "линкуем",
    "линкует",
    "мультипечь",
    "нахер",
    "неинтерактивная",
    "парсит",
    "перелинковать",
    "перелинковки",
    "перелогиньтесь",
    "пересоздаёт",
    "пиксельный",
    "подмодулем",
    "подмодуль",
    "подмодуля",
    "подмодулями",
    "прилинкуйте",
    "рассчёта",
    "репозитории",
    "репозитория",
    "сертралин",
    "симлинк",
    "симлинка",
    "симлинки",
    "симлинком",
    "херню",
    "эмодзи",
    "ютуба",
]
flag_words = ["todo", "fixme"]
include_paths = ["**"]
ignore_paths = ["target/**/*", "**/*.json", ".git/**/*"]
ignore_patterns = [
    '\b[ATCG]+\b',
    '\d{3}-\d{2}-\d{4}',
    "^[A-Z]{2,}$",
    'https?://[^\s]+',
]
include_tags = ["comment", "string"]
exclude_tags = ["string.heredoc"]

Steps to Reproduce

Expected Behavior

The case should not be taken into account.

Actual Behavior

English words are automatically converted to lowercase when added to the dictionary, but Russian words doesnt.

Code Sample

Log Output

Additional Context

No response

[blopker]

Interesting, I'd have to refresh my memory, but my understanding is that non Latin letters are not as easy to convert case with. I'd assume this is not a Russian-only issue.

I'd bet an LLM could find and fix the issue pretty quickly though, all this logic is in one place.