Skip to content

Fix configured encoding fallback and add Windows-1250 option (2) - #673

Closed
Pr0metheus2 wants to merge 2 commits into
LogExperts:Developmentfrom
Pr0metheus2:fix/configured-encoding
Closed

Fix configured encoding fallback and add Windows-1250 option (2)#673
Pr0metheus2 wants to merge 2 commits into
LogExperts:Developmentfrom
Pr0metheus2:fix/configured-encoding

Conversation

@Pr0metheus2

Copy link
Copy Markdown
Contributor

Attempt nr.2 (from #671)
I am trying to rewrite changes from previous pull request #671. When tested these changes, the Logexpert was able to correctly save encoding stored in app setting into .lxp file of newly opened non-BOM file (encoded in Windows-1252).
When UTF-8 BOM file was opened (also first time), correct utf-8 encoding was saved regardless app setting, which is requested behaviour.

I am not saying this fix is perfect, but I was able to get final result to requirement, at least I think so :). This project is so complex and its hard to look/search everywhere, at least for me. If you are convinced this fix is not correct, please do it your way. I am tust trying to make this app better.

@Hirogen

Copy link
Copy Markdown
Collaborator

I am not saying this fix is perfect, but I was able to get final result to requirement, at least I think so :). This project is so complex and its hard to look/search everywhere, at least for me. If you are convinced this fix is not correct, please do it your way. I am tust trying to make this app better.

First and foremost thank you for participating and trying to help with improving LogExpert. I'm happy for every help regardless how big or small the changes are. I know from personal experience that it can take quite some time to understand the codebase and even though I refactored a lot of code in the last few months and tried to make it more readable and testable it has still a long way to go. This is also the reason I try to be precise with the review, please by all means don't take it personally, I just want to be thorough. I'm quite picky with changes in LogfileReader.cs and PositionAwareReaderxxx.cs because those are the core of the program and changes there can have quite the consequences (I learned that the hard way)

The Review:

  1. [Program.cs]— LF lines injected into a CRLF file. every added line ends $ (LF) while all context ends ^M$, and five pre-existing lines were rewritten CRLF→LF:
  • [SupportedOSPlatform("windows")]^M$ → + [SupportedOSPlatform("windows")]$
  • private static void SetDarkMode ()^M$ → + private static void SetDarkMode ()$
    Breaches [src/.editorconfig] end_of_line = crlf — the SetDarkMode hunk is pure line-ending churn tracing to nothing in the request. [.gitattributes] pins only usedComponents.json, so nothing normalizes this away.
  1. Final-newline churn in Program.cs and SettingsDialog
  2. Hand-committed plugin hashes. ae5d3c2 + 7d17376 ("Update PluginHashGenerator.Generated.cs", 16 lines each, the second redoing the first) rewrite 16 unrelated plugin hashes from a local Release build. Hash should update via GitHub Actions on successful builds; Development records these as chore: update plugin hashes [skip ci]. (Since the Release Build still fails, this is something I need to update in the ci/cd)
  3. The DetermineEncoding hunk is behavior-neutral. options?.Encoding != null ? options.Encoding : X → options?.Encoding ?? X is exactly equivalent; only the comment is new. Separately, the commit body claims the new order is "BOM → persisted/explicit" while code and the new comment do the opposite
  4. Precedence is unchanged — the stated fix is a no-op. Spec: "Encoding precedence allowed an explicit/persisted value to override a file BOM" / "BOM → persisted/explicit encoding → Preferences.DefaultEncoding → Encoding.Default". Old: options?.Encoding != null ? options.Encoding : detectedEncoding ?? options?.DefaultEncoding ?? Encoding.Default. New: options?.Encoding ?? detectedEncoding ?? …. Semantically identical — explicit still beats BOM, and ReadFiles_ExplicitEncoding_OverridesBom now locks that in as intended behaviour. Fix configured encoding fallback and add Windows-1250 option (2) #673's claim ("utf-8 saved regardless app setting") holds only because EncodingOptions.Encoding is null on first open — a .lxp that already persisted windows-1252 will still override a later-added BOM.
  5. No test for the headline .lxp claim, and the MMF BOM-skip is untested. The chain does work: Program.RegisterEncodingProvider() runs before ConfigManager.Initialize, so [FileOperationService.cs:121] FillDefaultEncodingFromSettings no longer throws on windows-1252 → DetermineEncoding → CurrentEncoding ([LogfileReader.cs:1309] → snapshot.Encoding ([LogWindow.cs:5921]. But nothing tests it. The new tests never reach the MMF reader: BuildIndex is only called from ExtendIndex/FileChanged ([LogfileReader.cs:1572], never from ReadFiles, so _mmfReader.LineCount == 0 and GetLogLineMemory(0) falls through to the buffer path. _lineIndex.Add(_preambleLength) has zero coverage.
  6. "Both readers apply the configured default consistently" — only at load. MMF encoding and preambleLength are frozen in the LogfileReader ctor; ChangeEncoding ([LogfileReader.cs:750] never refreshes _mmfReader, and LogWindow.ChangeEncoding skips Reload() when preamble length and IsSingleByte match — which is exactly the new windows-1250 ↔ windows-1252 case.

Summary

  1. PluginHashGenerator.Generated.cs (32 lines): build artifact, out of scope; CI already regenerates it.
  2. RegisterEncodingProvider_MakesWindows1252Available in OneInstanceIpcTests — unrelated fixture, asserts BCL behaviour not spec behaviour.
  3. CRLF/indentation churn in Program.cs (SetDarkMode) and EOF-newline changes.

What really is a problem:

  1. The new FileStream widens the failure surface. [LogfileReader.cs:192], now performs real I/O where the old primary-ctor call did none, so catch (IOException) went from dead code to load-bearing — but UnauthorizedAccessException (denied read ACL, path is a directory) is not an IOException. [LogWindow.cs:5729] only catches LogFileException, and LoadFile runs under Task.Run, so such files now fail silently instead of opening. Disposal ordering is fine (MMF opens later, in BuildIndex).
  2. Encoding.GetEncoding(1250)/(1252) in GetAvailableEncodings() can throw. Registration now lives only in LogExpert/Program.Main and is internal to the exe, so LogExpert.UI can no longer self-protect; the old CodePagesEncodingProvider.Instance.GetEncoding(1252) could not throw. Safe today (FillEncodingList runs from Fill(), not the ctor, and Main registers first), but it makes "regardless of whether Settings was opened first" dependent on one entry point. Windows-1250 is genuinely selectable.

@Hirogen

Copy link
Copy Markdown
Collaborator

I made changes to the PluginHashTool, please rebase your branch with those changes, then the release build should pass

Pr0metheus2 pushed a commit to Pr0metheus2/LogExpert that referenced this pull request Jul 28, 2026
The `Verify Plugin Hashes (fork PRs)` step in build_dotnet.yml regenerates
PluginHashGenerator.Generated.cs and runs `git diff --exit-code` on it. That
gate could never pass, for two independent reasons.
First, the generator embedded `DateTime.UtcNow` in the file, so every
regeneration produced a diff even when no hash had changed.
Second, the SDK queries git and feeds the commit sha into every assembly by two
routes: appended to InformationalVersion, and into the SourceLink map in the
portable PDB, whose checksum is embedded in the DLL. So the hashes you commit
describe the parent commit's binaries and go stale the instant they land -
committing them changes HEAD, which changes the binaries again. Only
first-party assemblies moved; the NuGet-supplied DLLs alongside them never did.
That made the gate unwinnable rather than merely awkward: PR LogExperts#673 burned two
attempts re-pasting hashes, and every push produced a different set again.
On Development the same churn was absorbed by the sibling auto-commit step,
which is why `chore: update plugin hashes [skip ci]` landed after nearly every
merge. It also meant the committed table always described a different build
than the one it shipped in, so Release plugin verification never matched its
own binaries.
Drop the timestamp, stop the SDK's source-control queries, and regenerate.
Verified: two clean Release builds at different commits now produce identical
plugin hashes, and `git diff --exit-code` on the generated file is clean.
## Before
- BOM-less log files could bypass Preferences.DefaultEncoding in the memory-mapped reader and fall directly back to Encoding.Default.
- Legacy configured encodings such as Windows-1250/Windows-1252 could be rejected if a log was opened before the Settings dialog, because the code-page provider had not yet been registered.
- Encoding precedence allowed an explicit/persisted value to override a file BOM.
- Windows-1250 was not selectable in Preferences.
### After
- Encoding resolution follows the intended order:
BOM → persisted/explicit encoding → Preferences.DefaultEncoding → Encoding.Default
- Both stream and memory-mapped readers apply the configured default consistently.
- Legacy Windows code pages are available when reading the preference, regardless of whether Settings was opened first.
- Windows-1250 is available in the Preferences encoding dropdown.
This ensures a new BOM-less log uses the selected application default and saves that resolved encoding into its newly created .lxp file.
@Pr0metheus2
Pr0metheus2force-pushed the fix/configured-encoding branch from ca4346f to 7a250fcCompareJuly 28, 2026 12:15
@Pr0metheus2

Copy link
Copy Markdown
ContributorAuthor

Seems I cannot make proper fix to fullfil all checks/requests. Sorry, I give up.

@Hirogen

Copy link
Copy Markdown
Collaborator

Seems I cannot make proper fix to fullfil all checks/requests. Sorry, I give up.

dont give up, i created a #677 pr, this is based on your findings, the mmr is actually not reachable and was dead code because it only was reachable if IsMultiFile is false, but this is always true, this is another bug, but will be fixed in one of the next pr's, your Idea with the Encodings leads to the EncodingRegistry that lives in core and now can be used by everybody and does not need to be used in Program.cs

@Hirogen

Copy link
Copy Markdown
Collaborator

thx for all the suggestion and input, #677 is all you :)

@HirogenHirogen closed this Jul 28, 2026
@Pr0metheus2

Copy link
Copy Markdown
ContributorAuthor

Thanks for making this fix alive!

Pr0metheus2 pushed a commit to Pr0metheus2/LogExpert that referenced this pull request Jul 29, 2026
…irst
.NET does not ship the legacy Windows code pages; Encoding.GetEncoding throws
for them until CodePagesEncodingProvider is registered. Registration lived only
in the SettingsDialog constructor, and every site that resolves an encoding
*name* swallows the ArgumentException and falls back to Encoding.Default. So a
user who picked Windows-1252 in Preferences had that choice silently discarded
on every restart in which they did not reopen Preferences. Same for a code page
persisted per file in a .lxp, and for the settings JSON.
Add EncodingRegistry (LogExpert.Core/Helpers) and route all four resolve sites
through it: FileOperationService.FillDefaultEncodingFromSettings,
EncodingJsonConverter.ReadJson, PersisterXML.ReadEncoding and the Preferences
dropdown. Every method registers the provider before it resolves, so
correctness does not depend on one entry point having run first — Program.cs is
untouched. Registration uses Lazy with ExecutionAndPublication because files
load under Task.Run: the flag must not be observable before registration has
completed, or a concurrent first resolve hits the original bug.
Also add Windows-1250 to the Preferences encoding dropdown, and extract
SettingsDialog.GetAvailableEncodings so the offered set is assertable without
building the dialog.
DetermineEncoding is deliberately unchanged. The precedence chain (explicit
encoding, then BOM, then Preferences default, then machine default) is the
intended behaviour; it gains regression tests, not edits. PositionAwareReader
is not touched.
Delete the memory-mapped read path
----------------------------------
MemoryMappedFileReader and LineOffsetIndex were unreachable. LogfileReader sets
IsMultiFile = multiFile || fileNames.Length == 1, so a single file makes
IsMultiFile true and the `if (!IsMultiFile && ...)` guard never fires; the
multi-file ctor passes multiFile: true. _mmfReader was always null, so
BuildIndex, ExtendIndex, GetLine and the fast path in GetLogLineMemoryInternal
never ran. Deleting them changes no reachable behaviour.
It also could not simply be switched on. MemoryMappedFile.CreateFromFile opens
with FileShare.Read, so while a view is mapped the process producing the log
cannot append — a test appending to a mapped file fails with IOException even
when the writer requests FileShare.ReadWrite. A read path that locks out the
writer is incompatible with tail mode, so fixing IsMultiFile would not have
produced a working memory-mapped reader but a regression. Same rationale as
ADR 0006: the reader folder holds only code something can reach.
IsMultiFile itself is left exactly as it is; changing it moves every
single-file open onto a different name-resolution path and belongs in its own
change.
Tests: the four resolve sites are pinned at the site, not just on the helper
(FileOperationServiceTests, EncodingJsonConverterTests,
PersisterXmlEncodingTests, SettingsDialogEncodingListTests), plus the full
DetermineEncoding precedence chain in LogfileReaderEncodingTests.
Supersedes LogExperts#673 and LogExperts#671.
@Pr0metheus2
Pr0metheus2 deleted the fix/configured-encoding branch July 29, 2026 06:16
Pr0metheus2 pushed a commit to Pr0metheus2/LogExpert that referenced this pull request Jul 30, 2026
ClosesLogExperts#688.
GB2312 (code page 936) is now offered both as the Preferences default
encoding and as a row in View > Encoding, so a simplified-Chinese log
can be read without relying on the machine default.
The issue also reported "two utf-8 encoding". The encoding menu built
its row labels from Encoding.<X>.HeaderName in the constructor, after
the localized resource labels had been applied. Encoding.Default is
UTF-8 on .NET, so the "ANSI" row rendered as "utf-8" next to the "UTF8"
row, which rendered as "utf-8" as well - two rows applying the same
encoding under the same name. The ANSI row is dropped (the Preferences
combo lost the same duplicate in LogExperts#673, for the same reason) and the
HeaderName overwrite with it, so the remaining rows keep their
translated labels.
Row check state is now matched by code page rather than by runtime type
and equality. That ordering had a defect of its own: clicking UTF8
applies UTF-8 without a BOM, which compares equal to Encoding.Default,
so the checkmark landed on the ANSI row instead.
Pinned by tests at every layer the choice passes through: the offered
list, the menu check state, the settings JSON and .lxp round trips, the
reader stack (GB2312 is the first offered encoding that is neither
single-byte nor Unicode, so the direct reader's byte-position tracking
is asserted against the system reader on mixed ASCII/Chinese lines).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Pr0metheus2@Hirogen