Enable sourcelink for Visual fsharp tools repo

[KevinRansom]

Enable sourcelink for Visual F# tools repo.

1. Sourcelink is now enabled by default. set an environment variable
   set usesourcelink=false to turn it off
2. Release build now builds with portable pdb and sourcelink.
3. Debug build builds with portable pdb and sourcelink. + optimizations

Note: In order
Kevin

[KevinRansom]

@dotnet-bot test this please

[mmitche]

@dotnet-bot test this please

[KevinRansom]

@dotnet-bot test this please

[KevinRansom]

@dotnet-bot test this please

[ctaggart]

@KevinRansom What are your thoughts on using a debug type of embedded for the tools? We see only about a 20% increase in dll files, but it guarantees the debug info and source link.

[KevinRansom]

@ctaggart not embed in release builds. The working set of released product is important to minimize. Sourcelink works very well and only has a teeny tiny overhead. switching to portable PDBs is like getting free working set given how immense the old symbol files were.

Embedding symbols and not source may be interesting for nuget deployment I think we will need some experience to see what works best in that scenario.

[KevinRansom]

There is a coreclr bug that impacts this:

https://github.com/dotnet/coreclr/issues/3541 : JITTrackingEnabled on coreclr making internals visible to not work correctly. I expect it to be fixed in .NET Standard 2.0

Mainly this affects debugging the coreclr version of the FSC compiler. I'm going to see if I can find a work around.

[KevinRansom]

@dotnet-bot test this please

[KevinRansom]

Good lord, what has happened to the build?

[brettfo]

I'm concerned about adding a dependency to an alpha release.

[brettfo]

As it currently stands, it doesn't look like source link will do what you think:

1. Our builds come out of a private branch with master simply merged in so the git SHA will only be valid against our internal copy of the repo; it won't mean anything to GitHub.
2. The build system passes the git SHA via the %BUILD_SOURCEVERSION% environment variable and I don't see that being referenced in any of our files.

[KevinRansom]

@brett, does that mean that the internal repo doesn't have the sha's from GitHub? I figured once this was in, we could figure out the exact magic to get the correct sha.

[ctaggart]

I just downloaded an fsc.pdb from this build. You can use a tool I built to test it and look at it.

C:\Users\CameronTaggart\cs\sourcelink-test\ClassLibrary1 [fsharp ≡ +0 ~2 -0 !]> dotnet sourcelink
SourceLink 2.1.0-b446
Source Code On Demand
Usage: [options] [command]
Options:
-h|--help Show help information
Commands:
print-documents print the documents stored in the pdb or dll
print-json print the Source Link JSON stored in the pdb or dll
print-urls print the URLs for each document based on the Source Link JSON
test test each URL and verify that the checksums match
Use " [command] --help" for more information about a command.
C:\Users\CameronTaggart\cs\sourcelink-test\ClassLibrary1 [fsharp ≡ +0 ~2 -0 !]> dotnet sourcelink print-json C:\Users\Ca
meronTaggart\Downloads\fsc.pdb
{"documents": { "D:\\j\\workspace\\release_ci_pa---866fd2c3\\*" : "https://raw.githubusercontent.com/Microsoft/visualfsh
arp/532a5b763fc18fd9bd4795077f503a1a099dff8c/*" }}
C:\Users\CameronTaggart\cs\sourcelink-test\ClassLibrary1 [fsharp ≡ +0 ~2 -0 !]> dotnet sourcelink print-documents C:\Use
rs\CameronTaggart\Downloads\fsc.pdb
10db868b4c84ef9b97a08a4d2bc1ea94 md5 fsharp D:\j\workspace\release_ci_pa---866fd2c3\src\fsharp\fscmain.fs
C:\Users\CameronTaggart\cs\sourcelink-test\ClassLibrary1 [fsharp ≡ +0 ~2 -0 !]> dotnet sourcelink print-urls C:\Users\Ca
meronTaggart\Downloads\fsc.pdb
10db868b4c84ef9b97a08a4d2bc1ea94 md5 fsharp D:\j\workspace\release_ci_pa---866fd2c3\src\fsharp\fscmain.fs
https://raw.githubusercontent.com/Microsoft/visualfsharp/532a5b763fc18fd9bd4795077f503a1a099dff8c/src/fsharp/fscmain.fs
C:\Users\CameronTaggart\cs\sourcelink-test\ClassLibrary1 [fsharp ≡ +0 ~2 -0 !]> dotnet sourcelink test C:\Users\CameronT
aggart\Downloads\fsc.pdb
1 Documents with errors:
10db868b4c84ef9b97a08a4d2bc1ea94 md5 fsharp D:\j\workspace\release_ci_pa---866fd2c3\src\fsharp\fscmain.fs
https://raw.githubusercontent.com/Microsoft/visualfsharp/532a5b763fc18fd9bd4795077f503a1a099dff8c/src/fsharp/fscmain.fs
error: url hash does not match: 10db868b4c84ef9b97a08a4d2bc1ea94
sourcelink test failed

The commit hash works. The file hash does not. Guessing line endings.

[ctaggart]

Confirmed that is line endings. I downloaded the file from that URL, calculated the checksum, opened up the file in VS Code, changed the line endings to CRLF and then calculated it again. Git repos store the files as LF, so you must compile as LF to source linking to work. This should be easy.

C:\Users\CameronTaggart\Downloads> Get-FileHash .\fscmain.fs -Algorithm md5
Algorithm Hash Path
--------- ---- ----
MD5 B61F1298FB0798601F310A4BBFC51542 C:\Users\CameronTaggart\Downl...
C:\Users\CameronTaggart\Downloads> code .\fscmain.fs
C:\Users\CameronTaggart\Downloads> Get-FileHash .\fscmain.fs -Algorithm md5
Algorithm Hash Path
--------- ---- ----
MD5 10DB868B4C84EF9B97A08A4D2BC1EA94 C:\Users\CameronTaggart\Downl...

Hopefully adding something like

git config --global core.autocrlf input

somewhere in the netci.groovy file is simple. @mmitche ?

[mmitche]

@ctaggart All the git installations are on auto (not specified in the groovy file, though theoretically you could set the repo's git config before running any commands). This should be fixed on the repo side if possible since auto is what almost every developer is using too.

[KevinRansom]

@ctaggart, apparently VSCode hashes with both line endings in-order to figure things out, and VS needs a similar change.
I chatted with Tomas and he thinks that there is not really much the compiler can do to solve this. Picking hashing with line endings different than the local file, would break VS locally. Forcing the repo to a particular set of line endings, doesn't really solve the problem, it just makes our distribution work.

Also, we are still building our distributions on a VSO repo, which merges localization files each night and so the hash we use is not available on GitHub. There are hacks we can do to fix that, however, they give @brettfo nightmares, even to think about them. In order to allow him to be awake in the day, I'm going to not push for the hacks, just yet. He is confident that we will not have the closed source loc step for much longer, so the issue will go away.

[ctaggart]

Forcing the repo to a particular set of line endings, doesn't really solve the problem, it just makes our distribution work.

@KevinRansom You don't force the repo to a set of line endings. You simply have the git client on the build server not change the line endings. The setting that tells git to clone on Windows without changing the files is core.autocrlf input.