AsciiUtilityTests failing in CI and PR

[jaredpar]

Specifically the following tests:

• System.Text.Tests.AsciiUtilityTests.GetIndexOfFirstNonAsciiByte_Vector128InnerLoop
• System.Text.Tests.AsciiUtilityTests.GetIndexOfFirstNonAsciiChar_Boundaries
• System.Tests.UriCreateStringTests.Scheme_Authority_IdnHost
• System.Tests.UriMethodTests.GetComponents

Console Log Summary

Generally the stack trace is the following:

 System.Text.Tests.AsciiUtilityTests.GetIndexOfFirstNonAsciiByte_Vector128InnerLoop [FAIL]
 System.PlatformNotSupportedException : Operation is not supported on this platform.
 Stack Trace:
 /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Intrinsics/X86/Sse2.cs(673,0): at System.Runtime.Intrinsics.X86.Sse2.LoadVector128(Byte* address)
 /_/src/libraries/System.Private.CoreLib/src/System/Text/ASCIIUtility.cs(494,0): at System.Text.ASCIIUtility.GetIndexOfFirstNonAsciiByte_Sse2(Byte* pBuffer, UInt64 bufferLength)
 /_/src/libraries/System.Runtime/tests/System/Text/ASCIIUtilityTests.cs(394,0): at System.Text.Tests.AsciiUtilityTests.CallGetIndexOfFirstNonAsciiByte(ReadOnlySpan`1 buffer)
 /_/src/libraries/System.Runtime/tests/System/Text/ASCIIUtilityTests.cs(71,0): at System.Text.Tests.AsciiUtilityTests.GetIndexOfFirstNonAsciiByte_Vector128InnerLoop()
 System.PlatformNotSupportedException : Operation is not supported on this platform.

Builds

Build	Pull Request	Test Failure Count
#557918	Rolling	1
#558017	Rolling	1
#558485	Rolling	1
#558582	Rolling	1
#558716	Rolling	1
#559049	Rolling	1
#559300	Rolling	1
#559370	Rolling	2
#559451	Rolling	2
#559583	Rolling	2
#559682	Rolling	1
#559846	Rolling	2
#565608	Rolling	19
#565828	Rolling	18
#566046	Rolling	21
#566217	Rolling	22
#566391	Rolling	16
#566510	Rolling	19
#566655	Rolling	18
#566731	Rolling	17
#566876	Rolling	20
#567099	Rolling	19
#567233	Rolling	15
#566655	Rolling	3
#566731	Rolling	1
#566876	Rolling	3
#567099	Rolling	2
#567233	Rolling	3

Configurations

• netcoreapp5.0-Linux-Release-arm64-CoreCLR_release-(Alpine.38.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-a45aeeb-20190620184035
• netcoreapp5.0-Linux-Release-x64-CoreCLR_checked-(Alpine.311.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.11-helix-bfcd90a-20200123191053
• netcoreapp5.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Alpine.310.Amd64.Open)ubuntu.1604.amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.10-helix-bfcd90a-20200123191054
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Alpine.311.Amd64.Open)ubuntu.1604.amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.11-helix-bfcd90a-20200123191053
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Alpine.39.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.9-helix-bfcd90a-20200123191053
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Debian.10.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Fedora.29.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-29-helix-a12566d-20191210224553
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Fedora.30.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-helix-4f8cef7-20200121150022
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-(Ubuntu.1910.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-19.10-helix-amd64-cfcfd50-20191030180623
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Centos.7.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Debian.9.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1604.Amd64.Open
• netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
• netcoreapp5.0-OSX-Release-x64-CoreCLR_checked-OSX.1013.Amd64.Open
• netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open
• netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.10.Amd64.Server19H1.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.7.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x64-CoreCLR_release-Windows.81.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open

Helix Logs

Build	Pull Request	Console	Core	Test Results
#557918	Rolling	console.83a32cee.log
#558017	Rolling	console.ec8ffec8.log
#558485	Rolling	console.8f7d230b.log
#558582	Rolling	console.436c397a.log	core.1000.2960
#558716	Rolling	console.0b001431.log
#559049	Rolling	console.0c7e7d97.log
#559300	Rolling	console.5b15ebff.log
#559370	Rolling	console.dbe5b636.log
#559370	Rolling	console.6ce3108d.log
#559451	Rolling	console.8b9e16d4.log
#559451	Rolling	console.6fd13e28.log
#559583	Rolling	console.7cfbcdc3.log
#559583	Rolling	console.dcba481f.log
#559682	Rolling	console.4975c6df.log
#559846	Rolling	console.3ca57cb6.log
#559846	Rolling	console.eae472d4.log
#565608	Rolling	console.987810b3.log
#565608	Rolling	console.e3706475.log		testResults.xml
#565608	Rolling	console.1085ec09.log		testResults.xml
#565608	Rolling	console.dda7a6a0.log		testResults.xml
#565608	Rolling	console.b0e01646.log		testResults.xml
#565608	Rolling	console.68cb7b95.log		testResults.xml
#565608	Rolling	console.28e47de3.log		testResults.xml
#565608	Rolling	console.8ff82f76.log		testResults.xml
#565608	Rolling	console.3e2bdd81.log		testResults.xml
#565608	Rolling	console.a16fad20.log		testResults.xml
#565608	Rolling	console.a2c1afd9.log	core.1000.21
#565608	Rolling	console.cc17e1ec.log	core.1000.17424
#565608	Rolling	console.264fe44d.log		testResults.xml
#565608	Rolling	console.a2b931e5.log		testResults.xml
#565608	Rolling	console.76357b1b.log		testResults.xml
#565608	Rolling	console.c3ee9e88.log
#565608	Rolling	console.a48b1d88.log		testResults.xml
#565608	Rolling	console.51eed5af.log		testResults.xml
#565608	Rolling	console.4e325d13.log		testResults.xml
#565828	Rolling	console.41d34f41.log		testResults.xml
#565828	Rolling	console.f7bac871.log		testResults.xml
#565828	Rolling	console.cbd1229a.log		testResults.xml
#565828	Rolling	console.04b24086.log		testResults.xml
#565828	Rolling	console.04b052b7.log	core.1000.27551
#565828	Rolling	console.cb32b5d4.log		testResults.xml
#565828	Rolling	console.c17519d6.log		testResults.xml
#565828	Rolling	console.c9f1ca3e.log		testResults.xml
#565828	Rolling	console.ccbbba74.log		testResults.xml
#565828	Rolling	console.74c922b7.log		testResults.xml
#565828	Rolling	console.acad5777.log		testResults.xml
#565828	Rolling	console.924f9bcf.log		testResults.xml
#565828	Rolling	console.22bc780e.log	core.1000.21
#565828	Rolling	console.fbe3a5a8.log		testResults.xml
#565828	Rolling	console.93cbb0a6.log		testResults.xml
#565828	Rolling	console.b791831a.log		testResults.xml
#565828	Rolling	console.d9e4c6a9.log		testResults.xml
#565828	Rolling	console.32ddfd13.log
#566046	Rolling	console.867d9579.log
#566046	Rolling	console.1da95518.log		testResults.xml
#566046	Rolling	console.38a3c5a3.log		testResults.xml
#566046	Rolling	console.2119d075.log		testResults.xml
#566046	Rolling	console.929a12b5.log		testResults.xml
#566046	Rolling	console.86599e29.log		testResults.xml
#566046	Rolling	console.79e91e3a.log		testResults.xml
#566046	Rolling	console.b4af5e07.log		testResults.xml
#566046	Rolling	console.8646766e.log		testResults.xml
#566046	Rolling	console.a6e38e02.log		testResults.xml
#566046	Rolling	console.e095a30e.log		testResults.xml
#566046	Rolling	console.6aa1f45c.log		testResults.xml
#566046	Rolling	console.7e4e9ca5.log		testResults.xml
#566046	Rolling	console.fcce39b1.log	core.1000.21422
#566046	Rolling	console.97f1fe75.log	core.1000.21
#566046	Rolling	console.cdd19e57.log
#566046	Rolling	console.0ebe7567.log		testResults.xml
#566046	Rolling	console.739f0904.log		testResults.xml
#566046	Rolling	console.f97a8509.log		testResults.xml
#566046	Rolling	console.96508fd1.log		testResults.xml
#566046	Rolling	console.a31ab588.log		testResults.xml
#566217	Rolling	console.b1e38580.log	core.84478
#566217	Rolling	console.baabc936.log		testResults.xml
#566217	Rolling	console.9ae700dc.log		testResults.xml
#566217	Rolling	console.27687ad8.log		testResults.xml
#566217	Rolling	console.0bfa9792.log		testResults.xml
#566217	Rolling	console.8b7affd4.log		testResults.xml
#566217	Rolling	console.7cabf0be.log		testResults.xml
#566217	Rolling	console.f7d95c9b.log		testResults.xml
#566217	Rolling	console.e3b6daaf.log		testResults.xml
#566217	Rolling	console.63aab979.log		testResults.xml
#566217	Rolling	console.bdb54bbf.log		testResults.xml
#566217	Rolling	console.8a8e2507.log		testResults.xml
#566217	Rolling	console.2f54ab7f.log		testResults.xml
#566217	Rolling	console.a8550158.log		testResults.xml
#566217	Rolling	console.3f8ab106.log	core.1000.22
#566217	Rolling	console.82f1cb9e.log
#566217	Rolling	console.53f726db.log
#566217	Rolling	console.5d96ab0a.log		testResults.xml
#566217	Rolling	console.ba46036c.log		testResults.xml
#566217	Rolling	console.b6823471.log		testResults.xml
#566217	Rolling	console.d1cf54c6.log		testResults.xml
#566217	Rolling	console.5b248b0a.log		testResults.xml
#566391	Rolling	console.c89c8ae5.log	core.37343
#566391	Rolling	console.82ddec35.log		testResults.xml
#566391	Rolling	console.ada3b111.log		testResults.xml
#566391	Rolling	console.542ada7d.log		testResults.xml
#566391	Rolling	console.e6987d55.log		testResults.xml
#566391	Rolling	console.e305fb7b.log		testResults.xml
#566391	Rolling	console.8f911bef.log		testResults.xml
#566391	Rolling	console.ec32f8b2.log		testResults.xml
#566391	Rolling	console.03435f2e.log		testResults.xml
#566391	Rolling	console.ff835ee8.log		testResults.xml
#566391	Rolling	console.07979b9a.log	core.1000.21
#566391	Rolling	console.e9c2906a.log
#566391	Rolling	console.65b8a37e.log
#566391	Rolling	console.9137bac8.log		testResults.xml
#566391	Rolling	console.6d06efd3.log		testResults.xml
#566391	Rolling	console.6a6769fd.log		testResults.xml
#566510	Rolling	console.cac127ae.log
#566510	Rolling	console.b39840d8.log		testResults.xml
#566510	Rolling	console.b9d974d7.log		testResults.xml
#566510	Rolling	console.9851258f.log		testResults.xml
#566510	Rolling	console.c7db2ab0.log		testResults.xml
#566510	Rolling	console.614a7ff0.log		testResults.xml
#566510	Rolling	console.4c8fa680.log		testResults.xml
#566510	Rolling	console.164c69a4.log		testResults.xml
#566510	Rolling	console.aa212ed3.log		testResults.xml
#566510	Rolling	console.4bc7fc1a.log		testResults.xml
#566510	Rolling	console.47fb86ab.log		testResults.xml
#566510	Rolling	console.7a7d34c2.log		testResults.xml
#566510	Rolling	console.6a418d45.log	core.1000.21
#566510	Rolling	console.479ae0e9.log		testResults.xml
#566510	Rolling	console.81f9cc43.log		testResults.xml
#566510	Rolling	console.620c82b3.log
#566510	Rolling	console.40d99616.log		testResults.xml
#566510	Rolling	console.15197d18.log		testResults.xml
#566510	Rolling	console.99b6fa5d.log
#566655	Rolling	console.3c2e1132.log	core.65791
#566655	Rolling	console.664c367d.log		testResults.xml
#566655	Rolling	console.ca4fdd19.log		testResults.xml
#566655	Rolling	console.d28fcea9.log		testResults.xml
#566655	Rolling	console.ff811b19.log	core.1000.22
#566655	Rolling	console.84289a02.log
#566655	Rolling	console.2a269004.log		testResults.xml
#566655	Rolling	console.1d189ac4.log		testResults.xml
#566655	Rolling	console.d43e1bdb.log		testResults.xml
#566655	Rolling	console.a97e3d93.log		testResults.xml
#566655	Rolling	console.7765eb66.log		testResults.xml
#566655	Rolling	console.dfa4964b.log		testResults.xml
#566655	Rolling	console.1fe45d72.log		testResults.xml
#566655	Rolling	console.01e0a39e.log		testResults.xml
#566655	Rolling	console.265429d2.log		testResults.xml
#566655	Rolling	console.600958cd.log		testResults.xml
#566655	Rolling	console.a942f3c4.log		testResults.xml
#566655	Rolling	console.27e3286f.log
#566731	Rolling	console.d112584e.log
#566731	Rolling	console.b4692b58.log		testResults.xml
#566731	Rolling	console.3d724a14.log		testResults.xml
#566731	Rolling	console.a9af2c3b.log		testResults.xml
#566731	Rolling	console.01065070.log		testResults.xml
#566731	Rolling	console.925f1d43.log		testResults.xml
#566731	Rolling	console.1e9e306b.log		testResults.xml
#566731	Rolling	console.21f3d66c.log		testResults.xml
#566731	Rolling	console.7331e231.log		testResults.xml
#566731	Rolling	console.ef69abde.log		testResults.xml
#566731	Rolling	console.a2efc04a.log		testResults.xml
#566731	Rolling	console.1067e069.log	core.1000.22
#566731	Rolling	console.cc47b34d.log
#566731	Rolling	console.113fb683.log		testResults.xml
#566731	Rolling	console.ce851e7d.log		testResults.xml
#566731	Rolling	console.f3320756.log		testResults.xml
#566731	Rolling	console.d5ed4f3d.log
#566876	Rolling	console.2cce08e4.log
#566876	Rolling	console.460432d7.log		testResults.xml
#566876	Rolling	console.41aa2f84.log		testResults.xml
#566876	Rolling	console.7a8edb74.log		testResults.xml
#566876	Rolling	console.bedc7220.log		testResults.xml
#566876	Rolling	console.8f226e5c.log	core.1000.22
#566876	Rolling	console.68b93113.log		testResults.xml
#566876	Rolling	console.35ed572b.log		testResults.xml
#566876	Rolling	console.1fcc55bf.log		testResults.xml
#566876	Rolling	console.678c5b20.log		testResults.xml
#566876	Rolling	console.d93d59cd.log		testResults.xml
#566876	Rolling	console.097d3eb5.log		testResults.xml
#566876	Rolling	console.ab308057.log		testResults.xml
#566876	Rolling	console.492fb87a.log		testResults.xml
#566876	Rolling	console.0f5f3f07.log
#566876	Rolling	console.8b12edd3.log		testResults.xml
#566876	Rolling	console.23fee36c.log		testResults.xml
#566876	Rolling	console.0f0f72ec.log		testResults.xml
#566876	Rolling	console.9b8f097f.log		testResults.xml
#566876	Rolling	console.0a4a012e.log
#567099	Rolling	console.df6e1e6b.log
#567099	Rolling	console.1bb3878f.log
#567099	Rolling	console.247e575b.log		testResults.xml
#567099	Rolling	console.44090819.log		testResults.xml
#567099	Rolling	console.1156e6a4.log		testResults.xml
#567099	Rolling	console.4d274fed.log		testResults.xml
#567099	Rolling	console.73109f28.log		testResults.xml
#567099	Rolling	console.70cd87ae.log		testResults.xml
#567099	Rolling	console.d7c77e44.log		testResults.xml
#567099	Rolling	console.48892f34.log		testResults.xml
#567099	Rolling	console.d92c87ed.log		testResults.xml
#567099	Rolling	console.67eac0d2.log	core.1000.21
#567099	Rolling	console.251a7c42.log
#567099	Rolling	console.9d4a5385.log		testResults.xml
#567099	Rolling	console.82a94822.log		testResults.xml
#567099	Rolling	console.92e51d72.log		testResults.xml
#567099	Rolling	console.f15342b6.log
#567099	Rolling	console.df3c9e25.log		testResults.xml
#567099	Rolling	console.de2bd07b.log		testResults.xml
#567233	Rolling	console.e8b1aac6.log	core.93319
#567233	Rolling	console.d749a64b.log		testResults.xml
#567233	Rolling	console.0eb8836e.log		testResults.xml
#567233	Rolling	console.7f3fbf57.log		testResults.xml
#567233	Rolling	console.867514ae.log		testResults.xml
#567233	Rolling	console.e362ae39.log		testResults.xml
#567233	Rolling	console.d7af3399.log		testResults.xml
#567233	Rolling	console.f56db6ce.log		testResults.xml
#567233	Rolling	console.5782db9b.log		testResults.xml
#567233	Rolling	console.d6b333cd.log		testResults.xml
#567233	Rolling	console.972e9b80.log
#567233	Rolling	console.e9da89a1.log
#567233	Rolling	console.0a691b53.log		testResults.xml
#567233	Rolling	console.b9666e64.log		testResults.xml
#567233	Rolling	console.9e3ecabf.log		testResults.xml
#566655	Rolling	console.ca4fdd19.log		testResults.xml
#566655	Rolling	console.7765eb66.log		testResults.xml
#566655	Rolling	console.01e0a39e.log		testResults.xml
#566731	Rolling	console.01065070.log		testResults.xml
#566876	Rolling	console.ab308057.log		testResults.xml
#566876	Rolling	console.492fb87a.log		testResults.xml
#566876	Rolling	console.9b8f097f.log		testResults.xml
#567099	Rolling	console.48892f34.log		testResults.xml
#567099	Rolling	console.92e51d72.log		testResults.xml
#567233	Rolling	console.d6b333cd.log		testResults.xml
#567233	Rolling	console.0a691b53.log		testResults.xml
#567233	Rolling	console.9e3ecabf.log		testResults.xml

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[tannergooding]

CC. @GrabYourPitchforks

[tannergooding]

Maybe also @davidwrighton, I wouldn't expect this to fail unless either crossgen is doing something wrong or we have a bad copy of S.P.Corelib.

[tannergooding]

Likely related to #33897

In both cases, it is GetIndexOfFirstNonAsciiByte_Sse2 failing which is only called from one location and only if Sse2.IsSupported: https://source.dot.net/#System.Private.CoreLib/ASCIIUtility.cs,92

However, in the case of 33897, it looks like we trigger an assert that Sse2.IsSupported is false: https://source.dot.net/#System.Private.CoreLib/ASCIIUtility.cs,247 and in the case of this one, we hit the first SSE2 API and throw PNSE: https://source.dot.net/#System.Private.CoreLib/ASCIIUtility.cs,265

[jaredpar]

Updated the bug a bit. This is the root cause of pretty much all our CI failures at the moment. If more logs are needed I can help get them but feel like there are enough up there at the moment.

[tannergooding]

I've looked at the regular JIT importation logic a few times and the logic shouldn't allow a case where IsSupported returns true and an actual intrinsic causes a PNSE:

• https://github.com/dotnet/runtime/blob/master/src/coreclr/src/jit/importer.cpp#L3467-L3505
• https://github.com/dotnet/runtime/blob/master/src/coreclr/src/jit/importer.cpp#L4450-L4475
• https://github.com/dotnet/runtime/blob/master/src/coreclr/src/jit/hwintrinsic.cpp#L299-L339

We check if the ISA is supported as part of the lookup and return PNSE only if the ISA isn't supported or we are a recursive call for an unsupported platform.

I had checked out the assets from https://dev.azure.com/dnceng/public/_build/results?buildId=567233&view=logs&j=5daca27f-52cd-532c-7da3-66cd43b2fb44&t=762dfcbb-c544-5393-6139-962f790eaf9e (all four zips downloaded for OSX) and all copies of S.P.Corelib I could find were the x86/x64 copy and had recursive implementations.

[tannergooding]

Also CC. @CarolEidt, @echesakovMSFT

[davidwrighton]

This looks like my change #33730. I'm in the process of putting together a revert pr for this. I think I must have broken something in the runtime jit, as I verified before checkin that the crossgen output was binary identical for corelib.

[tannergooding]

Hmmm, I did R2RDump on all 3 instances of S.P.Corelib that I could find, and none of them have the failing asserts or even calls that would need to be jitted:

CoreCLRProduct_OSX_x64_checked.txt
libraries_bin_OSX_x64_Release.txt
dotnet-sdk-5.0.100-preview.3.20168.11-osx-x64.txt

[davidwrighton]

It turns out that due to tiered compilation, that the check around Sse2.IsSupported isn't actually handled correctly, the rules for how crossgen and tiered compilation work, make it unsafe to perform an IsSupported check and then call a helper function if the result of IsSupported may vary between crossgen time and runtime. However, that should only be a problem for the Avx variants of generated code in crossgen, not for stuff which is compiled as optimistically supported in crossgen.

My suspicion is that the problem is a runtime triggered tiered rejit not considering SSE2 to be active. I'm building the product now to verify, but I'm pretty sure its not the crossgen behavior, but the rejit behavior that is getting us in trouble here.