coreclr_initialize failure on SmartOS x64

[am11]

Without sudo

It fails earlier with coreclr_initialize failed - status: 0x8007ff02 and returns false from:

runtime/src/coreclr/src/pal/src/thread/process.cpp

Lines 3517 to 3524 in bacef40

	// Locking the page ensures that it stays in memory during the two mprotect
	// calls in the FlushProcessWriteBuffers below. If the page was unmapped between
	// those calls, they would not have the expected effect of generating IPI.
	int status = mlock(s_helperPage, GetVirtualPageSize());
	if (status != 0)
	{
	return FALSE;

(similar situation as reported in #34793)

truss -f result: http://sprunge.us/Uwi2jZ.

ulimit -a:

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 46117
virtual memory (kbytes, -v) unlimited

kstat -p -m memory_cap:

memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:anon_alloc_fail 0
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:anonpgin 0
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:class zone_memory_cap
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:crtime 64.486347335
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:execpgin 63
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:fspgin 1672
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:nover 0
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:pagedout 0
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:pgpgin 1735
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:physcap 15216934912
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:rss 66265088
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:snaptime 2458.006242087
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:swap 34607104
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:swapcap 15216934912
memory_cap:1:df333bd6-c7f8-63b6-b1c2-e947d7:zonename df333bd6-c7f8-63b6-b1c2-e947d740e084

kstat -p -m vmem:

Output collapsed

vmem:1:heap:alloc 22368
vmem:1:heap:class vmem
vmem:1:heap:contains 0
vmem:1:heap:contains_search 0
vmem:1:heap:crtime 0
vmem:1:heap:fail 0
vmem:1:heap:free 5922
vmem:1:heap:lookup 228
vmem:1:heap:mem_import 0
vmem:1:heap:mem_inuse 1637310464
vmem:1:heap:mem_total 2060927893504
vmem:1:heap:populate_fail 0
vmem:1:heap:populate_wait 0
vmem:1:heap:search 7792
vmem:1:heap:snaptime 7041.815844948
vmem:1:heap:vmem_source 0
vmem:1:heap:wait 0
vmem:2:vmem_metadata:alloc 63121
vmem:2:vmem_metadata:class vmem
vmem:2:vmem_metadata:contains 0
vmem:2:vmem_metadata:contains_search 0
vmem:2:vmem_metadata:crtime 0
vmem:2:vmem_metadata:fail 0
vmem:2:vmem_metadata:free 37
vmem:2:vmem_metadata:lookup 42
vmem:2:vmem_metadata:mem_import 278265856
vmem:2:vmem_metadata:mem_inuse 278212608
vmem:2:vmem_metadata:mem_total 278265856
vmem:2:vmem_metadata:populate_fail 0
vmem:2:vmem_metadata:populate_wait 0
vmem:2:vmem_metadata:search 0
vmem:2:vmem_metadata:snaptime 7041.815858463
vmem:2:vmem_metadata:vmem_source 1
vmem:2:vmem_metadata:wait 0
vmem:3:vmem_seg:alloc 62998
vmem:3:vmem_seg:class vmem
vmem:3:vmem_seg:contains 0
vmem:3:vmem_seg:contains_search 0
vmem:3:vmem_seg:crtime 0
vmem:3:vmem_seg:fail 0
vmem:3:vmem_seg:free 0
vmem:3:vmem_seg:lookup 0
vmem:3:vmem_seg:mem_import 258101248
vmem:3:vmem_seg:mem_inuse 258101248
vmem:3:vmem_seg:mem_total 258101248
vmem:3:vmem_seg:populate_fail 0
vmem:3:vmem_seg:populate_wait 0
vmem:3:vmem_seg:search 0
vmem:3:vmem_seg:snaptime 7041.815869908
vmem:3:vmem_seg:vmem_source 2
vmem:3:vmem_seg:wait 0
vmem:4:vmem_hash:alloc 66
vmem:4:vmem_hash:class vmem
vmem:4:vmem_hash:contains 0
vmem:4:vmem_hash:contains_search 0
vmem:4:vmem_hash:crtime 0
vmem:4:vmem_hash:fail 0
vmem:4:vmem_hash:free 35
vmem:4:vmem_hash:lookup 37
vmem:4:vmem_hash:mem_import 19832832
vmem:4:vmem_hash:mem_inuse 19830784
vmem:4:vmem_hash:mem_total 19832832
vmem:4:vmem_hash:populate_fail 0
vmem:4:vmem_hash:populate_wait 0
vmem:4:vmem_hash:search 0
vmem:4:vmem_hash:snaptime 7041.815881051
vmem:4:vmem_hash:vmem_source 2
vmem:4:vmem_hash:wait 0
vmem:5:vmem_vmem:alloc 91
vmem:5:vmem_vmem:class vmem
vmem:5:vmem_vmem:contains 0
vmem:5:vmem_vmem:contains_search 0
vmem:5:vmem_vmem:crtime 0
vmem:5:vmem_vmem:fail 0
vmem:5:vmem_vmem:free 13
vmem:5:vmem_vmem:lookup 5
vmem:5:vmem_vmem:mem_import 278528
vmem:5:vmem_vmem:mem_inuse 283920
vmem:5:vmem_vmem:mem_total 314928
vmem:5:vmem_vmem:populate_fail 0
vmem:5:vmem_vmem:populate_wait 0
vmem:5:vmem_vmem:search 11
vmem:5:vmem_vmem:snaptime 7041.815892208
vmem:5:vmem_vmem:vmem_source 2
vmem:5:vmem_vmem:wait 0
vmem:6:heap_core:alloc 52
vmem:6:heap_core:class vmem
vmem:6:heap_core:contains 0
vmem:6:heap_core:contains_search 0
vmem:6:heap_core:crtime 0
vmem:6:heap_core:fail 0
vmem:6:heap_core:free 7
vmem:6:heap_core:lookup 4
vmem:6:heap_core:mem_import 0
vmem:6:heap_core:mem_inuse 2265088
vmem:6:heap_core:mem_total 931049472
vmem:6:heap_core:populate_fail 0
vmem:6:heap_core:populate_wait 0
vmem:6:heap_core:search 0
vmem:6:heap_core:snaptime 7041.815903496
vmem:6:heap_core:vmem_source 0
vmem:6:heap_core:wait 0
vmem:7:heaptext:alloc 155
vmem:7:heaptext:class vmem
vmem:7:heaptext:contains 263
vmem:7:heaptext:contains_search 263
vmem:7:heaptext:crtime 0
vmem:7:heaptext:fail 0
vmem:7:heaptext:free 62
vmem:7:heaptext:lookup 21
vmem:7:heaptext:mem_import 0
vmem:7:heaptext:mem_inuse 7954432
vmem:7:heaptext:mem_total 67108864
vmem:7:heaptext:populate_fail 0
vmem:7:heaptext:populate_wait 0
vmem:7:heaptext:search 0
vmem:7:heaptext:snaptime 7041.815914727
vmem:7:heaptext:vmem_source 6
vmem:7:heaptext:wait 0
vmem:8:static:alloc 0
vmem:8:static:class vmem
vmem:8:static:contains 0
vmem:8:static:contains_search 0
vmem:8:static:crtime 0
vmem:8:static:fail 0
vmem:8:static:free 0
vmem:8:static:lookup 0
vmem:8:static:mem_import 0
vmem:8:static:mem_inuse 0
vmem:8:static:mem_total 0
vmem:8:static:populate_fail 0
vmem:8:static:populate_wait 0
vmem:8:static:search 0
vmem:8:static:snaptime 7041.815925952
vmem:8:static:vmem_source 1
vmem:8:static:wait 0
vmem:9:static_alloc:alloc 0
vmem:9:static_alloc:class vmem
vmem:9:static_alloc:contains 0
vmem:9:static_alloc:contains_search 0
vmem:9:static_alloc:crtime 0
vmem:9:static_alloc:fail 0
vmem:9:static_alloc:free 0
vmem:9:static_alloc:lookup 0
vmem:9:static_alloc:mem_import 0
vmem:9:static_alloc:mem_inuse 0
vmem:9:static_alloc:mem_total 0
vmem:9:static_alloc:populate_fail 0
vmem:9:static_alloc:populate_wait 0
vmem:9:static_alloc:search 0
vmem:9:static_alloc:snaptime 7041.815937300
vmem:9:static_alloc:vmem_source 8
vmem:9:static_alloc:wait 0
vmem:10:hat_memload:alloc 1348
vmem:10:hat_memload:class vmem
vmem:10:hat_memload:contains 0
vmem:10:hat_memload:contains_search 0
vmem:10:hat_memload:crtime 0
vmem:10:hat_memload:fail 0
vmem:10:hat_memload:free 76
vmem:10:hat_memload:lookup 2
vmem:10:hat_memload:mem_import 5210112
vmem:10:hat_memload:mem_inuse 5210112
vmem:10:hat_memload:mem_total 5210112
vmem:10:hat_memload:populate_fail 0
vmem:10:hat_memload:populate_wait 0
vmem:10:hat_memload:search 0
vmem:10:hat_memload:snaptime 7041.815948590
vmem:10:hat_memload:vmem_source 1
vmem:10:hat_memload:wait 0
vmem:11:kstat:alloc 1476
vmem:11:kstat:class vmem
vmem:11:kstat:contains 0
vmem:11:kstat:contains_search 0
vmem:11:kstat:crtime 0
vmem:11:kstat:fail 0
vmem:11:kstat:free 403
vmem:11:kstat:lookup 32
vmem:11:kstat:mem_import 364544
vmem:11:kstat:mem_inuse 404192
vmem:11:kstat:mem_total 430080
vmem:11:kstat:populate_fail 0
vmem:11:kstat:populate_wait 0
vmem:11:kstat:search 240
vmem:11:kstat:snaptime 7041.815960001
vmem:11:kstat:vmem_source 1
vmem:11:kstat:wait 0
vmem:12:kmem_metadata:alloc 65871
vmem:12:kmem_metadata:class vmem
vmem:12:kmem_metadata:contains 0
vmem:12:kmem_metadata:contains_search 0
vmem:12:kmem_metadata:crtime 0
vmem:12:kmem_metadata:fail 0
vmem:12:kmem_metadata:free 36
vmem:12:kmem_metadata:lookup 40
vmem:12:kmem_metadata:mem_import 288489472
vmem:12:kmem_metadata:mem_inuse 288411648
vmem:12:kmem_metadata:mem_total 288489472
vmem:12:kmem_metadata:populate_fail 0
vmem:12:kmem_metadata:populate_wait 0
vmem:12:kmem_metadata:search 17
vmem:12:kmem_metadata:snaptime 7041.815980704
vmem:12:kmem_metadata:vmem_source 1
vmem:12:kmem_metadata:wait 0
vmem:13:kmem_msb:alloc 65663
vmem:13:kmem_msb:class vmem
vmem:13:kmem_msb:contains 0
vmem:13:kmem_msb:contains_search 0
vmem:13:kmem_msb:crtime 0
vmem:13:kmem_msb:fail 0
vmem:13:kmem_msb:free 10
vmem:13:kmem_msb:lookup 0
vmem:13:kmem_msb:mem_import 268914688
vmem:13:kmem_msb:mem_inuse 268914688
vmem:13:kmem_msb:mem_total 268914688
vmem:13:kmem_msb:populate_fail 0
vmem:13:kmem_msb:populate_wait 0
vmem:13:kmem_msb:search 0
vmem:13:kmem_msb:snaptime 7041.815992342
vmem:13:kmem_msb:vmem_source 12
vmem:13:kmem_msb:wait 0
vmem:14:kmem_cache:alloc 559
vmem:14:kmem_cache:class vmem
vmem:14:kmem_cache:contains 0
vmem:14:kmem_cache:contains_search 0
vmem:14:kmem_cache:crtime 0
vmem:14:kmem_cache:fail 0
vmem:14:kmem_cache:free 105
vmem:14:kmem_cache:lookup 0
vmem:14:kmem_cache:mem_import 471040
vmem:14:kmem_cache:mem_inuse 461264
vmem:14:kmem_cache:mem_total 471040
vmem:14:kmem_cache:populate_fail 0
vmem:14:kmem_cache:populate_wait 0
vmem:14:kmem_cache:search 428
vmem:14:kmem_cache:snaptime 7041.816052250
vmem:14:kmem_cache:vmem_source 12
vmem:14:kmem_cache:wait 0
vmem:15:kmem_hash:alloc 374
vmem:15:kmem_hash:class vmem
vmem:15:kmem_hash:contains 0
vmem:15:kmem_hash:contains_search 0
vmem:15:kmem_hash:crtime 0
vmem:15:kmem_hash:fail 0
vmem:15:kmem_hash:free 117
vmem:15:kmem_hash:lookup 62
vmem:15:kmem_hash:mem_import 19025920
vmem:15:kmem_hash:mem_inuse 19006464
vmem:15:kmem_hash:mem_total 19025920
vmem:15:kmem_hash:populate_fail 0
vmem:15:kmem_hash:populate_wait 0
vmem:15:kmem_hash:search 0
vmem:15:kmem_hash:snaptime 7041.816066098
vmem:15:kmem_hash:vmem_source 12
vmem:15:kmem_hash:wait 0
vmem:16:kmem_log:alloc 9
vmem:16:kmem_log:class vmem
vmem:16:kmem_log:contains 0
vmem:16:kmem_log:contains_search 0
vmem:16:kmem_log:crtime 0
vmem:16:kmem_log:fail 0
vmem:16:kmem_log:free 0
vmem:16:kmem_log:lookup 0
vmem:16:kmem_log:mem_import 348160
vmem:16:kmem_log:mem_inuse 345888
vmem:16:kmem_log:mem_total 348160
vmem:16:kmem_log:populate_fail 0
vmem:16:kmem_log:populate_wait 0
vmem:16:kmem_log:search 2
vmem:16:kmem_log:snaptime 7041.816077952
vmem:16:kmem_log:vmem_source 1
vmem:16:kmem_log:wait 0
vmem:17:kmem_firewall_va:alloc 297
vmem:17:kmem_firewall_va:class vmem
vmem:17:kmem_firewall_va:contains 0
vmem:17:kmem_firewall_va:contains_search 0
vmem:17:kmem_firewall_va:crtime 0
vmem:17:kmem_firewall_va:fail 0
vmem:17:kmem_firewall_va:free 244
vmem:17:kmem_firewall_va:lookup 3
vmem:17:kmem_firewall_va:mem_import 230109184
vmem:17:kmem_firewall_va:mem_inuse 230109184
vmem:17:kmem_firewall_va:mem_total 230109184
vmem:17:kmem_firewall_va:populate_fail 0
vmem:17:kmem_firewall_va:populate_wait 0
vmem:17:kmem_firewall_va:search 0
vmem:17:kmem_firewall_va:snaptime 7041.816089715
vmem:17:kmem_firewall_va:vmem_source 1
vmem:17:kmem_firewall_va:wait 0
vmem:18:kmem_firewall:alloc 0
vmem:18:kmem_firewall:class vmem
vmem:18:kmem_firewall:contains 0
vmem:18:kmem_firewall:contains_search 0
vmem:18:kmem_firewall:crtime 0
vmem:18:kmem_firewall:fail 0
vmem:18:kmem_firewall:free 0
vmem:18:kmem_firewall:lookup 0
vmem:18:kmem_firewall:mem_import 0
vmem:18:kmem_firewall:mem_inuse 0
vmem:18:kmem_firewall:mem_total 0
vmem:18:kmem_firewall:populate_fail 0
vmem:18:kmem_firewall:populate_wait 0
vmem:18:kmem_firewall:search 0
vmem:18:kmem_firewall:snaptime 7041.816101378
vmem:18:kmem_firewall:vmem_source 17
vmem:18:kmem_firewall:wait 0
vmem:20:mod_sysfile:alloc 39
vmem:20:mod_sysfile:class vmem
vmem:20:mod_sysfile:contains 0
vmem:20:mod_sysfile:contains_search 0
vmem:20:mod_sysfile:crtime 0
vmem:20:mod_sysfile:fail 0
vmem:20:mod_sysfile:free 0
vmem:20:mod_sysfile:lookup 0
vmem:20:mod_sysfile:mem_import 4096
vmem:20:mod_sysfile:mem_inuse 1469
vmem:20:mod_sysfile:mem_total 4096
vmem:20:mod_sysfile:populate_fail 0
vmem:20:mod_sysfile:populate_wait 0
vmem:20:mod_sysfile:search 0
vmem:20:mod_sysfile:snaptime 7041.816113101
vmem:20:mod_sysfile:vmem_source 1
vmem:20:mod_sysfile:wait 0
vmem:21:kmem_oversize:alloc 297
vmem:21:kmem_oversize:class vmem
vmem:21:kmem_oversize:contains 0
vmem:21:kmem_oversize:contains_search 0
vmem:21:kmem_oversize:crtime 0
vmem:21:kmem_oversize:fail 0
vmem:21:kmem_oversize:free 244
vmem:21:kmem_oversize:lookup 3
vmem:21:kmem_oversize:mem_import 230109184
vmem:21:kmem_oversize:mem_inuse 230077567
vmem:21:kmem_oversize:mem_total 230109184
vmem:21:kmem_oversize:populate_fail 0
vmem:21:kmem_oversize:populate_wait 0
vmem:21:kmem_oversize:search 0
vmem:21:kmem_oversize:snaptime 7041.816124911
vmem:21:kmem_oversize:vmem_source 17
vmem:21:kmem_oversize:wait 0
vmem:22:kmem_va:alloc 5813
vmem:22:kmem_va:class vmem
vmem:22:kmem_va:contains 0
vmem:22:kmem_va:contains_search 0
vmem:22:kmem_va:crtime 0
vmem:22:kmem_va:fail 0
vmem:22:kmem_va:free 0
vmem:22:kmem_va:lookup 0
vmem:22:kmem_va:mem_import 799567872
vmem:22:kmem_va:mem_inuse 799567872
vmem:22:kmem_va:mem_total 799567872
vmem:22:kmem_va:populate_fail 0
vmem:22:kmem_va:populate_wait 0
vmem:22:kmem_va:search 0
vmem:22:kmem_va:snaptime 7041.816140750
vmem:22:kmem_va:vmem_source 1
vmem:22:kmem_va:wait 0
vmem:23:kmem_default:alloc 121408
vmem:23:kmem_default:class vmem
vmem:23:kmem_default:contains 0
vmem:23:kmem_default:contains_search 0
vmem:23:kmem_default:crtime 0
vmem:23:kmem_default:fail 0
vmem:23:kmem_default:free 7219
vmem:23:kmem_default:lookup 248
vmem:23:kmem_default:mem_import 797302784
vmem:23:kmem_default:mem_inuse 797302784
vmem:23:kmem_default:mem_total 797302784
vmem:23:kmem_default:populate_fail 0
vmem:23:kmem_default:populate_wait 0
vmem:23:kmem_default:search 0
vmem:23:kmem_default:snaptime 7041.816152623
vmem:23:kmem_default:vmem_source 22
vmem:23:kmem_default:wait 0
vmem:24:logminor_space:alloc 86
vmem:24:logminor_space:class vmem
vmem:24:logminor_space:contains 0
vmem:24:logminor_space:contains_search 0
vmem:24:logminor_space:crtime 0
vmem:24:logminor_space:fail 0
vmem:24:logminor_space:free 44
vmem:24:logminor_space:lookup 0
vmem:24:logminor_space:mem_import 0
vmem:24:logminor_space:mem_inuse 42
vmem:24:logminor_space:mem_total 262137
vmem:24:logminor_space:populate_fail 0
vmem:24:logminor_space:populate_wait 0
vmem:24:logminor_space:search 2
vmem:24:logminor_space:snaptime 7041.816179867
vmem:24:logminor_space:vmem_source 0
vmem:24:logminor_space:wait 0
vmem:25:taskq_id_arena:alloc 147
vmem:25:taskq_id_arena:class vmem
vmem:25:taskq_id_arena:contains 0
vmem:25:taskq_id_arena:contains_search 0
vmem:25:taskq_id_arena:crtime 0
vmem:25:taskq_id_arena:fail 0
vmem:25:taskq_id_arena:free 86
vmem:25:taskq_id_arena:lookup 28
vmem:25:taskq_id_arena:mem_import 0
vmem:25:taskq_id_arena:mem_inuse 61
vmem:25:taskq_id_arena:mem_total 2147483647
vmem:25:taskq_id_arena:populate_fail 0
vmem:25:taskq_id_arena:populate_wait 0
vmem:25:taskq_id_arena:search 0
vmem:25:taskq_id_arena:snaptime 7041.816192472
vmem:25:taskq_id_arena:vmem_source 0
vmem:25:taskq_id_arena:wait 0
vmem:26:kmem_io_64G:alloc 4867
vmem:26:kmem_io_64G:class vmem
vmem:26:kmem_io_64G:contains 0
vmem:26:kmem_io_64G:contains_search 0
vmem:26:kmem_io_64G:crtime 0
vmem:26:kmem_io_64G:fail 0
vmem:26:kmem_io_64G:free 0
vmem:26:kmem_io_64G:lookup 0
vmem:26:kmem_io_64G:mem_import 19935232
vmem:26:kmem_io_64G:mem_inuse 19935232
vmem:26:kmem_io_64G:mem_total 19935232
vmem:26:kmem_io_64G:populate_fail 0
vmem:26:kmem_io_64G:populate_wait 0
vmem:26:kmem_io_64G:search 0
vmem:26:kmem_io_64G:snaptime 7041.816204460
vmem:26:kmem_io_64G:vmem_source 1
vmem:26:kmem_io_64G:wait 0
vmem:27:kmem_io_4G:alloc 4
vmem:27:kmem_io_4G:class vmem
vmem:27:kmem_io_4G:contains 0
vmem:27:kmem_io_4G:contains_search 0
vmem:27:kmem_io_4G:crtime 0
vmem:27:kmem_io_4G:fail 0
vmem:27:kmem_io_4G:free 0
vmem:27:kmem_io_4G:lookup 0
vmem:27:kmem_io_4G:mem_import 16384
vmem:27:kmem_io_4G:mem_inuse 16384
vmem:27:kmem_io_4G:mem_total 16384
vmem:27:kmem_io_4G:populate_fail 0
vmem:27:kmem_io_4G:populate_wait 0
vmem:27:kmem_io_4G:search 0
vmem:27:kmem_io_4G:snaptime 7041.816217703
vmem:27:kmem_io_4G:vmem_source 1
vmem:27:kmem_io_4G:wait 0
vmem:28:kmem_io_2G:alloc 14
vmem:28:kmem_io_2G:class vmem
vmem:28:kmem_io_2G:contains 0
vmem:28:kmem_io_2G:contains_search 0
vmem:28:kmem_io_2G:crtime 0
vmem:28:kmem_io_2G:fail 0
vmem:28:kmem_io_2G:free 12
vmem:28:kmem_io_2G:lookup 0
vmem:28:kmem_io_2G:mem_import 12288
vmem:28:kmem_io_2G:mem_inuse 12288
vmem:28:kmem_io_2G:mem_total 12288
vmem:28:kmem_io_2G:populate_fail 0
vmem:28:kmem_io_2G:populate_wait 0
vmem:28:kmem_io_2G:search 0
vmem:28:kmem_io_2G:snaptime 7041.816230911
vmem:28:kmem_io_2G:vmem_source 1
vmem:28:kmem_io_2G:wait 0
vmem:29:kmem_io_16M:alloc 0
vmem:29:kmem_io_16M:class vmem
vmem:29:kmem_io_16M:contains 0
vmem:29:kmem_io_16M:contains_search 0
vmem:29:kmem_io_16M:crtime 0
vmem:29:kmem_io_16M:fail 0
vmem:29:kmem_io_16M:free 0
vmem:29:kmem_io_16M:lookup 0
vmem:29:kmem_io_16M:mem_import 0
vmem:29:kmem_io_16M:mem_inuse 0
vmem:29:kmem_io_16M:mem_total 0
vmem:29:kmem_io_16M:populate_fail 0
vmem:29:kmem_io_16M:populate_wait 0
vmem:29:kmem_io_16M:search 0
vmem:29:kmem_io_16M:snaptime 7041.816244297
vmem:29:kmem_io_16M:vmem_source 1
vmem:29:kmem_io_16M:wait 0
vmem:30:id32:alloc 1
vmem:30:id32:class vmem
vmem:30:id32:contains 0
vmem:30:id32:contains_search 0
vmem:30:id32:crtime 0
vmem:30:id32:fail 0
vmem:30:id32:free 0
vmem:30:id32:lookup 0
vmem:30:id32:mem_import 4096
vmem:30:id32:mem_inuse 4096
vmem:30:id32:mem_total 4096
vmem:30:id32:populate_fail 0
vmem:30:id32:populate_wait 0
vmem:30:id32:search 0
vmem:30:id32:snaptime 7041.816257721
vmem:30:id32:vmem_source 6
vmem:30:id32:wait 0
vmem:31:bp_map:alloc 172
vmem:31:bp_map:class vmem
vmem:31:bp_map:contains 0
vmem:31:bp_map:contains_search 0
vmem:31:bp_map:crtime 0
vmem:31:bp_map:fail 0
vmem:31:bp_map:free 168
vmem:31:bp_map:lookup 9
vmem:31:bp_map:mem_import 524288
vmem:31:bp_map:mem_inuse 524288
vmem:31:bp_map:mem_total 524288
vmem:31:bp_map:populate_fail 0
vmem:31:bp_map:populate_wait 0
vmem:31:bp_map:search 0
vmem:31:bp_map:snaptime 7041.816271868
vmem:31:bp_map:vmem_source 1
vmem:31:bp_map:wait 0
vmem:32:kvmm_arena:alloc 0
vmem:32:kvmm_arena:class vmem
vmem:32:kvmm_arena:contains 0
vmem:32:kvmm_arena:contains_search 0
vmem:32:kvmm_arena:crtime 0
vmem:32:kvmm_arena:fail 0
vmem:32:kvmm_arena:free 0
vmem:32:kvmm_arena:lookup 0
vmem:32:kvmm_arena:mem_import 0
vmem:32:kvmm_arena:mem_inuse 0
vmem:32:kvmm_arena:mem_total 96733233152
vmem:32:kvmm_arena:populate_fail 0
vmem:32:kvmm_arena:populate_wait 0
vmem:32:kvmm_arena:search 0
vmem:32:kvmm_arena:snaptime 7041.816285066
vmem:32:kvmm_arena:vmem_source 0
vmem:32:kvmm_arena:wait 0
vmem:33:zfs_file_data:alloc 69527
vmem:33:zfs_file_data:class vmem
vmem:33:zfs_file_data:contains 0
vmem:33:zfs_file_data:contains_search 0
vmem:33:zfs_file_data:crtime 0
vmem:33:zfs_file_data:fail 0
vmem:33:zfs_file_data:free 0
vmem:33:zfs_file_data:lookup 0
vmem:33:zfs_file_data:mem_import 0
vmem:33:zfs_file_data:mem_inuse 9084755968
vmem:33:zfs_file_data:mem_total 36274438144
vmem:33:zfs_file_data:populate_fail 0
vmem:33:zfs_file_data:populate_wait 0
vmem:33:zfs_file_data:search 0
vmem:33:zfs_file_data:snaptime 7041.816299059
vmem:33:zfs_file_data:vmem_source 0
vmem:33:zfs_file_data:wait 0
vmem:34:zfs_file_data_buf:alloc 1933785
vmem:34:zfs_file_data_buf:class vmem
vmem:34:zfs_file_data_buf:contains 0
vmem:34:zfs_file_data_buf:contains_search 0
vmem:34:zfs_file_data_buf:crtime 0
vmem:34:zfs_file_data_buf:fail 0
vmem:34:zfs_file_data_buf:free 11015
vmem:34:zfs_file_data_buf:lookup 791
vmem:34:zfs_file_data_buf:mem_import 9081225216
vmem:34:zfs_file_data_buf:mem_inuse 9081225216
vmem:34:zfs_file_data_buf:mem_total 9081225216
vmem:34:zfs_file_data_buf:populate_fail 0
vmem:34:zfs_file_data_buf:populate_wait 0
vmem:34:zfs_file_data_buf:search 0
vmem:34:zfs_file_data_buf:snaptime 7041.816311451
vmem:34:zfs_file_data_buf:vmem_source 33
vmem:34:zfs_file_data_buf:wait 0
vmem:35:device:alloc 50
vmem:35:device:class vmem
vmem:35:device:contains 0
vmem:35:device:contains_search 0
vmem:35:device:crtime 0
vmem:35:device:fail 0
vmem:35:device:free 39
vmem:35:device:lookup 0
vmem:35:device:mem_import 0
vmem:35:device:mem_inuse 1228800
vmem:35:device:mem_total 1073741824
vmem:35:device:populate_fail 0
vmem:35:device:populate_wait 0
vmem:35:device:search 0
vmem:35:device:snaptime 7041.816323822
vmem:35:device:vmem_source 0
vmem:35:device:wait 0
vmem:36:segkp:alloc 4878
vmem:36:segkp:class vmem
vmem:36:segkp:contains 0
vmem:36:segkp:contains_search 0
vmem:36:segkp:crtime 0
vmem:36:segkp:fail 0
vmem:36:segkp:free 3912
vmem:36:segkp:lookup 2658
vmem:36:segkp:mem_import 0
vmem:36:segkp:mem_inuse 24936448
vmem:36:segkp:mem_total 2147483648
vmem:36:segkp:populate_fail 0
vmem:36:segkp:populate_wait 0
vmem:36:segkp:search 104925
vmem:36:segkp:snaptime 7041.816337311
vmem:36:segkp:vmem_source 0
vmem:36:segkp:wait 0
vmem:37:umem_np:alloc 94
vmem:37:umem_np:class vmem
vmem:37:umem_np:contains 0
vmem:37:umem_np:contains_search 0
vmem:37:umem_np:crtime 0
vmem:37:umem_np:fail 0
vmem:37:umem_np:free 87
vmem:37:umem_np:lookup 0
vmem:37:umem_np:mem_import 917504
vmem:37:umem_np:mem_inuse 917504
vmem:37:umem_np:mem_total 917504
vmem:37:umem_np:populate_fail 0
vmem:37:umem_np:populate_wait 0
vmem:37:umem_np:search 0
vmem:37:umem_np:snaptime 7041.816351432
vmem:37:umem_np:vmem_source 1
vmem:37:umem_np:wait 0
vmem:38:ksyms:alloc 264
vmem:38:ksyms:class vmem
vmem:38:ksyms:contains 93
vmem:38:ksyms:contains_search 5854
vmem:38:ksyms:crtime 0
vmem:38:ksyms:fail 0
vmem:38:ksyms:free 92
vmem:38:ksyms:lookup 39
vmem:38:ksyms:mem_import 2420736
vmem:38:ksyms:mem_inuse 2145612
vmem:38:ksyms:mem_total 2420736
vmem:38:ksyms:populate_fail 0
vmem:38:ksyms:populate_wait 0
vmem:38:ksyms:search 1060
vmem:38:ksyms:snaptime 7041.816364030
vmem:38:ksyms:vmem_source 1
vmem:38:ksyms:wait 0
vmem:39:ctf:alloc 256
vmem:39:ctf:class vmem
vmem:39:ctf:contains 89
vmem:39:ctf:contains_search 4588
vmem:39:ctf:crtime 0
vmem:39:ctf:fail 0
vmem:39:ctf:free 89
vmem:39:ctf:lookup 28
vmem:39:ctf:mem_import 1662976
vmem:39:ctf:mem_inuse 1404781
vmem:39:ctf:mem_total 1662976
vmem:39:ctf:populate_fail 0
vmem:39:ctf:populate_wait 0
vmem:39:ctf:search 1009
vmem:39:ctf:snaptime 7041.816376785
vmem:39:ctf:vmem_source 1
vmem:39:ctf:wait 0
vmem:40:module_text:alloc 263
vmem:40:module_text:class vmem
vmem:40:module_text:contains 0
vmem:40:module_text:contains_search 0
vmem:40:module_text:crtime 0
vmem:40:module_text:fail 0
vmem:40:module_text:free 93
vmem:40:module_text:lookup 29
vmem:40:module_text:mem_import 7954432
vmem:40:module_text:mem_inuse 9799736
vmem:40:module_text:mem_total 10051584
vmem:40:module_text:populate_fail 0
vmem:40:module_text:populate_wait 0
vmem:40:module_text:search 1271
vmem:40:module_text:snaptime 7041.816390787
vmem:40:module_text:vmem_source 7
vmem:40:module_text:wait 0
vmem:41:module_data:alloc 365
vmem:41:module_data:class vmem
vmem:41:module_data:contains 0
vmem:41:module_data:contains_search 0
vmem:41:module_data:crtime 0
vmem:41:module_data:fail 0
vmem:41:module_data:free 125
vmem:41:module_data:lookup 64
vmem:41:module_data:mem_import 2260992
vmem:41:module_data:mem_inuse 1176783
vmem:41:module_data:mem_total 2568192
vmem:41:module_data:populate_fail 0
vmem:41:module_data:populate_wait 0
vmem:41:module_data:search 940
vmem:41:module_data:snaptime 7041.816403702
vmem:41:module_data:vmem_source 6
vmem:41:module_data:wait 0
vmem:42:mac_minor_ids:alloc 13
vmem:42:mac_minor_ids:class vmem
vmem:42:mac_minor_ids:contains 0
vmem:42:mac_minor_ids:contains_search 0
vmem:42:mac_minor_ids:crtime 0
vmem:42:mac_minor_ids:fail 0
vmem:42:mac_minor_ids:free 1
vmem:42:mac_minor_ids:lookup 0
vmem:42:mac_minor_ids:mem_import 0
vmem:42:mac_minor_ids:mem_inuse 12
vmem:42:mac_minor_ids:mem_total 130070
vmem:42:mac_minor_ids:populate_fail 0
vmem:42:mac_minor_ids:populate_wait 0
vmem:42:mac_minor_ids:search 2
vmem:42:mac_minor_ids:snaptime 7041.816419137
vmem:42:mac_minor_ids:vmem_source 0
vmem:42:mac_minor_ids:wait 0
vmem:43:namefs_inodes:alloc 1
vmem:43:namefs_inodes:class vmem
vmem:43:namefs_inodes:contains 0
vmem:43:namefs_inodes:contains_search 0
vmem:43:namefs_inodes:crtime 0
vmem:43:namefs_inodes:fail 0
vmem:43:namefs_inodes:free 0
vmem:43:namefs_inodes:lookup 0
vmem:43:namefs_inodes:mem_import 0
vmem:43:namefs_inodes:mem_inuse 64
vmem:43:namefs_inodes:mem_total 65536
vmem:43:namefs_inodes:populate_fail 0
vmem:43:namefs_inodes:populate_wait 0
vmem:43:namefs_inodes:search 0
vmem:43:namefs_inodes:snaptime 7041.816433042
vmem:43:namefs_inodes:vmem_source 0
vmem:43:namefs_inodes:wait 0
vmem:44:rctl_ids:alloc 45
vmem:44:rctl_ids:class vmem
vmem:44:rctl_ids:contains 0
vmem:44:rctl_ids:contains_search 0
vmem:44:rctl_ids:crtime 0
vmem:44:rctl_ids:fail 0
vmem:44:rctl_ids:free 0
vmem:44:rctl_ids:lookup 0
vmem:44:rctl_ids:mem_import 0
vmem:44:rctl_ids:mem_inuse 45
vmem:44:rctl_ids:mem_total 32767
vmem:44:rctl_ids:populate_fail 0
vmem:44:rctl_ids:populate_wait 0
vmem:44:rctl_ids:search 2
vmem:44:rctl_ids:snaptime 7041.816448729
vmem:44:rctl_ids:vmem_source 0
vmem:44:rctl_ids:wait 0
vmem:45:zoneid_space:alloc 1
vmem:45:zoneid_space:class vmem
vmem:45:zoneid_space:contains 0
vmem:45:zoneid_space:contains_search 0
vmem:45:zoneid_space:crtime 0
vmem:45:zoneid_space:fail 0
vmem:45:zoneid_space:free 0
vmem:45:zoneid_space:lookup 0
vmem:45:zoneid_space:mem_import 0
vmem:45:zoneid_space:mem_inuse 1
vmem:45:zoneid_space:mem_total 8190
vmem:45:zoneid_space:populate_fail 0
vmem:45:zoneid_space:populate_wait 0
vmem:45:zoneid_space:search 2
vmem:45:zoneid_space:snaptime 7041.816461754
vmem:45:zoneid_space:vmem_source 0
vmem:45:zoneid_space:wait 0
vmem:46:taskid_space:alloc 169
vmem:46:taskid_space:class vmem
vmem:46:taskid_space:contains 0
vmem:46:taskid_space:contains_search 0
vmem:46:taskid_space:crtime 0
vmem:46:taskid_space:fail 0
vmem:46:taskid_space:free 112
vmem:46:taskid_space:lookup 1
vmem:46:taskid_space:mem_import 0
vmem:46:taskid_space:mem_inuse 57
vmem:46:taskid_space:mem_total 999999
vmem:46:taskid_space:populate_fail 0
vmem:46:taskid_space:populate_wait 0
vmem:46:taskid_space:search 2
vmem:46:taskid_space:snaptime 7041.816474890
vmem:46:taskid_space:vmem_source 0
vmem:46:taskid_space:wait 0
vmem:47:pool_ids:alloc 0
vmem:47:pool_ids:class vmem
vmem:47:pool_ids:contains 0
vmem:47:pool_ids:contains_search 0
vmem:47:pool_ids:crtime 0
vmem:47:pool_ids:fail 0
vmem:47:pool_ids:free 0
vmem:47:pool_ids:lookup 0
vmem:47:pool_ids:mem_import 0
vmem:47:pool_ids:mem_inuse 0
vmem:47:pool_ids:mem_total 999998
vmem:47:pool_ids:populate_fail 0
vmem:47:pool_ids:populate_wait 0
vmem:47:pool_ids:search 0
vmem:47:pool_ids:snaptime 7041.816488031
vmem:47:pool_ids:vmem_source 0
vmem:47:pool_ids:wait 0
vmem:48:contracts:alloc 181
vmem:48:contracts:class vmem
vmem:48:contracts:contains 0
vmem:48:contracts:contains_search 0
vmem:48:contracts:crtime 0
vmem:48:contracts:fail 0
vmem:48:contracts:free 113
vmem:48:contracts:lookup 2
vmem:48:contracts:mem_import 0
vmem:48:contracts:mem_inuse 68
vmem:48:contracts:mem_total 2147483646
vmem:48:contracts:populate_fail 0
vmem:48:contracts:populate_wait 0
vmem:48:contracts:search 2
vmem:48:contracts:snaptime 7041.816501776
vmem:48:contracts:vmem_source 0
vmem:48:contracts:wait 0
vmem:49:ddi_periodic:alloc 0
vmem:49:ddi_periodic:class vmem
vmem:49:ddi_periodic:contains 0
vmem:49:ddi_periodic:contains_search 0
vmem:49:ddi_periodic:crtime 19.026255796
vmem:49:ddi_periodic:fail 0
vmem:49:ddi_periodic:free 0
vmem:49:ddi_periodic:lookup 0
vmem:49:ddi_periodic:mem_import 0
vmem:49:ddi_periodic:mem_inuse 0
vmem:49:ddi_periodic:mem_total 1023
vmem:49:ddi_periodic:populate_fail 0
vmem:49:ddi_periodic:populate_wait 0
vmem:49:ddi_periodic:search 0
vmem:49:ddi_periodic:snaptime 7041.816517611
vmem:49:ddi_periodic:vmem_source 0
vmem:49:ddi_periodic:wait 0
vmem:50:bootfs_minors:alloc 1
vmem:50:bootfs_minors:class vmem
vmem:50:bootfs_minors:contains 0
vmem:50:bootfs_minors:contains_search 0
vmem:50:bootfs_minors:crtime 19.057419741
vmem:50:bootfs_minors:fail 0
vmem:50:bootfs_minors:free 0
vmem:50:bootfs_minors:lookup 0
vmem:50:bootfs_minors:mem_import 0
vmem:50:bootfs_minors:mem_inuse 1
vmem:50:bootfs_minors:mem_total 2147483646
vmem:50:bootfs_minors:populate_fail 0
vmem:50:bootfs_minors:populate_wait 0
vmem:50:bootfs_minors:search 2
vmem:50:bootfs_minors:snaptime 7041.816542178
vmem:50:bootfs_minors:vmem_source 0
vmem:50:bootfs_minors:wait 0
vmem:51:ip_minor_arena_sa:alloc 1
vmem:51:ip_minor_arena_sa:class vmem
vmem:51:ip_minor_arena_sa:contains 0
vmem:51:ip_minor_arena_sa:contains_search 0
vmem:51:ip_minor_arena_sa:crtime 19.075826158
vmem:51:ip_minor_arena_sa:fail 0
vmem:51:ip_minor_arena_sa:free 0
vmem:51:ip_minor_arena_sa:lookup 0
vmem:51:ip_minor_arena_sa:mem_import 0
vmem:51:ip_minor_arena_sa:mem_inuse 64
vmem:51:ip_minor_arena_sa:mem_total 262140
vmem:51:ip_minor_arena_sa:populate_fail 0
vmem:51:ip_minor_arena_sa:populate_wait 0
vmem:51:ip_minor_arena_sa:search 0
vmem:51:ip_minor_arena_sa:snaptime 7041.816559828
vmem:51:ip_minor_arena_sa:vmem_source 0
vmem:51:ip_minor_arena_sa:wait 0
vmem:52:ip_minor_arena_la:alloc 1
vmem:52:ip_minor_arena_la:class vmem
vmem:52:ip_minor_arena_la:contains 0
vmem:52:ip_minor_arena_la:contains_search 0
vmem:52:ip_minor_arena_la:crtime 19.075837698
vmem:52:ip_minor_arena_la:fail 0
vmem:52:ip_minor_arena_la:free 0
vmem:52:ip_minor_arena_la:lookup 0
vmem:52:ip_minor_arena_la:mem_import 0
vmem:52:ip_minor_arena_la:mem_inuse 64
vmem:52:ip_minor_arena_la:mem_total 4294705152
vmem:52:ip_minor_arena_la:populate_fail 0
vmem:52:ip_minor_arena_la:populate_wait 0
vmem:52:ip_minor_arena_la:search 0
vmem:52:ip_minor_arena_la:snaptime 7041.816573339
vmem:52:ip_minor_arena_la:vmem_source 0
vmem:52:ip_minor_arena_la:wait 0
vmem:53:lib_va_32:alloc 20
vmem:53:lib_va_32:class vmem
vmem:53:lib_va_32:contains 0
vmem:53:lib_va_32:contains_search 0
vmem:53:lib_va_32:crtime 20.567654148
vmem:53:lib_va_32:fail 0
vmem:53:lib_va_32:free 0
vmem:53:lib_va_32:lookup 0
vmem:53:lib_va_32:mem_import 0
vmem:53:lib_va_32:mem_inuse 8347648
vmem:53:lib_va_32:mem_total 2138767360
vmem:53:lib_va_32:populate_fail 0
vmem:53:lib_va_32:populate_wait 0
vmem:53:lib_va_32:search 20
vmem:53:lib_va_32:snaptime 7041.816610354
vmem:53:lib_va_32:vmem_source 0
vmem:53:lib_va_32:wait 0
vmem:54:lport-instances:alloc 0
vmem:54:lport-instances:class vmem
vmem:54:lport-instances:contains 0
vmem:54:lport-instances:contains_search 0
vmem:54:lport-instances:crtime 20.621196542
vmem:54:lport-instances:fail 0
vmem:54:lport-instances:free 0
vmem:54:lport-instances:lookup 0
vmem:54:lport-instances:mem_import 0
vmem:54:lport-instances:mem_inuse 0
vmem:54:lport-instances:mem_total 65536
vmem:54:lport-instances:populate_fail 0
vmem:54:lport-instances:populate_wait 0
vmem:54:lport-instances:search 0
vmem:54:lport-instances:snaptime 7041.816624197
vmem:54:lport-instances:vmem_source 0
vmem:54:lport-instances:wait 0
vmem:55:rport-instances:alloc 0
vmem:55:rport-instances:class vmem
vmem:55:rport-instances:contains 0
vmem:55:rport-instances:contains_search 0
vmem:55:rport-instances:crtime 20.621205920
vmem:55:rport-instances:fail 0
vmem:55:rport-instances:free 0
vmem:55:rport-instances:lookup 0
vmem:55:rport-instances:mem_import 0
vmem:55:rport-instances:mem_inuse 0
vmem:55:rport-instances:mem_total 65536
vmem:55:rport-instances:populate_fail 0
vmem:55:rport-instances:populate_wait 0
vmem:55:rport-instances:search 0
vmem:55:rport-instances:snaptime 7041.816637693
vmem:55:rport-instances:vmem_source 0
vmem:55:rport-instances:wait 0
vmem:56:tl_minor_space:alloc 826
vmem:56:tl_minor_space:class vmem
vmem:56:tl_minor_space:contains 0
vmem:56:tl_minor_space:contains_search 0
vmem:56:tl_minor_space:crtime 21.252066609
vmem:56:tl_minor_space:fail 0
vmem:56:tl_minor_space:free 792
vmem:56:tl_minor_space:lookup 0
vmem:56:tl_minor_space:mem_import 0
vmem:56:tl_minor_space:mem_inuse 34
vmem:56:tl_minor_space:mem_total 262138
vmem:56:tl_minor_space:populate_fail 0
vmem:56:tl_minor_space:populate_wait 0
vmem:56:tl_minor_space:search 2
vmem:56:tl_minor_space:snaptime 7041.816679510
vmem:56:tl_minor_space:vmem_source 0
vmem:56:tl_minor_space:wait 0
vmem:57:keysock:alloc 0
vmem:57:keysock:class vmem
vmem:57:keysock:contains 0
vmem:57:keysock:contains_search 0
vmem:57:keysock:crtime 21.253319123
vmem:57:keysock:fail 0
vmem:57:keysock:free 0
vmem:57:keysock:lookup 0
vmem:57:keysock:mem_import 0
vmem:57:keysock:mem_inuse 0
vmem:57:keysock:mem_total 4294967295
vmem:57:keysock:populate_fail 0
vmem:57:keysock:populate_wait 0
vmem:57:keysock:search 0
vmem:57:keysock:snaptime 7041.816695124
vmem:57:keysock:vmem_source 0
vmem:57:keysock:wait 0
vmem:58:spdsock:alloc 1
vmem:58:spdsock:class vmem
vmem:58:spdsock:contains 0
vmem:58:spdsock:contains_search 0
vmem:58:spdsock:crtime 21.257343685
vmem:58:spdsock:fail 0
vmem:58:spdsock:free 0
vmem:58:spdsock:lookup 0
vmem:58:spdsock:mem_import 0
vmem:58:spdsock:mem_inuse 64
vmem:58:spdsock:mem_total 4294967295
vmem:58:spdsock:populate_fail 0
vmem:58:spdsock:populate_wait 0
vmem:58:spdsock:search 0
vmem:58:spdsock:snaptime 7041.816709123
vmem:58:spdsock:vmem_source 0
vmem:58:spdsock:wait 0
vmem:59:lofi_id:alloc 1
vmem:59:lofi_id:class vmem
vmem:59:lofi_id:contains 0
vmem:59:lofi_id:contains_search 0
vmem:59:lofi_id:crtime 23.588251440
vmem:59:lofi_id:fail 0
vmem:59:lofi_id:free 0
vmem:59:lofi_id:lookup 0
vmem:59:lofi_id:mem_import 0
vmem:59:lofi_id:mem_inuse 1
vmem:59:lofi_id:mem_total 67108863
vmem:59:lofi_id:populate_fail 0
vmem:59:lofi_id:populate_wait 0
vmem:59:lofi_id:search 1
vmem:59:lofi_id:snaptime 7041.816724322
vmem:59:lofi_id:vmem_source 0
vmem:59:lofi_id:wait 0
vmem:60:lib_va_64:alloc 134
vmem:60:lib_va_64:class vmem
vmem:60:lib_va_64:contains 0
vmem:60:lib_va_64:contains_search 0
vmem:60:lib_va_64:crtime 24.026199225
vmem:60:lib_va_64:fail 0
vmem:60:lib_va_64:free 0
vmem:60:lib_va_64:lookup 0
vmem:60:lib_va_64:mem_import 0
vmem:60:lib_va_64:mem_inuse 124862464
vmem:60:lib_va_64:mem_total 136888915828736
vmem:60:lib_va_64:populate_fail 0
vmem:60:lib_va_64:populate_wait 0
vmem:60:lib_va_64:search 134
vmem:60:lib_va_64:snaptime 7041.816739191
vmem:60:lib_va_64:vmem_source 0
vmem:60:lib_va_64:wait 0
vmem:61:dtrace:alloc 18550
vmem:61:dtrace:class vmem
vmem:61:dtrace:contains 0
vmem:61:dtrace:contains_search 0
vmem:61:dtrace:crtime 25.188633613
vmem:61:dtrace:fail 0
vmem:61:dtrace:free 18461
vmem:61:dtrace:lookup 32
vmem:61:dtrace:mem_import 0
vmem:61:dtrace:mem_inuse 89
vmem:61:dtrace:mem_total 4294967295
vmem:61:dtrace:populate_fail 0
vmem:61:dtrace:populate_wait 0
vmem:61:dtrace:search 37010
vmem:61:dtrace:snaptime 7041.816753478
vmem:61:dtrace:vmem_source 0
vmem:61:dtrace:wait 0
vmem:62:dtrace_minor:alloc 0
vmem:62:dtrace_minor:class vmem
vmem:62:dtrace_minor:contains 0
vmem:62:dtrace_minor:contains_search 0
vmem:62:dtrace_minor:crtime 25.188649726
vmem:62:dtrace_minor:fail 0
vmem:62:dtrace_minor:free 0
vmem:62:dtrace_minor:lookup 0
vmem:62:dtrace_minor:mem_import 0
vmem:62:dtrace_minor:mem_inuse 0
vmem:62:dtrace_minor:mem_total 4294967293
vmem:62:dtrace_minor:populate_fail 0
vmem:62:dtrace_minor:populate_wait 0
vmem:62:dtrace_minor:search 0
vmem:62:dtrace_minor:snaptime 7041.816767295
vmem:62:dtrace_minor:vmem_source 0
vmem:62:dtrace_minor:wait 0
vmem:68:overlay_target_minors:alloc 1
vmem:68:overlay_target_minors:class vmem
vmem:68:overlay_target_minors:contains 0
vmem:68:overlay_target_minors:contains_search 0
vmem:68:overlay_target_minors:crtime 26.544120711
vmem:68:overlay_target_minors:fail 0
vmem:68:overlay_target_minors:free 0
vmem:68:overlay_target_minors:lookup 0
vmem:68:overlay_target_minors:mem_import 0
vmem:68:overlay_target_minors:mem_inuse 1
vmem:68:overlay_target_minors:mem_total 2147483646
vmem:68:overlay_target_minors:populate_fail 0
vmem:68:overlay_target_minors:populate_wait 0
vmem:68:overlay_target_minors:search 2
vmem:68:overlay_target_minors:snaptime 7041.816804778
vmem:68:overlay_target_minors:vmem_source 0
vmem:68:overlay_target_minors:wait 0
vmem:69:vnd_minors:alloc 0
vmem:69:vnd_minors:class vmem
vmem:69:vnd_minors:contains 0
vmem:69:vnd_minors:contains_search 0
vmem:69:vnd_minors:crtime 26.550900013
vmem:69:vnd_minors:fail 0
vmem:69:vnd_minors:free 0
vmem:69:vnd_minors:lookup 0
vmem:69:vnd_minors:mem_import 0
vmem:69:vnd_minors:mem_inuse 0
vmem:69:vnd_minors:mem_total 2147483646
vmem:69:vnd_minors:populate_fail 0
vmem:69:vnd_minors:populate_wait 0
vmem:69:vnd_minors:search 0
vmem:69:vnd_minors:snaptime 7041.816820000
vmem:69:vnd_minors:vmem_source 0
vmem:69:vnd_minors:wait 0
vmem:70:logdmux_minor:alloc 0
vmem:70:logdmux_minor:class vmem
vmem:70:logdmux_minor:contains 0
vmem:70:logdmux_minor:contains_search 0
vmem:70:logdmux_minor:crtime 26.571093664
vmem:70:logdmux_minor:fail 0
vmem:70:logdmux_minor:free 0
vmem:70:logdmux_minor:lookup 0
vmem:70:logdmux_minor:mem_import 0
vmem:70:logdmux_minor:mem_inuse 0
vmem:70:logdmux_minor:mem_total 256
vmem:70:logdmux_minor:populate_fail 0
vmem:70:logdmux_minor:populate_wait 0
vmem:70:logdmux_minor:search 0
vmem:70:logdmux_minor:snaptime 7041.816833730
vmem:70:logdmux_minor:vmem_source 0
vmem:70:logdmux_minor:wait 0
vmem:71:ptms_minor:alloc 3
vmem:71:ptms_minor:class vmem
vmem:71:ptms_minor:contains 0
vmem:71:ptms_minor:contains_search 0
vmem:71:ptms_minor:crtime 26.574400673
vmem:71:ptms_minor:fail 0
vmem:71:ptms_minor:free 1
vmem:71:ptms_minor:lookup 0
vmem:71:ptms_minor:mem_import 0
vmem:71:ptms_minor:mem_inuse 2
vmem:71:ptms_minor:mem_total 16
vmem:71:ptms_minor:populate_fail 0
vmem:71:ptms_minor:populate_wait 0
vmem:71:ptms_minor:search 0
vmem:71:ptms_minor:snaptime 7041.816847406
vmem:71:ptms_minor:vmem_source 0
vmem:71:ptms_minor:wait 0
vmem:73:ipf_minor:alloc 31
vmem:73:ipf_minor:class vmem
vmem:73:ipf_minor:contains 0
vmem:73:ipf_minor:contains_search 0
vmem:73:ipf_minor:crtime 26.667819089
vmem:73:ipf_minor:fail 0
vmem:73:ipf_minor:free 30
vmem:73:ipf_minor:lookup 0
vmem:73:ipf_minor:mem_import 0
vmem:73:ipf_minor:mem_inuse 1
vmem:73:ipf_minor:mem_total 4294967294
vmem:73:ipf_minor:populate_fail 0
vmem:73:ipf_minor:populate_wait 0
vmem:73:ipf_minor:search 62
vmem:73:ipf_minor:snaptime 7041.816860961
vmem:73:ipf_minor:vmem_source 0
vmem:73:ipf_minor:wait 0
vmem:74:sppptun_minor:alloc 0
vmem:74:sppptun_minor:class vmem
vmem:74:sppptun_minor:contains 0
vmem:74:sppptun_minor:contains_search 0
vmem:74:sppptun_minor:crtime 26.690987998
vmem:74:sppptun_minor:fail 0
vmem:74:sppptun_minor:free 0
vmem:74:sppptun_minor:lookup 0
vmem:74:sppptun_minor:mem_import 0
vmem:74:sppptun_minor:mem_inuse 0
vmem:74:sppptun_minor:mem_total 16
vmem:74:sppptun_minor:populate_fail 0
vmem:74:sppptun_minor:populate_wait 0
vmem:74:sppptun_minor:search 0
vmem:74:sppptun_minor:snaptime 7041.816874679
vmem:74:sppptun_minor:vmem_source 0
vmem:74:sppptun_minor:wait 0
vmem:79:vmm_minors:alloc 0
vmem:79:vmm_minors:class vmem
vmem:79:vmm_minors:contains 0
vmem:79:vmm_minors:contains_search 0
vmem:79:vmm_minors:crtime 26.909226220
vmem:79:vmm_minors:fail 0
vmem:79:vmm_minors:free 0
vmem:79:vmm_minors:lookup 0
vmem:79:vmm_minors:mem_import 0
vmem:79:vmm_minors:mem_inuse 0
vmem:79:vmm_minors:mem_total 262142
vmem:79:vmm_minors:populate_fail 0
vmem:79:vmm_minors:populate_wait 0
vmem:79:vmm_minors:search 0
vmem:79:vmm_minors:snaptime 7041.816892260
vmem:79:vmm_minors:vmem_source 0
vmem:79:vmm_minors:wait 0
vmem:82:devfsadm_event_channel:alloc 1
vmem:82:devfsadm_event_channel:class vmem
vmem:82:devfsadm_event_channel:contains 0
vmem:82:devfsadm_event_channel:contains_search 0
vmem:82:devfsadm_event_channel:crtime 30.424926041
vmem:82:devfsadm_event_channel:fail 0
vmem:82:devfsadm_event_channel:free 0
vmem:82:devfsadm_event_channel:lookup 0
vmem:82:devfsadm_event_channel:mem_import 0
vmem:82:devfsadm_event_channel:mem_inuse 1
vmem:82:devfsadm_event_channel:mem_total 101
vmem:82:devfsadm_event_channel:populate_fail 0
vmem:82:devfsadm_event_channel:populate_wait 0
vmem:82:devfsadm_event_channel:search 2
vmem:82:devfsadm_event_channel:snaptime 7041.816915033
vmem:82:devfsadm_event_channel:vmem_source 0
vmem:82:devfsadm_event_channel:wait 0
vmem:83:devfsadm_event_channel:alloc 1
vmem:83:devfsadm_event_channel:class vmem
vmem:83:devfsadm_event_channel:contains 0
vmem:83:devfsadm_event_channel:contains_search 0
vmem:83:devfsadm_event_channel:crtime 30.424966800
vmem:83:devfsadm_event_channel:fail 0
vmem:83:devfsadm_event_channel:free 0
vmem:83:devfsadm_event_channel:lookup 0
vmem:83:devfsadm_event_channel:mem_import 0
vmem:83:devfsadm_event_channel:mem_inuse 1
vmem:83:devfsadm_event_channel:mem_total 2
vmem:83:devfsadm_event_channel:populate_fail 0
vmem:83:devfsadm_event_channel:populate_wait 0
vmem:83:devfsadm_event_channel:search 2
vmem:83:devfsadm_event_channel:snaptime 7041.816928983
vmem:83:devfsadm_event_channel:vmem_source 0
vmem:83:devfsadm_event_channel:wait 0
vmem:84:syseventconfd_door:alloc 0
vmem:84:syseventconfd_door:class vmem
vmem:84:syseventconfd_door:contains 0
vmem:84:syseventconfd_door:contains_search 0
vmem:84:syseventconfd_door:crtime 30.441343572
vmem:84:syseventconfd_door:fail 0
vmem:84:syseventconfd_door:free 0
vmem:84:syseventconfd_door:lookup 0
vmem:84:syseventconfd_door:mem_import 0
vmem:84:syseventconfd_door:mem_inuse 0
vmem:84:syseventconfd_door:mem_total 101
vmem:84:syseventconfd_door:populate_fail 0
vmem:84:syseventconfd_door:populate_wait 0
vmem:84:syseventconfd_door:search 0
vmem:84:syseventconfd_door:snaptime 7041.816942773
vmem:84:syseventconfd_door:vmem_source 0
vmem:84:syseventconfd_door:wait 0
vmem:85:syseventconfd_door:alloc 1
vmem:85:syseventconfd_door:class vmem
vmem:85:syseventconfd_door:contains 0
vmem:85:syseventconfd_door:contains_search 0
vmem:85:syseventconfd_door:crtime 30.441360871
vmem:85:syseventconfd_door:fail 0
vmem:85:syseventconfd_door:free 0
vmem:85:syseventconfd_door:lookup 0
vmem:85:syseventconfd_door:mem_import 0
vmem:85:syseventconfd_door:mem_inuse 1
vmem:85:syseventconfd_door:mem_total 2
vmem:85:syseventconfd_door:populate_fail 0
vmem:85:syseventconfd_door:populate_wait 0
vmem:85:syseventconfd_door:search 2
vmem:85:syseventconfd_door:snaptime 7041.816956539
vmem:85:syseventconfd_door:vmem_source 0
vmem:85:syseventconfd_door:wait 0
vmem:86:syseventd_channel:alloc 2
vmem:86:syseventd_channel:class vmem
vmem:86:syseventd_channel:contains 0
vmem:86:syseventd_channel:contains_search 0
vmem:86:syseventd_channel:crtime 30.444381239
vmem:86:syseventd_channel:fail 0
vmem:86:syseventd_channel:free 0
vmem:86:syseventd_channel:lookup 0
vmem:86:syseventd_channel:mem_import 0
vmem:86:syseventd_channel:mem_inuse 2
vmem:86:syseventd_channel:mem_total 101
vmem:86:syseventd_channel:populate_fail 0
vmem:86:syseventd_channel:populate_wait 0
vmem:86:syseventd_channel:search 2
vmem:86:syseventd_channel:snaptime 7041.816970462
vmem:86:syseventd_channel:vmem_source 0
vmem:86:syseventd_channel:wait 0
vmem:87:syseventd_channel:alloc 1
vmem:87:syseventd_channel:class vmem
vmem:87:syseventd_channel:contains 0
vmem:87:syseventd_channel:contains_search 0
vmem:87:syseventd_channel:crtime 30.444392367
vmem:87:syseventd_channel:fail 0
vmem:87:syseventd_channel:free 0
vmem:87:syseventd_channel:lookup 0
vmem:87:syseventd_channel:mem_import 0
vmem:87:syseventd_channel:mem_inuse 1
vmem:87:syseventd_channel:mem_total 2
vmem:87:syseventd_channel:populate_fail 0
vmem:87:syseventd_channel:populate_wait 0
vmem:87:syseventd_channel:search 2
vmem:87:syseventd_channel:snaptime 7041.816984513
vmem:87:syseventd_channel:vmem_source 0
vmem:87:syseventd_channel:wait 0
vmem:88:Hex0xfffffe1fee182518_minor:alloc 0
vmem:88:Hex0xfffffe1fee182518_minor:class vmem
vmem:88:Hex0xfffffe1fee182518_minor:contains 0
vmem:88:Hex0xfffffe1fee182518_minor:contains_search 0
vmem:88:Hex0xfffffe1fee182518_minor:crtime 36.762355537
vmem:88:Hex0xfffffe1fee182518_minor:fail 0
vmem:88:Hex0xfffffe1fee182518_minor:free 0
vmem:88:Hex0xfffffe1fee182518_minor:lookup 0
vmem:88:Hex0xfffffe1fee182518_minor:mem_import 0
vmem:88:Hex0xfffffe1fee182518_minor:mem_inuse 0
vmem:88:Hex0xfffffe1fee182518_minor:mem_total 4294967294
vmem:88:Hex0xfffffe1fee182518_minor:populate_fail 0
vmem:88:Hex0xfffffe1fee182518_minor:populate_wait 0
vmem:88:Hex0xfffffe1fee182518_minor:search 0
vmem:88:Hex0xfffffe1fee182518_minor:snaptime 7041.816998728
vmem:88:Hex0xfffffe1fee182518_minor:vmem_source 0
vmem:88:Hex0xfffffe1fee182518_minor:wait 0
vmem:89:Hex0xfffffe1fee182520_minor:alloc 0
vmem:89:Hex0xfffffe1fee182520_minor:class vmem
vmem:89:Hex0xfffffe1fee182520_minor:contains 0
vmem:89:Hex0xfffffe1fee182520_minor:contains_search 0
vmem:89:Hex0xfffffe1fee182520_minor:crtime 36.764578965
vmem:89:Hex0xfffffe1fee182520_minor:fail 0
vmem:89:Hex0xfffffe1fee182520_minor:free 0
vmem:89:Hex0xfffffe1fee182520_minor:lookup 0
vmem:89:Hex0xfffffe1fee182520_minor:mem_import 0
vmem:89:Hex0xfffffe1fee182520_minor:mem_inuse 0
vmem:89:Hex0xfffffe1fee182520_minor:mem_total 4294967294
vmem:89:Hex0xfffffe1fee182520_minor:populate_fail 0
vmem:89:Hex0xfffffe1fee182520_minor:populate_wait 0
vmem:89:Hex0xfffffe1fee182520_minor:search 0
vmem:89:Hex0xfffffe1fee182520_minor:snaptime 7041.817072438
vmem:89:Hex0xfffffe1fee182520_minor:vmem_source 0
vmem:89:Hex0xfffffe1fee182520_minor:wait 0
vmem:90:Hex0xfffffe200d9d6518_minor:alloc 0
vmem:90:Hex0xfffffe200d9d6518_minor:class vmem
vmem:90:Hex0xfffffe200d9d6518_minor:contains 0
vmem:90:Hex0xfffffe200d9d6518_minor:contains_search 0
vmem:90:Hex0xfffffe200d9d6518_minor:crtime 71.280637066
vmem:90:Hex0xfffffe200d9d6518_minor:fail 0
vmem:90:Hex0xfffffe200d9d6518_minor:free 0
vmem:90:Hex0xfffffe200d9d6518_minor:lookup 0
vmem:90:Hex0xfffffe200d9d6518_minor:mem_import 0
vmem:90:Hex0xfffffe200d9d6518_minor:mem_inuse 0
vmem:90:Hex0xfffffe200d9d6518_minor:mem_total 4294967294
vmem:90:Hex0xfffffe200d9d6518_minor:populate_fail 0
vmem:90:Hex0xfffffe200d9d6518_minor:populate_wait 0
vmem:90:Hex0xfffffe200d9d6518_minor:search 0
vmem:90:Hex0xfffffe200d9d6518_minor:snaptime 7041.817104515
vmem:90:Hex0xfffffe200d9d6518_minor:vmem_source 0
vmem:90:Hex0xfffffe200d9d6518_minor:wait 0
vmem:91:Hex0xfffffe200d9d6520_minor:alloc 0
vmem:91:Hex0xfffffe200d9d6520_minor:class vmem
vmem:91:Hex0xfffffe200d9d6520_minor:contains 0
vmem:91:Hex0xfffffe200d9d6520_minor:contains_search 0
vmem:91:Hex0xfffffe200d9d6520_minor:crtime 71.282533072
vmem:91:Hex0xfffffe200d9d6520_minor:fail 0
vmem:91:Hex0xfffffe200d9d6520_minor:free 0
vmem:91:Hex0xfffffe200d9d6520_minor:lookup 0
vmem:91:Hex0xfffffe200d9d6520_minor:mem_import 0
vmem:91:Hex0xfffffe200d9d6520_minor:mem_inuse 0
vmem:91:Hex0xfffffe200d9d6520_minor:mem_total 4294967294
vmem:91:Hex0xfffffe200d9d6520_minor:populate_fail 0
vmem:91:Hex0xfffffe200d9d6520_minor:populate_wait 0
vmem:91:Hex0xfffffe200d9d6520_minor:search 0
vmem:91:Hex0xfffffe200d9d6520_minor:snaptime 7041.817120442
vmem:91:Hex0xfffffe200d9d6520_minor:vmem_source 0
vmem:91:Hex0xfffffe200d9d6520_minor:wait 0

ppriv $$ (current process privileges):

5519: -bash
flags = <none>
 E: basic
 I: basic
 P: basic
 L: basic,contract_event,contract_identity,contract_observer,dtrace_proc,dtrace_user,file_chown,file_chown_self,file_dac_execute,file_dac_read,file_dac_search,file_dac_write,file_owner,file_setid,ipc_dac_read,ipc_dac_write,ipc_owner,net_bindmlp,net_icmpaccess,net_mac_aware,net_observability,net_privaddr,net_rawaccess,proc_audit,proc_chroot,proc_lock_memory,proc_owner,proc_prioup,proc_setid,proc_taskid,sys_acct,sys_admin,sys_audit,sys_fs_import,sys_ip_config,sys_iptun_config,sys_mount,sys_nfs,sys_ppp_config,sys_resource,sys_smb

---

With sudo

we cross the memory bridge and fail later:

sudo gdb --args artifacts/bin/coreclr/SunOS.x64.Debug/corerun \
 ./src/libraries/Common/tests/Data/TinyAssembly.dll

(gdb) r
Starting program: /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/corerun ./src/libraries/Common/tests/Data/TinyAssembly.dll
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
[New LWP 2 ]
[New Thread 2 (LWP 2)]

Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xfffffc7fe864d218 in InitializeSpinConstants() () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
(gdb) thread apply all bt

Thread 4 (Thread 2 (LWP 2)):
#0 0xfffffc7fef25c8da in __pollsys () from /lib/64/libc.so.1
#1 0xfffffc7fef247325 in _pollsys () from /lib/64/libc.so.1
#2 0xfffffc7fef1eda01 in poll () from /lib/64/libc.so.1
#3 0xfffffc7fe8a9df8c in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe (this=0x438820, iTimeout=-1, pRecvBuf=0xfffffc7feeacbd87 "", iBytes=1)
 at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:2234
#4 0xfffffc7fe8a9d6e9 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (this=0x438820, iPollTimeout=-1, pswcWorkerCmd=0xfffffc7feeacbe40, pshridMarshaledData=0xfffffc7feeacbef0, 
 pdwData=0xfffffc7feeacbe44) at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:2012
#5 0xfffffc7fe8a9c4fb in CorUnix::CPalSynchronizationManager::WorkerThread (pArg=0x438820) at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1715
#6 0xfffffc7fe8ab8989 in CorUnix::CPalThread::ThreadEntry (pvParam=0x439270) at /home/am11/runtime/src/coreclr/src/pal/src/thread/thread.cpp:1802
#7 0xfffffc7fef255a1c in _thrp_setup () from /lib/64/libc.so.1
#8 0xfffffc7fef255d30 in ?? () from /lib/64/libc.so.1
#9 0x0000000000000000 in ?? ()

Thread 3 (LWP 2 ):
#0 0xfffffc7fef25c8da in __pollsys () from /lib/64/libc.so.1
#1 0xfffffc7fef247325 in _pollsys () from /lib/64/libc.so.1
#2 0xfffffc7fef1eda01 in poll () from /lib/64/libc.so.1
#3 0xfffffc7fe8a9df8c in CorUnix::CPalSynchronizationManager::ReadBytesFromProcessPipe (this=0x438820, iTimeout=-1, pRecvBuf=0xfffffc7feeacbd87 "", iBytes=1)
 at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:2234
#4 0xfffffc7fe8a9d6e9 in CorUnix::CPalSynchronizationManager::ReadCmdFromProcessPipe (this=0x438820, iPollTimeout=-1, pswcWorkerCmd=0xfffffc7feeacbe40, pshridMarshaledData=0xfffffc7feeacbef0, 
 pdwData=0xfffffc7feeacbe44) at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:2012
#5 0xfffffc7fe8a9c4fb in CorUnix::CPalSynchronizationManager::WorkerThread (pArg=0x438820) at /home/am11/runtime/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:1715
#6 0xfffffc7fe8ab8989 in CorUnix::CPalThread::ThreadEntry (pvParam=0x439270) at /home/am11/runtime/src/coreclr/src/pal/src/thread/thread.cpp:1802
#7 0xfffffc7fef255a1c in _thrp_setup () from /lib/64/libc.so.1
#8 0xfffffc7fef255d30 in ?? () from /lib/64/libc.so.1
#9 0x0000000000000000 in ?? ()

Thread 2 (Thread 1 (LWP 1)):
#0 0xfffffc7fe864d218 in InitializeSpinConstants() () from /home/am11/runtime/artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
#1 0xfffffc7fe864ac43 in EEStartupHelper () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:773
#2 0xfffffc7fe864b484 in <lambda(PVOID)>::operator()(PVOID) const (__closure=0xfffffc7fffdff61a, p=0x0) at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1132
#3 0xfffffc7fe864b5de in EEStartup () at /home/am11/runtime/src/coreclr/src/vm/ceemain.cpp:1134

(for some reason gdb doesn't load libcoreclr.so symbols when running with corerun)

It fails in libcoreclr, where we try to assign value to a global variable defined in other binary (corerun):

runtime/src/coreclr/src/vm/syncblk.h

Line 146 in b41296a

g_SpinConstants.dwInitialDuration = g_pConfig->SpinInitialDuration();

it happens at the assignment; after g_pConfig->SpinInitialDuration() returns (value of 50). Before this line it successfully prints the initial value of g_SpinConstants.dwInitialDuration, which is also 50, so the memory is not corrupt.

truss -f result: http://sprunge.us/PfoIZl.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[am11]

cc @janvorli, @jclulow

[jclulow]

Looking at the truss output in the non-root case, it seems like the process could not lock the memory because it does not have the privilege to do so:

...
56362/1:	mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEF010000
56362/1:	memcntl(0xFFFFFC7FEF010000, 4096, MC_LOCK, 0, 0, 0) Err#1 EPERM [proc_lock_memory]
...

Note that our mlock(3C) is defined to return EPERM if you do not have the appropriate privilege:

ERRORS

 The mlock() and munlock() functions will fail if:
...
 EPERM
 The {PRIV_PROC_LOCK_MEMORY} privilege is not asserted in the
 effective set of the calling process.

You don't see an mlock() call in the truss output because it's looking at system calls, and our mlock() is a small wrapper around the memcntl(2) system call:

/*
 * Function to lock address range in memory.
 */
int
mlock(caddr_t addr, size_t len)
{
 return (memcntl(addr, len, MC_LOCK, 0, 0, 0));
}

In your ppriv output you can see that you inheritable/permitted/effective set is just basic, and the proc_lock_memory privilege is not part of the basic set that non-root processes generally start with.

[jclulow]

Note that the privilege model is described in privileges(5), including which privileges are in the basic set.

I'm not sure what's going on with the segfault you're seeing. I suspect you'll need to look at the program text in the compiled binary to see what's going on there -- what kind of relocation it's using to find the external variable, etc.

[am11]

Thank you @jclulow! Adding the proc_lock_memory privilege fixed the non-sudo issue.

sudo usermod -K defaultpriv='basic,proc_lock_memory' am11
# logout and `zlogin -lam11` back to the zone
ppriv $$

outputs:

56702: -bash
flags = <none>
 E: basic,proc_lock_memory
 I: basic,proc_lock_memory
 P: basic,proc_lock_memory
 L: basic,contract_event,contract_identity,contract_observer,dtrace_proc,dtrace_user,file_chown,file_chown_self,file_dac_execute,file_dac_read,file_dac_search,file_dac_write,file_owner,file_setid,ipc_dac_read,ipc_dac_write,ipc_owner,net_bindmlp,net_icmpaccess,net_mac_aware,net_observability,net_privaddr,net_rawaccess,proc_audit,proc_chroot,proc_lock_memory,proc_owner,proc_prioup,proc_setid,proc_taskid,sys_acct,sys_admin,sys_audit,sys_fs_import,sys_ip_config,sys_iptun_config,sys_mount,sys_nfs,sys_ppp_config,sys_resource,sys_smb

Now, both sudo and non-sudo invocations are aligned; fail with the same sigsegv.

@janvorli, can we devise some other mechanism to ensure mprotect calls will not desync, in case mlock fails? Otherwise, I think a more specific error message would help the user in this case.

[am11]

you'll need to look at the program text in the compiled binary to see what's going on there

with objdump -D, found this information in libcoreclr.so (there is nothing in corerun executable related to .*spin.*, so my assumption was wrong about extern, global is initialized and used within libcoreclr.so):

$ objdump -D artifacts/bin/coreclr/$(uname).x64.Debug/libcoreclr.so | grep -A15 -i initializespinconst.*:
0000000000add1f4 <_Z23InitializeSpinConstantsv>:
 add1f4: 55 push %rbp
 add1f5: 48 89 e5 mov %rsp,%rbp
 add1f8: 53 push %rbx
 add1f9: 48 83 ec 08 sub $0x8,%rsp
 add1fd: 48 8b 05 fc 63 92 00 mov 0x9263fc(%rip),%rax # 1403600 <_GLOBAL_OFFSET_TABLE_+0x3c8>
 add204: 48 8b 00 mov (%rax),%rax
 add207: 48 89 c7 mov %rax,%rdi
 add20a: e8 f1 fe ff ff callq add100 <_ZNK8EEConfig19SpinInitialDurationEv>
 add20f: 89 c2 mov %eax,%edx
 add211: 48 8b 05 60 79 92 00 mov 0x927960(%rip),%rax # 1404b78 <_GLOBAL_OFFSET_TABLE_+0x1940>
 add218: 89 10 mov %edx,(%rax)
 add21a: 48 8b 05 df 63 92 00 mov 0x9263df(%rip),%rax # 1403600 <_GLOBAL_OFFSET_TABLE_+0x3c8>
 add221: 48 8b 00 mov (%rax),%rax
 add224: 48 89 c7 mov %rax,%rdi
 add227: e8 fc fe ff ff callq add128 <_ZNK8EEConfig16SpinLimitProcCapEv>

$ objdump -D artifacts/bin/coreclr/$(uname).x64.Debug/libcoreclr.so| grep -A10 -i g_spin
000000000140d880 <g_SpinConstants>:
 140d880: 32 00 xor (%rax),%al
 140d882: 00 00 add %al,(%rax)
 140d884: 40 9c rex pushfq 
 140d886: 00 00 add %al,(%rax)
 140d888: 03 00 add (%rax),%eax
 140d88a: 00 00 add %al,(%rax)
 140d88c: 0a 00 or (%rax),%al
 140d88e: 00 00 add %al,(%rax)
 140d890: 00 00 add %al,(%rax)

from the bt above (0xfffffc7fe864d218 ending with 218), looks like sigsegv is raised at:
add218: 89 10 mov %edx,(%rax).

uploaded some files at: https://github.com/am11/runtime/releases/tag/sunos-wip

[janvorli]

can we devise some other mechanism to ensure mprotect calls will not desync, in case mlock fails

I am not sure I understand what you are asking for. If the membarrier syscall with MEMBARRIER_CMD_PRIVATE_EXPEDITED is not supported and the mlock fails, it is a dead end. We have no way to flush process write buffers in that case and runtime would crash intermittently during / after a GC.

[janvorli]

If SmartOS has some other mean to do this flushing, we can obviously add that.

[am11]

Thank you @janvorli. I did not realize that the mlock was already in a fallback path. It makes sense to keep it as is, unless we find some alternative way on SmartOS/Illumos to flush the write buffer.

---

So far, I could not figure out the global address issue from text search. The variables defined in .got section should be writable. We can read this variable (e.g. printf(g_SpinConstants.dwInitialDuration); just before the assignment prints the value of 50 correctly), but value assignment (to even a constant) fails. So it is still a mystery.

[am11]

Tracked it down to an mprotect call: in InitGSCookie() after we set protection back to READONLY (see line 534 below), assignment g_SpinConstants.dwInitialDuration = 50 fails:

runtime/src/coreclr/src/vm/ceemain.cpp

Lines 479 to 538 in 49f59a3

	void InitGSCookie()
	{
	CONTRACTL
	{
	THROWS;
	GC_NOTRIGGER;
	MODE_ANY;
	}
	CONTRACTL_END;
	volatile GSCookie * pGSCookiePtr = GetProcessGSCookiePtr();
	#ifdef TARGET_UNIX
	// On Unix, the GS cookie is stored in a read only data segment
	DWORD newProtection = PAGE_READWRITE;
	#else // TARGET_UNIX
	DWORD newProtection = PAGE_EXECUTE_READWRITE;
	#endif // !TARGET_UNIX
	DWORD oldProtection;
	if(!ClrVirtualProtect((LPVOID)pGSCookiePtr, sizeof(GSCookie), newProtection, &oldProtection))
	{
	ThrowLastError();
	}
	#ifdef TARGET_UNIX
	// PAL layer is unable to extract old protection for regions that were not allocated using VirtualAlloc
	oldProtection = PAGE_READONLY;
	#endif // TARGET_UNIX
	#ifndef TARGET_UNIX
	// The GSCookie cannot be in a writeable page
	assert(((oldProtection & (PAGE_READWRITE|PAGE_WRITECOPY|PAGE_EXECUTE_READWRITE|
	PAGE_EXECUTE_WRITECOPY|PAGE_WRITECOMBINE)) == 0));
	// Forces VC cookie to be initialized.
	void * pf = &__security_check_cookie;
	pf = NULL;
	GSCookie val = (GSCookie)(__security_cookie ^ GetTickCount());
	#else // !TARGET_UNIX
	// REVIEW: Need something better for PAL...
	GSCookie val = (GSCookie)GetTickCount();
	#endif // !TARGET_UNIX
	#ifdef _DEBUG
	// In _DEBUG, always use the same value to make it easier to search for the cookie
	val = (GSCookie) BIT64_ONLY(0x9ABCDEF012345678) NOT_BIT64(0x12345678);
	#endif
	// To test if it is initialized. Also for ICorMethodInfo::getGSCookie()
	if (val == 0)
	val ++;
	*pGSCookiePtr = val;
	if(!ClrVirtualProtect((LPVOID)pGSCookiePtr, sizeof(GSCookie), oldProtection, &oldProtection))
	{
	ThrowLastError();
	}
	}

one line above (L533) the assignment works. @janvorli, does this mean we do not expect g_SpinConstants etc. to be affected by these protection changes and somehow it is happening on SmartOS but not the others? or something else?

[am11]

A bit more context: assignment before InitGSCookie() function call also works. It seems like after GSCookie val = (GSCookie)GetTickCount();, val refers to the region of memory that should not have global variables such as g_SpinConstants, but in case of SmartOS, it does, hence renders the global variables readonly. Perhaps, it is an indication to address this review comment:

 // REVIEW: Need something better for PAL...
 GSCookie val = (GSCookie)GetTickCount(); 

One workaround to unblock this situation is to change the protection to PAGE_READWRITE in the beginning of InitializeSpinConstants(), then change it back to PAGE_READONLY at the end, after the assignment to global variable is completed.

[janvorli]

It seems that this means that the GSCookie is already stored in read-write section. On other Unixes, the fact that it is marked as const results in it being placed in read only section.
There are two options there:

1. Force the GSCookie to get placed into a readonly section on SmartOS somehow. I've played with this concept in the past on OSX and Linux. IIRC, then I've created a new section using a linker options and added __attribute__((section("sectionname)). But maybe there is some readonly data section already that we could add the cookie to.
2. Or change the oldProtection for SmartOS to PAGE_READWRITE. This would kind of defeat the purpose of the cookie, but you can use it at least as a quick mean to see if there are more issues.