Migrate nestboxes to lua and add to autobutcher

[Birkow]

Nestboxes rewritten in lua with enhanced functionality, added to autobutcher.
Started with DFHack/scripts#1296, moved to dfhack to add as part of autobutcher
Attached performance check after ~1 year with 14 nestboxes and birds assigned to all of them. Script checks created egg item when called from eventful on ITEM_CREATED event.

dfhack-perf-counters.dat

Summary

Measuring unpaused time only

elapsed 1299094 ms (21m 39s)
dfhack 118211 ms ( 9.10% of elapsed)

DFHack details

keybinding 16 ms ( 0.01% of dfhack, 0.00% of elapsed)
update 25355 ms ( 21.45% of dfhack, 1.95% of elapsed)
overlay 92809 ms ( 78.51% of dfhack, 7.14% of elapsed)
zscreen 31 ms ( 0.03% of dfhack, 0.00% of elapsed)

Update details

event manager 4087 ms ( 16.12% of update, 0.31% of elapsed)
plugin onUpdate 17178 ms ( 67.75% of update, 1.32% of elapsed)
lua timers 3194 ms ( 12.60% of update, 0.25% of elapsed)

Event manager per event type

 UNIT_NEW_ACTIVE 1380 ms ( 33.77% of event manager, 0.11% of elapsed)
JOB_STARTED 1296 ms ( 31.71% of event manager, 0.10% of elapsed)
JOB_COMPLETED 847 ms ( 20.72% of event manager, 0.07% of elapsed)
JOB_INITIATED 127 ms ( 3.11% of event manager, 0.01% of elapsed)
BUILDING 61 ms ( 1.49% of event manager, 0.00% of elapsed)
ITEM_CREATED 31 ms ( 0.76% of event manager, 0.00% of elapsed)
framework 345 ms ( 8.44% of event manager, 0.03% of elapsed)
all subtimers 3742 ms ( 91.56% of event manager, 0.29% of elapsed)

Event manager JOB_INITIATED event per plugin

 eventful 63 ms ( 49.61% of event type, 0.00% of elapsed)
framework 64 ms ( 50.39% of event type, 0.00% of elapsed)
all subtimers 63 ms ( 49.61% of event type, 0.00% of elapsed)

Event manager JOB_COMPLETED event per plugin

 framework 847 ms (100.00% of event type, 0.07% of elapsed)
all subtimers 0 ms ( 0.00% of event type, 0.00% of elapsed)

Event manager BUILDING event per plugin

 framework 61 ms (100.00% of event type, 0.00% of elapsed)
all subtimers 0 ms ( 0.00% of event type, 0.00% of elapsed)

Event manager UNIT_NEW_ACTIVE event per plugin

 framework 1380 ms (100.00% of event type, 0.11% of elapsed)
all subtimers 0 ms ( 0.00% of event type, 0.00% of elapsed)

Event manager ITEM_CREATED event per plugin

 eventful 16 ms ( 51.61% of event type, 0.00% of elapsed)
framework 15 ms ( 48.39% of event type, 0.00% of elapsed)
all subtimers 16 ms ( 51.61% of event type, 0.00% of elapsed)

Event manager JOB_STARTED event per plugin

 burrow 15 ms ( 1.16% of event type, 0.00% of elapsed)
framework 1281 ms ( 98.84% of event type, 0.10% of elapsed)
all subtimers 15 ms ( 1.16% of event type, 0.00% of elapsed)

Update per plugin

 overlay 12762 ms ( 74.29% of update, 0.98% of elapsed)
autochop 316 ms ( 1.84% of update, 0.02% of elapsed)
preserve-rooms 265 ms ( 1.54% of update, 0.02% of elapsed)
logistics 237 ms ( 1.38% of update, 0.02% of elapsed)
autofarm 203 ms ( 1.18% of update, 0.02% of elapsed)
dwarfvet 186 ms ( 1.08% of update, 0.01% of elapsed)
RemoteFortressReader 169 ms ( 0.98% of update, 0.01% of elapsed)
suspendmanager 140 ms ( 0.81% of update, 0.01% of elapsed)
stonesense 125 ms ( 0.73% of update, 0.01% of elapsed)
buildingplan 110 ms ( 0.64% of update, 0.01% of elapsed)
add-spatter 109 ms ( 0.63% of update, 0.01% of elapsed)
seedwatch 108 ms ( 0.63% of update, 0.01% of elapsed)
autobutcher 107 ms ( 0.62% of update, 0.01% of elapsed)
burrow 94 ms ( 0.55% of update, 0.01% of elapsed)
3dveins 93 ms ( 0.54% of update, 0.01% of elapsed)
autonestbox 79 ms ( 0.46% of update, 0.01% of elapsed)
autoclothing 78 ms ( 0.45% of update, 0.01% of elapsed)
tweak 77 ms ( 0.45% of update, 0.01% of elapsed)
tailor 63 ms ( 0.37% of update, 0.00% of elapsed)
cxxrandom 62 ms ( 0.36% of update, 0.00% of elapsed)
autoslab 62 ms ( 0.36% of update, 0.00% of elapsed)
hotkeys 48 ms ( 0.28% of update, 0.00% of elapsed)
pet-uncapper 47 ms ( 0.27% of update, 0.00% of elapsed)
zone 47 ms ( 0.27% of update, 0.00% of elapsed)
plant 47 ms ( 0.27% of update, 0.00% of elapsed)
spectate 32 ms ( 0.19% of update, 0.00% of elapsed)
deramp 32 ms ( 0.19% of update, 0.00% of elapsed)
misery 32 ms ( 0.19% of update, 0.00% of elapsed)
showmood 32 ms ( 0.19% of update, 0.00% of elapsed)
preserve-tombs 32 ms ( 0.19% of update, 0.00% of elapsed)
aquifer 31 ms ( 0.18% of update, 0.00% of elapsed)
changevein 31 ms ( 0.18% of update, 0.00% of elapsed)
nestboxes 31 ms ( 0.18% of update, 0.00% of elapsed)
autolabor 31 ms ( 0.18% of update, 0.00% of elapsed)
fix-occupancy 31 ms ( 0.18% of update, 0.00% of elapsed)
cursecheck 31 ms ( 0.18% of update, 0.00% of elapsed)
tiletypes 31 ms ( 0.18% of update, 0.00% of elapsed)
pathable 30 ms ( 0.17% of update, 0.00% of elapsed)
fastdwarf 16 ms ( 0.09% of update, 0.00% of elapsed)
getplants 16 ms ( 0.09% of update, 0.00% of elapsed)
dig 16 ms ( 0.09% of update, 0.00% of elapsed)
flows 16 ms ( 0.09% of update, 0.00% of elapsed)
liquids 16 ms ( 0.09% of update, 0.00% of elapsed)
sort 16 ms ( 0.09% of update, 0.00% of elapsed)
createitem 16 ms ( 0.09% of update, 0.00% of elapsed)
dig-now 16 ms ( 0.09% of update, 0.00% of elapsed)
faststart 16 ms ( 0.09% of update, 0.00% of elapsed)
filltraffic 16 ms ( 0.09% of update, 0.00% of elapsed)
regrass 16 ms ( 0.09% of update, 0.00% of elapsed)
changelayer 16 ms ( 0.09% of update, 0.00% of elapsed)
reveal 16 ms ( 0.09% of update, 0.00% of elapsed)
cleanconst 16 ms ( 0.09% of update, 0.00% of elapsed)
orders 16 ms ( 0.09% of update, 0.00% of elapsed)
stocks 15 ms ( 0.09% of update, 0.00% of elapsed)
changeitem 15 ms ( 0.09% of update, 0.00% of elapsed)
blueprint 15 ms ( 0.09% of update, 0.00% of elapsed)
design 15 ms ( 0.09% of update, 0.00% of elapsed)
tubefill 15 ms ( 0.09% of update, 0.00% of elapsed)
strangemood 15 ms ( 0.09% of update, 0.00% of elapsed)
eventful 15 ms ( 0.09% of update, 0.00% of elapsed)
framework 792 ms ( 4.61% of update, 0.06% of elapsed)
all subtimers 16386 ms ( 95.39% of update, 1.26% of elapsed)

State change per plugin

 framework 17178 ms (100.00% of update, 1.32% of elapsed)
all subtimers 0 ms ( 0.00% of update, 0.00% of elapsed)

Lua timer details

 control-panel/fix/noexert-exhaustion 578 ms ( 18.10% of lua timers, 0.04% of elapsed)
control-panel/combine 566 ms ( 17.72% of lua timers, 0.04% of elapsed)
control-panel/fix/ownership 328 ms ( 10.27% of lua timers, 0.03% of elapsed)
control-panel/fix/empty-wheelbarrows 297 ms ( 9.30% of lua timers, 0.02% of elapsed)
control-panel/fix/stuck-instruments 237 ms ( 7.42% of lua timers, 0.02% of elapsed)
control-panel/fix/stuck-worship 220 ms ( 6.89% of lua timers, 0.02% of elapsed)
control-panel/fix/general-strike 220 ms ( 6.89% of lua timers, 0.02% of elapsed)
autofish 140 ms ( 4.38% of lua timers, 0.01% of elapsed)
control-panel/fix/engravings 124 ms ( 3.88% of lua timers, 0.01% of elapsed)
control-panel/fix/dry-buckets 80 ms ( 2.50% of lua timers, 0.01% of elapsed)
control-panel/orders-reevaluate 16 ms ( 0.50% of lua timers, 0.00% of elapsed)
control-panel/fix/dead-units 15 ms ( 0.47% of lua timers, 0.00% of elapsed)
control-panel/automilk 15 ms ( 0.47% of lua timers, 0.00% of elapsed)
control-panel/autoshear 15 ms ( 0.47% of lua timers, 0.00% of elapsed)
framework 343 ms ( 10.74% of lua timers, 0.03% of elapsed)
all subtimers 2851 ms ( 89.26% of lua timers, 0.22% of elapsed)

Overlay details

 agitation-rebalance.monitor 13287 ms ( 14.32% of overlay, 1.02% of elapsed)
gui/notify.panel 7348 ms ( 7.92% of overlay, 0.57% of elapsed)
hotkeys.menu 3108 ms ( 3.35% of overlay, 0.24% of elapsed)
sort.location_selector 2701 ms ( 2.91% of overlay, 0.21% of elapsed)
suspendmanager.overlay 2041 ms ( 2.20% of overlay, 0.16% of elapsed)
sort.info 1292 ms ( 1.39% of overlay, 0.10% of elapsed)
sort.elevate_barony 1236 ms ( 1.33% of overlay, 0.10% of elapsed)
notes.map_notes 1110 ms ( 1.20% of overlay, 0.09% of elapsed)
gui/civ-alert.big_red_button 910 ms ( 0.98% of overlay, 0.07% of elapsed)
sort.slab 893 ms ( 0.96% of overlay, 0.07% of elapsed)
sort.world 871 ms ( 0.94% of overlay, 0.07% of elapsed)
sort.candidates 866 ms ( 0.93% of overlay, 0.07% of elapsed)
sort.places 784 ms ( 0.84% of overlay, 0.06% of elapsed)
confirm.overlay 63 ms ( 0.07% of overlay, 0.00% of elapsed)
gui/design.dimensions 16 ms ( 0.02% of overlay, 0.00% of elapsed)
zone.retirelocation 15 ms ( 0.02% of overlay, 0.00% of elapsed)
framework 56268 ms ( 60.63% of overlay, 4.33% of elapsed)
all subtimers 36541 ms ( 39.37% of overlay, 2.81% of elapsed)

ZScreen details

 warn-stranded 31 ms (100.00% of zscreen, 0.00% of elapsed)
framework 0 ms ( 0.00% of zscreen, 0.00% of elapsed)
all subtimers 31 ms (100.00% of zscreen, 0.00% of elapsed)

[Birkow]

pre-commit.ci autofix

[lethosor]

I may be missing some earlier discussion on this, but why the merge into autobutcher? The two tools don't really seem all that similar to me - sure, they both deal with animals, but autobutcher is intended to control which units get butchered automatically, and nestboxes is intended to protect the eggs of egglayers. I think autobutcher nestboxes makes less sense than nestboxes from an end-user perspective.

[myk002]

why the merge into autobutcher?

It's because of the configuration overlap. we're adding a nestboxes feature that essentially tracks targets per race, which is very aligned with autobutcher's concept and interface. if you configure one, you are very likely to want to configure the other the same way, so it seemed like it made sense to combine them

[lethosor]

Would some other name make more sense?

As a historical example, I feel like this combination makes about as much sense as combining autobutcher and zone, which were previously combined until we split them up a couple years back.

[myk002]

For the record, I didn't understand the autobutcher/zone combination at all. They didn't share code or configuration. Moreover, one was an automation tool and the other was a productivity tool. In this case, both are automation tools with aligned purposes.

I think this was the discussion on Discord: https://discord.com/channels/793331351645323264/807444515194798090/1284174784472547338

[Birkow]

Nestboxes is already recognized as egg handling thing, leaving it might make changes easier on end users already familiar with name. Although I'm not really attached to it. Started working on it with eggwatch name, any name would be fine with me. There is still possibility of renaming autobutcher like myk mentioned to something like animal-husbandry to cover expanded functionality.

Adding nestboxes as part of autobutcher makes sense to me as it can be used to make sure butcher will not be overworked by handling excess hatchlings. With default settings only protect enough fertile eggs to reach autobutcher targets for the population (plus few extra to deal with random gender).

[myk002]

I'm finding the proposed API a bit confusing. Let's hash out the behavior and interface in the docs before I look at the actual code

[myk002]

Maybe it would be better if you could explain/mock up how the new configuration will fit into gui/autobutcher:

I was hoping there would be one extra column and nothing else, but the complexity of the API makes me doubt that that is possible. However, I'm also not convinced that all the new options are necessary, and I think some of them could be trimmed.

[Birkow]

Edit: changed screenshot to include shortcuts for setting new per race fields:

I think from "global" variables for Nestboxes module it's worth to only expose enabled status, others can be hidden/removed. Added only that one to mock up.
For per race config added everything to mock up. I have no idea if any of those should be hidden/removed.

Obviously new fields would be filled only for egglayers and first 2 rows with default/all, just added it here as static text to everything.

Currently there are following variables saved in persisted site config:

• enabled
  enabled status for nestboxes module, added to mock up
• ignore_autobutcher
  would allow nestboxes work on it's own in without autobutcher enabled, good candidate to remove/not show in gui
• split_stacks
  if egg stacks should be split when stack goes partially over target amount of protected eggs, good candidate to
  remove and always split stacks
• default
  default settings for new races
• config_per_race
  table with per race settings
  Not exposed to user:
• verbose
  probably good idea to remove from persisted config and move to separate global variable, when enabled prints a details with debug info
• migration_from_cpp_to_lua_done
  flag to mark migration of enabled status from cpp nenstboxes

Settings per race:

• watched
  If eggs for race should be protected
• target
  base amount of protected eggs
• ama
  Add Missing Animals to base target for protected eggs, difference between current amount of animals and target set for autobutcher (both for kids and adults)
• stop
  stop protecting eggs once autobutcher target for live animals is reached