How to reproduce the evaluation table in the paper?

[KaminariOS]

Is there some push-button script? Or do I need to collect the numbers across all tasks and take the average?

[HacksonClark]

Some of the agent implementations in clients show how you can loop through each problem iteratively which is more push-button then just running each problem individually. The platform will output your scores on each task.

I would recommend doing multiple runs and taking averages since there will be some variance in how the LLMs respond.

[KaminariOS]

I think this should be built in the agent base class. Since every evaluation costs money, we should persist the numbers in a CSV or SQLite database.

…

On Sun, Sep 21, 2025, 6:16 PM Jackson Clark ***@***.***> wrote: *HacksonClark* left a comment (microsoft/AIOpsLab#79) <#79 (comment)> Some of the agent implementations in clients show how you can loop through each problem iteratively which is more push-button then just running each problem individually. The platform will output your scores on each task. I would recommend doing multiple runs and taking averages since there will be some variance in how the LLMs respond. — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR3ANA6SGFHMCBAH3MOB5R33T4PSFAVCNFSM6AAAAACHDNLXH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJWGI4TSMJSGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[KaminariOS]

I see it in aiopslab/data/results