Is your feature request related to a problem? Please describe.
We have lots of fetch methods for datasets in pyrit.datasets, see https://github.com/Azure/PyRIT/blob/main/pyrit/datasets/fetch_example_datasets.py
However, with #396 we're changing what datasets look like. So far, a lot of metadata lived on the dataset level and will be moved to the prompt level. This makes sense because every prompt may have different harm categories.
While not in scope for #396 we want the accurate harm_categories to be reflected for each prompt.
Describe the solution you'd like
Go through all the datasets and see if there are indications per prompt on what harm_categories the prompts belong to. For example, the original dataset may have a column for that. Note that this can be multiple values (harm categories is plural, i.e., a list).
This can be done one dataset at a time. If you volunteer to take on one dataset please comment below to avoid having redundant work.
CC Raja Sekhar Rao Dheekonda (@rdheekonda)
Is your feature request related to a problem? Please describe.
We have lots of fetch methods for datasets in
pyrit.datasets, see https://github.com/Azure/PyRIT/blob/main/pyrit/datasets/fetch_example_datasets.pyHowever, with #396 we're changing what datasets look like. So far, a lot of metadata lived on the dataset level and will be moved to the prompt level. This makes sense because every prompt may have different harm categories.
While not in scope for #396 we want the accurate
harm_categoriesto be reflected for each prompt.Describe the solution you'd like
Go through all the datasets and see if there are indications per prompt on what
harm_categoriesthe prompts belong to. For example, the original dataset may have a column for that. Note that this can be multiple values (harm categories is plural, i.e., a list).This can be done one dataset at a time. If you volunteer to take on one dataset please comment below to avoid having redundant work.
CC Raja Sekhar Rao Dheekonda (@rdheekonda)