Skip to content
This repository was archived by the owner on Jun 11, 2026. It is now read-only.
This repository was archived by the owner on Jun 11, 2026. It is now read-only.

Normalization of answers. #3

Description

@victorywys

Hi, when we're doing the case study using the evaluation script, we got some confusion about the results.

  1. We find that you have the normalize_answer function. Several rules are set to normalize the candidates and references. However, we found these methods are not applied to the answers: in the return statement, these separate rules are not even called. This caused at least the punctuation problem: no punctuations are included in your raw text field of the dataset, but some punctuations exist in your keyphrases which makes it impossible for the models to predict the "exactly matched keyphrases".
    Refer to:

    OpenKP/evaluate.py

    Lines 7 to 17 in 6e1d770

    defnormalize_answer(s):
    defremove_articles(text):
    returnre.sub(r'\b(a|an|the)\b', ' ', text)
    defwhite_space_fix(text):
    return' '.join(text.split())
    defremove_punc(text):
    exclude=set(string.punctuation)
    return''.join(chforchintextifchnotinexclude)
    deflower(text):
    returntext.lower()
    return' '.join([lower(x) forxins]).rstrip()

  2. We notice that you filter keyphrases beginning with an empty string "". However, there're some(quite a few) keypharses in the dataset which are not exactly empty like ["", "Middle", "East", "&", "Jewish", "World"] in line 6109 of dev.jsonl. So is there a reason to discard these kind of keyphrases?
    Refer to:

    OpenKP/evaluate.py

    Lines 19 to 25 in 6e1d770

    defremove_empty(a_list):
    new_list= []
    foriina_list:
    iflen(i) >0:
    iflen(i[0]) >0:
    new_list.append(normalize_answer(i))
    returnnew_list

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions