Information Gain Filtration

Information Gain Filtration (IGF) is a method for filtering domain-specific data during language model finetuning. IGF shows significant improvements over baseline fine-tuning without data filtration. This demo repo has now been moved to the HuggingFace research projects repo here.