perf: optimize license detection

[saquibsaifee]

💡 What: The _detect_license_from_file method has been optimized to query the Hugging Face hub for the model's file list using model_info() before iterating over the potential license filenames. This ensures it only ever downloads matching files.

🎯 Why: The previous implementation sequentially attempted to download every filename in license_filenames using hf_hub_download, catching the 404 exception when the file wasn't found. For models missing standard licenses, this resulted in numerous wasted, latency-inducing network calls.

📊 Measured Improvement:
I measured the performance improvement using a local Python script running over three different test models.

Original Time: 0.82s
Optimized Time: 0.28s

Improvement: 65.97% time reduction for sequential detections.

[saquibsaifee]

For reviewer reference, here is the script to measure the performance improvement locally:

Measured Execution:

Original Time: 0.82s
Optimized Time: 0.28s
Improvement: 65.97%

import time
import logging
from huggingface_hub import HfApi, hf_hub_download
from huggingface_hub.utils import RepositoryNotFoundError, EntryNotFoundError
import warnings

warnings.filterwarnings('ignore')
logging.getLogger("huggingface_hub").setLevel(logging.ERROR)

LICENSE_MAPPING = {
    "apache license version 2.0": "Apache-2.0",
    "mit license": "MIT",
    "bsd 3-clause": "BSD-3-Clause"
}

def original_detect_license_from_file(model_id: str):
    license_filenames = ["LICENSE", "LICENSE.txt", "LICENSE.md", "LICENSE.rst", "COPYING"]
    for filename in license_filenames:
        try:
            file_path = hf_hub_download(repo_id=model_id, filename=filename)
            with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
                snippet = f.read(4096).lower()
            for header, spdx_id in LICENSE_MAPPING.items():
                if header in snippet:
                    return spdx_id
            return "UNKNOWN"
        except (RepositoryNotFoundError, EntryNotFoundError):
            continue
        except Exception as e:
            continue
    return None

def optimized_detect_license_from_file(model_id: str):
    license_filenames = ["LICENSE", "LICENSE.txt", "LICENSE.md", "LICENSE.rst", "COPYING"]
    api = HfApi()
    try:
        repo_info = api.model_info(repo_id=model_id, files_metadata=False)
        repo_filenames = {sibling.rfilename for sibling in repo_info.siblings}
    except Exception:
        return None
            
    matching_files = [f for f in license_filenames if f in repo_filenames]
    for matching_file in matching_files:
        try:
            file_path = hf_hub_download(repo_id=model_id, filename=matching_file)
            with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
                snippet = f.read(4096).lower()
            for header, spdx_id in LICENSE_MAPPING.items():
                if header in snippet:
                    return spdx_id
            return "UNKNOWN"
        except Exception:
            pass
    return None

def run_benchmarks():
    models = ["microsoft/phi-2", "google/gemma-7b", "mistralai/Mistral-7B-v0.1"]
    
    print("Benchmarking Original...")
    start = time.time()
    for model in models:
        original_detect_license_from_file(model)
    orig_time = time.time() - start
    print(f"Original Time: {orig_time:.2f}s")
    
    print("\nBenchmarking Optimized...")
    start = time.time()
    for model in models:
        optimized_detect_license_from_file(model)
    opt_time = time.time() - start
    print(f"Optimized Time: {opt_time:.2f}s")
    
    if orig_time > 0:
        print(f"\nImprovement: {((orig_time - opt_time) / orig_time) * 100:.2f}%")

if __name__ == "__main__":
    run_benchmarks()

[saquibsaifee]

@eaglei15 this PR is ready to be reviewed.