RAG AutoML tool for automatically finding an optimal RAG pipeline for your data.
There are many RAG pipelines and modules out there, but you don’t know what pipeline is great for “your own data” and "your own use-case." Making and evaluating all RAG modules is very time-consuming and hard to do. But without it, you will never know which RAG pipeline is the best for your own use-case.
AutoRAG is a tool for finding the optimal RAG pipeline for “your data.” You can evaluate various RAG modules automatically with your own evaluation data and find the best RAG pipeline for your own use-case.
AutoRAG supports a simple way to evaluate many RAG module combinations. Try now and find the best RAG pipeline for your own use-case.
Explore our 📖 Document!!
AutoRAG.Tutorial.1.1.mp4
Muted by default, enable sound for voice-over
You can see on YouTube
- Step 1: Basic of AutoRAG | Optimizing your RAG pipeline
- Step 2: Data Creation | Create your own Data for RAG Optimization
- Step 3: Use Custom LLM & Embedding Model | Use Custom Model
We recommend using Python version 3.10 or higher for AutoRAG.
pip install AutoRAGIf you want to use the local models, you need to install gpu version.
pip install "AutoRAG[gpu]"Or for parsing, you can use the parsing version.
pip install "AutoRAG[gpu,parse]"
RAG Optimization requires two types of data: QA dataset and Corpus dataset.
- QA dataset file (qa.parquet)
- Corpus dataset file (corpus.parquet)
QA dataset is important for accurate and reliable evaluation and optimization.
Corpus dataset is critical to the performance of RAGs. This is because RAG uses the corpus to retrieve documents and generate answers using it.
modules:
- module_type: langchain_parseparse_method: pdfminerYou can also use multiple Parse modules at once. However, in this case, you'll need to return a new process for each parsed result.
You can parse your raw documents with just a few lines of code.
fromautorag.parserimportParserparser=Parser(data_path_glob="your/data/path/*")
parser.start_parsing("your/path/to/parse_config.yaml")modules:
- module_type: llama_index_chunkchunk_method: Tokenchunk_size: 1024chunk_overlap: 24add_file_name: enYou can also use multiple Chunk modules at once. In this case, you need to use one corpus to create QA and then map the rest of the corpus to QA Data. If the chunk method is different, the retrieval_gt will be different, so we need to remap it to the QA dataset.
You can chunk your parsed results with just a few lines of code.
fromautorag.chunkerimportChunkerchunker=Chunker.from_parquet(parsed_data_path="your/parsed/data/path")
chunker.start_chunking("your/path/to/chunk_config.yaml")You can create QA dataset with just a few lines of code.
importpandasaspdfromllama_index.llms.openaiimportOpenAIfromautorag.data.qa.filter.dontknowimportdontknow_filter_rule_basedfromautorag.data.qa.generation_gt.llama_index_gen_gtimport (
make_basic_gen_gt,
make_concise_gen_gt,
)
fromautorag.data.qa.schemaimportRaw, Corpusfromautorag.data.qa.query.llama_gen_queryimportfactoid_query_genfromautorag.data.qa.sampleimportrandom_single_hopllm=OpenAI()
raw_df=pd.read_parquet("your/path/to/parsed.parquet")
raw_instance=Raw(raw_df)
corpus_df=pd.read_parquet("your/path/to/corpus.parquet")
corpus_instance=Corpus(corpus_df, raw_instance)
initial_qa= (
corpus_instance.sample(random_single_hop, n=3)
.map(
lambdadf: df.reset_index(drop=True),
)
.make_retrieval_gt_contents()
.batch_apply(
factoid_query_gen, # query generationllm=llm,
)
.batch_apply(
make_basic_gen_gt, # answer generation (basic)llm=llm,
)
.batch_apply(
make_concise_gen_gt, # answer generation (concise)llm=llm,
)
.filter(
dontknow_filter_rule_based, # filter don't knowlang="en",
)
)
initial_qa.to_parquet('./qa.parquet', './corpus.parquet')
Here is the AutoRAG RAG Structure that only show Nodes.
Here is the image showing all the nodes and modules.
The metrics used by each node in AutoRAG are shown below.
Here is the detailed information about the metrics that AutoRAG supports.
First, you need to set the config YAML file for your RAG optimization.
We highly recommend using pre-made config YAML files for starter.
Here is an example of the config YAML file to use three retrieval nodes, prompt_maker, and generator nodes.
node_lines:
- node_line_name: retrieve_node_linenodes:
- node_type: lexical_retrievalstrategy:
metrics: [ retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr ]top_k: 3modules:
- module_type: bm25
- node_type: semantic_retrievalstrategy:
metrics: [ retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr ]top_k: 3modules:
- module_type: vectordbvectordb: default
- node_type: hybrid_retrievalstrategy:
metrics: [ retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr ]top_k: 3modules:
- module_type: hybrid_rrfweight_range: (4,80)
- node_line_name: post_retrieve_node_linenodes:
- node_type: prompt_maker # Set Prompt Maker Nodestrategy:
metrics: # Set Generation Metrics
- metric_name: meteor
- metric_name: rouge
- metric_name: sem_scoreembedding_model: openaimodules:
- module_type: fstringprompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
- node_type: generator # Set Generator Nodestrategy:
metrics: # Set Generation Metrics
- metric_name: meteor
- metric_name: rouge
- metric_name: sem_scoreembedding_model: openaimodules:
- module_type: openai_llmllm: gpt-4o-minibatch: 16You can evaluate your RAG pipeline with just a few lines of code.
fromautorag.evaluatorimportEvaluatorevaluator=Evaluator(qa_data_path='your/path/to/qa.parquet', corpus_data_path='your/path/to/corpus.parquet')
evaluator.start_trial('your/path/to/config.yaml')or you can use the command line interface
autorag evaluate --config your/path/to/default_config.yaml --qa_data_path your/path/to/qa.parquet --corpus_data_path your/path/to/corpus.parquetOnce it is done, you can see several files and folders created in your current directory.
At the trial folder named to numbers (like 0),
you can check summary.csv file that summarizes the evaluation results and the best RAG pipeline for your data.
For more details, you can check out how the folder structure looks like at here.
You can run a dashboard to easily see the result.
autorag dashboard --trial_dir /your/path/to/trial_dir
You can use an optimal RAG pipeline right away from the trial folder. The trial folder is the directory used in the running dashboard. (like 0, 1, 2, ...)
fromautorag.deployimportRunnerrunner=Runner.from_trial_folder('/your/path/to/trial_dir')
runner.run('your question')You can run this pipeline as an API server.
Check out the API endpoint at here.
importnest_asynciofromautorag.deployimportApiRunnernest_asyncio.apply()
runner=ApiRunner.from_trial_folder('/your/path/to/trial_dir')
runner.run_api_server()autorag run_api --trial_dir your/path/to/trial_dir --host 0.0.0.0 --port 8000The cli command uses extracted config YAML file. If you want to know it more, check out here.
you can run this pipeline as a web interface.
Check out the web interface at here.
autorag run_web --trial_path your/path/to/trial_path
Thanks go to these wonderful people:
We are developing AutoRAG as open-source.
So this project welcomes contributions and suggestions. Feel free to contribute to this project.
Plus, check out our detailed documentation at here.
@misc{kim2024autoragautomatedframeworkoptimization,
title={AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline},
author={Dongkyu Kim and Byoungwook Kim and Donggeon Han and Matouš Eibich},
year={2024},
eprint={2410.20878},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.20878},
}