Dataframes Haystack

---

• 📃 Description
• 🛠️ Installation
• 💻 Usage
  • DataFrameFileToDocument
  • pandas Converters
    • FileToPandasDataFrame
    • PandasDataFrameConverter
  • polars Converters
    • FileToPolarsDataFrame
    • PolarsDataFrameConverter
• 🤝 Contributing

---

📃 Description

dataframes-haystack is an extension for Haystack 2 that enables integration with dataframe libraries.

The dataframe libraries currently supported are:

• pandas
• Polars

The library offers various custom Converters components to transform dataframes into Haystack Document objects:

• DataFrameFileToDocument is a main generic converter that reads files using a dataframe backend and converts them into Document objects.
• FileToPandasDataFrame and FileToPolarsDataFrame read files and convert them into dataframes.
• PandasDataFrameConverter or PolarsDataFrameConverter convert data stored in dataframes into Haystack Documentobjects.

dataframes-haystack supports reading files in various formats:

• csv, json, parquet, excel, html, xml, orc, pickle, fixed-width format for pandas. See the pandas documentation for more details.
• csv, json, parquet, excel, avro, delta, ipc for polars. See the polars documentation for more details.

🛠️ Installation

# for pandas
pip install "dataframes-haystack[pandas]"# for polars
pip install "dataframes-haystack[polars]"

💻 Usage

Tip

See the Example Notebooks for complete examples.

DataFrameFileToDocument

Complete example

You can leverage both pandas and polars backends (thanks to narwhals) to read your data!

fromdataframes_haystack.components.convertersimportDataFrameFileToDocumentconverter=DataFrameFileToDocument(content_column="text_str")
documents=converter.run(files=["file1.csv", "file2.csv"])

>>>documents
{'documents': [
Document(id=0, content: 'Hello world', meta: {}),
Document(id=1, content: 'Hello everyone', meta: {})
]}

pandas Converters

Complete example

FileToPandasDataFrame

fromdataframes_haystack.components.converters.pandasimportFileToPandasDataFrameconverter=FileToPandasDataFrame(file_format="csv")
output_dataframe=converter.run(
file_paths=["data/doc1.csv", "data/doc2.csv"]
)

Result:

>>>output_dataframe
{'dataframe': <pandas.DataFrame>}

PandasDataFrameConverter

importpandasaspdfromdataframes_haystack.components.converters.pandasimportPandasDataFrameConverterdf=pd.DataFrame({
"text": ["Hello world", "Hello everyone"],
"filename": ["doc1.txt", "doc2.txt"],
})
converter=PandasDataFrameConverter(content_column="text", meta_columns=["filename"])
documents=converter.run(df)

Result:

>>>documents
{'documents': [
Document(id=0, content: 'Hello world', meta: {'filename': 'doc1.txt'}),
Document(id=1, content: 'Hello everyone', meta: {'filename': 'doc2.txt'})
]}

polars Converters

Complete example

FileToPolarsDataFrame

fromdataframes_haystack.components.converters.polarsimportFileToPolarsDataFrameconverter=FileToPolarsDataFrame(file_format="csv")
output_dataframe=converter.run(
file_paths=["data/doc1.csv", "data/doc2.csv"]
)

Result:

>>>output_dataframe
{'dataframe': <polars.DataFrame>}

PolarsDataFrameConverter

importpolarsasplfromdataframes_haystack.components.converters.polarsimportPolarsDataFrameConverterdf=pl.DataFrame({
"text": ["Hello world", "Hello everyone"],
"filename": ["doc1.txt", "doc2.txt"],
})
converter=PolarsDataFrameConverter(content_column="text", meta_columns=["filename"])
documents=converter.run(df)

Result:

>>>documents
{'documents': [
Document(id=0, content: 'Hello world', meta: {'filename': 'doc1.txt'}),
Document(id=1, content: 'Hello everyone', meta: {'filename': 'doc2.txt'})
]}

🤝 Contributing

Do you have an idea for a new feature? Did you find a bug that needs fixing?

Feel free to open an issue or submit a PR!

Setup development environment

Requirements: hatch, pre-commit

1. Clone the repository
2. Run hatch shell to create and activate a virtual environment
3. Run pre-commit install to install the pre-commit hooks. This will force the linting and formatting checks.

Run tests

• Linting and formatting checks: hatch run lint:fmt
• Unit tests: hatch run test-cov-all

✍️ License

dataframes-haystack is distributed under the terms of the MIT license.