Thinking Dataset: Leveraging Real-World Data for Strategic Business Insights

Table of Contents

• Overview
• Features
• Installation
• Usage
• Quick Start
• Project Structure
• Contributing
• Resources
• License
• Citations
• Acknowledgements
• Contact

Overview

Thinking Dataset leverages real-world data and efficient pipelines for business insights. We integrate robust data workflows (PDF to SQL ingestion), tactical TDD (details), and a streamlined SQLite design (details). Multi-stage AI generates STaR case studies for actionable strategies.

Features

• 🔄 End-to-End Pipeline: Download, process, transform, and store large datasets.
• 💾 SQLite Backend: Lightweight, fast, and easy to manage, with optional Parquet.
• ✅ Comprehensive Testing: Thorough TDD coverage and validation checks.
• 🖥️ Flexible CLI: Modular Click commands for quick execution of tasks.
• 🔀 Data Transformation: Granular pipes for cleaning, merging, and deriving data.
• 📚 STaR Case Studies: Generate synthetic scenarios alongside real data for deeper insights.
• ⚡ Parallel Execution: Efficiently process big data with optional concurrency.
• 🔒 Secure Config: Manage environment variables discretely with .env files.

Quick Start

Prerequisites

• Python 3.12 or later
• Git
• A cloud-based account (e.g., Huggingface) or a GPU (RTX 3090 or greater) for processing, or both

Setup

1. Clone the repository:

   git clone https://github.com/MultiTonic/thinking-dataset.git
   cd thinking-dataset

2. Install uv package manager:

   First add the package into the global environment:

   pip install uv

   Then add uv tools directory to PATH*:

   uv tool update-shell

3. Set up the project:

   uv run setup

   *You may need to restart your terminal session for the changes to update.

This will create a virtual environment, install the project dependencies, and activate the virtual environment.

4. Set up environment variables:

   Copy the .env.sample file to .env and change the values as needed:

   cp .env.sample .env

   Update the .env file with your credentials:

   # Required settingsHF_ORG="my_huggingface_organization"HF_USER="my_huggingface_username"HF_READ_TOKEN="my_huggingface_read_access_token"HF_WRITE_TOKEN="my_huggingface_write_access_token"# Required configurationCONFIG_PATH="config/config.yaml"# One or more providersOLLAMA_SERVER_URL="http://localhost:11434"OPENAI_API_TOKEN="your_openai_api_token"RUNPOD_API_TOKEN="your_runpod_api_token"

Usage

For complete usage instructions and examples, see the Usage Guide.

Running the Download Command

To download all parquet files from the Cablegate dataset using Hugging Face CLI:

thinking-dataset download

Running All CLI Commands

To execute all CLI commands for the project:

python assets/scripts/run_cli_commands.py

Project Structure

The following directory structure provides an overview of how the project is organized:

thinking-dataset/
├── config/ # Configuration files
├── assets/ # Assets directory for external resources
│ ├── prompts/ # Prompt templates
│ ├── scripts/ # Utility scripts
│ ├── resources/ # External project data
│ ├── templates/ # JSON prompt templates
├── data/ # Data directory
├── docs/ # Project documentation
├── reports/ # Generated reports
├── tests/ # Test files
├── thinking_dataset/ # Core project code
│ ├── commands/ # CLI command implementations
│ ├── connectors/ # Data connectors
│ ├── config/ # Configuration loaders and management
│ ├── datasets/ # Dataset definitions and processing
│ │ ├── operations/ # Data operations and transformations
│ ├── db/ # Database support
│ │ ├── operations/ # Database operations and transactions
│ ├── dto/ # Data Transfer Objects (DTO)
│ ├── io/ # File I/O operations
│ ├── pipeworks/ # Pipelines and pipes for data processing
│ │ ├── pipelines/ # Pipeline management and control
│ │ ├── pipes/ # Pipes used for data frame processing
│ ├── providers/ # AI data providers
│ ├── tonics/ # Data utility functions and helpers
│ ├── utils/ # General-purpose utility helpers
│ ├── main.py # Main execution file
└── setup.py # Project setup
└── .env # Private environment variables file

Contributing

Contributions are welcome! Fork the repository, make your changes, and create a pull request. Ensure your code follows the project's standards and includes tests. See Contributing for guidelines.

Resources

• GitHub Repository
• Python
• Ollama
• Discord: 🌟Tonic's Better Prompts

License

This dataset is licensed under the MIT License.

Citations

Please use the following BibTeX entry to cite this dataset:

@software{thinking-dataset,
author = {Kara Rawson, Joseph Pollack, and et al.},
title = {Thinking-Dataset: Leveraging Real-World Data for Strategic Business Insights and STaR Case Study Generation},
year = {2025},
howpublished = {\url{https://github.com/MultiTonic/thinking-dataset}},
note = {Accessed: 2025-01-25}
}

Acknowledgements

Special thanks to our contributors:

• Kara Rawson - Lead Engineer
• Joseph Pollack - Creator & Business Leader
• MultiTonic Team - Support and Collaboration
• Hugging Face - Robust tools and infrastructure for dataset management

Contact

For questions or support, please contact us at:

• Email: info@multitonics.ai
• Discord: Join our Discord