DIVE Framework

A blockchain digging framework for constructing vulnerability-tagged smart contract datasets.

---

🔍 Key Features

The DIVE framework provides a powerful pipeline for blockchain dataset creation through six core components:

---

1. 🧾 Feature Collection

Fetch smart contract and account data from public blockchains.

• ✅ Currently supports Ethereum.
• 🔗 Uses Etherscan.io as a data source.
• 📊 Collects:
  • Contract metadata
  • Account-level information
  • Opcodes

2. 🧠 Solidity Code Extraction

Retrieve and store verified contract source code as .sol files using Solidity.

3. 🧪 Feature Extraction

Extract structured features from various smart contract attributes, including: ABI, Timestamp, Library, TransactionIndex, Code Metrics, Input / Bytecode, and Opcode

4. 🏷️ Labeled Data Construction

Merge extracted features with ground-truth vulnerability labels to build a structured dataset.

5. 🧹 Data Preprocessing

Clean, normalize, and transform the data to prepare it for downstream analysis or machine learning tasks.

6. 📊 Statistical Analysis & Visualization

Generate statistical summaries and visualizations to better understand the dataset's structure and characteristics.

---

📦 Requirements

• Python = 3.12.2

• solidity-code-metrics = 0.0.26

  Install using one of the following:

  # Using Yarn
  yarn global add solidity-code-metrics@0.0.26
  # Or using npm
  npm install -g solidity-code-metrics@0.0.26

• 🔑 Etherscan API Key

  Create an account at Etherscan.io and follow their API key guide.
  ⚠️Do not share your API key publicly.

• Python dependencies are listed in requirements.txt.
  You can install them using:

  pip install -r requirements.txt

---

📁 Repository Structure

DIVE/
├── Datasets/ # Generated datasets
│ ├── InitialCombinedData/ # Merged raw features before preprocessing
│ └── PreprocessedData/ # Cleaned, transformed datasets for ML
│
├──Docs/
│ ├── initial-setup.md # Step-by-step guide for project installation and configuration
│ └── usage.md # Detailed documentation for using framework functions and scripts
│
├── Features/ # Extracted features
│ ├── API-based/ # Features collected from Etherscan APIs
│ │ ├── AccountInfo/ # Account-level features
│ │ ├── BlockInfo/ # Block transaction counts
│ │ ├── ContractsInfo/ # Contract metadata from Etherscan
│ │ └── Opcodes/ # Opcode data from Etherscan
│ ├── FE-based/ # Feature engineering outputs
│ │ ├── ABI-based/ # Features extracted from ABI
│ │ ├── CodeMetrics/ # Code metric data
│ │ │ ├── CodeMetrics/ # Parsed metric values
│ │ │ └── Reports/ # Raw/edited Markdown metric reports
│ │ │ ├── EditedReports/
│ │ │ ├── OriginalReports/
│ │ │ └── Raw_CodeMetrics/
│ │ ├── Input-based/ # Features derived from the Input attribute
│ │ ├── Library-based/ # Features derived from the Library attribute
│ │ ├── Opcode-based/ # Features derived from opcode-level analysis
│ │ ├── Timestamp-based/ # Features derived from the Timestamp attribute
│ │ └── TransactionIndex/ # Features derived from the TransactionIndex attribute
│
├── Labels/ # Ground-truth labels for contracts
│
├── RawData/ # Data collected or downloaded
│ ├── Samples/ # Extracted Solidity source code samples
│ ├── SamplesSummary/ # │ └── SC_Addresses/ # CSVs of smart contract addresses
│
├── Scripts/ # Main processing and utility scripts
│
│ ├── FeatureExtraction/ # Scripts for extracting low-level features
│ │ ├── EVM_Opcodes/ # Contains opcode-related resources
│ │ │ ├── EVM_Opcodes_*.xlsx # Excel file(s) listing EVM opcodes and metadata
│ │ ├── ABI_FeatureExtraction.py # Extracts features from ABI (Application Binary Interface)
│ │ ├── Bytecode_FeatureExtraction.py# Extracts bytecode-level features
│ │ ├── get_CodeMetrics.py # Calls external tools (i.e., solidity-code-metrics) to compute code metrics
│ │ ├── get_OpcodesList.py # Generates the EVM opcode reference list (EVM_Opcodes_*.xlsx)
│ │ ├── Library_FeatureExtraction.py # Extracts library-based features
│ │ ├── Opcode_FeatureExtraction.py # Extracts features from opcodes (e.g., opcode metrics)
│ │ ├── Timestamp_FeatureExtraction.py # Extracts timestamp-based features
│ │ └── transactionIndex_FeatureExtraction.py # Extracts transactionIndex-based features
│
│ ├── FeatureSelection/ # Script for selecting relevant features for analysis/modeling
│ │ └── get_FilteredFeatures.py # Applies feature selection (uses classification defined in Feature list.xlsx)
│
│ ├── apply_DataPreprocessing.py # Cleans, normalizes, and transforms data
│ ├── apply_FeatureExtraction.py # Coordinates the execution of multiple feature extraction steps
│ ├── construct_FinalData.py # Merges feature sets and labels to construct the final dataset
│ ├── extract_SourceCodes.py # Extracts Solidity source code (included in Etherscan API responses) │ ├── get_Addresses.py # Loads and filters smart contract addresses from input CSV files
│ ├── get_BlockFeatures.py # Retrieves transaction counts for each block
│ ├── get_ContractFeatures.py # Orchestrates retrieval of contract info from Etherscan
│ └── get_DataStatistics.py # Generates summary statistics and visualizations for the dataset
│
├── Statistics/ # Analysis outputs and statistical summaries
│
├── config.json # Configuration file for paths and API key
├── DIVE_pipeline.yaml # YAML config defining the full data creation pipeline execution
├── DIVE.ipynb # Interactive notebook for demonstrating the framework
├── Feature list.xlsx # Documentation of features and their descriptions
├── LICENSE.md # License: CC BY-NC 4.0
├── README.md # Project overview and usage instructions
├── requirements.txt # Python package dependencies
└── run_DIVE_Pipeline.py # Entrypoint to run the entire pipeline as a script

---

🧭 Getting Started

🔧 Initial Setup

• See full instructions in Docs/initial-setup.md

🛠️ Using Framework Functions

• Each function is explained in detail in Docs/usage.md

---

📦 License

This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).

🚫 Patent Rights Reserved

• This project may be covered by pending or granted patents. The authors reserve all rights under applicable patent laws.
• The use of this software does not grant any rights to use patented inventions.
• For commercial licensing or patent-related inquiries, please contact the authors directly.

🛡️ Disclaimer

• DIVE is provided as a research tool and is under active development. While we strive for reliability, we do not provide warranties or guarantees. Please use it responsibly and at your own discretion.