Sentiment Analysis on Primate Dataset

This project aims to perform sentiment analysis on a dataset containing posts related to primates. Leveraging transformer-based models, sentiment labels are predicted for the textual data. Below is an elaborate description of the project workflow:

Workflow Overview

1. Data Preprocessing:

   • The dataset (primate_dataset.json) is loaded and preprocessed to prepare it for model training.
   • The clean_text function in preprocess_data.py cleans the text data by converting it to lowercase, removing punctuation, tokenizing, removing stopwords, and stemming.

   importpandasaspdimportnltkimportstringfromnltkimportword_tokenizefromnltk.corpusimportstopwordsfromnltk.stem.porterimportPorterStemmernltk.download('punkt')
   nltk.download('stopwords')
   defclean_text(data):
   # Implementation of text cleaning
   ...
   returnresult

2. Data Splitting:

   • The preprocessed data is split into training and testing sets for model evaluation.

   fromsklearn.model_selectionimporttrain_test_splitX_train, X_test, Y_train, Y_test=train_test_split(X, Y, test_size=0.2, random_state=42)

3. Model Training:

   • The sentiment analysis model is trained using the training data.

   fromtransformersimportAutoTokenizer, AutoModelForSequenceClassificationimporttorchfromtorch.utils.dataimportDataLoader, TensorDatasettokenizer=AutoTokenizer.from_pretrained("sbcBI/sentiment_analysis_model")
   model=AutoModelForSequenceClassification.from_pretrained("sbcBI/sentiment_analysis_model")
   # Model training code...

4. Model Evaluation:

   • The trained model is evaluated using the testing data to assess its performance.

   fromsklearn.metricsimportaccuracy_score, precision_score, recall_score, f1_score, roc_auc_scoreimportnumpyasnp# Model evaluation code...

5. Optional Quantization:

   • Optionally, the trained model can be quantized to reduce memory usage and improve inference speed.

   fromtorch.quantizationimportquantize_dynamicquantized_model=quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

Code Structure

• preprocess_data.py: Contains functions for cleaning and preprocessing the dataset.
• train_model.py: Script for training the sentiment analysis model.
• evaluate_model.py: Script for evaluating the trained model on the test dataset.
• quantize_model.py: Optional script for quantizing the trained model.
• utils.py: Utility functions used across different scripts.

Setup and Dependencies

1. Install the required Python packages:

   pip install pandas transformers torch nltk
   

2. Download NLTK data:

   importnltknltk.download('punkt')
   nltk.download('stopwords')

3. Ensure GPU support for faster model training if available.

Usage

1. Preprocess the dataset using preprocess_data.py.
2. Train the sentiment analysis model using train_model.py.
3. Evaluate the trained model using evaluate_model.py.
4. Optionally, quantize the trained model using quantize_model.py.

Model Deployment

The trained and optionally quantized model can be deployed for inference in production environments. Ensure compatibility with the deployment platform and optimize for performance if necessary.