👋 Introduction

DTECT (Dynamic Topic Explorer & Context Tracker) is an end-to-end, open-source system designed to streamline the entire process of dynamic topic modeling.

---

🚀 Live Demo & Video

• Interactive Demo: Try DTECT live on Hugging Face Spaces!

  • https://huggingface.co/spaces/AdhyaSuman/DTECT

• Demo Video: Watch a walkthrough of DTECT's features.

---

💻 Code Examples

DTECT Preprocessing Pipeline

Here is an example of how to use the DTECT preprocessing pipeline for a custom dataset:

frombackend.datasets.preprocessimportPreprocessorfromnltk.corpusimportstopwordsimportosdataset_dir='../data/Sample_data/'stop_words=stopwords.words('english')
preprocessor=Preprocessor(
docs_jsonl_path=dataset_dir+'docs.jsonl',
output_folder=os.path.join(dataset_dir, 'processed'),
use_partition=False,
min_count_bigram=5,
threshold_bigram=20,
remove_punctuation=True,
lemmatize=True,
stopword_list=stop_words,
min_chars=3,
min_words_docs=3,
)
preprocessor.preprocess()

Training and Evaluation Pipeline (with CFDTM)

This snippet shows an example of the training and evaluation pipeline using the CFDTM model:

frombackend.datasetsimportdynamic_datasetfrombackend.models.CFDTM.CFDTMimportCFDTMfrombackend.models.dynamic_trainerimportDynamicTrainerfrombackend.evaluation.evalimportTopicQualityAssessor# Load datasetdata=dynamic_dataset.DynamicDataset('../data/Sample_data/processed')
# Initialize modelmodel=CFDTM(
vocab_size=data.vocab_size,
num_times=data.num_times,
num_topics=20,
pretrained_WE=data.pretrained_WE,
train_time_wordfreq=data.train_time_wordfreq
).to("cuda")
# Train modeltrainer=DynamicTrainer(model, data)
top_words, _=trainer.train()
top_words_list= [[topic.split() fortopicintimestamp] fortimestampintop_words]
train_corpus= [doc.split() fordocindata.train_texts]
# Evaluationassessor=TopicQualityAssessor(
topics=top_words_list,
train_texts=train_corpus,
topn=10,
coherence_type='c_npmi'
)
summary=assessor.get_dtq_summary()

---

📁 Repository Structure

├── app
│ └── ui.py
├── assets
├── backend
│ ├── datasets
│ ├── evaluation
│ ├── inference
│ ├── llm
│ ├── llm_utils
│ └── models
├── data
│ └── Sample_data
├── environment.yml
├── LICENSE
├── main.py
└── requirements.txt

---

📚 Supporting Resources

We list below the datasets, codebases, and evaluation resources referenced or integrated into DTECT:

Datasets

• ACL Anthology
• UN General Debates
• TCPD-IPD Finance

Dynamic Topic Modeling Codebases

• DTM
• DETM
• CFDTM

Preprocessing Toolkit

• OCTIS

Evaluation Metrics

• Evaluating Dynamic Topic Models:https://github.com/CharuJames/Evaluating-Dynamic-Topic-Models

---

🙏 Acknowledgements

We would like to acknowledge the following open-source projects that were instrumental in the development of DTECT:

🔍 TopMost Toolkithttps://github.com/bobxwu/TopMost 📌 Reference: Xiaobao Wu, Fengjun Pan, and Anh Tuan Luu. 2024. Towards the TopMost: A Topic Modeling System Toolkit. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 31–41, Bangkok, Thailand. Association for Computational Linguistics.

📦 OCTIShttps://github.com/MIND-Lab/OCTIS 📌 Reference: Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021. OCTIS: Comparing and Optimizing Topic Models is Simple!. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 263–270, Online. Association for Computational Linguistics.

📌 BibTeX

@misc{adhya2025dtect,
title={{DTECT}: Dynamic Topic Explorer & Context Tracker}, author={Suman Adhya and Debarshi Kumar Sanyal},
year={2025},
eprint={2507.07910},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2507.07910}, }