Scribe is a powerful and easy-to-use library for natural language processing (NLP) tasks. It integrates cutting-edge AI models to provide functionalities such as text summarization, question answering, grammar correction, and text similarity comparison.
Built on top of Google Gemini, Transformer models, and BERT-based similarity techniques, Scribe enables developers to leverage advanced AI capabilities with minimal setup.
Table of Contents
- ✅ Text Summarization – Generate concise summaries of long texts.
- ✅ Question Answering – Answer questions based on given context.
- ✅ Question Generation and Answer Evaluation.
- ✅ Grammar Correction – Fix grammar and spelling mistakes.
- ✅ Text Similarity – Compare texts using BERT, SBERT, and TF-IDF.
- ✅ Multi-language Support – Works in English, Arabic, French and more!.
You can install it using a virtualenv or Docker.
make a python virtual enviroment
python -m venv .env
source .env/bin/activateInstall python libraries using:
pip install -r requirementsThen download all the models
sh ./scripts/install.shThen you are good to go.
build the dockerfile
docker build -t scribe .
run it using
docker run --rm -it --net=host scribefastapi dev scribe/api.pyfromscribeimportSummary# Initialize the Summary classsummarizer=Summary()
# Summarize texttext="Artificial intelligence is transforming industries across the globe. It offers opportunities for innovation and growth."summary=summarizer.bart_summarize(text)
print("Summary:", summary)Keyword Extraction:
# Extract keywordskeywords=summarizer.spacy_extract_keywords(text, num_keywords=5)
print("Keywords:", keywords)BERT Similarity:
fromscribeimportSimilarity# Initialize the Similarity classsimilarity_tool=Similarity()
# Compute similarity between two sentences using BERTsentences= [
"Artificial intelligence is fascinating.",
"Machine learning is a subset of artificial intelligence."
]
similarity=similarity_tool.bert_similarity(sentences)
print("BERT Similarity:", similarity)SBERT Similarity:
# Compute similarity for multiple paragraphs using SBERTparagraphs= [
"Artificial intelligence powers many modern applications.",
"Deep learning and AI have revolutionized technology."
]
sbert_similarity=similarity_tool.sbert_similarity(paragraphs)
print("SBERT Similarity:", sbert_similarity)TF-IDF Cosine Similarity:
# Compute similarity for sentences using TF-IDFsentences= [
"Natural language processing is a key area of AI.",
"AI techniques are widely used in NLP."
]
tfidf_similarity=similarity_tool.tfidf_cosine_similarity(sentences)
print("TF-IDF Similarity:", tfidf_similarity)for simple one word answer.
fromscribeimportQnA# Initialize the QnA classqna_tool=QnA()
# Provide a question and contextquestion="What is artificial intelligence?"context="Artificial intelligence (AI) is the simulation of human intelligence in machines that are programmed to think and learn."answer=qna_tool.simple_question(question, context)
print("Answer:", answer)for question generation and evaluation (Requires LLM API Key)
qna=QnA()
text="""Photosynthesis is the process used by plants, algae, and some bacteria to convert light energy into chemical energy.It occurs in the chloroplasts of plant cells, primarily using chlorophyll to capture sunlight.The process involves the intake of carbon dioxide and water, which aretransformed into glucose and oxygen."""# generate questionsquestions=qna.generate_questions(text, num_questions=5)
print("Generated Questions:", questions)
# ['What is photosynthesis?', 'Which organisms utilize photosynthesis?', 'Where does photosynthesis take place in plant cells?', 'What is the primary pigment used in photosynthesis?', 'What are the inputs and outputs of photosynthesis?']# Evaluate answersevaluations=qna.evaluate_answers(
questions=["What is photosynthesis?"], # or use the generated questionsuser_answers=["the plants converts light energy into chemical energy"],
context=text
)
print(evaluations)
# [('What is photosynthesis?', 6, 'The answer is partially correct, but lacks# detail. It correctly identifies the conversion of light energy into chemical# energy. However, it omits key components such as the involvement of chlorophyll,# carbon dioxide, water, glucose, and oxygen, and the location of the process# (chloroplasts).')]fromscribeimportGrammarCorrectorgrammar_tool=GrammarCorrector()
# Grammar correctionoriginal_text="This is an sentence with errors."corrected_text=grammar_tool.correct(original_text)
print("Corrected Text:", corrected_text)
# Diff of changesdiff=grammar_tool.diff(original_text, corrected_text)
print("Diff:")
print(diff)fromscribeimportLLM# Initialize the LLM module with an API keyllm=LLM(api_key="your_api_key")
# Example 1: Summarizationtext_to_summarize= (
"Artificial intelligence (AI) is intelligence demonstrated by machines, ""in contrast to the natural intelligence displayed by humans and animals. ""Leading AI textbooks define the field as the study of intelligent agents: ""any device that perceives its environment and takes actions that maximize ""its chance of achieving its goals."
)
summary=llm.summarize(text_to_summarize, max_tokens=50, temperature=0.5)
print("Summary:", summary)
# Example 2: Answering Questionscontext= (
"The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, ""France. It is named after the engineer Gustave Eiffel, whose company ""designed and built the tower."
)
question="Who designed the Eiffel Tower?"answer=llm.answer_question(question, context, max_tokens=30, temperature=0.3)
print("Answer:", answer)
# Example 3: Grammar Correctionincorrect_text="She go to the market every day."corrected_text=llm.grammar_corrector(incorrect_text, max_tokens=30, temperature=0.3)
print("Corrected Text:", corrected_text)
# Example 4: Summarization with Additional Instructions and Language Supportsummary_french=llm.summarize(text_to_summarize, max_tokens=50, temperature=0.5, language="French")
print("Summary in French:", summary_french)
# Example 5: Answering a Question with Additional Instructionsquestion_with_instruction=llm.answer_question(
"What is the significance of the Eiffel Tower?",
context,
max_tokens=50,
temperature=0.3,
language="English"
)
print("Detailed Answer:", question_with_instruction)| Module | Model Name | Trained On | Accuracy / Performance |
|---|---|---|---|
| LLM | gemini-1.5-flash | Proprietary Google dataset | - |
| Similarity | t5-base | C4 (Clossal Clean Crawled Corpus) dataset | GLUE = ~83.38% |
| Summary | all-mpnet-base-v2 | MNLI dataset | 87.47% using nli-roberta-base for benchmark |
| QnA | roberta-base-squad2 | SQuAD 2.0 dataset | EM = 76.87%, F1=80.9% |
| GrammarCorrector | flan-t5-large-grammar-synthesis | jfleg dataset | GLEU ≈ 76% |
بالحب