MathPix OCR Batch

Program to batch convert images to LaTeX.

:License: GPLv3

Installation

Install local requirements:

make install

Usage

If you do not have a cluster running locally you can use docker-compose to avoid several headaches. By default the KAFKA_BOOTSTRAP_SERVER is kafka://localhost:29092.

make kafka-cluster

Then, start the Faust application:

make start-app

Settings

Settings are created based on local-settings package.

The only settings required if the KAFKA_BOOTSTRAP_SERVER environment variable.

SIMPLE_SETTINGS= {
'OVERRIDE_BY_ENV': True,
'CONFIGURE_LOGGING': True,
'REQUIRED_SETTINGS': ('KAFKA_BOOTSTRAP_SERVER', 'STORE_URI'),
}
# The following variables can be ovirriden from ENVKAFKA_BOOTSTRAP_SERVER="kafka://localhost:29092"TOPIC_ALLOW_DECLARE=TrueTOPIC_DISABLE_LEADER=FalseSSL_ENABLED=FalseSSL_CONTEXT=NoneifSSL_ENABLED:
# file in pem format containing the client certificate, as well as any ca certificates# needed to establish the certificate’s authenticityKAFKA_SSL_CERT=None# filename containing the client private keyKAFKA_SSL_KEY=None# filename of ca file to use in certificate verificationKAFKA_SSL_CABUNDLE=None# password for decrypting the client private keySSL_KEY_PASSWORD=NoneSSL_CONTEXT=ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile=KAFKA_SSL_CABUNDLE)
SSL_CONTEXT.load_cert_chain(KAFKA_SSL_CERT, keyfile=KAFKA_SSL_KEY, password=SSL_KEY_PASSWORD)

The settings also include a basic logging configuration:

LOGGING= {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'default': {
'format': '%(asctime)s %(levelname)s %(name)s %(message)s',
},
},
'handlers': {
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'default',
},
},
'loggers': {
'page_views': {
'handlers': ['console'],
'level': 'INFO',
},
},
}

Basic Commands

• Start application: make kafka-cluster. This command start both the Page Views application
• Stop and remove containers: make stop-kafka-cluster
• Install requirements: make install
• Start Faust application: make start-app
• List topics: make list-topics
• Create topic: make create-topic={topic-name}
• List agents: make list-agents
• Send events to page_view topic/agent: make send-page-view-event payload='{"id": "foo", "user": "bar"}'

Docker

The Dockerfile is based on python:3.7-slim.

Docker Compose

docker-compose.yaml includes zookeepeer, kafka and schema-registry (if included in the cookiecutter setup) based on confluent-inc. For more information you can go to confluentinc and see the docker compose example here

Useful ENVIRONMENT variables that you may change:

Variable	description	example
WORKER_PORT	Worker port	6066
KAFKA_BOOTSTRAP_SERVER	Kafka servers	kafka://localhost:29092
KAFKA_BOOTSTRAP_SERVER_NAME	Kafka server name	kafka
KAFKA_BOOTSTRAP_SERVER_PORT	Kafka server port	29092
SCHEMA_REGISTRY_SERVER	Schema registry server name	schema-registry
SCHEMA_REGISTRY_SERVER_PORT	Schema registry server port	8081
SCHEMA_REGISTRY_URL	Schema Registry Server url	http://schema-registry:8081

Run tests

make install-test && ./scripts/test

Lint code

./scripts/lint

Type checks

Running type checks with mypy:

mypy mathpix_ocr_batch