GitHub - 0bs-chat/crawler

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Further Information

This repository serves as a starting point for creating your own custom RunPod Serverless worker. It provides a basic structure and configuration that you can build upon.

Getting Started

Use this template: Create a new repository based on this template or clone it directly.
Customize: Modify the code and configuration files to implement your specific task.
Test: Run your worker locally to ensure it functions correctly.
Deploy: Connect your repository to RunPod or build and push the Docker image manually.

Customizing Your Worker

handler.py: This is the core of your worker.
- The handler(event) function is the entry point executed for each job.
- The event dictionary contains the job input under the "input" key.
- Modify this function to load your models, process the input and return the desired output.
- Consider implementing model loading outside the handler (e.g., globally or in an initialization function) if models are large and reused across jobs.
requirements.txt: Add any Python libraries your worker needs to this file. These will be installed via uv when the Docker image is built.
Dockerfile:
- This file defines the Docker image for your worker.
- It starts from a RunPod base image (runpod/base) which includes CUDA, mulitple versions of python, uv, jupyter notebook and common dependencies.
- It installs dependencies from requirements.txt using uv.
- It copies your src directory into the image.
- You might need to add system dependencies (apt-get install ...), environment variables (ENV), or other setup steps here if required by your specific application.
test_input.json: Modify this file to provide relevant sample input for local testing.

Testing Locally

You can test your handler logic locally using the RunPod Python SDK. For detailed steps on setting up your local environment (creating a virtual environment, installing dependencies) and running the handler, please refer to the RunPod Serverless Get Started Guide.

Prepare Input: Modify test_input.json with relevant sample input for your handler.
Run the Handler:
```
python handler.py
```
This will execute your handler function with the contents of test_input.json as input.

Deploying to RunPod

There are two main ways to deploy your worker:

GitHub Integration (Recommended):
- Connect your GitHub repository to RunPod Serverless. RunPod will automatically build and deploy your worker whenever you push changes to your specified branch.
- For detailed instructions on setting up the GitHub integration, authorizing RunPod, and configuring your deployment, please refer to the RunPod Deploy with GitHub Guide.
Manual Docker Build & Push:
- For detailed instructions on building the Docker image locally and pushing it to a container registry, please see the RunPod Serverless Get Started Guide.
- Once pushed, create a new Template or Endpoint in the RunPod Serverless UI and point it to the image in your container registry.

Name		Name	Last commit message	Last commit date
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
fly.toml		fly.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
fly.toml		fly.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
fly.toml		fly.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
fly.toml		fly.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
app.py		app.py
fly.toml		fly.toml
requirements.txt		requirements.txt

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Getting Started

Customizing Your Worker

Testing Locally

Deploying to RunPod

Further Information

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Getting Started

Customizing Your Worker

Testing Locally

Deploying to RunPod

Further Information

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Getting Started

Customizing Your Worker

Testing Locally

Deploying to RunPod

Further Information

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Getting Started

Customizing Your Worker

Testing Locally

Deploying to RunPod

Further Information

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files