Introduction

The EllarStorage Module enriches your Ellar application with robust support for managing both cloud and local file storage. Leveraging the capabilities of the Apache libcloud package to simplify file storage operations within your Ellar-powered projects.

Installation

$(venv) pip install ellar-storage

This library drew inspiration from sqlalchemy-file.

Usage

To integrate EllarStorage into your project, follow the standard Ellar project structure and then configure the module as follows:

StorageModule

Similar to other Ellar modules, the StorageModule can be configured directly where it's used or through the application configuration.

StorageModule.setup

You can set up the StorageModule using the setup method. Here's a quick example:

importosfrompathlibimportPathfromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule, get_driver, ProviderBASE_DIRS=Path(__file__).parent@Module(modules=[StorageModule.setup(files={"driver": get_driver(Provider.LOCAL),"options": {"key": os.path.join(BASE_DIRS, "media")}, },images={"driver": get_driver(Provider.LOCAL),"options": {"key": os.path.join(BASE_DIRS, "media")}, },documents={"driver": get_driver(Provider.LOCAL),"options": {"key": os.path.join(BASE_DIRS, "media")}, },default="files" )])classApplicationModule(ModuleBase):
pass

In this example, after application initialization, folders for files, images, and documents will be created in the specified directory. Each folder is configured to be managed by a local storage driver. You can explore other supported storage drivers.

StorageModule.register_setup

Alternatively, you can move the storage configuration to the application config:

## project_name/root_module.pyfromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule@Module(modules=[StorageModule.register_setup()])classApplicationModule(ModuleBase):
pass

Then, in config.py, you can define the storage configurations:

importosfrompathlibimportPathfromellar.core.confimportConfigDefaultTypesMixinfromellar_storageimportget_driver, ProviderBASE_DIRS=Path(__file__).parentclassDevelopmentConfig(ConfigDefaultTypesMixin):
DEBUG=TrueSTORAGE_CONFIG=dict(
storages=dict(
files={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
images={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
},
documents={
"driver": get_driver(Provider.LOCAL),
"options": {"key": os.path.join(BASE_DIRS, "media")},
}
),
default="files"
)

StorageController

StorageModule also registers StorageController which is useful when retrieving saved files. This can be disabled by setting disable_storage_controller to True.

Also, StorageController is not protected and will be accessible to the public. However, it can be protected by simply applying @Guard or @Authorize decorator.

Retrieving Saved Data

By using request.url_for, we can generate a download link for the file we wish to retrieve For example:

fromellar.commonimportInject, postfromellar.coreimportRequest@post('/get-books')defget_book_by_id(self, req: Request, book_id, session: Inject[Session]):
book=session.execute(
select(Book).where(Book.title=="Pointless Meetings")
).scalar_one()
return {
"title": book.title,
"cover": req.url_for("storage:download", path="{storage_name}/{file_name}"),
"thumbnail": req.url_for("storage:download", path=book.thumbnail.path)
}

With req.url_for("storage:download", path="{storage_name}/{file_name}"), we are able to create a download link to retrieve saved files.

StorageService

At the end of the StorageModule setup, StorageService is registered into the Ellar DI system. Here's a quick example of how to use it:

## project_name/server.pyimportosfromellar.appimportAppFactoryfromellar.commonimportdatastructures, constantsfromellar.coreimportLazyModuleImportaslazyLoadfromellar_storageimportStorageServiceapplication=AppFactory.create_from_app_module(
lazyLoad("project_name.root_module:ApplicationModule"),
config_module=os.environ.get(
constants.ELLAR_CONFIG_MODULE, "carapp.config:DevelopmentConfig"
),
)
storage_service: StorageService=application.injector.get(StorageService)
# Example: save a file in the 'files' folderstorage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'files' folder", name="file.txt"), upload_storage='files')
# Example: save a file in the 'images' folderstorage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'images' folder", name="image.txt"), upload_storage='images')
# Example: save a file in the 'documents' folderstorage_service.save(
file=datastructures.ContentFile(b"We can now save files in the 'documents' folder", name="docs.txt"), upload_storage='documents')

StorageService in Route Functions

You can inject StorageService into your controllers or route functions. For instance:

In Controller:

fromellar.commonimportControllerBase, Controllerfromellar_storageimportStorageService@Controller()classFileManagerController(ControllerBase):
def__init__(self, storage_service: StorageService):
self._storage_service=storage_service

In Route Function:

fromellar.commonimportUploadFile, Inject, postfromellar_storageimportStorageService@post('/upload')defupload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
pass

See Sample Project

Some Quick Cloud Setup

Google Cloud Storage

• For a service Account

  fromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule, get_driver, Provider@Module(modules=[StorageModule.setup(files={# For a service Account"driver": get_driver(Provider.GOOGLE_STORAGE),"options": {"key": "client_email","secret": "private_key","...": "..." }, }, )])classApplicationModule(ModuleBase):
  pass

• Installed Application

  fromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule, get_driver, Provider@Module(modules=[StorageModule.setup(files={# For a service Account"driver": get_driver(Provider.GOOGLE_STORAGE),"options": {"key": "client_id","secret": "client_secret","...": "..." }, }, )])classApplicationModule(ModuleBase):
  pass

• GCE instance

  fromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule, get_driver, Provider@Module(modules=[StorageModule.setup(files={# For a service Account"driver": get_driver(Provider.GOOGLE_STORAGE),"options": {"key": "GOOG0123456789ABCXYZ","secret": "key_secret","...": "..." }, }, )])classApplicationModule(ModuleBase):
  pass

See GCS

AWS S3

fromellar.commonimportModulefromellar.coreimportModuleBasefromellar_storageimportStorageModule, get_driver, Provider@Module(modules=[StorageModule.setup(files={"driver": get_driver(Provider.S3),"options": {"key": "api key","secret": "api secret key" }, }, )])classApplicationModule(ModuleBase):
pass

Specifying canned ACL when uploading an object

If you want to specify custom ACL when uploading an object, you can do so by passing extra argument with the acl attribute to the save or save_content methods.

Valid values for this attribute are:

• private (default)
• public-read
• public-read-write
• authenticated-read
• bucket-owner-read
• bucket-owner-full-control

For example

fromellar.commonimportUploadFile, Inject, postfromellar_storageimportStorageService@post('/upload')defupload_file(self, file: UploadFile, storage_service: Inject[StorageService]):
extra= {"content_type": "application/octet-stream", "acl": "public-read"}
stored_file=storage_service.save(file=file, extra=extra)
return {"message": f"{stored_file.filename} saved"}

API Reference

StorageService

• save(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Saves a file from an UploadFile object.
• save_async(self, file: UploadFile, upload_storage: Optional[str] = None) -> StoredFile: Asynchronously saves a file from an UploadFile object.
• save_content(self, **kwargs) -> StoredFile: Saves a file from content/bytes or through a file path.
• save_content_async(self, **kwargs) -> StoredFile: Asynchronously saves a file from content/bytes or through a file path.
• get(self, path: str) -> StoredFile: Retrieves a saved file if the specified path exists. The path can be in the format container/filename.extension or filename.extension.
• get_async(self, path: str) -> StoredFile: Asynchronously retrieves a saved file if the specified path exists.
• delete(self, path: str) -> bool: Deletes a saved file if the specified path exists.
• delete_async(self, path: str) -> bool: Asynchronously deletes a saved file if the specified path exists.
• get_container(self, name: Optional[str] = None) -> Container: Gets a libcloud.storage.base.Container instance for a configured storage setup.

StoredFile

StoredFile is a file-like object returned from saving and retrieving files. It extends some libcloud Object methods and has a reference to the libcloud Object retrieved from the storage container.

Key attributes include:

• name: File name
• size: File size
• filename: File name
• content_type: File content type
• object: libcloud Object reference
• read(self, n: int = -1, chunk_size: Optional[int] = None) -> bytes: Reads file content
• get_cdn_url(self) -> Optional[str]: Gets file CDN URL
• as_stream(self, chunk_size: Optional[int] = None) -> Iterator[bytes]: Creates a file stream
• delete(self) -> bool: Deletes the file from the container

License

Ellar is MIT licensed.