mosec

We have hosted the application mosec in order to run this application in our online workstations with Wine or directly.

Quick description about mosec:

Mosec is a high-performance and flexible model-serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.

Features:

• Web layer and task coordination built with Rust, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
• User interface purely in Python, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
• Aggregate requests from different users for batched inference and distribute results back
• Spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
• Designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
• Focus on the online serving part, users can pay attention to the model optimization and business logic

Programming Language: Python.
Categories: