Performant workflow orchestrator
Optimus is an open source performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Powering the best data teams
Framework
Modular architecture
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. It enables data analysts and engineers to transform their data by writing simple SQL queries and YAML configuration while Optimus handles dependency management, scheduling and all other aspects of running transformation jobs at scale.Zero dependency
Optimus is written in Go and compiles into a single binary with no external dependencies, and requires a very minimal memory footprint.Warehouse management
Optimus allows you to create and manage your data warehouse tables and views through YAML based configuration.Extensible
With the ease of plugin development build your own plugins. Optimus support Python transformation and allows for writing custom plugins.CLI
Optimus comes with a CLI which allows you to interact with workflows effectively. You can create, run, replay jobs and more.Proven
Battle tested at large scale across multiple companies. Largest deployment runs thousands of workflows on multiple data sources.Workflows
Optimus provides industry-proven workflows using git and REST/GRPC based specification management for data warehouse management.
Key Features
Built for ease
Optimus is an ETL orchestration tool that helps manage warehouse resources and schedule transformation over cron interval. Warehouses like Bigquery can be used to create, update, read, delete different types of resources(dataset/table/standard view). Similarly, jobs can be SQL transformations taking inputs from single/multiple source tables executing over fixed schedule interval. Optimus was made from start to be extensible, which is, adding support of different kind of warehouses, transformers can be done easily.Scheduling
Optimus provides an easy way to schedule your SQL transformation through a YAML based configuration.Dependency resolution
Optimus parses your data transformation queries and builds a dependency graphs automaticaly instead of users defining it in DAGs.Dry runs
Before SQL query is scheduled for transformation, during deployment query will be dry-run to make sure it passes basic sanity checks.Powerful templating
Optimus provides query compile time templating with variables, loop, if statements, macros, etc for allowing users to write complex tranformation logic.Cross tenant dependency
Optimus is a multi-tenant service. With more than two tenants registered Optimus can resolve cross tenant dependencies automatically.Hooks
Optimus provides hooks for post tranformation logic to extend the functionality of the transformation. e,g. You can sink BigQuery tables to Kafka.Workflow
Pre-built integrations
With Optimus data teams work directly with the data warehouse and data catalogs. Optimus provides a set of workflows which can be used to build data transformation pipelines, reporting, operational, machine learning workflows.Develop
Write your specifications in git using Optimus CLI or use Optimus APIs to prgramtically submit specifications through SDK.Test
Test your workflows prior to production with linting, dry runs and local execution from your machines.Deploy
Deploy your workflows safely with Optimus CLI and APIs to production.
Slack
Github
