Metadata collection framework
Meteor is an open source plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog or store.
Powering the best data teams
Framework
Modular architecture
Meteor agent uses recipes as a set of instructions which are configured by user. Recipes contains configurations about the source from which the metadata will be fetched, information about metadata processors and the destination to where the metadata will be sent.Extraction
Extractors are the set of plugins that are source of our metadata and include databases, dashboards, users, etc.Processing
Processors are the set of plugins that perform the enrichment or data processing for the metadata after extraction.Sink
Sinks are the set of plugins that act as the destination of our metadata after extraction and processing is done by agent.
Key Features
Built for ease
Meteor is a plugin driven agent for collecting metadata. Meteor has plugins to source metadata from a variety of data stores, services and message queues. It also has sink plugins to send metadata to variety of third party APIs and catalog services.Zero dependency
Meteor is written in Go and compiles into a single binary with no external dependencies, and requires a very minimal memory footprint.Coverage
With 50+ plugins and many more coming soon to extract and sink metadata, it is easy to start collecting metadata from various sources.Extensible
With the ease of plugin development you can build your own plugin to fit with your needs. It allows new sources, processors and sinks to be easily added.Proven
Battle tested at large scale across multiple companies. Largest deployment collect metadata from thousands of data sources.CLI
Meteor comes with a CLI which allows you to interact with agent effectively. You can list all plugins, start and stop agent, and more.Runtime
Meteor can run from your local machine, cloud server machine or containers with minium efforts required for deployment.Ecosystem
Pre-built integrations
Meteor’s plugin system allows new plugins to be easily added. With 50+ plugins and many more coming soon to extract and sink metadata, it is easy to start collecting metadata from various sources and sink to any data catalog or store.Extractors
Meteor supports source plugins to extract metadata from a variety of datastores services, and message queues, including BigQuery, InfluxDB, Kafka, Metabase, and many others.Processors
Meteor has in-built processors including enrichment and others. It is easy to add your own processors as well using custom plugins.Sinks
Meteor supports sink plugins to send metadata to a variety of third party APIs and catalog services, including Compass, HTTP, BigQuery, Kafka, and many others.
Slack
Github
