Frameworks turn resources into platforms.
ClusterD provides the shared resource layer. Framework schedulers and providers add workload-specific orchestration—from Kubernetes clusters and Compose-defined services to Apache Airflow task execution.
mesos-m3s
Run Kubernetes on top of ClusterD.
mesos-m3s is a Mesos® framework that deploys Kubernetes through K3s and supports multiple independent framework instances. ClusterD schedules each instance's components as cluster workloads, while users receive a regular kubeconfig and can work with familiar Kubernetes tools such as kubectl.
- Operate through ClusterDRun K3s server, agent, and etcd components as framework-managed workloads.
- Use the Kubernetes APIRetrieve kubeconfig from the selected framework and connect standard Kubernetes clients.
- Scale framework componentsThe mesos-m3s CLI supports scaling managers, agents, and etcd instances.
- Run many independent clustersStart multiple m3s framework instances side by side—practically as many Kubernetes clusters as available ClusterD/Mesos® resources and operational limits allow.
version: "3.9"
services:
web:
image: registry.example.invalid/demo/web:1.0
ports:
- "8080:80"
deploy:
replicas: 3
resources:
limits:
cpus: 0.5
memory: 256mesos-compose
Orchestrate container services from Compose YAML.
mesos-compose turns supported Compose-spec 3.9 definitions into workloads scheduled by ClusterD. It provides a familiar declarative format for services while retaining ClusterD's resource offers, placement, and distributed execution model.
- Describe complete servicesConfigure replicas, resource limits, ports, volumes, networks, health checks, and placement constraints.
- Manage the workload lifecycleLaunch, list, update, restart, or stop services and individual tasks through the Mesos® CLI plugin.
- Target cluster resourcesUse attributes and constraints for placement, with optional GPU configuration where available.
Compatibility note: mesos-compose implements its documented Compose 3.9 feature set; it does not claim complete Docker Compose compatibility.
Airflow Mesos® Provider
Run Apache Airflow tasks on ClusterD.
The community Airflow Mesos® Provider connects Apache Airflow with Mesos® infrastructure such as ClusterD. It offers two integration paths: a Mesos® Scheduler for distributing Airflow tasks and a Mesos® Operator for launching a specific DAG task directly as a Mesos® task.
- Distribute Airflow tasksUse the Mesos® Scheduler to execute Airflow work across the cluster infrastructure.
- Launch DAG tasks directlyUse the Mesos® Operator when an individual task should run directly as a Mesos® task.
- Connect secured clustersThe provider documentation includes optional Mesos® SSL and authentication settings.
Documented requirements: Apache Airflow 3.x, Python 3.x, and Apache Mesos® 1.6 or newer. The installable package is avmesos_airflow_provider.
Toil
Build portable, scalable workflows in Python.
Toil is a scalable, cross-platform workflow engine for bioinformatics and other data-processing pipelines. Workflows can be written with Toil's Python API or described with the Common Workflow Language (CWL) and Workflow Description Language (WDL), then run locally, on HPC infrastructure, or in the cloud.
- Define workflows in PythonCompose jobs and dependencies with Toil's Python API while keeping workflow logic separate from the execution environment.
- Use open workflow standardsRun portable CWL and WDL workflows alongside native Python workflows.
- Scale beyond one machineMove from local development to distributed execution across cluster or cloud resources.
Toil with Mesos®
Toil includes a Mesos® batch-system integration that can submit workflow jobs as Mesos® tasks. In a ClusterD deployment, Mesos® provides the shared resource pool and Toil supplies the workflow orchestration layer; this is a standard Toil Mesos® backend, not a separate native ClusterD integration.
ClusterD Firecracker Executor
Deploy one microVM per ClusterD task and run the task command inside it.
The ClusterD Firecracker Executor creates a separate microVM for each scheduled task. The task command runs inside that microVM, using Firecracker to provide lightweight virtual-machine isolation for task execution.
- Run one microVM per taskEach ClusterD task gets its own Firecracker microVM and executes its task command inside it.
- Use configuration optionsConfigure agent port (FIRECRACKER_AGENT_PORT=8085), payload file, workdir (/mnt/mesos/sandbox), vCPU (1), and MEM_MB (256).
- Configure the host integrationThe project documents container isolation settings and tap networking for access to the microVM.
Configuration note: The executor documents environment variables for the Firecracker agent, payload, work directory, vCPUs, and memory.
TensorFlow for Mesos
Run TensorFlow scripts against an Apache Mesos® cluster.
tensorflow-mesos is a Python module for TensorFlow V2 and Apache Mesos®. It provides the integration needed to run TensorFlow scripts against a Mesos® cluster, with the Mesos® master and connection settings supplied through environment variables.
- Use TensorFlow 2Install the
tfmesos2package for TensorFlow versions 2.0 and newer. - Connect to Mesos®Configure the Mesos® master and optional SSL and authentication settings through the documented environment variables.
- Run Python workloadsUse the module with TensorFlow scripts such as the examples included in the project.
Documented requirement: Apache Mesos® 1.6.x or newer. Install with pip install tfmesos2.
Bring a scheduler to the shared resource pool.
ClusterD's framework model keeps application-specific scheduling outside the cluster manager. Build a new framework or extend an existing one for your platform.