Big-data / Spark.
Managed Spark and distributed processing for large-scale transforms and ML feature pipelines.
Managed Spark and distributed processing on elastic clusters — run large transforms, joins and ML feature pipelines without standing up and babysitting a cluster.
- Category
- Analytics
- Deployment
- Managed → air-gapped
- Governance
- IAM · encryption · audit
Spark is the workhorse. The cluster operations are the tax.
For large transforms, joins over billions of rows, and ML feature pipelines, Spark is still the default engine — and it works. What doesn't work is the operational weight around it: sizing clusters for a worst-case job and paying for them at idle, babysitting a long-running cluster, and rebuilding the bridge between the data engineering platform and wherever the model training actually happens. The compute is solved; the economics and the seams are not.
Lake-native, so there's no ingestion step before the work.
Aether's Spark reads and writes the open table formats in your data lake directly. There is no load step that copies the lake into the processing engine's own storage before a job can run, and no second copy to keep consistent afterwards. A pipeline reads the same governed tables the warehouse and the model read, writes results back as open tables, and the lineage is continuous from source to feature to prediction — one catalog, not three.
From feature pipeline to training without crossing a boundary.
The reason Spark sits next to the model here is that most Spark in practice is feeding ML — computing features, building training sets, scoring batches. On a standalone Spark platform that handoff is an export to a separate training stack. On Aether the features a job computes are immediately trainable and servable by the model on the same platform and the same data, so the distance between data engineering and the model collapses from an integration to a function call.
A feature pipeline that reads the lake, computes a rolling aggregate, and writes a feature table the model trains on next — no copy out, no separate training stack.
from aether import spark
df = spark.read.table("lake.events") # open table, no ingest
features = (
df.groupBy("user_id")
.agg(spark.avg("amount").alias("avg_30d"))
)
features.write.table("lake.user_features") # back to the lake
# trainable immediately — same platform, same data, no export
spark.aether.train(table="lake.user_features", target="churn")On the lakehouse platform or EMR the last line is a different product and usually a different copy of the data. Here the feature table a Spark job just wrote is trained on in place, on the same governed lake, under the same identity — the gap between the pipeline and the model is a function call, not a pipeline.
Three steps to running.
Submit a Spark job or open an interactive notebook against an autoscaling cluster.
Process open table formats in the data lake directly, no ingestion step.
Clusters scale to the work and tear down after — run on spot for big, cheap batches.
Big-data / Spark, in full.
Autoscaling clusters that spin up for a job and tear down after.
Interactive exploration and scheduled production jobs alike.
Read and write open table formats in the data lake directly.
Run on spot with automatic recovery for big, cheap batches.
Provision it in a few lines.
Every service is reachable from the same SDK, CLI and infrastructure-as-code — one identity, one bill, one audit trail across the whole catalog.
# Provision big-data / spark on Aether Cloud
aether analytics create \
--service big-data-spark \
--name app \
--region us-1 \
--deploy managed # or vpc | on-prem | air-gappedAt a glance.
- Engine
- Managed Spark, autoscaling
- Interface
- Notebooks + scheduled jobs
- Storage
- Lake-native, open formats
- Economics
- Spot with automatic recovery
- Languages
- SQL · Python · Scala
Built for real work.
Large-scale ETL
ML feature pipelines
Log and event processing
On one model, not stitched together.
The usual stack runs big-data / spark in one product, the model in another and the data in a third — and the seams between them are the cost. Aether Cloud runs it on the same platform that serves the model, governs your identity and deploys into your boundary, with the rest of the catalog one hop away.
No stitching a vector DB to one place, a warehouse to another and a model to a third — big-data / spark sits next to the rest of the catalog, one identity, one bill.
The provider that runs Aether runs your big-data / spark — so the data and the model never leave the same governed boundary to talk to each other.
Need a capability that isn’t here yet? The model writes and deploys it into the same boundary — the catalog is a starting point, not a ceiling.
Good to know.
Managed Spark with notebooks and lake-native I/O — but on the platform that serves the model, so feature pipelines feed training without leaving the boundary.
Yes — the same runtime powers interactive notebooks and scheduled, retried production jobs.
Run on spot with automatic recovery; clusters spin up per job and tear down after.
Your boundary, your choice.
Pairs well with.
Exactly-once streaming compute for real-time pipelines, enrichment and detection.
Visual and code-first pipelines with scheduling, lineage and schema management.
Governed dashboards and self-serve exploration over the warehouse and lake.
Full-text and hybrid search as a service, with relevance tuning and analyzers.
Run Big-data / Spark on Aether Cloud.
Managed Spark and distributed processing for large-scale transforms and ML feature pipelines. Deployable managed, in your VPC, on-prem or fully air-gapped — talk to us about the configuration your workloads and your boundary require.