Lakeflow Designer: Building ETL in Databricks Without Writing a Line of Code

Let’s be honest: for many people, opening Databricks for the first time is a little intimidating. You see notebooks, Spark, SQL, Python, and the immediate thought is, “This is clearly built for engineers, not for me.” The code can be a real barrier, even when you understand the data and know exactly what you want to build.

Good news: that barrier is coming down. With the new Lakeflow Designer, you can build production-ready data pipelines in Databricks using a visual, drag-and-drop canvas, with no code required.

In this blog we’ll walk you through what Lakeflow Designer is, share hands-on experience building a real pipeline with it (including a few AI functions we put to the test), and give you an honest take on where it shines and where it still needs a critical eye.

What is Lakeflow Designer?

Lakeflow Designer is an AI-powered, no-code pipeline builder that is fully integrated into the Databricks Data Intelligence Platform. With a visual canvas and a built-in natural language interface, it lets you build scalable production pipelines and perform data analysis without writing a single line of code.

The clever part is what happens under the hood. Every pipeline you build in Designer creates a Lakeflow Declarative Pipeline behind the scenes, using the same ANSI SQL standard used across Databricks. In practice this means:

• Business analysts can build pipelines in a familiar, visual way.

• Data engineers can review, understand, and improve those same pipelines without switching tools or rewriting logic.

This solves a problem that has plagued traditional no-code tools for years. Legacy no-code tools live outside the data platform, creating silos, redundant work, and fragile pipelines that run without proper governance or observability. Because Designer’s output is a native Lakeflow pipeline, you get versioning, Unity Catalog governance, and full observability from day one, with no rewrites required.

My first impression: familiar territory

We recently got to try Lakeflow Designer myself, and our main takeaway is this: if you’ve worked in a low-code environment before, you’ll feel right at home almost immediately.

The canvas is intuitive. Navigating it, you can quickly:

• Select your sources

• Make joins between tables

• Add logic and transformations

• Add aggregations

• And even add AI functions directly into your flow

There is very little learning curve if you already think in terms of building blocks and visual flows rather than scripts.

Putting the AI functions to the test

The part we were most curious about was the AI functions. Designer lets you drop AI-powered transformations straight into your pipeline, and we wanted to see how they performed on real data rather than a polished demo.

For the test, we used a dataset of sales orders and applied the classify function to split products into two categories: bike parts and non-bike parts.

The result was a genuinely mixed bag, and that is worth being honest about:

• Most bike parts were classified correctly. The function understood the domain surprisingly well out of the box.

• But it was not flawless. The classification was also inconsistent: some non-bike parts were sometimes labelled as bike parts, and the very same parts were sometimes classified correctly on other rows. That inconsistency means the output needed a human review pass before we would trust it downstream.

This is a useful lesson, not a dealbreaker. AI functions can dramatically speed up tasks like categorisation, enrichment, or tagging, but they are an accelerator, not an autopilot. For anything feeding into decisions or reporting, you will still want validation logic or a sampling check in place.

Why this matters

The real significance of Lakeflow Designer is not just "another no-code tool." It is that the visual layer and the engineering layer are finally the same artifact. Because the output is a standard Lakeflow Declarative Pipeline, the work an analyst builds visually is the same code-based pipeline an engineer can open and edit directly in Databricks, with no rebuild required. Designer's AI is also grounded in your data's actual structure, semantics, and usage patterns, including table definitions, column names, and query history, so its suggestions are far more relevant than a generic assistant that sits outside your platform.

For teams, it means analysts and engineers can finally collaborate on the same governed pipelines without painful handoffs or full rebuilds. For individuals who have felt shut out by the code, it is an invitation: you can now build real, production-grade data products in Databricks on your own terms.

Want to try it yourself?

Curious whether Lakeflow Designer fits into your data landscape, or wondering how to combine it with your existing pipelines? We’d love to help you explore it.

Reach out to Joury.Jonkergouw@mccoy-partners.com or visit our expertise page to discuss a hands-on session or proof-of-concept with your own data.

Note: At the time of writing, Lakeflow Designer is rolling out in (Private) Preview, so availability may depend on your Databricks setup.