Back to all perspectives

Blog

Building Optionality with Databricks

Blog by Al McEwan and Ethan Hinton

Technology vendors naturally want organizations to make full use of their ecosystems. For Thorogood’s customers, the best long-term position is usually one that preserves choice. Requirements change, new technologies emerge, and commercial relationships evolve. An architecture that is difficult to adapt can turn yesterday’s sensible decision into tomorrow’s constraint.

Preserving that freedom has practical value. It allows organizations to adopt stronger AI models, take advantage of new cloud capabilities, and respond to changing costs without redesigning entire solutions. It also strengthens their position in vendor negotiations because future choices remain credible.

This is why, even on a relatively open platform like Databricks, optionality should be an important consideration. Choosing the platform is only the first step. Every use case is different, so the cost and value of each decision should be assessed against the organization’s business goals.

Make migration a point of renewal

Cloud migration is a process that many organizations go through, whether moving to a multi-cloud architecture or from one cloud provider to another. Alongside taking advantage of a new provider’s capabilities, this presents an opportunity to start building optionality into the architecture.

A typical Databricks migration could involve moving from an older SQL-based data warehouse or reporting environment to a modern data lakehouse. SQL remains widely supported in Databricks, and yet we often use PySpark for data transformation logic.

PySpark is based on the open-source Apache Spark framework, so transformation logic written in PySpark can run in environments other than Databricks. Unity Catalog also has an open-source implementation, although some capabilities remain platform-specific. The best approach is to use open languages and standards where they meet the need, while considering where a proprietary feature is the better option.

AI can also reduce the friction of migrations, despite creating its own optionality considerations. For example, one of our customers in the aerospace industry wanted to move its data loading and preparation code from Qlik Sense to PySpark. We were able to make a first pass quickly using AI to translate the code, before refining the logic to create the finished code in Databricks.

Balance convenience with portability 

An architecture with no dependencies at all is unrealistic, as every technology choice requires some degree of investment that will require costs to alter later. A proprietary service may deliver enough speed, simplicity, or performance to justify the additional switching effort.

Databricks Vector Search illustrates the tradeoff. It is a Databricks platform feature that can make a document index quick and simple to establish. If the surrounding solution uses open interfaces and keeps the index as a contained component, moving to another vector database later should be manageable. In that situation, the immediate convenience may comfortably outweigh the relatively small portability cost.

MLflow provides a second example. The framework is open source, while Databricks offers a managed version that removes much of the setup and administration. An organization can run MLflow elsewhere, although it would need to host and manage the supporting infrastructure. For many teams, the managed service is a sensible choice because the additional effort involved in moving remains clear and manageable.

At Thorogood, we make these assessments throughout the design and delivery for our customers. We consider what would have to change for each component if the organization moved platform, cloud or provider; whether the underlying code and data would remain accessible; and whether an alternative service could be substituted without rebuilding the wider solution.

Keep options open as AI evolves 

Databricks also supports optionality more directly through model serving. No organization can know which AI model will best meet every future use case. The platform allows teams to use proprietary, open-weight and custom models through consistent interfaces, chosen according to quality, cost, latency and risk.

The way agents are built matters too. Databricks supports open frameworks such as LangGraph, while Agent Bricks can generate code that teams can inspect and take elsewhere. This gives developers a quick route into building agents without leaving the organization dependent on a visual interface that conceals how the solution works.

Model Context Protocol, or MCP, offers another example. Databricks provides managed MCP servers for services such as AI/BI Genie and Vector Search, removing the need to build and host standard interfacing logic. Teams can also write custom MCP servers and host them on Databricks. Because MCP is an open standard, the custom server code can be hosted elsewhere, simplifying development while preserving future hosting choices.

Optionality is an architectural habit 

Databricks aligns well with an optionality-led approach because it uses open technologies, familiar interfaces and code that can be inspected and moved. However, maintaining that flexibility still requires active design decisions. At each stage, we assess the cost of each dependency, the value it creates, and what changing course would involve. The answer will vary according to the organization’s architecture, skills, and priorities.

Databricks often makes that assessment straightforward. Where the switching cost looks disproportionate, it becomes visible early enough to choose another design. This gives organizations room to benefit from the platform today while keeping future options open. The result is an architecture that reduces the cost and disruption of future change, protects existing investment, and can adapt as better technologies emerge.

Find out more

Contact Al McEwan. Al is a Data & AI Consultant, and Head of Capability Development at Thorogood

Contact Ethan Hinton. Ethan is a Data & AI Consultant at Thorogood based in the UK.

You might be interested in...

Blog

Databricks Data & AI Summit 2026: 6 Things We've Learned

Databricks Data & AI Summit 2026: 6 Things We've Learned

Upcoming

Webcast

Retailer Advantage through Data: Building an AI-Enabled C...

Retailer Advantage through Data: Building an AI-Enabled CPG Platform with Databricks

Webcast

Apollo Underwriting Business Case with Trevor Hebron & J...

Apollo Underwriting Business Case with Trevor Hebron & James Leonard

Blog

What Does a Good Data Platform Look Like?

What Does a Good Data Platform Look Like?