One of the biggest advancements in large-scale data analysis in recent years occurred with the launch of Databricks, a web-based platform that works with Apache Spark to provide automated cluster management.

As a Databricks partner for more than three years, Thorogood has collaborated closely with the firm while developing a variety of full-scale analytics systems for some of the world’s largest companies. Databricks is an essential tool for data engineers and data scientists alike, especially for companies who utilize multiple different cloud providers.

Here are five important things to know about this critical platform:

1) Databricks was built with a multi-cloud world in mind

As companies look to minimize costs and maximize the flexibility of their systems, they are increasingly breaking free of the single-vendor approach that was common in the early stages of the cloud revolution. Databricks can play a key role in facilitating this multi-cloud approach, given its compatibility with a variety of different providers, including Amazon Web Service, Google Cloud, and Microsoft Azure. The ease of transferring code between Databricks instances in different clouds means that investments can easily be repurposed as strategies evolve.

2) Databricks’ cluster-based distributed computing model enables its competitive pricing

Databricks is the latest step in an evolving line of platforms that originated with the AMPLab project at the University of California-Berekely, where Apache Spark was developed. Databricks is built on top of the Spark engine, a distributed computing architecture that uses clusters of computation resources (nodes) to execute and manage data workloads. Each cluster includes a driver node that assigns tasks and oversees the execution of a specific workload via a cluster of worker nodes, which perform the processing. Because Databricks only charges for the time that cluster is running, the platform is a cost-effective option that minimizes waste.

3) Databricks unifies data engineering and data science

One of the big challenges in computing at the enterprise level is creating and maintaining synergy between data engineers and data scientists. Because Databricks works well with the native platform offerings of the Amazon, Google and Microsoft Clouds, it offers a bridge of sorts between the needs of data engineering and data science teams on a given project. By nature, it is extremely effective at auto-scaling, using only the computing power needed at a specific point in time. Meanwhile, its notebooks can be written in a variety of popular languages utilized by engineers and scientists alike, including Python, SQL, and Scala.

On the data engineering side of things, Databricks offers plenty of architecture flexibility. It works well with Azure and AWS services likes ADLS and S3, but also offers its own proprietary data warehousing platform via Delta Lake, a new open-source offering that allows for a more structured environment.

On the data science side, Databricks is pre-installed with a number of popular libraries and packages (PyTorch, Keras, NumPy, Pandas, etc.) and features seamless integration with MLFlow, Databricks’ own machine learning platform.

4) Databricks was built to encourage and facilitate collaboration in data engineering processes

The platform’s notebooks are designed so that users can collaborate in real-time in an environment where developers can write code to the same notebook where the updates are made in real-time. Multiple different languages can be used within the same notebook. Databricks’ Spark engine optimizes the orchestration of workloads, whether using a third-party tool like Azure Data Factory or its internal management tool. Meanwhile, Delta Lake offers an open-format storage layer built on top of a data lake that provides enhanced reliability and security for both streaming and batch operations.

5) Databricks offers users the ability to perform complete, cutting-edge data science at any scale.

Databricks’ ML run times come pre-installed with so many different machine learning and artificial intelligence libraries. Meanwhile, it is constantly improving its MLOps capabilities. In addition to real-time scoring and its own Feature Stores service, Databricks offers fully-managed MLFlow for tracking, versioning, and storing experiments. Further enhancing the data science experience are its notebooks, which enable parallel development with automatic version control.

To explore the implementation of Databricks in your organization, reach out to Alaistair Jones.