Introduction

Dask DataFrame doesn't currently optimize your code for you (like Spark or a SQL database would). This means that users waste a lot of computation. Let's look at a common example which looks ok at first glance, but is actually pretty inefficient.

import dask.dataframe as dd

df = dd …

We recently pushed out two new and experimental features coiled jobs and coiled functions which is a deviation of coiled jobs. We are excited about both of them because they:

• Allow users to scale up any given program on any hardware in the cloud.
• Make GPUs easily accessible without going …

Explaining the pandas data model and its advantages

Introduction

pandas enables you to choose between different types of arrays to represent the data of your DataFrame. Historically, most DataFrames are backed by NumPy arrays. pandas 2.0 introduced the option to use PyArrow arrays as a storage format. There exists …

Getting notified of a significant performance regression the day before release sucks, but quickly identifying and resolving it feels great!

We were getting set up at our booth at JupyterCon 2023 when we received a notification: An engineer on our team had spotted a significant performance regression in Dask. With …

Or: How writing efficient pandas code matters

Introduction

I've regularly seen benchmarks that show how much faster Polars is compared to pandas. The fact that Polars is faster than pandas is not too surprising since it is multithreaded while pandas is mostly single-core. The big difference surprises me though. That's …

Get the most out of PyArrow support in pandas and Dask right now

Introduction

This post investigates where we can use PyArrow to improve our pandas and Dask workflows right now. General support for PyArrow dtypes was added with pandas 2.0 to pandas and Dask. This solves a bunch …

How the API is changing and how to leverage new functionalities

Introduction

After 3 years of development, the second pandas 2.0 release candidate was released on the 16th of March. There are many new features in pandas 2.0, including improved extension array support, pyarrow support for DataFrames and …

Improve performance when selecting data from a pandas object

Introduction

There exist different ways of selecting a subset of data from a pandas object. Depending on the specific operation, the result will either be a view pointing to the original data or a copy of the original data. This ties …

Get rid of annoying SettingWithCopyWarning messages

Introduction

Indexing operations in pandas are quite flexible and thus, have many cases that can behave quite different and therefore produce unexpected results. Additionally, it is hard to predict when a SettingWithCopyWarningis raised and what this means exactly. I’ll show a couple of …