Explaining the migration path for Copy-on-Write
The introduction of Copy-on-Write (CoW) is a breaking change that will have some impact on existing pandas-code. We will investigate how we can adapt our code to avoid errors when CoW will be enabled by default. This is currently planned for the pandas …
The most interesting things about the new release
pandas 2.1 was released on August 30th 2023. Let’s take a look at the things this release introduces and how it will help us improving our pandas workloads. It includes a bunch of improvements and also a set of new …
You can use Coiled Run and Coiled Functions for easily running scripts and functions on a VM in the cloud.
In this post we'll use Coiled Functions to train a RandomForestClassifer on a cloud-hosted machine that has enough cores to speed up our training process. The model parallelizes very well …
Explaining how Copy-on-Write optimizes performance
The first post explained how the Copy-on-Write mechanism works. It highlights some ares where copies are introduced into the workflow. This post will focus on optimizations that ensure that this won't slow the average workflow down.
We utilize a technique that pandas internals use …
Explaining how Copy-on-Write works internally
pandas 2.0 was released in early April and brought many improvements to the new Copy-on-Write (CoW) mode. The feature is expected to become the default in pandas 3.0, which is scheduled for April 2024 at the moment. There are no plans for …
We recently pushed out two new and experimental features Coiled Jobs and Coiled Functions for easily running scripts and functions in the cloud.
In this post we'll use Coiled Functions to process the 150 GB Uber-Lyft dataset on a single machine with DuckDB.
We start with creating the …
Dask DataFrame doesn't currently optimize your code for you (like Spark or a SQL database would). This means that users waste a lot of computation. Let's look at a common example which looks ok at first glance, but is actually pretty inefficient.
import dask.dataframe as dd
df = dd …
We recently pushed out two new and experimental features coiled jobs
and coiled functions
which is a deviation of coiled jobs. We are excited about both of them because they:
Explaining the pandas data model and its advantages
pandas enables you to choose between different types of arrays to represent the data of your DataFrame. Historically, most DataFrames are backed by NumPy arrays. pandas 2.0 introduced the option to use PyArrow arrays as a storage format. There exists …
Getting notified of a significant performance regression the day before release sucks, but quickly identifying and resolving it feels great!
We were getting set up at our booth at JupyterCon 2023 when we received a notification: An engineer on our team had spotted a significant performance regression in Dask. With …