There have been a number of engineering improvements to Dask Array like consistent chunksizes in Xarray rolling-constructs and improved efficiency in map_overlap. Notably, as of Dask version 2024.11.2, calculating quantiles is much faster and more reliable.
Calculating quantiles is a common operation for geospatial …
Running large-scale GroupBy-Map patterns with Xarray that are backed by Dask arrays is an essential part of a lot of typical geospatial workloads. Detrending is a very common operation where this pattern is needed.
In this post, we will explore how and why this caused so many pitfalls for Xarray …
Dask DataFrame scales out pandas DataFrames to operate at the 100GB-100TB scale.
Historically, Dask was pretty slow compared to other tools in this space (like Spark). Due to a number of improvements focused on performance, it's now pretty fast (about 20x faster than before). The new implementation moved Dask …
The most interesting things about the new release
pandas 2.2 was released on January 22nd 2024. Let’s take a look at the things this release introduces and how it will help us to improve our pandas workloads. It includes a bunch of improvements that will improve the user …
Explaining the migration path for Copy-on-Write
The introduction of Copy-on-Write (CoW) is a breaking change that will have some impact on existing pandas-code. We will investigate how we can adapt our code to avoid errors when CoW will be enabled by default. This is currently planned for the pandas …
The most interesting things about the new release
pandas 2.1 was released on August 30th 2023. Let’s take a look at the things this release introduces and how it will help us improving our pandas workloads. It includes a bunch of improvements and also a set of new …
You can use Coiled Run and Coiled Functions for easily running scripts and functions on a VM in the cloud.
In this post we'll use Coiled Functions to train a RandomForestClassifer on a cloud-hosted machine that has enough cores to speed up our training process. The model parallelizes very well …
Explaining how Copy-on-Write optimizes performance
The first post explained how the Copy-on-Write mechanism works. It highlights some ares where copies are introduced into the workflow. This post will focus on optimizations that ensure that this won't slow the average workflow down.
We utilize a technique that pandas internals use …
Explaining how Copy-on-Write works internally
pandas 2.0 was released in early April and brought many improvements to the new Copy-on-Write (CoW) mode. The feature is expected to become the default in pandas 3.0, which is scheduled for April 2024 at the moment. There are no plans for …
We recently pushed out two new and experimental features Coiled Jobs and Coiled Functions for easily running scripts and functions in the cloud.
In this post we'll use Coiled Functions to process the 150 GB Uber-Lyft dataset on a single machine with DuckDB.
We start with creating the …