Handling Partial Writes in InfluxDB 3.0

By Gary Fowler / Product
Sep 19, 2024

We recently adjusted how we handle “partial writes” with our InfluxDB Cloud Serverless product using the v2 Write API. This only applies to InfluxDB Cloud Serverless customers (those who created their Cloud accounts after January 31, 2023). In the near future, we will make this change for InfluxDB Cloud Dedicated and InfluxDB Clustered customers as well.

Partial writes will now always return a 400 status code. In most cases, you don’t have to do anything but be aware of the behavior. If you are using the v2 Write REST API directly, make sure your code handles 204 and 400 return codes and know that with a 400, it doesn’t mean all lines didn’t make it (the good lines will be written). Read on if you want more information and background on this change.

What are partial writes?

Most commonly, data gets written to InfluxDB in “batches,” meaning several lines at once. If there are lines within that batch that can’t be written for some reason, we refer to this as a “partial write.”

What conditions can cause a partial write?

The most common condition that causes a partial write is an error in the Line Protocol. Batch lines are formatted in the line protocol format, which requires a specific syntax. If there is an error in that syntax for a specific line, that line cannot be written.

What status code is returned on a partial write?

The status code for partial writes is now always 400.

What was it before?

With Cloud Serverless, a 204 could be returned in some cases. This matched the behavior in Cloud2. In Cloud2, data is first written to an ingestion queue (we used Apache Kafka), and then, as a secondary action, the data is written to the database. Not all line protocol errors can be detected when data is written to the ingestion queue (e.g., schema inconsistencies). Therefore, we acknowledge we have received the data and return a 204 response before it is fully validated. When the data is validated, if there is a problem with any write lines, those lines are written to a separate Cloud 2 bucket (_monitoring) so customers can see and correct any problematic lines. This “_monitoring” bucket does not exist with Cloud Serverless/InfluxDB 3.0, so it didn’t make sense to continue returning this status code on a partial write.

Partial Writes were not supported by default in InfluxDB Cloud Dedicated or InfluxDB Clustered, but for customers for whom we had enabled them, the status code could sometimes be a 201.

What else is returned upon a partial write?

When a 400 is returned, it also returns the first 100 failed lines in the batch—so, if there is an exception of just a few lines, your code can know which lines failed.

Future

Our plan is not to make any additional changes to this. However, we do have a v3 Write API planned that will completely revisit this, hopefully add a few more options, and handle partial writes better overall. We don’t plan on deprecating the v2 Write API; you can still use both (you can still use v1, in fact). This will be a new v3 option with more capabilities—including some new data types (teaser alert).

Contact our sales team to get started with InfluxDB Cloud Dedicated or Clustered.