golang write data to influxdb v2.7 in batches example

Lussy04

3 min read

·

24-01-2025

1

0

Share

InfluxDB is a popular time-series database, and Go (Golang) is a fantastic language for interacting with it. This article demonstrates how to efficiently write data to InfluxDB 2.7 in Go using batching. Batching significantly improves performance compared to writing individual data points. We'll cover writing both line protocol and using the InfluxDB Go client.

Setting Up Your Environment

Before we begin, ensure you have the necessary tools installed:

• Go: Make sure you have a recent version of Go installed. You can check with go version.
• InfluxDB: Download and install InfluxDB 2.7. You can download it from the official InfluxDB website.
• InfluxDB Go Client: Install the InfluxDB Go client using:

go get github.com/influxdata/influxdb-client-go/v2

Writing Data using Line Protocol

This method involves constructing the line protocol directly and sending it to InfluxDB. While less convenient than the client, it provides more control.

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/influxdata/influxdb-client-go/v2"
)

func main() {
	// Replace with your InfluxDB connection details
	url := "http://localhost:8086"
	token := "your-influxdb-token"
	org := "your-org"
	bucket := "your-bucket"

	client := influxdb2.NewClient(url, token)
	defer client.Close()

	writeAPI := client.WriteAPIBlocking(org, bucket) // Blocking write for simplicity
	defer writeAPI.Close()


	// Batch data
	batchSize := 1000
	batch := make([]influxdb2.Point, 0, batchSize)


	for i := 0; i < 2000; i++ {
		p := influxdb2.NewPoint("measurement", map[string]string{"tag": "value"}, map[string]interface{}{"field": i}, time.Now())
		batch = append(batch, p)

		if len(batch) >= batchSize {
			err := writeAPI.Write(context.Background(), batch)
			if err != nil {
				log.Fatal(err)
			}
			batch = batch[:0] // Reset the batch
		}
	}

	// Write any remaining data points
	if len(batch) > 0 {
		err := writeAPI.Write(context.Background(), batch)
		if err != nil {
			log.Fatal(err)
		}
	}

	fmt.Println("Data written successfully!")
}

Remember to replace placeholders like your-influxdb-token, your-org, and your-bucket with your actual values. This code creates a batch of points and writes them to InfluxDB when the batch reaches batchSize.

Using the InfluxDB Go Client

The InfluxDB Go client provides a more structured and user-friendly approach. It handles batching internally, making it simpler to use.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/influxdata/influxdb-client-go/v2"
)

func main() {
	// Replace with your InfluxDB connection details
	url := "http://localhost:8086"
	token := "your-influxdb-token"
	org := "your-org"
	bucket := "your-bucket"

	client := influxdb2.NewClient(url, token)
	defer client.Close()

    writeAPI := client.WriteAPI(org, bucket)
    defer writeAPI.Flush() // ensures all points are written before closing

	for i := 0; i < 2000; i++ {
		p := influxdb2.NewPoint("measurement", map[string]string{"tag": "value"}, map[string]interface{}{"field": i}, time.Now())
		writeAPI.WritePoint(p)
	}

	fmt.Println("Data written successfully!")

}

This example uses WritePoint which handles the underlying batching for you. The defer writeAPI.Flush() is crucial; it ensures all pending writes are sent before the program exits. The client's default batching settings are generally efficient.

Error Handling and Best Practices

For production systems, add robust error handling: check for network issues, authorization problems, and data validation errors. Consider implementing retry mechanisms to handle transient network failures.

Always choose the appropriate batch size based on your data volume and network conditions. Experiment to find the optimal size that balances performance and resource usage. Larger batches can reduce overhead but increase the risk of data loss if a write fails.

Conclusion

This article provided two approaches to efficiently write data to InfluxDB 2.7 in Go using batching. The choice between using the line protocol directly or the InfluxDB Go client depends on your needs and preferences. Remember to implement thorough error handling and optimize your batch size for optimal performance in a production environment. Efficient data ingestion is key to maximizing the performance of your time-series database.