Edge Telemetry Daemon

View the repository

Project Overview: This daemon captures 1000Hz IMU hardware data for a firearm 3D visualization pipeline. At 1000Hz, missed packets create visible jitter in the motion trail. Because UDP does not wait for the application. the primary goal of this daemon is to stream raw bytes to disk before the OS buffer fills or garbage collection pauses the runtime.

Bypassing the Database

Databases parse, index, manage locks and allocate memory. Putting a database in the real-time hot path introduces unnecessary latency and risks dropped packets.

The ingestion layer only needs one guarantee: save the bytes immediately. Querying and interpretation are offloaded to later stages.

Packet Shape & Binary Format

Each sensor sample is 20 bytes: one timestamp and six sensor values.

type SensorData struct {
 Timestamp int64
 Thumb     uint16
 Index     uint16
 Middle    uint16
 Ring      uint16
 Pinky     uint16
 Palm      uint16
}

Five samples are batched into a 100-byte UDP packet.

const (
 BatchSize  = 5
 SampleSize = 20
 PacketSize = BatchSize * SampleSize
)

Instead of JSON-which inflates payload size and requires parsing-the client packs values directly into a binary format. The daemon simply appends these bytes to a file.

func (s SensorData) Pack(buf []byte) {
 binary.LittleEndian.PutUint64(buf[0:8], uint64(s.Timestamp))
 binary.LittleEndian.PutUint16(buf[8:10], s.Thumb)
 binary.LittleEndian.PutUint16(buf[10:12], s.Index)
 binary.LittleEndian.PutUint16(buf[12:14], s.Middle)
 binary.LittleEndian.PutUint16(buf[14:16], s.Ring)
 binary.LittleEndian.PutUint16(buf[16:18], s.Pinky)
 binary.LittleEndian.PutUint16(buf[18:20], s.Palm)
}

Buffer Pooling (Zero Allocations)

Allocating a new byte slice for every packet gives the garbage collector (GC) a reason to trigger a stop-the-world pause. To keep the GC out of the capture path, the daemon uses a sync.Pool of fixed buffers.

func newBufferPool(size int) *sync.Pool {
 return &sync.Pool{
  New: func() any {
   b := make([]byte, size)
   return &b
  },
 }
}

The loop is strictly mechanical: Get a buffer → Read UDP → Send to writer → Return buffer.

we can use [100]byte array instead of sync.Pool, but i need it for concurrency features that are out of scope for now.

Read Path

The reader pulls a buffer, reads the packet and verifies the size.

bufPtr := r.bufferPool.Get().(*[]byte)
buffer := *bufPtr

n, err := r.conn.Read(buffer)
if err != nil {
 r.bufferPool.Put(bufPtr)
 if ctx.Err() != nil {
  return
 }
 log.Println("Read Error:", err)
 continue
}

if n != r.packetSize {
 r.bufferPool.Put(bufPtr)
 log.Printf("warning: received malformed packet of length %d", n)
 continue
}

It checks timestamps to reject out-of-order packets. It does not enforce full ordering for now, but acts as a cheap guard against stale data.

currentPacketTime := binary.LittleEndian.Uint64((*bufPtr)[0:8])
if currentPacketTime < r.lastPacketTime {
 atomic.AddUint64(&r.OldPacketsRejected, 1)
 r.bufferPool.Put(bufPtr)
 continue
}
r.lastPacketTime = currentPacketTime

Write Path

The writer appends packets to data.bin via a 32KB buffered writer.

writer := bufio.NewWriterSize(file, 32*1024)

n, err := writer.Write(*bufPtr)
if err != nil {
 log.Println(err)
} else {
 atomic.AddUint64(&d.PacketsWritten, 1)
 atomic.AddUint64(&d.BytesWritten, uint64(n))
}
d.bufferPool.Put(bufPtr)

Failing to return the buffer to the pool (d.bufferPool.Put) forces the system to allocate, defeating the pool’s purpose.

The writer flushes every two seconds to balance disk I/O pressure against the risk of data loss on power failure.

t := time.NewTicker(d.flushEvery)
defer t.Stop()

Channel Buffer

A buffered channel decouples the reader and writer. It absorbs short network bursts when packets arrive unevenly, giving the disk time to catch up.

packetBuffer := make(chan *[]byte, 10000)

Out of Scope

To maintain throughput, this layer explicitly excludes:

1. SQL/Indexing
2. Concurrency
3. Network retries
4. Dashboards

Results

Benchmarks confirm zero allocations in the hot path:

BenchmarkDiskWriter-12      7592066        166.5 ns/op  600.45 MB/s        0 B/op        0 allocs/op

During a 24-hour stress test, the daemon processed 1.55 GB of telemetry with zero dropped packets and zero GC cycles.

Total Run Time       : 24h0m0.001219672s
Total Rows Saved     : 81368675

Packets Received     : 16273735
Bytes Received       : 1627373500 (1551.98 MB)

Packets Written      : 16273735
Bytes Written        : 1627373500 (1551.98 MB)

Internal Drops       : 0 (Perfect Transfer)
Total GC Cycles Run  : 0