chaos-exporter VS statsd_exporter

Chaos interleaved dashboards (i.e., instrumented app dashboards/panels with prom functions on chaos metrics in order to indicate the period of chaos injection/experiment execution) is something that has been in practice and, like mentioned previously, used the event router. While this did help, it was by no means a perfect solution - as upstream implementations of the event-router didn’t have the desired event filtering capabilities (leading to more space on the TSDB) and events are relatively ephemeral entities that disappear from the etcd after a specified period of time. As a result, the chaos exporter was improved to generate more meaningful metrics to indicate the state & results of the chaos experiments - the source being a richer (schema-wise) ChaosResult CR.

It's part of the statsd_exporter.

package metrics import ( "fmt" "strings" "time" "github.com/alexcesaro/statsd" ) // Hybrid metrics type Hybrid struct { Statsd StatsdConfig Client *statsd.Client AttachLabels bool } // StatsdConfig ... type StatsdConfig struct { Host string Prefix string } // Init ... func (m *Hybrid) Init() error { // Set up the statsd client client, err := statsd.New( statsd.Address(m.Statsd.Host), statsd.Prefix(m.Statsd.Prefix), statsd.FlushPeriod(time.Second*1), ) if err != nil { return fmt.Errorf("Hybrid: Could not connect to %s: %s", m.Statsd.Host, err.Error()) } m.Client = client return nil } // Format key with labels func (m *Hybrid) formatKeyWithLabels(key string, labels map[string]string) string { // Do we want to attach labels? if m.AttachLabels == false { return key } // Use librato-style tags // metrim.name#tagName=val,tag2Name=val2:0|c // https://github.com/prometheus/statsd_exporter#tagging-extensions if len(key) == 0 { return key } formattedLabels := []string{} for labelKey, labelVal := range labels { formattedLabels = append(formattedLabels, fmt.Sprintf("%s=%s", labelKey, labelVal)) } joinedFormattedLabels := strings.TrimSpace(strings.Join(formattedLabels, ",")) if joinedFormattedLabels != "" { key = fmt.Sprintf("%s#%s", key, joinedFormattedLabels) } return strings.Trim(key, "#") } // Increment ... func (m *Hybrid) Increment(key string, labels map[string]string) { m.IncrementBy(key, 1, labels) } // IncrementBy ... func (m *Hybrid) IncrementBy(key string, by int, labels map[string]string) { updatedKey := m.formatKeyWithLabels(key, labels) if m.Client != nil { m.Client.Count(updatedKey, by) } } // Gauge ... func (m *Hybrid) Gauge(key string, value int, labels map[string]string) { updatedKey := m.formatKeyWithLabels(key, labels) if m.Client != nil { m.Client.Gauge(updatedKey, value) } }