Interview With Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry

Interview With Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry

The team decided to do some integration dogfooding by instrumenting our own code with OpenTelemetry alongside existing Beeline instrumentation.

At Honeycomb, we talk a lot about eating our own dogfood. Since we use Honeycomb to observe Honeycomb, we have many opportunities to try out UX changes ourselves before rolling them out to all of our users.

UX doesn’t stop at the UI though! Developer experience matters too, especially when getting started with observability. We often get questions about the difference between using our Beeline SDKs compared with other integrations, especially OpenTelemetry (abbreviated "OTel"). That’s why the team decided to do some integration dogfooding by instrumenting our own code with OpenTelemetry alongside existing Beeline instrumentation.

Poodle is the frontend Go service that renders visualizations in the Honeycomb UI after getting query results and traces from the backend. Engineer Chris Toshok has been working on adding OpenTelemetry to the Poodle code and comparing it with our existing Beeline integration. I talked to Chris about his experience setting up OpenTelemetry from the perspective of a practitioner and service owner.

Interview With Chris Toshok

What were your thoughts going into the effort?

The main concern was schema compatibility. We have existing triggers, boards, SLOs, other things that relied on the schema that the Beeline generated. If we only sent to the existing Poodle dataset, we’d have to make sure that the OTel data would end up with the same field names and values as what's already there, so it wouldn’t break the things that depend on the existing schema.

Alternatively, we could double-send: the Beeline data goes to one dataset and OTel data goes to another. We ended up going this way so that we can look at what the differences are between the two schemas without breaking existing dependencies in our Dogfood team.

Was there anything that surprised you in the process?

While there are lots of little differences between the APIs, the core concepts were the same. The only real sort of surprise was that the Beelines actually let you do stuff that’s kind of scary, because of how the data is stored.

With OTel, as soon as set a value for a field, you can’t modify that value, only replace it. In the Beeline, it’s just a reference to a Go struct. You can just change the values at any point before the data gets sent to Honeycomb. With some fields, like the team object within a span, changing that could be dangerous. But sometimes you want to change a certain value in the middle, like if you need to sanitize values before sending them to Honeycomb.

The configuration for OTel is a bit different from the Beeline, did you also work on getting set up with that?

We basically just set up the OpenTelemetry-Honeycomb Exporter for Go, written by [engineer] Alyson [van Hardenberg].

In the code, I used dependency injection to create an OTel-compatible initializer that works similarly to the Beeline. Beeline data and OTel data are stored a bit differently before they’re sent out, so we need to handle both cases. I added a shim that looks very similar to the OTel API, to wrap around both OTel and the Beeline and allow us to send to Honeycomb using either or both.

If you could advise a team starting today, would you recommend they use the Beeline or OTel for their Go app?

Beeline has a much more specific API that maps more closely to Honeycomb’s features. OTel is doing things at a more general layer, so it might not have all the interesting application bits that we have after several years of doing Honeycomb-specific work in the Go Beeline, which is an effect of the Beeline being something we built out alongside Honeycomb itself. We developed it to answer the questions we had about the service, and to make use of Honeycomb’s features.

What sort of differences have you found?

There were a couple of small, but notable differences. OTel sends JSON blobs as strings, which don’t work with the JSON unfurling feature in Honeycomb.

Also, Beelines allow us to set trace-level attributes. Any span that we send can add a trace-level field. OTel doesn’t support trace-level fields added from child spans.

Have you noticed any performance changes in Poodle after adding OTel?

No change, but I’m wishing we had time over time queries. This is something I was interested in checking on and seeing in time series form.

Observing the Impact

Toshok shared with me the queries he ran to check for any impact on Poodle's performance. Here's the baseline behavior from the week before Toshok's change went out:

Poodle performance from the week before the change went out

Here's the same query, run on the following week. Toshok's changes got rolled out on 7/21:

Poodle performance from the week the change went out

It makes sense that there would be a minimal impact in this case because of how Toshok implemented the change, but it's still neat to be able to query for it.

integration interview go golang observability instrumentation opentelemetry otel

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

What's new in the go 1.15

Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.

"Interview Prep - Golang" Course Released

We just launched Interview Prep - Golang, a quick course for those looking to brush up on some Go quirks before walking into an interview. The course is a

Top 10 Technical Go Interview Questions

Are you interviewing for a new Go developer position? Perhaps you are about to interview someone yourself? Let's review some good questions to be familiar.

“Go Has Indeed Become The Language Of Cloud Infrastructure“ - Rob Pike

We spoke to Rob Pike, the co-author of the Go programming language, about a career spanning four decades, the evolution of Go over the last ten years, and into the future.

Go Tutorial (Golang) - Looping Constructs in Golang | Golang For Loop

In this video we will see the Looping Constructs in Golang. Golang For Loop. Go is an open source programming language which was originally developed by Google. In this Go Tutorial we will Learn Go from the Basics with Code Examples. Go is a statically-typed language.