Sentry keeps you honest

We have recently moved to Sentry for error tracking and other than praising how much of a great product it can be I came to realize that it does something more. Before moving to Sentry we leveraged an internal pipeline for ingesting logs. You could easily query for specific errors, set alerts, or build dashboards for tracking. Given the size of MongoDB it wasn't transparent how much it costs to generate all those logs.

Sentry offers a very nice way to review, categorize and do much more with your errors. It also keeps you honest. While I recommend for everyone to have their own log decision tree (will talk about it another time but it's a simple tool to help you decide if it's should be an info, warning, error, debug), when you see how many logs just one error can generate you quickly realize that you navigated the decision tree poorly.

We had an error log make it into production that generated over 4M logs in a day. It was fairly new and Sentry yelled at us for having a high number of errors. I went ahead and fixed it assigning it the proper log level and life was good again. Unfortunately it affected our quota so we had to reel it in until the next cycle.

Thoughts

If it wasn't for Sentry we might have had a non-error log generate a LOT of entries every day. We still generate a ton of logs for audits, debugging and all the good stuff, but when you have that type of visibility on your errors you realize how a small change can affect your system.

Sentry was one way to surface this but it forced us to really think about our logs and what level they should be.

If you don't second guess if a log should be an error in your reviews, you should start doing it now! We keep the decision tree handy as a doc in our codebase and I regularly use to suggest changes where needed. Next time I'll talk about it I'll share it here!