Success stories and case studies on Grafana Labs

How Adform transformed 1,300 different data sources into one central observability system with Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Tech is filled with origin stories rooted in humble beginnings. From Hewlett Packard and Amazon (garage) to Facebook (dorm room), inspiration and innovation can happen anywhere.

In 2002, three men huddled in a Copenhagen basement bent on changing the way digital advertising buyers and sellers worked: How can we make that process better? Their answer was Adform, an advertising technology platform that now powers 25,000 clients across the world.

All of Adform’s consumer-facing usability and accolades — like a spot in Gartner’s Magic Quadrant and a Red Dot award for user experience — means a lot of behind-the-scenes efforts from their development and operations team to consistently innovate. As Louis e Kloster, Adform’s SVP of Marketing puts it: “Building an advanced ad tech platform is really a complex process.”

One that prompted Adform’s DevOps team to re-evaluate their observability solutions and wrestle with the same question that launched the company: How can we make that process better?

After years of operating in startup mode where developers were cherry-picking tools based on personal preferences and past experiences, it was time to take a step back and restructure a disparate and disorganized observability strategy into a centralized, cohesive approach. “We wanted to build monitoring as a service on the products we believe in,” says Adform’s DevOps Tech Lead Linas Daneliukas.

And they believed that Grafana could help them achieve that. By taking full advantage of Grafana’s open and composable platform and pairing it with Prometheus, Adform’s developers are able to maintain the flexibility in the tools they used while allowing the DevOps team to offer a centralized monitoring experience within the organization. Today, Adform runs more than 1,400 dashboards in Grafana, connecting more than 1,300 data sources and nearly 200 active users across 86 organizations.

“We, as an operator, can have control of the instance itself, but teams have control over their own smaller spaces and can grow as they see fit,” says Daneliukas. “Developers get the control they need, while still having someone who can manage the whole instance. So it was a win-win for us and them.”

Adform’s new observability slogan: One central system

In Adform’s early days, developers “were using what they knew or what they thought was good,” says Tomas Dabašinskas, a DevOps Services Delivery Manager.

But as the company matured and more teams were created, this disconnected ecosystem caused onboarding difficulties, reporting challenges, and infrastructure inefficiency when it came to maintenance and improvement. Any dreams of optimizing existing solutions ended as they faced the reality of constant troubleshooting and ad hoc problem solving. “We weren’t even able to keep things up-to-date,” says Daneliukas.

Around 2018, the DevOps team changed their approach. “We came to the decision as a company to move to centralized services,” adds Dabašinskas.

The first focus was to create a monitoring as a service solution. Initially, they tried to work with the existing software in place such as Graphite, Nagios, Zadig, Graylog, ELK Stack, and multiple Prometheus instances.

We tried to consolidate and do a selection of the tools that we can develop further. However, we view ourselves as a modern company and wanted to see what the best practices were and what’s emerging. So the decision was made to scrap everything we had in regards to observability and create one central and unified system, all powered by Grafana.

Linas Daneliukas, Devops Tech Lead, Adform

Still, the migration was not without challenges. Change, as always, is hard, especially when it includes people revising their processes. But Grafana’s ability to manage a huge number of data sources allowed Adform to slowly transition their teams to Prometheus for metrics by using Grafana as a core part of their stack. “We knew it was difficult for developers to move to Prometheus instantly,” says Dabašinskas. “But Grafana supporting options from Graphite to Prometheus and other choices allowed for this interim period [when developers could still use their preferred tools]."

The work was worth it. Grafana not only brought together data, but teams too — even if they didn’t share the same opinions about their observability tools. With Grafana’s organizations functionality, the DevOps team provided each organization in Adform its own isolated monitoring experience within Adform’s Grafana instance, which made the solution a more cost-effective and streamlined path compared to managing multiple instances. Users would then have view permissions for all of the other organization’s data while maintaining admin access over their own.

“It was a good way to separate our teams, but still allow them that visibility so they can find each other’s dashboards and visualizations,” Daneliukas says. “It also wouldn’t impose roadblocks down the line. We didn’t want to create a solution we would have to deprecate. We raised the criteria quite high for what the tool has to do so that we’re not stuck five or ten years down the line.”

Grafana’s open source roots and growing global community also aligned with Adform’s criteria. If developers get stuck on a problem, Grafana’s online communities have become a prime resource for staying in sync on how to optimize their stack. “Everything is solved via community forums,” says Daneliukas. With the help of the active community, “we’ve never had issues of getting stuck. Everything is just simple. We don’t run into roadblocks with Grafana.”

Instead they have a roadmap of new features and functionalities to always look forward to.

Choosing Grafana was a no-brainer for us because we saw that the tool has a future in front of it and a community behind it. It’s constantly being developed. It was the easiest path and the best path.

Linas Daneliukas, Devops Tech Lead, Adform

An inventory of wins with Grafana

Having a visualization tool across environments, while still maintaining multiple data sources, is key to Adform’s DevOps team turning their developers from hesitant participants to Grafana power users.

“The biggest impact is that people can find what they are looking for,” says Daneliukas. “When we had different tools and an alert fired, it was hard to know who to contact, where to find the dashboard. Then you needed credentials for that monitoring system, which you didn’t always have. You’re basically blind to most things that are happening. Now, with a central monitoring system, when you see an alert anywhere within the company, you can click on the alert and have access. You can go to their organization and view the corresponding dashboard. You have the data in front of you.”

This transparency — aided by built-in support for Prometheus Alertmanager in Grafana — facilitated greater communication and cross-collaboration across Adform’s teams. “Now everything is streamlined,” Daneliukas continues. “You can be sure, when you and the development team are talking about monitoring, you’re using the same language. You know Grafana is the place where we visualize metrics, and Prometheus is the tool we use to collect them. You don’t spend an hour talking to someone trying to fix a problem only to realize they are using their own monitoring solution, which is why nothing seemed to add up."

Consolidation also meant operational efficiency. Originally, development teams averaged 1-3 full-time employees (FTEs) per team and per month on infra maintenance and monitoring. With 25 teams, that adds up to 75 FTEs per month tied up with upkeep rather than innovation. Now, that number is down to 1 FTE per team. The team also provides better upkeep across the board, ensuring the latest versions of software are being used.

Walking around the office, you’ll see TV screens with Grafana dashboards all around. People continuously refer to them. In the end, with Grafana, the ease of troubleshooting has increased exponentially all while we’ve been able to cut our monitoring stack CPU usage by 50%.

Linas Daneliukas, Devops Tech Lead, Adform

Adform flows into the future

As they continue to scale, Adform is looking forward to exploring other ways to improve using Grafana, says Dabašinskas, pointing to Grafana Alerting. “We definitely will test it since for us, it would give us the best of both worlds of managing our alerts in Prometheus format as a code and also being able to see and manage all alerts in a user-friendly way via Grafana UI,” he says.

In the team’s continued effort to “make observability very simple” at Adform, they recently ran a successful PoC of Grafana Loki during a company-wide hackathon where the team collected all the logs across over 3,000 virtual machines with Loki. “We wanted to implement everything from zero overnight and it worked!" says Daneliukas. “We had a single Loki instance that was collecting OS-level logs from every single virtual machine. You could just open up a dashboard, select your machine, grab your logs. That was just astonishing. Now we have it in our roadmaps to implement it.”

The Adform DevOps team is also considering doing a PoC of Grafana Mimir to continue building out an easy-to-use observability experience within their growing Grafana stack.

We like things being a seamless, unified experience.Our developers want that too. We never decided to build everything on top of Grafana. It just happened naturally. We’re not favoring Grafana, but somehow it ends up on the top of the list time and time again.

Linas Daneliukas, Devops Tech Lead, Adform

What started as an effort to provide monitoring as a service naturally evolved to include logging as a service, before adding tracing. Soon, Daneliukas and Dabašinskas can realize the end vision of offering observability as a service.

With the constant ebbs and flows of business and technology, Daneliukas and Dabašinskas are reflective about how they can empower Adform’s teams to realize the company’s ambitions. “We want to make our development team’s life as easy as possible so they can focus on their daily job of building client-facing applications successfully,” says Dabašinskas. “They need to have the right tools. By us providing monitoring and logging as a service, they can focus on what’s important.”

How Dapper Labs uses Grafana Cloud to meet the global demand of NFT-mania

Mon, 01 Jan 0001 00:00:00 +0000

In March 2021, a JPEG created by the digital artist Beeple sold for more than $69 million dollars at Christie’s. The sale started a worldwide obsession with NFTs — or non-fungible tokens — that represent digital collectibles, art, and media. Amid all the headlines, the blockchain gaming studio Dapper Labs came to the forefront.

Launched in 2017 by the venture studio Axiom Zen, Dapper Labs leverages blockchain to build addictive games, verify authentic digital collectibles, and run fan tokens for sports personalities and music artists. The company’s first product, CryptoKitties, is a wildly popular blockchain game that allows players to breed, exchange, and collect virtual cats, and it’s often credited as one of the earliest forays in deploying blockchain technology for NFT-related recreation.

More recently, the blockchain-based start-up launched NBA Top Shot, a digital trading card system that has roughly 19 million transactions and has generated more than $230 million in the sales of NBA game highlights.

Amid the growing fervor around what has been dubbed “NFT Mania,” Dapper Labs has had to hyperscale to meet the global demand for crypto collectibles. That’s where Andrew Burian, principal SRE and engineering manager for the IT, security, and SRE teams at Dapper Labs, comes in.

“The SRE mantra is simple: You don’t have the ability to do anything if you don’t have observability down first,” says Burian.

To that end, Burian and his team built an observability stack for Dapper Labs that includes Grafana Cloud, PagerDuty, Prometheus, Kubernetes, and Google Cloud Platform as its core monitoring tools.

They also provide developers with a suite of out-of-the-box alerts so that as teams write applications, they can also implement shift-left observability and start exporting their metrics to Grafana and monitoring their work right away. “It’s up to devs to define how they want to leverage their alerts and make sure their application stays up,” explains Burian. “We just provide them with the tools, the guidance, and the expertise to do it right.”

That’s no small task. To date, the Dapper Labs SRE team monitors more than 4 million active series, which translates to roughly 12 million data points per minute. With Grafana Loki for logs (that’s up to 2TB of logs per day) and Grafana Tempo for traces, Dapper Labs funnels all of its observability data through Grafana Cloud.

“I have high-level Grafana dashboards that gather all of our disparate systems into one pane of glass because it’s the SRE’s job to have a pulse on the health of the system,” says Burian.

To do so, over the years Dapper Labs quickly evolved from using Grafana OSS to Grafana Cloud for its application and systems monitoring. As a result, Burian adds, “Grafana is heavily used in all facets of the company.”

We do pretty much everything through Prometheus, so for long-term storage, dashboards, and alerting we send all of that to Grafana. Grafana Cloud sits on it and allows us to make the data available so that any product team can dashboard and alert off of the information whenever and wherever they want.

Andrew Burian, Principal SRE and Engineering manager for the IT, security, and SRE teams, Dapper Labs

Graduating to Grafana Cloud

Dapper Labs started its monitoring journey by running its own Prometheus instance as the data store and layering Grafana on top as the open source visualization layer.

“I tend to favor open source projects because I like to know what the product is doing and where,” says Burian. “Also, if I get sufficiently frustrated, I can submit changes.”

Within six to eight months of using Grafana, however, data retention quickly became an issue.

“To spend time carefully managing where the storage goes, what our retention period is, and to make sure that the Prometheus node is beefy enough that we can actually do queries across the last six months of data … It was all a headache,” says Burian.

To eliminate the operational burden on his team, Burian chose Grafana to run the visualization as well as deal with data warehousing. “The fewer things I have to run, the better,” he says.

Since the team had already been a Grafana Cloud customer for metrics, migrating from open source to Grafana Cloud for dashboarding was a natural move when Dapper Labs upgraded its product in 2020.

“For v2, we put the new dashboards on Grafana Cloud,” says Burian. “We slowly deprecated the self-hosted ones. We still have a few old alerts and dashboards on our self-hosted Grafana instance because deprecations of consumer-facing products take forever.”

As Dapper Labs continued to scale, “our agreement with Grafana Cloud progressively added more and more of the Grafana Stack into production,” says Burian, who had considered other monitoring options in the market, but “they’ve all gone by the wayside.” Especially since their price tags were “ludicrous,” charging three to four times more when compared to Grafana Labs’ offering.

Even as their products experienced a 100-fold increase in users, a 1,000x increase in traffic, and their metrics ballooned from 200,000 to almost 4,000,000 active series, “I’ve always thought the price scaled fairly with our usage” on Grafana Cloud, says Burian.

His team also benefits from the time savings that come with Grafana. With only six people in the observability pod supporting an engineering organization of 100 total around Dapper Labs, Grafana Cloud allows Burian’s team to focus on bigger projects without having to worry about maintaining and upgrading every few months.

Says Burian: “Anything that requires babysitting is a lost opportunity cost for us.”

Proactive problem solving

When Burian and his team first started down the path of observability, “the data was hard to instrument, we had one APM stat and some CPU usage, and people just didn’t care.” says Burian. “We’d get performance issue reports from customers long before monitoring caught anything.”

Since Dapper Labs implemented its observability stack in 2019, “we’re at the point where our instrumentation is so good that we catch outages on Google Cloud before Google Cloud has publicly reported them,” says Burian.

When an issue arises, Grafana picks up the incident and sends it to PagerDuty which sends the developers an alert. Then the devs tap into Grafana to see what triggered the alert, using the Grafana learning engine to assess the metrics.

Typically, the devs know what they’re looking for and find it pretty quick. We can probably debug 80 percent of an issue with just metrics. Then the last bit they dive into either logs or our error-reporting tooling for specifics. But quite often, if it’s an issue with scaling, that can be debugged with one or two Grafana dashboards.

Andrew Burian, Principal SRE and Engineering manager for the IT, security, and SRE teams, Dapper Labs

In addition to being more agile in alerting and monitoring, “the empowerment to the teams has been the best single result because that feeds into everything else we do,” says Burian. “There is more productivity and fewer mistakes.”

Open editing season

“Our monitoring and observability platform is one of the first systems internally that we started to think about as its own product being offered to internal customers,” says Burian. “The power of this mindset is that it keeps you thinking about improvement like you would a product.”

Which is why their internal Grafana deployment is continuously shaped by the Dapper community. Within the company, “there are definitely more editors than viewers on our Grafana user list,” says Burian.

The collaborative environment at Dapper Labs encourages more eyeballs — and editing — on products and programs. There are roughly 240 active users in Grafana, including engineers as well as project and product managers who want insight into the data.

“The project management, product management layer, and executive layer are only consuming dashboards, but pretty much every engineer is editing them,” says Burian. “In our company, we really have an open editing culture. The Wiki for all of our products is one giant collaborative editing environment. There’s no real access control. I give everyone global editor access. If you want to tweak a dashboard or you want something from there, you can just jump in and do it.”

The low-activation threshold means there’s an open invitation to contribute to the monitoring work. “Most people are very enthusiastic, and I have not built any controls to stop them from sending as many metrics as their heart wants to Grafana, so they’ve taken advantage of that,” says Burian.

Even new hires tend to make meaningful contributions to the monitoring setup within their first two or three months.

It helps that “Grafana is visually appealing from the get-go,” says Burian. “It’s hard to make a dashboard that looks terrible.” As a result, Burian says that the Grafana dashboards drive usage because “more people look at things that look nice.”

The vast majority of the company’s dashboards are engineering-focused, with only a handful dedicated to high-level summaries of the company’s platforms. Recently, the team also started to formalize SLOs and SLIs into dashboards to track the company’s goals and progress throughout the year.

“That is gaining some traction with product managers and at the executive level,” says Burian. “The goal is to have one SLO and a dashboard that describes the SLO with supporting data you need to verify when things are going wrong.”

“Things can only get better from here”

In the beginning “the low watermaker when we first plugged in was probably around 30,000 series,” says Burian. “But I basically told them to throw everything at the system to get us where we are now.”

As Dapper Labs scaled, so did the company’s Grafana usage. However, there’s a difference between start-up mode — i.e. “We have to build all the things as fast as you can!” — and figuring out where efficiencies are — i.e. “Let’s talk about our bottom line.” Currently Dapper Labs is going through that transition.

“There’s already some tooling that exists to detect which series are absolutely useless. We’re going to start chopping those out,” says Burian.

What will remain is Burian’s overall vision for observability at Dapper Labs. “Real success is zeroing in on using our tooling and our observability to define really good customer-facing SLOs, make really impactful alerts and monitoring, and get what we need without just drowning ourselves in data,” says Burian.

As the team improves at ad hoc instrumentation, they’ll learn to export more metrics, more efficiently. And in the coming months and years as their monitoring use continues to go up, “we’ll ingest new metrics, more logs from new sources, and we want to lean into our tracing capacity now that we have Grafana Tempo,” says Burian.

“I’ve never been happier with our monitoring situation than we are right now,” adds Burian. “Things can only get better from here.”

To hear more about the observability journey at Dapper Labs, check out Andrew Burian’s complete talk at GrafanCONline 2021.

Itaú

Mon, 01 Jan 0001 00:00:00 +0000

Actian

Mon, 01 Jan 0001 00:00:00 +0000

Adevinta

Mon, 01 Jan 0001 00:00:00 +0000

Adevinta

Mon, 01 Jan 0001 00:00:00 +0000

Adobe

Mon, 01 Jan 0001 00:00:00 +0000

After trying to DIY, Wix embraces Grafana Cloud

Mon, 01 Jan 0001 00:00:00 +0000

Wix is the company behind the cloud-based web development platform wix.com. With its drag-and-drop visual site editor, wix.com has enabled millions of users to create beautiful HTML5 and mobile-friendly websites. Headquartered in Tel Aviv, Israel, Wix employs over 1,400 people in offices spread across the globe.

Seeking a fast, stable monitoring solution

Metrics have always been an important part of the culture at Wix. The company requires a super fast – and dependable – solution to generate good alerts. As Wix’s infrastructure has grown, so has the scope of their monitoring and alerting needs. False alerts are painful for any team, and if the underlying database is not stable and available at all times, the alerting data is unreliable. At Wix, “We don’t have the bandwidth to maintain it 24/7,” says Alex Ulstein, Head of Monitoring, “so availability was going be a big factor in choosing our new monitoring stack.”

After years of investigating commercial and open-source solutions, including two different stints of running and trying to scale Graphite on their own, Wix discovered Grafana Cloud, a high-performance, fully managed, production-ready Grafana stack that is 100% Graphite compatible. Alex and his team were immediately intrigued. The product checked a lot of boxes for them: It offered the speed, stability and scale they needed to monitor their critical infrastructure. They were also already using Grafana, Alex says, because of the “open source, pluggability, very rich functionality and of course awesome UI.”

Grafana Cloud would allow them to offload the hard parts—managing and scaling the stack. “When we started the search in 2016, there wasn’t a TSDB that worked for us,” says Alex. “There was a lot of work required, and it wasn’t possible for us to do what the Grafana team has done in terms of scaling, performance and reliability. It doesn’t make sense, price-to-performance, to do it ourselves, so we were looking for a fully managed solution from a team that had experience running monitoring at scale.”

We don’t have the bandwidth to maintain it 24/7, so availability was going be a big factor in choosing our new monitoring stack.

Alex Ulstein, Head of Monitoring, Wix

People talking to people

Throughout the process of implementing Grafana Cloud, Wix worked closely with many members of the core Grafana Labs team, primarily through its chat-based support channel. The ability for integral members of both teams to join together in a fully transparent, asynchronous manner allowed for a smooth deployment.

For Alex, this interaction revealed a lot of structural similarities between the two companies.

“At Wix, it’s totally the same; we have a flat structure. People talking to people; there are no open tickets,” says Alex. “I appreciate that level of deep knowledge and communication.”

Understanding what Wix valued most helped the Grafana Labs team tailor their support in order to speed up implementation.

It wasn’t possible for us to do what the Grafana Labs team has done in terms of scaling, performance and reliability.

Alex Ulstein, Head of Monitoring, Wix

A tale of two users

Wix has two main types of users on their teams. The first group of users are developers who view data from proprietary monitoring systems such as New Relic; the second group is composed of power users who have varying requirements.

For these 250+ power users, their needs are complex. They are spread across multiple teams and have their own specific subsets of metrics that they find valuable. The ability to create custom queries and dashboards is critical for these users. Grafana provides the power and flexibility to create their metrics the way they want to see them, no matter where the data lives. Much of the value Wix gets from Grafana is the ability to show different data types together in one dashboard. For example, Wix combines Grafana Cloud, Elasticsearch, Cloudwatch, and MySQL data on the same dashboard, seamlessly blending business intelligence client data with server metrics.

Even the less frequent users find Grafana extremely valuable because so much attention has been focused on the user experience. For these users, Wix has created a set of templates as a starting point, where queries can be adjusted and display settings and alerts can be tailored to visualize the data they find important to complete their tasks. Alex describes this shortcut as “an extremely fast way to give less-experienced users enough speed to do their jobs effectively.”

An eye toward the future

Now that Grafana Cloud is up and running at Wix, “stability of the platform is our primary focus,” says Alex. “But I am always thinking about how we can improve our monitoring. Prometheus and Kubernetes are interesting now and will continue to be in the future. Another key consideration is finding a way to increase the frequency of my collection so I can get alerts and address any performance issues faster.”

With Grafana as the front end, the Wix team know they have that ability to experiment with new tools—without having to sacrifice the stability and speed of their primary TSDB.

During implementation, the many active conversations with multiple members of the Grafana Labs team helped Wix tune their relays and other infrastructure to get the most out of Grafana Cloud. And as Wix continues to evolve their monitoring stack, Grafana Labs will keep those conversations going. It’s the Grafana Labs way: People working together, creating the best data back end while allowing them to experiment with new open-source tools and technologies.

Agrology

Mon, 01 Jan 0001 00:00:00 +0000

Ahold Delhaize

Mon, 01 Jan 0001 00:00:00 +0000