How to measure everything - in engineering teams, in company decisions, in life
Meet Adi Belan, VP R&D & Measurement at Appsflyer. We spoke to Adi about running engineering teams, transitioning from engineer to leadership roles, what can and cannot be measured, and how Adi finds focus in today’s cluttered environment. Watch the full interview, or read a lightly edited version below.
What was your first job?
My first real job was waiting tables at an expensive restaurant in Tel Aviv. It actually paid very well. What I most learned about from there was how to deal with all kinds of different people and understand what people are looking for and how to make them happy and feel good about their experience.
My first real industry job was also a funny story. I got brought into the industry by a friend, a company called Flash Networks; he brought me onto his team and left the company two weeks later. There I learned a ton about how to understand what is and isn’t important, what works and what doesn't. I was part of a CTO team there, and we worked mainly on prototypes. It was really a great experience to be exposed to so many different projects and aspects.
How do you use metrics in your day-to-day?
Metrics and measurement are like a double-edged sword. I think we should measure almost everything we can. And we measure a ton of metrics… like how much stuff costs. We have a big AWS bill. So we have a really detailed measurement of that. We have a measurement of how customers adopt features - how, what do they use? What don’t they use? We measure the traffic and the scale in our system and availability and everything you would think from the engineering aspect.
And that's great, but for some cases it's so many trees, you can't see the forest.You also need to be very careful that you really understand what the metrics and measurements mean. We use it mainly as a signal. No single metric is like the ‘golden one’ that will be the deciding factor. Everything is multidimensional.
We are very careful about the KPIs we decide on, because if you give people a KPI, they'll optimize for that. And that could create some weird optimizations and not exactly what you're looking for. So we use it more in a qualitative way, not necessarily quantitative. We actually say that common sense is not that common. So even if you see a ton of metrics, it’s still: does this even make sense? Because a lot of the time, metrics are meaningless. Sometimes, they're a mistake and it's very hard to tell. And we live in a time where there's so much information and everything has a million metrics. I have a hundred dashboards I can look at and they have numbers and trends and definitely going up and going down, you really have to find the root cause and understand what's going on. And correlation is not necessarily causality.
Basically, common sense is the best answer to everything.
What is your advice for transitioning from engineer to team lead?
The first leadership or management role you transition into from an engineer is probably the hardest one. A lot of the time you don't understand that it's not only a promotion. It's not like becoming a senior engineer instead of a junior engineer. It's a profession change and you need to do things that are different from what you did as an engineer. You're not necessarily measured on the amount of deliveries you personally deliver and their quality [as before].
It's a more complex role. It's very hard for many engineers. A lot of the time the companies will say, “Here's our best engineer, let's make him the team lead.” That's not necessarily the best idea because it's very important for people who are starting engineering management roles to go and talk to a couple of the people who do the same thing in your company or in a different company and hear their experience.
In Appsflyer we're doing much better now in facilitating this - we have a set training that lets people transition into a team lead role, and we have a team lead community and there's really more of a supporting net allowing them to do this transition.
For people thinking about it, the two most important things are to understand it's a profession change, and talk to people who've done this transition and think, is this really for me?
How will R&D teams evolve in the next 5 years?
Forecasting the future is very hard to do. If we take the past and try to forecast the future, teams are becoming more and more diverse and vertical. Fifteen years ago you had the development team, the quality team, the design team, the product team. Now, you want to have all these people on the same team trying to use all their skills and different disciplines to get to the same goal.
Hopefully, each team has a logical scope or something that makes sense and most of the time run autonomously towards their goal. This will probably continue and we'll see even more disciplines combined into the same teams, maybe even support this dev, like mini companies within companies.
The downside is, at least from the engineer's standpoint, that your life is much more complex than it used to be. Ten years ago, you wrote your C or C++ code. If it compiled and you managed to build the thing, it would, 99% of the time, work fine in production.
Now our engineers need to understand about 15 different services and tools, some of which are big obstructions behind the scenes, and they have no idea how it works and they're not expected to understand how it works, but it makes their lives very difficult. They need to understand a little bit about design, a little bit about product, a little bit about the business because all of these people are their interface and they need to have a common language with them. It's good for the team. I see the teams becoming more diverse but it makes everything more complex.
I find engineering teams or teams in general are like a smart organism. Teams are already recognizing that complexity is the problem. And I think that the teams will find a way to solve that problem or at least mitigate it. I'm a great believer in that if you let the team decide - give them a goal - they'll find their way there. They'll navigate between the problems.
What do you wish could be measured?
There's a lot that can't be measured and I wish it could be. In leadership and management, I look at something, I see a problem, and I think if I engineer the organization in some way I'll solve the problem or mitigate it - and then we create a different set of problems. So can we measure the impact of organizational change?
A lot of the time companies grow big and bigger - can we measure the impact of that, the positive or negative effect of that? It's very hard to do. I'm not sure it's possible.
In general measuring the impact of something: if we had a very exact formula that says, take the value, divide by effort, X the income it creates for the company - that's the impact of the fit. I would love to have that because that would make a lot of discussions very easy, right? Product side, that the company says we have to do this by then. And you tell them: why, what is the impact of that? There's a lot of sentences that start with: I think, I believe, a customer asks that we’ll do something, or even worse, we have a lot of something, where 'a lot' as a number is not 10, 20, 100, million. I don't know what 'a lot' is. So that's a lot of the discussions I would like to not have and instead say, here we have a magic formula: The impact is 517, and this is 403. So this wins, but we don't have that.
From a personal perspective, I’d like to have something more measurable about parenting because you're always thinking, Am I doing the right thing by letting this go or insisting or something. I find a lot of common lines between parenting and leadership. Both sometimes giving you the biggest of joys and the worst of frustrations.
What startup solution do you wish you had?
Going back to the complexity problem, if there was a startup solution that would take away some of that complexity without hiding away too many of the controls, I would be very happy to at least try it out.
It's very hard now to add another tool to an existing company's tech stack. You have to see that there's value and that it's enough value that the team should know another tool. Before you reduce complexity you add complexity because, hey there's another tool you need to use.
Or if there was something that engineers didn't need to know about and it'll take away some of the complexity and increase their productivity then I would be very happy to try it out.
The most scarce resource we have is developer time. And it's obviously not optimally used for a million reasons but even if I got like 10% more development time - for every 10 people I ‘hired’ another developer - which in the economic weather now is getting harder to do. But by being more effective with developer time, I get the value of ‘another developer’ without training. It's just the same people, but they have more context and are more productive.
That would be amazing. If I had a good idea I would start that startup myself.
What’s a mistake founders make when pitching you?
When somebody talks to me about their idea or their product, then honesty is the best policy. I know you're a 3-person startup or a 10-person startup. It's okay to say, we don't know. We don't have a solution for that. We only support this and this and that. It's in our roadmap, but frankly I'm not sure we're gonna get there.
A lot of the time you only have one chance to integrate. If you think it's not a good fit, it's better to say, okay let's revisit this six months from now or a year from now - when we think it will bring value. Instead of integrating, frustrating both sides. Getting a second chance is much harder than getting the first.
