Why is AI Platform Team Hard to Succeed & Escaping the Build Trap Reading Notes

by Melissa Perri reading notes & reflections v0.1

It has been a while since I worked on a user facing product. I have “proudly” worked on various infrastructure pieces of stack for the majority of my past few years. There had been this myth in my mind that infrastructure teams have the priviledge to operate with great autonomy since we are not constrainted by Product Managers. Looking back, that is a huge mistake and it is a misconception that still exists in most “infrastructure” teams.

It is probably right in some internal infra teams, like Networking team, all they need is a spec from users like: bisectional bandwidth, # of machines, etc. From there, the Networking team can design the data center topology to meet the requirements. In other words, the customer contract is clear and well established.

However, that is not the case for a lot of other infrastructure teams, for example, a Big Data Infrastructure team or an AI infrastructure team. In this post, we will focus more on the AI infrastructure team.

First off, the core of AI infra is not its infrastructure, but its product provided to the AI users. I would rename any AI infra team to AI Platform Team, or AI Service Team. There are many infra pieces in the AI Platform org, for example, some compute engines like TensorFlow/PyTorch etc. that uniquely serves the customers from a scalability perspective, however, most of the stack are tasked to solve a yet standardized user problem — how to make the AI developers (or product engineers) more productive building AI models with the platform?

AI platform is more a product problem than an infrastructure problem

With the above hypothesis, let’s discuss why most AI platform solutions/teams fail:
Hard to build the right team
In a typical product team, you have role clarity at job title level. You have a Product Manager responsible for the product direction, an Engineering Manager for typical management tasks, and a TL responsible for the technical direction.
What is the challenge for an AI platform team? There is no Product Manager , and that becomes a natural conflict between the manager and the tech lead. Usually the problem gets back to the identity of the team, is the team a product team or an infra team. When the team lacks a product manager, who is responsible for the direction of this platform, is this a technical direction or a product direction? Who owns that?

Hard to set the right culture
Many infrastructure engineers are zealous about technologies (including myself), and less enthusiastic about the customers they serve. Infrastructure teams also celebrate deep dive and solving complicated problems than solving user problems. This natural tendency leads to a team of technology specialist than product generalists — it’s incredibly challenging for one person to acquire both technical depth and product sensitivity. It is a challenge for the AI platform to balance the culture between deep dive & customer obsession.

Because of the above challenges, it is not uncommon to see AI platform teams running a product death cycle:

• Customers don’t like our AI platform.
• We ask users for what is missing.
• We build more features.
• Customers still don’t like our product.
• …

With these challenges in mind, it is time to discuss how to escape the Build Trap by reading through the book!

Besides the insights from the book, the book is also very fun to read. It describes a fictitious (based on real scenario) which ran into the build trap, and how the author worked with the leadership team to salvage the company.

Excerpts

Part I The Build Trap

The build trap is when organizations become stuck measuring their success by outputs rather than outcomes. It’s when they focus more on shipping and developing features rather than on he actual value those things produce.

I continued:

• Everything is number one on your project list. You are peanut-buttering your strategy — meaning that you have so many strategic initiatives spread over very few people.
• You tie people’s bonuses to shipping software, not to solving problems.
• You need to build up a proper product management organization that can explore how to achieve value for the business. This is a specialized skill set.

The build trap is a terrifying place for companies because it distracts them. Everyone is so focused on shipping more software that they lose sight of what is important: producing value for customers, hitting business goals, and innovating against competitors.

The Value Exchange System

Products and services are not inherently valuable. It’s what they do for the customer or user that has the value — solving a problem, for example, or fulfilling a desire or need. Doing this repeatedly and reliably is what guides a company to success.

…(the data platform) had a total of 30 features, with about 40 more on the backlog, people used only 2% of them consistently. And yet, development was underway to add more, instead of trying to reevaluate what they already had.

Constraints on the Value Exchange System

To realize the maximum value, organizations need to have the right individuals, the right processes, the right policies, the right strategy, and the right culture.

Yet most companies I encounter are stuck in output mode, and their entire organization Is optimized to increase the output.

To be strategic and to have people operate strategically, we need to stop judging teams based on the quantity of feature shipped. We should instead define and measure value and then celebrate them for delivering on outcomes for our business and users.

Projects Versus Products Versus Services

Products are vehicles of value. They deliver value repeatedly to customers and users, without requiring the company to build something new every time.

Services unlike products, use human labor to primarily deliver value to the user. Service-based organizations are design agencies that create logos or brands for business.

Project is a discrete scope f work that has a particular aim. It usually has a deadline, milestones, and specific outputs that will be delivered. Projects are an essential part of product development, but the mentality of thinking only in projects will cause damage.

The Product-Led Organization

Product-led companies understand that the success of their products is the primary driver of growth and value for their company… many companies are, instead, led by sales, visionaries, or technology. All of these ways of organizing can land you in the build trap.

Sales-led
Sales lead companies let their contracts define their product strategy. The product roadmap and direction were driven by what was promised to customers, without aligning back to the overall strategy.
When you have 50–100 customers or more, you cannot build everything uniquely to match the needs of each one, unless you want to be a bespoke agency. You need to change your strategy to building features that apply to everyone, without customization.

Visionary-led
The easiest way to think of a visionary-led company is to consider Apple. Visionary-led companies can be very powerful — when you have the right visionary. But there aren’t too many Steve Jobses floating around.

Technology-led
These companies are driven by the latest and coolest technology. The problem is that they often suffer from a lack of a market-facing, value-led strategy.

Technology is critical to a software company’s success, but it cannot drive the product strategy.

Product-led
Product-led companies optimize for their business outcomes, align their product strategy to these goals, and then prioritize the most effective projects that will help develop those products into sustainable drivers of growth. You need to begin focusing on outcomes and to adopt an experimental mindset to eliminate the uncertainty that what you are building will reach your goals.

What We Know and What We Don’t

— — — — Known Unknown
Known Facts Questions
Unknown Intuition Discovery

When kicking off a project, it’s best to begin by identifying what you know to be true about the situation — your known knowns.

Known unknowns are clarified enough that you know which question to ask. You use discovery methods and experimentation to clarify these, turn them into facts, and build to satisfy those facts.

Unknown knowns are those moments when you say, “I feel like this is the right thing to do.” This is intuition form years of experience. Although we should all listen to our intuition, you should also be cautious because this is often where bias thrives. It is imperative to check and experiment to see whether your intuition is right.

The unknown unknowns are the things you don’t know you don’t know.

Product management is the domain of recognizing and investigating the known unknowns and of reducing the universe around the unknown unknowns.
Product managers identify features and products that will solve customer problems while achieving business goals.
Product managers are the ones who fit right in the middle and translate needs into a product that will satisfy the customer while sustaining and growing the business.
Product managers are the key to becoming product-led.

Part II The Role of the Product Manager

Product management is a career, not just a role you play on a team. The product manager deeply understands both the business and the customer to identify the right opportunities to produce value. They are responsible for synthesizing multiple pieces of data, including user analytics, customer feedback, market research, and stakeholder opinions, and then determining in which direction the team should move. They keep the team focused on the why — why are we building this product, and what outcome will it produce?

A great product manager must be able to interface with business, technology, and design departments and to hardness their collective knowledge.

Bad Product Manager Archetypes

Agile does indeed promote a better way of collaboration and a faster method of building software, but it largely ignores how to do effective product management.

Being a great product manager takes a thorough understanding of your users, a careful analysis of your systems, and an ability to see and execute on opportunities for your market.

The Mini-CEO
Product managers can’t change many of the things a CEO can in an organization. They especially don’t have authority over people, because they are not people managers at the team level. Instead, they need to rely on influencing them to actually move in a certain direction.

He reminded me that my job was to produce value, not develop my own ideas.

The Waiter
The waiter Is a product manager who, at heart, is an order taker. They go to their stakeholders, customers, or managers, ask for what they want, and turn those wants into a list of items to be developed. There is no goal. There is no vision. There i no decision making involved.

Product death cycle:

No one uses our product -> ask customers what features are missing -> build the missing features -> No one uses our product

The product death cycle is a specific form of the build trap. You are implementing ideas without validating them. It’s not the customer’s job to come up with their own solutions. That is your job. You need to deeply understand their problems and then determine the best solutions for them.

It is very possible to find the waiter type paired with another one, like Project Management. They are not focused on why, but focus a lot on when.

The Former Project Manager
Product managers are not project managers. Product managers are responsible for the why? Why are we building this? How does it deliver value to our customers? How does it help meet the goals of the business?

A Great Product Manager

An effective product manager must understand many sides of the company in order to do their job effectively. They need to understand the market and how the business works. They need to truly understand the vision and goals of the company. They also need deep empathy for the users for whom they are building products, to understand their needs.

One of the biggest misconceptions about the role of a product manager is that they own the entire product and therefore can tell everyone what to build. Act this way, and you will only alienate the rest of your team. Product managers really own the “why” of what they are building.

The product manager works with the rest of the team to develop the idea and then jumps in. They then work to identify the product vision, crafting it and communicating it, and then championing it. But, at the end of the day, it’s the team, collectively, that really owns the product — the WHAT.

Figuring out what to build takes a strategic and experimental approach. Product managers connect the dots.

One of the worst traits a product manager can have is the lone wolf mentality — the idea that they are the only one responsible for the success of their product. Great product managers understand that they will get further by taking advantage of the skills and expertise of their team.

One of the biggest mistake companies make in hiring a product manager is trying to find either a technical or market expert. Product managers are not experts in either of these domains; they are experts in product management.

The product manager carefully balances the line between all disciplines to be able to strategize and decide what is best for the product.

The Product Manager Career Path

Tactical work focuses on the shorter term actions of building features and getting them out the door.
Strategic work is about positioning the product and the company to win in the market and achieve goals. It looks at the future state of the product and the company. And what it will take to get there.
Operational work is about tying the strategy back to the tactical work. Here is where product managers create a roadmap that connects the current state of the product to the future state and that aligns the teams around the work.

Associate PM: mostly tactical, some operational, strategic.
Product manager/Sr PM: less tactical, more strategical and operational.
Director of Product: little tactical, 60 operational, 40 strategical.
VP of Product: no tactical, 40 operational, 50 strategical. Owns entire product line.
CPO: 20 operational, 80 strategical. Connects products.

Organizing Your Teams

A value stream is all of the activities needed to deliver value to the customer. That includes the processes, from discovering the problem, setting the goals, and conceiving of the idea, to delivering the actual product or service. It makes sense to organize your teams around the value stream.

Part III Strategy

A good strategy is not a plan; it’s a framework that helps you make decisions.
Executing better on the core mission is the way to win.

What Is Strategy

Thinking of strategy as a plan is what gets us into the build trap. Strategy is a deployable decision-making framework, enabling action to achieve desired outcomes, constrained by current capabilities, coherently aligned to the existing context.

Strategic Gaps

Outcomes -> Plans -> Actions
Knowledge Gap is between outcome -> plans. The difference between what we would like to know and what we actually know. Organizations try to fill this gap by providing and demanding more detailed information.

Alignment Gap is between plans -> actions. The difference between what we want people to do and what they actually do. Organizations try to fill this gap by providing more detailed instructions; whereas, instead, they should allow each level within the company to define how it will achieve the intent of the next level up.
Product exams need the freedom to explore solutions and to adjust their actions according to the data they receive. Instead of sending down mandates, organizations should, instead, turn to aligning every level of the company around the why and should give the new layer down the opportunity to figure out the how and report back.

Effects Gap is between Action -> Outcome. When organizations do not see the results they want, they try to fill this gap by putting more controls in place.

You scale an organization by enabling action through Autonomous Teams

Creating a Good Strategic Framework

A good company strategy should be made up of two parts: the operational framework, or how to keep day-to-day activities of a company moving; and the strategic framework, or how the company releases the vision through product and service development in the market.

Spotify’s DIBB framework: Data, Insights, Beliefs, and Bets.

Strate by Deployment
Executives are really good at telling five-year stories, but a team cannot act on a five-year story when they’re used to thinking in 2–4 weeks. There is too much space to explore.

Not having the right level of direction lands us in the build trap. Teams are given instructions that are either too prescriptive or too broad.

In most product organizations, there should be 4 major levels in strategy deployment:
Vision -> what do we want to be in 5–10 years? Value for customers, position in market, what our business looks like -> CEO/Senior leadership
Strategic intent -> what business challenges are standing in out way of reaching our vision -> Senior leadership/Business leads
Product initiative -> What problems can we address to tackle the challenge from a product perspective? -> Product leadership team
Options -> what are the different ways I can address those problems to reach my goals? -> Product dev teams

Strategy is about how you take the organization from where you currently and reach the vision.

Company-Level Vision and Strategic Intents

Company vision: A good mission explains why the company exists. A vision on the other hand, explains where the company is going based on that purpose. I find that the best things a company can do is to combine both the mission and the vision into one statement to provide the value proposition of the company — what the company does, why it does it, and how it wins doing that.
Strategic intents: Strategic intents are few concise, outcome-oriented goals focus the company around how to reach the vision. Strategic intents are always aligned to the current state of the business.

To understand how to set our strategic intents, we had to first understand what business value really means. A great model for thinking about business value:
Increase Revenue: Increasing sales to new or existing customers. Delighting or disrupting to increase market share and size.
Protect Revenue: Improvements and incremental innovation to sustain current market share and revenue figures.
Reduce Costs: Costs that we are currently incurring, that can be reduced. More efficient, improved margin or contribution.
Avoid Costs: Improvements to sustain current cost base. Costs we are not currently incurring but may do in the future.

Product Vision and Portfolio

Product initiatives translate the business goals into the problems that we will solve with our products. The product initiatives answer how? How can I reach these business goals by optimizing my products or building new ones?

The product vision communicates why you are building something and what the value proposition is for the customer.

Part IV Product Management Process

The best solutions are linked to real problems that users want solved. Product managers use a process to identify which of those problems the team can solve to further the business and achieve the strategy.

The Product Kata

Context Matters
It’s important to take a step back and understand where you are and what is needed in that stage before you jump into any work. After we have set the goals, we begin walking through the product Kata. We ask ourselves the following:

1. What is our goal?
2. Where are we now in relation to that goal?
3. What is the biggest problem or obstacle standing in the way of me reaching that goal?
4. How do I try to solve the problem?
5. What do I expect to happen? (Hypothesis)
6. What actually happened, and what did we learn?

Many times, they’re experiencing unnecessarily when the problem is not yet known or when there is already a good idea about the solution.

Don’t spend your time overdesigning and creating unique, innovative solutions for things that are not core to your value propositions. If someone has already solved that problem with a best practice, learn from that, implement their solutions,gather data and iterate. Reserve your time and energy for the things that will make or break your value propositions.

The fewer features, the better. That is how you reduce the complexity of products. Otherwise, you can quickly run into feature fatigue from customers. Remember, it’s about quality, not quantity.

Understanding the Direction and Setting Success Metrics

Product metrics tell you how healthy your product is, and, ultimately, your business, given that a healthy product contributes to overall health of the business.
Pirate Metrics
Acquisition: users find you
Activation: users’ first experience with your product.
Retention: means and rates of users returned.
Referral: users tell the others about you.
Revenue: the profit you gain.

Problem Exploration

Understanding the Problem
Product managers are often spoken about as the “voice of the customer,” yet too many of us are not getting out and talking to customers as much as we should. It’s essential that we all go talk to actual humans to get to the heart of their problems.
Problem-based user research is generative research, meaning that its purpose is to find the problems you want to solve. You’re trying to identify the pain point and the root cause of the problem.

It’s easy to fall into the trap of solving problems before you find their root causes. We’re all prone to problem solve, even if we don’t know what the problem is.

“Nobody wants to hear that their baby is ugly.”

Users Don’t Want an App
The company had rushed into building features for the sake of getting something out the door rather than trying to understand what its customers wanted and needed.

Remember, its not the customer’s job to solve their own problems. It’s your job to ask them the right questions.

Validating the Problem
Your website sucks, but that isn’t my biggest problem. It takes me so long to create a course because I have to learn how to edit videos and create videos that are engaging.

Solutions Exploration

Experimenting to Learn
The most importance piece of the MVP is the learning, which is why my definition has always been “the minimum amount of effort to learn.” This keeps us anchored on outcomes rather than outputs.
Due to the misconception of this term, I have stopped using MVP altogether. Instead, I talk more about solution experimentation.

Because these are not designed to be long-lasting solutions, you want to limit exposure to your customers.

3 ways to experiment:
CONCIERGE
Concierge experiments deliver the end result to your client manually, but they do not look like the final solution at all… it’s far faster and less expensive to iterate on a service than on a coded feature.

WIZARD OF OZ
The idea behind the Wizard of Oz is that, unlike the concierge experiment, it looks and feels like a real, finished product. Customers don’t know that, on the backend, its all manual. Someone is pulling the strings — just like the Wizard of Oz.

Companies are tempted to leave Wizard of Oz experiments up for a long time because they look real to the customers. This is not wise, because it is still manual on th backend. After you know which direction you wan tot go in, you can begin to think about the full solution or move on to other forms of experimentation.

CONCEPT TESTING
Concept testing is another solution experiment that focuses more on high-tough interaction with the customer. The idea here is to pitch the solution idea in the fastest, lightest way possible to convey the message.

Building and Optimizing Your Solution

Build < Partner < Buy
Evolving the Product Vision
A North Star document explains the product in a way that can be visualized by the entire team and company. This includes the problem it is solving, the proposed solution, the solution factors that matter for success, and the outcomes the product will result in.

Story mapping helps teams break down their work and align around goals.

Prioritizing Work
Cost of delay is a numeric value that describes the impact of time on the outcomes you hope to achieve. It combines urgency and value so that you can measure impact and prioritize what you should be doing first.

The Real Definition of Done
When teams create their definition of done, it’s usually around finishing building features required to ship a product. Although this is a definitely a useful concept to make sure the team finished what it needs to, it sets the wrong expectation that what a finished feature is.

We are DONE developing or iterating on a feature only when it has reached its goals. Often, teams shop a first version of the feature and then move on to the next, without measuring the outcomes for the user. “Version 2 is the biggest lie in software development.”

These are the marks of a product-led company. Process and frameworks get you only so far on your way to success. Culture, policies, and structure are the things that really set a company apart to thrive in product management.

Part V The Product-Led Organization

The product-led organization is characterized by a culture that understands and organizes around outcomes over outputs, including a company cadence that revolves around evaluating its strategy in accordance to meeting outcomes. In product-led organizations, people are rewarded for learning and achieving goals.

Without the organization to sustain it, the efforts are too little, too late.

Outcome-Focused Communication
If there is one main reason I have seen companies fail to make a transition, it’s the lack of leadership buy-in to move to an outcome-oriented company.

Cadences and Communication
The more leaders can understand where teams are, the more they will step back and let the teams execute.
Quarterly business reviews: review revenue for the quarter, churn of customers, and costs associated with development or operations.
Product initiative review: this is the place for product managers to talk about the results of preliminary experimentation, research, or first releases, as they relate to overall goals.
Release reviews: provide the opportunity for teams to show off the hard work they have done and to talk about success metrics.

Roadmaps and Sales Teams
I recommend aligning your company around certain terminology to determine stages of development so everyone understands which activities are happening. We use 4 phases:
Experiment: understand the problem and determine whether it’s worth solving. Conduct problem exploration, no production code.
Alpha: a minimum feature set or a robust solution experiment, but built in production code and live for a small set of users.
Beta: this phase is to determine whether the solution is scalable, from a technical standpoint.
GA: the solution is widely available to all of our clients.

Product Operations
The team is in charge of streamlining all operational and process work that product teams need to be successful. This includes:

• Create automated and streamlined ways to collect data on process toward goals and outcomes across teams.
• Report on goals, outcomes, roadmaps, progress, capacity, and costs across the product organization, translating these activities into financial implications for the company execs.
• Set up and maintain a product analysis platform to report on product engagement metrics across the organization.
• Standardize product processes that go across teams.
• Organize and run critical product meetings for strategy creation, strategy deployment, and releases.
• Conduct any coaching or training for the product teams.

The point of this team is not to dictate how the members of a team work together to build the product, but instead to create the criteria for inputs and outputs of the work.

Rewards and Incentives

The biggest issues I see with companies tying to transition to becoming product-led is that they don’t evaluate their current reward structures to make sure they incentivize the right behavior.

Even though it’s difficult to change many of the policies, if you don’t have the seniority, you can still try to change the minds of the people who can bring those messages up the chain.

Safety and Learning

I want to be clear: it is not a success if you fail and do not learn. Learning should be at the core of every product-led organization. It should be what drives us as an organization.

When you don’t have safety built in to your company, your product managers won’t fell comfortable trying something new. No one will.

Product managers should be risk mitigations who say: Hey I think the most costly thing we can do is build this product without knowing it’s the right product to build. How do I test it and ensure that this is the actually what we want? How do I become more confident that we’re on the right path before I invest money in this?

Instead of launching the product to everyone, start with a small representative population, learn from them, and then expand to more people as you feel more confident.

Customer Centricity

Jeff Bezos: The most important single thing is to focus obsessively on the customer. Our goal is to be earth’s most customer-centric company.

Marquetly: The Product Led Company

One of the biggest mistakes that companies make in these transitions is having leadership think that it’s everyone else’s job to change instead of theirs.

Afterword: Escaping the Build Trap to Become Product-Led

When I was first starting off as a product manager, I needed to learn about humility. I learned that my role was not that of the big idea generator but that of the bad idea terminator. I need to learn to be humble and to gain the support and buy-in of my team in order to make great products.

People will get in the way of a good product every time. Even if it is the best idea for the company, if it doesn’t meet the personal agendas of senior stakeholders, it can be squashed. To mitigate the risk, you need to deeply understand what motivates people and to know how you can address their personal motivations by introducing information and data that wins them over.
Being agile, being customer-centric — these things are already baked into their culture. They understand that the fundamental criterion for building a product is that the product solves a problem for a user. They do not just build things for the sake of checking boxes.

Appendix: Six Questions to Determine Whether a Company is Product-Led

Who came up with the last feature or product idea you built?
What was the last product you decided to kill?
What’s the last time you talked with your customers?
What is your goal?
What are you currently working on?
What are your product managers like?

• The dream organization for product people is one that sees product managers as leaders who help shape the direction of the company and the services they provide to their customers.