If you don’t understand it you can’t manage it
So if I have a “AI and ML Platform Product Manager” in my title, I’m imagining that people are going to ask me about how to be a PM in the AI and ML space. Wrong. Somehow no one ever asked me. I get a lot of questions on product management in general and on how to become a product manager, however, no one ever asked me “what does it mean to be a product manager of X”, as if the specific product you are managing doesn’t really matter once you are a product manager. Some people believe that if you are a good product manager, you will be a good product manager regardless of which product or domain you’re operating in. I couldn’t disagree more.
In the past I’ve been in a room with other product managers who I reputed good at their job, each in a different domain and on very different products, and I felt like we were all talking different languages. It’s not that we were not agreeing or not being constructive. We were approaching problems from different universes and there was no common ground of understanding.
The domain or product that you tackle requires different expertises and different styles of product management. In this article I’ll cover one of those that I’ve handled during my career and that is particularly hot at the moment (May 2026): AI and Machine Learning.
Quick note: there is a section later on the differences between PMing ML and AI. I’m not going to go through the technical differences of ML and AI, so that’s a pre-read you can do, since you will need to know that and an LLM is likely better at explaining that in a way that is tailored to your knowledge. Given you’re at it, also look up the difference between:
- Structured and unstructured data
- Online and offline models
How do you know if your role is an AI or ML Product Manager?
If you have used an LLM once, that doesn't make you an AI or ML product manager. That depends on the scope of your team. There are usually two types of AI or ML teams in companies:
- Platform AI or ML teams
- Applied AI or ML teams
For small companies, these two teams can be a single one (I’ll cover it in the applied section).
The big caveat here is that this distinction is true for companies for which the main business is not developing frontier models or new machine learning packages. For companies such as OpenAI or Anthropic, that develop the models that everyone else uses, there will be a lot more AI and ML teams with associated PMs.
Most of the companies of the world though are not research companies, so I will ignore the research company type and focus on all the others.
Platform teams
An ML or AI platform team serves other teams inside (or outside) the company with the functionality that they need to run AI or ML, ideally with the lowest friction possible for the model development lifecycle (check out the other articles on platforms). The major focus of this team is usually developer experience and infrastructure. They want to make it easy for developers to set up, train and run machine learning or AI models, ideally by abstracting all the complexity in setting up the data, infrastructure and the systems needed to run models optimally.
In short, they serve Applied AI or ML teams.
Applied teams
These teams “apply” AI or ML in trying to solve a specific problem. Their work is usually comprised of feature / context engineering (curating, selecting and optimizing the data used by the model), model selection and training, model deployment, monitoring / evals and retraining.
In a company you don't necessarily need a platform team to enable an applied team. In small companies, the ML / AI team is supposed to do both, then as the company matures, it makes more sense to have a center of excellence platform that then allows applied teams to move and iterate faster.
In order for a team to be an applied team, model development should be the bulk of their work. Also, the model they’re working on should be AI or Machine Learning based. A group of analysts that have a complicated excel sheet to set prices is not an applied team (though could likely be helped by an applied ML team). Likewise, a team that does mostly other tasks and that occasionally uses an ML model or an LLM to solve a problem is not an applied ML or AI team.
Data and Objectives
Why is it important to define whether a team is an ML or AI team? Because, for that type of team, the responsibility of the product manager is very simple:
- (input) You have to define and expand the data available to the team
- (output) You have to set the measurable objectives (aka objective function, loss function etc) of the team and the models
That’s it.
It’s not easy by the way. But any energy spent elsewhere is not going to contribute meaningfully to the team.
Data and objectives is also a language that a highly technical team will understand. I’ve seen plenty of cases of non technical PMs that were assigned to a highly technical and skilled ML team and failed because they simply couldn’t communicate effectively with the team. Let’s see why that is.
Data: garbage in, garbage out
In the age of AI, data is everything. As we will see in a few sections, AI and LLMs are great at knowing all the possible model types and configurations. However, they are less great at finding the data that can help the model make a decision. They are even less good at identifying data that your company should have, but somehow never collected.
It doesn’t matter how complex or cutting edge your model architecture is. If you don’t give it the right data, it won’t be able to make good decisions.
And this is where the product manager really enhances an ML or AI team.
I’ve noticed that teams of engineers or data scientists tend to use the data that has always been there. Every time a new engineer or scientist gets onboarded, they get briefed about the data that the team currently uses and, usually, they never ask themself about what other data there could be, or what other data should exist. Usually because they much prefer to do modeling work rather than data collection work, so the topic of data collection and organization is politely avoided.
That’s where the Product Manager shines. When you join an ML or AI team and find a problem to solve with a model, you should think about:
- What data is currently available and the team is using
- What data is available at the company but the team is not using
- What data should be available at the company, but somehow the company is not collecting
Without a PM, teams usually do (1) and, at best (2). The real value that a product manager brings to the team is to think hard about which data points would help the model and then go out and make them available to the team.
Usually (2) means that the product manager has to inform the team about the existence of the data and usually build relationships with the teams owning the data to make sure the model can use it seamlessly. On (3) instead the PM has to go out and make sure the data is collected or bought! Whether it is because the company never logged it or because the retention policy is too short, a product manager of an ML or AI team has to be the biggest advocate for collection, storage and sharing of data inside the company.
Engineers will say that a few very well selected data points are better than a lot of random data points. That is true, though in the age of AI, AI can usually figure out the best data points out of too many data points. In the past that would require a lot of time from machine learning engineers, though with AI it is easier (more on autoresearch later). The important part is that data should be as diverse as possible, and that is what the product manager should be mostly focusing on.
Objectives: if you don’t know what you want, an algorithm won’t know either
Once you have a lot of amazing data, you have to put it to use! A ML or AI model can put it to use, the main question though becomes… for what? And that’s where objectives come in. Or rather, objective.
The first thing is that a model really can only optimize for one thing. In most cases it is a number, calculated by an “objective function” (can also be called loss function). Given the world is non-linear, there are usually multiple things that a model has to achieve. However, all of those have to be somehow coalesced into a single value.
Many problems are so called “multi objective” problems, however, at some point in the model process, the objectives do have to be compared with each other. If you don’t consciously tackle that, some automated package or process will, outputting results that might not be the ones that you are looking for. Hence the importance of the objective function.
The product manager is the main person inside the team that should be aware of the business context and of what the company really cares about. Hence, the PM has to make sure that the objective function does indeed represent business value, and that the model is optimizing something that is worth optimizing for. You will work with machine learning or AI engineers or data scientists on it, though it is important for the PM to be very present in the process and provide continuous and constant feedback, until a satisfactory objective is agreed to.
I’ve seen a lot of product managers or even teams fail because they request too vague or high level objectives, which are not deterministic and not clearly measurable. An objective has to be very deterministic. It has to answer the question “did it happen or did it not happen”, or “what is the specific value, given a specific time and location”.
The objective has to be measurable according to your model cadence and decision unit. If you run a model once a month, the objective should be measured once a month. If a model runs on every app open, the objective has to be measured every app open (at app close, if the unit of decision is the app session). Usually the sequence should be:
- The user triggers the decision
- The model makes a decision
- You wait X time for the decision to have its effects
- You measure the objective
- Now you know what objective value is associated to that specific decision and can learn from it
For example “customer satisfaction” is not a great objective. Do you measure it through surveys? Every how often? Do all your customers complete a survey? That is one which is a nightmare to measure and optimize for.
In the context of Lyft, whether an app open resulted in a ride is a very specific and measurable objective.
For every app open, it is very clear, after the user has finished their app session, whether the user has taken a ride or not. We have specific definitions for what counts as a successful ride (which takes into account things such as cancellations, lack of drivers etc), so that we can clearly label each app session.
An even better objective that we usually use is the amount of money that a rider has spent in the app during the session. Once the session finishes, it is very easy to measure that and tag the session with the value.
This helps us understand whether the decisions that we took in the session contributed to an increase in the overall spend or not.
Everything else: autoresearch and the black box
Once you have a great set of data and the measurable objective, then the model can do its thing. From the perspective of a product manager, I consider the model a black box. You don't need to know what's inside of it. With the latest (as of May 2026) advances in artificial intelligence, there is an approach which is meant specifically to tune what is inside the black box. It is known as “auto research”. You effectively tell an LLM what the data to be used is and what the objective you want to optimize for. And the LLM will try a bunch of models, tune them, measure their result and ultimately spit out the best model that it has found and tune it for your problem. LLMs are already trained with all the latest paper and research on models, so likely it will know more than many people out there (albeit it might take a while and a good amount of tokens to try out all the options and find out which one works for your problem).
If you then place auto research in the hands of a knowledgeable machine learning engineer or scientist, you will get the absolute best results, given auto research performs better when it is actually given ideas for models or techniques to try.
That’s the black box. Throw data and objectives at your ML or AI team and they will take care of the rest. Your job is done.
Differences between PMing ML and AI (LLMs in particular)
Up until now I have mostly considered AI and ML similar. Whereas that is true in some regards, in this section I’ll cover the practical differences between the two.
Data: structured vs context
In ML, usually the most effective way to structure the input data to a model is through traditional structured tables. The data type, format and cardinality is what, as a PM, you’ll be looking to define together with your engineers. There are a lot more advanced ways to structure data beyond traditional SQL tables (for training), however, from what I’ve seen, most companies still use that format and that is going to be mostly enough for those use cases. Even when you get to embeddings or semantic IDs, the input to creating those often does go back to structured data tables. If the origin of your data is unstructured, you’ll usually want to have an aggregation / structure layer before feeding it to models, given a lot of the existing packages (that your autoresearch layer would try out) handle structured data better, or have some pre-processing function which takes structured data as an input.
For LLMs instead, given they (as of May 2026) process text better, you will want to have unstructured data. LLMs are NOT deterministic, meaning that if you pass them the same set of data, they will most likely provide different (albeit similar) results every time. For LLMs, think of input data more as “context”. You are providing the LLM all the information it needs to make a decision on what to answer. The “real” input to an LLM is a string of text, explaining the situation. If you have data that is structured, you will want to figure out a way to pass it to the LLM efficiently. This can either be in a JSON style file where each data point comes with its description, or you can even pre-summarize all the data and put it in a paragraph format. A usual tradeoff that you have to understand also is what you put in the “prompt” to the LLM (the set of instructions that you pre configure to apply to each request to it) and what you put in the data description. If a data point is always going to be passed to the LLM, you might as well describe it in the prompt.
Data: when more is more and when less is more
For ML, a key part of the model development is feature selection. That is a process which identifies all the data worth keeping and discards all the data that you don’t need. A lot of data scientists spend a lot of time selecting data features before training the model. As a PM, for ML I find it a lot more efficient in trying to collect as much data as possible upfront, then letting the feature selection in the training process tell me which data points are relevant (keeping an eye out for overfitting and collinearity). So for ML start with more data and remove as you go.
In LLMs instead, the LLM will use all the data that you provide it all the time and it is not as straightforward to figure out what is helpful vs what isn’t. LLMs (as of May 2026) actually tend to try and forcefully use all the context you provide it. The only way to find out what the right context is, is to try out different evaluation runs with different context, which is expensive. So for LLMs I suggest to start with less context and increase it as you go if you don’t see the performance that you want.
Measuring the objective
The evaluation for ML is easy. You define the unit of decision making (the rider app session for a rideshare app for example) and then you measure your objective at the end of each of those. Given models usually spit out a number or a defined artifact, the evaluation on what the model did is fairly straightforward.
For LLMs, given the output is usually text, the evaluation is more murky. There’s a lot of ways to word pretty much the same meaning. How do you evaluate the meaning?
The standard for evaluating objectives is the “eval / evals”. Evals are a set of “tests” that the Product Manager configures, to which the LLM is subjected to and scored based on the output.
There’s offline evals, which you run your system through usually before launch, every time you change the LLM model and periodically (just to check how things are going).
Then there’s online evals, which you run on select example runs of the model as they happen, to check how your model is doing in production.
There are different types of evals:
- Code based
- LLM as a judge
- Human score based
For offline evals, you’ll need an evals dataset. That is usually structured as a set of inputs to the model. Each input is then run through the model independently. Depending on the eval type, each run is scored differently.
- Code based evals are usually a set of functions that take the output and score it, either boolean (easiest) or with a numeric scale (harder to define exactly how it should scale). Then you would have a way to get the score of each function and merge them all into a single score per eval line. The functions can either be NLP (natural language processing) on the output or take a look at the intermediate processes that the model does in order to get to the answer (for example, calling a specific tool). The functions represent what you care about in the response. Example functions include:
- Seeing whether the LLM returns a response with particular words, such as swear words. This would score high if the words are not present in the response
- Seeing whether the LLM calls the right tools. For example, if the LLM needs to return the weather forecast, you’d imagine that it has to get some kind of weather related information. That likely is a tool call that you’d expect to see in those evals
- LLM as a judge means that you write a prompt explaining an LLM how to score the output of… the other LLM. You’re essentially using an LLM to check an LLM. Research is still being done around it, though my sense is that it is better to use a different model from the one you are trying to check. There was a paper that described how a model tends to agree with itself, so better critique of the answer is obtained when a different model checks the other’s outputs. So use model B to check model A. The LLM as a judge prompt is usually the same for each eval, and should include all of the criteria that you would consider for what a “good” response is. It can either provide a single result, or multiple scores for each eval, representing different ways to evaluate the message (brand voice, style, grammatical correctness, accuracy in response…). This one is easier to do as a product manager, since you don’t have to have any technical capabilities, though it requires more manual quality assurance, to make sure that the judge is working as expected. When using this, I usually check the first few runs manually and that lets me realize what parts are missing both in the model prompt and the judge prompt. A key learning here is that, given it is probabilistic, it doesn’t necessarily return the same results every time
- Human score. This is the good old “have someone go through the whole list”. It is analogous to the LLM as a judge use case, except that you have a human giving the scores instead of another LLM. You would provide a rubric with the types of scores you want the person to judge the responses on, then every time you do an iteration of the product or the model, you get someone to go through the responses again. The pros of this method vs LLM as a judge, is that the human can likely provide you a summary of where the system doesn’t work, usually better than an LLM, and can do so as development is still in progress (without having to wait for the testing stage). The downside is that you can run it less often because it is slower and it costs more. So you have to be strategic
I put the responsibility to write evals squarely in the product manager’s responsibilities. In some of the teams at Lyft, I notice that engineers or data scientists write the evals, with product managers outlining the major themes to cover and engineers writing the evals sets, usually with AI. The product manager should be responsible for the functionality of the product and hence the product manager should be responsible to score the completeness of it, much like the PM is responsible for the objective in ML based products.
Should AI write evals?
This is one of the key questions which I don’t think people have spent enough time thinking about. Writing evals is boring. You want to make sure they cover all the potential uses of the product, all the edge cases. There are usually hundreds of hundreds of evals for even simple LLM based products. The more you identify the better the product should be (in theory, though I’ve also seen that in practice). Given writing and testing evals is a laborious process, most people tend to use AI to write evals. They provide the LLM the major cases they want tested and then ask LLM to write detailed evals to make sure that all the cases are covered.
For me that is a critical mistake.
It is faster, for sure, and it does cover the first type of things that would usually go wrong, however AI written evals are usually very shallow and concentrated on the specific cases around which the original prompt was written.
To get awesome evals, I’ve found that the human touch is still superior, which can then be enhanced with AI review (and formatting, which should for sure be done by LLMs).
The order of preference of how I’d like to do evals is:
- Human writes all evals
- Human writes a good set of evals, then asks AI to extend them and to fill in the gaps
- Human asks AI to write evals, then adds more manually, then asks AI to fill in the gaps
In all cases, the process for coming up with evals should be:
- Write evals
- Ask AI to review them and critique them
- Do edits and review every single one of them, ideally after some time of writing to face them with a fresh mind
The success of (1) is directly proportional to the skill of the product manager. (2) can work, however usually grows the eval count exponentially. (3) has the same drawbacks of (2), though also results in some aspects being completely missed, since the human gets anchored by AI on the topics to check for.
How to be a great AI or ML product manager
Seems fairly easy based on what I wrote above, but it is actually not.
In order to be an effective product manager in this realm, you do need to have experience running models. You do need to understand the nuances of how similar data points don’t necessarily help the model, on how designing the train / test split is critical in avoiding overfitting and on the order of magnitude in performance across different models or just different data points.
You need to understand the infrastructure aspects of the model and how speed in model training and execution drastically improves your iteration capabilities, which usually result in better models. You need to understand which models benefit from GPU and which models don’t.
Given the inherent probabilistic nature of machine learning and AI, theory can only get you so far at some point, trial and error works better compared to analysis.
So ultimately the best way to become good is to roll up your sleeves and do some on-the-side model development. Kaggle is a great place to find preset datasets and problems that you can play around with. Learning even just a little bit of it will earn you a lot of trust and respect for your team other than will enable you to call out the right things when planning for a model. It will also really help you understand how to write an objective.
Platform side
Working yourself through a few models will also help you to understand what a good machine learning or AI platform needs. When you are learning by yourself, you will most likely not have access to any platform. And hence, you will have to do most of the work yourself.
Once you get in a work setting, you should first of all familiarize yourself with any tools or platform that your company already has. Then shadow some existing machine learning or AI engineers to understand their workflows. Benchmarking those versus best practices (that you can find online) will usually make it clear as to what is needed next for the platform.
There will be a follow-on article on how to be a great platform product manager, so I’ll defer to that for platform product management best practices that transcend ML and AI.
Common mistakes for product managers working with AI and ML
No, in 90% of cases you don’t need to reinvent the wheel
Engineers in most cases want to do cool things. And lately the cool thing is to fine tune and retrain your own LLMs or complex AI models. What that means is essentially to take an off-the-shelf LLM and provide it with more custom data for the company to make it better at tackling a specific problem.
I find that usually it is a step too far, especially given the huge amount of impact that can be driven by using plain off-the-shelf models or known machine learning packages. Fine-tuning LLMs usually takes a lot of infrastructure and cost, which doesn't really justify it. Engineers do like to prey on inexperienced product managers, trying to sell them the vision of needing to fine tune LLMs, mostly because they want to have that on their resume. It is usually overkill and similar results can be achieved by much faster and cost efficient ways.
The one case where that makes sense is if you have a very clear high volume and high critical use case for which an LLM or AI algorithm has been proven effective and hence it makes sense, for cost reason, to transition to an open source fine tuned LLM instead of using some from Claude, OpenAI etc (which usually cost more).
However, what many people fail to account for is the fact that the frontier models are always progressing, whereas once you fine tune your LLM, in order to keep it up to the frontier, you also have to continuously reinvest in it. It is usually better just to count on off-the-shelf hosted models, especially if you want to leverage the latest improvements.
Buying data is an option, but not the best option
This is a common one regarding the data. Sometimes companies don't have all the data that they want available and hence business development folks are quick to suggest the path of buying data. In itself, it is not a bad idea. Usually, buying data gives you access to data points that the company cannot collect organically. However, purchased data usually comes with a lot of caveats and legal restrictions, other than it usually is not applicable to all your decision points. If you're buying data on your users, you most likely will not be able to buy data on all your users. If you’re buying data on areas in which you operate, you most likely will not be able to buy data on all the regions in which you operate. This leaves gaps in the data which you need to solve through data augmentation.
So I do usually suggest trying to 100% leverage the data that your company can directly collect before resorting to buying third-party data. Third-party data looks easy to come by but it actually takes a long time to figure out on the legal and contractual side. And you usually have to renew contracts every one to a few years, which adds risk to your model (if you lose access to a data source, the model will either go stale or require retraining with alternative data).
Yes, you do need to understand ML and AI
Going back to the previous section on how to be a great AI and ML product manager, I reiterate that yes you do need to understand both the theory and the practice, at least at a basic level. I have seen way too many PMs fail at the job because they thought that they could just write a user-story-based PRD and the team will take care of the rest.
Results first, optimization later
The product manager must be the engine of the team, pushing for fast results. Often very good machine learning engineers and data scientists get caught in premature optimization, looking for an efficient solution other than an accurate solution. I’ve seen a lot of time spent trying to optimize a model before realizing that there were major issues in either the data, the objective or the overall approach.
As the PM, the fastest way to get to a good model is to make sure that the approach is validated and produces good results first. Initially, it doesn’t matter if the latency is too long or if the cost is too high. Get a model that works offline (meaning, in a testing setting) and that can then be optimized to make it economically and latency wise viable. If you can’t even get a good enough model without latency and cost constraints, there is no point aiming to optimize for those.
At NVIDIA, that is the concept of “speed of light”. You aim to optimize one thing (accuracy / performance of the model) and assume that everything else is going to work, even though it is not guaranteed. Once you get to an acceptable level of that, you can go and optimize all the other aspects.
Rules are ok to get started
A colleague at Lyft once told me “if manual rules don’t bring some incremental impact, then a model won’t either”. Models ultimately automate decisions. If you can’t start to improve the decision with manually defined and human intelligible static rules, then tackling the problem with a more advanced model will likely not work.
Applied teams see static rules as the enemy: the team is there to replace them with more advanced and capable models. As the PM you will have to fight hard to start by trying out a rules based solution, but it does pay off. If you can’t identify a few static rules to try and address the question, then you likely won’t be able to identify the input data that can affect the decision.
Rules are good also to make sure that there is the right infrastructure around the decision. Ultimately a model replaces the rule, though the part on providing the data to the model and measuring the outcome of the decision is the same regardless of model or rule.
Rules should be easy to set up, so before embarking on a lengthy modeling project, try out a rule and see if you see improvement in your objective. If you do, then transitioning to a model may yield even more success!
In conclusion
It is all about the input and the objectives. Once you are very clear to your team about those, they will surprise you with models that look magical.
Without clarity on those, you will get very long projects which don’t do as good as simple rules.