This post was originally published on this site.
Yobi is a behavioral AI company building foundation models that predict future behavior for ad tech, marketing, and more.
Connect with Frank via fportman.com or at yobi.ai.
Congrats to Hooked on winning a Populist badge for their answer to Removing whitespace around a saved image.
TRANSCRIPT
Ryan Donovan (00:01.103)
Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I’m Ryan Donovan, your host. And today we are talking about how LLMs fail on things that aren’t language, things like intent prediction. And my guest for that is Frank Portman, who is the CTO at Yobi. So welcome to the show, Frank. Of course. So before we get into the topic of the day,
Frank Portman (00:21.72)
Thank you for having me.
Ryan Donovan (00:27.993)
Let’s, we like to get to know our guests. How did you get into software and technology?
Frank Portman (00:32.832)
It’s a good question. So I’ve been doing this for a little while now, but actually way back in the day, I wanted to be a mathematician.
Ryan Donovan (00:41.627)
Close enough, right?
Frank Portman (00:42.894)
Close enough, and I think I have a very similar origin story to many people working in technology, which is that when I was very young, I liked computers, I liked video games, I liked to tinker, but I definitely wasn’t one of those people that was coding something at 13. I was just like, I knew how to open the terminal on my way.
Ryan Donovan (00:53.115)
Mm-hmm.
Frank Portman (01:03.134)
computer which was advanced but not you know not too crazy but I was like that’s cool technology is cool but I like math so I actually went to college for pure math
Ryan Donovan (01:06.031)
Mm-hmm.
Frank Portman (01:15.628)
Which, which I did and I finished and I still find it super interesting, but along the way we had to take a couple of courses like outside of just our direct major, which I think is pretty common. So I was like, okay, let’s do applied math. I’m not doing pure math. Specifically, let’s do the ones where we have to implement our own matrix multiplication and all that stuff and see. And suddenly I wasn’t the one that was asking people for help. was like.
Ryan Donovan (01:30.019)
Right, right.
Ryan Donovan (01:42.171)
Yeah. Yeah.
Frank Portman (01:42.988)
You know what I mean? someone is asking me for help. And I was like, maybe there’s something here because it’s always fun to kind of do what you’re good at. And I think it was just super fun for me to, know, I think of writing software and solving problems with software is just like a series of mini puzzles. Which is not, by the way, that’s not the software engineering job. I think that’s like the coding that you do is kind of like, software engineering job is all about like solving business problems, which is we can get into that.
Ryan Donovan (02:06.683)
Mmm.
Frank Portman (02:12.422)
but I founded challenging and rewarding, and this was right around the time data science and machine learning was becoming like a very hot buzzword. So, okay, despite no formal training or education here, I do have the math and stats backgrounds. seem to be good at coding thing. Let me try. And climbing into that was tough. Like you’re for, you know, nothing’s harder than getting your first job, I think. But ever since then, that’s, that’s what I’ve been doing.
Ryan Donovan (02:22.639)
Right.
Ryan Donovan (02:41.199)
Yeah, and obviously matrix multiplication is pretty hot these days, isn’t it?
Frank Portman (02:46.752)
It is pretty hot these days. That’s what everything comes down to these days.
Ryan Donovan (02:50.677)
Yeah. So talk about, you know, I’ve heard of folks using LLMs for a sort of generic abstracted next token prediction where the next token could be intent signals, could be steps in a process, could be, you know, behaviors in a chain. Why do you think that is a sort of lesser approach or a flawed approach?
Frank Portman (03:20.104)
I think there’s a couple of things and I definitely welcome you to challenge me on any of these points because I only see what we do and also I see kind what the industry does at a high level. So one, the nice thing about models is they’ll do exactly what you train them to do. Also, there’s also the worst thing about models.
Ryan Donovan (03:36.667)
Mm.
Frank Portman (03:39.454)
And it’s not clear to me why the inductive bias of let’s gather all the texts in the world and maybe all the video in the world get very good at predicting sort of sometimes very long sequences to be fair. and then maybe fine tuning it a bit so that it’s more, more pleasant or more correct in conversation, which is so subjective. It’s not clear to me that that inductive bias would make these LLMs good for.
like forecasting or prediction or decision making, but they are phenomenal at synthesizing information within your context. They’re phenomenal at like writing code.
they can do all sorts of emergent things such as writing rap in the form of Shakespeare. But decision-making under uncertainty is sort of like the thing around modeling intent and making predictions. And it’s not clear to me that the inductive bias of like, let’s just train to predict next token can just build that into existence. However, I do think those technologies such as LLMs in combination with the right tools.
Ryan Donovan (04:27.227)
Okay.
Frank Portman (04:49.826)
you could start to get there. And I think that’s why people are pretty excited about agentic everything versus fairly like the one LLMs rule them all, if that makes sense.
Ryan Donovan (04:54.562)
Yeah, yeah.
Ryan Donovan (04:59.257)
Yeah. Yeah, I think you mentioned it there. It’s like to get this sort of inductive bias, you need a lot of foundational data. I think where I’ve talked to people doing Next Token Prediction for intent is on very constrained use cases like job hiring or something. So you’re looking at a sort of more generalized intent prediction. Is that right?
Frank Portman (05:18.51)
Mm-hmm.
Frank Portman (05:27.374)
Yeah, the way I think about what we’re building and maybe I’ll give this bill now. So it’ll be, I think of us as a behavioral AI company, building a foundation model of behavior.
Ryan Donovan (05:35.323)
Okay.
Frank Portman (05:38.926)
which I think is different than a foundation model of, let’s say, text generation, video generation, image generation. There’s a couple of reasons why. I think the inductive biases that are largely different are the data that goes into it and the way we train the model. mean, maybe tautologically, those are like the two biggest ones of any model,
Ryan Donovan (05:59.169)
Yeah, yeah, yeah.
Frank Portman (06:00.334)
For us, those are the different ones. And the data that we train on is, is proprietary. It’s sensitive. We can talk about like why we’re making this bed given that a lot of people look at us and say, like, this is, this is too hard to do, but it’s proprietary. It’s sensitive. it’s, it’s oftentimes identifiable even to an anonymous ID, like an anonymous, you know, browser session or something like that. and the model we, it’s not always text. Sometimes it is text, but sometimes it’s just product A, which is.
again, something that we sometimes do for privacy considerations. And then the way we train it is we’re not literally trying to create a chatbot experience. We’re trying to create a base representation that we call a broadly predictive, a future behavior. And that’s our, that’s our variants of the foundation and foundation model. Meaning if it’s a broadly predictive, suddenly you can run an ad campaign for a product that was nowhere near a training data set. And it does well the same way that foundation for LLMs means
Clearly in history, there was never like meets Shakespeare in the training data. And yet it’s pretty good at that, right? That’s kind of the foundation emergence.
Ryan Donovan (07:04.325)
Sure. Yeah.
Ryan Donovan (07:09.593)
Yeah. So when you say behavior prediction, is that just limited to ads? Or do you got like a long tail, like minority report angle going?
Frank Portman (07:20.926)
It’s a good question. I’ll say no minority report, But we are not a ad tech company and my CEO would be very upset if I spent too much time talking about ads here. We are a behavioral AI company. We can talk about why we’re working in the ad space to start. I actually think that’s interesting because I’m very new to it. I mean, I’ve been here four years now, but in theory, I’m new to the ads world.
Ryan Donovan (07:25.797)
Good, good.
Frank Portman (07:46.498)
But we see ourselves going and building products anywhere a personalization decision or recommendation could be made. Ads is obviously a very economically viable place to play first.
Ryan Donovan (07:59.704)
I mean, the internet is built on ads. So it makes sense that that’s where you’re applying behavioral prediction. So if not LLMs, what sort of models are you working with?
Frank Portman (08:02.156)
Yes.
Frank Portman (08:13.25)
So at the end of the day, these architectures all tend to look very similar. Like Attention Is All You Need is the title of the paper from back in the day. And it’s kind of true. So yes, we use large scale transformers. We also use graph models because we have interesting questions around identity that language models don’t. For example, there’s all these anonymous identifiers that we want to be able to.
Ryan Donovan (08:22.157)
Mm-hmm. Right. Yeah.
Frank Portman (08:39.062)
connect in some way while still respecting privacy. So we have flavors of kind of graph neural networks in our stack, but also at the end of the day, attention is all you need, right? Scale transformer is the right training process, the right amount of electricity to be able to train all this. That’s the thing that I have to worry about now, stuff like that.
Ryan Donovan (08:51.052)
Right.
Ryan Donovan (08:54.587)
Yeah.
Ryan Donovan (09:01.143)
Sure. Yeah. So the training process, how does that differ from a transformer based on language?
Frank Portman (09:12.334)
The literal training process by the time we’re just running PyTorch on GPUs distributed in our data centers, very similar. We have a couple of interesting things, you know, feel free to cut me off if I’m going too deep into the rabbit hole, but in the graphical worlds or in the world where certain…
tokens are discrete and high cardinality, such as like all the websites or maybe like specific hashed user IDs and stuff like that. We have questions around inductive versus transductive models. And a inductive model can induct any new sort of node or row or whatever in a, in a satisfying way, I will say. And then a transductive model is the state that it’s trained with is all that it has a representation for. And then you kind of need heuristics to induct new representations in.
to it. And this is all I picked up from my time working on recommender systems in tech. But for us, it’s very applicable as well, because for, let’s say, the behavior side is maybe not rapidly changing. Like, yes, a new franchise opens up somewhere, and maybe we should get to including that in our model. But it’s not super important. But the user side changes quite a bit.
Like you’re doing new things, you’re diverging from some paths that you might have looked similar to this before, but now you’re taking it a different direction. And so we do have to be able to kind of new nodes relatively satisfyingly. And so we spend a lot of time working on architectures that are inductive and not just transductive.
Ryan Donovan (10:43.579)
Mm-hmm.
Frank Portman (10:52.364)
But LLMs are kind of inductive by definition. Every one of these transformer models, at the end of the day, somewhere inside them, I mean, maybe someone listening is going to be like, actually, I found the one. But at end of the day, somewhere inside them, there is some hookup table for the embeddings related to certain tokens.
Ryan Donovan (10:57.551)
Right, yeah.
Ryan Donovan (11:05.989)
Mm-hmm.
Frank Portman (11:14.398)
And that could be trainable, right? I’m not saying it’s frozen or fixed, but somewhere in there that exists. And in the language world, I think on average, it’s like maybe six, maybe seven digits, probably six, like hundreds of thousands of tokens. And the way that those are combined with transformers and all these contextually aware ways, it’s amazing that given 300,000, 500,000 base representations, like you can do what they’re doing now.
the number of behaviors in the world is like three orders of magnitude bigger. And that’s, and that’s basically how we cut it today. Like we can make it even higher.
Ryan Donovan (11:47.237)
Right.
Ryan Donovan (11:50.553)
Right. It seems like with language, you can sort of predict the various words that will come out. You have the embeddings. You have the sort of semantic map. But for behaviors, that’s, do you have the fallback to heuristics for that?
Frank Portman (12:10.382)
for running inference, for example, or…
Ryan Donovan (12:12.855)
Yeah, if you run into a sort of unknown behavior.
Frank Portman (12:16.164)
got it. So right now our models are transductive on the behavior side. We have ways that we can add new behaviors that are unseen, but it’s done in a sort of decoupled process from our core user representation side. It’s something we want to get to, but it’s also, I don’t want to call it unimportant, but we’re definitely hitting kind of diminishing returns on that.
Ryan Donovan (12:23.769)
Mm-hmm.
Frank Portman (12:41.506)
Because while there are a lot of behaviors, they’re not necessarily like churning all that much. Like yes, new websites get created, new whatever.
Ryan Donovan (12:46.863)
Right? Right. But the fundamental person doesn’t change. Like, humanity’s fundamental operating system doesn’t change very much.
Frank Portman (12:56.236)
Yeah, the state of things that they might do doesn’t change, but what they do changes a lot. And that’s why the product exists.
Ryan Donovan (13:00.443)
Yeah.
Ryan Donovan (13:04.247)
Yeah. So I want to get a speculative, because there’s been a lot of physical AI stuff. And this seems like it might have physical AI applications. Are you thinking about that, or are you sticking to the ads for now?
Frank Portman (13:20.438)
Is the physical AI do you mean sort of like robots, for example? I think we’re sticking to software and personalization and prediction. I do think the, the initial question you raised around why aren’t elements good at this? Like that’s what we see there is opportunity to be the decision layer, regardless of where.
Ryan Donovan (13:23.523)
Yeah, basically robots,
Ryan Donovan (13:42.085)
Mm-hmm.
Frank Portman (13:47.106)
this LLM company goes or that LLM company goes or whether it’s like.
Maybe you don’t need some agents. Maybe all you need is tools. Like think that the proprietary data and the proprietary way that we build products on top of it can be kind of the decision forecasting inference layer. and then the actual product experience, whether it’s a chat bot or whether it’s a real time million QPS API and an ad server. Almost doesn’t matter. Obviously one of those is a little like sexier to talk about today, which is the agentic one.
Ryan Donovan (14:17.421)
Yeah, so let’s talk about that agentic application, because I think there’s a lot of people either building agents or getting into agents. What does a agentic process with a decision layer look like?
Frank Portman (14:34.222)
So all the times where you’ve just tried to will into existence within a conversation, like what should I do? Predict this should be able to reach for a tool. Possibly. mean, I think this required deep collaboration or maybe us building it ourselves. Possibly there’s a way that this is baked into the model itself, like kind of our representation and what we do. But if effectively it’s a, when the context or the task at hand calls for some sort of prediction or recommendation.
use us. We’re not that general purpose today in terms of our literal interface and integration, but I think we could be. I think separately that begs the question of are we in a fad with sort of chat as the best way people want to communicate with each other or like between chatbots? I don’t know if we are, to be clear I think I phrased that in a kind of loaded way, but I’m not sure.
Ryan Donovan (15:30.629)
Sure. Well, yeah, it seems like a natural, understandable way for humans, right? Like we talk to each other using language. Like why not have the robots do that too?
Frank Portman (15:41.132)
Yeah. And it, it worked. Right. I think if you look back to the genesis of GPT, a lot of these now frontier labs, like DeepMind, I think they were, and I was in some of these circles at the time. People were a little bit salty because they were like, well, yeah, like, duh. Like we could have done that. It just didn’t seem that cool or interesting at the time. And it’s like, yeah, well, someone did it, right. Someone got all the time and they did it and it worked.
Ryan Donovan (16:01.317)
Right. Right. Yeah. Yeah. I I talked to one of the people on the original attention is all you need paper. And he said that he had the idea over lunch. He spent an afternoon kind of coded up. It was like there it was. Right? Yeah.
Frank Portman (16:21.624)
There was, and you’re right, is something, there’s definitely something that’s special about language to me and you, like to people. But clearly there might be as well more abstractly from a general intelligence standpoint, which I think that part is very cool. Like really there’s something going on with language.
Ryan Donovan (16:29.691)
Mm.
Ryan Donovan (16:42.543)
Yeah. So to the agentic decision-makings, I wonder with agents, are there processes that don’t require decisions? And I guess what I’m asking is, what does a decision mean for an agent?
Frank Portman (16:58.83)
That’s a good question. So I think there are definitely things that don’t need decisions. think two economically valuable problems that agents are solving today, like no question. One is coding. And I think the other I would loosely classify as like in your pocket, LLM.
It’s clearly economically valuable because people will pay for it. think that one’s a little murkier for like, what are we doing there and why, but also it will get better, right? Sort of the personal assistant in your pocket is makes sense to me. And then for coding, mean, I see that every day. Like it’s no question economically valuable. We spend a lot of money on.
Ryan Donovan (17:38.841)
A lot of companies do, yeah.
Frank Portman (17:41.184)
And I don’t really think either of those use cases, I mean, the coding one may be a little more than the like, let’s talk about X, let’s automate Y. I don’t think either of them really requires a ton of decision-making. And I use the word a ton, which I think maybe is like hedging a little bit. But the coding thing, like, yes, it needs to make a decision that like, I’m literally gonna write this and I may or may not for approval.
Ryan Donovan (17:55.524)
Yeah.
Frank Portman (18:05.318)
but that at least seems more possible to me within a context that it has, and also with whatever pre-training it was trained with. And then similarly like, Hey, can you go up to my calendar? Like it needs to make a decision to do that. But you’re, you’ve kind of told it exactly what to do. when I think of the hard decisions that it can’t do today, it’s like, Hey, Chad GPT, like how much should I spend on marketing today?
Ryan Donovan (18:21.723)
Yeah.
Ryan Donovan (18:29.499)
I’m sure. I mean, do we want it to make those kind of decisions? Like, you know, I read a paper about sort of belief offloading and situational disempowerment and like giving up these parts of our own personal decision making. Not great, yeah.
Frank Portman (18:47.852)
I agree. I look, I, I’ll this carefully because I don’t want to get skewered by whatever, like all the different camps here, but I think that the agenda coding, if it’s not used in a smart way, it, it is slowly dulling us as well. And we have that conversation internally all the, the time. Like, feel like we’re getting a little slower, like in terms of the like deep thoughts stuff.
Ryan Donovan (19:14.681)
Yeah, and I read that that’s a sort of function of any sort of offloading to tools. It’s not just agents, right? Like, I can’t use a sextant, unbat it navigating a city without a maps application, right? Yeah. So I think for the, I want to sort of get a little focus on the decisions too, because things like word choice,
Frank Portman (19:28.792)
Sure, yeah.
Ryan Donovan (19:44.955)
the sort of function naming are small level decisions, but you’re talking about something else as a decision, right?
Frank Portman (19:52.94)
Yeah. So I guess one concrete example is our current product runs at.
millions and millions of queries per second to basically determine, Hey, someone is on this publisher. They’re watching this TV show. I have this ad slots. Here’s the context. Here’s like the show or here’s the article on ESPN. Do I show anything? And then yes, what do I show? Like what exact creative what’s, what, campaign is it for and so on and so forth. And
They’re even forgetting the latency thing. Cause I think sometimes people are like, well, yeah, sure. Chatbots aren’t that fast. So, but let’s pretend they were that fast. That’s in the, everything that they’re pre-trained with and given everything that’s in context, which is very little actually in that example, right? It’s just, here’s like a row.
Ryan Donovan (20:38.245)
Yeah.
Ryan Donovan (20:46.149)
Yeah, yeah.
Frank Portman (20:50.734)
How are they going to predict the best possible thing to do? They need to predict it. It’s not just a, like, I’m summoning up the next word and sort of get numbers is a good name for a function that queries the numbers in database. It’s like, oh my god, I need to predict. This depends on expected value. Expected value, if it’s to be encoded in the conditional probabilities of generating a sequence, I think that would be very interesting model to see.
Ryan Donovan (21:16.601)
Yeah. Well, and I think to make a good prediction, you need sort of prior data, not just like the sort of baked into the model prior data, but sort of context sensitive prior data. How do you think about get and apply that sort of context to the decision? Yeah.
Frank Portman (21:35.502)
So for our use case, our context is in the form of this highly specialized model’s weights. Because we’re not building a chatbot, our context about this specific publisher or page or the maybe like tokens on the page, those are baked into the model because they are a little bit more specialized around kind of the behavioral prediction side.
Ryan Donovan (21:43.803)
Mm.
Ryan Donovan (21:57.861)
you
Frank Portman (21:57.934)
but in a pure language model where everything is inductive kind of from this 300,000, 500,000 token table with of course, once again, super complex transformers on top, but everything’s inductive from that. The context on any one sort of like publisher is like it is there because these things do have world models, but I don’t think it’s as nuanced for the purposes of personalization and stuff like that.
Ryan Donovan (22:17.765)
Mm-hmm.
Frank Portman (22:22.7)
It will tell you at length about McDonald’s and Burger King being more similar than McDonald’s and H Those are like parlor tricks for the purposes of recommendation.
Ryan Donovan (22:28.485)
Right.
Ryan Donovan (22:32.421)
Hmm, right. Well, I mean, that’s a interesting sort of getting into the different sheets. Like it’s a parallel trick to differentiate between two burger chains where it’s like it’s almost like a brand loyalty thing, right? It’s like do you like the the jets of the Giants? Well, So, you walk you talked about running a pretty significant amount of queries per second?
Frank Portman (23:02.167)
and
Ryan Donovan (23:02.715)
What sort of inferencing stacks and optimizations do you have to do for that?
Frank Portman (23:11.686)
It’s a question. mean, we’re definitely still in our era of let’s maximize margin later. As long as we are not looking at any architectural decision and saying like, this just feels like there’s no way we can fix this. We’re happy to do something that’s slightly inefficient if it helps us move faster because that’s sort of the software business model is.
Ryan Donovan (23:19.44)
Mm.
Ryan Donovan (23:25.936)
Mm-hmm.
Frank Portman (23:35.776)
Adoption then margin. we’re kind of in that mindset. That being said, at that scale, you’re right. is like, you know, we can run out of money if we’re not smart, right? The two biggest things I would say for this specifically, one is pre-compute as much as possible. But have these things and kind of embedding lookup tables versus like on the fly, you need to generate anything. So you trade sort of the memory for your model inference latency budget.
Ryan Donovan (23:44.847)
Sure, yeah, yeah.
Frank Portman (24:05.142)
And the second would be batching. So million requests come in. A lot of these code paths are not that much slower to do two, three, 500 things versus one. but building in batching and queuing kind of at various parts of the stack is difficult because the sort of the straightforward thing to do is one comes in, one comes out. So we spend a lot of time on, batching.
Ryan Donovan (24:07.397)
Mm.
Ryan Donovan (24:26.651)
Mm-hmm.
Ryan Donovan (24:30.189)
All right. You said for the lookup tables, I imagine there’s a lot of similarities in requests. Is that sort of what happens with the lookup tables, sort of a caching of prompts?
Frank Portman (24:44.994)
We cache the actual feature requests for any hot features to be even more easily accessible or accessible in an even quicker way. But we’re not necessarily caching the full response because each…
You know, this is an ad example, right? So each bid request is at least a little bit different. I mean, they are sometimes nearly identical, but we haven’t had to worry too much about caching near identical ones yet just to get a quicker model output. That’s actually a good point though. I mean, maybe as our models get more sophisticated, we might want to do something like that. Maybe as like a…
These ad auctions are interesting, right? There’s all sorts of heuristics you could do. Like you could have a backup bit that’s like good enough, but worse than your main model. But like, if you exceed your latency budget, send the backup one, because the expected value is little greater than zero, stuff like that.
Ryan Donovan (25:27.141)
Mm-hmm.
Ryan Donovan (25:36.507)
Yeah, I mean, I could see a pretty decent chain of conditions on a backup. To that question, how much benefit is there to a just intent prediction model versus a simple wall of if statements?
Frank Portman (25:42.669)
Yeah.
Frank Portman (25:58.702)
Wall of this statements is hard. I mean, you if you decompose any model, I’m sure someone has proven that these can all be like XORs or something. At some point it unironically gets harder to think of and manage the state of heuristics than it is to train a model, honestly.
Ryan Donovan (26:06.691)
Yeah.
Ryan Donovan (26:14.331)
Mm. Mm.
Frank Portman (26:17.302)
Not for everybody, but I think for machine learning team, certainly like we’re better at training models than maintaining like walls of heuristics. but actually a lot of the industry is very heavy on the heuristics. And I think one of our, definitely have a bit of a second mover advantage. I mean, we’re not second, but you know, everyone is not first that’s last, right? We have a second mover advantage here because we don’t have a lot of this. Like pre LLM.
Ryan Donovan (26:35.087)
Yeah, yeah, early stage, yeah.
Frank Portman (26:46.702)
pre-sophisticated machine learning kind of tech debt and just kind of, what I always say internally is can we just solve the right problem? Can we solve the right problem? Yeah.
Ryan Donovan (26:48.855)
Okay. Yeah.
Ryan Donovan (26:55.087)
Mm.
Right, solve the right problem with the tools we have now. You’re not tied to an existing solution, right?
Frank Portman (27:03.968)
Exactly. Can we, at the end of the day, this needs to predict the expected value of semastron. Can we solve that? Is there a way? And sometimes there’s not. Sometimes you have to solve a proxy and you add a heuristic. And the heuristic could be pretty good heuristic. It’s not literally like, if whatever, do whatever. Sometimes these heuristics are pretty thoughtful and intentional. But we love solving the unsent problem correctly if we can. We’ll take the time to.
Ryan Donovan (27:09.455)
Mm-hmm. Yeah.
Ryan Donovan (27:30.723)
Yeah. So I want to save your scalp from the CEO. What’s the non-ads use cases for this?
Frank Portman (27:39.83)
Okay. So the half scalp saving answer is I think we’re talking a lot about advertising and like ad tech. There is also kind of martech and marketing, which is not that different, but it is, I think the natural place to go next because it’s not that different. And that’s more about engaging with your kind of existing customers.
Ryan Donovan (27:51.131)
Mm-hmm. Sure.
Frank Portman (28:02.946)
That has a big industry around it because I think a lot of companies that don’t have a lot of first party data, that’s their way of getting relationships with their customers. Email, SMS, personalized product stuff. And people like it. They don’t see it as intrusively as ads because they’re like, I have already given my money to this company. I like that. I think it’s…
cool or at least not annoying that they’re now telling me like this new shoe is out. People also like shopping, Like people, told multiple times that people would use an app of Instagram, that was all ads. People like shopping, right? So marketing is an interesting place to enter. But I think going a little bit beyond that, fraud and risk thing as well. And then of course, like the long tail of personalization and prediction tasks.
Ryan Donovan (28:29.775)
Right. Yeah.
Ryan Donovan (28:38.639)
Hahaha
Ryan Donovan (28:47.291)
Hmm.
Ryan Donovan (28:57.307)
So you said you have different models for each client, is that right?
Frank Portman (29:04.45)
We have the typical case where we have a base foundation model.
And then we fine tune on a per outcome basis where the outcomes today are typically campaigns, sometimes customized kind of at the customer advertiser level, sometimes at the campaign level. but our, our ability to amortize the pre-training across the huge amount of data and then do a relatively light fine tuning process on a net new campaign or net new data. that’s kind of how this stays economically viable. And then of course, the fact that we can fine tune on such little data and still perform is.
At least that’s my definition of foundation model.
Ryan Donovan (29:41.369)
Yeah, that’s kind of everybody’s foundation model definition, I think. You also mentioned world models earlier. Do you use world models for this?
Frank Portman (29:54.67)
Not explicitly. I think I mentioned it in the context of, you know, because you always hear, I always hear and I always tell, I guess, don’t use LLMs for facts. Like they’re not factoid machines. Maybe if they’re doing research and they can cite it, sure. But don’t just ask it to will from existence, like tell me about McDonald’s. But at end of the day, they kind of can do that in many cases. And that’s like the sort of world model flavor that I’m thinking with LLMs.
Ryan Donovan (30:05.913)
Mm. Right.
Ryan Donovan (30:18.971)
Yeah.
Ryan Donovan (30:24.173)
OK. Yeah, because I’ve heard that a lot of world models kind of use diffusion as their basis. And I’ve talked to some folks doing diffusion for text. So I wonder if diffusion plays a part in this as an error correction.
Frank Portman (30:41.368)
Got it. Yeah, not for us. Yeah.
Ryan Donovan (30:48.879)
So we’re coming up on 30 minutes. Was there anything you wanted to talk about that we didn’t cover? Otherwise, I can ask a future-looking question.
Frank Portman (30:57.166)
Let’s go with the future looking question.
Ryan Donovan (30:59.547)
All right, so what is the spaces you’re looking for beyond, maybe even beyond commerce or the ways to sort of optimize the intent prediction that you’re already doing?
Frank Portman (31:16.644)
so I’ll actually give an answer that’s less related to the product or how we stand to make money and more what motivates kind of me and my team on a technical academic level. So we think the privacy preserving machine learning recommender system world is very interesting. and very nascent.
Ryan Donovan (31:26.523)
Yeah, perfect.
Ryan Donovan (31:38.394)
Yes.
Frank Portman (31:38.87)
And there’s a good amount of work on differential privacy and stuff like that. And there is work on homomorphic machine learning, which is how you can train a model without ever quote unquote seeing the data, just kind of manipulate the symbols.
But it’s very academic. We want to take that stuff into the industry and not just use it, but also contribute to that world. Because at end of the day, we are executing on a difficult bet, which is that companies have to trust us with their sensitive consumer data.
And the way we earn that trust is by having the best security measures, but also the best algorithmic privacy measures in the industry. We don’t have the luxury of running a massive social network and having everything tied to a hermetic email ID.
Ryan Donovan (32:21.782)
Hahaha.
Frank Portman (32:24.526)
But we still think that we can win in the space by applying the best technology. And so I think that privacy, centric machine learning worlds, all sorts of, some are maybe tactical and heuristic-y, such as how do we train our model in your environments? But I think some are just like hard math and science and we can publish papers on it.
Ryan Donovan (32:44.347)
Yeah, I mean, with behavior prediction and the data behind it, I imagine there’s a huge privacy component.
Frank Portman (32:53.464)
Totally, yeah, it’s first class, I will say.
Ryan Donovan (32:57.753)
Yeah. So what is differential privacy?
Frank Portman (33:03.31)
So differential privacy and like, you know, people know this stuff a lot better than me, but it’s a way of looking at a notion of privacy in a mathematical way. I think a a key tenant of it is the concept of being K anonymous, meaning K is a thing you choose like five, 10, 25. And that’s the sort of…
pool of people that let’s say might be you. mean, I think I’m maybe defining it a little bit imprecisely, but it’s a, if someone were to go in the dataset looking for Ryan Donovan, there’s a sort of 25 people, like if they do their hardest, hardest, hardest to like query and filter and all this other stuff, to try to get just you, there’ll be some ambiguity on the level of about like 25 people, for example. And that…
Ryan Donovan (33:40.923)
Mm-hmm.
Ryan Donovan (33:50.587)
Yeah.
Frank Portman (33:58.794)
model works reasonably well for, like data analysis, like querying filter group by measurements. It’s not to my knowledge, not super applicable to training models. think some of the concepts and core tenants are, but I think when you go and do kind of a literature review of differential privacy and what sort of things are differentially private or not, it’s rarely around like, how can you make that into your model training process?
But the homomorphic machine learning, and that’s really cool because that’s, know, homomorphic is a math term. So it goes back to my roots.
Ryan Donovan (34:24.603)
watching.
Ryan Donovan (34:33.467)
I mean, the differential sounds like a diff EQ, differential equations back in the day. All right, we’ll take us to the outro. OK, I’m going to shout out somebody who came on to Stack Overflow earned a badge. And then I’m going to say my name, my title, a CTA where I can be found on the internet. And then you can do the same.
Frank Portman (34:41.516)
There you go. Yeah.
Frank Portman (34:47.587)
See you.
Frank Portman (34:59.512)
Sounds good.
Ryan Donovan (35:01.121)
All right, it’s that time of the show where we shout out somebody who came onto Stack Overflow, dropped some knowledge, shared some curiosity, and got themselves a badge. So today we’re shouting out a populist badge winner, somebody who dropped an answer that was so good it outscored the accepted answer. And so congrats to Hookd for answering, removing whitespace around a saved image. If you’re curious about that, we’ll have the answer for you in the show notes. I am Ryan Donovan. I host the podcast, edit the blog here at Stack Overflow.
If you have questions, concerns, comments, topics to cover, email me at podcast at stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn. Frank.
Frank Portman (35:41.644)
Awesome. Yeah. Thank you so much for having me on. This has been great. I’m Frank Portman, CTO at Yobai. If you want to find out a little bit more about me or maybe my academic background, you can go on fportman.com. So first initial lastname.com and then yobai.ai. got the ai tld. And then I also want to actually just shout out Stack Overflow. I think this whole industry is kind of built on the back of that. I think it’s awesome. So.
Ryan Donovan (35:59.813)
That’s right.
Ryan Donovan (36:07.259)
Well, we appreciate it. And thank you for listening, everyone. And we’ll talk to you next time.
