Skip to playerSkip to main content
  • 3 weeks ago
Pennymac’s Jensen Szakaly and Vesta CEO Mike Yu talk with HousingWire’s Allison LaForgia about how Pennymac moved 40% of its backend fulfillment operations to AI agents, how it trains and manages those AI agents and how its workforce changes as automation scales toward 80%.

Category

🗞
News
Transcript
00:12From Chicago with HousingWire, I'm Alison LaBorgia and I am sitting down with Mike Yu,
00:18the CEO of Vesta and Jensen Zikeli, the Managing Director of Mortgage Fulfillment Division at
00:25PennyMac Loan Services. I want to highlight what is actually different about how PennyMac
00:32manufactures a loan now versus what it looked like 18 months ago. What's the difference?
00:37I mean, it's an easy answer. 18 months ago, we were in the development process or deployment
00:43process with Vesta. So that was pre-full launch. It was the historical mortgage fulfillment process
00:54where you had a lot of stare and compare. You had manual document indexing. There were limited OCR
01:00and kind of data extraction, but we didn't have a platform that really supported what we were trying
01:08to accomplish. And prior to that, dating back to 2018, I know that's well before the 18 months,
01:14but we had switched to a task-based workflow. And we learned very quickly that having a task-based
01:21workflow, while not critical to deploying agents, certainly enabled it because instead of trying
01:27to kind of boil the ocean, you could take a task or a series of tasks and apply an agent
01:34against that
01:36task, which is much easier to train, deploy, and have a more rapid deployment cycle,
01:42versus I want to, you know, have an AI agent that processes A through Z, right? That's very
01:49complicated to do. And then, you know, fast forward, we went to Vesta. We were fully deployed
01:55in our consumer direct lending channel towards the end of February of this year. So call it,
02:02you know, eight months-ish. You know, immediately we saw a 30 to 50% efficiency gain just based on
02:10the base system and, you know, better automation, better APIs, a more modern, intuitive navigation,
02:19just the way work is queued up. All that made us significantly more efficient. And that was before
02:25we even introduced agents. So then agents came and call it late April, May, I would say is, you know,
02:33when we pulled together a pretty elite team to really, as their day job, focus on deploying
02:40AI agents. And so with that now, you know, about 40, 45% of our backend fulfillment operations
02:49are handled by AI agents. In development, not fully released is another, call it 15 to 20%-ish. So
02:58we're getting close to the 60, 65%, you know, either fully deployed or in the process of being deployed.
03:06So we're talking reduced cycle times, right? Across all functions. We're starting to measure,
03:13you know, tasks in minutes, not days. Our conditional approval timelines have been cut in half. And kind
03:21of an interesting phenomenon behind all this is the cultural buy-in from the team, right? Like it's
03:28actually been a cultural boost, which I didn't expect that. And you hear all this negative AI,
03:34you know, news and the team's genuinely excited about it. We, we have all day sessions where we
03:42deep dive these agents and we'll have, you know, our title review agent leader kind of poking fun at
03:49our closing agent saying, my agent's better than your agent, right? So it's really, it's become this
03:54really cool experience. And, you know, it's only been call it four or five months since we've been
04:00really, really going hard at this and to see the progress, you know, I think we're at the frontier of
04:07the frontier and we just simply could not have done it that fast without the Vesta platform and the
04:14foundation that Vesta provides. Now, I want to dig into this AI agent situation a little bit because
04:25it sounds amazing. And I mean, I love to hear about the competition between even teams on my agents
04:32doing this and mine's doing this. How do you actually teach an AI agent its job? Yeah. So that,
04:40that's actually my favorite question. So I'm glad you asked that. Um, we train AI agents the same way
04:48we train new hires, right? So if you think about training a new hire, it usually starts with, you
04:56know, you give them some sort of test case and like a sanitized environment just to make sure either as
05:01part of the recruiting hiring process, make sure somebody says they're an underwriter, they actually
05:06know how to calculate income, right? So that's kind of phase one. Phase two is you, you onboard them,
05:13hire them and you may give them five, 10, 20 kind of live production test case, but you have their
05:20manager second sign a hundred percent of risk, right? And now in theory, they, they prove their,
05:26their competence. You expand their school, their scope and responsibility over time. So maybe you start
05:32with conventional underwriting and then maybe you go, okay, you have a DE. So we're going to let you
05:39do five FHA loans and we're going to second sign those and, and gradually scale up until you have
05:45full trust and confidence that this new team member is executing at the right, you know, level
05:52and within your risk tolerance. So that's, you know, how we, we train employees. I'll tell you how we
05:59train agents. It's literally the exact same. So we start with, you know, in the, the lower development
06:06environments within Vesta, we spin up kind of the technical details of an agent. We run it manually on
06:15a handful of test loans just to make sure it, it, it kind of mechanically operates. And then once we
06:22get
06:23comfortable with that, we go to what we call manually invoking the agent and it's literally
06:28myself and a group of leaders and even frontline associates, we'll sit there and it's almost like
06:34you want to pop a bottle of champagne after or something because somebody will, will grab a, a loan in
06:40the
06:40system, we'll assign the agent and we'll just watch in real time as the agents processing the work. And with
06:48Vesta has the full agent trace, you can see the chain of thought. And so we'll all sit there and
06:53stare and watch it, you know, anywhere from 20 seconds to five minutes, however long it takes.
06:59Then, you know, that starts the ramp up. So that's kind of the first loan to give to somebody in
07:04production. As we advance, you know, we iterate daily, we do deployments daily. And we always start
07:12with a hundred percent human in the loop. So gradually what we do is we usually go 10 percent
07:18automation. So 10 percent of the work going to the agent, 90 percent go into a human. As we get
07:24more
07:24and more comfortable, further refine, train, educate that agent, we get them to a hundred percent. They're
07:30considered, you know, certified, if you will. And we no longer require a hundred percent human in the loop.
07:38We do have embedded in there, you know, really no different than managing people or managing our
07:45team members is if there's a question or something's unknown, we ask the agent to escalate.
07:52And when they escalate, it's almost like the agent raising their hand, asking their manager for help.
07:58And with that, by design, we have escalation around edge cases. Or another example would be,
08:05let's say the agent recommended declining a loan, right? Or an application. I want a human in the
08:11loop a hundred percent of the time on that. And we want to try and find a solution for that
08:17customer.
08:18And so I know that was a lot of words, but hopefully the key message is we train them like
08:25we train our
08:25new team members. I learned something fun there, which is my design team has been saying we need to
08:30redesign the screen where you can watch the agent think. And I was like, no one watches the agent
08:35think, you know, like the agent just works. And then everyone goes, I just learned Jensen wants
08:39to drink champagne while he watches the agent think. So maybe I'll let them redesign that screen.
08:42Yeah. I'd keep the screen.
08:46So now we've talked about how to train the agent. So once the agent's live, we've had our champagne
08:52moment. We've watched him do its work. We've watched it train itself. We've made sure that it's in its
08:58parameters. How do we know that it's doing its job right? It's a great question. So I mentioned
09:04earlier through the entire iteration and deployment, we have a hundred percent human in the loop, right?
09:08We absolutely love AI. We love the potential. We see it in action, but we understand we're taking risks.
09:15No different than you take, you know, with having a new team member, people make mistakes, agents can make
09:21mistakes. So we have a very robust ongoing QA and QC kind of, uh, behind every agent. And so the
09:30QA is
09:31in near real time. And then obviously through investor feedback, our quality control team feedback,
09:37we have that layered feedback to make sure that we're not, you know, taking on any unnecessary risks.
09:45And then I mentioned earlier, kind of escalations being baked in. That's another way for us to
09:51understand what the agent's capable of doing, what they're not capable of doing. And it's just a way
09:57to continuously improve the agent. So, you know, this sounds kind of like a generic answer, but
10:03it's really no different than what we do for team members. Uh, it's just, we get a lot better notes
10:09because they list everything it's thinking versus, you know, you ask, uh, Jimmy when they underwrote a
10:15loan two weeks ago, what were they thinking? They may not remember, right? Versus with investor,
10:21you have the entire history and, and can really drill into where the agent may have went wrong and
10:27where do you need to make prompt, prompt changes. And then that, that sampling, that QA sampling near real
10:34time, it depends if it's a high risk, you know, action such as rendering a decision, right? That's
10:42going to be at a higher selection rate or sampling rate. Then something is, you know, uh, less risk,
10:49which would be, you know, a customer uploads a document, the agent evaluates it and says, yes,
10:55this document's good. Knowing that further down the line, we have a human in the loop.
11:01The other thing where you have, you have to make sure agents don't get worse over time.
11:05You don't have this problem with people, right? Like normally if my employee comes in today,
11:09I'm pretty sure they at least know all the things they knew yesterday. And agents have this like a
11:13slightly weird behavior where we're actually doing a lot behind the scenes as well to make sure we're
11:17running really comprehensive evaluation sets. So every time you make any change to the code in general,
11:23like not any LOS release, not just an agent release, but an LOS release, you've got to make sure you
11:27are
11:27rerunning your entire evaluation set. You've got to make sure the scores look like they did yesterday.
11:31You've got a bunch of like automated monitoring that we've partnered with PennyMac on, on like,
11:35is the agent now taking 50% longer to do each task than it used to be? That may be
11:39fine because no
11:40one is like sitting there waiting for the agent, but it may be, uh, kind of indicative that, yeah,
11:45that's something that maybe it did get a little bit dumber. Maybe it is starting to do something
11:48that it wasn't doing before. So you're sitting back there watching all of this. We've got to make sure
11:52there's, there's all sorts of, and then there's all sorts of other tricks you can do. You can take
11:55smarter LLMs and, you know, that are much more expensive that you wouldn't actually run on every
11:59single task and run them on a sampling. So you can do some automated sampling and some insights on,
12:04you know, anytime we want to make a notable change, we have to do pretty significant A-B testing.
12:08So you've got to make sure that, you know, if Jensen hired me to work at PennyMac on the processing
12:14line,
12:15I would need to make sure that every day I was putting in high quality work or Jensen probably fired
12:18me.
12:18It's no different for agents. You've got to make sure that the agent is reliable,
12:22more reliable than human and more consistent. And so there's a lot of responsibility there,
12:26I think to have really comprehensive evaluation sets and production analytics and all those things.
12:29Yeah. And we have a whole suite of analytics as well. So Mike's really,
12:33Mike and his team are looking at kind of the backend agent infrastructure, if you will.
12:38We're looking at it almost from the customer, the operational layer where we're saying,
12:43okay, why is this agent not, not running as fast? We need to insert.
12:48We have analytics around, you know, are we seeing us elevated escalation rates? Right. So we mentioned,
12:55go ahead.
12:55So I want, I want to be really specific in this answer. How do you measure accuracy and how is
13:03it audited?
13:04Because those are the two things that in a highly regulated industry, we need to be hyper specific on.
13:10So the accuracy starts with, look, we do a hundred percent human in the loop and we have leaders and
13:16senior, you know, team members, instead of doing the work today that the agents are doing, they're
13:21now auditing the work. So we have that near real time feedback, right? We strive for zero defects at
13:27PennyMac. That's just who we are. It's in our DNA. Um, you know, what we determine in, in terms of
13:34acceptability is sub 2% error rate. And we track that every single day, every morning we start,
13:41we have a group of about 20 of us. We pull up our dashboards. We evaluate what were the QA
13:47results
13:47from the prior day? What were the escalation rates from the prior day? Where do we need to
13:53do deep dive sessions where we have kind of the, um, the program owner that's doing the prompt
13:59configuration, a business owner and somebody from our business transformation team and potentially
14:05even more get in a room and, and figure it out. And then we'll deploy that night retest. And if
14:11we see
14:12anything go astray, we can very quickly, the way we've built, you know, part of it's the way Vesta's
14:17built, but part of it's the way I think PennyMac's configured it is if we need to revert to a
14:22hundred percent
14:23manual again, or add a hundred percent human in the loop, it's the click of a button.
14:28And then behind that, uh, you know, we have our, our three lines of defense at PennyMac.
14:34So we have our frontline, I mentioned the QA, we have our second line, that's our quality control group
14:39and they're doing testing. And then even our third line, which is our internal audit, you know,
14:44they've collaborated with us. There's, you know, AI is new and, and there's a lot of people that are
14:49scared of it. There's absolutely a strong foundation around risk management and responsible use and
14:56making sure that, you know, if an agent goes rogue, we would identify it ideally proactively,
15:02but if reactively we can shut it down immediately. Taking out your crystal balls.
15:09What is next for AI agents at PennyMac? As the agents take on more,
15:15what does that working partnership between your operators and the agents look like?
15:22Sure. So on the first part, you know, we have a committed plan really through the end of 2026,
15:27a roadmap and through the end of 2027. For that roadmap, I believe we can get to north of 80
15:34%
15:35automation within our fulfillment. Um, our kind of current focus, look, calculating income,
15:44it has been kind of the industry issue for years, right? We are very close to having an agent that
15:51all types of income lights out, calculates it, right? So that's one that we're focused on.
15:59We want to get that fully deployed by the end of the year. Closing is another highly complex agent
16:04that, you know, when you're in closing and the customer's waiting for docs at the table,
16:09you can't have light, light issues, right? So that's an area that's ripe for automation,
16:15right for art, artificial intelligence. Uh, so that's really, you know, the two big focus items
16:21right now. And then beyond Vesta, we're also, you know, working on an aggressive timeline around
16:28natural language, virtual assistants, where that's, that's kind of where we're building the back office
16:35now to accelerate everything. And then the, the NLVA coming, you know, on top of that outside of Vesta,
16:43that's what's going to enable that 24 by seven customer interaction. Now those natural language
16:49virtual assistants, they're going to interact with Vesta. And so if you're a consumer or customer,
16:55and you want to, you know, get a preapproval at, at midnight, and there happens to not, you know,
17:01it's outside of PennyMax hours, you upload your documentation, we have agents that review it,
17:07you have an NLVA that you can talk to about options and pricing and, and then, you know,
17:13you can lock your loan then and there. Right. And so that's, that's the roadmap. The end of 2026 will
17:21be
17:22heavy Vesta focused. Then the, you know, 2027 will be much more on the natural language virtual
17:29assistants, but then also replicating what we've done in our consumer direct lending channel in
17:35our third party origination channel. That's moving on to Vesta, you know, later this year and early
17:41into next year. All I can say is 2027 is a long way away in AI with the rate the
17:47models are improving.
17:48I am cooking things up that Jensen doesn't even know about yet. And he is going to be very excited
17:51about it. I better find out timely this time. That's all I'm saying. You will.
17:55I'm excited. I love the idea of the nurse getting off of night shift, being able to go through,
18:01ask questions in real time, outside of hours, and being able to have a really great customer
18:07experience. I love that PennyMax is focused on that. The other thing, real quick, just to answer
18:12your second comment. When we laid all this out, we had this concept of our workforce evolution, right?
18:20So we're evolving the entire way we fulfill a mortgage and originate a mortgage. And
18:26in late April, early May, I kind of put that on the back burner. I will tell you from what
18:32I've seen
18:32over the past two to three months, in terms of our ability to deploy at scale and quickly, right?
18:38That workforce evolution is now like top of mind because it's how do you empower your team members?
18:46How do you upskill them? How do you redeploy resources to do QA and escalations versus performing
18:52the task culturally? How do you maintain that energy and excitement about AI and start to remove kind of the
19:03mundane stare and compare data entry and really upskill those associates to tackle the things
19:09that are complicated that require a human in the world? And so that piece to me, thanks to Vesta,
19:18that piece is going to be the hardest, is really that workforce evolution.
19:22We even have it.
19:23And that's not a PennyMax problem either. That's a, I think industry societal problem.
19:28I think it's ubiquitous. I think everybody is figuring out the best way to power their humans,
19:36to lean into the tasks that require that human touch. And mortgage is such an emotional transaction
19:43when we can let our people shine. And I think PennyMax does this great. And I'd love to see this
19:48partnership where you're letting your people take on more people focused tasks to better serve the
19:55person who we're all trying to serve, which is the borrower.
19:58Exactly.
20:00Mike Jensen, thank you so much for sitting down with me. I can't wait to see what's next.
20:04Thank you. This was fun. Appreciate it.

Recommended