- 1 week ago
Category
🤖
TechTranscript
00:00OpenAI released hundreds of new mathematics findings this week on a wide range of topics,
00:05including algebra, number theory, and logic. This follows an announcement last month that
00:10it solved the Navier-Stokes equation, which is one of the math problems of the Millennium
00:14Prizes. That offers a $1 million reward for each solution to long-standing math puzzles.
00:21Joining us right now is NYU mathematics professor Tristan Buckmaster. He says that some of the
00:26recent advances by him and another researcher may have inspired OpenAI's prompts that led to that
00:32solution that he'd been working on for quite a while. Tristan, welcome.
00:36All right, so let's talk, first of all, what the Navier-Stokes problem is, why it's important,
00:40how long you've been working on it?
00:42Yeah, it's one of the most fundamental equations in mathematics and physics, one of the oldest
00:47sort of partial differential equations. So it's used to describe fluid and water,
00:54blood flow, air, you know, along an air wing.
01:00Airplanes, yeah.
01:01Yeah, it's the most fundamental equation in fluids. And it's been an open problem for hundreds
01:09of years of whether the solutions can break down.
01:12And you've been working on this for not hundreds of years, but...
01:16No, it's sort of been a north star for me for my whole research career.
01:21And it's so bizarre. Something goes to infinity in a finite amount of time.
01:26Yeah, so...
01:27Are you sure?
01:28To infinity and beyond? It's to infinity.
01:31The velocity goes to infinity. So imagine a fluid particle goes... So the physics breaks.
01:36Right, exactly.
01:37The equations don't make sense anymore.
01:39I immediately thought, would this change anything in how you design an airplane, now that you know this?
01:43I mean, it could be used in all sorts of ways, sort of test fluid models and, like, sort of,
01:50test fluid models.
01:53But it's something that's so rare that it wouldn't fundamentally...
01:57But it has been solved now.
01:58This has been solved, yeah.
02:00You were at the point, you and your partner, Levent Apoge, right?
02:05You were at the point of solving this. And then OpenAI came out and said, hey, lo and behold, we
02:11have solved this.
02:12What did you think? What happened?
02:15That's right. So, end of August, we were... I think we thought we were roughly a month away from solving
02:22it.
02:22From solving it or from publishing it?
02:26You know, I don't know. Until we had the result.
02:31And that's when a rumor, sort of, there was a leak from Anthropic to OpenAI, that Anthropic was close to
02:39solving two Millennium Prize problems.
02:41Not this problem, but two other Millennium Prize problems.
02:43Well, this problem and the Hodge conjecture, it's another Millennium Prize problem.
02:48So, that had spread from OpenAI to the tech world and then on to Twitter.
02:53And it had, sort of, gone a bit crazy.
02:56And then, so, OpenAI started working on these, started spending all these millions of dollars on compute trying to solve
03:03these Millennium Prize problems.
03:04For them, it was, sort of, a life or death situation because, you know, if Anthropic had come out and
03:11they resolved a Millennium Prize problem,
03:13it shows that, you know, they had their... their internal models were better.
03:16So, Levent works for Anthropic.
03:18That's right.
03:18We should back this up.
03:19So, there would be the association with Anthropic solving this problem because he is an employee there.
03:24So, what happened with, when you heard that they had solved this, when they put this out, you contacted them?
03:30So, no, they contacted us before.
03:33So, first of all, the rumor was spreading.
03:35We heard things were getting a bit crazy.
03:37So, we contacted them and we told them, look, we're using your product as well.
03:45You know, this is not a big Anthropic project.
03:48It's just me and Levent.
03:51And, you know, we hoped, actually, we could have worked together, you know.
03:55And they said, like, let's not compete.
03:58They offered us, you know, millions of dollars.
04:01They offered you millions of dollars?
04:02Well, not millions.
04:03They offered us millions of...
04:06Tokens?
04:06Tokens for...
04:09To complete the project.
04:10Okay.
04:11So, this is where you get into the, did they steal this from you?
04:14So, let's talk about that.
04:15And then, on Saturday, we got this, we were told that they're about to do something stupid and to get
04:23on the phone call.
04:24And so, this is when we had this infamous phone call.
04:27Okay.
04:27So, what does that mean?
04:27They were going to do something stupid?
04:29We don't know.
04:30We don't know what it was.
04:31But it was enough to convince us to get on the phone.
04:32And what happened when you got on the phone?
04:35Well, they first tried to bribe us.
04:37They said, you know, they, or bribe me.
04:39They said, I could publish their paper as sole author as long as Levent wouldn't be part of it because
04:49he was an anthropic employee.
04:51They then threatened me.
04:53So, they said, when I said, you know, this was completely unethical what their author was.
04:58And I said, look, I'm going to go public with this.
05:00They said, why would you ruin your career?
05:03And he said, and Sebastian said, look, I've been nice to you.
05:09Sebastian Bubeck from OpenAI.
05:11So, I've been nice to you.
05:13I don't have to be nice.
05:15And at that point, I knew the negotiation was over.
05:18I don't understand what the, I don't, but I don't understand what the negotiation is about.
05:21Either you've done the work or they've done the work.
05:24So, we had solved the, a different problem, the Euler equation, which is sort of a step below the Navier
05:29-Stokes problem.
05:30Right.
05:30And the negotiation was, they'd realized that they'd done something wrong.
05:35But when you say they've realized they did something wrong, let's just go straight at that.
05:38Because that's, by the way, that, that to me is the whole story.
05:41Either they did something wrong or they didn't do something wrong.
05:43Sure.
05:43What are you, what are you accusing them of actually doing?
05:45We haven't really got to that part of this.
05:47Sure.
05:47Well, when, after I'd revealed my name, it took maybe a day for them to get exactly where we were
05:56in our, in our research program.
05:59So, we, they had duplicated our results.
06:01So, you revealed your name, you revealed your name when to them?
06:03On the 3rd, I think it was.
06:05So, Tristan, let's just back this up.
06:06Yeah.
06:06To say you were using OpenAI for some of your work.
06:09Explain that.
06:10So, yeah, I was using OpenAI to, to help interpret some of the work and help develop the work that
06:17I was receiving from Levent.
06:19And so, what that meant was that all our work was on OpenAI servers.
06:25So, yeah.
06:26But can I ask a very technical question?
06:28Do you have an enterprise account?
06:29Do you have a personal account?
06:31Personal.
06:31You have a personal account.
06:32You're, you're in a paid, paid personal account.
06:34That's right.
06:35And you're working using, um, ChatGPT.
06:40You're chatting with it.
06:41You're, you're code, you're using Codex.
06:42What are you using?
06:43Codex.
06:44Using Codex.
06:44And just so we're all on the same page.
06:47Uh, you have turned on or off their ability, uh, to train or use any of your information.
06:53That's actually a significant component part of this story.
06:56Uh, it is and isn't.
06:58So, but there was, there was two, there was two accounts, I think, that I had turned it off.
07:03And there was one account that it was still on.
07:05On, meaning that they.
07:06The default is on.
07:07They're allowed to train on.
07:08The default is on.
07:08The default is on.
07:09The default is on.
07:10So, there was one that I hadn't turned off.
07:12But, uh, you know, there, there, there isn't the walls you believe.
07:17Uh, you know, in, but from user data in, uh, open AI.
07:22I mean, you know, just recently they had a white hacker who used Claude to hack into their system
07:27and gain access to their, uh, internals and get access to user data.
07:32I mean, the, the whole, you know, the whole sandbox is, you know, aren't holding up whatsoever.
07:38So, I, I don't know what happened.
07:39I also don't know.
07:40But you have to genuinely believe that somebody on a, on a very individualistic basis saw, saw you, saw exactly
07:48what you were doing and decided let's go collect all the information in this personal guy's account.
07:54Well, somebody or something, the agents, if they were doing, sending agents, is that what you suspect?
07:57Exactly.
07:58I, I don't think there was one person that said, like, steal this person's data.
08:01I mean, honestly, I don't think they even know whether, you know, I mean, the initial statement is they couldn't
08:07determine.
08:08Let me say this.
08:08That what they now say is that we can say, this is open AI.
08:13We can say categorically, categorically that it is impossible for Dr. Buckmaster's codex prompts over the last two months to
08:21have influenced the system in any way, including training.
08:24Now, there is some, some room in that.
08:26It says the last two months, the last two months, would they have needed that to get to the answer?
08:32I mean, they wouldn't have needed much.
08:35I mean, the, the issue is that they make these statements and they keep changing it.
08:40So when they made that statement to the New York Times, they also, they originally started with, it was April
08:45and they shifted it to July.
08:47You know, they've been lying the whole time.
08:48When I got onto the phone call, they said, uh, they one-shotted Navier Stokes.
08:55Basically what they took, they, they claimed is they.
08:57One shot is a single prompt.
08:59Single prompt.
09:00To create the answer.
09:00They took the, the prize problem.
09:02They, they put it into the prompt, uh, and then they press enter and out came the money in prize.
09:07Right.
09:07Now, this, this evaporated within minutes.
09:10They admitted this was not true.
09:12Uh, you know, they had internal people, uh, communicating them on their internal messaging.
09:17And it, you know, it turned out they worked on all these other problems, including the, the work that we
09:21were doing ourselves.
09:23And, you know, they used 10,000 agents to get work from our result, which is the oil equations, the
09:29Navier Stokes.
09:30They used a hundred agents and they got to exactly where we were with exactly the same argument and exactly
09:37the same methods, uh, within a day.
09:42I mean, look, I don't have any, you know, I, I can't say how they did that, but how did
09:48they, they, they, they went from, before they knew my name, they had no progress.
09:53After they knew my name, they suddenly started from where we left off.
09:59So you find it suspicious?
10:01I find it suspicious.
10:02I, I, I think maybe more interesting is what you are describing as kind of hard pressure and, and, and
10:10threatening you if you would not publish on your own with this.
10:13I mean, that may say more about the tactics that were used.
10:16Now, they have denied those tactics too.
10:17What do you say in response to that?
10:19Well, actually, they haven't denied any of the tactics.
10:21They admitted that they said that they, uh, that, uh, they admitted that they offered me the, to be sole
10:29author.
10:29They admitted that they said, why would you end my career?
10:34They haven't actually denied.
10:36Sebastian has never denied that.
10:37He said, I've been nice.
10:39Uh, I don't need to be nice.
10:41They haven't denied actually anything that I've said.
10:44Uh, so, I mean, I mean, let's, can I just go backwards for a second?
10:49How did you get in touch with them originally?
10:51How did they get in touch with you and why?
10:54Uh, so we, so Levent contacted an open AI mathematician.
11:00Uh, and we originally got in contact with them just because this rumor was going crazy.
11:06I mean, there was betting markets making bets on whether Anthropic would, uh, solve the million price.
11:11So we just wanted to calm things down.
11:13Well, calm them down for what reason?
11:16Well, first of all, the rumor was false, right?
11:19We hadn't solved the Navier-Stokes problem.
11:20You're saying the rumor was out there that you had solved it?
11:22The rumor that we'd solved it was actually false.
11:25We'd sold something slightly less.
11:27And so we realized that this was, uh, uh, you know, the Navier-Stokes problem was, you
11:33know, a pet project of, of the tech companies.
11:36And it was, if their rival had solved it, this was, you know.
11:40Okay.
11:41How much money did they offer you in all of this and why?
11:43They didn't offer me money.
11:44They offered me to, uh, they offered me to be the sole author of the paper.
11:49So I would have got the million dollars.
11:50You guessed it.
11:50Am I wrong?
11:51Didn't you say they were, they were offering millions of dollars?
11:53No, they offered me millions of tokens.
11:54Tokens.
11:54Millions of tokens.
11:55They offered me-
11:56So why were they offering you millions of tokens?
11:58Initially, when I came to them and I said, like, uh, let's, let's, let's talk.
12:03Initially, they said, let's not compete.
12:05Uh, we can offer you compute.
12:08So as in to work together.
12:10Right.
12:10That wasn't by email.
12:11That was before the phone call.
12:13Okay.
12:14And then they started competing, of course.
12:16So do you use OpenAI still and Anthropic still?
12:19Uh, I still use Anthropic.
12:21I still, I still have, uh, OpenAI, uh, codex, but I don't put any sort of ideas in there.
12:28Anything I decided to be original.
12:31And what's come of all of this?
12:32So you're not going to get the million dollar prize money.
12:34What, where does that kind of leave things?
12:36How are you feeling about it all?
12:38I mean, for me, I mean, honestly, I, you know, I wasn't, it was never my goal to solve the
12:42Navier-Stokes problem.
12:43So, I mean, like to solve the Euler problem, I'm super happy.
12:46Like, uh, so, I mean, in all of this, like, uh, it was never my goal.
12:51I mean, I think the bigger picture is, uh, like this recent dump of papers on Tuesday,
12:58which destroyed the careers of early, uh, early, uh, mathematicians, early career mathematicians.
13:04So we've had whole research programs wiped out on Tuesday by their dump.
13:10And you have to understand, they have access to all our research proposals.
13:16And, uh, and the reason they have access, you don't even have to use the LLM.
13:20So if you submit a research proposal, a panelist will look at that research proposal and they have to write
13:25a summary.
13:26And often how they write the summary is they, they put it into the, uh, so they have a massive
13:32conflict of interest, right?
13:34They have all, they have, for the next few years, they have all, uh, all the mathematicians and scientists' research
13:41programs in their training data
13:43and how they're going to go about it.
13:45And, and you would, your message would be user beware.
13:48Yeah, I mean, the, the fact that they would front run their customers, I mean, it shows their sort of
13:54disdain for their customers.
13:56Uh, I mean, like, why, why would a company, uh, buy their product when, like, if you get an interesting
14:05ideas,
14:06they can look at your private notes and beat you to market.
14:09Yeah.