OpenAI Email Archives (from Musk v. Altman and OpenAI blog) - Part I

date
Dec 16, 2024
slug
open_ai_email_archives_musk_altman
status
Published
tags
AI
summary
type
Post

Subject: question

Sam Altman to Elon Musk - May 25, 2015 9:10 PM

Been thinking a lot about whether it's possible to stop humanity from developing AI.
I think the answer is almost definitely not.
If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first.
Any thoughts on whether it would be good for YC to start a Manhattan Project for AI? My sense is we could get many of the top ~50 to work on it, and we could structure it so that the tech belongs to the world via some sort of nonprofit but the people working on it get startup-like compensation if it works. Obviously we'd comply with/aggressively support all regulation.
Sam

Elon Musk to Sam Altman - May 25, 2015 11:09 PM

Probably worth a conversation

Sam Altman to Elon Musk - Jun 24, 2015 10:24 AM

The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest. More generally, safety should be a first-class requirement.
I think we’d ideally start with a group of 7-10 people, and plan to expand from there. We have a nice extra building in Mountain View they can have.
I think for a governance structure, we should start with 5 people and I’d propose you, Bill Gates, Pierre Omidyar, Dustin Moskovitz, and me. The technology would be owned by the foundation and used “for the good of the world”, and in cases where it’s not obvious how that should be applied the 5 of us would decide. The researchers would have significant financial upside but it would be uncorrelated to what they build, which should eliminate some of the conflict (we’ll pay them a competitive salary and give them YC equity for the upside). We’d have an ongoing conversation about what work should be open-sourced and what shouldn’t. At some point we’d get someone to run the team, but he/she probably shouldn’t be on the governance board.
Will you be involved somehow in addition to just governance? I think that would be really helpful for getting work pointed in the right direction getting the best people to be part of it. Ideally you’d come by and talk to them about progress once a month or whatever. We generically call people involved in some limited way in YC “part-time partners” (we do that with Peter Thiel for example, though at this point he’s very involved) but we could call it whatever you want. Even if you can’t really spend time on it but can be publicly supportive, that would still probably be really helpful for recruiting.
I think the right plan with the regulation letter is to wait for this to get going and then I can just release it with a message like “now that we are doing this, I’ve been thinking a lot about what sort of constraints the world needs for safefy.” I’m happy to leave you off as a signatory. I also suspect that after it’s out more people will be willing to get behind it.
Sam

Elon Musk to Sam Altman - Jun 24, 2015 11:05 PM

Agree on all

Subject: Re: AI docs 📎

Sam Altman to Elon Musk - Nov 20, 2015 10:48AM

Elon–
Plan is to have you, me, and Ilya on the Board of Directors for YC AI, which will be a Delaware non-profit. We will also state that we plan to elect two other outsiders by majority vote of the Board.
We will write into the bylaws that any technology that potentially compromises the safety of humanity has to get consent of the Board to be released, and we will reference this in the researchers’ employment contracts.
At a high level, does that work for you?
I’m cc’ing our GC <redacted> here–is there someone in your office he can work with on the details?
Sam

Elon Musk to Sam Altman - Nov 20, 2015 12:29PM

I think this should be independent from (but supported by) YC, not what sounds like a subsidiary.
Also, the structure doesn’t seem optimal. In particular, the YC stock along with a salary from the nonprofit muddies the alignment of incentives. Probably better to have a standard C corp with a parallel nonprofit.

Subject: follow up from call

Greg Brockman to Elon Musk, (cc: Sam Altman) - Nov 22, 2015 6:11 PM

Hey Elon,
Nice chatting earlier.
As I mentioned on the phone, here's the latest early draft of the blog post: https://quip.com/6YnqA26RJgKr. (Sam, Ilya, and I are thinking about new names; would love any input from you.)
Obviously, there's a lot of other detail to change too, but I'm curious what you think of that kind of messaging. I don't want to pull any punches, and would feel comfortable broadcasting a stronger message if it feels right. I think it's mostly important that our messaging appeals to the research community (or at least the subset we want to hire). I hope for us to enter the field as a neutral group, looking to collaborate widely and shift the dialog towards being about humanity winning rather than any particular group or company. (I think that's the best way to bootstrap ourselves into being a leading research institution.)
I've attached the offer letter template we've been using, with a salary of $175k. Here's the email template I've been sending people:
Attached is your official YCR offer letter! Please sign and date the your convenience. There will be two more documents coming:
  • A separate letter offering you 0.25% of each YC batch you are present for (as compensation for being an Advisor to YC).
  • The At-Will Employment, Confidential Information, Invention Assignment and Arbitration Agreement
(As this is the first batch of official offers we've done, please forgive any bumpiness along the way, and please let me know if anything looks weird!)
We plan to offer the following benefits:
  • Health, dental, and vision insurance
  • Unlimited vacation days with a recommendation of four weeks per year
  • Paid parental leave
  • Paid conference attendance when you are presenting YC AI work or asked to attend by YC AI
We're also happy to provide visa support. When you're ready to talk about visa-related questions, I'm happy to put you in touch with Kirsty from YC.
Please let me know if you have any questions — I'm available to chat any time! Looking forward to working together :).
  • gdb

Elon Musk to: Greg Brockman (cc Sam Altman) - Nov 22, 2015 7:48PM

Blog sounds good, assuming adjustments for neutrality vs being YC-centric.
I'd favor positioning the blog to appeal a bit more to the general public -- there is a lot of value to having the public root for us to succeed -- and then having a longer, more detailed and inside-baseball version for recruiting, with a link to it at the end of the general public version.
We need to go with a much bigger number than $100M to avoid sounding hopeless relative to what Google or Facebook are spending. I think we should say that we are starting with a $1B funding commitment. This is real. I will cover whatever anyone else doesn't provide.
Template seems fine, apart from shifting to a vesting cash bonus as default, which can optionally be turned into YC or potentially SpaceX (need to understand how much this will be) stock.

Subject: Draft opening paragraphs

Elon Musk to Sam Altman - Dec 8, 2015 9:29 AM

It is super important to get the opening summary section right. This will be what everyone reads and what the press mostly quotes. The whole point of this release is to attract top talent. Not sure Greg totally gets that.
  • --- OpenAI is a non-profit artificial intelligence research company with the goal of advancing digital intelligence in the way that is most likely to benefit humanity as a whole, unencumbered by an obligation to generate financial returns.
The underlying philosophy of our company is to disseminate AI technology as broadly as possible as an extension of all individual human wills, ensuring, in the spirit of liberty, that the power of digital intelligence is not overly concentrated and evolves toward the future desired by the sum of humanity.
The outcome of this venture is uncertain and the pay is low compared to what others will offer, but we believe the goal and the structure are right. We hope this is what matters most to the best in the field.

Sam Altman to Elon Musk - Dec 8, 2015 10:34 AM

how is this?
__
OpenAI is a non-profit artificial intelligence research company with the goal of advancing digital intelligence in the way that is most likely to benefit humanity as a whole, unencumbered by an obligation to generate financial returns.
Because we don't have any financial obligations, we can focus on the maximal positive human impact and disseminating AI technology as broadly as possible. We believe AI should be an extension of individual human wills and, in the spirit of liberty, not be concentrated in the hands of the few.
The outcome of this venture is uncertain and the pay is low compared to what others will offer, but we believe the goal and the structure are right. We hope this is what matters most to the best in the field.

Subject: just got word...

Sam Altman to Elon Musk - Dec 11, 2015 11:30AM

that deepmind is going to give everyone in openAI massive counteroffers tomorrow to try to kill it.
do you have any objection to me proactively increasing everyone's comp by 100-200k per year? i think they're all motivated by the mission here but it would be a good signal to everyone we are going to take care of them over time.
sounds like deepmind is planning to go to war over this, they've been literally cornering people at NIPS.

Elon Musk to Sam Altman - Dec 11, 2015

Has Ilya come back with a solid yes?
If anyone seems at all uncertain, I’m happy to call them personally too. Have told Emma this is my absolute top priority 24/7.

Sam Altman to Elon Musk - Dec 11, 2015 12:15 PM

yes committed committed. just gave his word.

Elon Musk to Sam Altman - Dec 11, 2015 12:32 PM

awesome

Sam Altman to Elon Musk - Dec 11, 2015 12:35 PM

everyone feels great, saying stuff like "bring on the deepmind offers, they unfortunately dont have 'do the right thing' on their side"
news out at 130 pm pst

Subject: The OpenAI Company

Elon Musk to: Ilya Sutskever, Pamela Vagata, Vicki Cheung, Diederik Kingma, Andrej Karpathy, John D. Schulman, Trevor Blackwell, Greg Brockman, (cc:Sam Altman) - Dec 11, 2015 4:41 PM

Congratulations on a great beginning!
We are outmanned and outgunned by a ridiculous margin by organizations you know well, but we have right on our side and that counts for a lot. I like the odds.
Our most important consideration is recruitment of the best people. The output of any company is the vector sum of the people within it. If we are able to attract the most talented people over time and our direction is correctly aligned, then OpenAI will prevail.
To this end, please give a lot of thought to who should join. If I can be helpful with recruitment or anything else, I am at your disposal. I would recommend paying close attention to people who haven't completed their grad or even undergrad, but are obviously brilliant. Better to have them join before they achieve a breakthrough.
Looking forward to working together,
Elon

Subject: Fwd: congrats on the falcon 9

<redacted> to: Elon Musk - Jan 2, 2016 10:12 AM CST

Hi Elon Happy new year to you, ██████████!
Congratulations on landing the Falcon 9, what an amazing achievement. Time to build out the fleet now!
I've seen you (and Sam and other OpenAI people) doing a lot of interviews recently extolling the virtues of open sourcing AI, but I presume you realise that this is not some sort of panacea that will somehow magically solve the safety problem? There are many good arguments as to why the approach you are taking is actually very dangerous and in fact may increase the risk to the world. Some of the more obvious points are well articulated in this blog post, that I'm sure you've seen, but there are also other important considerations: http://slatestarcodex.com/2015/12/17/should-ai-be-open/
I'd be interested to hear your counter-arguments to these points.
Best,
████
[Elon forwards the above email to Sam Altman, Ilya Sutskever and Greg Brockman on Jan 2, 2016 8:18AM]

Ilya Sutskever to: Elon Musk, Sam Altman, Greg Brockman - Jan 2, 2016 9:06 AM

The article is concerned with a hard takeoff scenario: if a hard takeoff occurs, and a safe AI is harder to build than an unsafe one, then by opensorucing everything, we make it easy for someone unscrupulous with access to overwhelming amount of hardware to build an unsafe AI, which will experience a hard takeoff. As we get closer to building AI, it will make sense to start being less open. The Open in openAI means that everyone should benefit from the fruits of AI after its built, but it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes).

Elon Musk to: Ilya Sutskever - Jan 2, 2016 9:11 AM

Yup

Subject: Re: Followup thoughts 📎

Elon Musk to: Ilya Sutskever, Greg Brockman, Sam Altman - Feb 19, 2016 12:05 AM

Frankly, what surprises me is that the AI community is taking this long to figure out concepts. It doesn't sound super hard. High-level linking of a large number of deep nets sounds like the right approach or at least a key part of the right approach. ███████████████████████████████
The probability of DeepMind creating a deep mind increases every year. Maybe it doesn't get past 50% in 2 to 3 years, but it likely moves past 10%. That doesn't sound crazy to me, given their resources.
In any event, I have found that it is far better to overestimate than underestimate competitors.
This doesn't mean we should rush out and hire weak talent. I agree that nothing good would be achieved by that. What we need to do is redouble our efforts to seek out the best people in the world, do whatever it takes to bring them on board and imbue the company with a high sense of urgency.
It will be important for OpenAI to achieve something significant in the next 6 to 9 months to show that we are for real. Doesn't need to be a whopper breakthrough, but it should be enough for key talent around the world to sit up and take notice.
████████████████████████████████████████████████████████████████████████████████████████████████████████████

Ilya Sutskever to: Elon Musk, (cc: Greg Brockman, Sam Altman) - Feb 19, 2016 10:28 AM

Several points:
  • It is not the case that once we solve "concepts," we get AI. Other problems that will have to be solved include unsupervised learning, transfer learning, and lifetime learning. We're also doing pretty badly with language right now. It does not mean that these problems will not see significant progress in the coming years, but it is not the case that there is only one problem that stands between us and full human level AI.
  • We can't build AI today because we lack key ideas (computers may be too slow, too, but we can't tell). Powerful ideas are produced by top people. Massive clusters help, and are very worth getting, but they play a less important role.
  • We will be able to achieve a conventionally significant result in the next 6 to 9 months, simply because the people we already have are very good. Achieving a field-altering result will be harder, riskier, and take longer. But we have a not unreasonable plan for that as well.

Subject: compensation framework

Greg Brockman to Elon Musk, (cc: Sam Altman) - Feb 21, 2016 11:34 AM

Hi all,
We're currently doing our first round of full-time offers post-founding. It's obviously super important to get these right, as the implications are very long-term. I don't yet feel comfortable making decisions here on my own, and would love any guidance.
Here's what we're currently doing:
Founding team: $275k salary + 25bps of YC stock
  • Also have option of switching permanently to $125k annual bonus or equivalent in YC or SpaceX stock. I don't know if anyone's taken us up on this.
New offers: $175k annual salary + $125k annual bonus || equivalent in YC or SpaceX stock. Bonus is subject to performance review, where you may get 0% or significantly greater than 100%.
Special cases: gdb + Ilya + Trevor
The plan is to keep a mostly flat salary, and use the bonus multiple as a way to reward strong performers.
Some notes:
  • We use a 20% annualized discount for the 8 years until the stock becomes liquid, the $125k bonus equates to 12bps in YC. So the terminal value is more like $750k. This number sounds a lot more impressive, though obviously it's hard to value exactly.
  • The founding team was initially offered $175k each. The day after the lab launched, we proactively increased everyone's salary by $100k, telling them that we are financially committed to them as the lab becomes successful, and asking for a personal promise to ignore all counteroffers and trust we'll take care of them.
  • We're currently interviewing Ian Goodfellow from Brain, who is one of the top 2 scientists in the field we don't have (the other being Alex Graves, who is a DeepMind loyalist). He's the best person on Brain, so Google will fight for him. We're grandfathering him into the founding team offer.
Some salary datapoints:
  • John was offered $250k all-in annualized at DeepMind, thought he could negotiate to $300k easily.
  • Wojciech was verbally offered ~$1.25M/year at FAIR (no concrete letter though)
  • Andrew Tulloch is getting $800k/year at FB. (A lot is stock which is vesting.)
  • Ian Goodfellow is currently getting $165k cash + $600k stock/year at Google.
  • Apple is a bit desperate and offering people $550k cash (plus stock, presumably). I don't think anyone good is saying yes.
Two concrete candidates that are on my mind:
  • Andrew is very close to saying yes. However, he's concerned about taking such a large paycut.
  • Ian has stated he's not primarily concerned with money, but the Bay Area is expensive / wants to make sure he can buy a house. I don't know what will happen if/when Google starts throwing around the numbers they threw at Ilya.
My immediate questions:
1. I expect Andrew will try to negotiate up. Should we stick to his offer, and tell him to only join if he's excited enough to take that kind of paycut (and that others have left more behind)?
2. Ian will be interviewing + (I'm sure) getting an offer on Wednesday. Should we consider his offer final, or be willing to slide depending on what Google offers?
3. Depending on the answers to 1 + 2, I'm wondering if this flat strategy makes sense. If we keep it, I feel we'll have to really sell people on the bonus multiplier. Maybe one option would be using a signing bonus as a lever to get people to sign?
4. Very secondary, but our intern comp is also below market: $9k/mo. (FB offers $9k + free housing, Google offers like $11k/mo all-in.) Comp is much less important to interns than to FT people, since the experience is primary. But I think we may have lost a candidate who was on the edge to this. Given the dollar/hour is so much lower than for FT, should we consider increasing the amount?
I'm happy to chat about this at any time.
  • gdb

Elon Musk to Greg Brockman, (cc: Sam Altman) - Feb 22, 2016 12:09 AM

We need to do what it takes to get the top talent. Let's go higher. If, at some point, we need to revisit what existing people are getting paid, that's fine.
Either we get the best people in the world or we will get whipped by Deepmind.
Whatever it takes to bring on ace talent is fine by me.
Deepmind is causing me extreme mental stress. If they win, it will be really bad news with their one mind to rule the world philosophy. They are obviously making major progress and well they should, given the talent level over there.

Greg Brockman to Elon Musk, (cc: Sam Altman) - Feb 22, 2016 12:21 AM

Read you loud and clear. Sounds like a plan. Will plan to continue working with sama on specifics, but let me know if you'd like to be kept in the loop.
  • gdb

Subject: wired article

Greg Brockman to Elon Musk, (cc: Sam Teller) - Mar 21, 2016 12:53 AM

Hi Elon,
I was interviewed for a Wired article on OpenAI, and the fact checker sent me some questions. Wanted to sync with you on two in particular to make sure they sound reasonable / aligned with what what you'd say:
Would it be accurate to say that OpenAI is giving away ALL of its research?
At any given time, we will take the action that is likely to most strongly benefit the world. In the short term, we believe the best approach is giving away our research. But longer-term, this might not be the best approach: for example, it might be better not to immediately share a potentially dangerous technology. In all cases, we will be giving away all the benefits of all of our research, and want those to accrue to the world rather than any one institution.
Does OpenAI believe that getting the most sophisticated AI possible in as many hands as possible is humanity's best chance at preventing a too-smart AI in private hands that could find a way to unleash itself on the world for malicious ends?
We believe that using AI to extend individual human wills is the most promising path to ensuring AI remains beneficial. This is appealing because if there are many agents with about the same capabilities they could keep any one bad actor in check. But I wouldn't claim we have all the answers: instead, we're building an organization that can both seek those answers, and take the best possible action regardless of what the answer turns out to be.
Thanks!
  • gdb

Elon Musk to Greg Brockman, (cc: Sam Teller) - Mar 21, 2016 6:53:47 AM

Sounds good

Subject: Re: Maureen Dowd

Sam Teller received this email from Alex Thompson and forwards it to Elon Musk - April 27, 2016 7:25 AM

Hi Sam,
I hope you are having a great day and I apologize for interrupting it with another question. Maureen wanted to see if Mr. Musk had any reaction to some of Mr. Zuckerberg's public comments since their interview. In particular, his labelling of Mr. Musk as "hysterical" for his A.I. fears and lectured those who "fearmonger" about the dangers of A.I.. I have included more details below of Mr. Zuckerberg's comments.
Asked in Germany recently about Musk’s forebodings, Zuckerberg called them “hysterical’’ and praised A.I. breakthroughs, including one system he claims can make cancer diagnoses for skin lesions on a mobile phone with the accuracy of “the best dermatologist.’’
“Unless we really mess something up,’’ he said, the machines will always be subservient, not “superhuman.”
“I think we can build A.I. so it works for us and helps us...Some people fearmonger about how A.I. is a huge danger, but that seems farfetched to me and much less likely than disasters due to widespread disease, violence, etc.’’ Or as he put his philosophy at an April Facebook developers conference: “Choose hope over fear.’’
  • -
Alex Thompson
The New York Times

Elon Musk to Sam Teller - Apr 27, 2016 12:24 PM

History unequivocally illustrates that a powerful technology is a double-edged sword. It would be foolish to assume that AI, arguably the most powerful of all technologies, only has a single edge.
The recent example of Microsoft's AI chatbot shows how quickly it can turn incredibly negative. The wise course of action is to approach the advent of AI with caution and ensure that its power is widely distributed and not controlled by any one company or person.
That is why we created OpenAI.

Subject: MSFT hosting deal

Sam Altman to Elon Musk, (cc: Sam Teller) - Sep 16, 2016 2:37 PM

Here are the MSFT terms. $60MM of compute for $10MM, and input from us on what they deploy in the cloud. LMK if you have any feedback.
Sam
Microsoft/OpenAI Terms[2]

Elon Musk to Sam Altman, (cc: Sam Teller) - Sep 16, 2016 3:10 PM

This actually made me feel nauseous. It sucks and is exactly what I would expect from them.
Evaluation, Evangelization, and Usage of CNTK v2, Azure Batch and HD-Insight: OpenAI will evaluate CNTK v2, Azure Batch, and HD-Insight for their research, provide feedback on how Microsoft can improve these products. OpenAI will work with Microsoft to evangelize these products to their research and developer ecosystems, and evangelize Microsoft Azure as their preferred public cloud provider. At their sole discretion, and as it makes sense for their research, OpenAI will adopt these products
Let’s just say that we are willing to have Microsoft donate spare computing time to OpenAI and have that be known, but we want do any contract or agree to “evangelize”. They can turn us off at any time and we can leave at any time.

Sam Altman to Elon Musk, (cc: Sam Teller) - Sep 16, 2016 3:33 PM

I had the same reaction after reading that section and they've already agreed to drop.
We had originally just wanted space cycles donated but the team wanted more certainty that capacity will be available. But I'll work with MSFT to make sure there are no strings attached

Elon Musk to Sam Altman, (cc: Sam Teller) - Sep 16, 2016

We should just do this low key. No certainty either way. No contract.

Sam Altman to Elon Musk, (cc: Sam Teller) - Sep 16, 2016 6:45 PM

ok will see how much $ I can get in that direction.

Sam Teller to Elon Musk - Sep 20, 2016 8:05 PM

Microsoft is now willing to do the agreement for a full $50m with “good faith effort at OpenAI's sole discretion” and full mutual termination rights at any time. No evangelizing. No strings attached. No looking like lame Microsoft marketing pawns. Ok to move ahead?

Elon Musk to Sam Teller - Sep 21, 2016 12:09 AM

Fine by me if they don't use this in active messaging. Would be worth way more than $50M not to seem like Microsoft's marketing bitch.

Subject: Bi-weekly updates 📎

Ilya Sutskever to: Greg Brockman, [redacted], Elon Musk - Jun 12, 2017 10:39 PM

This is the first of our bi-weekly updates. The goal is to keep you up to date, and to help us make greater use from your visits.
Compute:
  • Compute is used in two ways: it is used to run a big experiment quickly, and it is used to run many experiments in parallel.
  • 95% of progress comes from the ability to run big experiments quickly. The utility of running many experiments is much less useful.
  • In the old days, a large cluster could help you run more experiments, but it could not help with running a single large experiment quickly.
  • For this reason, an academic lab could compete with Google, because Google's only advantage was the ability to run many experiments. This is not a great advantage.
  • Recently, it has become possible to combine 100s of GPUs and 100s of CPUs to run an experiment that's 100x bigger than what is possible on a single machine while requiring comparable time. This has become possible due to the work of many different groups. As a result, the minimum necessary cluster for being competitive is now 10–100x larger than it was before.
  • Currently, every Dota experiment uses 1000+ cores, and it is only for the small 1v1 variant, and on extremely small neural network policies. We will need more compute to just win the 1v1 variant. To win the full 5v5 game, we will need to run fewer experiments, where each experiment is at least 1 order of magnitude larger (possibly more!).
  • TLDR: What matters is the size and speed of our experiments. In the old days, a big cluster could not let anyone run a larger experiment quickly. Today, a big cluster lets us run a large experiment 100x faster.
  • In order to be capable of accomplishing our projects even in theory, we need to increase the number of our GPUs by a factor of 10x in the next 1–2 months (we have enough CPUs). We will discuss the specifics in our in-person meeting.
Dota 2:
  • We will solve the 1v1 version of the game in 1 month. Fans of the game care about 1v1 a fair bit.
  • We are now at a point where a single experiment consumes 1000s of cores, and where adding more distributed compute increases performance.
Rapid learning of new games:
  • Infra work is underway
  • We implemented several baselines
  • Fundamentally, we're not where we want to be, and are taking action to correct this.
Robotics:
  • Current status: The HER algorithm (https://www.youtube.com/watch?v=Dz_HuzgMzxo) can learn to solve many low-dimensional robotics tasks that were previously unsolvable very rapidly. It is non-obvious, simple, and effective.
  • The above will be deployed on the robotic hand: [Link to Google Drive] [this video is human controlled, not algorithmic controlled. Need to be logged in to the OpenAI account to see the video].
Self play as a key path to AGI:
  • Self play in multiagent environments is magical: if you place agents into an environment, then no matter how smart (or not smart) they are, the environment will provide them with the exact level of challenge, which can be faced only by outsmarting the competition. So for example, if you have a group of children, they will find each other's company to be challenging; likewise for a collection of super intelligences of comparable intelligence. So the "solution" to self-play is to become more and more intelligent, without bound.
  • Self-play lets us get "something out of nothing." The rules of a competitive game can be simple, but the best strategy for playing this game can be immensely complex. [motivating example: https://www.youtube.com/watch?v=u2T77mQmJYI].
  • Training agents in simulation to develop very good dexterity via competitive fighting, such as wrestling. Here is a video of ant-shaped robots that we trained to struggle: <redacted>
We have a few more cool smaller projects. Updates to be presented as they produce significant results.

Elon Musk to: Ilya Sutskever, (cc: Greg Brockman, [redacted]) - Jun 12, 2017 10:52 PM

Thanks, this is a great update.

Elon Musk to: Ilya Sutskever, (cc: Greg Brockman, [redacted]) - Jun 13, 2017 10:24 AM

Ok. Let's figure out the least expensive way to ensure compute power is not a constraint...

Subject: The business of building AGI 📎

Ilya Sutskever to: Elon Musk, Greg Brockman - Jul 12, 2017 1:36 PM

We usually decide that problems are hard because smart people have worked on them unsuccessfully for a long time. It’s easy to think that this is true about AI. However, the past five years of progress have shown that the earliest and simplest ideas about AI — neural networks — were right all along, and we needed modern hardware to get them working.
Historically, AI breakthroughs have consistently happened with models that take between 7–10 days to train. This means that hardware defines the surface of potential AI breakthroughs. This is a statement about human psychology more than about AI. If experiments take longer than this, it’s hard to keep all the state in your head and iterate and improve. If experiments are shorter, you’ll just use a bigger model.
It’s not so much that AI progress is a hardware game, any more than physics is a particle accelerator game. But if our computers are too slow, no amount of cleverness will result in AGI, just like if a particle accelerator is too small, we have no shot at figuring out how the universe works. Fast enough computers are a necessary ingredient, and all past failures may have been caused by computers being too slow for AGI.
Until very recently, there was no way to use many GPUs together to run faster experiments, so academia had the same “effective compute” as industry. But earlier this year, Google used two orders of magnitude more compute than is typical to optimize the architecture of a classifier, something that usually requires lots of researcher time. And a few months ago, Facebook released a paper showing how to train a large ImageNet model with near-linear speedup to 256 GPUs (given a specially-configured cluster with high-bandwidth interconnects).
Over the past year, Google Brain produced impressive results because they have an order of magnitude or two more GPUs than anyone. We estimate that Brain has around 100k GPUs, FAIR has around 15–20k, and DeepMind allocates 50 per researcher on question asking, and rented 5k GPUs from Brain for AlphaGo. Apparently, when people run neural networks at Google Brain, it eats up everyone’s quotas at DeepMind.
We're still missing several key ideas necessary for building AGI. How can we use a system's understanding of “thing A” to learn “thing B” (e.g. can I teach a system to count, then to multiply, then to solve word problems)? How do we build curious systems? How do we train a system to discover the deep underlying causes of all types of phenomena — to act as a scientist? How can we build a system that adapts to new situations on which it hasn’t been trained on precisely (e.g. being asked to apply familiar concepts in an unfamiliar situation)? But given enough hardware to run the relevant experiments in 7–10 days, history indicates that the right algorithms will be found, just like physicists would quickly figure out how the universe works if only they had a big enough particle accelerator.
There is good reason to believe that deep learning hardware will speed up 10x each year for the next four to five years. The world is used to the comparatively leisurely pace of Moore’s Law, and is not prepared for the drastic changes in capability this hardware acceleration will bring. This speedup will happen not because of smaller transistors or faster clock cycles; it will happen because like the brain, neural networks are intrinsically parallelizable, and new highly parallel hardware is being built to exploit this.
Within the next three years, robotics should be completely solved, AI should solve a long-standing unproven theorem, programming competitions should be won consistently by AIs, and there should be convincing chatbots (though no one should pass the Turing test). In as little as four years, each overnight experiment will feasibly use so much compute capacity that there’s an actual chance of waking up to AGI, given the right algorithm — and figuring out the algorithm will actually happen within 2–4 further years of experimenting with this compute in a competitive multiagent simulation.
To be in the business of building safe AGI, OpenAI needs to:
  • Have the best AI results each year. In particular, as hardware gets exponentially better, we’ll have dramatically better results. Our DOTA and Rubik’s cube projects will have impressive results for the current level of compute. Next year’s projects will be even more extreme, and what’s realistic depends primarily on what compute we can access.
  • Increase our GPU cluster from 600 GPUs to 5000 GPUs ASAP. As an upper bound, this will require a capex of $12M and an opex of $5–6M over the next year. Each year, we’ll need to exponentially increase our hardware spend, but we have reason to believe AGI can ultimately be built with less than $10B in hardware.
  • Increase our headcount: from 55 (July 2017) to 80 (January 2018) to 120 (January 2019) to 200 (January 2020). We’ve learned how to organize our current team, and we’re now bottlenecked by number of smart people trying out ideas.
  • Lock down an overwhelming hardware advantage. The 4-chip card that <redacted> says he can build in 2 years is effectively TPU 3.0 and (given enough quantity) would allow us to be on an almost equal footing with Google on compute. The Cerebras design is far ahead of both of these, and if they’re real then having exclusive access to them would put us far ahead of the competition. We have a structural idea for how to do this given more due diligence, best to discuss on a call.
2/3/4 will ultimately require large amounts of capital. If we can secure the funding, we have a real chance at setting the initial conditions under which AGI is born. Increased funding needs will come lockstep with increased magnitude of results. We should discuss options to obtain the relevant funding, as that’s the biggest piece that’s outside of our direct control.
Progress this week:
  • We’ve beat our top 1v1 test player (he’s top 30 in North America at 1v1, and beats the top 1v1 player about 30% of the time), but the bot can also be exploited by playing weirdly. We’re working on understanding these exploits and cracking down on them.
    • Repeated from Saturday, here’s the first match where we beat our top test player: https://www.youtube.com/watch?v=FBoUHay7XBI&feature=youtu.be&t=345
    • Every additional day of training makes the bot stronger and harder to exploit.
  • Robot getting closer to solving Rubik’s cube.
    • The improved cube simulation teleoperated by a human: <redacted>.
  • Our defense against adversarial examples is starting to work on ImageNet.
    • We will completely solve the problem of adversarial examples by the end of August.
████████████████████████████████████
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
███████████████████████████████████████████████████████████████████████████████████████████████████████████████

iMessages on OpenAI for-profit 📎

Shivon Zilis to: Greg Brockman - Jul 13, 2017 10:35 PM

How did it go?

Greg Brockman to: Shivon Zilis - Jul 13, 2017 10:35 PM

Went well!
ocean: agreed on announcing around the international; he suggested playing against the best player from the winning team which seems cool to me. I asked him to call <redacted> and he said he would. I think this is better than our default of announcing in advance we’ve beaten the best 1v1 player and then having our bot playable at a terminal at TI ████ ██████████ █████████████████ ██.
gpus: said do what we need to do
cerebras: we talked about the reverse merger idea a bit. independent of cerebras, turned into talking about structure (he said non-profit was def the right one early on, may not be the right one now — ilya and I agree with this for a number of reasons). He said he’s going to Sun Valley to ask <redacted> to donate.

Shivon Zilis to: Greg Brockman - Jul 13, 2017 10:43 PM

<redacted> and others. Will try to work it for ya.

© Chris Song 2021 - 2025