What does AI "pacing" actually mean?

Hi!

It's a strange time to be talking about tech because AI not only dominates the conversation but has put a pretty dark spin on it recently.

AI doom has been the elephant in the room for a long time, and it's a good thing that it's finally come to the fore and that the big companies, governments, even royalty are being forced to acknowledge it.

Let's just hope that we get some level-headed agreement on slowdown, regulations, alignment, etc.

In the meantime, I thought the most useful thing for me to do is break down the different arguments, so you've got a bit of clarity in your conversations with friends and family about this.

There are also the usual updates on interesting stuff that's crossed my feed.

This week’s top stories are:

“AI pacing” is the phrase doing the rounds. The basic idea is to slow the development of the most powerful AI when safety work can't keep up, rather than treating maximum speed as the only possible setting.

Anthropic boss Dario Amodei has set out a proposal. It's about the frontier: the systems pushing beyond what today's AI can do. About not letting them run away so that we lose control.

There are a few arguments tangled together here.

1. The optimistic case: agree to slow down together.

It can't just be one company slowing itself down, allowing its competitors to just keep going. Amodei proposes independent evaluators inside the labs, coordination between leading companies, and ultimately government agreements that include China, so that absolutely everyone's on the same page.

The hope is that everyone gains from avoiding a race nobody can safely control. The difficulty is getting competitors and governments to agree on what counts as dangerous, how to verify compliance, and what happens when someone cheats (very possible!). A proposal is a long way from an enforceable global deal.

2. The government case: this shouldn't be the manufacturers' decision.

Imagine a food manufacturer or car company openly discussing the possibility that its next product could kill huge numbers of people. We'd expect governments to investigate, and stop it reaching the market, right?

Why do we seem so comfortable leaving AI companies to lead the conversation about how dangerous their own products might be, and how quickly they should build them?

To be fair, Amodei's proposal explicitly includes independent scrutiny and government action. It isn't simply “trust us”. But I'd like to see governments setting and enforcing the boundaries, rather than waiting for the industry to agree on boundaries it finds acceptable.

3. The argument against slowing down: you could lose the benefits and the race.

Opponents worry that restrictions would delay useful technology and hand an advantage to countries that refuse to participate. Nvidia's Jensen Huang has pushed back against the slowdown argument.

Do you really want the same people who actively avoid and evade regulation to end up being the people with the most powerful AI?

The argument against slowing down is that we just need to make sure the good guys have the best tech. And at least we know who they are, and we can demand transparency from them.

4. The sceptical case: how solid are the catastrophe claims?

Some researchers think the dramatic, doomy predictions are a bit short on hard evidence. You have to remember that the current storm around AI doom was triggered by an offhand tweet quoting this 10% risk figure. There's so much more we'd need to know. It's not a simple topic.

The Guardian's explainer is useful on these disagreements. There isn't an agreed scientific probability that tells us exactly how dangerous the next system will be.

I’m building a site to lay out clearly where the biggest 35 voices in AI stand. I’ll link to it next week when it’s finished!

Results in next week’s email.

The AI safety debate reached Dumfries House in Scotland today. King Charles brought tech leaders together in Ayrshire, including Nvidia's Jensen Huang, DeepMind co-founder Sir Demis Hassabis and OpenAI's chief financial officer Sarah Friar. Anthropic was represented too, alongside UK AI minister Kanishka Narayan.

In his published speech, Charles warned of the “existential dangers” of AI falling into the wrong hands. He called for effective controls and international cooperation, while recognising the technology's potential to improve and save lives.

The responses showed some of the differences already running through this debate. Huang said companies should hold back products that aren't safe enough and keep working on them. Hassabis argued for a middle course that gives people time to address the risks. He also suggested AI matching or exceeding human abilities across many tasks could be only a few years away. That's his forecast, not an agreed deadline.

The Ditchley Foundation facilitated the discussions. According to the Royal Household's account, delegates explored whether they could establish shared principles for AI. It doesn't announce a binding agreement, a development pause or new regulation.

The practical test is what follows the speeches. Who decides whether a system is safe enough, who checks, and who can stop an unsafe release? Getting influential people into the same room is a useful start. Those are the questions that still need answers.

Claude is merging Cowork and chat, so you can ask a quick question or hand over a bigger task in the same conversation. The merger is rolling out to Pro and Max users over the coming weeks.

It's also adding Docs and Slides, with direct editing, comments and presentations you can export to PowerPoint or PDF. Those tools are in beta on paid plans, with Enterprise access controlled by admins.

I've been waiting for this. Making people decide which part of the product to use before they can explain what they want just adds confusion. Most people don't care which mode handles a task. They care whether the thing gets done.

Documents and presentations are also exactly the kind of everyday knowledge work that makes this useful. There's a clear job to do, and a recognisable output at the end of it.

I'll do a video breakdown when I get access!

OpenAI is testing Sponsored Agents: click an ad in ChatGPT and you can start a conversation with an agent representing that business.

It's a separate, labelled sponsored conversation, distinct from ChatGPT's own answers. The test currently involves selected advertisers in the US.

For marketers, the appeal is obvious. Someone could ask whether a product suits their needs, work through objections and arrive at your website already much closer to a decision. An ad that can qualify a potential customer before the click through to your site.

The question is whether people actually want that conversation. Or even ads. Remember that ChatGPT was free to use and ad-free for a long time, and it's yet to be seen if the user base will accept and interact with advertising here. Remember that one of the top use cases for ChatGPT is therapy, and now your therapist's getting paid to show you ads? Not sure that's going to go down well.

Apple's new Siri AI has started rolling out, with more natural conversations, awareness of what's on your screen and the ability to draw on information in your messages, emails and photos.

Apple's example is pretty ordinary: find something a relative mentioned in a message, locate the relevant recipe in an email, then add the ingredients to your shopping list. That sort of small, fiddly task is a much more understandable pitch than another claim about how intelligent a model is.

I haven't tried it yet, so I can't tell you whether this is finally the Siri people have been waiting for. The interesting test is whether it can reliably join up the things already on your phone, without making you explain everything from scratch.

It starts as an English-language beta on compatible devices with the new software releases. There are regional restrictions, including no initial iPhone, iPad or Apple Watch rollout in the EU. Some server-based features have daily limits, with paid expanded access planned.

ChatGPT turns four in November. I would have thought more people would be using AI in more sophisticated ways by now, but this data and my experience say that is very much not the case.

Gallup's May figures show that 52% of US workers used AI at least occasionally, but only 15% used it daily. That's a big gap between trying something and making it part of your working day.

More recent research shows how company adoption figures can hide that gap. The New York Fed's August business surveys, published on 1 September, found AI use spreading across businesses in New York and northern New Jersey. But among the businesses using it, the median share of employees doing so was just 17% in service firms and 7% in manufacturing. A company can tick the “we use AI” box while most of its people aren't using it themselves.

I'm hearing from a lot of people that company adoption drives aren't going well. As an educator, I'm at the coalface of this, and I'm seeing much less of the transformation we imagined.

The cliché I hear all the time is that senior management are saying “use AI more”. But there's no real instruction on how and for what. That leaves people with a mixture of confusion, fear and the suspicion that they're being asked to automate themselves out of a job, and so (I suspect) they resist it further.

The support isn't necessarily keeping up, either. In an Eagle Hill/Ipsos survey released on 8 September, only 31% of leaders said employee development for AI was built into broader business and workforce planning. This was an August survey of 306 senior leaders at large US companies where AI adoption was already established.

I think we're often imagining the wrong jobs for AI.

Honestly, there's not a huge amount of my work that I've replaced with AI. There is, however, a lot I can now do that I couldn't have done before, and would never have paid someone else to do.

It's enhanced my work. It's made more things possible. But if I hadn't had time to explore those things and thought it was just a copywriting or web searching tool, maybe I wouldn't be that excited by it.

Meta is reportedly developing camera-free smart glasses, codenamed Luna. The reported design has microphones and speakers for talking to its AI services, with a possible unveiling at Meta Connect next week.

I find it staggering that people thought putting cameras on your face was a good idea.

I’m glad the product exists though: we needed to have the conversation about wearable cameras in public, at some point. The person choosing to record isn't the only person affected. Everyone around them becomes part of that decision, without any way to opt out.

So they’re a bad idea… but we do talk to our tech more than ever, and glasses with a built in microphone, and therefore an AI assistant sounds like a fair idea. But I’d rather have my regular RayBans and Airpods thanks 👍

Blimey, that tweet really had an impact on the public perception of AI risk.

A POLITICO/Public First poll found that 63% of US adults saw at least a moderate risk that advanced AI could destroy humanity. That is a measure of people's concern, not a 63% probability of extinction. In a separate question, 48% favoured a pause in development, compared with 31% who wanted it to continue.

The poll surveyed 2,064 US adults on 13–15 September. It doesn't tell us which headline, post or argument shaped their answers. It’s just interesting that this AI doom thing has properly hit the mainstream now.

Phew! A very AI-heavy issue this week. There will be more marketing, media, and tech stories next week.

Thanks for reading 🙂

Will