Who controls what these institutions do?

The 2025 AI Index Report

Whooooooooooooosh!

“In 2023, researchers introduced new benchmarks—MMMU, GPQA, and SWE-bench—to test the limits of advanced AI systems. Just a year later, performance sharply increased: scores rose by 18.8, 48.9, and 67.3 percentage points on MMMU, GPQA, and SWE-bench, respectively. Beyond benchmarks, AI systems made major strides in generating high-quality video, and in some settings, language model agents even outperformed humans in programming tasks with limited time budgets.”

“Global AI optimism is rising—but deep regional divides remain. In countries like China (83%), Indonesia (80%), and Thailand (77%), strong majorities see AI products and services as more beneficial than harmful. In contrast, optimism remains far lower in places like Canada (40%), the United States (39%), and the Netherlands (36%). Still, sentiment is shifting: Since 2022, optimism has grown significantly in several previously skeptical countries, including Germany (+10%), France (+10%), Canada (+8%), Great Britain (+8%), and the United States (+4%).

“Given that most local policymakers support some form of AI regulation, which specific policies do they favor? At 80.4%, the strongest support is for stricter data privacy regulations. In addition, 76.2% support retraining programs for the unemployed, and 72.5% support AI deployment regulations (Figure 8.2.2). In contrast, there is significantly less backing for redistributive measures. Just 33.9% support wage subsidies to offset wage declines and just 24.6% support universal basic income.”

“Business is all in on AI, fueling record investment and usage, as research continues to show strong productivity impacts. In 2024, U.S. private AI investment grew to $109.1 billion—nearly 12 times China’s $9.3 billion and 24 times the U.K.’s $4.5 billion. Generative AI saw particularly strong momentum, attracting $33.9 billion globally in private investment—an 18.7% increase from 2023. AI business usage is also accelerating: 78% of organizations reported using AI in 2024, up from 55% the year before. Meanwhile, a growing body of research confirms that AI boosts productivity and, in most cases, helps narrow skill gaps across the workforce.”

Smaller, Stronger Models: Phi-3-mini (3.8B parameters) matched PaLM's 2022 performance with 142× fewer parameters.

https://hai.stanford.edu/ai-index/2025-ai-index-report

Understanding of Trade

AAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHRG

“This paper sheds new light on two questions: How do people perceive and understand trade and trade policy, and what shapes their support for different trade policies?”

“First, respondents perceive gains from trade as consumers to be vague and unclear but perceive potential losses as workers to be concentrated and salient. Actual and perceived exposure to trade through the labor market is significantly associated with policy views. When respondents are experimentally prompted to think about their benefits from trade as consumers, there is no change in their support for free trade. On the contrary, respondents who are prompted to think about the impacts of trade on their job reduce their support for free trade. In a nutshell, consumer gains matter less than job impacts for views on trade.

“Second, people’s policy views on trade do not only reflect self-interest. Respondents also care about trade’s distributional and efficiency impacts on others and the US economy. The belief that trade leads to efficiency gains is correlated with stronger support for free trade. The belief that trade has adverse distributional impacts is important, too but is modulated by the belief that losers can be compensated with appropriate policies. Respondents who think it is possible to have compensatory transfers support more free trade, even if they believe that it has negative distributional consequences. They also support more compensatory redistribution to help those who have been hurt. These links are confirmed experimentally by showing respondents’ pedagogical videos on the efficiency or distributional impacts of trade.”

“Third, respondents’ experience, as measured by their exposure to trade through their sector, occupation, and local labor market, shapes their policy views directly (through self-interest) and indirectly by influencing their understanding and reasoning about the broader efficiency and distributional impacts of trade. Respondents who are or feel more negatively affected by trade appear to extrapolate from their experience and hold more negative beliefs about the effects of trade on other groups (e.g., the middle class or small businesses), inequality, prices, innovation, and competitiveness. A Gelbach decomposition shows that both the direct and indirect channels matter substantially.”

Stantcheva, S. (2022). Understanding of trade (No. w30040). National Bureau of Economic Research.

https://socialeconomicslab.org/wp-content/uploads/2024/05/understanding_trade.pdf

https://bsky.app/profile/s-stantcheva.bsky.social

What failure looks like

Whomp Whomp

“The stereotyped image of AI catastrophe is a powerful, malicious AI system that takes its creators by surprise and quickly achieves a decisive advantage over the rest of humanity. I think this is probably not what failure will look like, and I want to try to paint a more realistic picture.”

“Right now humans thinking and talking about the future they want to create are a powerful force that is able to steer our trajectory. But over time human reasoning will become weaker and weaker compared to new forms of reasoning honed by trial-and-error. Eventually our society’s trajectory will be determined by powerful optimization with easily-measurable goals rather than by human intentions about the future.”

“As this world goes off the rails, there may not be any discrete point where consensus recognizes that things have gone off the rails.”

“Optimization daemons

We can see natural selection spitting out humans as a special case of "If you dump enough computing power into optimizing a Turing-general policy using a consequentialist fitness criterion, it spits out a new optimizer that probably isn't perfectly aligned with the fitness criterion." After repeatedly optimizing brains to reproduce well, the brains in one case turned into general optimizers in their own right, more powerful ones than the original optimizer, with new goals not aligned in general to replicating DNA.

This could potentially happen anywhere inside an AGI subprocess where you were optimizing inside a sufficiently general solution space and you applied enough optimization power - you could get a solution that did its own, internal optimization, possibly in a way smarter than the original optimizer and misaligned to its original goals.

When heavy optimization pressure on a system crystallizes it into an optimizer - especially one that's powerful, or more powerful than the previous system, or misaligned with the previous system - we could term the crystallized optimizer a "daemon" of the previous system. Thus, under this terminology, humans would be daemons of natural selection.”

“The concern exists whether that reasoning occurs within a single computer, or is implemented in a messy distributed way by a whole economy of interacting agents---whether trial and error takes the form of gradient descent or explicit tweaking and optimization by engineers trying to design a better automated company.”

“The catastrophe might look like a rapidly cascading series of automation failures: A few automated systems go off the rails in response to some local shock. As those systems go off the rails, the local shock is compounded into a larger disturbance; more and more automated systems move further from their training distribution and start failing. Realistically this would probably be compounded by widespread human failures in response to fear and breakdown of existing incentive systems---many things start breaking as you move off distribution, not just ML.”

“One day leaders may find that despite their nominal authority they don’t actually have control over what these institutions do.”

https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like

Reader Feedback

“…’The should’s and ought’s underlining stakeholders’ expectations’ … where do the expectations come from?”

Footnotes

Because Chatham House Rules apply, I’ll summarize and generalize.

In several environments, I noticed an uptick in fear and alienation. These aren’t quite new seeds and not quite old roots. It simply sounded different last week. It was more specific. Grittier. Biting.

In several environments, I noticed an uptick in courage and daring. These aren’t quite new qualities. They’ve been around for awhile. It sounded quite a bit different last week. It was more direct. Calculated. Thoughtful.

Keep at it friends.

Never miss a single issue

Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox