Feedback loops with the outside world

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Do we tend to feel safer when we’re disconnected?

“Its “environment” is purely computational: it processes a static dataset under a predefined prior and has no feedback loop with the outside world.”

“The crucial advantage of using a Scientist AI in this research program is that we would be able to trust it, whereas if we try to use an untrusted agentic AI to help us figure out how to build future and supposedly safe ASI, it may fool us into building something that would advance its goals and endanger us, for example by proposing code with back-doors that we are not able to detect. One may however ask why we would want to build ASI at all, if we do not have the answers to these questions. One motivation is that a safe agentic ASI may be necessary to defend humanity against a rogue system. Such a system could emerge if hostile actors transform a non-agentic AI into a dangerous agent, or if an uncontrolled ASI is exploited as a geopolitical threat. Regulations and treaties can reduce these risks but cannot remove them entirely. Alternative measures must be in place to ensure that any ASI developed is both safe and able to protect humanity.”

Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., ... & Williams-King, D. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?. arXiv preprint arXiv:2502.15657.

The necessity of construct and external validity for deductive causal inference

It would seem as though a wider machine intelligence would need to be connected though?

“We show that ignoring construct and external validity within identification strategies undermines the Credibility Revolution’s own goal of understanding causality deductively. Without assumptions regarding construct validity, one cannot accurately label the cause or outcome. Without assumptions regarding external validity, one cannot label the conditions enabling the cause to have an effect. If any of the assumptions regarding internal, construct, and external validity are missing, the claim is not deductively supported.”

“But internal validity sheds no light on what were the cause and effect, or why the effect occurred. As a result, internal validity provides no guidance on how the presence of causality shown in a given study or across studies can accumulate into deductively formed knowledge. Instead, in their view, any substantive claims about what was the cause, what was the effect, and what were the enabling conditions can at best be – in their words – “speculative” (see also [20, p. 959]). Without explicit assumptions about construct and external validity, however, such speculations remain tentative in a way that undermines the deductive “credibility” of a causal claim, contradicting the very purpose of the Credibility Revolution.”

“To demonstrate the consequences of neglecting construct and external validity, we develop a framework we call causal specification.6 This framework formalizes assumptions regarding internal, construct, and external validity within a single causal expression and shows that causal deduction requires a rebalancing that equally values all three validities. Within a deductive understanding of causality, internal validity has no special status or lexical priority. The causal specification framework explicitly recognizes the contributions of theory and substantive knowledge to quantitative causal inference, and charts a way forward for social scientists who aim to make deductive causal generalizations.”

“Second, we clarify that external validity requires the correct specification of the conditions (γ) under which the causal generalization operates. Thus, defending a claim of external validity requires evidence or assumptions about what conditions enable the treatment to produce its effects. According to the traditional definition, a claim is externally valid if it generalizes across settings. We say a claim is externally valid to the extent one has accurately specified why or how the effect generalizes across settings. This means specifying the range of conditions in settings γ across which the effect generalizes. Hence, we define external validity as the correct specification of the conditions that enable/disable or augment/weaken a causal effect (for a similar definition, see refs [17,33]). External validity is present when that specification is true – i.e., when γ is correctly labeled.”

Esterling, K. M., Brady, D., & Schwitzgebel, E. (2025). The necessity of construct and external validity for deductive causal inference. Journal of Causal Inference, 13(1), 20240002.

Reconstructing Trust in the Knowledge Economy

Automotive industry in the 1990’s or machine intelligence in the 2020’s?

“The application of ‘exit’ and ‘voice’ to supplier relations captures the orientation of the parties towards their relationship (as opposed to just a single transaction) and frames the issues of information sharing that crucially affect the relationship beyond transaction-specific costs. ‘Exit’ is characterized by the creation and exploitation of information asymmetries by both parties, even when the relationship endures over long periods of time. ‘Voice’ requires shared norms of reciprocity that balance the willingness of the customer to undertake investments in the supplier’s capabilities against the supplier’s responsibilities to invest in new technology and capacity. As we discuss below, both exit and voice have been profit-maximizing strategies in the past, depending on such conditions as firm strategy and market structure.”

Heckscher, C., & Adler, P. (2006). Reconstructing Trust in the Knowledge Economy. Adler P Heckscher C (eds.), 142-155.

https://faculty.wharton.upenn.edu/wp-content/uploads/2012/05/MacDuffie_FirmCollab_Chap10.pdf

Terence Tao - Machine-Assisted Proofs (February 19, 2025)

The ragged edge

“I think we need to change, or at least broaden, our workflows in the ways we do mathematics.” (44:53)

https://www.youtube.com/watch?v=5ZIIGLiQWNM

Reader Feedback

“I have been reading your newsletter, and I am finding it something I very much look forward to.”

Footnotes

Last week marked the switch from pure problem discovery to problem testing. What stays the same is the extreme curiosity and the double clicking on the odd, wild, and divergent ideas you share. These conversations can be extraordinarily strange and I’m thankful for all your time and honesty. It’s been a treat. I heard a lot. And for most of the problems, there is nothing I will, and in most instances, can do about them.

But there is a sub-set of problems that I will and can do something about!

“tHere hAs gOt tO bE a bEtTar wAy”

What changes in problem testing? There’s a stated value proposition and a direct ask. I state assumptions and listen for the feedback. I make a direct ask and I listen for the feedback. Ping! Pong. Ping! Pong.

I might be pinging you soon.

Never miss a single issue

Be the first to know. Subscribe now to get the gatodo newsletter delivered straight to your inbox!