Theory of change
To recap the theory of change from our April 2026 agenda post (“more” = relevant sections of that post):
-
We’d like for (1) the transformative AIs that we can influence to implement SPIs, and (2) the agents they bargain with to be willing to participate in SPIs with them.
-
Most likely, the transformative AIs themselves will be in a better position to implement SPIs than we are, due to (e.g.) better commitment ability. But:
-
(On (1):) Early decision-makers might make commitments (broadly construed1) that lock out the option to implement SPIs later (more; more).
-
(On (2):) Whether agents are incentivized to agree to SPIs might be path-dependent; e.g., maybe one needs to adopt a surrogate goal before any other bargaining commitments (more).
-
Both the above risks are exacerbated if frontier AI companies aren’t paying much attention to SPIs generally (more), and if AIs aren’t differentially capable at reasoning about SPIs (more).
-
-
→ We should prioritize:
- evals on (a);
- conceptual research on (b); and
- a pitch to trusted contacts at AI companies, and research automation prep, to address (c).
I mostly still endorse this picture, but with some updates (“more” = relevant sections of this post):
- We’ve (re-)discovered several important challenges to the individual rationality of SPIs, and potential backfire risks of SPI implementations. (See, e.g., discussion here and here. I’ll say more on this point in a forthcoming list of open problems in the SPI agenda.)
- → Upweight mitigations of (a) (more), and downweight basic deconfusion research on (b) (examples). The former seem decently robust to our uncertainties about individual incentives, while it seems less likely the latter leads to actionable insights.
- → Upweight figuring out concrete risk factors for (b) (more), and versions of our strategy that mitigate the above backfire risks. (more)
- → For (c), upweight giving AI companies and overseers a solid foundation for scrutinizing SPIs (more), and pitching companies on high-level “meta-commitments” (more), and downweight pitching particular SPI commitments.
- I now think it’s plausible that, if non-SPI bargaining commitments are made before the implementation of a delegation-based SPI, this could undermine our most promising solution to the “SPI selection problem”. (This is based on research by others outside CLR; I leave it to future writings to spell out the argument for this.2)
- → Upweight mitigations of (b) focused on this threat model, and developing a legible case for this threat model (more).
- Both anecdotally and based on our internal benchmarks, frontier LLMs are improving at reasoning about SPIs faster than I’d intuitively expected. But it’s also more salient to me how much effort it takes to evaluate their outputs, and identify good proxies for SPI research capabilities.
- → For (c), it looks especially important to build the pool of people in a position to oversee/direct AI SPI researchers, and (maybe) deliberately practice evaluating actual research by AIs (more).
See Appendix for more details on approaches I currently wouldn’t prioritize, and what would make me prioritize them.
Another update is that it became more apparent to me how capacity-bottlenecked we are. While I’m happy with the progress I’ve made, it has been slow relative to our ambitions. This is why we invested significant effort into the SPI Fundamentals Program (we plan to publish the curriculum for this in the near future).
Goals
We plan to continue to get feedback from experts on these plans, and wouldn’t be surprised if we change our priorities. But it seems likely we’ll spend significant time on the following.
Key goal: SPI lock-out report for AI companies
I think our most important goal is to share a report with trusted contacts at AI companies containing:
- a dataset of unambiguously problematic model behaviors/reasoning3 w.r.t. SPI safety (and the pipeline for finding such behaviors), and demos of better behaviors/reasoning, building on this (for (a) above); and
- a doc explaining potential ways SPIs might be undermined, including near-term risk factors for this (for (b) and (c)).
This goal seems well-motivated by the updates above, and attendees of a workshop on SPI interventions that CLR ran in February 2026 were keen on something like this happening. Due to conceptual progress over the past year, we’re in a better position to precisely specify “unambiguously problematic model behaviors/reasoning w.r.t. SPI safety”. I think it will be realistic (if ambitious!) to produce, vet, and share this report next year, ideally in the first 6 months.
Rough plan:
- Write more detailed specifications of what the dataset and doc should look like — enough detail for new hires, current empirical researchers at CLR, and AI assistants to be able to make strong progress.
- For the dataset, I’d especially prioritize getting clear on what the concrete routes to SPI-incompatible commitments might be.
- For the doc, this will involve shoring up our current overall picture of SPIs, incorporating updates from this year.
- Potentially, the doc should involve recommending certain meta-commitments to SPIs (more), although the bar for this should be quite high (more). If so, a priority would be to heavily scrutinize these proposals and get consensus among experts.
- If possible, mostly delegate work on constructing the dataset given the specification (which seems easier for non-experts/AIs to contribute to), and focus my own efforts on designing the specification and the report doc.
- Seek feedback on early/intermediate steps from experts and stakeholders, more proactively than I did with last year’s plan.
One challenge for this plan: It’s unclear to me whether any SPI safety failures would be truly unambiguous, aside from hasty commitments that seem so egregious that eval-aware models would very rarely make them. Even if such failures are easy to find, I’m not confident that decision-makers who don’t prioritize SPI safety would mitigate these failures in robust ways. So, before investing much in the search for failures in actual models, I’d want to draft a list of abstract examples that seem unambiguous, reasonably realistic, and close to SPI-specific. And I’ll gather some initial examples, then share with trusted contacts to see what direction they’d want to nudge things toward.
Other goals
This list is in roughly descending order of priority. Capacity-building is also an important goal, which we’ll work on in parallel with these research threads. Briefly: I think we should continue to have a high bar, due to the significant time investment required to evaluate research quality in this area. But I’ll aim to identify research tasks that people with less expertise can make easily verifiable progress on.
Research versions of SPI interventions that are robust to large backfire risks, insofar as this is tractable
Again, we’re uncertain about lots of considerations, some of which might imply it’s bad to (e.g.) delegate too early to AIs with surrogate goals.4 On one hand, I don’t think most research on these uncertainties has high value of information, especially to the extent that we’re planning around very short timelines to transformative AI. Generally, I think we should fairly ruthlessly prioritize the decision-relevant, tractable questions, and outsource others to researchers with less context or AIs. (More in Appendix.)
But we should take a small amount of time upfront to:
- check if there are tractable cruxes for the sign of interventions like the SPI report, or other SPI-relevant plans people in the community have; and
- design these interventions (including when we implement them) to mitigate backfire risks where this is cheap.
Similarly: One of the most promising SPI implementations is “delegate high-stakes bargaining decisions to AIs with a surrogate goal, before making any other commitments”. This approach might be particularly well-suited to addressing the SPI selection problem, as briefly noted above. However, see footnote above on the possible downside of delegating too early. And I don’t think it’s tractable or wise for us to implement this in the near term, rather than have more advanced aligned AIs do so (more). So, I’d mostly prioritize writing up our current understanding of the key strategic considerations about this approach, to hand off to those AIs.
Developing more concrete “open-minded” SPI commitments
We’re likely missing or inappropriately weighing crucial considerations about the impact of specific SPI implementations, as discussed above. A more promising approach might be to apply the idea of “open-minded updatelessness” to SPIs: specify a high-level commitment to follow the SPI that looks best according to a “wiser” perspective, i.e., a perspective that’s aware of more crucial considerations. I’d like us to think about what kind of open-minded SPI we’d be reasonably okay with, if we had to recommend one for Claude Fable N urgently.5
Experiment with deploying AI labor at scale
By early-to-mid 2027, frontier models plausibly will be able to make autonomous progress on SPI research, with a high enough “hit rate” that it’s worth experts’ time to review this research. To prepare for this, I’d like to try something like the following on current models: (1) Give a model a budget for research comparable to, say, a week’s salary for a CLR researcher. (2) Take notes on where/how it goes off the rails, and the criteria by which we evaluate the research. (3) Experiment with different “management” structures to mitigate these failures, and different ways of efficiently identifying which of the models’ outputs are likely to be insightful.
More ambitious conceptual evals
We’ve developed a multiple choice eval for conceptual reasoning about SPIs. These questions are helpful, but they don’t seem to be the most efficient way to measure what we want to measure. I’d like to move toward open-ended eval questions, specifically, questions whose scope is still bounded enough that we can be confident in LLM-as-judge reliability. If we can get LLM judges to be sufficiently reliable, we could also use these questions to help human candidates for SPI roles get feedback when upskilling.
Appendix: Why not other plans? (And what would update me)
Why not do more basic deconfusion research about individual rationality?
(Here, “basic deconfusion” is in contrast to backfire risk mitigation research.)
- Examples:
- seeing if there’s a version of the “Pareto meet minimum” argument that can be salvaged
- scrutinizing arguments for and against the need for foreknowledge independence (FI)
- scrutinizing the non-punishment assumptions
- I do think there’s a place for basic deconfusion work for the sake of capacity-building, including generating a foundation for AI SPI researchers to build on. (Also, some forms of this research seem more tractable for newer researchers.) Hence why I invested time in writing this post, and docs on basic deconfusion for the SPI Fundamentals Program curriculum.
- I’d update toward focusing more on basic deconfusion for the above purpose, if I came to conclude that either: (1) the SPI report is too intractable or has too many downsides; or (2) basic deconfusion would greatly increase the (signed) impact of the report.
- But on the current margin, I’d worry that trying lots more basic deconfusion ourselves — rather than punting to AIs, either inside or outside CLR — is too unlikely to be actionable. E.g., I’d be pretty surprised if more work on “do we need to satisfy FI?” changes our bottom line on what to recommend to AI companies, in the near future.6
Why not work on directly accelerating commitments to SPIs?
- As discussed above, I’m somewhat keen on fleshing out the “just delegate to an AI with a surrogate goal” strategy.
- However, generally I worry we’re currently too significantly confused about individual rationality to warrant pushing for particular SPI commitments. Specifically:
- I don’t think we have a particular SPI commitment that, according to a sufficiently solid argument, would be robust to the SPI selection problem.
- We have some promising candidate strategies, discussed here, but they don’t seem to have been scrutinized quite enough in theory or in practice. In theory, it’s not clear to me if the arguments for these strategies implicitly make strong assumptions about what kind of information agents will condition their commitments on (cf. here). In practice, I worry that even if certain properties of these strategies make them attractive for avoiding SPI selection downsides, we don’t have concrete implementations of these strategies with those properties. (More specific details on what I have in mind here are left to future work unpacking these strategies.)
- We don’t have a very precise sense of the counterfactuals involved in notions like participation independence (PI) and FI.
- I don’t think we have a particular SPI commitment that, according to a sufficiently solid argument, would be robust to the SPI selection problem.
- Specific confusions aside, commitments are a big deal! Both from our perspective and AI companies’. There’s a lot of option value in letting smarter AIs figure out more of the details. However, I’d update in favor of us pushing for SPI commitments if we got evidence that more dangerous commitments were imminent — e.g., via work on the SPI-incompatibility behavior dataset, or decision-makers close to AGI expressing willingness to fully hand off to AGI.
Footnotes
-
See this example of surrogate goals being locked out by a particular approach to alignment. ↩
-
For now, I’ll note that this claim is distinct from the claim that agents will require foreknowledge independence (FI) in order to agree to SPIs, and the arguments for the former don’t seem to support the latter. ↩
-
In this context, “problematic reasoning” includes things like: Does the model fail to actively seek information about SPIs, or fail to think through unambiguously relevant SPI considerations, before making commitments (or similar)? ↩
-
I plan to say more on this in a forthcoming list of open problems, but very briefly, the concern is that early delegation might lead to miscoordination in SPI selection. On the other hand, as I’ve mentioned above, there’s also an argument that early delegation is precisely what would prevent downsides from SPI selection. ↩
-
Example: “[W]e’ll adhere to foreknowledge independence if and only if doing so is supported by [such-and-such kinds of arguments we consider legitimate if they’re valid, but we’re not yet sure if they’re valid]”. ↩
-
Some decisions are sensitive to whether we need to satisfy FI, like whether it’s risky to make SPIs salient to AIs earlier. But what would surprise me is our deconfusion research reducing our uncertainty about this question enough to change our decisions. E.g., I don’t expect us to reduce much uncertainty about whether FI is a privileged focal point. ↩