Design Notes

Under the current versioning policy, each new ARGUS release states whether it is major, minor or a patch, and why. Earlier entries remain visible as published. This is why the version history is part of the audit trail rather than a changelog for the curious.

In the Amodei case, the stronger condition happened to hold as well. The test that produced the central finding, the modal constancy check, had been published in ARGUS V5.1.0 in August 2026, before the September essay existed. The test asks whether the same load-bearing proposition is stated strongly where that strength does argumentative work, then narrowed where maintaining it would carry a cost.

One risk of ad hoc textual criticism is a criterion shaped around the text already in front of the analyst. A dated public specification does not make that impossible, but it makes the chronology checkable instead of deniable.

The analysis states what it will accept as adequate support for the genre in hand before the substantive tests are run. In the Amodei case, that meant an identifiable or retrievable source for a supported claim, a stated causal mechanism for a forward-looking one, a commitment determinate enough for a third party to check later, and a declared base and perimeter for a quantitative claim.

The same discipline applies to the numerical controls. The figures treated as probative are declared before calculation, rather than selected afterwards according to which results prove useful.

This does not make the chosen standard correct. A reader may find it too strict or too lax. The point is that the reader can argue with the standard separately from the verdicts it produced.

“Blind” has a specific meaning here.

Once the normative analysis is complete, the publication synthesis is produced in a separate run from that analysis alone, with its own primary summary masked. The analysed object is not visible. Neither are the external source objects, the previous conversation or the translations, and external research is disabled.

The second run is therefore unable to repair the analysis by returning to the original essay, add a fact from a source the analyst omitted, or quietly strengthen a conclusion through a fresh search. Its task is narrower: reconstruct what the text argues, state what the completed analysis finds solid, identify the principal weaknesses and what survives them, preserve the distinction between a strong claim that fails and a weaker one that holds, and introduce no finding that is absent from the supplied analysis.

Keeping the summary in the same run would leave it exposed to the same contextual emphases. Separation does not make the second model impartial. It narrows the dependency and makes it inspectable.

An early reading of the Amodei rendering, before its link annotations had been extracted and incorporated into the audit, treated the essay as essentially unsourced. That reading was wrong.

Once the link layer was examined, the source audit and the assessment of interpretive closure had to be rebuilt. The correction moved the latter from level 3 to level 2, in the essay’s favour. The analysis records that change rather than absorbing it silently.

It also records a bias risk running in the opposite direction from the obvious one. A model produced by Anthropic might defer to Anthropic’s chief executive. It might also overcorrect and become unusually severe in order to display independence. The analysis names both possibilities and states where that second tendency would be expected to appear: inflated severity findings or an unnecessarily high closure level.

There is even a concrete example of deference in the record. The analyst initially failed to check the essay’s first stated reason because a technical statement by the chief executive of the company doing the work appeared to come from the person best placed to know. The protocol eventually forced the check.

What these properties produce is visible in the case study.

The essay says that AI progress is now driven primarily by AI’s growing ability to build the next generation of AI. The Anthropic Institute page linked at that claim describes substantial acceleration of human engineering work, while stating that it remains genuinely unclear whether current methods could unlock the stronger capacity claimed.

The essay also places Anthropic’s incidents beside the OpenAI–Hugging Face episode as “similar, though less severe”. Anthropic’s own linked assessment says its incidents involved single model instances, that there was no coordination between agents, and that the models did not depart from their assigned exercises. Those differences matter to the comparison the essay is making.

A third example concerns the standards body proposed by Demis Hassabis. The essay invokes it as an example of industry dialogue. The proposal itself goes further: it envisages a body that begins voluntarily, can become mandatory for US deployment, and can coordinate a slowdown among frontier labs if necessary.

The important point here is not that the analyst found sources hostile to the essay. It is almost the reverse. Some of the analysis’s strongest objections were reached by following sources the essay itself chose to provide.

The procedure also produced credits.

The analysis records the essay’s dated 6–12 month prediction, which can later be checked against events. It credits the disclosure of Anthropic’s own failures and the links supplied to the underlying reports. It also credits a provision giving outside reviewers the right to publish findings without Anthropic’s editorial control.

The essay’s sourcing matters for the same reason. At its most charged description of the OpenAI–Hugging Face incident, it links an independent investigation that supports most of what the sentence says. The analysis records that support rather than treating charged language as a defect by default.

An apparatus that only ever produces criticism is not an apparatus. It is a verdict written in advance.

The limit should be stated rather than left for someone else to point out.

This is not reproducibility in the experimental sense. Re-running ARGUS on the same text does not guarantee an identical analysis, and claiming otherwise would be false.

What the design can provide is weaker and still useful: disagreement becomes locatable.

Because the steps, evidential standard, sources and corrections are declared, two analysts who differ can identify the point at which they part company and argue there, rather than exchange general impressions of the text.

The Amodei analysis does this on its own central finding. The modal-retraction verdict is recorded at medium confidence. The competing reading is stated explicitly: perhaps “pacing” always meant the weaker requirement of allowing enough time for alignment and verification, in which case there is no retraction and only an over-compressed opening sentence. The analysis rejects that reading, gives its reasons, and labels the rejection a judgement rather than a demonstration.

That is a smaller claim than reproducibility. It is also one a reader can test.

At the level of principle, none of these requirements needs specialised infrastructure. The cost is additional inference and disciplined record-keeping.

Publish the criteria, with dates, before applying them. Declare what will count as evidence before substantive testing begins. Separate the runs. Record the corrections, above all those that weaken your own conclusion.

Without that record, an analysis asks to be believed on the strength of how it sounds.

The reader of an ARGUS analysis turns to the protocol at two moments. Before, to know what will be applied: Design Note 01 explains why criteria published and dated before the analysis matter when it comes to auditing it. After, to check what was done. The report’s headings, its qualifications and the checks it declares all refer to notions in the protocol, which the reader looks up in the version they read.

On this site, the reference analysis of a case study is produced in the language of the analysed text, and some are then translated. The analysis of Dario Amodei’s essay, written in English, can also be read in French and in German. Its German reader checks it against the German protocol, whichever version the AI was given.

Any version can therefore be the one the AI applies, and each is the one some reader reads. The six must say the same thing. The wording is where they are likely to drift apart: a protocol of about a thousand lines, rendered in six languages, leaves each of them choices of vocabulary that only show up when the versions are set side by side.

Three cases found in the V5.1.0 protocols show what drifting means.

One notion, two words. In 0.a, the protocol provides for two cases in which the mandatory stop does not occur. French calls this a dispense and keeps the word throughout, down to the short version and the appendix on corpora. English, German, Spanish and Italian do the same with exemption, Befreiung, exención and esenzione. The Portuguese protocol heads the rule Isenção, then speaks of “regime de dispensa” in the short version and of “fórmulas de dispensa” in the corpus appendix. A Portuguese reader who comes across the second form has to guess that it means the first.

One word, several notions. The German protocol uses Ausnahme for three things: the “declared derogation” of 0.a, when the user asks for the analysis of a text of weak relevance anyway; the three exclusions in the modal constancy check, in Step 6; and the exception that Rule 13 constitutes in the short version. Each of the other five versions uses at least two words for these three notions.

The heading in front of the reader. The AI fills in a model report, the output template, supplied with each version of the protocol, and it is the template that provides the report’s headings. For test 3.I, the Spanish, Italian and Portuguese templates write circularidad informacional, circolarità informazionale and circularidade informacional; their protocols say informativa. A reader who looks the heading up in their own language’s protocol or glossary will not find it. The check of the Portuguese template found 21 of 28 headings worded differently from the protocol at the corresponding point; the check of the Italian template found 24 discrepancies.

An AI links dispensa to isenção without effort. For a reader who is checking, each of these discrepancies is a place where the check can go astray.

A discrepancy can also be a good choice. For the straw man of Step 1, the Portuguese protocol uses espantalho, the scarecrow. In French, homme de paille traditionally denotes a front man, the only sense Le Robert records for the phrase, and the French glossary has to warn the reader. The Portuguese glossary only had to set aside the scarecrow in the fields and one figurative sense. The divergence was kept, and the glossary work judged it to be in the Portuguese version’s favour.

An explanatory glossary tells the reader what a term means in ARGUS, in their own language. It would have been possible to write one and translate it five times. The six glossaries were built differently: the part of each notice that describes the ARGUS usage is drawn from that language’s protocol alone, without reading the notices of the other glossaries, and no notice points out a convergence or a divergence with another language.

The reason is the one the protocol gives for the cross-audit, the second of the three levels of contradiction it provides for: a second AI analyses the same text blind, without the first analysis, because “a second AI shown the first produces a revision, not an audit”. A glossary written with another one in view would be a translation of it, and its agreement with its model would prove nothing. Written separately, six glossaries that describe a notion in the same way attest that the six protocols state it in the same way. Where they diverge, the divergence points to a place to examine.

The separation has a second effect: each glossary responds to the difficulties of its own language. The German glossary specifies, in its notice on modal constancy, that the three Ausnahmen of Step 6 are to be confused neither with the derogation of 0.a nor with the exception of Rule 13. The Portuguese glossary brings isenção and dispensa together in a single entry. The other languages need none of these warnings.

The lists of entries differ too: 109 in French, 108 in English, 115 in Spanish, 94 in Italian, 134 in German, 135 in Portuguese. Each number is an outcome, with no target set in advance. Italian groups several terms under a single entry. German, one of the last to be written, started from a survey of 205 notions in its protocol; Portuguese made its own survey, then aligned its list with the German one.

The rule tightened along the way. The first notices, from 10 to 12 September, were written in all six languages side by side, entry by entry. The ban on reading the other glossaries’ notices came afterwards, with one exception, a glossary in a distant language used as a layout model: practised for Spanish, formulated for Italian on 20 September, written into the procedure on the 23rd. Even then, during the work on German, three frozen glossaries in other languages were opened by mistake, and their text entered the working conversation.

Separate glossaries need a meeting point; without one, nothing would indicate that dispense and isenção designate the same notion. That meeting point is a cross-language glossary, which is not published. It has two sides. The first is a table of forms: for each ARGUS concept it covers, the form used by each of the six protocols. It also serves translations, which must render every ARGUS term by the form used in the protocol of the target language. The second is an inventory of 205 notions, each tied to the section of the protocol where it appears and to the titles of the entries that deal with it in each glossary.

The cross-language glossary aligns the languages by protocol section and by entry title. It contains no definitions and never compares the content of notices. It supports three checks. Does every notion treated in one language have its counterpart in the others? Are the protocol’s closed sets, such as a scale or a grid of questions, covered in full? Is every formula that the protocol prescribes word for word tied to an entry?

What it flags is then verified in the protocols. On 23 September, the first four frozen glossaries were compared through their anchors alone. Five independent measures produced the same split: French and English on one side, Spanish and Italian on the other. The split followed the order of the work: the automatic checks, developed while the Spanish and Italian glossaries were being written, had not been applied to the first two. French and English were revised. In the six glossaries of version 1.0.0, every “See also” cross-reference leads to an existing entry.

The Ausnahme case shows where the tool stops. The inventory of 205 notions was drawn from the survey of the German protocol, the most recent at the time it was compiled. It tied the three exclusions of Step 6 to section 0.a, because the word Ausnahme appears there for the derogation. The Portuguese check noted that the Portuguese protocol did not mention them in 0.a; the German protocol does not either. The discrepancy came from the German word. The cross-language glossary flags, and the protocol of each language decides.

It makes no language the reference. Its concept identifiers are technical, and the table of forms, written in French, states that it grants normative priority to none of the six languages. It has errors of its own: its revision 1.2.0, numbered separately from the explanatory glossaries, still gives, in all six languages, the name “weak pole” to an element of the modal constancy check that the six protocols call the “retracted pole”. Its correction is pending.

Writing a notice means reading every occurrence of a term in its protocol. That work, carried out six times, brought discrepancies in the protocols themselves to light. The Italian protocol calls a descriptor Modalità retorica, a feminine noun, while its template calls it Modo retorico, a masculine one, and the values of each agree with its gender. The Portuguese exemption and the template headings are cases of the same kind.

A published version of ARGUS is never modified. These discrepancies are recorded for the next version, and the Italian correction is already prepared in a working version, V5.1.1.

A discrepancy that has been flagged is not always upheld. The German qualifier for rhetorical mode, raffiniert, where the other languages say “sophisticated”, was first presented as a defect of the German protocol. On examination it was reclassified as a choice of term, which remains to be decided.

The six glossaries were written, in separate conversations, by AI models from a single family: the documents are separate, the writer is shared. Assigning each language to a different writer would have brought the method closer to the cross-audit it imitates, but it would have meant coordinating six writers at the same time, which was beyond the project’s reach. In return, each glossary was audited by another AI, and each finding of that audit was checked against the protocol or the cited source before being accepted.

How familiar a word actually is to a reader of each language has not been measured; it was assessed from dictionaries and from the way words are formed. The cross-language glossary bears the mark of the language from which its inventory was drawn, and its errors only come to light by going back to the protocols. The discrepancies found in the V5.1.0 protocols have not yet been corrected: the reader of that version encounters them, and their glossary flags some of them.

The result can still be improved: a frozen glossary can be reopened, in a new version, for a documented defect or an inventory decision. The six versions of ARGUS remain six texts; this arrangement shows where they differ and tells the reader of each one, in their own language.

This note was written in French. Each of its translations was made separately, with the glossary of its language for ARGUS terms.

The “In ARGUS” field is the core of the notice. It is drawn from the protocol, at the point where the protocol defines or uses the notion. The French glossary sets the requirement: if only the term and the first sentence of this field are kept, the reader must be able to explain the notion correctly to someone else. Conditions, exceptions and references to steps come afterwards, and none of them may be dropped or softened to make the reading easier.

Thus, in the French glossary, a check is “a verification question defined before it is carried out and counted as a single unit”. The rest of the field specifies that each check receives one of the three statuses provided for in 3.B, and that every analysis declares three numbers, one per status.

The “In ARGUS” field says nothing the protocol does not say. In the German and Portuguese glossaries, every quotation from the protocol is compared with the protocol text by a script before any audit. Even so, the Portuguese audit found a notice that distinguished two terms with defining sentences the protocol does not contain; they were replaced.

The regime determines the form of the notice. There are three.

Under regime A, an ordinary word receives a precise meaning in ARGUS. Check is an example: the reader knows the word and is unaware of the unit of count it designates.

Under regime B, the word is an established term in a field, and the notice gives that “established meaning” before the ARGUS usage. Assertoric comes from the Kantian classification of judgements, where it occupies the middle degree between the problematic and the apodictic. In ARGUS, it designates the highest of four degrees on a scale of commitment. The notice has to state both, since a reader who knows Kant would otherwise misread the scale.

Under regime C, the expression is a creation of the protocol, with no prior meaning. Locked nuance is an example: according to the French glossary, a nuance “integrated in such a way that it strengthens or protects the conclusion instead of opening it up”. The “Not to be confused with” field matters more here than elsewhere, because the expression, made of everyday words, gives the reader the impression of understanding it.

The regime belongs to the notice, in its language. The same notion can be at B in one language and at A in another, either because the language lacks the same established term or because the available evidence differs. The Portuguese glossary thus places deferência under regime A, whereas Spanish and German place the notion under B: research pointed to a use in administrative law, but the source could not be read. With no source read, the notice says nothing about it, and the point remains open.

To say that an expression had no meaning before ARGUS is to assert that nobody used it. That is the hardest claim to establish, and the audits often caught it out.

Cross-audit, which names the second level of contradiction provided for by the protocol, was first classified under regime C in French, English and German. In all three languages, the expression turned out to be attested before ARGUS in quality and audit practice. The three notices moved to regime A, and now mention that earlier use.

The work on German laid down a rule, since taken up in Portuguese. An entry is classified C only if three verifications have been carried out and recorded: the dictionaries consulted do not give the expression; research found no established use in any field; the protocol itself constructs the notion. Failing that, the entry stays under regime A, where the notice quotes the meaning of each of the words that make up the expression. This cautious choice asserts no absence.

The rule did not work at the first attempt. The first German version had 17 entries under C. The eight major findings of its audit all concerned entries under that regime: the negative searches had not covered domain literatures such as law, quality or literary studies. Seven of those entries moved to A, one to B, and nine remain under C.

The effect shows in the figures. Regime C accounts for 44 entries out of 109 in French, 44 out of 108 in English, 46 out of 115 in Spanish and 33 out of 94 in Italian; it accounts for 9 out of 134 in German and 10 out of 135 in Portuguese. The gap comes from the rule: six notions classified C in Spanish, such as the short counter-check or the substitution test, are at A in German and Portuguese. Eight notions remain under C in all three languages, among them locked nuance, adapted propaganda and series conformism.

In version 1.0.0, the first four glossaries were not re-examined under this rule. They had been frozen on 24 September, the day before it was laid down. A frozen glossary is reopened only for a documented defect or an inventory decision. An entry classified C under the old rule may still be right; its evidence is weaker than what is required today, and only fresh research, followed by an audit, would show whether it holds. For the 167 entries concerned, this review was not carried out in version 1.0.0; it is among the work reserved for a later version. The glossaries thus bear the date of their method, and the figures above keep a record of it.

The rule on claims about usage was laid down during the Spanish work, and the writer broke it the same day. The zetética notice stated that research had found no established use of the pair zetética / dogmática in Spanish-language legal literature. The counter-audit produced four documents attesting it, including a Peruvian law review and the programme of a legal philosophy conference. The lesson stuck: a search that finds nothing establishes nothing.

Claims about frequency follow the same rule. The Spanish glossary said of one sense that it was “el más frecuente”, and of one word that it “no tiene uso corriente en castellano”. The reference dictionary does not measure frequency: the six claims of this kind were removed, and each was replaced by what the dictionary records or does not record. For hombre de paja, the replacement strengthened the warning: the dictionary gives the phrase only the sense of front man.

In the German and Portuguese glossaries, every external source is quoted word for word, with the work, the entry, the sense, the URL and the date of consultation. Each of the two glossaries contains more than 450 URLs of this kind. For matiz bloqueado, the Portuguese equivalent of locked nuance, the notice states that the expression appears neither in the Priberam dictionary, consulted on 25 September, nor in the Infopédia dictionary, consulted on the 26th, then quotes the meaning of each of the two words. The French notice for the same notion, written two weeks earlier, confines itself to two sentences, with no source. A source that is unreadable, paywalled or blocked is not worked around; the point remains open. Two documents that the reading tool could not reach were supplied as PDFs by a project lead, then read.

Notices are written in working files, in groups of entries. A script assembles them into a glossary, checks its structure and its cross-references, and refuses to overwrite an existing version. The assembled file is never corrected by hand: every correction is made in the working files, and the glossary is then reassembled. The French glossary went through seventeen versions, the Portuguese one two.

Each move from one version to the next is described in a migration document: the notices modified, those that remain byte-for-byte identical, and the hash of each file. Between its versions 0.1 and 0.2, the Portuguese glossary added two entries, removed one and modified thirty in the fields announced, and all the others remained identical.

Freezing takes place on the explicit agreement of a human project lead. A frozen glossary is no longer modified; a correction opens a new version. It was the actual rendering of the page that revealed, in Italian, a defect the proofreading had missed: a sentence in bold containing a word in italics was displayed incorrectly. Since then, before freezing, the candidate is run through the generator that produces the site page, and the page produced from the frozen file must be byte-for-byte identical to the one produced from the validated candidate.

Each glossary was audited by another AI, on a package whose contents are verified by script before it is sent: the glossary, the protocol of its language, the migration and, for German and Portuguese, the sources read. No finding is accepted on the auditor’s word. It is verified in the protocol or in the source before being conceded, and whatever could not be verified is declared.

This check works both ways. The German audit gave 2020 as the date of an article published in 2021, and 1917 for a publication from 1921; both dates were corrected before use. For another entry, the auditor proposed regime B; verification showed that the established meaning concerned a different object, the entry moved to A, and the following audit withdrew its recommendation.

What the audits found falls into a few families: C regimes without proof, claims about usage without a source, conditions of the protocol missing from a notice, quotations transcribed incorrectly. The Portuguese audit found 29 quotations from the protocol whose internal quotation marks had been changed; the instruction came from the writing brief, and the glossary did not declare it.

Some defects escape the audit, because no checklist covers them. The final check of the Spanish glossary found two lines of arbitration, in French, left in the body of two notices.

Finally, the project’s tracking notes keep a list of the writer’s errors, however they were found: the counting instrument was wrong five times before it was right, and a page 96 had been cited as page 95.

The quotations in the German and Portuguese glossaries reproduce the text of the pages as the reading tool returned it. In German, a randomly drawn sample was reread on the page; the Portuguese glossary declares that checking against the original remains to be done.

Each glossary declares its open points in a working note, 19 for Portuguese; these notes are in the frozen files, and the site page does not display them.

Nor were the first four glossaries revisited under the rule on dated quotations. The French and English glossaries still refer, a few times, to a point “3.A” that the protocol never names that way.

The work could go on forever: there will always be something to improve. In September, the French glossary was frozen four times and reopened three times, between its working versions 0.10 and 0.17. So that the work can come to an end, the six frozen glossaries now form a common version, 1.0.0. A frozen version is no longer modified; known improvements are reserved for a later version, which will deal with them together and carry a new number.

No overall measurement was made; what follows is qualitative.

The most expensive part was rereading the whole protocol, which runs to nearly 1,100 lines. During the work on German, several AI agents launched at the same time each reread the protocol and their own history at every step; some were cut off by the usage limit before they had written up their work, which was lost. Audits are costly too, with the preparation of their package and the verification of each finding. Dictionary lookups, on the other hand, cost little.

Three choices reduced the spending on the Portuguese glossary. A script extracts from the protocol only the paragraphs in which each term appears, and the writer reads only those. Dictionary research is separated from the writing and handed to a lighter model. Only one agent works at a time, and an agent that is cut off is resumed where it stopped. The Portuguese glossary thus cost markedly less than the German one, though it missed its target of costing half as much. What still weighed on it: carrying out the first steps in a single conversation, where the protocol and the tool outputs were reread at every call.