Method, schema and rubric

How the reports are produced and how to read them.

Pipeline

Extract — src/extract_text.py converts each PDF in pdf/ into page-tagged text (text/<id>.txt) so every claim can cite a PDF page.
Analyse — following docs/ANALYST_GUIDE.md (prompt bundle: src/make_prompt.py), an analyst (here an LLM; in future runs, another LLM) reads the full text and writes json/<id>.json following src/schema/paper.schema.json. Optionally, a corpus synthesis json/_corpus.json (src/schema/corpus.schema.json) records cross-paper findings.
Validate — src/validate.py checks schema conformance and referential integrity (every cited evidence/issue/claim id exists; relations form a graph; thesis present; empirical claims rated adequate+ cite evidence).
Render — src/build_site.py produces this site. Argument maps use Graphviz when available and a built-in layout otherwise.

Access modes

Every record declares how it was produced (document only or web-verified).

Document only: only the document is used, and its data are assumed correctly reported. Checks test internal consistency (arithmetic, text vs tables, agreement across documents), not truth against the world. General domain knowledge may appear only as a labelled analyst note.
Web-verified: after the document-only analysis, the analyst may consult external sources. These include publisher records (corrections, retractions), cited works (citation checks: does the source say what the paper claims?), original statistics and datasets (external checks, matched on vintage), and replication packages (replication checks). Every source is listed with URL, access date, reliability and version.

The two judgements are kept separate. A claim's support rating always answers "does the paper itself support this?". Its external status (ext: corroborated ext: partially corroborated ext: contradicted ext: not found ext: not checked ) answers "is it consistent with outside evidence?". External findings can change the verdict and the evidential-support score, and the record explains how in an external-verification summary.

What a paper record contains

Block	Purpose
bibliographic, summary	Identification; analyst paraphrase of abstract, research questions, contribution and argument outline.
methodology	Approach, design, data sources (D*), methods, scope, transparency notes, limitations acknowledged by authors.
concepts (K*)	Key terms with definitions, origin (coined/adapted/borrowed/legal/operational) and analyst notes on consistency.
evidence (E*)	Tables, statistics, excerpts, documents… with locations and key values, linked to data sources.
claims (C*)	Faithful restatement, type, role, location, quote, linked evidence and issues, and a support rating with rationale and "obviousness".
relations	Directed edges between claims/concepts (supports, explains, enables, motivates, defines, qualifies, illustrates, generalizes, contradicts) with strength: the logical structure behind the argument map.
issues (I*)	Problems with severity, category, location, affected claims and impact on conclusions.
verification_checks (V*)	Recomputations with inputs, result, status and scope (internal, cross paper, external, replication).
external_sources (S), citation_checks (R)	Web mode only: every outside source consulted, and checks of whether cited works support what is attributed to them.
assessment	Evaluation frame (what standard suits this genre), six rubric scores with rationales, verdict, strengths/weaknesses, robust/fragile claims, what would strengthen the work.

Claim types

definitional
theoretical
analytical
empirical descriptive
empirical comparative
empirical causal
interpretive
normative
predictive
methodological
background

Definitional, theoretical and normative claims are usually rated not applicable for empirical support; their quality is reflected in conceptual clarity and coherence instead.

Support ratings

strong	Evidence directly and convincingly establishes the claim; figures reconcile.
adequate	Reasonable support for the claim as stated; minor caveats.
partial	Supports part of the claim, or the claim is stated more strongly than the evidence allows.
weak	Evidence is thin, indirect, or open to clear alternative explanations.
unsupported	No evidence offered, or evidence contradicts the claim.
not applicable	Definitions, framing premises, normative recommendations.

Obviousness (self-evident / common knowledge / non-trivial / contested) records how much evidence a claim needs: self-evident claims are not penalised for lacking data.

Rubric dimensions (1–5)

conceptual clarity	Are key concepts defined, operationalised and used consistently?
logical coherence	Do conclusions follow from premises; are there internal contradictions or gaps?
evidential support	Is the evidence presented adequate, relevant and correctly reported for the claims made?
methodological rigor	Is the design appropriate (sampling, measurement, inference) for the questions asked?
transparency	Could a reader reconstruct what was done (data, definitions, procedures, dates, N)?
proportionality	Are conclusions and their certainty proportionate to the evidence (no overreach)?

Scale: 1 = serious deficiencies, 2 = weak, 3 = acceptable with notable problems, 4 = good, 5 = exemplary. Scores are judged relative to the genre stated in each record's evaluation frame (a theoretical essay is not penalised for lacking regressions, but its empirical quantifiers are still checked).

Derived indicators

Rubric score = mean of the six dimensions.
Claim support index = mean over rated claims of strong 100, adequate 80, partial 50, weak 25, unsupported 0; the core index restricts to thesis/main/conclusion claims.
Verdict (analyst judgement): well supported mostly supported partially supported weakly supported unsupported

Issue severity

critical invalidates a central conclusion.
major substantially weakens a main claim or headline figure.
moderate qualifies a claim or its magnitude.
minor presentational or peripheral; does not change conclusions.

Limitations of these reports

The analyses are LLM-generated and, although grounded in page-cited text and explicit recomputation, can contain misreadings; each claim links to its page for verification.
Charts and figures whose values are not printed in the text layer could not be checked.
Document-only records cannot detect errors in the underlying data or misrepresented citations; web-verified records can, but only for what the analyst checked (see each record's access notes).
Ratings are judgements; the rationales are the substantive content and should be read alongside the scores.