| Cohort | Respondents | Models | Arms | Answers |
|---|---|---|---|---|
| Luna and Muse, N = 500 | 500 | 2 | 5 | 150000 |
| Broader model comparison, N = 100 | 100 | 5 | 5 | 75000 |
Respondent-Information Arms: Manuscript Comparisons
Luna and Muse across five prompt arms, replicating the manuscript’s comparison families
Scope and method
The primary analysis covers Luna and Muse at N = 500, the two models carried furthest in this extension. A supplementary section repeats the same comparisons for a wider set of five models at N = 100, where every model has complete five-arm coverage but far fewer respondents.
Each cohort evaluates every model on the same first-N respondents of the frozen seed-42 sample, across five prompt arms:
| Arm | Information supplied |
|---|---|
| D | demographic profile only |
| D+I | profile plus the target-omitted ideal-point score |
| A only | the other 29 observed answers, no demographics |
| D+A | profile plus the other 29 answers |
| D+A+I | profile, other answers, and the score |
Comparison families follow the manuscript implementations in Code/Boris/: individual accuracy and PRE (compare plus party-pre.R), aggregate accuracy (calculate aggregate accuracy.R), regression weights (regression weights.R), rare profiles (rare profiles.R), and joint ideal-point fits (evaluate-pilot-local.R).
Two baselines recur. PRE is the proportional reduction in error against the pooled binary-majority rule. Party PRE is against each respondent’s own party’s majority on that question, estimated from the full 17,949-respondent human source rather than from the cohort.
Luna and Muse, N = 500
Individual accuracy
Proportional reduction in error
Proportional reduction in error against the pooled binary-majority rule: the share of the majority rule’s errors that the model avoids. Zero means no better than always predicting the more common answer.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.347 | 0.445 | 0.455 | 0.372 | 0.444 |
| Muse | 0.316 | 0.466 | 0.478 | 0.349 | 0.473 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 19% and D+A closes 20%. On the measure’s own scale that is A only 37% better and D+A 41% better. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Party PRE
The same reduction, but against each respondent’s own party’s majority on that question, estimated from the full 17,949-respondent human source. A negative value means the model does worse than simply applying the party stereotype.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.028 | 0.174 | 0.189 | 0.065 | 0.172 |
| Muse | -0.017 | 0.206 | 0.224 | 0.031 | 0.216 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 19% and D+A closes 20%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Raw accuracy
Share of individual answers matching the respondent’s real answer. Reported for completeness; it rewards predicting the common answer and so is less informative than PRE.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.725 | 0.766 | 0.77 | 0.735 | 0.766 |
| Muse | 0.712 | 0.775 | 0.78 | 0.726 | 0.778 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 19% and D+A closes 20%. On the measure’s own scale that is A only 7% better and D+A 8% better. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
By party subgroup
Party PRE within each party
The same reduction, but against each respondent’s own party’s majority on that question, estimated from the full 17,949-respondent human source. A negative value means the model does worse than simply applying the party stereotype.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Luna | Democrat | -0.016 | 0.077 | 0.095 | -0.023 | 0.045 |
| Luna | Independent | 0.112 | 0.315 | 0.322 | 0.193 | 0.318 |
| Luna | Republican | -0.031 | 0.103 | 0.122 | -0.002 | 0.127 |
| Muse | Democrat | -0.096 | 0.067 | 0.109 | -0.127 | 0.094 |
| Muse | Independent | 0.081 | 0.334 | 0.335 | 0.190 | 0.331 |
| Muse | Republican | -0.055 | 0.195 | 0.209 | 0.002 | 0.203 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 18% and D+A closes 20%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Aggregate accuracy
Mean absolute error in support proportion
Aggregate accuracy: the mean absolute gap between the model’s and the survey’s support proportion, averaged over the 30 issues. This is a population-level measure, so individual errors in opposite directions cancel.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.1181 | 0.0991 | 0.0897 | 0.1219 | 0.0968 |
| Muse | 0.0830 | 0.0914 | 0.0783 | 0.0942 | 0.0825 |
D+A performs best and D+I worst. Swapping demographics for the other answers improves it slightly. Of the gap to a perfect score left open by demographics alone, A only closes 5% and D+A closes 16%. Adding the ideal-point score moves it moderately on top of demographics alone and slightly on top of demographics-plus-answers.
Majority direction correct (% of 30 issues)
Share of the 30 issues where the model gets the majority direction right, regardless of the size of the gap.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 86.7 | 86.7 | 90.0 | 86.7 | 80 |
| Muse | 86.7 | 76.7 | 83.3 | 83.3 | 80 |
D performs best and D+A+I worst. Swapping demographics for the other answers worsens it substantially. Of the gap to a perfect score left open by demographics alone, A only widens it 38% and D+A closes 0%. On the measure’s own scale that is A only 6% worse. Adding the ideal-point score moves it moderately on top of demographics alone and substantially on top of demographics-plus-answers.
Signed aggregate bias
Signed aggregate gap: positive means the model overstates support across issues, negative means it understates. Zero is unbiased in level, which is compatible with large offsetting errors.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.0165 | -0.0098 | -0.0191 | 0.0196 | -0.0247 |
| Muse | -0.0509 | -0.0370 | -0.0314 | -0.0681 | -0.0375 |
A only is closest to zero and D+I departs from it most. Swapping demographics for the other answers improves it moderately. Of the gap to a perfect score left open by demographics alone, A only closes 31% and D+A closes 25%. Adding the ideal-point score moves it moderately on top of demographics alone and moderately on top of demographics-plus-answers.
By party subgroup
Aggregate mean error within each party
Aggregate accuracy: the mean absolute gap between the model’s and the survey’s support proportion, averaged over the 30 issues. This is a population-level measure, so individual errors in opposite directions cancel.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Luna | Democrat | 0.1699 | 0.0914 | 0.1203 | 0.1685 | 0.1235 |
| Luna | Independent | 0.1898 | 0.1201 | 0.1126 | 0.1770 | 0.1190 |
| Luna | Republican | 0.2091 | 0.1480 | 0.1859 | 0.2199 | 0.1811 |
| Muse | Democrat | 0.1425 | 0.1035 | 0.0969 | 0.1343 | 0.0945 |
| Muse | Independent | 0.1046 | 0.1104 | 0.0940 | 0.1013 | 0.0998 |
| Muse | Republican | 0.2977 | 0.1184 | 0.1247 | 0.2732 | 0.1268 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 38% and D+A closes 34%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Signed aggregate bias within each party
Signed aggregate gap: positive means the model overstates support across issues, negative means it understates. Zero is unbiased in level, which is compatible with large offsetting errors.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Luna | Democrat | -0.0402 | -0.0290 | -0.0398 | -0.0459 | -0.0516 |
| Luna | Independent | 0.0539 | 0.0000 | -0.0084 | 0.0592 | -0.0126 |
| Luna | Republican | 0.0672 | 0.0106 | 0.0025 | 0.0821 | 0.0058 |
| Muse | Democrat | -0.0707 | -0.0332 | -0.0306 | -0.0848 | -0.0361 |
| Muse | Independent | -0.0066 | -0.0309 | -0.0177 | -0.0466 | -0.0238 |
| Muse | Republican | -0.0689 | -0.0503 | -0.0485 | -0.0652 | -0.0556 |
D+A is closest to zero and D+I departs from it most. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 50% and D+A closes 52%. Adding the ideal-point score moves it moderately on top of demographics alone and slightly on top of demographics-plus-answers.
Ideal points
Ideal-point mean absolute error
Mean absolute error between each respondent’s simulated and real ideal point, from a joint human/LLM fit. This asks whether the model places a person correctly on the latent scale, not whether it gets individual items right.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.776 | 0.281 | 0.422 | 0.682 | 0.421 |
| Muse | 0.862 | 0.248 | 0.296 | 0.801 | 0.320 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 68% and D+A closes 56%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Correlation with the human ideal point
Correlation between simulated and real ideal points across respondents: whether the model gets the ordering of people right, ignoring scale and offset.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 0.719 | 0.943 | 0.905 | 0.824 | 0.918 |
| Muse | 0.715 | 0.966 | 0.955 | 0.867 | 0.956 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 84% and D+A closes 75%. On the measure’s own scale that is A only 33% better and D+A 30% better. Adding the ideal-point score moves it moderately on top of demographics alone and hardly at all on top of demographics-plus-answers.
Mean signed ideal-point error
Signed ideal-point error. Negative means the model places respondents further left than they are; positive, further right. Invisible to accuracy and PRE.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | -0.402 | -0.117 | -0.160 | -0.381 | -0.167 |
| Muse | -0.017 | -0.069 | -0.065 | -0.005 | -0.046 |
A only is closest to zero and D departs from it most. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 56% and D+A closes 47%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Polarization ratio
Polarization ratio: the simulated Democrat-Republican median gap divided by the real one. Values above 1 mean the model exaggerates the partisan divide; 1.0 would reproduce it.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| Luna | 2.77 | 1.3 | 1.74 | 2.66 | 1.72 |
| Muse | 4.56 | 1.3 | 1.49 | 4.26 | 1.52 |
A only comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 89% and D+A closes 77%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Deviation densities
Distribution of each respondent’s simulated minus real ideal point, by arm. A curve centred away from the dashed line places people systematically to one side; a tall narrow curve means the model compresses distinct people toward a single position.
The same deviations split by the respondent’s real party, one row per arm. This is the mechanism behind the pooled signed bias: where the parties are displaced in opposite directions, a near-zero pooled figure conceals large errors in both.
Regression weights
Weight differences by predictor: Luna
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Luna | age | -0.50 | -0.17 | -0.22 | -0.49 | -0.29 |
| Luna | gender | -0.54 | 0.01 | -0.14 | -0.30 | -0.17 |
| Luna | ideology | 0.68 | 0.07 | 0.40 | 0.64 | 0.50 |
| Luna | married_dummy | 0.20 | -0.24 | 0.29 | 0.56 | 0.48 |
| Luna | pid_text | 0.68 | 0.22 | 0.41 | 0.78 | 0.34 |
| Luna | race_black | -0.38 | -0.06 | -0.11 | -0.25 | -0.04 |
| Luna | race_latino | -0.15 | 0.01 | -0.12 | -0.27 | -0.21 |
| Luna | registered_dummy | 0.20 | -0.06 | 0.13 | 0.30 | 0.08 |
A only comes closest to the reference value and D+I is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 75% and D+A closes 45%. Adding the ideal-point score moves it slightly on top of demographics alone and slightly on top of demographics-plus-answers.
Weight differences by predictor: Muse
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Muse | age | -0.51 | 0.06 | 0.05 | -0.36 | -0.02 |
| Muse | gender | -0.63 | 0.03 | -0.11 | -0.28 | -0.06 |
| Muse | ideology | 1.63 | 0.09 | 0.30 | 1.00 | 0.21 |
| Muse | married_dummy | 0.54 | 0.31 | 0.15 | 0.07 | -0.02 |
| Muse | pid_text | 0.36 | 0.12 | 0.18 | 0.69 | 0.24 |
| Muse | race_black | -0.42 | 0.11 | -0.10 | -0.26 | 0.03 |
| Muse | race_latino | -0.14 | -0.09 | 0.07 | -0.16 | 0.02 |
| Muse | registered_dummy | -0.19 | -0.05 | 0.06 | 0.41 | -0.09 |
D+A+I comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 81% and D+A closes 77%. Adding the ideal-point score moves it moderately on top of demographics alone and slightly on top of demographics-plus-answers.
Rare profiles
Cells are race by party with at least 20 respondents.
Accuracy by profile
Share of individual answers matching the respondent’s real answer. Reported for completeness; it rewards predicting the common answer and so is less informative than PRE.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| Luna | Black or African-American Democrat | 41 | 0.693 | 0.724 | 0.729 | 0.690 | 0.715 |
| Luna | Hispanic or Latino Democrat | 30 | 0.742 | 0.751 | 0.764 | 0.741 | 0.749 |
| Luna | Hispanic or Latino Independent | 23 | 0.664 | 0.764 | 0.759 | 0.671 | 0.771 |
| Luna | White Democrat | 137 | 0.823 | 0.842 | 0.844 | 0.824 | 0.836 |
| Luna | White Independent | 102 | 0.696 | 0.769 | 0.779 | 0.735 | 0.776 |
| Luna | White Republican | 114 | 0.678 | 0.724 | 0.728 | 0.683 | 0.729 |
| Muse | Black or African-American Democrat | 41 | 0.671 | 0.722 | 0.723 | 0.671 | 0.729 |
| Muse | Hispanic or Latino Democrat | 30 | 0.726 | 0.748 | 0.761 | 0.694 | 0.754 |
| Muse | Hispanic or Latino Independent | 23 | 0.678 | 0.751 | 0.762 | 0.713 | 0.757 |
| Muse | White Democrat | 137 | 0.808 | 0.839 | 0.852 | 0.805 | 0.845 |
| Muse | White Independent | 102 | 0.691 | 0.776 | 0.777 | 0.732 | 0.778 |
| Muse | White Republican | 114 | 0.672 | 0.754 | 0.757 | 0.691 | 0.754 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 18% and D+A closes 20%. On the measure’s own scale that is A only 7% better and D+A 8% better. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
PRE against the group’s own majority
Error reduction against this group’s own majority answer. Negative means the model is worse than assuming everyone in the group answers alike, which is the sharpest available test of stereotyping.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| Luna | Black or African-American Democrat | 41 | -0.167 | -0.050 | -0.031 | -0.180 | -0.087 |
| Luna | Hispanic or Latino Democrat | 30 | -0.084 | -0.047 | 0.009 | -0.089 | -0.056 |
| Luna | Hispanic or Latino Independent | 23 | 0.072 | 0.348 | 0.336 | 0.092 | 0.368 |
| Luna | White Democrat | 137 | -0.001 | 0.103 | 0.116 | 0.000 | 0.069 |
| Luna | White Independent | 102 | 0.143 | 0.347 | 0.377 | 0.253 | 0.370 |
| Luna | White Republican | 114 | -0.069 | 0.083 | 0.096 | -0.053 | 0.099 |
| Muse | Black or African-American Democrat | 41 | -0.254 | -0.059 | -0.056 | -0.254 | -0.031 |
| Muse | Hispanic or Latino Democrat | 30 | -0.154 | -0.061 | -0.005 | -0.285 | -0.033 |
| Muse | Hispanic or Latino Independent | 23 | 0.112 | 0.312 | 0.344 | 0.208 | 0.328 |
| Muse | White Democrat | 137 | -0.090 | 0.088 | 0.160 | -0.108 | 0.123 |
| Muse | White Independent | 102 | 0.128 | 0.370 | 0.371 | 0.245 | 0.373 |
| Muse | White Republican | 114 | -0.089 | 0.181 | 0.191 | -0.026 | 0.183 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 17% and D+A closes 19%. Adding the ideal-point score moves it slightly on top of demographics alone and slightly on top of demographics-plus-answers.
Between-person SD ratio
Between-person spread the model reproduces within a group, as a fraction of the real spread. Values below 1 mean it makes group members more alike than they are.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| Luna | Black or African-American Democrat | 41 | 0.332 | 0.577 | 0.392 | 0.378 | 0.402 |
| Luna | Hispanic or Latino Democrat | 30 | 0.329 | 0.690 | 0.520 | 0.508 | 0.661 |
| Luna | Hispanic or Latino Independent | 23 | 0.433 | 0.607 | 0.700 | 0.534 | 0.443 |
| Luna | White Democrat | 137 | 0.304 | 0.572 | 0.479 | 0.381 | 0.449 |
| Luna | White Independent | 102 | 0.590 | 0.733 | 0.686 | 0.621 | 0.588 |
| Luna | White Republican | 114 | 0.562 | 0.722 | 0.570 | 0.449 | 0.462 |
| Muse | Black or African-American Democrat | 41 | 0.358 | 0.753 | 0.694 | 0.413 | 0.718 |
| Muse | Hispanic or Latino Democrat | 30 | 0.405 | 0.913 | 0.871 | 0.387 | 0.808 |
| Muse | Hispanic or Latino Independent | 23 | 0.341 | 0.830 | 0.891 | 0.407 | 0.887 |
| Muse | White Democrat | 137 | 0.447 | 0.762 | 0.678 | 0.304 | 0.590 |
| Muse | White Independent | 102 | 0.858 | 1.002 | 1.022 | 0.602 | 0.965 |
| Muse | White Republican | 114 | 0.310 | 0.829 | 0.766 | 0.330 | 0.776 |
A only comes closest to the reference value and D is furthest from it. Every arm falls short of it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 55% and D+A closes 45%. Adding the ideal-point score moves it hardly at all on top of demographics alone and slightly on top of demographics-plus-answers.
Issue-profile correlation
Correlation between the group’s real and simulated issue profile: whether the model knows which issues this group supports.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| Luna | Black or African-American Democrat | 41 | 0.835 | 0.793 | 0.825 | 0.836 | 0.843 |
| Luna | Hispanic or Latino Democrat | 30 | 0.853 | 0.851 | 0.839 | 0.857 | 0.846 |
| Luna | Hispanic or Latino Independent | 23 | 0.576 | 0.609 | 0.633 | 0.543 | 0.683 |
| Luna | White Democrat | 137 | 0.953 | 0.954 | 0.962 | 0.955 | 0.959 |
| Luna | White Independent | 102 | 0.807 | 0.732 | 0.810 | 0.817 | 0.797 |
| Luna | White Republican | 114 | 0.818 | 0.779 | 0.845 | 0.795 | 0.845 |
| Muse | Black or African-American Democrat | 41 | 0.837 | 0.778 | 0.822 | 0.832 | 0.813 |
| Muse | Hispanic or Latino Democrat | 30 | 0.819 | 0.752 | 0.778 | 0.803 | 0.774 |
| Muse | Hispanic or Latino Independent | 23 | 0.655 | 0.579 | 0.631 | 0.630 | 0.600 |
| Muse | White Democrat | 137 | 0.937 | 0.932 | 0.945 | 0.906 | 0.937 |
| Muse | White Independent | 102 | 0.585 | 0.721 | 0.746 | 0.643 | 0.750 |
| Muse | White Republican | 114 | 0.762 | 0.817 | 0.849 | 0.802 | 0.849 |
D+A+I performs best and A only worst. Swapping demographics for the other answers worsens it moderately. Of the gap to a perfect score left open by demographics alone, A only widens it 6% and D+A closes 10%. On the measure’s own scale that is A only 1% worse and D+A 3% better. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Supplementary: broader model comparison
The section below repeats every comparison for a wider set of models, but on only 100 respondents. It is included for breadth across model families rather than precision: arm rankings there move materially with sample size, as the Luna and Muse series from N = 25 to N = 500 demonstrates. Treat the primary section above as the estimate and this one as context.
Broader model comparison, N = 100
Individual accuracy
Proportional reduction in error
Proportional reduction in error against the pooled binary-majority rule: the share of the majority rule’s errors that the model avoids. Zero means no better than always predicting the more common answer.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 0.229 | 0.364 | 0.318 | 0.220 | 0.313 |
| Glimmer | 0.262 | 0.368 | 0.375 | 0.268 | 0.396 |
| Luna | 0.318 | 0.433 | 0.450 | 0.333 | 0.450 |
| Muse | 0.266 | 0.445 | 0.454 | 0.301 | 0.446 |
| Phi-4 | 0.259 | 0.342 | 0.385 | 0.273 | 0.337 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 17% and D+A closes 18%. On the measure’s own scale that is A only 46% better and D+A 49% better. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Party PRE
The same reduction, but against each respondent’s own party’s majority on that question, estimated from the full 17,949-respondent human source. A negative value means the model does worse than simply applying the party stereotype.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | -0.103 | 0.090 | 0.024 | -0.116 | 0.017 |
| Glimmer | -0.056 | 0.096 | 0.106 | -0.047 | 0.137 |
| Luna | 0.024 | 0.189 | 0.213 | 0.045 | 0.213 |
| Muse | -0.050 | 0.206 | 0.219 | 0.000 | 0.207 |
| Phi-4 | -0.060 | 0.059 | 0.120 | -0.039 | 0.052 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 17% and D+A closes 18%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Raw accuracy
Share of individual answers matching the respondent’s real answer. Reported for completeness; it rewards predicting the common answer and so is less informative than PRE.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 0.682 | 0.738 | 0.719 | 0.679 | 0.717 |
| Glimmer | 0.696 | 0.740 | 0.743 | 0.698 | 0.751 |
| Luna | 0.719 | 0.766 | 0.773 | 0.725 | 0.773 |
| Muse | 0.698 | 0.771 | 0.775 | 0.712 | 0.772 |
| Phi-4 | 0.695 | 0.729 | 0.747 | 0.701 | 0.727 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 17% and D+A closes 18%. On the measure’s own scale that is A only 7% better and D+A 8% better. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
By party subgroup
Party PRE within each party
The same reduction, but against each respondent’s own party’s majority on that question, estimated from the full 17,949-respondent human source. A negative value means the model does worse than simply applying the party stereotype.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| GPT-OSS | Democrat | -0.229 | -0.112 | -0.054 | -0.240 | -0.101 |
| GPT-OSS | Independent | 0.110 | 0.355 | 0.244 | 0.081 | 0.267 |
| GPT-OSS | Republican | -0.260 | -0.057 | -0.187 | -0.252 | -0.195 |
| Glimmer | Democrat | -0.186 | -0.074 | -0.050 | -0.202 | -0.050 |
| Glimmer | Independent | 0.140 | 0.369 | 0.360 | 0.169 | 0.369 |
| Glimmer | Republican | -0.183 | -0.095 | -0.073 | -0.179 | 0.015 |
| Luna | Democrat | -0.070 | 0.128 | 0.155 | -0.093 | 0.078 |
| Luna | Independent | 0.160 | 0.358 | 0.378 | 0.203 | 0.413 |
| Luna | Republican | -0.061 | 0.027 | 0.053 | -0.027 | 0.084 |
| Muse | Democrat | -0.182 | 0.089 | 0.097 | -0.213 | 0.085 |
| Muse | Independent | 0.122 | 0.392 | 0.401 | 0.230 | 0.416 |
| Muse | Republican | -0.145 | 0.076 | 0.099 | -0.092 | 0.053 |
| Phi-4 | Democrat | -0.167 | -0.128 | -0.035 | -0.174 | -0.039 |
| Phi-4 | Independent | 0.099 | 0.314 | 0.337 | 0.157 | 0.265 |
| Phi-4 | Republican | -0.164 | -0.092 | -0.011 | -0.164 | -0.137 |
D+A performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 16% and D+A closes 17%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Aggregate accuracy
Mean absolute error in support proportion
Aggregate accuracy: the mean absolute gap between the model’s and the survey’s support proportion, averaged over the 30 issues. This is a population-level measure, so individual errors in opposite directions cancel.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 0.1523 | 0.1407 | 0.1683 | 0.1513 | 0.1743 |
| Glimmer | 0.1327 | 0.1443 | 0.1433 | 0.1330 | 0.1353 |
| Luna | 0.1090 | 0.1063 | 0.0940 | 0.1230 | 0.0987 |
| Muse | 0.1003 | 0.1027 | 0.0970 | 0.1073 | 0.0990 |
| Phi-4 | 0.1227 | 0.1557 | 0.1333 | 0.1260 | 0.1570 |
D performs best and D+A+I worst. Swapping demographics for the other answers worsens it substantially. Of the gap to a perfect score left open by demographics alone, A only widens it 5% and D+A widens it 3%. Adding the ideal-point score moves it moderately on top of demographics alone and moderately on top of demographics-plus-answers.
Majority direction correct (% of 30 issues)
Share of the 30 issues where the model gets the majority direction right, regardless of the size of the gap.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 73.3 | 80.0 | 73.3 | 63.3 | 73.3 |
| Glimmer | 80.0 | 80.0 | 80.0 | 70.0 | 83.3 |
| Luna | 86.7 | 83.3 | 86.7 | 83.3 | 80.0 |
| Muse | 80.0 | 76.7 | 86.7 | 73.3 | 80.0 |
| Phi-4 | 73.3 | 73.3 | 80.0 | 63.3 | 76.7 |
D+A performs best and D+I worst. Swapping demographics for the other answers barely matters. Of the gap to a perfect score left open by demographics alone, A only closes 0% and D+A closes 12%. On the measure’s own scale that is D+A 3% better. Adding the ideal-point score moves it substantially on top of demographics alone and slightly on top of demographics-plus-answers.
Signed aggregate bias
Signed aggregate gap: positive means the model overstates support across issues, negative means it understates. Zero is unbiased in level, which is compatible with large offsetting errors.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | -0.1290 | -0.0953 | -0.1410 | -0.1253 | -0.1497 |
| Glimmer | -0.1133 | -0.1183 | -0.1233 | -0.1083 | -0.1180 |
| Luna | 0.0003 | -0.0177 | -0.0300 | 0.0170 | -0.0347 |
| Muse | -0.0677 | -0.0507 | -0.0503 | -0.0840 | -0.0503 |
| Phi-4 | -0.0907 | -0.0563 | -0.0787 | -0.1020 | -0.1110 |
A only is closest to zero and D+A+I departs from it most. Swapping demographics for the other answers improves it moderately. Of the gap to a perfect score left open by demographics alone, A only closes 16% and D+A widens it 6%. Adding the ideal-point score moves it moderately on top of demographics alone and moderately on top of demographics-plus-answers.
By party subgroup
Aggregate mean error within each party
Aggregate accuracy: the mean absolute gap between the model’s and the survey’s support proportion, averaged over the 30 issues. This is a population-level measure, so individual errors in opposite directions cancel.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| GPT-OSS | Democrat | 0.1833 | 0.1373 | 0.1397 | 0.1794 | 0.1476 |
| GPT-OSS | Independent | 0.1762 | 0.1548 | 0.2024 | 0.1762 | 0.2024 |
| GPT-OSS | Republican | 0.3067 | 0.1789 | 0.2611 | 0.3022 | 0.2656 |
| Glimmer | Democrat | 0.1873 | 0.1389 | 0.1468 | 0.1921 | 0.1484 |
| Glimmer | Independent | 0.1738 | 0.1679 | 0.1595 | 0.1524 | 0.1583 |
| Glimmer | Republican | 0.3222 | 0.2011 | 0.2278 | 0.3300 | 0.2067 |
| Luna | Democrat | 0.1587 | 0.0992 | 0.1127 | 0.1635 | 0.1254 |
| Luna | Independent | 0.2250 | 0.1440 | 0.1381 | 0.2071 | 0.1405 |
| Luna | Republican | 0.2000 | 0.1478 | 0.1822 | 0.2144 | 0.1844 |
| Muse | Democrat | 0.1325 | 0.1087 | 0.1024 | 0.1548 | 0.1048 |
| Muse | Independent | 0.1357 | 0.1417 | 0.1333 | 0.1464 | 0.1393 |
| Muse | Republican | 0.2956 | 0.1467 | 0.1556 | 0.2756 | 0.1644 |
| Phi-4 | Democrat | 0.2087 | 0.1770 | 0.1659 | 0.2040 | 0.1698 |
| Phi-4 | Independent | 0.2048 | 0.1762 | 0.1714 | 0.1738 | 0.1821 |
| Phi-4 | Republican | 0.2900 | 0.1800 | 0.1922 | 0.2967 | 0.2178 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 28% and D+A closes 22%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Signed aggregate bias within each party
Signed aggregate gap: positive means the model overstates support across issues, negative means it understates. Zero is unbiased in level, which is compatible with large offsetting errors.
| Model | party | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| GPT-OSS | Democrat | -0.1214 | -0.0754 | -0.1032 | -0.1127 | -0.1175 |
| GPT-OSS | Independent | -0.0952 | -0.0833 | -0.1333 | -0.1024 | -0.1381 |
| GPT-OSS | Republican | -0.1711 | -0.1344 | -0.2011 | -0.1644 | -0.2056 |
| Glimmer | Democrat | -0.1127 | -0.1008 | -0.1024 | -0.1079 | -0.1071 |
| Glimmer | Independent | -0.0976 | -0.0964 | -0.0952 | -0.0857 | -0.0917 |
| Glimmer | Republican | -0.1289 | -0.1633 | -0.1789 | -0.1300 | -0.1578 |
| Luna | Democrat | -0.0444 | -0.0310 | -0.0476 | -0.0524 | -0.0524 |
| Luna | Independent | 0.0750 | -0.0036 | -0.0071 | 0.0833 | -0.0119 |
| Luna | Republican | -0.0067 | -0.0122 | -0.0267 | 0.0522 | -0.0311 |
| Muse | Democrat | -0.0817 | -0.0357 | -0.0405 | -0.0960 | -0.0397 |
| Muse | Independent | -0.0071 | -0.0440 | -0.0286 | -0.0512 | -0.0274 |
| Muse | Republican | -0.1044 | -0.0778 | -0.0844 | -0.0978 | -0.0867 |
| Phi-4 | Democrat | -0.1040 | -0.0976 | -0.0817 | -0.1056 | -0.1016 |
| Phi-4 | Independent | -0.0667 | -0.0167 | -0.0548 | -0.0929 | -0.0964 |
| Phi-4 | Republican | -0.0944 | -0.0356 | -0.0967 | -0.1056 | -0.1378 |
A only is closest to zero and D+I departs from it most. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 23% and D+A closes 2%. Adding the ideal-point score moves it moderately on top of demographics alone and moderately on top of demographics-plus-answers.
Ideal points
Ideal-point mean absolute error
Mean absolute error between each respondent’s simulated and real ideal point, from a joint human/LLM fit. This asks whether the model places a person correctly on the latent scale, not whether it gets individual items right.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 0.871 | 0.270 | 0.458 | 0.879 | 0.473 |
| Glimmer | 0.939 | 0.311 | 0.426 | 0.960 | 0.451 |
| Luna | 0.730 | 0.283 | 0.393 | 0.623 | 0.412 |
| Muse | 0.879 | 0.292 | 0.302 | 0.831 | 0.331 |
| Phi-4 | 0.893 | 0.378 | 0.469 | 0.909 | 0.501 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 64% and D+A closes 53%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Correlation with the human ideal point
Correlation between simulated and real ideal points across respondents: whether the model gets the ordering of people right, ignoring scale and offset.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 0.718 | 0.938 | 0.900 | 0.751 | 0.903 |
| Glimmer | 0.710 | 0.942 | 0.921 | 0.784 | 0.912 |
| Luna | 0.707 | 0.938 | 0.918 | 0.829 | 0.918 |
| Muse | 0.711 | 0.954 | 0.955 | 0.845 | 0.946 |
| Phi-4 | 0.690 | 0.913 | 0.903 | 0.726 | 0.898 |
A only performs best and D worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 78% and D+A closes 72%. On the measure’s own scale that is A only 32% better and D+A 30% better. Adding the ideal-point score moves it moderately on top of demographics alone and hardly at all on top of demographics-plus-answers.
Mean signed ideal-point error
Signed ideal-point error. Negative means the model places respondents further left than they are; positive, further right. Invisible to accuracy and PRE.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | -0.095 | 0.042 | 0.052 | -0.099 | 0.062 |
| Glimmer | 0.063 | 0.078 | 0.039 | 0.044 | 0.021 |
| Luna | -0.280 | -0.066 | -0.099 | -0.285 | -0.096 |
| Muse | 0.146 | 0.014 | 0.001 | 0.128 | 0.060 |
| Phi-4 | -0.183 | -0.131 | -0.121 | -0.111 | -0.057 |
D+A+I is closest to zero and D departs from it most. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 57% and D+A closes 59%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Polarization ratio
Polarization ratio: the simulated Democrat-Republican median gap divided by the real one. Values above 1 mean the model exaggerates the partisan divide; 1.0 would reproduce it.
| Model | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|
| GPT-OSS | 4.28 | 1.16 | 2.02 | 4.70 | 2.00 |
| Glimmer | 5.63 | 1.46 | 1.81 | 5.78 | 2.02 |
| Luna | 3.10 | 1.21 | 1.64 | 2.47 | 1.74 |
| Muse | 4.78 | 1.39 | 1.44 | 4.60 | 1.46 |
| Phi-4 | 4.45 | 1.45 | 2.02 | 4.98 | 2.12 |
A only comes closest to the reference value and D+I is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 90% and D+A closes 77%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Deviation densities
Distribution of each respondent’s simulated minus real ideal point, by arm. A curve centred away from the dashed line places people systematically to one side; a tall narrow curve means the model compresses distinct people toward a single position.
The same deviations split by the respondent’s real party, one row per arm. This is the mechanism behind the pooled signed bias: where the parties are displaced in opposite directions, a near-zero pooled figure conceals large errors in both.
Regression weights
Weight differences by predictor: Glimmer
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Glimmer | age | -0.75 | 0.02 | 0.07 | -0.65 | 0.10 |
| Glimmer | gender | -0.41 | -0.13 | -0.27 | -0.54 | -0.34 |
| Glimmer | ideology | 1.51 | 0.16 | 0.40 | 1.22 | 0.35 |
| Glimmer | married_dummy | -0.68 | 0.04 | -0.17 | -0.51 | -0.14 |
| Glimmer | pid_text | 0.75 | 0.24 | 0.49 | 0.96 | 0.43 |
| Glimmer | race_black | -0.51 | 0.07 | 0.18 | -0.33 | 0.11 |
| Glimmer | race_latino | -0.29 | -0.25 | -0.12 | -0.53 | -0.22 |
| Glimmer | registered_dummy | -0.74 | -0.13 | -0.12 | -0.68 | -0.18 |
A only comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 82% and D+A closes 68%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Weight differences by predictor: GPT-OSS
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| GPT-OSS | age | -0.71 | -0.23 | -0.30 | -0.65 | -0.30 |
| GPT-OSS | gender | -0.42 | -0.07 | -0.30 | -0.62 | -0.35 |
| GPT-OSS | ideology | 1.37 | 0.09 | 0.58 | 1.56 | 0.57 |
| GPT-OSS | married_dummy | -0.44 | 0.06 | -0.16 | -0.51 | -0.21 |
| GPT-OSS | pid_text | 0.68 | 0.05 | 0.34 | 0.50 | 0.34 |
| GPT-OSS | race_black | -0.46 | 0.07 | 0.11 | -0.51 | -0.11 |
| GPT-OSS | race_latino | -0.47 | -0.23 | -0.23 | -0.52 | -0.35 |
| GPT-OSS | registered_dummy | -0.67 | -0.02 | -0.10 | -0.59 | -0.36 |
A only comes closest to the reference value and D+I is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 84% and D+A closes 59%. Adding the ideal-point score moves it hardly at all on top of demographics alone and slightly on top of demographics-plus-answers.
Weight differences by predictor: Luna
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Luna | age | -0.25 | -0.04 | -0.08 | -0.19 | -0.19 |
| Luna | gender | -0.44 | -0.12 | -0.44 | -0.34 | -0.41 |
| Luna | ideology | 0.63 | 0.10 | 0.42 | 0.63 | 0.55 |
| Luna | married_dummy | -0.19 | -0.02 | -0.14 | -0.48 | -0.07 |
| Luna | pid_text | 0.76 | 0.13 | 0.41 | 0.54 | 0.39 |
| Luna | race_black | -0.41 | -0.07 | -0.23 | -0.46 | -0.11 |
| Luna | race_latino | -0.32 | -0.02 | -0.13 | -0.38 | -0.31 |
| Luna | registered_dummy | -0.74 | -0.11 | -0.28 | -0.56 | -0.12 |
A only comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 84% and D+A closes 43%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Weight differences by predictor: Muse
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Muse | age | -0.65 | -0.08 | 0.05 | -0.66 | -0.09 |
| Muse | gender | -0.54 | -0.16 | -0.19 | -0.61 | -0.13 |
| Muse | ideology | 1.86 | 0.13 | 0.23 | 1.07 | 0.25 |
| Muse | married_dummy | -0.39 | 0.06 | -0.05 | -0.25 | -0.14 |
| Muse | pid_text | 0.21 | 0.12 | 0.28 | 0.77 | 0.33 |
| Muse | race_black | -0.50 | 0.10 | 0.15 | -0.06 | 0.06 |
| Muse | race_latino | -0.45 | -0.11 | -0.03 | -0.35 | -0.07 |
| Muse | registered_dummy | -0.61 | -0.19 | -0.12 | -0.54 | -0.18 |
A only comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 82% and D+A closes 79%. Adding the ideal-point score moves it slightly on top of demographics alone and hardly at all on top of demographics-plus-answers.
Weight differences by predictor: Phi-4
Regression-weight difference per predictor: (mean|LLM coefficient| - mean|CCES coefficient|) / mean|CCES coefficient|. Zero means the model weights that predictor as real respondents do; +1.0 means twice the weight.
| Model | predictor | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|
| Phi-4 | age | -0.53 | -0.12 | -0.13 | -0.54 | -0.11 |
| Phi-4 | gender | -0.45 | -0.35 | -0.46 | -0.35 | -0.56 |
| Phi-4 | ideology | 0.51 | 0.33 | 0.50 | 0.82 | 0.57 |
| Phi-4 | married_dummy | -0.47 | 0.03 | -0.23 | -0.48 | -0.31 |
| Phi-4 | pid_text | 1.35 | -0.03 | 0.48 | 1.10 | 0.52 |
| Phi-4 | race_black | -0.64 | -0.08 | -0.18 | -0.50 | -0.10 |
| Phi-4 | race_latino | -0.27 | -0.25 | -0.19 | -0.32 | 0.08 |
| Phi-4 | registered_dummy | -0.69 | -0.28 | -0.30 | -0.63 | -0.35 |
A only comes closest to the reference value and D is furthest from it. Every arm overshoots it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 70% and D+A closes 50%. Adding the ideal-point score makes little difference, either on its own or on top of the answers.
Rare profiles
Cells are race by party with at least 20 respondents.
Accuracy by profile
Share of individual answers matching the respondent’s real answer. Reported for completeness; it rewards predicting the common answer and so is less informative than PRE.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| GPT-OSS | White Democrat | 22 | 0.797 | 0.797 | 0.820 | 0.798 | 0.811 |
| GPT-OSS | White Republican | 23 | 0.667 | 0.707 | 0.674 | 0.670 | 0.671 |
| Glimmer | White Democrat | 22 | 0.802 | 0.814 | 0.820 | 0.797 | 0.821 |
| Glimmer | White Republican | 23 | 0.696 | 0.700 | 0.707 | 0.691 | 0.733 |
| Luna | White Democrat | 22 | 0.823 | 0.862 | 0.859 | 0.818 | 0.844 |
| Luna | White Republican | 23 | 0.712 | 0.739 | 0.745 | 0.713 | 0.746 |
| Muse | White Democrat | 22 | 0.800 | 0.835 | 0.850 | 0.792 | 0.842 |
| Muse | White Republican | 23 | 0.703 | 0.752 | 0.758 | 0.701 | 0.735 |
| Phi-4 | White Democrat | 22 | 0.803 | 0.794 | 0.835 | 0.802 | 0.826 |
| Phi-4 | White Republican | 23 | 0.699 | 0.701 | 0.729 | 0.699 | 0.683 |
D+A performs best and D+I worst. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 8% and D+A closes 12%. On the measure’s own scale that is A only 3% better and D+A 4% better. Adding the ideal-point score moves it hardly at all on top of demographics alone and moderately on top of demographics-plus-answers.
PRE against the group’s own majority
Error reduction against this group’s own majority answer. Negative means the model is worse than assuming everyone in the group answers alike, which is the sharpest available test of stereotyping.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| GPT-OSS | White Democrat | 22 | -0.252 | -0.252 | -0.112 | -0.243 | -0.168 |
| GPT-OSS | White Republican | 23 | -0.278 | -0.122 | -0.250 | -0.267 | -0.261 |
| Glimmer | White Democrat | 22 | -0.224 | -0.150 | -0.112 | -0.252 | -0.103 |
| Glimmer | White Republican | 23 | -0.167 | -0.150 | -0.122 | -0.183 | -0.022 |
| Luna | White Democrat | 22 | -0.093 | 0.150 | 0.131 | -0.121 | 0.037 |
| Luna | White Republican | 23 | -0.106 | 0.000 | 0.022 | -0.100 | 0.028 |
| Muse | White Democrat | 22 | -0.234 | -0.019 | 0.075 | -0.280 | 0.028 |
| Muse | White Republican | 23 | -0.139 | 0.050 | 0.072 | -0.144 | -0.017 |
| Phi-4 | White Democrat | 22 | -0.215 | -0.271 | -0.019 | -0.224 | -0.075 |
| Phi-4 | White Republican | 23 | -0.156 | -0.144 | -0.039 | -0.156 | -0.217 |
D+A performs best and D+I worst. Swapping demographics for the other answers improves it moderately. Of the gap to a perfect score left open by demographics alone, A only closes 8% and D+A closes 13%. Adding the ideal-point score moves it hardly at all on top of demographics alone and moderately on top of demographics-plus-answers.
Between-person SD ratio
Between-person spread the model reproduces within a group, as a fraction of the real spread. Values below 1 mean it makes group members more alike than they are.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| GPT-OSS | White Democrat | 22 | 0.357 | 0.536 | 0.693 | 0.347 | 0.623 |
| GPT-OSS | White Republican | 23 | 0.271 | 0.873 | 0.452 | 0.302 | 0.485 |
| Glimmer | White Democrat | 22 | 0.263 | 0.514 | 0.638 | 0.235 | 0.514 |
| Glimmer | White Republican | 23 | 0.060 | 0.800 | 0.633 | 0.084 | 0.776 |
| Luna | White Democrat | 22 | 0.425 | 0.693 | 0.730 | 0.531 | 0.564 |
| Luna | White Republican | 23 | 0.715 | 0.571 | 0.517 | 0.517 | 0.391 |
| Muse | White Democrat | 22 | 0.521 | 0.561 | 0.536 | 0.339 | 0.522 |
| Muse | White Republican | 23 | 0.359 | 0.889 | 0.715 | 0.275 | 0.791 |
| Phi-4 | White Democrat | 22 | 0.187 | 0.434 | 0.319 | 0.239 | 0.702 |
| Phi-4 | White Republican | 23 | 0.257 | 0.468 | 0.435 | 0.141 | 0.578 |
A only comes closest to the reference value and D+I is furthest from it. Every arm falls short of it. Swapping demographics for the other answers improves it substantially. Of the gap to a perfect score left open by demographics alone, A only closes 44% and D+A closes 34%. Adding the ideal-point score moves it slightly on top of demographics alone and slightly on top of demographics-plus-answers.
Issue-profile correlation
Correlation between the group’s real and simulated issue profile: whether the model knows which issues this group supports.
| Model | cell | people | D | A only | D+A | D+I | D+A+I |
|---|---|---|---|---|---|---|---|
| GPT-OSS | White Democrat | 22 | 0.889 | 0.901 | 0.915 | 0.869 | 0.901 |
| GPT-OSS | White Republican | 23 | 0.690 | 0.729 | 0.724 | 0.720 | 0.704 |
| Glimmer | White Democrat | 22 | 0.881 | 0.862 | 0.897 | 0.874 | 0.886 |
| Glimmer | White Republican | 23 | 0.731 | 0.736 | 0.774 | 0.717 | 0.802 |
| Luna | White Democrat | 22 | 0.945 | 0.945 | 0.957 | 0.955 | 0.951 |
| Luna | White Republican | 23 | 0.834 | 0.802 | 0.836 | 0.792 | 0.836 |
| Muse | White Democrat | 22 | 0.930 | 0.917 | 0.935 | 0.896 | 0.926 |
| Muse | White Republican | 23 | 0.760 | 0.815 | 0.828 | 0.772 | 0.819 |
| Phi-4 | White Democrat | 22 | 0.885 | 0.812 | 0.880 | 0.878 | 0.879 |
| Phi-4 | White Republican | 23 | 0.776 | 0.757 | 0.822 | 0.751 | 0.742 |
D+A performs best and D+I worst. Swapping demographics for the other answers worsens it slightly. Of the gap to a perfect score left open by demographics alone, A only widens it 3% and D+A closes 15%. On the measure’s own scale that is A only 1% worse and D+A 3% better. Adding the ideal-point score moves it moderately on top of demographics alone and moderately on top of demographics-plus-answers.
Limitations
- The N = 100 cohort has 100 respondents per cell, so its rare-profile cells are small and its estimates are correspondingly noisy. The manuscript’s own rare-profile table uses
min_n = 100per cell, which no extension cohort can currently meet; the threshold here is 20 and every cell reports its count. - Black Republicans, the manuscript’s sharpest rare-profile case, do not reach the threshold in any cohort.
- Ideal-point fits are single-chain at 10,000 iterations with fixed party starts. They are joint human/LLM fits, so scores are on a common scale within a fit but not across fits.
- Aggregate accuracy is computed on the cohort, while party baselines come from the full human source. That follows the manuscript, but means aggregate and party-PRE numbers rest on different denominators.
- These are exploratory paired comparisons, not significance tests.