Counting arguments flawed for AI risk predictions
article argues that counting arguments (eg vast space of possible goals) provide no evidence for ai doom because they rely on uniform priors that are incoherent without knowing the sampling process.
the article challenges a common argument in ai safety: that because the space of possible goals for a superintelligence is vast, and human-compatible goals are a tiny subset, doom is the default outcome. the author demonstrates that this reasoning is mathematically unsound by referencing bertrand's paradox, where different 'reasonable' methods of random selection yield different probabilities for the same event. the text concludes that projecting the complex structure of future ai minds onto a simple subspace of 'goals' creates a distorted map similar to the mercator projection, making counting arguments unreliable for predicting ai behavior without specific knowledge of the training process.