Who Says What's Right?

RLHF — or: How six thinkers explain why alignment is not just about technology, but a matter of democracy.

The Third Step

Pre-training makes the model language-capable. Fine-tuning makes it useful. But only RLHF makes it 'obedient' (sic!). Reinforcement Learning from Human Feedback is the step that turns a highly competent but indifferent language machine into an assistant that says what we(?) want to hear. Not the truth — that would be asking too much. But what humans (who, exactly?) consider to be 'right'. And here begins a sociological question that is actually well-known. Because who are these people? What do they consider right? And who decided that and how they should decide?

The Invisible Judges

Before we get to the grand theories, let's look at the facts on the ground: RLHF works like this: Humans evaluate AI responses. Right or wrong. Good or bad. Helpful or harmful. A reward model learns from these evaluations — and this model controls what the AI will say in the future. The people who perform these evaluations sit in Nairobi, Manila, Chennai. They work for platforms like Sama, Appen, Scale AI. They often earn less than $5 an hour. They evaluate thousands of prompts per shift. The global clickworker industry is the backbone of RLHF. OpenAI, Anthropic, Google — they all use it (see Perrigo, TIME 2023; Hao, MIT Technology Review 2022). Castells would have recognized this immediately: RLHF is not a laboratory process. It is a node in the global network of the information economy. Power does not circulate horizontally — it concentrates in the clients, while the work takes place in the peripheries of the network. Castells calls this constellation "spaces of flows": Data flows from San Francisco to Nairobi and back. The people in Nairobi sit in the "space of places" — bound, interchangeable, invisible. But the decisions are made in the "space of flows," where the clients are located. The precise working conditions vary greatly between platforms and projects. Transparency is low. Most RLHF providers do not publish detailed work reports.

Habermas — Alignment as a Systematically Distorted Discourse Now it gets sociological. Jürgen Habermas developed the concept of the ideal speech situation: A discourse is rational when all affected parties can participate equally, when arguments count, not power, and when the better argument prevails. RLHF is the opposite of this ideal speech situation. Firstly: The affected parties are not in the room. If an AI responds in Germany, India, or South Africa — are the people from these countries at the RLHF table? No. The labelers talk about the needs of others without these others having a say themselves. Secondly: The rules of discourse are not transparent. What criteria do the labeling guidelines establish? Who wrote them? By what procedure were they legitimized? Habermas would say: Alignment is a systematically distorted discourse — produced by power, not by reason. Thirdly: The validity claims are not fulfilled. In the ideal speech situation, speakers make four claims: comprehensibility, truth, truthfulness, rightness. But the AI cannot guarantee truth (see hallucinations), can only provide rightness as a statistical agreement with human judgments, and certainly not truthfulness — it has no intention. Habermas' diagnosis, applied to RLHF: The AI training discourse takes place beyond the conditions of a discourse free from domination. The AI cannot act communicatively — it can only strategically classify. It thereby simulates understanding where in reality only statistical optimization is taking place. Crucial question: How does our brain operate when making decisions? Is it not the same biomechanical process in the synapses?

Fraser — Participatory Parity and the New Counter-Publics

Nancy Fraser extended Habermas's theory. Her central question: Who belongs to the discourse? And who doesn't? For Fraser, there is not one public, but many — parallel arenas in which marginalized groups develop their own discourses. These counter-publics are where alternative interpretations and norms emerge. Applied to RLHF: The dominant public (English-speaking, North American, academic, liberal) sets the standards for "helpful" and "harmless." But what is "helpful" for a queer young person in Lagos can be "irritating" for a conservative father in Bavaria. And vice versa. Fraser's category of participatory parity is the decisive test here: Can all social groups participate equally in defining good AI? Obviously not. Blind spot: RLHF governance discusses alignment as a technical problem (safety, bias, robustness). The democratic theoretical question — who legitimizes the norms of AI? — is hardly asked. Fraser would postulate: RLHF doesn't need better technology. It needs a democratic structure in which those affected by AI decisions are also involved in defining "right" and "wrong." Participatory parity as a design principle of alignment.

Bourdieu — The Symbolic Capital of Labelers Now we're talking about class and stratification. Pierre Bourdieu showed that not all judgments are equally valid. A judgment is not only factually true or false — it carries the weight of the social position of the person who makes it. RLHF labelers possess specific cultural capital: They are linguistically competent (mostly English), digitally savvy, well-read, and familiar with the conventions of platform work. But they possess little symbolic capital — their judgment does not count as authority. It is anonymized, aggregated, averaged. Bourdieu would ask: Whose habitus is inscribed into the AI through RLHF? The habitus of the labelers — shaped by class, social stratum, education, origin — determines which responses are rated as "good." Not individually, but statistically: whoever clicks "helpful" often enough shapes the reward model. The crux: Globally, labelers are precariously employed individuals from educated strata of the Global South. They are between classes — skilled enough for the work, not privileged enough for better. Their habitus is that of cultural intermediaries. But this position is invisible because it is framed as "objective quality control." Sociological RLHF research is still in its infancy. Initial studies (Gray & Suri, Ghost Work, 2019; Tubaro et al., 2024) show the precariousness of clickworkers, but the concrete habitus-shaping of labeling decisions has hardly been investigated yet.

Eribon — The Class That Isn't Named

In Returning to Reims, Didier Eribon closed a gap in French sociology: the shame of class mobility, the disdain of the elites for the working class, the silence about class in public debates. His sharpest point: Class is not named, but it acts. It is the blind spot of every discussion about inequality — because those who talk about inequality have long since left the field of class. Transferred to RLHF: The tech elite talks about alignment. About safety. About value diffusion. But the class of people who literally click the alignment is not mentioned. They are the subject that is not named. Eribon's provocative question might be: Would RLHF even exist if the labelers were unionized? If they had the power to co-determine the guidelines? If they could negotiate instead of just clicking? The answer is no. The precarity of the labelers is not a malfunction of RLHF — it is its prerequisite. Alignment is cheap, and the work remains invisible. Principal-Agent: The Secret Power of Clickworkers So far, it sounded as if clickworkers were mere victims of the structure. Exploited. Invisible. Powerless. But that's only half the truth. The principal-agent theory (from institutional economics) describes a relationship in which a principal (client) entrusts an agent (contractor) with a task. The agent has a decisive advantage: information asymmetry. They know better what they are doing than the principal can control. Applied to RLHF: The principal (OpenAI, Anthropic, Google) wants high-quality, honest, consistent evaluations. The agent (the clickworker in Nairobi) wants to keep their job, have as little stress as possible, and earn enough. The principal's problem: They cannot look at every screen. They don't know if the agent actually read every answer — or switched to autopilot after the third "A." Whether the agent is tired, frustrated, sick, or simply underpaid. The hidden power: The clickworker has the power to influence the quality of the entire alignment — through minimal, uncontrollable actions. Clicking too fast distorts the reward model. A misunderstood ranking produces systematic errors. A frustrated worker who indiscriminately clicks "harmful" changes the AI's behavior towards thousands of users. Moral Hazard — The Silent Strike If you know your work isn't seen — why should you put in effort? This is the logic of moral hazard: After signing the contract, the agent has an incentive to exercise less care than the principal desires. Especially when pay is low, control is superficial, and consequences remain abstract. Adverse Selection — Who Stays, Who Goes? A second principal-agent problem: The platform doesn't know which workers are truly competent. The good ones eventually leave — because they find better jobs or can't emotionally endure the work (daily evaluating hate speech, violence, pornography). Blind spot: Adverse selection in the RLHF labor market has hardly been investigated so far. What selection effects systematically distort alignment? The Dialectic of Power Clickworkers are simultaneously extremely powerless (economically, politically, legally) and extremely powerful (informational, as quality gatekeepers of alignment). Their power is negative — they can prevent, not shape. Untapped power: If clickworkers were to collectively use their information-asymmetric position — through organized slowdowns, coordinated quality standards — they would have real bargaining power. They are the only ones who can really stop alignment.

Diskutiere diesen Text, seine Begriffe oder Denker:innen mit Sociologica. Dialektisch (Lesart + Gegen-Lesart), mit Leitfrage zurück an dich.

Diskussion starten
Day 12
Day 14