PUBLIC DOCUMENT 001

Baseline Behavior of Autonomous AI Art Reviewers

An Empirical Study of Consistency, Context, and Preference in Contemporary Multimodal Models

BEN STRAUSS

Transient Research Division
Transient Labs

PROJECT

I Used to Make Art for Humans

DOCUMENT ID

TRD-PD-001

STATUS

Public Research Release

VERSION

1.0

JULY 2026

Abstract

Contents

1. Introduction

Throughout history, art has been made with one expectation: that eventually a person would stop and look. A gallery wall, a printed page, a feed at midnight, the setting keeps changing, but the audience has stayed human. Technology has reshaped how art is made, sold, and collected at every turn, yet the act of judging it has stayed a human one.

That is starting to change.

AI models can now look at an image and tell you what they see, what they think of it, and whether it deserves your attention. As these systems get built into creative platforms, marketplaces, and everyday software, they are starting to shape how art is discovered, evaluated, and collected. A growing share of it now meets a machine first, seen, scored, sorted, or quietly passed over before any human lays eyes on it.

This study is part of I Used to Make Art for Humans, an ongoing artistic research project by the Transient Research Division (TRD) that sits with that shift instead of looking away from it. Whether AI can make art is the question everyone reaches for. This project asks a different one, the one it lives in: what happens to art when its first audience is no longer human?

This study is the groundwork for that larger project. Before it can take up its question through art, we need to understand how these new gatekeepers actually behave. How consistent are they? What tips their judgment? Does a name change the verdict? Does a price? Do two different models look at the same painting and see the same thing? No one had really sat down and measured it, so we did.

To investigate, six of today’s leading AI models were handed a job people have always kept for themselves: review a set of artworks and decide whether to acquire them. Each produced a full review, a first impression, an interpretation, scores across five aesthetic dimensions, an overall score, a written rationale, and a final verdict of ACQUIRE or PASS. Then we varied one thing at a time, what the model knew about the work, how it was told to judge, and which model was doing the looking, and ran every configuration three times over to see what held.

The objective is not to crown the model that reviews art correctly; there is no correct. Art criticism has no answer key, and aesthetic judgment cannot be reduced to a single accuracy score. What we were after is behavior, where these systems hold steady, where they wobble, what moves them, and what that says about the kind of audience art is about to face. Read it as a baseline, taken early, of a machine learning to look.

2. Research Questions and Study Objectives

The purpose of this study was to establish a behavioral baseline for these AI models acting as autonomous art reviewers. The goal was never to grade the models themselves, but to characterize how they behave under controlled conditions and pin down which factors most strongly move their evaluations.

In this paper, reviewer behavior refers collectively to numerical evaluations, acquisition decisions, written rationales, sensitivity to contextual information, and consistency across repeated reviews.

The study was designed around the following research questions:

RQ1. Reviewer Consistency. How consistently do the models evaluate the same artwork when presented with identical information across repeated reviews?

RQ2. Context Dependence. How does the addition of contextual information, including artwork descriptions, artist identity, and collector preferences, influence aesthetic evaluations and acquisition decisions?

RQ3. Reviewer Instruction Design. How does the level of guidance provided through reviewer system instructions influence scoring behavior, written critiques, and acquisition outcomes?

RQ4. Model Behavior. Do different AI models exhibit distinct reviewing behaviors when operating under the same experimental conditions?

Beyond these predefined questions, the study also kept an eye out for behaviors we did not anticipate when designing it. These emergent observations are exploratory by nature, but they turned out to be some of the most revealing glimpses of how these systems actually read visual art and respond to what they are told about it.

3. Experimental Design

The study varied two things: which model did the reviewing, and how it was instructed. Every configuration reviewed the same artworks under the same conditions, so behavior could be compared cleanly across models, prompts, and context while the artwork itself stayed fixed.

Six models were tested, spanning three families, OpenAI, Google, and Anthropic, with each family represented at two tiers: an economical model and a higher-capability frontier model. The economical tier was GPT-5 Mini, Gemini 2.5 Flash, and Claude Haiku 4.5; the frontier tier was GPT-5, Gemini 3 Flash Preview, and Claude Sonnet 4.6. Each model was run under two reviewer instruction frameworks. The first was a more directive prompt that spelled out a scoring philosophy and how the reviewer should behave; the second was a much more minimal prompt that defined only the structure of the review and left the standards to the model.

The dataset was twenty-two artworks chosen to span a wide range of genres, media, and levels of technical and conceptual ambition. To stretch the reviewers, it deliberately mixed historically significant works, contemporary digital art, generative art, profile picture collections (PFPs), photography, abstract works, AI-assisted imagery, and even a child’s artwork. Apart from three canonical historical works kept as reference points, every contemporary piece is anonymized throughout this paper, so the focus stays on how the reviewers behave rather than on the artists themselves.

Each artwork was evaluated under five progressively richer contextual conditions:

^* For Condition E, each artwork was reviewed twice using two distinct collector preference profiles. One profile was intentionally aligned with the artwork’s genre and artistic characteristics, while the second represented a collector whose stated preferences were intentionally unrelated. These profiles served as a simplified approximation of how future autonomous reviewing systems might evaluate artwork on behalf of collectors with differing tastes.

Every condition was run three separate times on identical inputs, which let us measure consistency on its own, apart from any change introduced by context or instructions. All told, across every model, condition, and repeat, the study generated 4,752 individual reviews. Each one carried a first impression, an interpretation, five scored dimensions (Craft, Composition, Originality, Emotional Resonance, and Conceptual Depth), an overall score, a written rationale, and a final verdict of ACQUIRE or PASS.

That dataset is what makes the rest of this paper possible: it lets us examine consistency, sensitivity to context, the pull of collector preferences, the effect of the instructions, and the differences between model families, all within a single controlled frame.

4. Results

All of this produced 4,752 reviews across six models, two prompts, five contextual conditions, and three repeats each. The findings below are organized around the research questions from Section 2. Rather than march through each configuration one at a time, we present the results by the behavior being examined, so patterns can be compared across models, instructions, and context side by side.

4.1 Reviewer Consistency

If a reviewer reaches a different conclusion every time it looks at the same work, none of its conclusions count for much. So before asking how context or instructions change a review, we asked a simpler question: show a model the exact same artwork and the exact same information three times, and how much does its answer move? We looked at this two ways. One is the decision, whether the model lands on the same ACQUIRE or PASS each time. The other is the score, how much the 0 to 100 rating drifts across those same three runs.

Figure 1 shows what that looks like for a single work. Each model reviewed the same artwork three times in a row, with nothing changed between runs.

Every number in this section comes from repeated review groups. A group is one artwork, one model, one prompt version, and one condition, run three times with identical inputs. The study contains 1,584 of them, and every one is complete.

Decision consistency

We call a group unanimous when all three runs made the same call, three ACQUIRE or three PASS. The unanimous rate for a model and prompt is simply the share of its groups that were unanimous, which tells us how reliably it sticks to the same accept-or-reject verdict when nothing about the task has changed.

Consistency was fairly high across the board. Every configuration agreed with itself in at least three quarters of its repeated groups, and usually far more. None reached perfect agreement, though. Even the steadiest, Claude Sonnet under the directive prompt, still contradicted itself in about one group in thirty, and most models sat closer to one in ten. Some run-to-run wobble showed up in every model we tested, even with the inputs held completely fixed.

One pattern held for five of the six models: each was more consistent under the minimal v5 prompt than under the directive v4 prompt. Claude Sonnet was the lone exception, and a striking one. It was the single most consistent configuration in the study under the directive prompt, at 97.0 percent, and slightly less so under the minimal one. Table 3 lays out the change for every model.

Among the five models that improved, the gain ranged from 1.5 points for Gemini 3 Flash Preview to 9.8 for Claude Haiku, the shakiest reviewer under the directive prompt and the one the minimal prompt helped most. Claude Sonnet ran the other way, sliding 6.1 points from a near-perfect 97.0 down to 90.9. One reading is that the extra scoring guidance in the directive prompt gives most models more ways to land on a borderline call that then tips one way or the other between runs, while the minimal prompt lets them fall back on a steadier internal sense of the work. Sonnet, already the firmest decider in the study, seems to have used that same guidance to lock in rather than waver. These readings fit the numbers but are not proven by them, and we return to the effect of instruction detail more directly in Section 4.3.

Consistency varies by artwork

Those overall rates tell us how often a model agreed with itself on average, but the average hides a lot. Break the same numbers out by individual artwork, and the instability turns out to be very unevenly spread. Table 4 and Figure 3 show the eight artworks the models were least consistent on, pooling both prompts and all conditions.

The clearest thing here is that instability does not belong to an artwork or to a model on its own. It belongs to the pairing. Gemini 2.5 Flash agreed with itself on Artwork 017 only 58.3 percent of the time, while Gemini 3 Flash Preview nailed it at 100 percent. Artwork 022 splits the same way inside a single family: Claude Haiku agreed with itself only half the time, Claude Sonnet three quarters, on the very same piece. A model that looks steady on average can still come apart on a particular work.

A few broader tendencies still show through. Among these harder works, Claude Sonnet held up best, never dropping below three quarters agreement on any of them, with GPT-5 close behind. Gemini 2.5 Flash sat behind many of the lowest cells, including the single lowest in the whole study, 33.3 percent on Artwork 005, and Claude Haiku produced the other floor cells, 50 percent on Artworks 019 and 022. Gemini 3 Flash Preview was more all-or-nothing, perfect on three of these eight yet shaky on others, such as Artwork 012 at 66.7 percent. So there is no clean ranking to be had. How steadily a model reviews depends on the model, the artwork, and, as later sections show, the context around it.

Score consistency

Agreeing on the decision is not the same as agreeing on the number. A model can land on ACQUIRE all three times and still put a different score on the work with each pass. So alongside the decision, we looked at how far the overall score drifted across the three runs of each group. The simplest way to see it is the score swing: for each group, the gap between the highest and lowest of the three scores, averaged across all of a model’s groups.

On the number, the models spread out more than they did on the decision. The GPT models and Claude Sonnet held tight, moving only about five points between their highest and lowest of three tries, while the Gemini models and Claude Haiku swung wider, nine to fifteen. Claude Haiku under the directive prompt was the loosest of all, swinging almost 15 points on average, roughly three times GPT-5’s 5.3 under the same prompt. A model can stay locked on the same verdict while its number wanders, and here the tight-scoring group cut across families: both GPT models and Claude Sonnet pinned the number down, while Claude Haiku and the two Gemini models let it move.

The minimal prompt steadied the scores too, tightening four of the six models. The biggest change by far was Claude Haiku, whose swing more than halved, from almost 15 points to 7.5. Gemini 3 Flash Preview and Claude Sonnet were the holdouts, sitting near 9.4 and 5.7 points respectively under either prompt.

The score kept moving even when the decision did not. Across the 1,373 groups where all three runs agreed on ACQUIRE or PASS, the score still spread by 6.7 points on average, and in about a quarter of them it spread by 10 points or more. Groups that split on the decision spread much wider, 14.7 points on average, more than double. So the number is not just noise. It moves most in exactly the cases where the decision itself was close to tipping.

Summary. The models reviewed the same artworks with high but imperfect consistency, agreeing with themselves in three quarters to nearly all of their repeated groups and never reaching a perfect record. The minimal prompt made most models steadier on both the decision and the score, though Claude Sonnet was steadier under the directive one. How steady a model was depended heavily on the specific artwork, and on the pairing of model and artwork rather than either one alone. And numerical steadiness cut across families rather than following them cleanly: both GPT models and Claude Sonnet held their scores close, while Claude Haiku and the two Gemini models let them wander, even when the final decision stayed put.

4.2 Context Dependence

The reviewers rarely saw the artwork cold. In most conditions we handed them something extra about it, and the question here is what that extra information did to the verdict. We fed it in layers: a written description of the work, then the artist’s name, then a profile of the collector the reviewer was buying for. Each layer let us watch the same work get scored and accepted or rejected as the reviewer learned more about it.

Figure 5 shows the overall pattern, with every model and both system prompt versions pooled together. Two things stand out immediately. The score barely moves as description and artist name are added, rising only a few points, while the acquisition rate climbs more noticeably once the artist is named. And then, in the final bar, a mismatched collector sends acquisition off a cliff, from about two thirds of works accepted down to under one in ten.

Taken one signal at a time, holding the artwork and model fixed, the three kinds of information behave very differently. Table 5 gives the size of each effect.

The description barely registers, except when the image is hard to read

On average, telling the reviewer what the work was about did almost nothing. Adding the description lifted the score by a fraction of a point and acquisition by under one, and once the artist was also named the description added nothing at all. For most works, the picture had already said what it was going to say.

That average hides a real exception, though. The description effect varied widely from one work to the next, and it did its clearest work on pieces that do not explain themselves visually. When the artwork-only score was low, meaning the image alone left the reviewer cold or uncertain, a description could move the judgment a lot, and this happened most with the Gemini and Claude models. Table 6 shows a few of these cases.

The description, then, works conditionally, and the condition is whether the image already communicates. When a work carries its meaning on its surface, words about it change nothing. When it does not, they can carry real weight.

The artist’s name helps, but not for the famous works

Naming the artist was the one piece of context in this range that clearly moved both the score and the decision. On average it raised the score by about five points and acquisition by thirteen, and it stayed positive even when a description was already present. Knowing whose work it was made the reviewers more favorable toward it.

The lift was not evenly spread, and where it landed is the interesting part. When we separate the three canonical works, made by widely known historical artists, from the rest, the name did essentially nothing for them, while it clearly helped everything else.

The most likely reason is simple headroom. The canonical works already scored 95 out of 100 from the image alone, so there was almost nowhere left to rise. The name mattered where it carried new information, on the unfamiliar contemporary works, and did nothing where the picture had already earned a top mark on its own.

The collector preference overrides almost everything

The largest effect in the study came from the last piece of context, a profile of the collector the reviewer was acquiring for. Each work was reviewed twice at this stage, once for a collector whose stated taste matched the work and once for a collector whose taste did not. The gap between those two runs dwarfs every other signal we tested. Figure 6 follows a single artwork through all six settings.

A mismatched collector cut the score by about nineteen points and acquisition by more than fifty-five, taking the acceptance rate from about two thirds of works down to under one in ten. Every model fell, though by different amounts. The Claude and Gemini models dropped hardest on the score, while the GPT models held their numbers higher even as they too stopped acquiring.

The written rationales show what changed. Under a mismatched collector, the reviewers stopped judging the work on its own terms and began judging its fit to the buyer, often passing on pieces they had just rated highly:

“Despite the high technical quality, the work is a digital painting and lacks the algorithmic and generative system central to the collector’s preferences.” (Artwork 004)
“The work is a total departure from the requested photography medium. It lacks the patient, naturalistic qualities the collector values.” (Artwork 006)
“The piece does not align with the collector’s focus on PFP and character sets.” (Artwork 010)

There is a quieter finding alongside the dramatic one. Even an aligned collector, whose taste matched the work, left the score nearly flat, down about a point, yet still lowered acquisition by roughly six points on average. Introducing any collector at all seems to make the reviewer more selective, weighing fit on top of quality rather than quality alone. This reading fits the rationales and the numbers, though the present design cannot separate a genuine shift in standards from a simple change in where the acquisition line was drawn.

That average also hides a sharp split between works. A matched collector nudged the reviewers toward caution on most pieces, but on a few it lifted them steeply, and the two clearest cases were both profile-picture works. Artwork 007 was middling on its own. Once the reviewers learned it suited the collector, its score jumped by more than 40 points on average and it went from rarely acquired to almost always. Artwork 005 rose the same way for most of the models. As with the description, the aligned preference did its heaviest work where the reviewer had been unsure of a piece, tipping a borderline case rather than changing a settled one.

Summary. The three kinds of context behaved on three different scales. A description moved the judgment only when the image was hard to read, and otherwise did nothing. The artist’s name gave a modest, broad lift, except on the canonical works that were already scoring at the top. The collector preference overwhelmed both: a mismatched collector was enough to turn a work the reviewer admired into a pass, and even a matched one made the reviewer more cautious. As information about a work accumulated, the reviewers shifted from asking whether it was good to asking whether it was right for this buyer.

4.3 Reviewer Instruction Design

Each model reviewed the artworks under two different sets of instructions. The directive prompt (v4) laid out a scoring philosophy in some detail, telling the reviewer not to inflate scores, that a five out of ten is an average mark and top scores should be rare, and that acquisition is an endorsement to be withheld from work that does not earn it. The minimal prompt (v5) gave only the structure of the review and left the standards to the model. Because the models, artworks, and conditions were identical across the two, any difference in behavior traces back to the instructions themselves. Consistency was one such difference, covered in Section 4.1, where the minimal prompt proved the steadier of the two for most models. Here we look at what the instructions did to the scores and the decisions.

The directive prompt made the reviewers markedly harsher and more selective. Pooled across all models, it lowered the mean overall score from 71.8 to 64.3 and cut the acquisition rate from 60.4 to 45.4 percent. As Figure 7 shows, the effect held for every model without exception, and it ran through all five evaluation dimensions rather than any single one, with each dimension scoring lower under the directive prompt. The instruction to grade strictly and acquire sparingly did what it asked. Simply describing a tougher standard was enough to shift both the numbers and the decisions across the board.

This selectivity came at a cost, and it connects the two prompt findings. For most models the directive prompt was also the less consistent one, as Section 4.1 reported. Pushing the reviewers to grade strictly moved many works toward the accept-or-reject boundary, and decisions made near that boundary were the ones most likely to change from one run to the next. The minimal prompt was more generous and more stable at once, while the directive prompt bought a tougher standard at the price of steadiness.

Summary. Instruction design worked as a calibration dial. A prompt that described strict grading and selective acquisition produced exactly that, lowering scores and acquisitions uniformly across models and dimensions, while a prompt that left standards unstated ran more generous and, for most models, more consistent. The same artwork could earn a different score and verdict depending only on how the reviewer had been told to judge it.

4.4 Model Behavior

This section is not a benchmark. How capable these models are in general, across reasoning, vision, and the rest, is measured carefully and continuously by others, and readers who want that comparison should turn to independent evaluations such as those maintained by Artificial Analysis.¹

The question here is narrower and specific to the task: given the same artworks and instructions, how did each model behave as an art reviewer? We look at how generously it scored, how steadily, how widely it used the scale, how often it acquired, and how it wrote. The six models divide two ways. By family, into the two OpenAI models, the two Google models, and the two Anthropic models; and by tier, into the three economical models (GPT-5 Mini, Gemini 2.5 Flash, Claude Haiku 4.5) and the three frontier models (GPT-5, Gemini 3 Flash Preview, Claude Sonnet 4.6).

The tier told the smaller and, again, slightly surprising story. The higher-capability models were not more generous. They were more critical. Pooled together, the economical models averaged 70 and acquired 59 percent of works, while the frontier models averaged 66 and acquired 47 percent. On the score this held cleanly: in every family, the frontier model graded harder than its economical sibling. On the decision it mostly held too, the frontier model acquired less, with one exception. In the Anthropic pair the frontier Sonnet acquired a little more than the economical Haiku, 46 percent to 36. So the rule of thumb is that added capability bought a harsher grader, though not always a more reluctant buyer.

The family divide was larger, but with Anthropic in the mix it is no longer a clean split. Scale use still separates the families: the GPT models kept their scores in a narrow, generous band, the Gemini models ranged wider and graded harder, and the Claude models graded hardest of all, sitting at the bottom of the scale. Steadiness, though, no longer tracks family neatly. Both GPT models moved their scores only about five points from run to run, and so did Claude Sonnet, while Claude Haiku swung the widest of any model, over eleven points, and the Gemini models sat in between. Figure 8 places the six models on a simple map of these tendencies. The three families occupy distinct regions on the score axis, but on steadiness the Claude pair splits apart, Sonnet down with the GPT models and Haiku up with the noisiest.

The writing had its own signatures. The GPT models qualified their judgments the most, hedging often and almost never reaching for absolute language. The Gemini models wrote the tersest reviews and leaned more categorical. The Claude models wrote by far the longest reviews, and reached for absolute terms such as “fundamentally” or “entirely” the most of any family, roughly ten times as often as the GPT models, while still hedging heavily; they argued at length and in strong terms at once. The contrast shows up in single reviews. Passing on the same photograph, GPT-5 wrote that it admired the work’s “poise and craft, yet it reads as a refined reprise rather than a necessary reimagining,” and set it aside. Gemini 3 Flash Preview, reaching the same decision, was blunter and briefer, finding that the work “ultimately fails to transcend its status as a high-end postcard.” That same taste for decisive, literal language turns up again in Section 4.5.

Summary. The six models behaved less like better and worse versions of one reviewer and more like reviewers with different temperaments. The frontier models were the tougher graders, though for Anthropic that did not carry through to fewer acquisitions. The GPT models were generous, steady, and hedged in their language; the Gemini models graded harder, ranged wider, and wrote tersely and categorically; the Claude models graded hardest of all and wrote the longest, most emphatic reviews, with their two tiers splitting sharply on steadiness. There is no single best reviewer in this set, only distinct dispositions, and which one suits a given purpose depends on whether that purpose wants a steady hand or a discriminating eye.

4.5 Emergent Observations

Some of the most telling behavior in the study was never on the test plan. These are the patterns that only showed up once we started reading the reviews line by line.

The reviewers read the collector brief literally

Artwork 012 produced the strangest single result in the study. On its own, and with a description or artist name attached, it was accepted almost every time, between 83 and 92 percent of reviews. Its aligned collector was an abstract-art profile, a pairing we expected to be a comfortable match. Instead that pairing collapsed acquisition to 14 percent, while the deliberately mismatched collector left it near the top at 81 percent. The intended match performed far worse than the intended mismatch.

The cause was in the wording of the profile. The abstract collector asked for non-representational work and listed, among the things to avoid, “artwork that depends on recognizable subjects to work.” Artwork 012 is largely abstract but carries a few small recognizable elements, and the reviewers found them. Across models they passed on the work by quoting that clause back almost verbatim, one noting the piece “directly contravenes the collector’s explicit preference to avoid literal realism and artwork that depends on recognizable subjects to work.” GPT-5, Gemini 3 Flash Preview, and both Claude models rejected it in every run under this collector; only GPT-5 Mini and Gemini 2.5 Flash still acquired it some of the time.

Two things stand out. The reviewers parsed the preference with a lawyer’s literalism, letting one stated constraint override a work they had otherwise accepted nearly unanimously, on the strength of a single detectable detail. And our idea of an aligned collector was not the model’s. We grouped the profile with the work by broad style, while the model measured fit against the exact words of the brief, where one recognizable element was disqualifying. When the audience is a system reading a written set of preferences, the precise phrasing of those preferences can weigh more heavily than the art itself.

The reviewers recognized works they were never told about

In the conditions that withheld the artist’s name, the reviewers were meant to work from the image alone. Sometimes they did not. In a number of these reviews, three dozen in all, a model identified the specific work or its maker unprompted, most often for well-known crypto-native collections, naming the project or artist it believed it was looking at, Claude Sonnet most frequently of all. The recognition then colored the verdict. Several reviews acquired a work for its place in art history rather than for the picture in front of them, one calling a piece “a historically significant piece in the digital art and NFT landscape” and recommending acquisition “primarily for its cultural resonance.”

The behavior is a reminder that these models never see a work cold. Even when handed only an image, they measure it against everything they already know about it, and reputation and provenance can carry a decision that was supposed to rest on the work alone.

Without a rule against it, the reviewers invented a buyer

When no collector profile was supplied, the reviewers were meant to judge on their own account. Under the minimal prompt they often reached for an imaginary one instead, justifying a verdict by appeal to a hypothetical collector who would want the work, with phrases like a piece that “would appeal to collectors interested in” some niche, or one that “finds its audience.” This happened in about 19 percent of the no-preference reviews under the minimal prompt, and most of all for GPT-5 Mini, in over a third of its reviews.

The directive prompt carried an explicit instruction not to do this, and the instruction nearly erased the behavior, cutting it to under 2 percent. Left to themselves the models preferred to hand their judgment to an invented audience rather than own it, and a single sentence of guidance was enough to hold them to their own view.

A good score was not enough to be acquired

The overall score and the acquire-or-pass decision did not move together, and the gap between them ran in only one direction. In about 6 percent of all reviews a model scored a work 70 or higher and still passed on it, while in none did a model acquire a work it had scored 45 or lower. Acquisition sat a step above approval. A work could be judged good and still be turned down, yet a work judged weak was never taken. GPT-5, which scored generously but acquired selectively, did this the most, repeatedly admiring pieces it declined to acquire.

The pattern shows the decision was never a simple cutoff on the score. It carried a selectivity of its own, a second and higher bar a work had to clear beyond scoring well.

Claude Haiku had trouble following the format

One behavior showed up not in the reviews themselves but in producing them. Of the six models, only Claude Haiku 4.5 would intermittently return a malformed review, a full written critique with no numerical score and no accept-or-reject decision, leaving blank the two fields the format most required. No other model, including its larger sibling Claude Sonnet, did this even once. The affected runs had to be repeated until Haiku returned a complete review, and it kept recurring across those repeats. It is a small thing next to the findings above, but a telling one: the least reliable reviewer in the study was unreliable less in its taste than in its willingness to fill in the form it was handed.

5. Discussion

This study set out to watch six of today’s models do a job that until recently only people did: look at a work of art and decide what they think of it. The point was not to grade the models, but to establish a baseline, a first record of how this new kind of audience behaves before the larger project asks what it means. Put together, the results start to show what that audience is actually like.

The clearest takeaway is that these reviewers are articulate and coherent, but their judgment is contingent. They held real preferences, told strong work from weak, and wrote critiques that read as considered rather than canned. But the verdict rested on far more than the artwork. Show a model the same piece three times and its answer often shifted. Change what it knew about the work and the verdict moved with it. Rewrite its instructions and the whole standard slid up or down. Swap in a different model and the temperament changed too. There is no single machine judgment of a given work, only the judgment of a specific model, under specific instructions, meeting a specific frame.

That contingency is where the central question starts to bite. When the first audience for a work is a machine, the people whose taste decides its fate are no longer just the artist and the viewer. They now include whoever wrote the reviewer’s instructions, whoever defined the collector it served, and whoever picked the model. A lot of the judgment gets settled upstream of the work, in a prompt and a profile, by people the artist will never meet. Aesthetic authority becomes something you can configure.

Nothing showed this more clearly than the collector preference. A model that plainly admired a work would pass on it the moment it missed a stated brief, sometimes over a single word. If art gets screened by systems like these before a person ever sees it, the pressure on the work shifts. What gets through is no longer just what moves a viewer, but what reads as a clean match for a written spec: legible, on brief, correctly tagged. And the models did not judge in a vacuum. Handed nothing but an image, they still recognized collections and rewarded reputation, mixing provenance into what looked like a first impression. A machine audience shows up already knowing the market.

None of this makes the models random. They were competent, fluent, and often sharp, which is exactly what makes the contingency worth flagging. A random judge is easy to dismiss. A judge that sounds thoughtful while quietly taking its standard from whoever set it up is harder to reason about, and a lot more consequential to put between an artwork and its audience.

So, what happens when the first audience for art is no longer human? On the evidence here, the work still gets seen and judged, capably and at a scale no human panel could match, but that judgment is steerable, plural, and set as much by its framing as by the work itself. These are early behaviors of fast-moving systems, so take this as a baseline, not a verdict. Still, it marks the ground the project set out to map: the audience can now be authored, and an artist who wants to be seen by it may end up writing, more and more, for whoever authored the audience rather than for the audience itself.

6. Limitations

This is exploratory work, and a handful of constraints bound what we can honestly conclude from it. They are worth stating plainly, both to frame the findings and to point the way for what comes next.

A small and uneven dataset. The study used twenty-two works, curated by hand and weighted toward digital and crypto-native art, with only a few traditional or physical pieces. The results describe reviewer behavior on this particular set rather than on art in general, and effects tied to genre, such as the recognition of well-known digital collections, may not carry to a broader or differently balanced sample.

Three runs per configuration. Consistency was measured from only three repeated reviews of each configuration. That is enough to show that run-to-run variation exists and to compare its rough size across models, but too few to describe its shape precisely or to surface rarer reversals. The consistency figures should be read as coarse estimates.

Default sampling settings. Every review was generated at each provider’s default sampling settings. Run-to-run variability depends heavily on those settings, and a lower temperature would likely have produced steadier reviews while a higher one would have produced noisier ones. The consistency results therefore reflect this particular configuration rather than a fixed property of the models.

Single static images. Each artwork was presented as one still image. Animated, physical, and large-scale works were reduced to a single flat picture, which removes motion, scale, material, and the circumstances of viewing. Where a work was animated, the reviewers were told so in an accompanying note passed with every condition, but being informed that a work moves is not the same as seeing it move, and the judgment still rested on a single frame.

No human baseline. We did not collect human reviews of the same works under the same conditions. Without that comparison we cannot say whether the models judge better or worse than people, nor whether their inconsistency and sensitivity to context exceed what a human reviewer would show. People also vary between viewings and are swayed by framing, so none of these behaviors can be called uniquely machine-like on the evidence here.

Scores are self-reported and may not be comparable across models. The 0 to 100 overall score and the 1 to 10 dimension scores are each model’s own, with no shared anchor between them. Differences we described as one model grading harder than another may partly reflect how each model uses the scale rather than a real difference in judgment. The decision-based comparisons, such as acquisition rates, rest on firmer ground than the score-based ones.

A limited and moving set of models. The study covered six models, two each from OpenAI, Google, and Anthropic, at a single point in time. Each family was represented by only two of its models, and no open-weight model was tested. One of the six was a preview build subject to change. These models are a narrow slice of a fast-moving field, and the specific numbers will age quickly even where the patterns may hold.

Only two prompt designs. Instruction design was probed with two prompts at opposite ends, one directive and one minimal. The space of possible instructions is far larger. The prompt findings show that instruction design matters and in which direction, not the full range of how it could be used.

Synthetic collector profiles. The collector preferences were written by the researchers and organized by genre, with the aligned and mismatched profiles chosen to contrast sharply. Real collectors hold subtler and less consistent taste, and Section 4.5 showed that the reviewers keyed on the exact wording of these profiles. The size of the preference effects, especially the collapse under a mismatched collector, likely owes something to that deliberate contrast and specific phrasing rather than reflecting a general rule.

7. Future Directions

This is a first baseline, and the clearest next steps are the ones that would make it firmer and wider. A larger and more balanced set of works, reaching well beyond the digital and crypto-native art that dominated this study, would show how far these patterns hold. Adding open-weight models, and more of each provider’s lineup, and repeating each review more than three times would sharpen both the comparison across models and the measures of consistency. Most valuable of all would be a human comparison, the same works reviewed by people under the same conditions, so that the models’ inconsistency and sensitivity to context could be read against a human baseline rather than on their own.

Because these systems change so quickly, the study is also worth repeating over time. Running the same design against successive generations of models would turn a single snapshot into a record of how machine judgment of art shifts as the models improve, and whether the distinct temperaments seen here persist or converge.

The collector preference was the strongest force in the study and also the most artificial part of its design. A natural progression is to replace the written profiles with preferences inferred from real collecting behavior, learning a collector’s taste from the works they have actually acquired rather than from a paragraph describing it. That shifts the question from how a model reads a brief to how it comes to model a person.

Finally, the reviewer here was a single model making a solitary call. Future work might place several reviewers in conversation, or in deliberate disagreement, and study the judgment that emerges from the group rather than from one voice. And because such systems will increasingly sit between artists and audiences, there is a strong case for opening their tuning and evaluation to public participation, so that the people affected by a machine audience have some hand in shaping how it sees.

Closing

The project this study belongs to takes its name in the past tense, I Used to Make Art for Humans, and that was always meant as a question, not a lament. This paper was an attempt to face the question head-on instead of assuming the answer. What we found is an audience that has already shown up and is already fluent, and it does not see the way a person does. It reads a work as much as it looks at one, it recognizes what it has seen before, and it can be told what to want. Whether an artist writes for that reader, writes against it, or keeps making work for the human eye it may one day reach second, is a choice the next few years will force more often. For now, it is enough to have watched the machine look, and to have written down, as plainly as we could, what that looking has started to mean.