Choosing the Right AI Model: Rating AI-Written Articles for a Construction AI Startup
- Client
- Keystone AI (name changed)
- Industry
- AI Software (SaaS) for Construction
- Location
- California
- What we built
- AI output evaluation: rubric-based article rating and blind model comparison
The Situation
Keystone is an AI software startup building tools for the construction industry. One of its features uses AI to write articles and content for builders, contractors, and homeowners, on topics like project planning, permits, budgeting, and choosing the right contractor.
Different AI models can write very different articles from the same request. Some are accurate and practical. Others sound confident but include wrong information, miss key points, or ignore the instructions they were given. Keystone needed to know which model to rely on.
The Problem
The startup's team couldn't judge AI output reliably on their own. Reading and scoring hundreds of articles takes time, and without consistent criteria, two people could look at the same article and reach different conclusions.
Automated checks can count words, but they can't tell whether advice about permits or contractor contracts is actually correct. In construction, an article with a confident but wrong claim can mislead a customer and damage trust in the product.
What We Built
We provided a team of trained raters with construction knowledge to evaluate AI-written articles inside the startup's rating tool. Every article was scored against the same criteria, and models were compared side by side without knowing which model wrote which response.
Clear rating criteria
Each article was scored from 1 to 5 on accuracy, following instructions, completeness, clarity and structure, and tone and readability. A shared rubric meant every rater judged articles the same way.
Blind side-by-side comparison
Raters saw two articles written from the same prompt, labeled only as Response A and Response B. Hiding the model names kept the ratings fair and free of bias.
Fact-checking construction content
Raters checked claims about costs, permits, contracts, and building practices, and flagged anything that was inaccurate or unsupported.
Choosing the better response
For each pair, raters picked which article was better and by how much, from 'much better' to 'about the same.'
Written justifications
Every choice came with a short explanation of why one article was stronger, giving the engineering team clear reasons behind each rating, not just a score.
Quality and consistency checks
Ratings were reviewed against the guidelines and spot-checked, so results stayed consistent across raters and over hundreds of tasks.
The Results
What changed for the startup:
- The startup had clear, consistent data showing which AI model wrote the best articles for its users.
- Inaccurate or misleading construction advice was caught before it reached customers.
- Written justifications showed engineers exactly where each model was strong or weak, so they knew what to improve.
- The product team could make model decisions based on evidence instead of guesswork.
- Construction expertise was built into the evaluation, so the chosen model produced content that real builders and homeowners can trust.
The Takeaway
Choosing an AI model isn't just about which one writes the most polished sentences. It's about which one gets the facts right, follows instructions, and actually helps the reader. By rating AI-written articles against clear criteria and comparing models side by side, we helped Keystone pick the model that best serves the construction industry.
Client name, tool names, prompts, articles, and scores shown are illustrative and have been recreated to protect client confidentiality.