All Field Notes

How to Know When Your Vibe-Coded App Has Outgrown the Vibe

A practical scorecard for the moment when an AI-built prototype needs product maturity, delivery discipline, and a rewrite-versus-remediate plan.

AI-assisted developmentMaintainabilityProduct maturity

AI-assisted iteration works best at the beginning — when the product is still finding its shape. Prompts are cheap, the surface area is small, and each change teaches you something useful about the idea.

That same workflow starts to strain when the prototype becomes real software. The app still moves quickly, but every new feature has to rediscover the rules. Screens contradict each other. Agents fix one bug by creating another. There's still a sense of momentum, but the team is quietly paying for it with manual testing, nervous deploys, and long prompts that re-explain the same context every time.

We reviewed an app recently where the warning signs weren't one dramatic failure. They were ordinary, useful things that had quietly piled up. The scripts directory had dozens of one-off files for backfills, cleanup jobs, data imports, and sync fixes. Database migrations had repeated numeric prefixes because nobody had set up ordering checks. A planning document said one system was the source of truth, but the running application actually used another. And a cron job generated a production wrapper that handled git sync, defaults, agent invocation, consolidation, and notifications in a single shell script.

None of those choices is surprising in a fast prototype. But together, they show the moment when "a few helpful scripts" has become an operating system with no clear owner.

The Inflection Point

An app has outgrown the vibe when the cost of continuing without structure exceeds the speed benefit that justified the approach in the first place.

That doesn't mean the prototype was a mistake. Most useful products are messy before they are coherent — that's normal. The issue is subtler: the product has crossed into a phase where users, money, and data depend on decisions still trapped in scattered prompts and local patches. Nobody planned for this phase because the last phase was working fine.

Once you recognize the shift, the question isn't "should we rewrite?" It's "what do we keep, what do we harden, and what do we actually need to rebuild so the next change is safer than the last one?"

The Scorecard

Use a simple scorecard before reaching for a grand rewrite. Mark each area pass, partial, or gap, and write down the evidence in the repo or running product. The app I reviewed scored like this:

  • Ownership: Can someone name the module, job, or service that owns each important workflow? Gap — the cron wrapper handled git sync, defaults, agent invocation, consolidation, and notifications in a single script; nobody owned it end to end.
  • Source of truth: Does the product have one clear home for each critical fact, or do docs, scripts, database rows, and UI state disagree? Gap — a planning document named one system as authoritative while the running application actually depended on another.
  • Regression control: Are recurring bugs captured in tests, types, lint rules, or migration checks? Gap — the scripts directory existed because recurring problems got patched by hand each time, not because a check caught them first.
  • Release discipline: Can the team tell how often it deploys, how long changes take, how often deploys need intervention, and how long recovery takes? Partial — deploys ran through the generated cron wrapper, but nothing enforced migration ordering, which is why prefixes collided.
  • Security baseline: Are protected routes, data access, secrets, model inputs, and logs treated as trust boundaries? Partial — the same one-off scripts that ran backfills and cleanups could reach production data directly, with nothing in the repo saying which ones were safe to run against live records.
  • Operations: Are background jobs observable, versioned, and owned, or are they shell scripts that only one person understands? Gap — the cron wrapper was the entire operations layer, with no logs separating its five responsibilities from one another.
  • Agent workflow: Can a coding agent make a safe small change from the repository instructions, or does every task require a custom briefing? Partial — an agent could extend the wrapper in an afternoon, but every task still needed a human to explain which of its five jobs was safe to touch. This is the one row you can score today for free — the Post Code Labs repo audit checks it automatically.

This is deliberately boring. Boring is the point. A product that is ready to grow should not depend on heroic memory.

The same scorecard, in a form you can paste into an issue:

Vibe-code maturity scorecard — mark pass / partial / gap and add evidence
 
- [ ] Ownership: can you name who owns each critical workflow?
- [ ] Source of truth: one home per fact, or do docs/scripts/DB/UI disagree?
- [ ] Regression control: are recurring bugs caught by tests, types, lint,
      or migration checks?
- [ ] Release discipline: do you know deploy frequency, lead time,
      intervention rate, and recovery time?
- [ ] Security baseline: are routes, data access, secrets, model inputs,
      and logs treated as trust boundaries?
- [ ] Operations: are background jobs observable, versioned, and owned?
- [ ] Agent workflow: can an agent make a safe small change from the repo
      instructions alone?

Early Signals

Real users start exposing hidden product rules

Early users are forgiving — they're collaborators, they know it's early, they work around the rough edges. That changes when the app becomes part of someone's actual job. Suddenly the form validation that was "fine for the demo" decides whether a customer can finish onboarding. A dashboard calculation shows up in a sales conversation. An admin screen changes data that support has to explain to an unhappy customer.

When real users arrive, product rules need a home. If pricing logic lives in both a server route and a client component, if role checks are scattered across three files, or if a background job uses a different definition of "complete" than the UI, throwing more prompts at the problem just widens the gaps. What the system actually needs is fewer write paths, named domain concepts, and tests around the workflows that matter.

Real data and money change the security standard

Security tolerances shift alongside the data. A prototype can get away with thin boundaries because the only data is sample data and the only users are the team. A live app doesn't get that luxury.

OWASP's Top 10 is a useful sanity check here, especially insecure design, software and data integrity failures, and security logging and monitoring failures. For AI-enabled products, the OWASP Top 10 for LLM Applications adds risks such as prompt injection and sensitive information disclosure.

You don't need to turn a small product into a compliance program overnight. But you do need to know where protected data enters, where it's stored, where it leaves, what a coding agent can see, and what ends up in logs. (The most common of these gaps — committed secrets and missing authorization — are the subject of The Secrets Your AI Already Leaked.) If answering those questions requires digging through shell scripts and deployment notes, the app has outgrown its original operating assumptions.

Delivery friction becomes visible

The clearest signal is often release pain — especially once a second or third person starts contributing. The team can still ship, but every deploy feels bespoke. A local script has to be run in the right order. A migration name has to be checked by hand. A cron job has production behavior inside a generated wrapper. A fix that looked safe in preview creates a production cleanup task two days later.

DORA's four key metrics are useful here because they separate speed from stability. Deployment frequency and lead time tell you whether the team can move. Change failure rate and recovery time tell you whether that movement is controlled. When throughput stays high only because people absorb the instability — staying late to babysit deploys, hand-checking migration order — the vibe has been outgrown.

Technical debt becomes a product decision

People use "technical debt" loosely, but the concept is specific: take a shortcut now, pay for it later. Martin Fowler's Technical Debt Quadrant is still the clearest framework for separating deliberate, prudent shortcuts from reckless or accidental ones. With AI-built apps, the question is sharper: does the team even know what kind of debt it's carrying?

Knowingly deferring a test suite to validate a market — that may have been prudent. Discovering six months later that migrations can collide, jobs fail silently, and two systems claim to own the same data is a different situation entirely. The first kind of debt can be scheduled. The second kind is already product risk.

What Changes Next

The next phase is usually not a full rewrite. It's a deliberate professionalization pass — the move we map stage by stage in From Prompting to Engineering — and the key is knowing which parts get which treatment. The behavior that proves the product is valuable? Keep it. The paths where users, money, or data depend on correctness? Harden those. The areas where every change requires a long explanation to the next developer? Refactor them. Only rebuild where the current shape genuinely prevents safe change.

In the full Post Code Labs consultation, which scores twelve categories from security to delivery, that thinking becomes a concrete set of artifacts: a scored checklist, file-level evidence, a risk register, a rewrite-versus-remediate recommendation, and a 30-60-90 day plan — not to make the repo look respectable, but to figure out what needs to be boring and stable before the next round of AI-assisted work lands on top of it.

What to Do Next Monday

Start with a small audit rather than a cleanup sprint.

  1. List every script, cron job, queue worker, migration, and manual deployment step. Mark which ones are still used, who owns them, and what happens when they fail.
  2. Trace one critical workflow from UI to database to background work. Write down the source of truth for each product decision along the path.
  3. Pick one recurring bug and convert it into an executable guardrail: a regression test, a stricter type, a lint rule, or a migration check.
  4. Add one release gate that currently depends on memory. Start with typecheck, lint, tests, build, or migration ordering before inventing a custom process.
  5. Update the agent instructions with the findings: forbidden shortcuts, required verification commands, ownership rules, and the point where a human must review.

If those steps feel straightforward, the app probably just needs routine hardening — running the free repo audit is a quick way to confirm the repository is ready for the next round of agent work. If they uncover conflicts, unowned jobs, untestable flows, and unclear data boundaries, the app has already outgrown the vibe — and that's not a failure. It means the product is real enough to deserve proper engineering, which is exactly what a full consultation maps out.

References