Computer Vision Edge Cases: Why Plant Models Agree on Species But Disagree on Health

In modern computer vision, image classification accuracy for botanical species identification (pflanze erkennen) has largely reached production maturity. With datasets like PlantNet or iNaturalist fine-tuned on ResNet or Vision Transformer (ViT) backbones, top-1 accuracy for identifying plant species routinely exceeds 95%.

However, when developers attempt to apply the exact same neural network architectures to plant health diagnosis (braune blätter an pflanzen / chlorosis / pest damage), model performance frequently collapses.

Why can a Vision Transformer instantly identify a Ficus benjamina leaf, yet completely fail to distinguish whether its yellowing is caused by overwatering, root rot, low light (zimmerpflanzen mit wenig licht), or nitrogen deficiency?

In this technical article, we'll analyze the root causes of model divergence in plant pathology. We'll explore the mathematical problem of overlapping visual feature spaces, examine why a single deterministic label is the wrong output shape for this problem, and walk through how a production system — GreenLens, specifically — handles this ambiguity by prompting a multimodal LLM to reason like a differential diagnosis rather than a hard classifier, backed by a two-tier model strategy for the cases that are genuinely hard to call.

1. The Overlapping Feature Problem in Plant Pathology

In taxonomic classification, species possess distinct, stable visual boundaries—such as leaf serration, venation geometry, or petal arrangement.

In contrast, plant stress responses are bottlenecked by plant physiology. Plants have a limited repertoire of visual stress signals. Chlorosis (loss of chlorophyll leading to yellow leaves) looks visually near-identical whether triggered by:

Overwatering: Oxygen-starved roots cannot uptake nutrients.
Underwatering: Dehydration causes lower leaves to drop.
Low Ambient Light (zimmerpflanzen mit wenig licht): Plant re-absorbs mobile nitrogen from old leaves to fuel new top growth.
Nitrogen Deficiency: Chlorophyll synthesis halts in lower foliage.

flowchart TD
    SubGraph1[Root Cause A: Overwatering] --> Feature[Symptom: Interveinal Chlorosis / Yellow Leaf]
    SubGraph2[Root Cause B: Underwatering] --> Feature
    SubGraph3[Root Cause C: Low Light Exposure] --> Feature
    SubGraph4[Root Cause D: Nitrogen Starvation] --> Feature

    Feature --> Model{Standard CNN Model}
    Model -->|High Ambiguity| ConfusedPrediction[Incorrect Diagnosis / False Confidence]

Because four fundamentally different care conditions produce overlapping RGB feature vectors, a pure image-only model faces a mathematically ill-posed problem.

2. Analyzing Spatial & Color Feature Breakdown

Let's break down the visual features of chlorosis and tip burn (braune spitzen pflanze) in different color spaces:

RGB vs. HSV Color Space Analysis

In standard RGB space, yellowing leaves display elevated Red and Green channels (R \approx 200, G \approx 200, B \approx 50). However, RGB channels are tightly coupled to ambient light intensity and shadow variations.

Converting image tensors to HSV (Hue, Saturation, Value) space isolates the true chlorophyll decay rate:

H = \arctan2(\sqrt{3} \cdot (G - B), 2R - G - B)

Healthy Foliage: Hue angle H \in [80^\circ, 140^\circ] (Deep Emerald Green).
Chlorosis / Fading: Hue angle H \in [45^\circ, 75^\circ] (Pale Yellow/Lime).
Necrosis / Crispy Edges (braune blätter): Hue angle H \in [10^\circ, 35^\circ] (Brown/Amber).

While HSV transformations help segment where the damage occurs, they still can't tell us why it occurred — and this is exactly the point at which a lot of plant-ID tooling either gives up (returns a single guess with false confidence) or, on paper, reaches for sensor fusion (soil moisture probes, light meters) that most consumer apps simply don't have access to. A phone camera has no idea how often the plant was watered last week.

3. The Actual Fix: Prompting for a Differential Diagnosis, Not a Label

GreenLens doesn't have soil moisture telemetry, and it doesn't train a fusion network. What it does have is a photo, and a general-purpose multimodal model (OpenAI's gpt-5-mini by default, gpt-5 for a higher-accuracy review pass, both behind an OPENAI_HEALTH_MODEL_CHAIN fallback to gpt-4.1-mini) that already has broad world knowledge about plant care baked into its pretraining. The engineering problem isn't "how do we fuse more input modalities" — it's "how do we get the model to admit ambiguity instead of confidently picking a wrong single cause."

Concretely, the /v1/health-check prompt is designed around the overlapping-feature problem described above:

It asks for multiple plausible causes with independent confidence scores, not a single winning label. If interveinal yellowing could be overwatering, underwatering, low light, or nitrogen deficiency, a well-designed prompt gets the model to say so explicitly — "likely overwatering (based on the described leaf softness), but low light is also plausible" — rather than forcing a false single answer the way a Softmax output head would.
It asks the model to point at the visual evidence that discriminates between causes. A model that's actually reasoning (rather than pattern-matching a single label) can call out things like leaf turgor, spot pattern, distribution across the plant (all leaves vs. just older ones), which map onto real diagnostic heuristics botanists use — nitrogen deficiency shows up in older/lower leaves first because the plant is mobilizing nitrogen from them; overwatering tends to be more uniform.
It leans on the model's own knowledge of species-typical vulnerabilities instead of a hand-maintained lookup table, since that knowledge generalizes across far more species than any small team could realistically curate rules for.
Genuinely ambiguous cases get escalated to the review-tier model. When the fast-tier pass comes back with low confidence or multiple competing causes close in probability, that's a signal worth spending the extra cost of a gpt-5 pass on — the same two-tier strategy described in the companion piece on GreenLens's diagnostic architecture.

Here's a simplified version of what that prompt-and-parse flow looks like server-side (the real implementation lives in server/lib/openai.js, but the shape is representative):

// server/lib/openai.js (simplified)
const HEALTH_CHECK_SYSTEM_PROMPT = `
You are a plant health diagnostic assistant. Given a photo of a plant,
identify visible symptoms and return STRICT JSON matching this shape:

{
  "species": string,
  "possibleCauses": [
    { "cause": string, "confidence": number, "evidence": string }
  ],
  "recommendedActions": string[]
}

Do not collapse ambiguous symptoms into a single cause. If multiple
causes are plausible from the image alone, list them ranked by
confidence and explain what visual evidence supports each one.
`;

async function runHealthCheck(imageBase64, { tier = 'standard' } = {}) {
  const modelChain = tier === 'pro'
    ? [process.env.OPENAI_HEALTH_REVIEW_MODEL || 'gpt-5']
    : (process.env.OPENAI_HEALTH_MODEL_CHAIN || 'gpt-5-mini,gpt-4.1-mini').split(',');

  let lastError;
  for (const model of modelChain) {
    try {
      const response = await callOpenAI({
        model,
        systemPrompt: HEALTH_CHECK_SYSTEM_PROMPT,
        image: imageBase64,
      });
      const parsed = parseAndValidateHealthJson(response);

      // Ambiguous fast-tier result — escalate to the review model
      const topTwo = parsed.possibleCauses.slice(0, 2);
      const isAmbiguous = topTwo.length === 2 &&
        Math.abs(topTwo[0].confidence - topTwo[1].confidence) < 0.15;

      if (isAmbiguous && model !== 'gpt-5') {
        return runHealthCheck(imageBase64, { tier: 'pro' });
      }
      return parsed;
    } catch (err) {
      lastError = err; // fall through to the next model in the chain
    }
  }
  throw lastError;
}

Nothing here is trained. There's no loss function, no gradient descent, no held-out validation split. The "model" in the traditional ML sense is entirely OpenAI's — what GreenLens owns is the prompt design, the JSON schema, the retry/fallback logic, and the escalation heuristic that decides when a case is worth a second, more expensive pass.

4. Why This Beats Forcing a Single Label — and Where It Still Breaks

The differential-diagnosis framing is a genuine improvement over a hard classifier for this problem, but it's worth being honest about where it still struggles:

It's only as good as the photo. No amount of prompt engineering recovers information that isn't in the image — a single leaf photographed without context (no visible stem, no soil, no sense of scale) gives the model less to work with than a wider shot, and the model's confidence scores should (and generally do) reflect that.
It can't see root rot. The single most common cause of chlorosis and wilting in houseplants is invisible from a leaf photo until the plant is already in serious trouble. This is a fundamental limitation of any vision-only system, not something a bigger model fixes.
User-supplied context helps more than better prompting. A one-line answer to "how often do you water this?" disambiguates overwatering vs. underwatering far more reliably than any amount of visual reasoning about leaf color — which is why the most useful next step for this kind of system is usually collecting a little bit of user context, not chasing marginal gains on image analysis alone.
Confidence isn't free lunch. Asking a model to express uncertainty is better than false certainty, but it also means the product has to be designed to show that uncertainty usefully — a ranked list of three possible causes is only helpful if the UI and the copy make clear that's not a definitive diagnosis.

Summary & Key Insights

Plant symptoms genuinely overlap. Visual signals like yellowing leaves or brown tips can stem from multiple, opposing care mistakes, and no image classifier — trained or prompted — can fully resolve that ambiguity from pixels alone.
A single deterministic label is the wrong output shape for this problem. Prompting a multimodal model to return ranked, confidence-scored possible causes is a better fit than forcing a Softmax-style single answer.
You don't need sensor fusion to make progress — you need honest uncertainty. GreenLens doesn't have soil-moisture telemetry; it gets more mileage from asking the model to reason explicitly about competing causes and from escalating ambiguous cases to a higher-accuracy review pass.
The real lever for disambiguation is user-supplied context, not a bigger model. A single follow-up question about watering habits often resolves what a photo alone cannot.

To explore how AI diagnostics isolate root causes for yellowing leaves and plant stress, visit the GreenLens Pro Symptom Guide.

12 KiB Raw Blame History