Production AI at the CRM boundary

Context

Revenue teams run on CRM objects, meeting artifacts, and chat. An AI product in that environment has to assemble a trustworthy picture of what happened, then produce output a salesperson can act on.

The engineering surface is not “call a model.” It is ingestion, identity, synchronization, background work, and safe writes back into operational systems.

Problem

CRM APIs, meeting providers, and Slack (or similar) do not share a clock, an identity model, or a delivery guarantee. Events arrive out of order. Webhooks retry. Objects are partially updated. The same meeting can appear as several records.

If generation runs against whatever is in the database at that moment, the system produces fluent intelligence that is locally plausible and globally wrong. Those failures are hard to reproduce because the missing piece is often an event that never arrived, arrived twice, or arrived under a different ID.

Constraints

Customer and employer data cannot leak into logs, prompts, or eval sets without controls.

CRM writes must be conservative: duplicates and retries should not create junk records.

Latency budgets differ by workflow. A Slack reply and a CRM backfill are not the same job.

The implementation has to be debuggable by a human at 2 a.m., not only by the person who wrote the prompt.

Architecture

External systems emit events. An ingestion layer accepts webhooks and API fetches, persists the raw payload, and enqueues work. Processing resolves identity, updates internal state, and only then assembles context for a model. Downstream actions (Slack messages, CRM notes, tasks) are explicit jobs with their own retry policy.

The important boundary: persistence of evidence happens before interpretation. AI is a consumer of assembled context, not the system of record.

Engineering decisions

Persist raw events before transforming them. If interpretation is wrong, you can replay. If you only store the model’s output, you cannot.

Make processing idempotent on a stable event key. Retries are normal; double writes are the bug.

Separate collection from generation. Mixing “fetch the deal” and “write the coaching note” in one request makes failure modes illegible.

Treat prompt and retrieval configuration as versioned artifacts. When output quality changes, you need to know whether the model, the context, or the data contract moved.

Tradeoffs

Synchronous generation is easier to demo and easier to reason about in a single request. It collapses under webhook bursts, slow CRM reads, and model latency.

Queued workers add operational surface (visibility, poison messages, retry storms) but they are how you survive partial failure.

Fetching CRM state on demand is fresher; caching it is faster and cheaper. The choice depends on whether stale context is a correctness bug or a UX delay. For qualification and coaching, stale context is usually a correctness bug.

Failure modes

Duplicate webhooks creating duplicate downstream actions.

Identity mismatch: the same person or meeting represented under different IDs across systems.

Auth expiry on a long-running sync.

Retrieval that omits the one field that would have contradicted the model’s summary.

A deployment that changes chunking, filters, or tool schemas without a regression harness.

Validation

Replay captured (redacted) event sequences through the worker and assert idempotency.

Compare generated artifacts against source evidence: every material claim should point at a record the system actually stored.

Watch retry metrics and dead-letter queues as first-class signals, not afterthoughts.

Lessons

In production AI, context assembly is the product. The model is a component.

Most “the AI is inconsistent” reports are integration, timing, or retrieval problems wearing a language-model costume.