Operations

This page is for the people who run a dms-saas deployment. Every money-moving workflow persists its intent before any Stripe or lifecycle effect, refuses timed takeovers, and stops in a reconciliation required state rather than replaying an effect it cannot prove. That trades automatic crash recovery for protection against double charges and conflicting external effects.

Start with the recorded outcome

Suppose a customer-credit request times out. The timeout does not tell you whether Stripe applied the credit. First inspect the existing operation:

GET /api/saas/workspaces/<tenantId>/operations/<operationId>

This endpoint requires platform-owner authentication. Keep the operation ID and original values: a new ID would describe a new action, not a retry.

Reconciliation means investigating the recorded effects and proving the previous executor has stopped. It is not a button that forces success. The sections below identify the evidence to inspect for each workflow.

Operator actions

The platform-owner workspace page provides suspension, reactivation, immediate Stripe-backed upgrades, and Stripe customer-balance credits. Every request uses a caller-supplied operation ID and creates a private durable command intent before any Stripe or lifecycle effect. The command state retains the actor, tenant, action, redacted-safe values, attempts, outcome, and effective time. Retrying the same ID and values resumes an eligible failed operation; reusing it with different values is rejected.

Retry or reconcile

Operator attempts do not use leases or timed takeover. The outcome determines whether a retry is safe:

The 23-hour retry window preserves a one-hour margin inside Stripe's minimum 24-hour idempotency retention.

Suspension, reactivation, and immediate plan changes persist a subscription transition before effects. Its revision fences competing local transitions and deletion; the operator journal must succeed before its intent is cleared. An ordinary subscription write rejects pending transitions and deletion markers. This does not provide distributed transactions or fence an external request that has already left the process.

Inspect and resolve an operation

Inspect an operation through the platform-owner-only endpoint GET /api/saas/workspaces/:tenantId/operations/:operationId. A running or reconciliation-required operation may have an old executor still active.

Before resolving it, establish both:

1. The previous executor is stopped or quiescent.
2. What happened externally, using the persisted operation, provider request details, and provider records.

A matching Stripe read alone does not establish quiescence. There is deliberately no force-success endpoint that clears the intent without those proofs. This trades automatic crash recovery and availability for protection against conflicting external effects.

Plan migrations

Bulk plan migrations capture an immutable workspace list and resolved target configuration before admitting an execution. Each workspace gets a persisted outcome and a subscription transition before provider effects. A crash after changing planId does not remove that workspace from the snapshot. Running jobs never use timed takeover; startup marks interrupted jobs as reconciliation_required, retaining their per-workspace evidence and pending subscription intents instead of blindly replaying effects.

The migrate-and-delete endpoint requests a migration only. Inspect GET /api/saas/plans-deletion/migrations/:migrationId for the snapshot, outcomes, and reconciliation requirements. Snapshot completion does not retire the source plan. The source remains available even at zero observed references; workspaces arriving after capture are outside this job. Any later retirement requires a separate procedure that establishes writer quiescence. A current reference count cannot exclude a suspended writer that already validated the plan.

Migration reconciliation requires stopping or proving quiescence of the prior executor and establishing the provider, subscription, permission-cleanup, and notification outcomes. A target planId or matching Stripe response alone is not sufficient. Failed core lifecycle admissions may also remain closed to destruction until their evidence is resolved. There is no automatic force-success or effect-replay endpoint; arbitrary hooks are not made idempotent by a job ID.

Self-service changes and checkout

Self-service cancellation, deferred plan changes, and administrative free grants also admit subscription intent before Stripe effects. An ambiguous schedule replacement retains the previous mirror and pending intent; it does not attempt to restore or erase provider state blindly. Owners can inspect these intents and the irreversible deletion marker through the workspace admin-state endpoint.

Checkout holds a checkout intent until the exact recorded session completes or expires. Completion admits one callback executor and writes a terminal session receipt only after its awaited provider effects finish. Unknown callback outcomes retain intent rather than enabling timed replay. A session whose ID was never acknowledged locally requires manual reconciliation, including executor quiescence. Trial reservations are permanent across checkout expiration; uncertain attempts do not receive another trial. Checkout automation remains best-effort, outside the terminal receipt, and does not promise exactly-once delivery or external order.

Stripe webhooks

Stripe webhook dispatch admits an event once and fences completion with its admitted revision. An old pending record never permits timed takeover. Handler failures become reconciliation_required, which blocks replay because the attempt may have partial effects. Redelivery receives an error while unresolved, rather than a successful acknowledgement or another execution.

Inspect stripe_webhook_events by Stripe event ID for its result, revision, and error message. Recovery requires establishing the prior executor's quiescence and the handler's database and external outcomes; do not delete the admission or resend blindly. There is no automatic recovery endpoint. Unresolved records remain indefinitely. The existing retention policy still removes old terminal success/skipped receipts, so deduplication is not guaranteed after that window.

Revision mutations require the published database interface 0.1.6 or later and an adapter implementing that contract. The MongoDB test adapter uses 1.3.1, and any DMS release carrying this module satisfies the requirement. Every record is written with a revision, and these mutations refuse a record without one; see Upgrading for rows written by earlier builds.

Tenant lifecycle consumers

Lifecycle-dependent modules register a required consumer through the public interface:

import {
  RegisterTenantLifecycleConsumer,
  type TenantLifecycleConsumerRegistration,
} from "@antelopejs/interface-dms-saas/tenant-lifecycle";

let lifecycleRegistration: TenantLifecycleConsumerRegistration | undefined;

export function construct(): void {
  lifecycleRegistration = RegisterTenantLifecycleConsumer({
    name: "cloud.workspaces",
    transitions: ["suspended", "reactivation_requested"],
    async consume(message) {
      const receipt = await applyWorkspaceLifecycle(message);
      return {
        receiptId: receipt.id,
        effectiveAt: receipt.effectiveAt,
      };
    },
  });
}

export function stop(): void {
  lifecycleRegistration?.unregister();
  lifecycleRegistration = undefined;
}

Registration lifecycle

Register required consumers during module construction, before module startup reconciliation. Registration may happen before dms-saas construction: dms-saas replays buffered consumers when it attaches its module-owned registry.

Registrations are removed automatically when their owning module unloads; the explicit handle remains available for module stop compatibility.

Delivery guarantees and access

Messages carry stable tenantId and operationId values. Deliveries and receipts are persisted; a persisted success is skipped on later retries.

Concurrent reconcilers may invoke the same consumer more than once before its receipt is persisted. Consumers must deduplicate concurrent calls by operation identity; the outbox does not provide exactly-once execution or ordering of external effects. Failed deliveries remain available to startup reconciliation and a five-minute retry worker.

Access depends on the transition:

• Suspension closes SaaS access before consumer delivery.
• Reactivation keeps SaaS access closed until every required consumer returns a durable receipt.

The consumer therefore owns any product-specific sleep or wake work.

Successful workspace creation

Provisioning reserves free-workspace capacity per card and trial eligibility per email/card identity before issuing provider effects. Reservations do not expire. An unknown provider or persistence outcome keeps the reservation and the account for reconciliation; the service never replays opaque provisioning hooks or stores their extras. Known provider handles and the attempt ID are persisted separately from billing profiles. Stripe customer metadata also contains the attempt ID for investigating a lost creation response.

Platform owners inspect the first 100 unresolved records through GET /api/saas/workspaces/provisioning-attempts and an individual record through GET /api/saas/workspaces/:tenantId/provisioning-attempt. Preparing records can represent an executor that disappeared before recording its error. Establish executor quiescence, provider cancellation or successful creation, and tenant closure before repairing a record or releasing an allocation. This release does not provide an automated repair command. Never release uncertain reservations by age or re-run non-idempotent hooks with guessed input.

Every consumer declares the transitions it handles, for example transitions: ["created"] for creation only or all three when it also handles suspension and reactivation. A consumer must return a durable receipt and deduplicate effects by operationId: workspace-created:<tenantId> remains stable across delivery retries.

Self-service provisioning writes a creation marker in the existing lifecycle outbox before inserting the tenant. Only after ownership, billing records and every TENANT_BEING_PROVISIONED listener succeed does it allow payment confirmation. Trials and creations without an invoice require no payment; zero-total invoices already marked paid are not paid again. An invoice-backed creation becomes committed only after Stripe reports its first invoice paid. Consumer failures never roll back committed workspaces.

Inventory reconcilers must check the public provisioning gate before creating external resources. This check prevents a concurrent inventory scan from provisioning a tenant whose membership, hooks or payment are still incomplete. It does not replace tenant existence, authorization or suspension checks.

import { IsTenantProvisioningCommitted } from
  "@antelopejs/interface-dms-saas/tenant-lifecycle";

async function reconcileTenant(tenantId: string): Promise<void> {
  if (!(await IsTenantProvisioningCommitted(tenantId))) return;
  await ensureWorkspaceResources(tenantId);
}

The gate returns true for legacy/default/bootstrap tenants without a creation marker. It returns false for preparing, unpaid or cancelled creations, and propagates storage failures; callers must fail closed. Cancellation leaves a tombstone so even a partial rollback cannot make an orphaned tenant eligible. Do not delete these markers while the tenant may remain in an inventory.

Reconciliation recovers a crash after successful payment but before delivery, partial consumer fan-out and lost receipts. It may deliver a message again when a consumer commits its effect before the receipt is saved. Consumers must make their resource creation and ownership linkage idempotent. Delivery skips tenants already deleted when the attempt starts; consumers must also tolerate deletion racing with an in-flight attempt.

A definitive card decline permits rollback only after tenant lifecycle admission closes and all required deletion hooks succeed. An incomplete closure preserves data, account ownership, and recovery markers. Rollback never drops the tenant schema or the lifecycle/invitation tombstones. An ambiguous payment error preserves the tenant and account because a timeout can hide a successful charge. Reconciliation checks payment status but never starts another charge. A crash before payment, an unpaid ambiguous result, or an interrupted pre-payment preparation requires payment/provisioning recovery; it never emits created merely because the tenant exists. HTTP creation requests are not made idempotent by the delivery operation ID.

Register consumers before startup reconciliation; use the existing tenant inventory and gate for bootstrap/backfill rather than treating created as a historical inventory API.