Salesbuildr - Slow or failing distributor product search – Incident details

All systems operational

Slow or failing distributor product search

Resolved
Major outage
Started 5 months agoLasted 4 hours 47 minutes

Affected

Distributor Hub

Operational from 7:30 AM to 12:17 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved (but we'll keep actively monitoring)

  • Monitoring
    UTC
    Monitoring

    The Elasticsearch cluster backing distributor product search has been restored to a healthy state. Search response times and indexing latency are back to normal. We have applied configuration changes to the index to prevent recurrence of this specific failure mode and are following up with engineering work to make this class of incident self-healing in the future.

  • Update
    UTC
    Update

    The root cause has been identified as a search index that fell behind on background maintenance, leading to high resource pressure on our Elasticsearch cluster. We are applying tuning changes to the index and clearing the backlog. Search results may continue to be slow during recovery.

  • Identified
    UTC
    Identified

    We are investigating elevated latency and intermittent failures when searching distributor product catalogs. Some searches may take significantly longer than usual or time out. Indexing of price and inventory updates from distributors is also delayed. Other parts of the platform are unaffected.