Event-Driven Run Ledger

Engineered an event-driven observability stack for monitoring concurrent AI enrichment pipelines. An EventBridge bus ingests job and project lifecycle events, a writer Lambda applies incremental updates to DynamoDB state tables, and a read-only REST API (secured with Cognito JWT authorization) exposes curated run data. A static S3/CloudFront web dashboard gives stakeholders real-time visibility into pipeline health, client configurations, and error states, while a CloudWatch dashboard with Logs Insights widgets supports operational triage.

Key Achievements

• Event-driven state tracking: Designed an EventBridge bus and rule that routes job_update and project_update events to a writer Lambda, which applies atomic DynamoDB updates using ADD-based delta counters to safely handle concurrent pipeline execution.
• Authenticated read API: Built a REST API with Cognito User Pool authorization (invite-only users and client-credentials for external systems) exposing endpoints for jobs, projects, clients, and client configuration with secret parameters automatically omitted.
• Intelligent query strategy: Implemented a multi-date scan strategy in the read API that automatically scans recent date partitions to fill the requested result limit when no specific filters are provided, keeping queries fast against time-partitioned DynamoDB tables.
• Live web dashboard: Delivered a static S3/CloudFront site that authenticates via the Cognito Hosted UI and renders live job statuses, project details, and client configurations by calling the secured API with bearer tokens.
• Operational observability: Provisioned a CloudWatch dashboard with Logs Insights widgets and direct links to log consoles, giving operations teams a single pane for filtering jobs by client, status, and result.