> **Multi-Agent Orchestration Platform** > Agentic AI patterns and autonomous workflow coordination through a production multi-agent orchestration platform. > > Source: https://jasonstiltner.com/projects/agentic-orchestration/ --- # Multi-Agent Orchestration Platform Exploring agentic AI patterns and autonomous workflow coordination Updated Sep 9, 2026 Agentic AI · NestJS + GPT-4 · Full-Stack Production · Technical Exploration ## Design Notes I built [Agentic-PR](https://agentic-pr.com) to work through multi-agent orchestration patterns end-to-end, including the business layer around them. The technical problem was the substantive one: coordinating autonomous AI agents, managing state across complex workflows, building production-grade systems for unpredictable LLM behavior. The architecture had to survive LLM output that doesn't behave like a normal API response — retryable failures, partial completions, and variance that a typed backend can't assume away. Solo-building the full stack (backend, AI service layer, web and mobile clients) forced every one of those decisions to be made explicitly rather than deferred to a team's division of labor. **Result:** a working demonstration of multi-agent coordination at production scale — ~8,000 lines of backend, a hybrid model-routing layer, and a real-time A/B testing system — that remains operational. The system design below is what shipped. ## System Architecture [Diagram: Multi-agent orchestration architecture diagram: a user request flows through an API gateway into a Bull/Redis job queue, which dispatches to Content, Sentiment, and Pitch agents; those agents call a shared AI service backed by MongoDB and Redis, while a WebSocket service pushes real-time status updates to the response. Links are colored by flow type: data, control, and real-time.] 👤 ## What I Personally Built #### Complete Backend Architecture (~8,000 lines) Designed and implemented the entire NestJS backend from scratch: modular service architecture, MongoDB/PostgreSQL data layer, Bull job queues with Redis, and WebSocket real-time communication layer. NestJS · TypeScript · MongoDB · Redis #### AI Service Layer Built the LLM integration layer: template-based prompt engineering, structured output parsing, and error handling with retries and fallbacks. A model-preference switch exists on the sentiment path, but only its GPT-4 branch calls a model — see [Limitations](#limitations). OpenAI GPT-4 · NestJS #### A/B Testing Infrastructure Implemented the A/B testing system: variant generation, random assignment, open/click tracking, weighted engagement scoring to pick a winner, and real-time dashboard updates via WebSocket. Queues · WebSocket · Real-time #### Full-Stack Web & Mobile Applications (~7,000 lines) Built the Next.js web application and React Native mobile apps (iOS/Android) with shared business logic, Chakra UI components, and real-time WebSocket integration for live updates. Next.js · React Native · Chakra UI #### Infrastructure & DevOps Set up the complete deployment pipeline: Docker containerization, Terraform infrastructure as code, nginx reverse proxy, OAuth 2.0 integration (Google, LinkedIn, Twitter), and JWT session management. Docker · Terraform · OAuth 2.0 ## Results & Outcomes 15,000+ Lines of TypeScript 3 Platform targets (Expo: web, iOS, Android) 5 AI Agents Orchestrated These count what is in the repository, not what was measured in production. A fourth figure — a ~60% API cost reduction from model routing — sat alongside them until it was checked against the source; it has been removed, and the reason is in [Limitations](#limitations). #### Technical Achievements - Production-grade LLM reliability with exponential backoff and fallbacks - Real-time WebSocket updates for live dashboard feedback - Per-call token and latency telemetry captured for every AI operation - A/B test harness for PR pitches and social posts, scored on weighted engagement ## Repository Structure Monorepo architecture with shared types and utilities agentic-pr/ ├── apps/ │ ├── api/ # NestJS Backend (~8,000 lines) │ │ ├── src/ │ │ │ ├── ai/ │ │ │ │ ├── ai.service.ts # LLM integration │ │ │ │ ├── ai-queue.producer.ts # Job queue producer │ │ │ │ ├── ai-queue.consumer.ts # Job queue consumer │ │ │ │ └── prompts/ # Template prompts │ │ │ ├── pr/ │ │ │ │ ├── pr.service.ts # PR pitch management │ │ │ │ ├── pr.controller.ts # REST endpoints │ │ │ │ └── dto/ # Data transfer objects │ │ │ ├── sentiment/ │ │ │ │ ├── sentiment.service.ts # Hybrid analysis │ │ │ │ ├── distilbert.service.ts # Fast model │ │ │ │ └── alerts.service.ts # Change detection │ │ │ ├── websocket/ │ │ │ │ ├── websocket.gateway.ts # Socket.io gateway │ │ │ │ └── websocket.service.ts # Real-time updates │ │ │ ├── auth/ │ │ │ │ ├── oauth.strategy.ts # Google/LinkedIn/Twitter │ │ │ │ └── jwt.strategy.ts # Session management │ │ │ └── common/ │ │ │ ├── filters/ # Error handling │ │ │ └── interceptors/ # Logging, transforms │ │ └── test/ # Integration tests │ │ │ ├── web/ # Next.js Web App (~4,000 lines) │ │ ├── src/ │ │ │ ├── pages/ │ │ │ ├── components/ │ │ │ │ ├── dashboard/ # Real-time dashboards │ │ │ │ ├── ab-testing/ # A/B test UI │ │ │ │ └── sentiment/ # Sentiment charts │ │ │ ├── hooks/ │ │ │ │ └── useWebSocket.ts # Real-time hook │ │ │ └── lib/ │ │ │ └── api.ts # API client │ │ └── public/ │ │ │ └── mobile/ # React Native (~3,000 lines) │ ├── src/ │ │ ├── screens/ │ │ ├── components/ │ │ └── services/ │ ├── ios/ │ └── android/ │ ├── packages/ │ ├── shared-types/ # TypeScript interfaces │ └── shared-utils/ # Common utilities │ ├── infrastructure/ │ ├── terraform/ │ │ ├── main.tf │ │ ├── variables.tf │ │ └── outputs.tf │ └── docker/ │ ├── Dockerfile.api │ ├── Dockerfile.web │ └── docker-compose.yml │ └── docs/ ├── architecture.md ├── api-reference.md └── deployment.md ## The Technical Challenge Traditional automation requires explicit programming of every step—if-then logic that breaks when conditions change. Agentic AI enables autonomous agents to make decisions and execute tasks, but introduces new challenges around coordination and reliability. The core question: **How do you orchestrate multiple autonomous AI agents reliably in production?** #### LLM Integration at Scale Production LLM systems need more than API calls—they need prompt engineering, response validation, cost optimization, and graceful handling of model failures and rate limits. #### Async Workflow Coordination AI tasks are inherently async and unpredictable. The system needs message queues, job prioritization, retry logic with exponential backoff, and real-time status updates. #### Multi-Model Orchestration Different tasks benefit from different models—a small local classifier for sentiment, GPT-4 for complex content generation. The dispatch that selects between them is built; the local classifier behind it is a stub, so what this demonstrates is the seam, not the saving. ## What I Built A complete production platform with ~15,000+ lines of TypeScript across backend, frontend, and mobile applications. The system handles AI-powered content generation, A/B testing, sentiment analysis, and real-time workflow coordination. ### AI Service Layer (~2,000 lines) Comprehensive LLM integration with template-based prompt engineering, structured output parsing, and fallback handling for API failures. AI Service - LLM Integration Production prompt engineering with template variables and structured completions 36 linesTypescriptNestJS + OpenAI View code 36 lines · Typescript · NestJS + OpenAI Copy ```typescript @Injectable() export class AiService { private readonly logger = new Logger(AiService.name); private openai: OpenAIApi; constructor(private configService: ConfigService) { const apiKey = this.configService.get('OPENAI_API_KEY'); const configuration = new Configuration({ apiKey }); this.openai = new OpenAIApi(configuration); } /** * Generate a completion using the OpenAI API */ async generateCompletion(prompt: string, params: Record = {}): Promise { // Replace template variables in the prompt let processedPrompt = prompt; for (const [key, value] of Object.entries(params)) { const placeholder = `{{${key}}}`; if (typeof value === 'string') { processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value); } else if (Array.isArray(value)) { processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value.join(', ')); } } const response = await this.openai.createCompletion({ model: 'gpt-4', prompt: processedPrompt, max_tokens: 2000, temperature: 0.7, }); return response.data.choices[0].text.trim(); } } ``` ### PR Service with A/B Testing (~770 lines) Automated A/B testing for AI-generated content with real-time tracking, statistical analysis, and WebSocket-based dashboard updates. PR Service - A/B Testing & Real-time Updates Variant selection, tracking, and WebSocket coordination for live dashboards 49 linesTypescriptNestJS + MongoDB + WebSocket View code 49 lines · Typescript · NestJS + MongoDB + WebSocket Copy ```typescript @Injectable() export class PRService { constructor( @InjectModel(PRPitch.name) private prPitchModel: Model, private aiService: AiService, private journalistService: JournalistService, private webSocketService: WebSocketService, ) {} /** * Send a PR pitch with A/B testing */ async sendPitch(id: number): Promise { const pitch = await this.findOne(id); const journalist = await this.journalistService.findOne(pitch.journalistId); // Randomly select variant A or B for this recipient const variant = Math.random() > 0.5 ? 'A' : 'B'; const subjectLine = variant === 'A' ? pitch.subjectLineA : pitch.subjectLineB; const content = variant === 'A' ? pitch.contentA : pitch.contentB; // Update pitch status and send real-time update via WebSocket pitch.status = 'sent'; const savedPitch = await pitch.save(); this.webSocketService.sendPRUpdate({ action: 'pitch_sent', pitch: { id: savedPitch._id, variant, timestamp: new Date().toISOString() } }); return savedPitch; } /** * Track email open for A/B test analysis */ async trackOpen(id: number, variant: 'A' | 'B'): Promise { const pitch = await this.findOne(id); variant === 'A' ? pitch.opensA += 1 : pitch.opensB += 1; // Real-time dashboard update this.webSocketService.sendABTestUpdate({ action: 'variant_opened', pitch: { id, variant, opensA: pitch.opensA, opensB: pitch.opensB } }); return pitch.save(); } } ``` ### Hybrid Sentiment Analysis (~940 lines) Multi-model sentiment analysis intended to combine DistilBERT for speed with GPT-4 for accuracy, plus automated alerting when sentiment changes significantly. Only the GPT-4 branch reaches a model; the DistilBERT branch is simulated. Sentiment Service - Hybrid ML Approach Model dispatch and alerting; the DistilBERT branch is a stub 55 linesTypescriptNestJS + OpenAI View code 55 lines · Typescript · NestJS + OpenAI Copy ```typescript @Injectable() export class SentimentService { private distilBertModel: any; private isModelLoaded: boolean = false; constructor( @InjectModel(SentimentAnalysis.name) private sentimentAnalysisModel: Model, private aiService: AiService, private userSettingsService: UserSettingsService, ) { this.initializeModels(); } /** * Analyze text sentiment using preferred model (DistilBERT or GPT-4) */ async analyzeSentiment(userId: string, dto: CreateSentimentAnalysisDto): Promise { const userSettings = await this.userSettingsService.getUserSettings(userId); const preferredModel = dto.preferredModel || userSettings?.preferredSentimentModel || SentimentModel.DISTILBERT; const sentimentAnalysis = new this.sentimentAnalysisModel({ userId, content: dto.content, modelUsed: preferredModel, isProcessed: false, }); const savedAnalysis = await sentimentAnalysis.save(); // Process asynchronously - hybrid DistilBERT + GPT-4 approach this.processSentimentAnalysis(savedAnalysis._id, preferredModel); return this.mapToResponseDto(savedAnalysis); } /** * Check if sentiment change warrants an alert */ private async checkForSentimentAlert(analysis: SentimentAnalysisDocument): Promise { const previousAnalyses = await this.sentimentAnalysisModel .find({ userId: analysis.userId, source: analysis.source }) .sort({ createdAt: -1 }).limit(5); const avgPreviousScore = previousAnalyses.reduce((sum, a) => sum + a.score, 0) / previousAnalyses.length; const changeMagnitude = Math.abs(analysis.score - avgPreviousScore); if (changeMagnitude >= this.alertThreshold) { await this.createSentimentAlert(analysis.userId, { title: 'Significant sentiment change detected', severity: changeMagnitude >= 0.7 ? AlertSeverity.CRITICAL : AlertSeverity.MEDIUM, recommendedActions: this.generateRecommendedActions(analysis, changeMagnitude) }); } } } ``` ### Async Queue System Production job queue with priority scheduling, retry logic, and exponential backoff for handling AI workloads reliably. Queue System - Async AI Processing Bull queue with priority, retries, and exponential backoff 35 linesTypescriptNestJS + Bull + Redis View code 35 lines · Typescript · NestJS + Bull + Redis Copy ```typescript // AI Queue Producer - Async job processing @Injectable() export class AiQueueProducer { constructor(@InjectQueue('ai-queue') private queue: Queue) {} async generateSocialMediaContent(params: SocialMediaContentParams, priority: number = 2) { const job = await this.queue.add( AiJobType.CONTENT_GENERATION, { params }, { priority, attempts: 3, backoff: { type: 'exponential', delay: 5000 }, }, ); return job.id; } } // AI Queue Consumer - Process jobs with retry logic @Processor('ai-queue') export class AiQueueConsumer { constructor(private readonly aiService: AiService) {} @Process(AiJobType.CONTENT_GENERATION) async processContentGeneration(job: Job) { this.logger.log(`Processing content generation job ${job.id}`); try { return await this.aiService.generateSocialMediaContent(job.data.params); } catch (error) { this.logger.error(`Error in job ${job.id}: ${error.message}`); throw error; // Triggers retry with exponential backoff } } } ``` ## Technical Stack #### Backend - **NestJS** (TypeScript) - Modular architecture - **MongoDB** - Document storage for workflows - **PostgreSQL** - Relational data - **Redis** - Caching & queue backend - **Bull** - Job queue with priorities - **Socket.io** - Real-time WebSocket #### AI/ML - **OpenAI GPT-4** - Content generation - **Template prompts** - Structured generation - **A/B testing** - Variant optimization - **Model dispatch** - Selection seam (local branch stubbed) #### Frontend - **Next.js** (React) - Web application - **React Native** - iOS/Android mobile - **Chakra UI** - Component library - **WebSocket** - Real-time updates - **TypeScript** - Type safety #### Infrastructure - **Docker** - Containerization - **Terraform** - Infrastructure as code - **nginx** - Reverse proxy - **OAuth 2.0** - Google, LinkedIn, Twitter - **JWT** - Session management ## Technical Challenges Solved #### Production LLM Reliability LLM APIs fail, rate limit, and produce unexpected outputs. Built comprehensive error handling with retries, fallbacks, and output validation. The queue system ensures no work is lost during failures. #### Real-time Coordination AI tasks run asynchronously but users need immediate feedback. Implemented WebSocket-based real-time updates so dashboards show live status of A/B tests, sentiment changes, and content generation. #### Cost Optimization GPT-4 API calls are expensive at scale, so the sentiment path was designed to send cheap classification to a local DistilBERT and reserve GPT-4 for generation. The dispatch was built; the local model behind it never was, so the saving is a design intent rather than an outcome. See [Limitations](#limitations). #### Multi-Platform Deployment Single codebase serving web, iOS, and Android with shared business logic. React Native mobile apps connect to the same backend APIs with real-time WebSocket support. ## Mathematical Formulation The formulations a harness like this one needs in order to make its outputs trustworthy. These are the standard results, written out to fix what the platform is aiming at — not a description of code that runs. The A/B tests currently pick a winner by weighted-score comparison, with none of the significance machinery below; the [Limitations](#limitations) section says which parts are missing. ### A/B Testing Statistical Analysis **Conversion Rate Estimation:** where \= opens/clicks, \= emails sent per variant. **Z-Test for Significance:** where is the pooled proportion. Reject null hypothesis if (95% confidence). **Sample Size for Power:** Minimum samples needed to detect effect size with 80% power. ### Sentiment Analysis & Alerting **Sentiment Score (DistilBERT):** where is the pooled representation from DistilBERT, = sigmoid activation. **Change Detection (Moving Average):** Alert triggered when (default ). **Alert Severity Classification:** ### Job Queue Priority & Retry Logic **Exponential Backoff:** where \= retry attempt, ms base delay, = jitter to prevent thundering herd. **Priority Queue Ordering:** Jobs ordered by priority (lower = higher priority), then by timestamp (FIFO within priority). ## Key Technical Learnings > "Production LLM systems are 20% prompt engineering and 80% infrastructure—error handling, retries, cost management, and observability. The happy path is easy; reliability is hard." > "Leaving the dispatch seam in place while the cheap model behind it stays a stub is how a cost saving gets claimed without ever being earned. The branch is the easy part; the model is the work." > "Real-time feedback transforms user experience with AI systems. WebSocket updates showing live progress make async AI tasks feel responsive and trustworthy." #### Production Insights - Exponential backoff for API failures - Output validation before storage - Per-call token and latency tracking - Structured logging for debugging #### Architecture Patterns - Event-driven async processing - Priority queues for job scheduling - WebSocket for real-time updates - Shared-type monorepo ## Limitations ### The limitation that conditions the rest: you cannot open the repository `jstiltner/agentic-pr` is private, and this page does not link to it. Every file named below — `sentiment.service.ts`, `ai-metrics.service.ts`, `openai.service.ts`, `docs/PlanToLowerCost.md`, `app.json` — is a path you have no way to open. The line counts and component totals higher up the page have the same problem. So do the corrections in this section: they are specific because the code is specific, but specificity is not evidence, and on this page you have only my word for any of it. Every other project on this site can be checked against something. This one cannot, and that is a real difference in what the two kinds of page are worth. The orchestration patterns on this page are implemented. Several things an earlier version of this page implied were also implemented are not, and they are listed here because a reader has no way to find them in the source themselves. #### The cheap half of the “hybrid” is a stub returning random numbers The dispatch is real: `sentiment.service.ts` branches on a user model preference and sends work to either a DistilBERT path or GPT-4. The DistilBERT path does not load DistilBERT. `initializeModels()` sleeps for a second and sets `isModelLoaded = true` without loading anything, and `analyzeWithDistilBERT()` sleeps 500 ms and returns `Math.random()` as the sentiment score, `0.7 + Math.random() * 0.3` as its own confidence, and randomly chosen three-word slices of the input as “key phrases.” The TensorFlow import sits commented out above it. Routing half the traffic to that branch would cut the API bill, which is the sense in which the saving was real; it would also replace the answers with noise, which is the sense in which it was not. #### The ~60% figure came from a plan, not a run This page reported a ~60% API cost reduction from hybrid model routing. The number traces to `docs/PlanToLowerCost.md`, which lists “Estimated Savings: 60-90%” beneath a heading reading *Implementation Plan* — a proposal for work not yet done, published as though it were a result. Nothing was ever compared against a GPT-4-only baseline. The generation paths make the point plainly: `openai.service.ts` and `ai.service.ts` both hard-code `'gpt-4'`, and `podcast-youtube-discovery.service.ts` reads a single env var defaulting to `'gpt-4-turbo'`. No content generation is routed anywhere but GPT-4. #### Cost is not actually measured — tokens and latency are `ai-metrics.service.ts` records token counts, latency and success per operation, which is the instrumentation a cost comparison would be built on. It is not that comparison. There is no per-model price table, no baseline run, and the counters live in memory and reset with the process — nothing is persisted, so no historical series exists to analyse. #### The A/B tests pick winners, but not significantly The harness is real — dedicated modules, controllers, queue processors and schemas for both PR pitches and social posts. A winner is chosen by comparing hand-weighted sums of engagement rates (`openRate × 1 + clickRate × 2 + responseRate × 3`) and taking the larger. There is no p-value, confidence interval or power calculation in the codebase. Describing this as “statistically rigorous” overstated it, and that wording is gone. #### Three platforms are targeted, not shipped The mobile client is an Expo app whose `app.json` declares iOS and Android identifiers, so all three targets are configured. No native project has been generated and no build has been produced for either store. The declared bundle identifiers contain a space (`com.Agentic PR.mobile`), which would fail a native build outright — good evidence that one has never been attempted. **Why the page still stands:** the queue topology, the retry and fallback behaviour, the WebSocket progress channel and the A/B harness are what the project was for, and they are the parts no claim here depends on measuring. The removed claims were the ones that needed a measurement nobody took. Any number that returns here should arrive with the run that produced it — and, given the repository is private, with something a reader can actually open. ## Scope Full-stack, solo build: backend architecture, the AI service layer, the real-time coordination system, and both the web and mobile clients. The business layer (customer development, sales, marketing) ran alongside it but isn't what's documented here — this page covers the system design. [See production ML work: Document Understanding](https://jasonstiltner.com/projects/document-understanding/)