Multi-Agent Orchestration Platform

Exploring agentic AI patterns and autonomous workflow coordination

Updated Sep 9, 2026
Agentic AINestJS + GPT-4Full-Stack ProductionTechnical Exploration

Design Notes

I built Agentic-PR to work through multi-agent orchestration patterns end-to-end, including the business layer around them. The technical problem was the substantive one: coordinating autonomous AI agents, managing state across complex workflows, building production-grade systems for unpredictable LLM behavior.

The architecture had to survive LLM output that doesn't behave like a normal API response — retryable failures, partial completions, and variance that a typed backend can't assume away. Solo-building the full stack (backend, AI service layer, web and mobile clients) forced every one of those decisions to be made explicitly rather than deferred to a team's division of labor.

Result: a working demonstration of multi-agent coordination at production scale — ~8,000 lines of backend, a hybrid model-routing layer, and a real-time A/B testing system — that remains operational. The system design below is what shipped.

System Architecture

👤

What I Personally Built

Complete Backend Architecture (~8,000 lines)

Designed and implemented the entire NestJS backend from scratch: modular service architecture, MongoDB/PostgreSQL data layer, Bull job queues with Redis, and WebSocket real-time communication layer.

NestJSTypeScriptMongoDBRedis

AI Service Layer

Built the LLM integration layer: template-based prompt engineering, structured output parsing, and error handling with retries and fallbacks. A model-preference switch exists on the sentiment path, but only its GPT-4 branch calls a model — see Limitations.

OpenAI GPT-4NestJS

A/B Testing Infrastructure

Implemented the A/B testing system: variant generation, random assignment, open/click tracking, weighted engagement scoring to pick a winner, and real-time dashboard updates via WebSocket.

QueuesWebSocketReal-time

Full-Stack Web & Mobile Applications (~7,000 lines)

Built the Next.js web application and React Native mobile apps (iOS/Android) with shared business logic, Chakra UI components, and real-time WebSocket integration for live updates.

Next.jsReact NativeChakra UI

Infrastructure & DevOps

Set up the complete deployment pipeline: Docker containerization, Terraform infrastructure as code, nginx reverse proxy, OAuth 2.0 integration (Google, LinkedIn, Twitter), and JWT session management.

DockerTerraformOAuth 2.0

Results & Outcomes

15,000+
Lines of TypeScript
3
Platform targets (Expo: web, iOS, Android)
5
AI Agents Orchestrated

These count what is in the repository, not what was measured in production. A fourth figure — a ~60% API cost reduction from model routing — sat alongside them until it was checked against the source; it has been removed, and the reason is in Limitations.

Technical Achievements

  • Production-grade LLM reliability with exponential backoff and fallbacks
  • Real-time WebSocket updates for live dashboard feedback
  • Per-call token and latency telemetry captured for every AI operation
  • A/B test harness for PR pitches and social posts, scored on weighted engagement

Repository Structure

Monorepo architecture with shared types and utilities

agentic-pr/
├── apps/
│   ├── api/                          # NestJS Backend (~8,000 lines)
│   │   ├── src/
│   │   │   ├── ai/
│   │   │   │   ├── ai.service.ts           # LLM integration
│   │   │   │   ├── ai-queue.producer.ts    # Job queue producer
│   │   │   │   ├── ai-queue.consumer.ts    # Job queue consumer
│   │   │   │   └── prompts/                # Template prompts
│   │   │   ├── pr/
│   │   │   │   ├── pr.service.ts           # PR pitch management
│   │   │   │   ├── pr.controller.ts        # REST endpoints
│   │   │   │   └── dto/                    # Data transfer objects
│   │   │   ├── sentiment/
│   │   │   │   ├── sentiment.service.ts    # Hybrid analysis
│   │   │   │   ├── distilbert.service.ts   # Fast model
│   │   │   │   └── alerts.service.ts       # Change detection
│   │   │   ├── websocket/
│   │   │   │   ├── websocket.gateway.ts    # Socket.io gateway
│   │   │   │   └── websocket.service.ts    # Real-time updates
│   │   │   ├── auth/
│   │   │   │   ├── oauth.strategy.ts       # Google/LinkedIn/Twitter
│   │   │   │   └── jwt.strategy.ts         # Session management
│   │   │   └── common/
│   │   │       ├── filters/                # Error handling
│   │   │       └── interceptors/           # Logging, transforms
│   │   └── test/                           # Integration tests
│   │
│   ├── web/                          # Next.js Web App (~4,000 lines)
│   │   ├── src/
│   │   │   ├── pages/
│   │   │   ├── components/
│   │   │   │   ├── dashboard/              # Real-time dashboards
│   │   │   │   ├── ab-testing/             # A/B test UI
│   │   │   │   └── sentiment/              # Sentiment charts
│   │   │   ├── hooks/
│   │   │   │   └── useWebSocket.ts         # Real-time hook
│   │   │   └── lib/
│   │   │       └── api.ts                  # API client
│   │   └── public/
│   │
│   └── mobile/                       # React Native (~3,000 lines)
│       ├── src/
│       │   ├── screens/
│       │   ├── components/
│       │   └── services/
│       ├── ios/
│       └── android/
│
├── packages/
│   ├── shared-types/                 # TypeScript interfaces
│   └── shared-utils/                 # Common utilities
│
├── infrastructure/
│   ├── terraform/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── docker/
│       ├── Dockerfile.api
│       ├── Dockerfile.web
│       └── docker-compose.yml
│
└── docs/
    ├── architecture.md
    ├── api-reference.md
    └── deployment.md

The Technical Challenge

Traditional automation requires explicit programming of every step—if-then logic that breaks when conditions change. Agentic AI enables autonomous agents to make decisions and execute tasks, but introduces new challenges around coordination and reliability.

The core question: How do you orchestrate multiple autonomous AI agents reliably in production?

LLM Integration at Scale

Production LLM systems need more than API calls—they need prompt engineering, response validation, cost optimization, and graceful handling of model failures and rate limits.

Async Workflow Coordination

AI tasks are inherently async and unpredictable. The system needs message queues, job prioritization, retry logic with exponential backoff, and real-time status updates.

Multi-Model Orchestration

Different tasks benefit from different models—a small local classifier for sentiment, GPT-4 for complex content generation. The dispatch that selects between them is built; the local classifier behind it is a stub, so what this demonstrates is the seam, not the saving.

What I Built

A complete production platform with ~15,000+ lines of TypeScript across backend, frontend, and mobile applications. The system handles AI-powered content generation, A/B testing, sentiment analysis, and real-time workflow coordination.

AI Service Layer (~2,000 lines)

Comprehensive LLM integration with template-based prompt engineering, structured output parsing, and fallback handling for API failures.

AI Service - LLM Integration
Production prompt engineering with template variables and structured completions
36 linesTypescriptNestJS + OpenAI
1@Injectable()
2export class AiService {
3  private readonly logger = new Logger(AiService.name);
4  private openai: OpenAIApi;
5
6  constructor(private configService: ConfigService) {
7    const apiKey = this.configService.get<string>('OPENAI_API_KEY');
8    const configuration = new Configuration({ apiKey });
9    this.openai = new OpenAIApi(configuration);
10  }
11
12  /**
13   * Generate a completion using the OpenAI API
14   */
15  async generateCompletion(prompt: string, params: Record<string, any> = {}): Promise<string> {
16    // Replace template variables in the prompt
17    let processedPrompt = prompt;
18    for (const [key, value] of Object.entries(params)) {
19      const placeholder = `{{${key}}}`;
20      if (typeof value === 'string') {
21        processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value);
22      } else if (Array.isArray(value)) {
23        processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value.join(', '));
24      }
25    }
26
27    const response = await this.openai.createCompletion({
28      model: 'gpt-4',
29      prompt: processedPrompt,
30      max_tokens: 2000,
31      temperature: 0.7,
32    });
33
34    return response.data.choices[0].text.trim();
35  }
36}

PR Service with A/B Testing (~770 lines)

Automated A/B testing for AI-generated content with real-time tracking, statistical analysis, and WebSocket-based dashboard updates.

PR Service - A/B Testing & Real-time Updates
Variant selection, tracking, and WebSocket coordination for live dashboards
49 linesTypescriptNestJS + MongoDB + WebSocket
1@Injectable()
2export class PRService {
3  constructor(
4    @InjectModel(PRPitch.name) private prPitchModel: Model<PRPitchDocument>,
5    private aiService: AiService,
6    private journalistService: JournalistService,
7    private webSocketService: WebSocketService,
8  ) {}
9
10  /**
11   * Send a PR pitch with A/B testing
12   */
13  async sendPitch(id: number): Promise<PRPitchDocument> {
14    const pitch = await this.findOne(id);
15    const journalist = await this.journalistService.findOne(pitch.journalistId);
16    
17    // Randomly select variant A or B for this recipient
18    const variant = Math.random() > 0.5 ? 'A' : 'B';
19    const subjectLine = variant === 'A' ? pitch.subjectLineA : pitch.subjectLineB;
20    const content = variant === 'A' ? pitch.contentA : pitch.contentB;
21    
22    // Update pitch status and send real-time update via WebSocket
23    pitch.status = 'sent';
24    const savedPitch = await pitch.save();
25    
26    this.webSocketService.sendPRUpdate({
27      action: 'pitch_sent',
28      pitch: { id: savedPitch._id, variant, timestamp: new Date().toISOString() }
29    });
30    
31    return savedPitch;
32  }
33
34  /**
35   * Track email open for A/B test analysis
36   */
37  async trackOpen(id: number, variant: 'A' | 'B'): Promise<PRPitchDocument> {
38    const pitch = await this.findOne(id);
39    variant === 'A' ? pitch.opensA += 1 : pitch.opensB += 1;
40    
41    // Real-time dashboard update
42    this.webSocketService.sendABTestUpdate({
43      action: 'variant_opened',
44      pitch: { id, variant, opensA: pitch.opensA, opensB: pitch.opensB }
45    });
46    
47    return pitch.save();
48  }
49}

Hybrid Sentiment Analysis (~940 lines)

Multi-model sentiment analysis intended to combine DistilBERT for speed with GPT-4 for accuracy, plus automated alerting when sentiment changes significantly. Only the GPT-4 branch reaches a model; the DistilBERT branch is simulated.

Sentiment Service - Hybrid ML Approach
Model dispatch and alerting; the DistilBERT branch is a stub
55 linesTypescriptNestJS + OpenAI
1@Injectable()
2export class SentimentService {
3  private distilBertModel: any;
4  private isModelLoaded: boolean = false;
5
6  constructor(
7    @InjectModel(SentimentAnalysis.name) private sentimentAnalysisModel: Model<SentimentAnalysisDocument>,
8    private aiService: AiService,
9    private userSettingsService: UserSettingsService,
10  ) {
11    this.initializeModels();
12  }
13
14  /**
15   * Analyze text sentiment using preferred model (DistilBERT or GPT-4)
16   */
17  async analyzeSentiment(userId: string, dto: CreateSentimentAnalysisDto): Promise<SentimentAnalysisResponseDto> {
18    const userSettings = await this.userSettingsService.getUserSettings(userId);
19    const preferredModel = dto.preferredModel || userSettings?.preferredSentimentModel || SentimentModel.DISTILBERT;
20    
21    const sentimentAnalysis = new this.sentimentAnalysisModel({
22      userId,
23      content: dto.content,
24      modelUsed: preferredModel,
25      isProcessed: false,
26    });
27    
28    const savedAnalysis = await sentimentAnalysis.save();
29    
30    // Process asynchronously - hybrid DistilBERT + GPT-4 approach
31    this.processSentimentAnalysis(savedAnalysis._id, preferredModel);
32    
33    return this.mapToResponseDto(savedAnalysis);
34  }
35
36  /**
37   * Check if sentiment change warrants an alert
38   */
39  private async checkForSentimentAlert(analysis: SentimentAnalysisDocument): Promise<void> {
40    const previousAnalyses = await this.sentimentAnalysisModel
41      .find({ userId: analysis.userId, source: analysis.source })
42      .sort({ createdAt: -1 }).limit(5);
43    
44    const avgPreviousScore = previousAnalyses.reduce((sum, a) => sum + a.score, 0) / previousAnalyses.length;
45    const changeMagnitude = Math.abs(analysis.score - avgPreviousScore);
46    
47    if (changeMagnitude >= this.alertThreshold) {
48      await this.createSentimentAlert(analysis.userId, {
49        title: 'Significant sentiment change detected',
50        severity: changeMagnitude >= 0.7 ? AlertSeverity.CRITICAL : AlertSeverity.MEDIUM,
51        recommendedActions: this.generateRecommendedActions(analysis, changeMagnitude)
52      });
53    }
54  }
55}

Async Queue System

Production job queue with priority scheduling, retry logic, and exponential backoff for handling AI workloads reliably.

Queue System - Async AI Processing
Bull queue with priority, retries, and exponential backoff
35 linesTypescriptNestJS + Bull + Redis
1// AI Queue Producer - Async job processing
2@Injectable()
3export class AiQueueProducer {
4  constructor(@InjectQueue('ai-queue') private queue: Queue) {}
5
6  async generateSocialMediaContent(params: SocialMediaContentParams, priority: number = 2) {
7    const job = await this.queue.add(
8      AiJobType.CONTENT_GENERATION,
9      { params },
10      { 
11        priority,
12        attempts: 3,
13        backoff: { type: 'exponential', delay: 5000 },
14      },
15    );
16    return job.id;
17  }
18}
19
20// AI Queue Consumer - Process jobs with retry logic
21@Processor('ai-queue')
22export class AiQueueConsumer {
23  constructor(private readonly aiService: AiService) {}
24
25  @Process(AiJobType.CONTENT_GENERATION)
26  async processContentGeneration(job: Job) {
27    this.logger.log(`Processing content generation job ${job.id}`);
28    try {
29      return await this.aiService.generateSocialMediaContent(job.data.params);
30    } catch (error) {
31      this.logger.error(`Error in job ${job.id}: ${error.message}`);
32      throw error; // Triggers retry with exponential backoff
33    }
34  }
35}

Technical Stack

Backend

  • NestJS (TypeScript) - Modular architecture
  • MongoDB - Document storage for workflows
  • PostgreSQL - Relational data
  • Redis - Caching & queue backend
  • Bull - Job queue with priorities
  • Socket.io - Real-time WebSocket

AI/ML

  • OpenAI GPT-4 - Content generation
  • Template prompts - Structured generation
  • A/B testing - Variant optimization
  • Model dispatch - Selection seam (local branch stubbed)

Frontend

  • Next.js (React) - Web application
  • React Native - iOS/Android mobile
  • Chakra UI - Component library
  • WebSocket - Real-time updates
  • TypeScript - Type safety

Infrastructure

  • Docker - Containerization
  • Terraform - Infrastructure as code
  • nginx - Reverse proxy
  • OAuth 2.0 - Google, LinkedIn, Twitter
  • JWT - Session management

Technical Challenges Solved

Production LLM Reliability

LLM APIs fail, rate limit, and produce unexpected outputs. Built comprehensive error handling with retries, fallbacks, and output validation. The queue system ensures no work is lost during failures.

Real-time Coordination

AI tasks run asynchronously but users need immediate feedback. Implemented WebSocket-based real-time updates so dashboards show live status of A/B tests, sentiment changes, and content generation.

Cost Optimization

GPT-4 API calls are expensive at scale, so the sentiment path was designed to send cheap classification to a local DistilBERT and reserve GPT-4 for generation. The dispatch was built; the local model behind it never was, so the saving is a design intent rather than an outcome. See Limitations.

Multi-Platform Deployment

Single codebase serving web, iOS, and Android with shared business logic. React Native mobile apps connect to the same backend APIs with real-time WebSocket support.

Mathematical Formulation

The formulations a harness like this one needs in order to make its outputs trustworthy. These are the standard results, written out to fix what the platform is aiming at — not a description of code that runs. The A/B tests currently pick a winner by weighted-score comparison, with none of the significance machinery below; the Limitations section says which parts are missing.

A/B Testing Statistical Analysis

Conversion Rate Estimation:

where = opens/clicks, = emails sent per variant.

Z-Test for Significance:

where is the pooled proportion. Reject null hypothesis if (95% confidence).

Sample Size for Power:

Minimum samples needed to detect effect size with 80% power.

Sentiment Analysis & Alerting

Sentiment Score (DistilBERT):

where is the pooled representation from DistilBERT, = sigmoid activation.

Change Detection (Moving Average):

Alert triggered when (default ).

Alert Severity Classification:

Job Queue Priority & Retry Logic

Exponential Backoff:

where = retry attempt, ms base delay, = jitter to prevent thundering herd.

Priority Queue Ordering:

Jobs ordered by priority (lower = higher priority), then by timestamp (FIFO within priority).

Key Technical Learnings

"Production LLM systems are 20% prompt engineering and 80% infrastructure—error handling, retries, cost management, and observability. The happy path is easy; reliability is hard."
"Leaving the dispatch seam in place while the cheap model behind it stays a stub is how a cost saving gets claimed without ever being earned. The branch is the easy part; the model is the work."
"Real-time feedback transforms user experience with AI systems. WebSocket updates showing live progress make async AI tasks feel responsive and trustworthy."

Production Insights

  • • Exponential backoff for API failures
  • • Output validation before storage
  • • Per-call token and latency tracking
  • • Structured logging for debugging

Architecture Patterns

  • • Event-driven async processing
  • • Priority queues for job scheduling
  • • WebSocket for real-time updates
  • • Shared-type monorepo

Limitations

The limitation that conditions the rest: you cannot open the repository

jstiltner/agentic-pr is private, and this page does not link to it. Every file named below — sentiment.service.ts, ai-metrics.service.ts, openai.service.ts, docs/PlanToLowerCost.md, app.json — is a path you have no way to open. The line counts and component totals higher up the page have the same problem. So do the corrections in this section: they are specific because the code is specific, but specificity is not evidence, and on this page you have only my word for any of it. Every other project on this site can be checked against something. This one cannot, and that is a real difference in what the two kinds of page are worth.

The orchestration patterns on this page are implemented. Several things an earlier version of this page implied were also implemented are not, and they are listed here because a reader has no way to find them in the source themselves.

The cheap half of the “hybrid” is a stub returning random numbers

The dispatch is real: sentiment.service.ts branches on a user model preference and sends work to either a DistilBERT path or GPT-4. The DistilBERT path does not load DistilBERT. initializeModels() sleeps for a second and sets isModelLoaded = true without loading anything, and analyzeWithDistilBERT() sleeps 500 ms and returns Math.random() as the sentiment score, 0.7 + Math.random() * 0.3 as its own confidence, and randomly chosen three-word slices of the input as “key phrases.” The TensorFlow import sits commented out above it. Routing half the traffic to that branch would cut the API bill, which is the sense in which the saving was real; it would also replace the answers with noise, which is the sense in which it was not.

The ~60% figure came from a plan, not a run

This page reported a ~60% API cost reduction from hybrid model routing. The number traces to docs/PlanToLowerCost.md, which lists “Estimated Savings: 60-90%” beneath a heading reading Implementation Plan — a proposal for work not yet done, published as though it were a result. Nothing was ever compared against a GPT-4-only baseline. The generation paths make the point plainly: openai.service.ts and ai.service.ts both hard-code 'gpt-4', and podcast-youtube-discovery.service.ts reads a single env var defaulting to 'gpt-4-turbo'. No content generation is routed anywhere but GPT-4.

Cost is not actually measured — tokens and latency are

ai-metrics.service.ts records token counts, latency and success per operation, which is the instrumentation a cost comparison would be built on. It is not that comparison. There is no per-model price table, no baseline run, and the counters live in memory and reset with the process — nothing is persisted, so no historical series exists to analyse.

The A/B tests pick winners, but not significantly

The harness is real — dedicated modules, controllers, queue processors and schemas for both PR pitches and social posts. A winner is chosen by comparing hand-weighted sums of engagement rates (openRate × 1 + clickRate × 2 + responseRate × 3) and taking the larger. There is no p-value, confidence interval or power calculation in the codebase. Describing this as “statistically rigorous” overstated it, and that wording is gone.

Three platforms are targeted, not shipped

The mobile client is an Expo app whose app.json declares iOS and Android identifiers, so all three targets are configured. No native project has been generated and no build has been produced for either store. The declared bundle identifiers contain a space (com.Agentic PR.mobile), which would fail a native build outright — good evidence that one has never been attempted.

Why the page still stands: the queue topology, the retry and fallback behaviour, the WebSocket progress channel and the A/B harness are what the project was for, and they are the parts no claim here depends on measuring. The removed claims were the ones that needed a measurement nobody took. Any number that returns here should arrive with the run that produced it — and, given the repository is private, with something a reader can actually open.

Scope

Full-stack, solo build: backend architecture, the AI service layer, the real-time coordination system, and both the web and mobile clients. The business layer (customer development, sales, marketing) ran alongside it but isn't what's documented here — this page covers the system design.