Multi-Agent Orchestration Platform
Exploring agentic AI patterns and autonomous workflow coordination (2024)
What This Project Taught Me
In 2024, I built Agentic-PR to deeply understand multi-agent orchestration patterns and explore the business applications of agentic AI. The technical challenge was fascinating—coordinating autonomous AI agents, managing state across complex workflows, building production-grade systems for unpredictable LLM behavior.
The experience taught me something important: while I enjoyed architecting the technical systems, the business development aspects didn't energize me. I realized I want to focus on hard technical problems at scale, not running a company. This clarity led me to seek senior ML engineering roles where I can work on frontier research implementation and production systems full-time.
The takeaway: The platform demonstrated the technical feasibility of multi-agent coordination at scale and remains operational as a technical demonstration. More importantly, it clarified my career direction: I'm an engineer who loves building sophisticated systems, not an entrepreneur who wants to run a business.
System Architecture
The Technical Challenge
Traditional automation requires explicit programming of every step—if-then logic that breaks when conditions change. Agentic AI enables autonomous agents to make decisions and execute tasks, but introduces new challenges around coordination and reliability.
The core question: How do you orchestrate multiple autonomous AI agents reliably in production?
LLM Integration at Scale
Production LLM systems need more than API calls—they need prompt engineering, response validation, cost optimization, and graceful handling of model failures and rate limits.
Async Workflow Coordination
AI tasks are inherently async and unpredictable. The system needs message queues, job prioritization, retry logic with exponential backoff, and real-time status updates.
Multi-Model Orchestration
Different tasks benefit from different models—DistilBERT for fast sentiment analysis, GPT-4 for complex content generation. The system routes tasks to optimal models based on requirements.
What I Built
A complete production platform with ~15,000+ lines of TypeScript across backend, frontend, and mobile applications. The system handles AI-powered content generation, A/B testing, sentiment analysis, and real-time workflow coordination.
AI Service Layer (~2,000 lines)
Comprehensive LLM integration with template-based prompt engineering, structured output parsing, and fallback handling for API failures.
1@Injectable()
2export class AiService {
3 private readonly logger = new Logger(AiService.name);
4 private openai: OpenAIApi;
5
6 constructor(private configService: ConfigService) {
7 const apiKey = this.configService.get<string>('OPENAI_API_KEY');
8 const configuration = new Configuration({ apiKey });
9 this.openai = new OpenAIApi(configuration);
10 }
11
12 /**
13 * Generate a completion using the OpenAI API
14 */
15 async generateCompletion(prompt: string, params: Record<string, any> = {}): Promise<string> {
16 // Replace template variables in the prompt
17 let processedPrompt = prompt;
18 for (const [key, value] of Object.entries(params)) {
19 const placeholder = `{{${key}}}`;
20 if (typeof value === 'string') {
21 processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value);
22 } else if (Array.isArray(value)) {
23 processedPrompt = processedPrompt.replace(new RegExp(placeholder, 'g'), value.join(', '));
24 }
25 }
26
27 const response = await this.openai.createCompletion({
28 model: 'gpt-4',
29 prompt: processedPrompt,
30 max_tokens: 2000,
31 temperature: 0.7,
32 });
33
34 return response.data.choices[0].text.trim();
35 }
36}PR Service with A/B Testing (~770 lines)
Automated A/B testing for AI-generated content with real-time tracking, statistical analysis, and WebSocket-based dashboard updates.
1@Injectable()
2export class PRService {
3 constructor(
4 @InjectModel(PRPitch.name) private prPitchModel: Model<PRPitchDocument>,
5 private aiService: AiService,
6 private journalistService: JournalistService,
7 private webSocketService: WebSocketService,
8 ) {}
9
10 /**
11 * Send a PR pitch with A/B testing
12 */
13 async sendPitch(id: number): Promise<PRPitchDocument> {
14 const pitch = await this.findOne(id);
15 const journalist = await this.journalistService.findOne(pitch.journalistId);
16
17 // Randomly select variant A or B for this recipient
18 const variant = Math.random() > 0.5 ? 'A' : 'B';
19 const subjectLine = variant === 'A' ? pitch.subjectLineA : pitch.subjectLineB;
20 const content = variant === 'A' ? pitch.contentA : pitch.contentB;
21
22 // Update pitch status and send real-time update via WebSocket
23 pitch.status = 'sent';
24 const savedPitch = await pitch.save();
25
26 this.webSocketService.sendPRUpdate({
27 action: 'pitch_sent',
28 pitch: { id: savedPitch._id, variant, timestamp: new Date().toISOString() }
29 });
30
31 return savedPitch;
32 }
33
34 /**
35 * Track email open for A/B test analysis
36 */
37 async trackOpen(id: number, variant: 'A' | 'B'): Promise<PRPitchDocument> {
38 const pitch = await this.findOne(id);
39 variant === 'A' ? pitch.opensA += 1 : pitch.opensB += 1;
40
41 // Real-time dashboard update
42 this.webSocketService.sendABTestUpdate({
43 action: 'variant_opened',
44 pitch: { id, variant, opensA: pitch.opensA, opensB: pitch.opensB }
45 });
46
47 return pitch.save();
48 }
49}Hybrid Sentiment Analysis (~940 lines)
Multi-model sentiment analysis combining DistilBERT for speed with GPT-4 for accuracy, plus automated alerting when sentiment changes significantly.
1@Injectable()
2export class SentimentService {
3 private distilBertModel: any;
4 private isModelLoaded: boolean = false;
5
6 constructor(
7 @InjectModel(SentimentAnalysis.name) private sentimentAnalysisModel: Model<SentimentAnalysisDocument>,
8 private aiService: AiService,
9 private userSettingsService: UserSettingsService,
10 ) {
11 this.initializeModels();
12 }
13
14 /**
15 * Analyze text sentiment using preferred model (DistilBERT or GPT-4)
16 */
17 async analyzeSentiment(userId: string, dto: CreateSentimentAnalysisDto): Promise<SentimentAnalysisResponseDto> {
18 const userSettings = await this.userSettingsService.getUserSettings(userId);
19 const preferredModel = dto.preferredModel || userSettings?.preferredSentimentModel || SentimentModel.DISTILBERT;
20
21 const sentimentAnalysis = new this.sentimentAnalysisModel({
22 userId,
23 content: dto.content,
24 modelUsed: preferredModel,
25 isProcessed: false,
26 });
27
28 const savedAnalysis = await sentimentAnalysis.save();
29
30 // Process asynchronously - hybrid DistilBERT + GPT-4 approach
31 this.processSentimentAnalysis(savedAnalysis._id, preferredModel);
32
33 return this.mapToResponseDto(savedAnalysis);
34 }
35
36 /**
37 * Check if sentiment change warrants an alert
38 */
39 private async checkForSentimentAlert(analysis: SentimentAnalysisDocument): Promise<void> {
40 const previousAnalyses = await this.sentimentAnalysisModel
41 .find({ userId: analysis.userId, source: analysis.source })
42 .sort({ createdAt: -1 }).limit(5);
43
44 const avgPreviousScore = previousAnalyses.reduce((sum, a) => sum + a.score, 0) / previousAnalyses.length;
45 const changeMagnitude = Math.abs(analysis.score - avgPreviousScore);
46
47 if (changeMagnitude >= this.alertThreshold) {
48 await this.createSentimentAlert(analysis.userId, {
49 title: 'Significant sentiment change detected',
50 severity: changeMagnitude >= 0.7 ? AlertSeverity.CRITICAL : AlertSeverity.MEDIUM,
51 recommendedActions: this.generateRecommendedActions(analysis, changeMagnitude)
52 });
53 }
54 }
55}Async Queue System
Production job queue with priority scheduling, retry logic, and exponential backoff for handling AI workloads reliably.
1// AI Queue Producer - Async job processing
2@Injectable()
3export class AiQueueProducer {
4 constructor(@InjectQueue('ai-queue') private queue: Queue) {}
5
6 async generateSocialMediaContent(params: SocialMediaContentParams, priority: number = 2) {
7 const job = await this.queue.add(
8 AiJobType.CONTENT_GENERATION,
9 { params },
10 {
11 priority,
12 attempts: 3,
13 backoff: { type: 'exponential', delay: 5000 },
14 },
15 );
16 return job.id;
17 }
18}
19
20// AI Queue Consumer - Process jobs with retry logic
21@Processor('ai-queue')
22export class AiQueueConsumer {
23 constructor(private readonly aiService: AiService) {}
24
25 @Process(AiJobType.CONTENT_GENERATION)
26 async processContentGeneration(job: Job) {
27 this.logger.log(`Processing content generation job ${job.id}`);
28 try {
29 return await this.aiService.generateSocialMediaContent(job.data.params);
30 } catch (error) {
31 this.logger.error(`Error in job ${job.id}: ${error.message}`);
32 throw error; // Triggers retry with exponential backoff
33 }
34 }
35}Technical Stack
Backend
- • NestJS (TypeScript) - Modular architecture
- • MongoDB - Document storage for workflows
- • PostgreSQL - Relational data
- • Redis - Caching & queue backend
- • Bull - Job queue with priorities
- • Socket.io - Real-time WebSocket
AI/ML
- • OpenAI GPT-4 - Content generation
- • DistilBERT - Fast sentiment analysis
- • Template prompts - Structured generation
- • A/B testing - Variant optimization
- • Hybrid routing - Model selection
Frontend
- • Next.js (React) - Web application
- • React Native - iOS/Android mobile
- • Chakra UI - Component library
- • WebSocket - Real-time updates
- • TypeScript - Type safety
Infrastructure
- • Docker - Containerization
- • Terraform - Infrastructure as code
- • nginx - Reverse proxy
- • OAuth 2.0 - Google, LinkedIn, Twitter
- • JWT - Session management
Technical Challenges Solved
Production LLM Reliability
LLM APIs fail, rate limit, and produce unexpected outputs. Built comprehensive error handling with retries, fallbacks, and output validation. The queue system ensures no work is lost during failures.
Real-time Coordination
AI tasks run asynchronously but users need immediate feedback. Implemented WebSocket-based real-time updates so dashboards show live status of A/B tests, sentiment changes, and content generation.
Cost Optimization
GPT-4 API calls are expensive at scale. Implemented intelligent model routing—using DistilBERT for fast sentiment analysis, reserving GPT-4 for complex content generation. Added caching and response truncation where appropriate.
Multi-Platform Deployment
Single codebase serving web, iOS, and Android with shared business logic. React Native mobile apps connect to the same backend APIs with real-time WebSocket support.
Mathematical Formulation
Key algorithms underlying the agentic orchestration platform.
A/B Testing Statistical Analysis
Conversion Rate Estimation:
where = opens/clicks, = emails sent per variant.
Z-Test for Significance:
where is the pooled proportion. Reject null hypothesis if (95% confidence).
Sample Size for Power:
Minimum samples needed to detect effect size with 80% power.
Sentiment Analysis & Alerting
Sentiment Score (DistilBERT):
where is the pooled representation from DistilBERT, = sigmoid activation.
Change Detection (Moving Average):
Alert triggered when (default ).
Alert Severity Classification:
Job Queue Priority & Retry Logic
Exponential Backoff:
where = retry attempt, ms base delay, = jitter to prevent thundering herd.
Priority Queue Ordering:
Jobs ordered by priority (lower = higher priority), then by timestamp (FIFO within priority).
Key Technical Learnings
"Production LLM systems are 20% prompt engineering and 80% infrastructure—error handling, retries, cost management, and observability. The happy path is easy; reliability is hard."
"Hybrid model architectures work well in practice. Fast models (DistilBERT) for high-volume tasks, powerful models (GPT-4) for complex generation. Route intelligently based on task requirements."
"Real-time feedback transforms user experience with AI systems. WebSocket updates showing live progress make async AI tasks feel responsive and trustworthy."
Production Insights
- • Exponential backoff for API failures
- • Output validation before storage
- • Cost tracking per API call
- • Structured logging for debugging
Architecture Patterns
- • Event-driven async processing
- • Priority queues for job scheduling
- • WebSocket for real-time updates
- • Hybrid model routing
What This Experience Taught Me About My Career
Building this platform gave me hands-on experience with production agentic AI—LLM integration at scale, async workflow coordination, and the real challenges of deploying AI systems reliably.
What I Loved
- • Architecting the multi-service system
- • Solving LLM reliability challenges
- • Building real-time coordination
- • Optimizing cost/performance trade-offs
What Didn't Energize Me
- • Customer development interviews
- • Sales and business development
- • Marketing and brand building
- • Fundraising conversations
Why I'm Seeking Senior ML Engineering Roles
This project taught me I want to focus on hard engineering problems at scale—implementing frontier research, building production ML systems, solving the technical challenges that make AI actually work in the real world. I want to wake up thinking about model architectures and system reliability, not customer acquisition strategies. That clarity is why I'm focused on senior ML engineering roles at companies doing cutting-edge AI work.