Files
postgenerator/.planning/research/ARCHITECTURE.md
Michele fe6cd4d516 docs: complete project research
Files:
- STACK.md
- FEATURES.md
- ARCHITECTURE.md
- PITFALLS.md
- SUMMARY.md

Key findings:
- Stack: FastAPI 0.135.1 + React 19 + Vite 7 + Tailwind v4, single-container deploy
- Architecture: FastAPI serves React SPA via catch-all, file-based storage (Docker volume), LLMService with retry/backoff
- Critical pitfall: All 9 pitfalls map to Phase 1 — Italian prompts, Canva field constants, UTF-8 BOM, root_path config, per-item bulk isolation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 14:06:44 +01:00

493 lines
26 KiB
Markdown

# Architecture Research
**Domain:** Content Automation System (FastAPI + React SPA, file-based storage, LLM integration)
**Researched:** 2026-03-07
**Confidence:** HIGH
## Standard Architecture
### System Overview
```
┌─────────────────────────────────────────────────────────────────────┐
│ EXTERNAL LAYER │
│ │
│ lab.mlhub.it/postgenerator/ │
│ nginx lab-router (strips /postgenerator/ prefix, forwards to :8000) │
└─────────────────────────────────────┬───────────────────────────────┘
┌──────────────────────────────────────▼──────────────────────────────┐
│ SINGLE DOCKER CONTAINER │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ FastAPI (port 8000) │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ /api/prompts│ │ /api/calendar│ │ /api/generate│ │ │
│ │ │ Router │ │ Router │ │ Router │ │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────┬────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌──────▼───────┐ ┌──────▼───────┐ ┌──────▼────────┐ │ │
│ │ │ PromptService│ │CalendarService│ │ LLMService │ │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────┬────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌──────▼──────────────────▼──────────────────▼────────┐ │ │
│ │ │ Storage Layer │ │ │
│ │ │ /data/prompts/*.txt /data/outputs/*.csv │ │ │
│ │ │ /data/config/*.json /data/swipe-files/*.json │ │ │
│ │ └───────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ [GET /*, /assets/*] → SPAStaticFiles → /app/dist/ │ │
│ │ (catch-all route serves React index.html for SPA routing) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────┘
┌────────▼──────────┐
│ Claude API │
│ (Anthropic) │
│ External HTTPS │
└────────────────────┘
```
### Component Responsibilities
| Component | Responsibility | Boundary |
|-----------|----------------|----------|
| nginx lab-router | Strip `/postgenerator/` prefix, forward to container port 8000 | External → Container |
| FastAPI main.py | Mount routers, serve React SPA via catch-all, configure root_path | App entry point |
| API Routers | Validate HTTP requests, call services, return responses | HTTP → Business logic |
| Services | Business logic, orchestration, no HTTP concerns | Business logic → Storage |
| LLMService | Claude API calls with retry, JSON validation, prompt loading | Business logic → External API |
| PromptService | Load/list/save .txt prompt files from /prompts/ directory | Business logic → Filesystem |
| CalendarService | Generate weekly/monthly post schedules from campaign config | Business logic → Domain logic |
| CSVBuilder | Transform generated content into Canva Bulk Create format | Business logic → File output |
| SwipeFileManager | Read/write curated content collections from JSON | Business logic → Filesystem |
| Storage layer | JSON config files, CSV outputs, prompt .txt files, swipe files | Filesystem |
| React SPA | UI: form inputs, results display, calendar views, file downloads | Browser → API |
## Recommended Project Structure
```
postgenerator/
├── backend/
│ ├── main.py # FastAPI app, router includes, root_path config, SPA mount
│ ├── config.py # Settings from env vars (paths, API keys)
│ ├── routers/
│ │ ├── __init__.py
│ │ ├── prompts.py # GET/POST/DELETE /api/prompts
│ │ ├── formats.py # GET /api/formats (carousel, post, story)
│ │ ├── calendar.py # POST /api/calendar/generate
│ │ ├── campaigns.py # CRUD /api/campaigns
│ │ ├── generate.py # POST /api/generate (LLM call endpoint)
│ │ ├── csv.py # POST /api/csv/build, GET /api/csv/download/{id}
│ │ └── swipe.py # CRUD /api/swipe-files
│ ├── services/
│ │ ├── __init__.py
│ │ ├── llm_service.py # Claude API calls, retry logic, JSON validation
│ │ ├── prompt_service.py # Load/list/save .txt files from /data/prompts/
│ │ ├── calendar_service.py # Generate date-indexed content schedules
│ │ ├── csv_builder.py # Build Canva Bulk Create CSV (max 300 rows, 150 cols)
│ │ └── swipe_service.py # Read/write swipe file JSON collections
│ ├── schemas/
│ │ ├── __init__.py
│ │ ├── prompt.py # Pydantic models for prompt request/response
│ │ ├── calendar.py # CalendarRequest, CalendarResponse
│ │ ├── generate.py # GenerateRequest, GeneratedPost, GenerateResponse
│ │ └── csv.py # CSVBuildRequest, CSVRow schema
│ └── data/ # Runtime data (Docker volume mount)
│ ├── prompts/ # Editable .txt prompt templates
│ │ ├── carousel_hook.txt
│ │ ├── carousel_slides.txt
│ │ └── caption_cta.txt
│ ├── outputs/ # Generated CSV files (named by timestamp)
│ ├── campaigns/ # JSON campaign configs
│ └── swipe-files/ # JSON swipe file collections
├── frontend/
│ ├── src/
│ │ ├── App.tsx
│ │ ├── pages/
│ │ │ ├── PromptManager.tsx
│ │ │ ├── CalendarView.tsx
│ │ │ ├── Generator.tsx
│ │ │ └── SwipeFile.tsx
│ │ ├── components/
│ │ └── api/ # API client (axios or fetch wrappers)
│ │ └── client.ts # Base URL, request helpers
│ ├── dist/ # Build output — FastAPI serves from here
│ └── vite.config.ts # base: '/postgenerator/' for subpath SPA routing
├── Dockerfile # Multi-stage: Node build → Python runtime
├── docker-compose.yml
└── .env # ANTHROPIC_API_KEY, DATA_PATH=/data
```
### Structure Rationale
- **routers/ vs services/:** Routers own HTTP (request parsing, status codes, response shaping). Services own business logic with no HTTP imports. This separation makes services testable independently.
- **schemas/:** Pydantic models live separately from logic. Shared between router (validation) and service (type hints).
- **data/ as Docker volume:** Prompts and outputs persist across container restarts. Mount as named volume in docker-compose. This is the "database" for a file-based system.
- **frontend/dist/ inside container:** React build output is copied into the container during Docker build. FastAPI serves it via SPAStaticFiles catch-all.
## Architectural Patterns
### Pattern 1: FastAPI Serving React SPA via Catch-All Route
**What:** FastAPI mounts React's build output with a custom StaticFiles handler that falls back to `index.html` for any path not found — enabling React Router client-side navigation.
**When to use:** Single container deployment where React and FastAPI coexist. Eliminates CORS complexity and separate web server process.
**Trade-offs:** API routes must be prefixed (e.g., `/api/`) to avoid colliding with the SPA catch-all. The SPA catch-all must be registered LAST after all API routers.
**Example:**
```python
# backend/main.py
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from starlette.responses import FileResponse
import os
app = FastAPI(root_path="/postgenerator") # Required for subpath deployment
# API routers first
app.include_router(prompts_router, prefix="/api/prompts")
app.include_router(calendar_router, prefix="/api/calendar")
app.include_router(generate_router, prefix="/api/generate")
app.include_router(csv_router, prefix="/api/csv")
app.include_router(swipe_router, prefix="/api/swipe")
# SPA catch-all MUST be last
class SPAStaticFiles(StaticFiles):
async def get_response(self, path: str, scope):
try:
return await super().get_response(path, scope)
except Exception:
# Fall back to index.html for SPA client-side routing
return await super().get_response("index.html", scope)
app.mount("/", SPAStaticFiles(directory="frontend/dist", html=True), name="spa")
```
### Pattern 2: LLM Service with Retry and Structured Output
**What:** Wrap all Claude API calls in a single service class. Use Anthropic's structured outputs (GA since Nov 2025 for Claude Sonnet/Opus) to guarantee JSON schema compliance. Fall back to prompt-based JSON enforcement with validation and retry for older approaches.
**When to use:** Any LLM integration in production. Retry + validation prevents cascading failures from transient Claude API errors or malformed responses.
**Trade-offs:** Structured outputs add latency (grammar compilation). Retry with exponential backoff adds latency on failure. Worth it for reliability. Max 3 retries before returning 503 to client.
**Example:**
```python
# backend/services/llm_service.py
import anthropic
import time
from typing import Type
from pydantic import BaseModel
class LLMService:
def __init__(self, api_key: str, max_retries: int = 3, base_delay: float = 1.0):
self.client = anthropic.Anthropic(api_key=api_key)
self.max_retries = max_retries
self.base_delay = base_delay
def generate(self, prompt: str, response_schema: Type[BaseModel]) -> BaseModel:
"""Call Claude with retry. Returns validated Pydantic model instance."""
last_error = None
for attempt in range(self.max_retries):
try:
message = self.client.messages.create(
model="claude-sonnet-4-5",
max_tokens=4096,
messages=[{"role": "user", "content": prompt}],
)
raw = message.content[0].text
return response_schema.model_validate_json(raw)
except (anthropic.APIError, anthropic.RateLimitError) as e:
last_error = e
time.sleep(self.base_delay * (2 ** attempt))
except Exception as e:
raise # Don't retry validation errors — bad prompt
raise RuntimeError(f"Claude API failed after {self.max_retries} attempts: {last_error}")
```
### Pattern 3: File-Based Storage with Explicit Path Management
**What:** All storage operations go through service classes that know the data directory layout. No scattered `open()` calls in routers or business logic. Path constants defined once in `config.py`.
**When to use:** This project. File-based storage is the right choice here — no concurrent writes, data is config-like (prompts, campaigns), outputs are ephemeral CSV downloads.
**Trade-offs:** No transactions. Concurrent writes from multiple users could corrupt JSON files (acceptable for single-user tool). If multi-user needed later, add file locking (fcntl) or migrate to SQLite.
**Example:**
```python
# backend/config.py
from pathlib import Path
import os
DATA_PATH = Path(os.getenv("DATA_PATH", "./backend/data"))
PROMPTS_PATH = DATA_PATH / "prompts"
OUTPUTS_PATH = DATA_PATH / "outputs"
CAMPAIGNS_PATH = DATA_PATH / "campaigns"
SWIPE_PATH = DATA_PATH / "swipe-files"
# backend/services/prompt_service.py
from backend.config import PROMPTS_PATH
class PromptService:
def list_prompts(self) -> list[str]:
return [f.stem for f in PROMPTS_PATH.glob("*.txt")]
def get_prompt(self, name: str) -> str:
path = PROMPTS_PATH / f"{name}.txt"
if not path.exists():
raise FileNotFoundError(f"Prompt '{name}' not found")
return path.read_text(encoding="utf-8")
def save_prompt(self, name: str, content: str) -> None:
(PROMPTS_PATH / f"{name}.txt").write_text(content, encoding="utf-8")
```
### Pattern 4: Calendar/Campaign Generation Engine
**What:** Calendar generation is pure Python — no LLM involved. Takes a campaign config (topics, frequency, date range, format mix) and produces a date-indexed schedule. Then LLM Generator fills each slot.
**When to use:** Separate the "what to generate" (calendar) from "generate it" (LLM). This way the calendar can be previewed, edited, and reused without burning API credits.
**Trade-offs:** Two-step UX (plan calendar → generate content) is more powerful but slightly more complex than one-shot generation.
**Example data flow:**
```
CampaignConfig (topics, freq, date_range, format_mix)
↓ CalendarService.generate()
CalendarSlots: [{date, topic, format, status: "pending"}, ...]
↓ User reviews calendar, confirms
↓ GenerateService.fill_slot(slot, prompt_name)
GeneratedContent: [{slot, hook, slides: [...], caption, cta}, ...]
↓ CSVBuilder.build(content_list)
CSV file ready for Canva Bulk Create download
```
## Data Flow
### Request Flow: LLM Content Generation
```
User (React)
│ POST /api/generate {prompt_name, topic, format, slot_count}
generate.py Router
│ Validate with GenerateRequest schema
│ Load prompt text via PromptService
│ Build final prompt string (template + user params)
LLMService.generate(prompt, GenerateResponse schema)
│ Call Claude API (with retry)
│ Parse and validate JSON response
GenerateResponse (list of GeneratedPost)
generate.py Router
│ Return 200 with JSON response
React SPA
│ Render results, offer CSV download
POST /api/csv/build {posts: [...]}
CSVBuilder.build()
│ Write CSV to /data/outputs/{timestamp}.csv
GET /api/csv/download/{timestamp}
│ FileResponse streaming download
User downloads CSV → uploads to Canva Bulk Create
```
### Request Flow: Calendar Generation
```
User (React)
│ POST /api/calendar/generate {campaign_config}
calendar.py Router
│ Validate CalendarRequest
CalendarService.generate(config)
│ Pure Python: iterate date range
│ Apply frequency rules (3x/week, mix of formats)
│ Assign topics from rotation
CalendarResponse: list of CalendarSlot
React renders calendar grid
│ User edits slots, confirms
│ User triggers bulk generation per slot
LLM Generator loop (one API call per slot)
```
### Request Flow: Prompt Management
```
User (React)
│ GET /api/prompts → list all prompts
│ GET /api/prompts/{name} → get content
│ PUT /api/prompts/{name} → save edited prompt
prompts.py Router
PromptService
│ Read/write /data/prompts/*.txt
File system (Docker volume)
```
### Key Data Flows Summary
1. **Prompt → LLM → CSV:** Template text (from file) + user params → Claude API → structured JSON → CSV rows → Canva Bulk Create download
2. **Campaign Config → Calendar → Bulk Generate:** Campaign settings → date-slot schedule → per-slot LLM calls → aggregated CSV
3. **Swipe File:** Curated content JSON stored as reference — read-only during generation, write-only during curation
## Subpath Deployment: Critical Configuration
Deploying at `/postgenerator/` requires coordinated config at three levels:
### 1. nginx lab-router (strips prefix)
```nginx
location /postgenerator/ {
proxy_pass http://lab-postgenerator-app:8000/; # Trailing slash strips prefix
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
}
```
### 2. FastAPI (root_path for OpenAPI docs)
```python
app = FastAPI(root_path="/postgenerator")
```
### 3. React/Vite (base path for assets)
```typescript
// vite.config.ts
export default defineConfig({
base: '/postgenerator/',
})
```
React Router must also use `<BrowserRouter basename="/postgenerator">`.
### 4. React API client (relative path)
```typescript
// frontend/src/api/client.ts
const API_BASE = '/postgenerator/api' // Absolute path including subpath
```
## Scaling Considerations
| Scale | Architecture Adjustments |
|-------|--------------------------|
| Single user (MVP) | Current architecture — file-based storage, single container, acceptable |
| 2-5 concurrent users | Add file locking on JSON writes (fcntl/portalocker). Current CSV output naming by timestamp may collide — use UUID instead |
| 10+ users or teams | Migrate JSON config to SQLite (drop-in with no infra change). Add task queue (Celery + Redis) for async LLM calls — they're slow (5-15s each) |
| Production-grade | Separate frontend container, PostgreSQL, background worker for LLM generation with WebSocket progress updates |
### Scaling Priorities
1. **First bottleneck:** LLM call latency. Each Claude API call takes 5-15 seconds. If user triggers bulk calendar generation (20 posts), that's 100-300 seconds synchronously. Fix: move to async background tasks with polling endpoint before any other optimization.
2. **Second bottleneck:** File write conflicts. Two simultaneous saves to the same JSON campaign file corrupts it. Fix: file locking or SQLite.
## Anti-Patterns
### Anti-Pattern 1: Catch-All Route Registered Before API Routes
**What people do:** Mount the SPAStaticFiles before including API routers.
**Why it's wrong:** The catch-all intercepts all requests including API calls. Every `/api/generate` request returns `index.html` instead of the actual handler.
**Do this instead:** Always register all API routers first, SPAStaticFiles last. FastAPI matches routes in registration order.
### Anti-Pattern 2: Hardcoding Paths Instead of Using Config
**What people do:** Scatter `open("/data/prompts/foo.txt")` calls throughout services and routers.
**Why it's wrong:** Impossible to test locally (path doesn't exist). Impossible to change deployment path without grep-replace. Docker volume mount path change breaks everything.
**Do this instead:** Single `config.py` with `DATA_PATH = Path(os.getenv("DATA_PATH", "./backend/data"))`. All services import path constants from config.
### Anti-Pattern 3: LLM Calls in Routers
**What people do:** Call `anthropic.messages.create()` directly inside the FastAPI route handler.
**Why it's wrong:** Untestable without mocking the entire Anthropic client. Retry logic duplicated if called from multiple routes. Error handling scattered.
**Do this instead:** All Claude API calls go through `LLMService`. Router calls `llm_service.generate(prompt, schema)`. LLMService can be mocked in tests.
### Anti-Pattern 4: React Base Path Mismatch
**What people do:** Build React with default base `/` but deploy at `/postgenerator/`. Or configure `base: '/postgenerator/'` but use absolute paths like `/api/generate` in fetch calls.
**Why it's wrong:** Asset 404s (JS/CSS files request `/assets/index.js` instead of `/postgenerator/assets/index.js`). API calls fail because they don't include the subpath.
**Do this instead:** `vite.config.ts` sets `base: '/postgenerator/'`. React Router uses `basename="/postgenerator"`. API client uses the full path `/postgenerator/api`.
### Anti-Pattern 5: Synchronous Bulk LLM Generation Without Feedback
**What people do:** Loop over 20 calendar slots, call Claude 20 times synchronously, return a 200 after 3 minutes.
**Why it's wrong:** Browser timeouts (default 60s). User has no progress feedback. Single failure aborts entire batch.
**Do this instead (MVP version):** Generate one slot at a time with streaming response or server-sent events. Client calls `/api/generate/slot` in a loop with progress bar. Each call is independent — failure is isolated.
## Integration Points
### External Services
| Service | Integration Pattern | Notes |
|---------|---------------------|-------|
| Anthropic Claude API | HTTP via `anthropic` Python SDK. Sync client in service layer. | ANTHROPIC_API_KEY from env. claude-sonnet-4-5 recommended for structured output reliability. Rate limit: 429 handled in retry loop. |
| Canva Bulk Create | No direct API. Generate CSV file, user uploads manually to Canva. | CSV constraints: max 300 rows, max 150 columns, headers must match Canva template field names. Image URLs not supported — text fields only in MVP. |
### Internal Boundaries
| Boundary | Communication | Notes |
|----------|---------------|-------|
| React SPA ↔ FastAPI | REST JSON over HTTP. No WebSockets in MVP. | All API routes under `/api/` prefix. Vite dev proxy handles CORS in development. In production: same origin, no CORS needed. |
| LLMService ↔ PromptService | Direct Python call. LLMService receives prompt string, does not load files itself. | Router loads prompt via PromptService, passes text to LLMService. Keeps LLMService pure (no file I/O). |
| CSVBuilder ↔ Storage | CSVBuilder writes to `/data/outputs/`. Returns filename. Router returns download URL. | Files accumulate — no auto-cleanup in MVP. Add manual delete or TTL later. |
| CalendarService ↔ LLMService | CalendarService generates slots (pure Python). Router then calls LLMService per slot. | CalendarService has zero LLM coupling. Can generate and display calendar without burning API credits. |
## Build Order Implications
The component dependencies suggest this implementation sequence:
1. **Foundation first:** Docker setup, FastAPI skeleton, root_path config, nginx routing, React Vite project with correct base path. Verify the plumbing works before writing any business logic.
2. **Storage layer second:** Config.py paths, data directory structure, PromptService (read/write .txt files). This is the dependency for everything else.
3. **LLMService third:** Retry logic, JSON validation. Can be tested with a single hardcoded prompt before routing exists.
4. **Feature services in dependency order:**
- PromptService (no dependencies)
- CalendarService (no LLM dependency — pure Python)
- CSVBuilder (depends on knowing the output schema)
- LLMService (depends on PromptService for templates)
- SwipeService (independent, lowest priority)
5. **API routers last:** Wire services to HTTP. Each router is thin — validates input, calls service, returns output.
6. **React UI per feature:** Build UI for each backend feature after that backend feature is working. Don't build UI speculatively.
## Sources
- FastAPI official docs — Behind a Proxy / root_path: https://fastapi.tiangolo.com/advanced/behind-a-proxy/
- FastAPI official docs — Bigger Applications: https://fastapi.tiangolo.com/tutorial/bigger-applications/
- FastAPI official docs — Static Files: https://fastapi.tiangolo.com/tutorial/static-files/
- Serving React from FastAPI (SPAStaticFiles pattern): https://davidmuraya.com/blog/serving-a-react-frontend-application-with-fastapi/
- FastAPI + React single container: https://dakdeniz.medium.com/fastapi-react-dockerize-in-single-container-e546e80b4e4d
- Anthropic Structured Outputs (Nov 2025): https://techbytes.app/posts/claude-structured-outputs-json-schema-api/
- Canva Bulk Create CSV requirements: https://www.canva.com/help/bulk-create/
- FastAPI best practices (service layer): https://orchestrator.dev/blog/2025-1-30-fastapi-production-patterns/
---
*Architecture research for: PostGenerator — Instagram carousel automation system*
*Researched: 2026-03-07*