graphs/CONTEXT.md

Context

1. Project Overview

This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a real-time sales and inventory dashboard. Its primary purpose is to ingest live webhook payloads from an external ERP (Tiny ERP) via n8n, securely store that data, and provide a visually rich, responsive dashboard for business analytics. It tracks total sales, product performance, customer behavior, and live inventory levels, while also providing tools for WhatsApp marketing campaigns.

2. Tech Stack & Tooling

Frontend:

• Library: React 19.2.5
• Language: TypeScript 6.0.2
• Build Tool: Vite 8.0.10
• Styling: Tailwind CSS 4.2.4
• Icons: Lucide React 1.14.0
• Charts: Recharts 3.8.1
• Routing: React Router DOM 7.14.2

Backend:

• Environment: Node.js
• Framework: Express 5.2.1
• Database: PostgreSQL (via pg 8.20.0)
• Authentication: JWT (jsonwebtoken 9.0.3)
• CORS & Middleware: cors, body-parser

Infrastructure & CI/CD:

• Containerization: Docker & Docker Compose
• Proxy: Nginx
• CI/CD: Gitea Actions (deploy.yml)
• Automation: n8n (External trigger source)

3. Architecture & Design Decisions

• Decoupled Client-Server: The frontend is a statically built SPA served by Nginx, communicating with an isolated Node.js API.
• Idempotent Database Operations: Due to potential retry/spam from n8n webhooks, database insertions strictly use INSERT ... ON CONFLICT DO UPDATE SET (UPSERTs). The backend dynamically generates fallback IDs using composite keys (Name_Date_Value) to prevent historical data squashing when explicit ID_Pedido fields are missing.
• Persistent Campaign Queue: Stock increases are no longer held in memory. Positive stock deltas are inserted into stock_campaign_queue, grouped by normalized base product, and processed later by a scheduled n8n workflow. This survives restarts and lets operators inspect/retry campaign work.
• Scheduled Campaign Processing: n8n calls POST /api/internal/process-stock-campaigns on fixed schedules, currently intended for 12:00 and 18:00 BRT. The endpoint claims ready product groups where accumulated delta is at least 100, combines all ready products into one campaign payload, and sends it to N8N_WHATSAPP_TRIGGER_URL.
• All-Time Top Buyers: Campaign targeting uses the top 100 buyers across all order history. There is no date filter in this query. Customers are grouped by cliente_fone and ordered by total spend (SUM(quantidade * valor_unitario)).
• Smart Polling vs. SSE: Real-time UI updates are handled via client-side polling (setInterval) rather than Server-Sent Events (SSE) to bypass persistent connection drops caused by production reverse proxies.
• Backend Analytics Foundation: Raw /api/data still exists for legacy pages, but dashboard/product/client aggregate endpoints now exist under /api/analytics/*. Dashboard already uses the backend dashboard aggregation endpoint so it does not wait on the full raw order download.
• Route Code Splitting: Frontend routes are lazy-loaded with React.lazy/Suspense so the first JS payload stays smaller. Recharts-heavy page chunks are loaded on demand.
• Analytics Client Cache: Frontend analytics requests use a small shared cache/deduping layer in src/dataService.ts. Navigating between Dashboard, RFV, Products, and Clients should reuse fresh responses instead of refetching every route mount unless query params change or the cache expires.

4. Directory Structure

/
├── .gitea/workflows/      # CI/CD pipeline definitions
├── backend/               # Node.js Express API
│   ├── Dockerfile         # Backend container definition
│   ├── index.js           # Backend entry point
│   ├── db.js              # PostgreSQL pool and startup schema/migration SQL
│   ├── routes/            # Express route modules
│   ├── services/          # Business logic and campaign processing
│   ├── mappers/           # Payload normalization helpers
│   ├── test/              # Node test runner tests
│   └── package.json       
├── public/                # Static assets (Favicons)
├── src/                   # React Frontend Application
│   ├── components/        # Reusable UI elements (Layout, DateRangePicker)
│   ├── pages/             # Route-level views (Dashboard, Products, Clients)
│   ├── dataService.ts     # Centralized API fetch logic and JWT handling
│   ├── types.ts           # Shared TypeScript interfaces
│   └── main.tsx           # React entry point
├── docker-compose.yml     # Local orchestration and environment variable mapping
├── nginx.conf             # Production web server routing
└── vite.config.ts         # Frontend build configuration

5. Core Business Rules & Domain Entities

• Order Entity (orders table): Tracks cliente_nome, data_pedido, normalized data_pedido_date, valor_pedido, produto_id, quantidade, valor_unitario, pedido_id, and cliente_fone. Tiny ERP metadata is also persisted when present: cliente_nome_fantasia, id_vendedor, nome_vendedor, marketplace, canal_venda, and numero_ecommerce.
• Order Metadata Preservation: Order upserts preserve existing non-empty phone and Tiny metadata values when incoming backfill rows send NULL or empty strings. Core mutable order/item fields still overwrite normally.
• Client Identity Rule: Analytics groups clients by a stable normalized identity so adding a phone number later does not split an existing buyer into a duplicate client. When a phone is added after historical purchases without phone, the old and new rows should resolve to the same client profile and history.
• Stock Entity (stock table): Tracks produto_id, nome, saldo (absolute current inventory), and delta_estoque. The database treats the ERP's saldo as the Absolute Truth (overwriting existing values rather than performing math) to prevent desynchronization.
• Campaign Queue Entity (stock_campaign_queue table): Tracks queued product restock deltas by base_product_name, product ID/name, delta, status (pending, processing, sent, failed, skipped), attempts, errors, and timestamps.
• WhatsApp Campaign Payload: The backend sends one n8n webhook payload per scheduled campaign run. The payload includes baseProduct, productsText, total_delta, sizes, products, and customers.
• Campaign Product Display Names: Product display text is customer-facing and may differ from internal grouping. Current aliases: BASE LISA CAMISETA ... -> Camiseta Premium ..., BASE LISA OVER SIZE ... -> Camiseta Premium Over Size ..., and BASE LISA MOLETOM CANGURU ... -> Moletom Canguru Premium ....
• Campaign Product List Format: Because Meta templates collapsed/ignored newlines inside one parameter, productsText is a single-line separator list: Produto 1 • Produto 2 • Produto 3. There is no leading bullet before the first product.
• Base Product Normalization: Apparel size suffixes like TAMANHO - P, - M, - G, - GG, and - M/G/GG are stripped for campaign grouping. ETIQUETA... products are special-cased and preserve their full variant name, e.g. ETIQUETA BRANCA TAMANHO GG.
• Campaign Observability: The frontend has a Campanhas page backed by /api/campaigns, /api/campaigns/preview, /api/campaigns/process, and /api/campaigns/retry.
• WhatsApp Marketing Integration: The system extracts phone numbers from incoming n8n payloads (checking Fone_Cliente, fone, or celular). Numbers are exposed in the UI for direct "Click-to-Chat" links and exported to CSV files.
• Filter Persistence: User preferences for Date Ranges, Sort options, and Auto-Refresh intervals are persisted to localStorage to survive page reloads.
• RFV Segmentation: The Portuguese UI calls customer segmentation RFV (Recência, Frequência, Valor). Internal code/API names may still use rfm to avoid route/type churn. The visible app labels should use RFV.
• RFV Lifecycle Thresholds: The active lifecycle is monthly, not yearly. Recency score 3 means last purchase in 0-7 days, score 2 means 8-15 days, score 1 starts at 16+ days, and clients with no purchase for 30+ days are forced into Perdido.
• RFV Date Semantics: Date filters use local calendar-day boundaries and send stable YYYY-MM-DD query params. Backend analytics treats start and end as inclusive data_pedido_date bounds.
• RFV Tag Aggregation Rule: RFV is not recalculated from only the selected period. For a selected period, the backend first calculates each buyer's RFV tag using history before the period starts. It then aggregates only the purchases inside the selected period by that prior tag. Example: if a client was Champions before today and buys today, Champions gets +1 client and that period revenue. If an Em Risco client does not buy today, Em Risco does not get counted for today's filter.
• RFV Display Rule: The RFV table's Última Compra date is the purchase date inside the selected period. Date-only strings must be parsed as local dates in the frontend to avoid UTC shifting (2026-06-15 must render as 15/06/2026, not 14/06/2026). The small recency text below it displays Hoje, Ontem, or X dias relative to the selected range end.
• Client Page Filters: The main Clients page has one filter button plus client search. The filter panel contains date shortcuts/custom dates, sorting, marketplace, sales channel, seller, and client type. Marketplace/channel/seller options come from distinct non-empty order metadata across all orders so options remain visible regardless of the current date window.
• Seller Metadata Display: Seller labels are cleaned for UI display. All-caps seller names are title-cased, trailing Tiny IDs like #977226210 are stripped from labels, and leading fantasy-name prefixes like [3] are removed for display. IDs may still be used internally as stable filter values.
• Seller Dashboard Charts: The main Dashboard includes seller performance charts above the existing charts, using backend aggregates for revenue and order count by seller.

6. CI/CD & Deployment

• Gitea Actions: A workflow located in .gitea/workflows/deploy.yml triggers on pushes to the main branch.
• Docker Registry: The pipeline builds the frontend and backend Docker images and pushes them directly to gitea.blyzer.com.br/blyzer/.
• Production Deployment: Updates are deployed manually via Portainer by pulling the latest image tags from the Gitea registry and redeploying the stack.
• Environment Variables: Security secrets (API_KEY, JWT_SECRET, POSTGRES_PASSWORD, N8N_WHATSAPP_TRIGGER_URL) are injected via the Portainer stack configuration and passed into containers via docker-compose.yml.

7. Environment Setup & Scripts

Running Locally:

1. Start the database:

   docker compose up -d db
   

2. Start the Backend (from /backend):

   npm install
   npm start
   

3. Start the Frontend (from project root):

   npm install
   npm run dev      # For HMR development
   npm run preview  # For production build testing
   

Building the Frontend:

npm run build

Backend Tests:

cd backend
npm test

Frontend Date/RFV Helper Test:

node --experimental-strip-types --test src/dateRanges.test.ts

Persistent Local Stack:

docker compose up -d --build

The local Docker services use restart: unless-stopped, so containers should come back after laptop restart if Docker starts.

8. Coding Standards & AI Directives

• Strict Type Safety: Use explicit TypeScript interfaces (defined in types.ts). Avoid any where possible. Do not bypass type checks with // @ts-ignore.
• Idiomatic React: Use functional components and hooks (useState, useEffect, useMemo). Complex data transformations (like merging arrays into chart-ready datasets) MUST be wrapped in useMemo to prevent unnecessary re-renders.
• Tailwind Architecture: All styling must be handled via Tailwind CSS utility classes. Avoid custom CSS files unless defining global font families or root variables in index.css.
• Robust Data Handling: Always implement graceful fallbacks for missing data. Never assume an API payload will contain all keys. (e.g., item.id || item.ID_Pedido || '').
• Database Migrations: There is no ORM (like Prisma or Sequelize). Table schemas, indexes, and lightweight data repairs are managed via raw SQL statements inside initDB() in backend/db.js using IF NOT EXISTS and safe startup execution patterns.
• API Security: All backend modifications exposing or altering data MUST use the verifyToken middleware for frontend requests or authenticateAPIKey for external n8n webhooks.
• Build Discipline: After frontend/backend behavior changes, run npm run lint, npm run build, and relevant backend tests. The user prefers builds after changes to catch issues before deployment.

9. Recent RFV Work

Recent commits related to RFV/date and client metadata behavior:

• 4131bf9 Update RFV lifecycle to monthly windows - RFV recency/lost thresholds changed to 0-7, 8-15, 16-29, and 30+ lost.
• ef27cff Add seller performance dashboard charts - dashboard now includes seller revenue/order charts.
• ca9615a Clean seller metadata display names - seller/fantasy labels are normalized for display.
• b129307 Fix seller filter options query - seller filter options are selected reliably from order metadata.
• b40f6fa Restore seller dropdown filter - seller filter uses a dropdown option list again.
• aeb801c Allow searching sellers in client filters - seller filter UI supports searching.
• b7a2059 Show all client metadata filter options - client metadata filter options are global, not date-scoped.
• 06faada Refresh client filter options reliably - filter option fetching was made more robust.
• 7d60073 Smooth analytics page reloads - analytics fetches now reuse cached/deduped results between pages.
• 335d231 Fix client identity when phone is added - client grouping prevents phone-added duplicate profiles.
• 73033ca Aggregate RFM period buyers by prior tag - backend RFV aggregation now groups period buyers by their prior tag.
• ae73a54 Fix RFM period date display - frontend date-only display and period recency fixed.
• 9a58b2a Improve RFM recency labels - period recency labels show Hoje/Ontem/X dias.
• f811d3c Rename visible RFM labels to RFV - visible UI wording changed from RFM to RFV.

Files most relevant to RFV:

• backend/services/analyticsService.js - getRfmAnalytics, date filters, RFV score/tag logic.
• backend/test/analyticsService.test.js - unit coverage for date filters and RFV aggregation.
• src/pages/Rfm.tsx - RFV page UI, matrix/table/export labels, period recency display.
• src/components/DateRangePicker.tsx and src/dateRanges.ts - local calendar date ranges and ISO query param formatting.
• src/dataService.ts - analytics fetch/cache helpers and client filter option requests.
• src/displayFormatters.ts - shared seller/fantasy/client metadata display cleanup.