graphs/CONTEXT.md

Context

1. Project Overview

This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a real-time sales and inventory dashboard. Its primary purpose is to ingest live webhook payloads from an external ERP (Tiny ERP) via n8n, securely store that data, and provide a visually rich, responsive dashboard for business analytics. It tracks total sales, product performance, customer behavior, and live inventory levels, while also providing tools for WhatsApp marketing campaigns.

2. Tech Stack & Tooling

Frontend:

• Library: React 19.2.5
• Language: TypeScript 6.0.2
• Build Tool: Vite 8.0.10
• Styling: Tailwind CSS 4.2.4
• Icons: Lucide React 1.14.0
• Charts: Recharts 3.8.1
• Routing: React Router DOM 7.14.2

Backend:

• Environment: Node.js
• Framework: Express 5.2.1
• Database: PostgreSQL (via pg 8.20.0)
• Authentication: JWT (jsonwebtoken 9.0.3)
• CORS & Middleware: cors, body-parser

Infrastructure & CI/CD:

• Containerization: Docker & Docker Compose
• Proxy: Nginx
• CI/CD: Gitea Actions (deploy.yml)
• Automation: n8n (External trigger source)

3. Architecture & Design Decisions

• Decoupled Client-Server: The frontend is a statically built SPA served by Nginx, communicating with an isolated Node.js API.
• Idempotent Database Operations: Due to potential retry/spam from n8n webhooks, database insertions strictly use INSERT ... ON CONFLICT DO UPDATE SET (UPSERTs). The backend dynamically generates fallback IDs using composite keys (Name_Date_Value) to prevent historical data squashing when explicit ID_Pedido fields are missing.
• Persistent Campaign Queue: Stock increases are no longer held in memory. Positive stock deltas are inserted into stock_campaign_queue, grouped by normalized base product, and processed later by a scheduled n8n workflow. This survives restarts and lets operators inspect/retry campaign work.
• Scheduled Campaign Processing: n8n calls POST /api/internal/process-stock-campaigns on fixed schedules, currently intended for 12:00 and 18:00 BRT. The endpoint claims ready product groups where accumulated delta is at least 100, combines all ready products into one campaign payload, and sends it to N8N_WHATSAPP_TRIGGER_URL.
• All-Time Top Buyers: Campaign targeting uses the top 100 buyers across all order history. There is no date filter in this query. Customers are grouped by cliente_fone and ordered by total spend (SUM(quantidade * valor_unitario)).
• Smart Polling vs. SSE: Real-time UI updates are handled via client-side polling (setInterval) rather than Server-Sent Events (SSE) to bypass persistent connection drops caused by production reverse proxies.
• Backend Analytics Foundation: Raw /api/data still exists for legacy pages, but dashboard/product/client aggregate endpoints now exist under /api/analytics/*. Dashboard already uses the backend dashboard aggregation endpoint so it does not wait on the full raw order download.
• Route Code Splitting: Frontend routes are lazy-loaded with React.lazy/Suspense so the first JS payload stays smaller. Recharts-heavy page chunks are loaded on demand.
• Analytics Client Cache: Frontend analytics requests use a small shared cache/deduping layer in src/dataService.ts. Navigating between Dashboard, RFV, Products, and Clients should reuse fresh responses instead of refetching every route mount unless query params change or the cache expires.
• Tiny as Current Source, Graphs as Future Operating Layer: Tiny ERP is the current source for sales, product/order metadata, and stock. Graphs should increasingly become the place where the business manages production intelligence: cut configuration, SKU families, raw material relationships, consumption references, and operational charts. NECESSIDADE DE CORTE.xlsx is reference material only and must not become a runtime dependency.
• Tiny/Olist Composition Import: Product structures from Tiny/Olist V3 are imported into product_compositions and product_composition_components. Sellable finished-product structures are also promoted into local catalog products/materials and consumption_references with source tiny_structure, so purchase planning can use real component consumption instead of guessed kg-per-piece rules.

4. Directory Structure

/
├── .gitea/workflows/      # CI/CD pipeline definitions
├── backend/               # Node.js Express API
│   ├── Dockerfile         # Backend container definition
│   ├── index.js           # Backend entry point
│   ├── db.js              # PostgreSQL pool and startup schema/migration SQL
│   ├── routes/            # Express route modules
│   ├── services/          # Business logic and campaign processing
│   ├── mappers/           # Payload normalization helpers
│   ├── test/              # Node test runner tests
│   └── package.json       
├── public/                # Static assets (Favicons)
├── src/                   # React Frontend Application
│   ├── components/        # Reusable UI elements (Layout, DateRangePicker)
│   ├── pages/             # Route-level views (Dashboard, Products, Clients)
│   ├── dataService.ts     # Centralized API fetch logic and JWT handling
│   ├── types.ts           # Shared TypeScript interfaces
│   └── main.tsx           # React entry point
├── docker-compose.yml     # Local orchestration and environment variable mapping
├── nginx.conf             # Production web server routing
└── vite.config.ts         # Frontend build configuration

5. Core Business Rules & Domain Entities

• Order Entity (orders table): Tracks cliente_nome, data_pedido, normalized data_pedido_date, valor_pedido, produto_id, quantidade, valor_unitario, pedido_id, and cliente_fone. Tiny ERP metadata is also persisted when present: cliente_nome_fantasia, id_vendedor, nome_vendedor, marketplace, canal_venda, and numero_ecommerce.
• Order Metadata Preservation: Order upserts preserve existing non-empty phone and Tiny metadata values when incoming backfill rows send NULL or empty strings. Core mutable order/item fields still overwrite normally.
• Client Identity Rule: Analytics groups clients by a stable normalized identity so adding a phone number later does not split an existing buyer into a duplicate client. When a phone is added after historical purchases without phone, the old and new rows should resolve to the same client profile and history.
• Stock Entity (stock table): Tracks produto_id, nome, saldo (absolute current inventory), and delta_estoque. The database treats the ERP's saldo as the Absolute Truth (overwriting existing values rather than performing math) to prevent desynchronization.
• Campaign Queue Entity (stock_campaign_queue table): Tracks queued product restock deltas by base_product_name, product ID/name, delta, status (pending, processing, sent, failed, skipped), attempts, errors, and timestamps.
• WhatsApp Campaign Payload: The backend sends one n8n webhook payload per scheduled campaign run. The payload includes baseProduct, productsText, total_delta, sizes, products, and customers.
• Campaign Product Display Names: Product display text is customer-facing and may differ from internal grouping. Current aliases: BASE LISA CAMISETA ... -> Camiseta Premium ..., BASE LISA OVER SIZE ... -> Camiseta Premium Over Size ..., and BASE LISA MOLETOM CANGURU ... -> Moletom Canguru Premium ....
• Campaign Product List Format: Because Meta templates collapsed/ignored newlines inside one parameter, productsText is a single-line separator list: Produto 1 • Produto 2 • Produto 3. There is no leading bullet before the first product.
• Base Product Normalization: Apparel size suffixes like TAMANHO - P, - M, - G, - GG, and - M/G/GG are stripped for campaign grouping. ETIQUETA... products are special-cased and preserve their full variant name, e.g. ETIQUETA BRANCA TAMANHO GG.
• Campaign Observability: The frontend has a Campanhas page backed by /api/campaigns, /api/campaigns/preview, /api/campaigns/process, and /api/campaigns/retry.
• WhatsApp Marketing Integration: The system extracts phone numbers from incoming n8n payloads (checking Fone_Cliente, fone, or celular). Numbers are exposed in the UI for direct "Click-to-Chat" links and exported to CSV files.
• Filter Persistence: User preferences for Date Ranges, Sort options, and Auto-Refresh intervals are persisted to localStorage to survive page reloads.
• RFV Segmentation: The Portuguese UI calls customer segmentation RFV (Recência, Frequência, Valor). Internal code/API names may still use rfm to avoid route/type churn. The visible app labels should use RFV.
• RFV Lifecycle Thresholds: The active lifecycle is monthly, not yearly. Recency score 3 means last purchase in 0-7 days, score 2 means 8-15 days, score 1 starts at 16+ days, and clients with no purchase for 30+ days are forced into Perdido.
• RFV Date Semantics: Date filters use local calendar-day boundaries and send stable YYYY-MM-DD query params. Backend analytics treats start and end as inclusive data_pedido_date bounds.
• RFV Tag Aggregation Rule: RFV is not recalculated from only the selected period. For a selected period, the backend first calculates each buyer's RFV tag using history before the period starts. It then aggregates only the purchases inside the selected period by that prior tag. Example: if a client was Champions before today and buys today, Champions gets +1 client and that period revenue. If an Em Risco client does not buy today, Em Risco does not get counted for today's filter.
• RFV Display Rule: The RFV table's Última Compra date is the purchase date inside the selected period. Date-only strings must be parsed as local dates in the frontend to avoid UTC shifting (2026-06-15 must render as 15/06/2026, not 14/06/2026). The small recency text below it displays Hoje, Ontem, or X dias relative to the selected range end.
• Client Page Filters: The main Clients page has one filter button plus client search. The filter panel contains date shortcuts/custom dates, sorting, marketplace, sales channel, seller, and client type. Marketplace/channel/seller options come from distinct non-empty order metadata across all orders so options remain visible regardless of the current date window.
• Seller Metadata Display: Seller labels are cleaned for UI display. All-caps seller names are title-cased, trailing Tiny IDs like #977226210 are stripped from labels, and leading fantasy-name prefixes like [3] are removed for display. IDs may still be used internally as stable filter values.
• Seller Dashboard Charts: The main Dashboard includes seller performance charts above the existing charts, using backend aggregates for revenue and order count by seller. Long date ranges are bucketed into week/month views for readability. Seller labels like 0/empty values are displayed as Sem vendedor. The seller moving-average line is optional behind the Tendência toggle because it can be confused with another seller series.
• Product Chart Readability: Product detail charts use automatic date bucketing, cleaner x-axis labels, clickable point drill-downs, and a moving average on the single-product view. Product detail KPIs include daily average sold and estimated stock coverage based on the selected date range.

6. Supply, Cutting, and Data Flow Requirements

Tiny provides sales and stock facts, but it does not fully describe how a finished SKU consumes raw material. For Suprimentos and Corte to work reliably, Graphs needs internal domain data that complements Tiny.

Current operating assumption:

• Tiny sends orders, product IDs/names, seller/order metadata, and current stock.
• Graphs derives demand, sales velocity, replenishment pressure, and cut/material needs from Tiny data plus local configuration.
• Excel files are examples of current manual management only. They should guide UX/data modeling but not be imported as the long-term source of truth.

Core missing relationship:

Finished SKU -> raw material -> consumption/yield

Example:

BASE LISA CAMISETA PRETA G
uses: MALHA 100% ALGODAO PRETA
yield: 4.8 units/kg

Data Graphs must own or enrich locally:

• Finished SKU catalog: SKU/id, name, color, size, product family, cut family, active/inactive.
• Raw material catalog: material SKU/id, name, type (malha, ribana, fio, etc.), color, supplier, unit (kg, metro, unidade), and whether it is raw input instead of finished stock.
• Consumption references: finished SKU, material product, yield per kg, optional yield by size, optional color/family overrides, and secondary materials when needed.
• Imported product compositions: Tiny/Olist structures are stored as immutable-ish sync facts: finished SKU/Tiny ID/unit, component SKU/Tiny ID/name, quantity per finished unit, and component unit (KG, UN, etc.). These records support Product Details composition display and drive tiny_structure consumption references.
• Cut family mapping: rules that say which colors/sizes/products can be planned together for cutting.
• Open production/cutting data: quantities already in production, expected finish date, linked finished SKU, and linked raw material when available.
• Business rules: target coverage days, minimum stock, safety stock, purchase lead time, production lead time, and discontinued/ignored SKU rules.

Current UI/data-flow behavior:

• Cadastros has an Importar composições JSON upload action for exports like composicoes_produtos_YYYY-MM-DD.json. The same import is available to scripts through POST /api/production-orders/tiny-compositions/import using x-api-key.
• Composition imports are idempotent. Re-importing the same Tiny/Olist export updates composition rows/components and refreshes derived tiny_structure consumption references instead of duplicating them.
• Product Details > Composição shows the synced component list for the current product when a composition exists.
• Suprimentos > Necessidade de Compra derives material pressure from demand and stock, but rows without a consumption reference are marked as missing reference instead of pretending to calculate kg.
• Imported tiny_structure consumption references now allow Necessidade de Compra to calculate mixed-unit material demand from real BOM lines, e.g. malha/ribana in KG and etiqueta/ilhós/atacador in UN.
• Missing-reference rows link into Cadastros > Referência de Consumo with SKU context. If the SKU does not exist in the local catalog yet, Cadastros opens the product form first.
• Cadastros > Referência de Consumo labels reference sources as Manual, Tiny OP, or Tiny Estrutura.
• Plano de Corte SKU edit actions deep-link into the cutting configuration drawer for that SKU.
• Product tables and group detail tables are tuned for high-volume values, including large Total Vendido quantities above 100k.

Current composition import status:

• Local import tested with /home/farelos/Downloads/composicoes_produtos_2026-07-31.json.
• Imported successfully: 437 compositions, 1,552 component rows, and 1,316 tiny_structure consumption references.
• Known data issues from that export: 4 sellable finished products are missing SKU; 54 structures are raw material/service structures and are stored as compositions but intentionally not promoted into sellable demand-planning references; 236 component/reference lines were skipped because their parent structure was non-sellable or the finished SKU was missing.
• Sellable-family detection currently includes DTF, CAMISETA, CAMISA, MOLETOM, CANGURU, REGATA, and POLO. Raw/input/service structures are excluded from demand-reference promotion when descriptions contain terms like MALHA, RIBANA, FIO, TECIDO, SERVIÇO, TINTURARIA, TECELAGEM, or FRETE.

Recommended next steps now that composition data exists:

• Make Necessidade de Compra clearer and more auditable: show demanda do SKU -> consumo por unidade -> material necessário -> estoque atual -> comprar.
• Add a data-health/review screen for missing finished SKU, missing component stock link, suspicious quantities, products with sales but no composition, and non-sellable structures.
• Connect material stock more explicitly to imported component SKUs/Tiny IDs so material purchase needs subtract the right current stock and pending receipts.
• Upgrade Planejamento de Corte to show material blockers by family/color/size, using imported composition references and current stock.
• Add production-plan status flow only after material needs and composition health are reliable.

7. CI/CD & Deployment

• Gitea Actions: A workflow located in .gitea/workflows/deploy.yml triggers on pushes to the main branch.
• Docker Registry: The pipeline builds the frontend and backend Docker images and pushes them directly to gitea.blyzer.com.br/blyzer/.
• Production Deployment: Updates are deployed manually via Portainer by pulling the latest image tags from the Gitea registry and redeploying the stack.
• Environment Variables: Security secrets (API_KEY, JWT_SECRET, POSTGRES_PASSWORD, N8N_WHATSAPP_TRIGGER_URL) are injected via the Portainer stack configuration and passed into containers via docker-compose.yml.

8. Environment Setup & Scripts

Running Locally:

1. Start the database:

   docker compose up -d db
   

2. Start the Backend (from /backend):

   npm install
   npm start
   

3. Start the Frontend (from project root):

   npm install
   npm run dev      # For HMR development
   npm run preview  # For production build testing
   

Building the Frontend:

npm run build

Backend Tests:

cd backend
npm test

Frontend Date/RFV Helper Test:

node --experimental-strip-types --test src/dateRanges.test.ts

Persistent Local Stack:

docker compose up -d --build

The local Docker services use restart: unless-stopped, so containers should come back after laptop restart if Docker starts.

Importing Tiny/Olist Product Compositions:

• From the app: open Cadastros and use Importar composições with the exported JSON file.
• From a script/API client:

  curl -X POST http://localhost:3004/api/production-orders/tiny-compositions/import \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: nexstar_secret_key_123' \
    --data-binary @/path/to/composicoes_produtos_YYYY-MM-DD.json
  

• Expected response includes imported, failed, componentCount, referenceCount, skippedReferenceCount, and issues.

9. Coding Standards & AI Directives

• Strict Type Safety: Use explicit TypeScript interfaces (defined in types.ts). Avoid any where possible. Do not bypass type checks with // @ts-ignore.
• Idiomatic React: Use functional components and hooks (useState, useEffect, useMemo). Complex data transformations (like merging arrays into chart-ready datasets) MUST be wrapped in useMemo to prevent unnecessary re-renders.
• Tailwind Architecture: All styling must be handled via Tailwind CSS utility classes. Avoid custom CSS files unless defining global font families or root variables in index.css.
• Robust Data Handling: Always implement graceful fallbacks for missing data. Never assume an API payload will contain all keys. (e.g., item.id || item.ID_Pedido || '').
• Database Migrations: There is no ORM (like Prisma or Sequelize). Table schemas, indexes, and lightweight data repairs are managed via raw SQL statements inside initDB() in backend/db.js using IF NOT EXISTS and safe startup execution patterns.
• API Security: All backend modifications exposing or altering data MUST use the verifyToken middleware for frontend requests or authenticateAPIKey for external n8n webhooks.
• Build Discipline: After frontend/backend behavior changes, run npm run lint, npm run build, and relevant backend tests. The user prefers builds after changes to catch issues before deployment.

10. Recent Work

Recent commits related to RFV/date and client metadata behavior:

• 4131bf9 Update RFV lifecycle to monthly windows - RFV recency/lost thresholds changed to 0-7, 8-15, 16-29, and 30+ lost.
• ef27cff Add seller performance dashboard charts - dashboard now includes seller revenue/order charts.
• ca9615a Clean seller metadata display names - seller/fantasy labels are normalized for display.
• b129307 Fix seller filter options query - seller filter options are selected reliably from order metadata.
• b40f6fa Restore seller dropdown filter - seller filter uses a dropdown option list again.
• aeb801c Allow searching sellers in client filters - seller filter UI supports searching.
• b7a2059 Show all client metadata filter options - client metadata filter options are global, not date-scoped.
• 06faada Refresh client filter options reliably - filter option fetching was made more robust.
• 7d60073 Smooth analytics page reloads - analytics fetches now reuse cached/deduped results between pages.
• 335d231 Fix client identity when phone is added - client grouping prevents phone-added duplicate profiles.
• 73033ca Aggregate RFM period buyers by prior tag - backend RFV aggregation now groups period buyers by their prior tag.
• ae73a54 Fix RFM period date display - frontend date-only display and period recency fixed.
• 9a58b2a Improve RFM recency labels - period recency labels show Hoje/Ontem/X dias.
• f811d3c Rename visible RFM labels to RFV - visible UI wording changed from RFM to RFV.

Files most relevant to RFV:

• backend/services/analyticsService.js - getRfmAnalytics, date filters, RFV score/tag logic.
• backend/test/analyticsService.test.js - unit coverage for date filters and RFV aggregation.
• src/pages/Rfm.tsx - RFV page UI, matrix/table/export labels, period recency display.
• src/components/DateRangePicker.tsx and src/dateRanges.ts - local calendar date ranges and ISO query param formatting.
• src/dataService.ts - analytics fetch/cache helpers and client filter option requests.
• src/displayFormatters.ts - shared seller/fantasy/client metadata display cleanup.

Recent commits related to supply/cutting/data-flow and chart readability:

• 37d4a77 Import Tiny product compositions - adds the Tiny/Olist composition JSON import flow, derived tiny_structure consumption references, app upload button, API-key import endpoint, and import tests.
• 800eb97 Derive consumption references from Tiny OP sync - Tiny OP composition sync now creates catalog products/materials and reusable consumption references from OP component lines.
• c07938c Add Tiny production order detail sync - stores OP detail rows, components, steps, and observations from Tiny/Olist sync payloads.
• 060b4da Connect supplies to project demand - Suprimentos purchase needs now derive from project/order demand and stock, while missing consumption references are surfaced explicitly.
• bc05fb4 Add SKU edit actions - product/cutting/replenishment/group tables gained compact SKU view/edit actions.
• 63efb47 Route SKU actions to focused editors - cut-plan edit opens SKU cutting configuration; missing-reference supply rows open the consumption reference flow.
• 9489318 Improve chart readability and drilldowns - dashboard/product charts gained date bucketing, cleaner unknown labels, click drill-downs, and product stock coverage context.
• ba9cc3a Fix large sold quantity icon layout - product/group sold quantity cells handle large values without icon compression.
• f923489 Make dashboard trend line optional - dashboard moving-average trend line is hidden by default and available through Tendência.

Files most relevant to supply/cutting/data-flow:

• src/pages/Supplies.tsx - supply dashboard, receipts, inventory, movements, and purchase-need UI.
• src/pages/Cutting.tsx - cut plan, cut settings drawer, SKU correction/deep-link flow.
• src/pages/Registrations.tsx - local catalog, raw materials, finished products, and consumption references.
• src/catalogLinks.ts - shared deep-link builders for SKU edit, cutting config, and consumption references.
• src/analytics/cutting.ts - cut-family parsing, cut needs, stock coverage, and product override logic.
• src/pages/Products.tsx and src/pages/ProductGroupDetails.tsx - product/group velocity, stock coverage, and high-volume display.
• src/chartUtils.ts - shared date bucketing, moving average, unknown-label cleanup, and chart date helpers.
• backend/services/productionOrderService.js - Tiny OP detail sync, Tiny/Olist structure import, composition storage, sellable structure filtering, and derived consumption-reference creation.
• backend/routes/productionOrderRoutes.js and backend/routes/catalogRoutes.js - API-key script import and authenticated app import endpoints for composition JSON.
• backend/test/productCompositionService.test.js - coverage for Tiny/Olist structure sync and bulk composition import behavior.