# Context ## 1. Project Overview This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a real-time sales and inventory dashboard. Its primary purpose is to ingest live webhook payloads from an external ERP (Tiny ERP) via n8n, securely store that data, and provide a visually rich, responsive dashboard for business analytics. It tracks total sales, product performance, customer behavior, and live inventory levels, while also providing tools for WhatsApp marketing campaigns. ## 2. Tech Stack & Tooling **Frontend:** * **Library:** React 19.2.5 * **Language:** TypeScript 6.0.2 * **Build Tool:** Vite 8.0.10 * **Styling:** Tailwind CSS 4.2.4 * **Icons:** Lucide React 1.14.0 * **Charts:** Recharts 3.8.1 * **Routing:** React Router DOM 7.14.2 **Backend:** * **Environment:** Node.js * **Framework:** Express 5.2.1 * **Database:** PostgreSQL (via `pg` 8.20.0) * **Authentication:** JWT (`jsonwebtoken` 9.0.3) * **CORS & Middleware:** `cors`, `body-parser` **Infrastructure & CI/CD:** * **Containerization:** Docker & Docker Compose * **Proxy:** Nginx * **CI/CD:** Gitea Actions (deploy.yml) * **Automation:** n8n (External trigger source) ## 3. Architecture & Design Decisions * **Decoupled Client-Server:** The frontend is a statically built SPA served by Nginx, communicating with an isolated Node.js API. * **Idempotent Database Operations:** Due to potential retry/spam from n8n webhooks, database insertions strictly use `INSERT ... ON CONFLICT DO UPDATE SET` (UPSERTs). The backend dynamically generates fallback IDs using composite keys (`Name_Date_Value`) to prevent historical data squashing when explicit `ID_Pedido` fields are missing. * **Persistent Campaign Queue:** Stock increases are no longer held in memory. Positive stock deltas are inserted into `stock_campaign_queue`, grouped by normalized base product, and processed later by a scheduled n8n workflow. This survives restarts and lets operators inspect/retry campaign work. * **Scheduled Campaign Processing:** n8n calls `POST /api/internal/process-stock-campaigns` on fixed schedules, currently intended for 12:00 and 18:00 BRT. The endpoint claims ready product groups where accumulated delta is at least 100, combines all ready products into one campaign payload, and sends it to `N8N_WHATSAPP_TRIGGER_URL`. * **All-Time Top Buyers:** Campaign targeting uses the top 100 buyers across all order history. There is no date filter in this query. Customers are grouped by `cliente_fone` and ordered by total spend (`SUM(quantidade * valor_unitario)`). * **Smart Polling vs. SSE:** Real-time UI updates are handled via client-side polling (`setInterval`) rather than Server-Sent Events (SSE) to bypass persistent connection drops caused by production reverse proxies. * **Backend Analytics Foundation:** Raw `/api/data` still exists for legacy pages, but dashboard/product/client aggregate endpoints now exist under `/api/analytics/*`. Dashboard already uses the backend dashboard aggregation endpoint so it does not wait on the full raw order download. * **Route Code Splitting:** Frontend routes are lazy-loaded with `React.lazy`/`Suspense` so the first JS payload stays smaller. Recharts-heavy page chunks are loaded on demand. * **Analytics Client Cache:** Frontend analytics requests use a small shared cache/deduping layer in `src/dataService.ts`. Navigating between Dashboard, RFV, Products, and Clients should reuse fresh responses instead of refetching every route mount unless query params change or the cache expires. * **Tiny as Current Source, Graphs as Future Operating Layer:** Tiny ERP is the current source for sales, product/order metadata, and stock. Graphs should increasingly become the place where the business manages production intelligence: cut configuration, SKU families, raw material relationships, consumption references, and operational charts. `NECESSIDADE DE CORTE.xlsx` is reference material only and must not become a runtime dependency. * **Tiny/Olist Composition Import:** Product structures from Tiny/Olist V3 are imported into `product_compositions` and `product_composition_components`. Sellable finished-product structures are also promoted into local catalog products/materials and `consumption_references` with source `tiny_structure`, so purchase planning can use real component consumption instead of guessed kg-per-piece rules. ## 4. Directory Structure ```text / ├── .gitea/workflows/ # CI/CD pipeline definitions ├── backend/ # Node.js Express API │ ├── Dockerfile # Backend container definition │ ├── index.js # Backend entry point │ ├── db.js # PostgreSQL pool and startup schema/migration SQL │ ├── routes/ # Express route modules │ ├── services/ # Business logic and campaign processing │ ├── mappers/ # Payload normalization helpers │ ├── test/ # Node test runner tests │ └── package.json ├── public/ # Static assets (Favicons) ├── src/ # React Frontend Application │ ├── components/ # Reusable UI elements (Layout, DateRangePicker) │ ├── pages/ # Route-level views (Dashboard, Products, Clients) │ ├── dataService.ts # Centralized API fetch logic and JWT handling │ ├── types.ts # Shared TypeScript interfaces │ └── main.tsx # React entry point ├── docker-compose.yml # Local orchestration and environment variable mapping ├── nginx.conf # Production web server routing └── vite.config.ts # Frontend build configuration ``` ## 5. Core Business Rules & Domain Entities * **Order Entity (`orders` table):** Tracks `cliente_nome`, `data_pedido`, normalized `data_pedido_date`, `valor_pedido`, `produto_id`, `quantidade`, `valor_unitario`, `pedido_id`, and `cliente_fone`. Tiny ERP metadata is also persisted when present: `cliente_nome_fantasia`, `id_vendedor`, `nome_vendedor`, `marketplace`, `canal_venda`, and `numero_ecommerce`. * **Order Metadata Preservation:** Order upserts preserve existing non-empty phone and Tiny metadata values when incoming backfill rows send `NULL` or empty strings. Core mutable order/item fields still overwrite normally. * **Client Identity Rule:** Analytics groups clients by a stable normalized identity so adding a phone number later does not split an existing buyer into a duplicate client. When a phone is added after historical purchases without phone, the old and new rows should resolve to the same client profile and history. * **Stock Entity (`stock` table):** Tracks `produto_id`, `nome`, `saldo` (absolute current inventory), and `delta_estoque`. The database treats the ERP's `saldo` as the Absolute Truth (overwriting existing values rather than performing math) to prevent desynchronization. * **Campaign Queue Entity (`stock_campaign_queue` table):** Tracks queued product restock deltas by `base_product_name`, product ID/name, delta, status (`pending`, `processing`, `sent`, `failed`, `skipped`), attempts, errors, and timestamps. * **WhatsApp Campaign Payload:** The backend sends one n8n webhook payload per scheduled campaign run. The payload includes `baseProduct`, `productsText`, `total_delta`, `sizes`, `products`, and `customers`. * **Campaign Product Display Names:** Product display text is customer-facing and may differ from internal grouping. Current aliases: `BASE LISA CAMISETA ...` -> `Camiseta Premium ...`, `BASE LISA OVER SIZE ...` -> `Camiseta Premium Over Size ...`, and `BASE LISA MOLETOM CANGURU ...` -> `Moletom Canguru Premium ...`. * **Campaign Product List Format:** Because Meta templates collapsed/ignored newlines inside one parameter, `productsText` is a single-line separator list: `Produto 1 • Produto 2 • Produto 3`. There is no leading bullet before the first product. * **Base Product Normalization:** Apparel size suffixes like `TAMANHO - P`, `- M`, `- G`, `- GG`, and `- M/G/GG` are stripped for campaign grouping. `ETIQUETA...` products are special-cased and preserve their full variant name, e.g. `ETIQUETA BRANCA TAMANHO GG`. * **Campaign Observability:** The frontend has a `Campanhas` page backed by `/api/campaigns`, `/api/campaigns/preview`, `/api/campaigns/process`, and `/api/campaigns/retry`. * **WhatsApp Marketing Integration:** The system extracts phone numbers from incoming n8n payloads (checking `Fone_Cliente`, `fone`, or `celular`). Numbers are exposed in the UI for direct "Click-to-Chat" links and exported to CSV files. * **Filter Persistence:** User preferences for Date Ranges, Sort options, and Auto-Refresh intervals are persisted to `localStorage` to survive page reloads. * **RFV Segmentation:** The Portuguese UI calls customer segmentation **RFV** (`Recência`, `Frequência`, `Valor`). Internal code/API names may still use `rfm` to avoid route/type churn. The visible app labels should use RFV. * **RFV Lifecycle Thresholds:** The active lifecycle is monthly, not yearly. Recency score 3 means last purchase in `0-7` days, score 2 means `8-15` days, score 1 starts at `16+` days, and clients with no purchase for `30+` days are forced into `Perdido`. * **RFV Date Semantics:** Date filters use local calendar-day boundaries and send stable `YYYY-MM-DD` query params. Backend analytics treats `start` and `end` as inclusive `data_pedido_date` bounds. * **RFV Tag Aggregation Rule:** RFV is not recalculated from only the selected period. For a selected period, the backend first calculates each buyer's RFV tag using history **before the period starts**. It then aggregates only the purchases inside the selected period by that prior tag. Example: if a client was `Champions` before today and buys today, `Champions` gets +1 client and that period revenue. If an `Em Risco` client does not buy today, `Em Risco` does not get counted for today's filter. * **RFV Display Rule:** The RFV table's `Última Compra` date is the purchase date inside the selected period. Date-only strings must be parsed as local dates in the frontend to avoid UTC shifting (`2026-06-15` must render as `15/06/2026`, not `14/06/2026`). The small recency text below it displays `Hoje`, `Ontem`, or `X dias` relative to the selected range end. * **Client Page Filters:** The main Clients page has one filter button plus client search. The filter panel contains date shortcuts/custom dates, sorting, marketplace, sales channel, seller, and client type. Marketplace/channel/seller options come from distinct non-empty order metadata across all orders so options remain visible regardless of the current date window. * **Seller Metadata Display:** Seller labels are cleaned for UI display. All-caps seller names are title-cased, trailing Tiny IDs like `#977226210` are stripped from labels, and leading fantasy-name prefixes like `[3]` are removed for display. IDs may still be used internally as stable filter values. * **Seller Dashboard Charts:** The main Dashboard includes seller performance charts above the existing charts, using backend aggregates for revenue and order count by seller. Long date ranges are bucketed into week/month views for readability. Seller labels like `0`/empty values are displayed as `Sem vendedor`. The seller moving-average line is optional behind the `Tendência` toggle because it can be confused with another seller series. * **Product Chart Readability:** Product detail charts use automatic date bucketing, cleaner x-axis labels, clickable point drill-downs, and a moving average on the single-product view. Product detail KPIs include daily average sold and estimated stock coverage based on the selected date range. ## 6. Supply, Cutting, and Data Flow Requirements Tiny provides sales and stock facts, but it does not fully describe how a finished SKU consumes raw material. For Suprimentos and Corte to work reliably, Graphs needs internal domain data that complements Tiny. **Current operating assumption:** * Tiny sends orders, product IDs/names, seller/order metadata, and current stock. * Graphs derives demand, sales velocity, replenishment pressure, and cut/material needs from Tiny data plus local configuration. * Excel files are examples of current manual management only. They should guide UX/data modeling but not be imported as the long-term source of truth. **Core missing relationship:** ```text Finished SKU -> raw material -> consumption/yield ``` Example: ```text BASE LISA CAMISETA PRETA G uses: MALHA 100% ALGODAO PRETA yield: 4.8 units/kg ``` **Data Graphs must own or enrich locally:** * **Finished SKU catalog:** SKU/id, name, color, size, product family, cut family, active/inactive. * **Raw material catalog:** material SKU/id, name, type (`malha`, `ribana`, `fio`, etc.), color, supplier, unit (`kg`, `metro`, `unidade`), and whether it is raw input instead of finished stock. * **Consumption references:** finished SKU, material product, yield per kg, optional yield by size, optional color/family overrides, and secondary materials when needed. * **Imported product compositions:** Tiny/Olist structures are stored as immutable-ish sync facts: finished SKU/Tiny ID/unit, component SKU/Tiny ID/name, quantity per finished unit, and component unit (`KG`, `UN`, etc.). These records support Product Details composition display and drive `tiny_structure` consumption references. * **Cut family mapping:** rules that say which colors/sizes/products can be planned together for cutting. * **Open production/cutting data:** quantities already in production, expected finish date, linked finished SKU, and linked raw material when available. * **Business rules:** target coverage days, minimum stock, safety stock, purchase lead time, production lead time, and discontinued/ignored SKU rules. **Current UI/data-flow behavior:** * `Cadastros` has an `Importar composições` JSON upload action for exports like `composicoes_produtos_YYYY-MM-DD.json`. The same import is available to scripts through `POST /api/production-orders/tiny-compositions/import` using `x-api-key`. * Composition imports are idempotent. Re-importing the same Tiny/Olist export updates composition rows/components and refreshes derived `tiny_structure` consumption references instead of duplicating them. * `Product Details > Composição` shows the synced component list for the current product when a composition exists. * `Suprimentos > Necessidade de Compra` derives material pressure from demand and stock, but rows without a consumption reference are marked as missing reference instead of pretending to calculate kg. * Imported `tiny_structure` consumption references now allow `Necessidade de Compra` to calculate mixed-unit material demand from real BOM lines, e.g. malha/ribana in `KG` and etiqueta/ilhós/atacador in `UN`. * Missing-reference rows link into `Cadastros > Referência de Consumo` with SKU context. If the SKU does not exist in the local catalog yet, `Cadastros` opens the product form first. * `Cadastros > Referência de Consumo` labels reference sources as `Manual`, `Tiny OP`, or `Tiny Estrutura`. * `Plano de Corte` SKU edit actions deep-link into the cutting configuration drawer for that SKU. * Product tables and group detail tables are tuned for high-volume values, including large `Total Vendido` quantities above 100k. **Current composition import status:** * Local import tested with `/home/farelos/Downloads/composicoes_produtos_2026-07-31.json`. * Imported successfully: `437` compositions, `1,552` component rows, and `1,316` `tiny_structure` consumption references. * Known data issues from that export: `4` sellable finished products are missing SKU; `54` structures are raw material/service structures and are stored as compositions but intentionally not promoted into sellable demand-planning references; `236` component/reference lines were skipped because their parent structure was non-sellable or the finished SKU was missing. * Sellable-family detection currently includes `DTF`, `CAMISETA`, `CAMISA`, `MOLETOM`, `CANGURU`, `REGATA`, and `POLO`. Raw/input/service structures are excluded from demand-reference promotion when descriptions contain terms like `MALHA`, `RIBANA`, `FIO`, `TECIDO`, `SERVIÇO`, `TINTURARIA`, `TECELAGEM`, or `FRETE`. **Recommended next steps now that composition data exists:** * Make `Necessidade de Compra` clearer and more auditable: show `demanda do SKU -> consumo por unidade -> material necessário -> estoque atual -> comprar`. * Add a data-health/review screen for missing finished SKU, missing component stock link, suspicious quantities, products with sales but no composition, and non-sellable structures. * Connect material stock more explicitly to imported component SKUs/Tiny IDs so material purchase needs subtract the right current stock and pending receipts. * Upgrade `Planejamento de Corte` to show material blockers by family/color/size, using imported composition references and current stock. * Add production-plan status flow only after material needs and composition health are reliable. ## 7. CI/CD & Deployment * **Gitea Actions:** A workflow located in `.gitea/workflows/deploy.yml` triggers on pushes to the `main` branch. * **Docker Registry:** The pipeline builds the `frontend` and `backend` Docker images and pushes them directly to `gitea.blyzer.com.br/blyzer/`. * **Production Deployment:** Updates are deployed manually via Portainer by pulling the `latest` image tags from the Gitea registry and redeploying the stack. * **Environment Variables:** Security secrets (`API_KEY`, `JWT_SECRET`, `POSTGRES_PASSWORD`, `N8N_WHATSAPP_TRIGGER_URL`) are injected via the Portainer stack configuration and passed into containers via `docker-compose.yml`. ## 8. Environment Setup & Scripts **Running Locally:** 1. Start the database: ```bash docker compose up -d db ``` 2. Start the Backend (from `/backend`): ```bash npm install npm start ``` 3. Start the Frontend (from project root): ```bash npm install npm run dev # For HMR development npm run preview # For production build testing ``` **Building the Frontend:** ```bash npm run build ``` **Backend Tests:** ```bash cd backend npm test ``` **Frontend Date/RFV Helper Test:** ```bash node --experimental-strip-types --test src/dateRanges.test.ts ``` **Persistent Local Stack:** ```bash docker compose up -d --build ``` The local Docker services use `restart: unless-stopped`, so containers should come back after laptop restart if Docker starts. **Importing Tiny/Olist Product Compositions:** * From the app: open `Cadastros` and use `Importar composições` with the exported JSON file. * From a script/API client: ```bash curl -X POST http://localhost:3004/api/production-orders/tiny-compositions/import \ -H 'Content-Type: application/json' \ -H 'x-api-key: nexstar_secret_key_123' \ --data-binary @/path/to/composicoes_produtos_YYYY-MM-DD.json ``` * Expected response includes `imported`, `failed`, `componentCount`, `referenceCount`, `skippedReferenceCount`, and `issues`. ## 9. Coding Standards & AI Directives * **Strict Type Safety:** Use explicit TypeScript interfaces (defined in `types.ts`). Avoid `any` where possible. Do not bypass type checks with `// @ts-ignore`. * **Idiomatic React:** Use functional components and hooks (`useState`, `useEffect`, `useMemo`). Complex data transformations (like merging arrays into chart-ready datasets) MUST be wrapped in `useMemo` to prevent unnecessary re-renders. * **Tailwind Architecture:** All styling must be handled via Tailwind CSS utility classes. Avoid custom CSS files unless defining global font families or root variables in `index.css`. * **Robust Data Handling:** Always implement graceful fallbacks for missing data. Never assume an API payload will contain all keys. (e.g., `item.id || item.ID_Pedido || ''`). * **Database Migrations:** There is no ORM (like Prisma or Sequelize). Table schemas, indexes, and lightweight data repairs are managed via raw SQL statements inside `initDB()` in `backend/db.js` using `IF NOT EXISTS` and safe startup execution patterns. * **API Security:** All backend modifications exposing or altering data MUST use the `verifyToken` middleware for frontend requests or `authenticateAPIKey` for external n8n webhooks. * **Build Discipline:** After frontend/backend behavior changes, run `npm run lint`, `npm run build`, and relevant backend tests. The user prefers builds after changes to catch issues before deployment. ## 10. Recent Work Recent commits related to RFV/date and client metadata behavior: * `4131bf9 Update RFV lifecycle to monthly windows` - RFV recency/lost thresholds changed to `0-7`, `8-15`, `16-29`, and `30+` lost. * `ef27cff Add seller performance dashboard charts` - dashboard now includes seller revenue/order charts. * `ca9615a Clean seller metadata display names` - seller/fantasy labels are normalized for display. * `b129307 Fix seller filter options query` - seller filter options are selected reliably from order metadata. * `b40f6fa Restore seller dropdown filter` - seller filter uses a dropdown option list again. * `aeb801c Allow searching sellers in client filters` - seller filter UI supports searching. * `b7a2059 Show all client metadata filter options` - client metadata filter options are global, not date-scoped. * `06faada Refresh client filter options reliably` - filter option fetching was made more robust. * `7d60073 Smooth analytics page reloads` - analytics fetches now reuse cached/deduped results between pages. * `335d231 Fix client identity when phone is added` - client grouping prevents phone-added duplicate profiles. * `73033ca Aggregate RFM period buyers by prior tag` - backend RFV aggregation now groups period buyers by their prior tag. * `ae73a54 Fix RFM period date display` - frontend date-only display and period recency fixed. * `9a58b2a Improve RFM recency labels` - period recency labels show `Hoje`/`Ontem`/`X dias`. * `f811d3c Rename visible RFM labels to RFV` - visible UI wording changed from RFM to RFV. Files most relevant to RFV: * `backend/services/analyticsService.js` - `getRfmAnalytics`, date filters, RFV score/tag logic. * `backend/test/analyticsService.test.js` - unit coverage for date filters and RFV aggregation. * `src/pages/Rfm.tsx` - RFV page UI, matrix/table/export labels, period recency display. * `src/components/DateRangePicker.tsx` and `src/dateRanges.ts` - local calendar date ranges and ISO query param formatting. * `src/dataService.ts` - analytics fetch/cache helpers and client filter option requests. * `src/displayFormatters.ts` - shared seller/fantasy/client metadata display cleanup. Recent commits related to supply/cutting/data-flow and chart readability: * `37d4a77 Import Tiny product compositions` - adds the Tiny/Olist composition JSON import flow, derived `tiny_structure` consumption references, app upload button, API-key import endpoint, and import tests. * `800eb97 Derive consumption references from Tiny OP sync` - Tiny OP composition sync now creates catalog products/materials and reusable consumption references from OP component lines. * `c07938c Add Tiny production order detail sync` - stores OP detail rows, components, steps, and observations from Tiny/Olist sync payloads. * `060b4da Connect supplies to project demand` - Suprimentos purchase needs now derive from project/order demand and stock, while missing consumption references are surfaced explicitly. * `bc05fb4 Add SKU edit actions` - product/cutting/replenishment/group tables gained compact SKU view/edit actions. * `63efb47 Route SKU actions to focused editors` - cut-plan edit opens SKU cutting configuration; missing-reference supply rows open the consumption reference flow. * `9489318 Improve chart readability and drilldowns` - dashboard/product charts gained date bucketing, cleaner unknown labels, click drill-downs, and product stock coverage context. * `ba9cc3a Fix large sold quantity icon layout` - product/group sold quantity cells handle large values without icon compression. * `f923489 Make dashboard trend line optional` - dashboard moving-average trend line is hidden by default and available through `Tendência`. Files most relevant to supply/cutting/data-flow: * `src/pages/Supplies.tsx` - supply dashboard, receipts, inventory, movements, and purchase-need UI. * `src/pages/Cutting.tsx` - cut plan, cut settings drawer, SKU correction/deep-link flow. * `src/pages/Registrations.tsx` - local catalog, raw materials, finished products, and consumption references. * `src/catalogLinks.ts` - shared deep-link builders for SKU edit, cutting config, and consumption references. * `src/analytics/cutting.ts` - cut-family parsing, cut needs, stock coverage, and product override logic. * `src/pages/Products.tsx` and `src/pages/ProductGroupDetails.tsx` - product/group velocity, stock coverage, and high-volume display. * `src/chartUtils.ts` - shared date bucketing, moving average, unknown-label cleanup, and chart date helpers. * `backend/services/productionOrderService.js` - Tiny OP detail sync, Tiny/Olist structure import, composition storage, sellable structure filtering, and derived consumption-reference creation. * `backend/routes/productionOrderRoutes.js` and `backend/routes/catalogRoutes.js` - API-key script import and authenticated app import endpoints for composition JSON. * `backend/test/productCompositionService.test.js` - coverage for Tiny/Olist structure sync and bulk composition import behavior.