blyzer/graphs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document supply and Olist operating model

Browse Source
This commit is contained in:
Cauê Faleiros
2026-08-25 09:14:24 -03:00
parent 11443a9202
commit 713cc4a9fc
1 changed files with 61 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
CONTEXT.md
View File

@@ -1,7 +1,7 @@
# Context
## 1. Project Overview
This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a real-time sales and inventory dashboard. Its primary purpose is to ingest live webhook payloads from an external ERP (Tiny ERP) via n8n, securely store that data, and provide a visually rich, responsive dashboard for business analytics. It tracks total sales, product performance, customer behavior, and live inventory levels, while also providing tools for WhatsApp marketing campaigns.
This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a real-time sales, inventory, and operations dashboard. It ingests webhook payloads from Tiny/Olist through n8n, securely stores that data, and provides business analytics, WhatsApp campaign tools, and a local Suprimentos/PCP operating layer. The operational flow is: **Necessidade de Compra → Plano de Corte → Ordens de Produção**.
## 2. Tech Stack & Tooling
@@ -37,8 +37,8 @@ This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a rea
* **Backend Analytics Foundation:** Raw `/api/data` still exists for legacy pages, but dashboard/product/client aggregate endpoints now exist under `/api/analytics/*`. Dashboard already uses the backend dashboard aggregation endpoint so it does not wait on the full raw order download.
* **Route Code Splitting:** Frontend routes are lazy-loaded with `React.lazy`/`Suspense` so the first JS payload stays smaller. Recharts-heavy page chunks are loaded on demand.
* **Analytics Client Cache:** Frontend analytics requests use a small shared cache/deduping layer in `src/dataService.ts`. Navigating between Dashboard, RFV, Products, and Clients should reuse fresh responses instead of refetching every route mount unless query params change or the cache expires.
* **Tiny as Current Source, Graphs as Future Operating Layer:** Tiny ERP is the current source for sales, product/order metadata, and stock. Graphs should increasingly become the place where the business manages production intelligence: cut configuration, SKU families, raw material relationships, consumption references, and operational charts. `NECESSIDADE DE CORTE.xlsx` is reference material only and must not become a runtime dependency.
* **Tiny/Olist Composition Import:** Product structures from Tiny/Olist V3 are imported into `product_compositions` and `product_composition_components`. Sellable finished-product structures are also promoted into local catalog products/materials and `consumption_references` with source `tiny_structure`, so purchase planning can use real component consumption instead of guessed kg-per-piece rules.
* **Tiny as Current Source, Graphs as Operating Layer:** Tiny/Olist remains the source for ERP sales, product/order metadata and imported stock. Graphs owns operational enrichment: cut configuration, SKU families, manual material details, consumption references, local production orders, and audited physical-stock overrides. `NECESSIDADE DE CORTE.xlsx` is reference material only and must not become a runtime dependency.
* **Tiny/Olist Composition Import:** Product structures from Tiny/Olist V3 are imported into `product_compositions` and `product_composition_components`. Sellable finished-product structures are promoted into local catalog products/materials and `consumption_references` with source `tiny_structure`, so purchase planning uses real component consumption rather than guessed kg-per-piece rules. Operators can also add, edit, and remove local composition lines when the ERP structure is missing or requires a controlled correction.
## 4. Directory Structure
```text
@@ -69,7 +69,8 @@ This project (often referred to as "Nexstar Graphs" or simply "Graphs") is a rea
* **Order Entity (`orders` table):** Tracks `cliente_nome`, `data_pedido`, normalized `data_pedido_date`, `valor_pedido`, `produto_id`, `quantidade`, `valor_unitario`, `pedido_id`, and `cliente_fone`. Tiny ERP metadata is also persisted when present: `cliente_nome_fantasia`, `id_vendedor`, `nome_vendedor`, `marketplace`, `canal_venda`, and `numero_ecommerce`.
* **Order Metadata Preservation:** Order upserts preserve existing non-empty phone and Tiny metadata values when incoming backfill rows send `NULL` or empty strings. Core mutable order/item fields still overwrite normally.
* **Client Identity Rule:** Analytics groups clients by a stable normalized identity so adding a phone number later does not split an existing buyer into a duplicate client. When a phone is added after historical purchases without phone, the old and new rows should resolve to the same client profile and history.
* **Stock Entity (`stock` table):** Tracks `produto_id`, `nome`, `saldo` (absolute current inventory), and `delta_estoque`. The database treats the ERP's `saldo` as the Absolute Truth (overwriting existing values rather than performing math) to prevent desynchronization.
* **Imported Stock Entity (`stock` table):** Tracks `produto_id`, `nome`, `saldo` (the absolute current balance reported by Tiny), and `delta_estoque`. Tiny stock remains the imported/audit value and is never overwritten by Graphs.
* **Graphs Physical-Balance Override:** `stock_graphs_balances` stores an operator-verified, absolute physical balance with reason, effective date, and editor. `stock_graphs_balance_history` records each set/removal along with the Tiny value at that point. Planning uses the Graphs balance when present; otherwise it uses Tiny. Removing the override returns planning to Tiny without modifying Tiny data.
* **Campaign Queue Entity (`stock_campaign_queue` table):** Tracks queued product restock deltas by `base_product_name`, product ID/name, delta, status (`pending`, `processing`, `sent`, `failed`, `skipped`), attempts, errors, and timestamps.
* **WhatsApp Campaign Payload:** The backend sends one n8n webhook payload per scheduled campaign run. The payload includes `baseProduct`, `productsText`, `total_delta`, `sizes`, `products`, and `customers`.
* **Campaign Product Display Names:** Product display text is customer-facing and may differ from internal grouping. Current aliases: `BASE LISA CAMISETA ...` -> `Camiseta Premium ...`, `BASE LISA OVER SIZE ...` -> `Camiseta Premium Over Size ...`, and `BASE LISA MOLETOM CANGURU ...` -> `Moletom Canguru Premium ...`.
@@ -112,7 +113,7 @@ yield: 4.8 units/kg
* **Finished SKU catalog:** SKU/id, name, color, size, product family, cut family, active/inactive.
* **Raw material catalog:** material SKU/id, name, type (`malha`, `ribana`, `fio`, etc.), color, supplier, unit (`kg`, `metro`, `unidade`), and whether it is raw input instead of finished stock.
* **Consumption references:** finished SKU, material product, yield per kg, optional yield by size, optional color/family overrides, and secondary materials when needed.
* **Imported product compositions:** Tiny/Olist structures are stored as immutable-ish sync facts: finished SKU/Tiny ID/unit, component SKU/Tiny ID/name, quantity per finished unit, and component unit (`KG`, `UN`, etc.). These records support Product Details composition display and drive `tiny_structure` consumption references.
* **Imported and local product compositions:** Tiny/Olist structures retain finished SKU/Tiny ID/unit, component SKU/Tiny ID/name, quantity per finished unit, and component unit (`KG`, `UN`, etc.). They support Product Details composition display and `tiny_structure` consumption references; local maintenance can add or correct composition data where operationally required.
* **Cut family mapping:** rules that say which colors/sizes/products can be planned together for cutting.
* **Open production/cutting data:** quantities already in production, expected finish date, linked finished SKU, and linked raw material when available.
* **Business rules:** target coverage days, minimum stock, safety stock, purchase lead time, production lead time, and discontinued/ignored SKU rules.
@@ -120,14 +121,20 @@ yield: 4.8 units/kg
**Current UI/data-flow behavior:**
* `Cadastros` has an `Importar composições` JSON upload action for exports like `composicoes_produtos_YYYY-MM-DD.json`. The same import is available to scripts through `POST /api/production-orders/tiny-compositions/import` using `x-api-key`.
* Composition imports are idempotent. Re-importing the same Tiny/Olist export updates composition rows/components and refreshes derived `tiny_structure` consumption references instead of duplicating them.
* `Product Details > Composição` shows the synced component list for the current product when a composition exists.
* `Product Details > Composição` shows the effective component list and supports manual add, edit, and delete actions. This is intentionally available even when no synced composition exists.
* `Suprimentos > Necessidade de Compra` derives material pressure from demand and stock, but rows without a consumption reference are marked as missing reference instead of pretending to calculate kg.
* Imported `tiny_structure` consumption references now allow `Necessidade de Compra` to calculate mixed-unit material demand from real BOM lines, e.g. malha/ribana in `KG` and etiqueta/ilhós/atacador in `UN`.
* Missing-reference rows link into `Cadastros > Referência de Consumo` with SKU context. If the SKU does not exist in the local catalog yet, `Cadastros` opens the product form first.
* `Cadastros > Referência de Consumo` labels reference sources as `Manual`, `Tiny OP`, or `Tiny Estrutura`.
* `Plano de Corte` SKU edit actions deep-link into the cutting configuration drawer for that SKU.
* `Plano de Corte` SKU edit actions open the focused cut-configuration dialog for that SKU. Family yields and SKU corrections are managed locally; historical Olist OP data can provide a suggested yield but never silently overwrites a configured value.
* Product tables and group detail tables are tuned for high-volume values, including large `Total Vendido` quantities above 100k.
**Planning safety rules:**
* A finished SKU without a reliable imported/manual composition must **not** block other SKUs. It produces no material demand or purchase need, but can still intentionally enter a cut plan and a local OP.
* A negative Tiny or Graphs balance is visible for audit, but counts as **zero available stock** in planning. It neither blocks the supply flow nor inflates purchase suggestions. Example: demand `500` and Tiny stock `-180` means purchase `500`, not `680`.
* `Dados Pendentes` is an informational/audit centre, not a blocker. It separates negative balances, SKUs without compositions, and incomplete cut registration data. Active queues can be exported to CSV for correction work.
* Material details are editable in a shared modal: display name, preferred supplier (from the Graphs supplier list), operational notes, and the optional audited Graphs physical balance. This keeps material editing consistent from Supply and Data Pending screens.
**Current composition import status:**
* Local import tested with `/home/farelos/Downloads/composicoes_produtos_2026-07-31.json`.
* Imported successfully: `437` compositions, `1,552` component rows, and `1,316` `tiny_structure` consumption references.
@@ -141,13 +148,54 @@ yield: 4.8 units/kg
* Upgrade `Planejamento de Corte` to show material blockers by family/color/size, using imported composition references and current stock.
* Add production-plan status flow only after material needs and composition health are reliable.
## 7. CI/CD & Deployment
## 7. Current Operations: Supply, Cutting, Production, and Olist
### 7.1 Suprimentos and PCP
* The six supply views are: the Suprimentos hub, Necessidade de Compra, Plano de Corte, Ordens de Produção, Estoque e Recebimentos, and Planejamento de Malha/Prioridades de Estoque.
* The hub prioritises real material purchases, then active local production orders, then cutting work. It does not elevate missing compositions or negative balances as operational blockers.
* Purchase calculations use actual BOM consumption and the effective planning balance. Negative balances are displayed for audit, but `getAvailablePlanningStock()` clamps availability to zero.
* `Cadastros` paginates Products, Categories, and Consumption References with 10 rows by default. Products, categories, consumption references, and compositions have local maintenance actions. Categories use the same compact action style as other tables.
* The cut plan has a compact demand table, filters, pagination, direct product actions, and a centred configuration dialog. The configuration manages cut-family yields and product corrections without a right-side drawer. Numeric yield inputs hide native browser spinners.
* The OP preview is paginated and selectable. Operators can select individual SKUs, all visible/eligible SKUs, or use the ready-data subset before confirming generation. Existing open local OPs for the same cut plan and SKU are ignored to prevent duplicate local OPs.
* Graphs-created OPs are local records, with incremental display numbers such as `OP-01999`; the number is not random and is not an Olist OP number. An open OP can be returned to the cut plan, restoring its quantity; progressed/finished work must not be returned casually.
* The production-order screen uses a compact list with inline expansion rather than a side detail page. Each OP has one status control (`Em aberto`, `Em andamento`, `Finalizada`, `Cancelada`), and expanded details show product markers, composition, and material consumption context. Material consumption is an OP-level action, relevant once production has started rather than a permanent page-level form.
### 7.2 Olist V3 connection and composition synchronisation
* Graphs has a Super Admin **Olist** monitor at `/#/admin/olist`. The connection action, manual sync, reconnection, stop action, live status, run history, run logs, and affected-product view are all on this page; Cadastros does not own the Olist connection flow.
* OAuth uses Olist/Tiny V3 with an authorization code and refresh token. Tokens are encrypted before storage in `olist_connections`; access tokens are refreshed automatically before expiry. The monitor shows the token expiry as an operational status, not an indication that the connection has failed.
* Required production variables are `OLIST_CLIENT_ID`, `OLIST_CLIENT_SECRET`, `OLIST_REDIRECT_URI`, `OLIST_FRONTEND_URL`, `OLIST_TOKEN_ENCRYPTION_KEY`, and `OLIST_SYNC_ENABLED=true`. The encryption key must be stable (a 32-byte base64 key or 64-character hex key): changing it makes already stored tokens unreadable and requires reconnection.
* The OAuth callback route is `GET /api/olist/oauth/callback`; `OLIST_REDIRECT_URI` must be exactly that publicly reachable backend URL. After authorization Graphs redirects to `/#/admin/olist` on `OLIST_FRONTEND_URL`.
* Native sync first lists Olist products, then checks each candidate with `GET /public-api/v3/produtos/{idProduto}/fabricado`. Products with a manufactured BOM are upserted through the existing composition importer; products without a BOM are recorded as checked but do not produce fictional material demand.
* The first run is a full catalogue/BOM scan. Later runs request product changes using `dataAlteracao`; cached products whose structure has not yet been checked remain pending and are resumed before a fully incremental run. This makes a stopped or server-interrupted scan resumable instead of starting the BOM checks over.
* Olist requests are intentionally throttled to a minimum `2,500 ms` apart (`OLIST_SYNC_REQUEST_DELAY_MS`, floor 2500). This is at most 24 requests/minute, deliberately below Tiny's 60 requests/minute limit so other ERP automations retain capacity.
* With `OLIST_SYNC_ENABLED=true`, backend startup schedules the automatic composition sync daily at **11:00 America/Sao_Paulo** (`OLIST_SYNC_HOUR = 11`). It is an in-process scheduler: the backend must be running at that time. Manual sync remains the fallback. The UI reports the next scheduled run and the status/error from the latest run.
* Sync execution is single-run protected by a PostgreSQL advisory lock. Runs, progress, errors, affected products, and events are persisted in `olist_sync_runs` and `olist_sync_events`. The user may request a stop; the worker finishes the current safe step, marks the run cancelled, and later resumes pending structures. Server restart marks a running run failed with an explicit interruption message; pending cache entries still permit a later resume.
* Olist monitor history is paginated with five rows by default. Run-detail event logs and affected-product lists are paginated as well, so status polling does not require reading an unbounded history.
### 7.3 Confirmed Olist API scope and the OP CSV bridge
* **Confirmed by Olist AI/documentation:** V3 and V2 do not provide a supported endpoint to list or read existing Production Orders (OPs), OP status/history, production steps, notes, lots, suppliers, roll quantities, fabric kg, or yield. There is no OP webhook and no incremental OP endpoint. Graphs must not scrape the ERP interface.
* The only production-order endpoint reported by Olist is generation from a sales order (`POST /public-api/v3/pedidos/{idPedido}/gerar-ordem-producao`; V2 equivalent `gerar.ordem.producao.pedido.php`). It does **not** provide subsequent OP reading and is not used as a history feed.
* Product/BOM sync therefore needs only **Products / Read** permission. Additional production permission in the application screen cannot unlock an endpoint that the API does not expose.
* Historical finalized OPs enter Graphs through the manual **Importar OPs finalizadas** CSV bridge. The CSV must include the required columns: `id_op`, `numero_op`, `sku_produto`, `descricao_produto`, `quantidade_produzida`, `data_emissao`, `data_conclusao`, `situacao`, and `observacoes`. Useful optional columns are `pedido_relacionado`, `fornecedor`, `lote`, `quantidade_rolos`, `quilos_malha`, `quilos_ribana`, and `rendimento`.
* The importer accepts only `Finalizada` records with a valid ID, number, SKU, description, and positive quantity. It is idempotent: it matches the existing record by Tiny internal ID or OP number, updating it rather than duplicating it. The import history explicitly reports new, updated, ignored, and failed rows.
* Optional operational values may be provided directly as columns or extracted from `observacoes`. Supported conventions include `FORNECEDOR:`, `RENDIMENTO:`, `BOBINAS:`/`ROLOS:`/`QUANTIDADE DE ROLOS:`, `KG:`, `QUILOS DE MALHA:`, `RIBANA:`, and lot variants such as `Lote 01`, `Lote: 01`, and `- Lote 01`. Bobbin multipliers such as `07(X2)` are recognised. Supplier in this historical OP context is treated as the fabric supplier.
* Importing completed OP history stores the raw operational facts (supplier, lot, rolls, fabric/rib kg, and pieces/kg yield) in `production_orders`, records each file in `production_order_import_runs`, and recalculates yield recommendations. Recommendations are available by family and by family + supplier. The cut configuration shows a suggested **units per roll** only when enough imported factual data exists; using a suggestion is an explicit operator action.
* The current historical-yield aggregation covers `BLCS`, `BLOS`, `BLMC`, `BLPM`, and `BLRM`. Family reading is: `BLCS` Camiseta Regular, `BLOS` Oversize, `BLMC` Moletom Canguru, `BLPM` Camiseta Pima, and `BLRM` Regata Base. `BLRM` is a regata family and must not be grouped with `BLMC` or its moletom materials. Supplier-specific variation is meaningful and should be treated as a recommendation to validate with the cutting team, not an automatic truth.
* Graphs can export a **missing finalized OP data** CSV from its imported history. That file is the correction queue to send to the Olist AI/ERP extraction process. Olist AI exports must be requested in manageable OP-number batches when large note fields cause failures. Re-importing corrected batches is safe because they update the matching OPs.
* The Olist page keeps family recommendation cards visible in both the supplier-reference and imported-file tabs. Supplier references and imported file runs are tables with five rows per page, keeping the previous dense tag wall out of the main workflow.
### 7.4 Supply audit and supplier sources
* Tiny negative balances were investigated as an operational-data problem, typically stock exits without matching entries or historic migration/balance errors. The balance must remain visible for auditing but must not turn into an additional procurement deficit.
* A Graphs physical balance is the way to record a verified count while preserving Tiny's reported balance. The correction must include a reason and is fully auditable through the balance history; it is not a hidden manipulation of imported stock.
* The selectable preferred-supplier list is intentionally built from local material details, lots, receipts, fabric plans, and the imported OP supplier recommendations. It is a controlled Graphs list, not an assumption that Olist exposes a dedicated supplier master through this integration.
## 8. CI/CD & Deployment
* **Gitea Actions:** A workflow located in `.gitea/workflows/deploy.yml` triggers on pushes to the `main` branch.
* **Docker Registry:** The pipeline builds the `frontend` and `backend` Docker images and pushes them directly to `gitea.blyzer.com.br/blyzer/`.
* **Production Deployment:** Updates are deployed manually via Portainer by pulling the `latest` image tags from the Gitea registry and redeploying the stack.
* **Environment Variables:** Security secrets (`API_KEY`, `JWT_SECRET`, `POSTGRES_PASSWORD`, `N8N_WHATSAPP_TRIGGER_URL`) are injected via the Portainer stack configuration and passed into containers via `docker-compose.yml`.
## 8. Environment Setup & Scripts
## 9. Environment Setup & Scripts
**Running Locally:**
1. Start the database:
@@ -199,7 +247,7 @@ The local Docker services use `restart: unless-stopped`, so containers should co
```
* Expected response includes `imported`, `failed`, `componentCount`, `referenceCount`, `skippedReferenceCount`, and `issues`.
## 9. Coding Standards & AI Directives
## 10. Coding Standards & AI Directives
* **Strict Type Safety:** Use explicit TypeScript interfaces (defined in `types.ts`). Avoid `any` where possible. Do not bypass type checks with `// @ts-ignore`.
* **Idiomatic React:** Use functional components and hooks (`useState`, `useEffect`, `useMemo`). Complex data transformations (like merging arrays into chart-ready datasets) MUST be wrapped in `useMemo` to prevent unnecessary re-renders.
* **Tailwind Architecture:** All styling must be handled via Tailwind CSS utility classes. Avoid custom CSS files unless defining global font families or root variables in `index.css`.
@@ -208,7 +256,7 @@ The local Docker services use `restart: unless-stopped`, so containers should co
* **API Security:** All backend modifications exposing or altering data MUST use the `verifyToken` middleware for frontend requests or `authenticateAPIKey` for external n8n webhooks.
* **Build Discipline:** After frontend/backend behavior changes, run `npm run lint`, `npm run build`, and relevant backend tests. The user prefers builds after changes to catch issues before deployment.
## 10. Recent Work
## 11. Historical Recent Work
Recent commits related to RFV/date and client metadata behavior:
* `4131bf9 Update RFV lifecycle to monthly windows` - RFV recency/lost thresholds changed to `0-7`, `8-15`, `16-29`, and `30+` lost.
* `ef27cff Add seller performance dashboard charts` - dashboard now includes seller revenue/order charts.
@@ -236,7 +284,7 @@ Files most relevant to RFV:
Recent commits related to supply/cutting/data-flow and chart readability:
* `37d4a77 Import Tiny product compositions` - adds the Tiny/Olist composition JSON import flow, derived `tiny_structure` consumption references, app upload button, API-key import endpoint, and import tests.
* `800eb97 Derive consumption references from Tiny OP sync` - Tiny OP composition sync now creates catalog products/materials and reusable consumption references from OP component lines.
* `c07938c Add Tiny production order detail sync` - stores OP detail rows, components, steps, and observations from Tiny/Olist sync payloads.
* `c07938c Add Tiny production order detail sync` - an earlier internal/import-oriented OP detail foundation. Current Olist API knowledge supersedes the assumption of native OP history reads: existing Olist OP history is now imported through the explicit CSV bridge documented in section 7.3.
* `060b4da Connect supplies to project demand` - Suprimentos purchase needs now derive from project/order demand and stock, while missing consumption references are surfaced explicitly.
* `bc05fb4 Add SKU edit actions` - product/cutting/replenishment/group tables gained compact SKU view/edit actions.
* `63efb47 Route SKU actions to focused editors` - cut-plan edit opens SKU cutting configuration; missing-reference supply rows open the consumption reference flow.
@@ -252,6 +300,6 @@ Files most relevant to supply/cutting/data-flow:
* `src/analytics/cutting.ts` - cut-family parsing, cut needs, stock coverage, and product override logic.
* `src/pages/Products.tsx` and `src/pages/ProductGroupDetails.tsx` - product/group velocity, stock coverage, and high-volume display.
* `src/chartUtils.ts` - shared date bucketing, moving average, unknown-label cleanup, and chart date helpers.
* `backend/services/productionOrderService.js` - Tiny OP detail sync, Tiny/Olist structure import, composition storage, sellable structure filtering, and derived consumption-reference creation.
* `backend/services/productionOrderService.js` - local production-order management, composition storage, sellable structure filtering, and derived consumption-reference creation. Historical finalized Olist OP CSV import lives in `backend/services/productionOrderCsvImportService.js`.
* `backend/routes/productionOrderRoutes.js` and `backend/routes/catalogRoutes.js` - API-key script import and authenticated app import endpoints for composition JSON.
* `backend/test/productCompositionService.test.js` - coverage for Tiny/Olist structure sync and bulk composition import behavior.