dtf-system/docs/LOCAL_SETUP.md

DTF local development

Read CONTEXT.md first. The service lives in app/; older portal, Kanban, agent, requirements, and SQL files are historical prototypes.

Start

Install Docker Engine/Desktop with Compose v2 or later. Docker must be running and your account must have permission to use it. The initial build downloads public container images and Python packages; running integrations are local.

From this repository directory:

docker compose -f compose.local.yaml up --build

docker-compose.yml is the production/R2 stack and will not start locally; the localhost stack is always compose.local.yaml.

No .env is required: Compose has the same disposable defaults as .env.example. To customize, copy .env.example to .env and edit it. Never use real credentials. For a detached start with readiness verification:

docker compose -f compose.local.yaml up --build -d --wait
docker compose -f compose.local.yaml ps

Component	Local URL / port	Purpose
Site DTF	http://localhost:8080	Existing Site with local checkout bridge
Customer portal	http://localhost:8080/portal.html	Local account, order history, corrections and files
Kanban	http://localhost:8081	Quote review, orders, movement, original downloads
API health	http://localhost:8000/health	Checks PostgreSQL and MinIO
MinIO S3	http://localhost:9000	Direct signed multipart uploads and downloads
MinIO console	http://localhost:9001	Inspect private local storage
PostgreSQL	db:5432, internal only	Shared persistent data
ClamAV	scanner:3310, internal only	Isolated malware scanner with bundled signatures
Worker	worker:8002/health, internal only	Mock outbox delivery, scanning and combined health

Use localhost consistently; mixing it with 127.0.0.1 creates a different browser session. Published ports bind only to 127.0.0.1.

Kanban default login: operator@example.test / local-operator-only. MinIO default login: dtf_local / local-storage-only. These are public, disposable development values, not real credentials. Interactive API documentation is disabled; /docs and /redoc are not local operator interfaces.

Browser test order

1. Open the Site. Choose Arquivo por metro and the manual/table-price path (CDR/AI/PSD/TIFF). Select tests/fixtures/local-test.cdr. This is deliberately harmless text for upload testing, not a printable CDR file.
2. Enter 1.01 metres. The original calculation bills 1.10 m × R$19.90 = R$21.89, with grade 0 and pickup. You can also use your own non-sensitive artwork with the original product controls, repetitions, and mixed cart.
3. Use test CNPJ 11.222.333/0001-81, phone 11999999999, email local-test@example.test, and pickup. For simulated freight, select delivery, enter an eight-digit CEP, and click quote. The default mock price is R$15.00.
4. Click the Site checkout button. Files upload directly to MinIO, remain quarantined until ClamAV returns clean, and then become eligible for a quote. The local banner displays a quote ID awaiting commercial review.
5. Open Kanban and log in. Open the Cotações tab, pick the quote, download the original if needed, confirm/correct metres and grade, tick the confirmation at the bottom and click Aprovar cotação. For the fixture keep 1.01 m and grade 0.
6. Return to the Site and click its checkout action again. Inspect the authoritative server total, then click Criar pedido de teste. No real payment occurs.
7. Click Atualizar on the Kanban. The paid order appears in Arte recebida on the Produção tab. Click its card to open the order panel, then Mover para Arte tratada (dragging the card to a column also works). In the panel's Arquivos finais, select the manually prepared final file for every item, enter a review note, tick the confirmation and click Aprovar arquivos finais. Use the harmless fixture again only for this local test. The text fixture is not an image, so its print file shows Preparar à mão; with a PNG, JPEG or PDF artwork the generated file is preselected instead (see "Print files"). Continue with the panel's main button through Fila de impressão → Imprimindo → Finalizado. Pedir correção asks for the reason the customer will see; from Correção the order returns to Arte recebida or Arte tratada.
8. The panel's Histórico lists every move. The Pagamentos e integrações tab shows the Tiny connection, payments that need a person, and the send log; with the fake adapters every send reads "Registrado (simulado)". Download links expire after five minutes; click again for a new one. The previous Kanban is kept in git tag ui-v1.

The manual quote step is necessary to enforce the backend trust boundary while automatic pre-flight is deferred. Browser measurements and grade are proposals. Only an authenticated operator can approve them; the server derives prices and freight. Payment accepts a quote ID only. Approved quotes are immutable for 24 hours, and repeated/concurrent payment requests return the same order. This local review step does not settle the future automated production checkout. Malware scanning is a separate safety gate and does not inspect dimensions, resolution, colors, printability, or any other pre-flight property.

The existing client-side previews/analysis remain unchanged and advisory. No new automatic print pre-flight or artwork validation runs on upload; malware scanning is the separate gate described above. Original downloads and manually approved final files are separately labeled. Never print the local text fixture.

Customer accounts, corrections, and cart recovery

Open Minha conta on the Site, or /portal.html. Create a local account with a test CNPJ, phone, email and password of at least 12 characters. Passwords use the current stronger scrypt format; legacy local hashes are verified and upgraded on successful login. Sessions are stored in PostgreSQL, expire after seven days, and are revoked on logout. Login/registration attempts are limited. No email service is used: email verification and password recovery remain absent.

Registration associates only the current guest session's uploads, quotes and orders with the new account. Signing in from another browser restores access to that account's orders. Matching an email or CNPJ never grants access to an order. Guests can still order and track within their current session.

In the portal, Ver detalhes e arquivos displays status history, correction reasons, files and expiry dates. When Kanban requests Correção, upload corrected files against the relevant item and add a note. The order stays in correction until the operator reviews it. The old final set is invalidated; the operator must upload/approve another complete set before re-entering the queue. Concurrent or repeated submissions with a stale order version are rejected.

The browser saves unfinished cart items and their File blobs in IndexedDB for 24 hours. Reloading restores them as cart rows; remove/replace a row to alter its artwork settings, or add more products. Existing calculations are preserved and delivery must be requoted. Payment clears the saved cart. Cart recovery is local to the browser, not a cross-device artwork library; very large files can exceed browser storage capacity, which is reported visibly. Expired browser entries are removed the next time the Site opens. Server retention does not erase downloaded files or copies stored by the browser/operating system. Logout also clears local checkout keys and IndexedDB File blobs across open Site tabs. Operator authentication uses an expiring, revocable HttpOnly session; browser storage never retains the operator password or a Basic-auth credential.

Automated checks

Pricing parity requires Python 3.10+ and Node 22+ on the host, no package install:

python3 -m unittest local.test_pricing -v

This executes the real pricing constants/functions extracted from web/site-config.js and compares all four modes, 101 grades, and 11 lengths (4,444 cases) to backend pricing, plus assembly and invalid-input checks.

With the stack healthy, run the integration test (Python standard library only):

python3 -m tests.smoke_test
python3 -m tests.workflow_test

It checks direct two-part upload/resume, missing parts, session isolation, private downloads and byte identity, customer validation, tampered totals, operator corrections, all four products, freight, immutable quotes, concurrent payment retries, state rules, history, and durable fake integration receipts. Each run leaves one clearly labeled test order and an approximately 8 MiB object. The workflow test also checks guest-to-account migration, cross-session login, revoked sessions, ownership isolation, correction submissions, final revision invalidation, secure downloads, and mandatory final-file approval before queueing.

Security and malware regressions are separate so an authorized harmless EICAR test is unmistakable:

python3 -m tests.security_test
python3 -m tests.scanning_test
docker compose -f compose.local.yaml exec -T api python3 -m tests.runtime_security_test
docker compose -f compose.local.yaml exec -T api python3 -m tests.retention_test

local.scanning_test stores an EICAR fixture as SECURITY-EICAR.cdr; ClamAV must reject it, and quote/download gates must remain closed. The rejected fixture is retained for at most three days, so it temporarily appears in alert summaries.

For an automated real-browser walkthrough, install Chrome and use Node 22+:

node tests/browser_test.mjs

Set CHROME_BIN if Chrome is not at /usr/bin/google-chrome-stable. The test uses a separate temporary browser profile, walks through the actual Site and Kanban controls, and saves screenshots to output/local/. It leaves its local test order for inspection. Both integration scripts read .env automatically.

Configuration and storage

.env.example lists local ports, database/MinIO values, operator login, adapter selection, mock freight amount, transport ceiling (5 GiB), and multipart size (8 MiB by default). The API and customer picker admit only files within the malware scanner's effective 128 MiB limit by default (SCAN_MAX_BYTES); the larger-file path remains a Week 2 decision. S3_ENDPOINT=http://storage:9000, database hostname db, and the internal service ports are fixed Compose wiring. The public S3 endpoint must resolve from the browser; keep http://localhost:9000 for this stack. Parts use 15-minute presigned URLs and unfinished reservations expire after one hour. Clients can cancel a reservation through the upload DELETE endpoint.

Fake integration adapters and s3-local storage are the default. Mercado Pago and Tiny can be selected for sandbox testing only (see "Provider sandboxes"); startup fails if their credentials are missing, if any other adapter is selected, or if a production environment or nonlocal S3 endpoint is used. The API, worker, and database run on an internal Docker network; web gateways and MinIO also join a network that permits loopback port publishing.

Objects are private, use UUID keys rather than filenames, and persist in a named volume. MinIO lifecycle rules expire objects after 30 days and abandon incomplete multipart uploads after one day as a backstop; the API lease and worker release unfinished reservations after one hour. The API also blocks expired downloads. Order history remains in PostgreSQL. Completed files start in pending; unknown, scanner-error, over-limit, encrypted/unsafe, and malware results fail closed. Only clean files can cross quote, payment, download, final-approval, and queue gates. Rejected/error objects expire within three days. The worker deletes original bytes within seven days of manual final artwork approval, and final/correction bytes by 30 days from the order's first upload. Expired files remain visible as metadata but cannot be downloaded. Retention runs once a minute; MinIO lifecycle is a backstop.

Upload retries resume completed parts. The saved cart retains files when browser storage permits; otherwise reselect them. Saved quote IDs also survive reloads. Clearing cookies loses a guest session, while registered customers can sign in again. The operator can still inspect order records.

Print files

Every paid order queues one print-file job per item. The worker renders a PDF exactly as wide as the film and as long as the approved layout, placing each copy at the position, rotation and mirror the customer reviewed. Each source image is embedded once at its original resolution (JPEG bytes unchanged, PNG transparency kept as a soft mask), so nothing is resampled. The Kanban card shows the status per item, the page size and the lowest DPI, and offers Baixar PDF. In Arquivos de produção the generated file is preselected as the final file; untick it to upload one by hand instead.

JPEG, PNG, WebP and TIFF are embedded as images. A single-page PDF is placed as a vector form (never rasterised), using the page's CropBox and /Rotate exactly as the Site measured it with pdf.js; the card then shows PDF vetorial. PSD, AI and CDR artwork, multi-page or password-protected PDFs, a file whose proportions do not match the quoted size, a layout longer than was billed, or an image above PRINT_MAX_PIXELS (250 Mpx by default) goes to preparar à mão with the reason, and the operator prepares it as before. Gerar arquivos de impressão retries those items and generates files for orders paid before the generator existed. After a customer correction the generated file is no longer offered: it reproduces the replaced artwork.

Layouts longer than about 5 m use the PDF UserUnit page scale instead of being split into pages. Confirm on the factory's FlexiPRINT that such a file imports at full length before relying on it for long orders.

docker compose -f compose.local.yaml exec -T api python -m unittest tests.test_printfile -v
docker compose -f compose.local.yaml exec -T api python -m tests.print_file_test

With PyMuPDF installed locally (pip install pymupdf), the unit suite also draws each page and checks where every quadrant of the artwork lands.

Provider sandboxes

app/mercadopago.py and app/tiny.py follow the providers' public API documentation and pass their unit suites against a fake transport. They are not verified integrations until they pass with the client's accounts.

The default local stack gives the API and worker no route to the internet, so provider testing adds compose.providers.yaml, which does:

docker compose -f compose.local.yaml -f compose.providers.yaml up -d --wait

Put only test credentials in .env:

PAYMENT_ADAPTER=mercadopago
MP_ACCESS_TOKEN=TEST-...
MP_WEBHOOK_SECRET=...            # "Assinatura secreta" in the webhook settings
MP_NOTIFICATION_URL=https://<public tunnel>/api/payments/webhook

Tiny (API v3)

Tiny uses OAuth2. In the client's Tiny (Construa plan or above, with the "Gestão de Aplicativos" extension): **Configurações → Geral → Aplicativos →

• novo aplicativo**, with the redirect URL set to this system's callback. That gives a client ID and secret:

TINY_CLIENT_ID=...
TINY_CLIENT_SECRET=...
TINY_REDIRECT_URI=http://localhost:8081/api/operator/tiny/callback   # exactly as registered in Tiny
TINY_PRODUCT_TEXTIL_FOLHA=...            # Tiny product id for each Site product
TINY_PRODUCT_TEXTIL_AVULSA=...
TINY_PRODUCT_UV_FOLHA=...
TINY_PRODUCT_UV_AVULSA=...

With the client ID and secret set, the Kanban header shows Conectar Tiny. Someone with a Tiny login approves access once; the tokens are stored in the database (the refresh token rotates on every use) and the worker keeps the connection alive. Only then set TINY_ADAPTER=tiny, which starts creating an order in Tiny for every paid order: find or create the customer's contact by CNPJ, then POST /pedidos with numeroOrdemCompra = DTF-<order number>. A retry searches the customer's recent orders for that number first, so it does not create a second one. Tiny has no sandbox: every test order is real, so agree the test with the client and cancel the test orders afterwards.

tests.tiny_oauth_test checks the connection flow against the real database with a fake token server; it saves and restores any existing connection.

With Mercado Pago selected, an approved quote shows Pagar com PIX on the Site instead of the local test button, and Pagar com cartão when MP_PUBLIC_KEY is set. The card form is Mercado Pago's Card Payment Brick: the card is typed into Mercado Pago's secure fields and only a one-time token reaches the API. It loads from Mercado Pago, so the Site's CSP must allow it:

MP_PUBLIC_KEY=TEST-...
PAYMENT_CSP_SOURCES=https://sdk.mercadopago.com https://*.mercadopago.com https://*.mlstatic.com https://*.mercadolibre.com

PAYMENT_CSP_SOURCES is empty by default, which keeps the Site at script-src 'self'. Once a payment for a quote is approved, or a card payment is in review, the API refuses any further attempt for that quote. The order is created only by the signed notification, after the payment is fetched from the Mercado Pago API and its BRL amount matches the approved total. Mercado Pago must be able to reach the webhook, so a local run needs a public HTTPS tunnel to the Site port. A paid notification that cannot become an order, or a refund on an existing order, appears under Pagamentos que precisam de atenção on the Kanban until an operator records the resolution.

Local backup and restore check

python3 -m ops.backup create-and-verify

This writes a private four-file bundle in backups/ (ignored by Git and Docker builds): a PostgreSQL custom-format dump, a gzip object archive, a JSON manifest, and the manifest's SHA-256 sidecar. The object archive includes only complete, unexpired files that ClamAV has marked clean; pending, rejected, errored, expired, and purged objects remain excluded. Run the command while uploads and retention are idle so the database dump and subsequent object selection describe the same local state.

Verification restores the dump into a newly created UUID-named database and the object bytes under a UUID-named originals/restore-verification/ MinIO prefix. It checks database counts, bundle hashes, every archived object's hash, and the bytes downloaded after restore, then removes only the temporary database and objects. It never restores over active data. Keep every bundle file private: it contains customer data, password hashes, and customer artwork. To verify it again, run python3 -m ops.backup verify followed by the printed backups/...manifest.json path. Legacy database-only .dump backups remain verifiable. Scheduling, offsite copies, and a production restore runbook remain unfinished.

After building the current image, test retention with synthetic files:

docker compose -f compose.local.yaml exec -T api python3 -m tests.retention_test

This checks that expired bytes are removed while unexpired files survive. It cleans up its own synthetic object bytes and retains their metadata.

Operations and troubleshooting

docker compose -f compose.local.yaml logs --tail=100 api worker scanner
docker compose -f compose.local.yaml restart api worker
docker compose -f compose.local.yaml ps
docker compose -f compose.local.yaml down

down stops the stack and preserves named database/storage volumes. Restart with docker compose -f compose.local.yaml up -d --wait. Do not add --volumes unless you intend to permanently erase all local orders and artwork. No reset is required for tests.

Seven long-running services have Docker health checks; the database and storage initialization jobs must exit successfully. API health queries PostgreSQL and MinIO; worker health fails if its database-processing loop, scanning thread, or live ClamAV PING is unavailable. Both web gateways expose /health. MinIO exposes /minio/health/ready.

Run the redacted local alert summary inside the API network namespace:

docker compose -f compose.local.yaml exec -T api python3 -m ops.security_status

Exit status 1 means attention is required. Review blocked artwork, rate limits, failed logins, scanner availability, and signature age. Immediately after the security tests, expected synthetic login failures/rate limits and SECURITY-EICAR.cdr rejections will trigger the alert; confirm names and test timing before classifying them as expected. Unexpected failures, scanner errors, stale signatures (more than seven days), or non-test rejected files require investigation. Events are retained for 30 days. See SECURITY_REPORT.md.

If a host port is occupied, change SITE_PORT, KANBAN_PORT, or API_PORT in .env and recreate the stack. MinIO currently reserves 9000/9001. If Docker says permission denied, fix Docker access or use your OS-approved Docker workflow. If image/package download fails, check development internet access. A warning about missing Buildx can fall back to Docker's classic builder; install your platform's Buildx plugin for the supported modern build path.

If API is unhealthy, inspect its logs and check local adapter settings. Database and MinIO credentials initialize persistent volumes only once; changing values later also requires updating the existing database/storage account deliberately. If an upload fails, keep the session and click checkout again. If parts expired, reselect the file to start another upload. For a stale Kanban move, refresh; the backend rejects stale versions instead of overwriting another operator.

The Site retains its existing remote visual assets (fonts/logo) and optional PDF.js 3.11.174 CDN dependency. Its known eval-based advisory is mitigated in the current call with isEvalSupported:false; the old CDN dependency is still an upgrade/vendor-pinning limitation, not a claim that it is current. The local API, manual fixture flow, storage, payment, freight, Kanban, and mock integrations work without these external services. Offline PDF previews may fall back to the prototype's manual/table-price path.

Dependency lock

The API, worker, and database initializer install every Python dependency from infra/requirements.lock with --require-hashes. infra/requirements.txt remains the human-maintained direct dependency list. After deliberately changing a direct pin, regenerate the lock in the same Python 3.12 environment and rebuild:

./infra/lock_dependencies.sh
docker compose -f compose.local.yaml up --build -d --wait

The generator downloads public package metadata in a disposable container and does not modify the host Python environment. Review the resolved versions and run the exact-runtime audit plus regression suite before accepting an updated lock.

Staging readiness gate

compose.staging.yaml is intentionally not a staging deployment. It runs only a network-disabled validator for non-secret decisions. After completing PRODUCTION_INPUTS.md, copy staging/staging.env.example to the ignored staging/staging.env, enter non-secret metadata, and run:

docker compose -f compose.staging.yaml run --rm readiness

A pass does not authorize deployment or prove provider access. The real staging composition and external secret injection still require approved provider contracts and owners. See staging/README.md.

Next production connection step

Keep this stack local. Before connecting real services, confirm the production checkout trust workflow, complete PRODUCTION_INPUTS.md, and pass the isolated staging-readiness gate. Then implement the actual staging composition using the adapter contracts in app/adapters.py: start with private S3 staging storage, CORS, signed multipart contract tests and scoped credentials injected outside Git. Add provider sandbox adapters one at a time. Mercado Pago requires authenticated, signed, idempotent webhook handling before real payment is allowed; Tiny/Olist and WhatsApp need confirmed contracts and provider idempotency/delivery handling. Do not simply change the local fake flags or point this Compose file at production.

Production delivery package

The repository now includes a separate Docker Swarm and Gitea Actions delivery package in deploy/ and .gitea/workflows/. It is not used by this localhost Compose stack and does not alter its volumes. Production images run as unprivileged users and install the hash-locked Python runtime. Gitea publishes latest for the normal Portainer webhook plus the full commit SHA for rollback. The stack expects pre-provisioned external Swarm secrets and an external PostgreSQL volume. Only Site and Kanban ports are published for the existing TLS reverse proxy; API, database, worker, and scanner remain private.

Run the source-only gate without credentials:

python3 deploy/production_preflight.py --source-only

It must remain blocked while app/ supports only local fake adapters and does not load Docker secret *_FILE settings. Do not bypass or delete this check. After approved production implementations and inputs exist, follow PORTAINER.md and deploy/PRODUCTION_CHECKLIST.md; release and deployment still require the Gitea scan/test gates and explicit approvals.