dtf-system/docs/LOCAL_SETUP.md

DTF local development

Read CONTEXT.md first. The service lives in app/; older portal, Kanban, agent, requirements, and SQL files are historical prototypes.

Start

Install Docker Engine/Desktop with Compose v2 or later. Docker must be running and your account must have permission to use it. The initial build downloads public container images and Python packages; running integrations are local.

From this repository directory:

docker compose -f compose.local.yaml up --build

docker-compose.yml is the production/R2 stack and will not start locally; the localhost stack is always compose.local.yaml.

No .env is required: Compose has the same disposable defaults as .env.example. To customize, copy .env.example to .env and edit it. Never use real credentials. For a detached start with readiness verification:

docker compose -f compose.local.yaml up --build -d --wait
docker compose -f compose.local.yaml ps

Component	Local URL / port	Purpose
Site DTF	http://localhost:8080	Existing Site with local checkout bridge
Customer portal	http://localhost:8080/portal.html	Local account, order history, corrections and files
Kanban	http://localhost:8081	Quote review, orders, movement, original downloads
API health	http://localhost:8000/health	Checks PostgreSQL and MinIO
MinIO S3	http://localhost:9000	Direct signed multipart uploads and downloads
MinIO console	http://localhost:9001	Inspect private local storage
PostgreSQL	db:5432, internal only	Shared persistent data
ClamAV	scanner:3310, internal only	Isolated malware scanner with bundled signatures
Worker	worker:8002/health, internal only	Mock outbox delivery, scanning and combined health

Use localhost consistently; mixing it with 127.0.0.1 creates a different browser session. Published ports bind only to 127.0.0.1.

Kanban default login: operator@example.test / local-operator-only. MinIO default login: dtf_local / local-storage-only. These are public, disposable development values, not real credentials. Interactive API documentation is disabled; /docs and /redoc are not local operator interfaces.

Browser test order

1. Open the Site. Choose Arquivo por metro and the manual/table-price path (CDR/AI/PSD/TIFF). Select tests/fixtures/local-test.cdr. This is deliberately harmless text for upload testing, not a printable CDR file.
2. Enter 1.01 metres. The original calculation bills 1.10 m × R$19.90 = R$21.89, with grade 0 and pickup. You can also use your own non-sensitive artwork with the original product controls, repetitions, and mixed cart.
3. Use test CNPJ 11.222.333/0001-81, phone 11999999999, email local-test@example.test, and pickup. For simulated freight, select delivery, enter an eight-digit CEP, and click quote. The default mock price is R$15.00.
4. Click the Site checkout button. Files upload directly to MinIO, remain quarantined until ClamAV returns clean, and then become eligible for a quote. The local banner displays a quote ID awaiting commercial review.
5. Open Kanban and log in. In Cotações, download the original if needed, confirm/correct total metres and grade, tick the manual confirmation, and click Aprovar cotação. For the fixture keep 1.01 m and grade 0.
6. Return to the Site and click its checkout action again. Inspect the authoritative server total, then click Criar pedido de teste. No real payment occurs.
7. Refresh Kanban. The paid order starts in Arte recebida. Move it using the buttons or drag and drop to Arte tratada. Open Arquivos de produção, select the manually prepared final files for every item, enter a review note, tick the confirmation and click Aprovar arquivos finais. Use the harmless fixture again only for this local test; no printable file is generated. Continue through Fila de impressão → Imprimindo → Finalizado. A move to Correção requires a reason; it can return to Arte recebida or Arte tratada. Finalizado is terminal locally.
8. Inspect Histórico and Eventos locais de integração. Worker receipts should say recorded locally. Download links expire after five minutes; request another link from Kanban when needed.

The manual quote step is necessary to enforce the backend trust boundary while automatic pre-flight is deferred. Browser measurements and grade are proposals. Only an authenticated operator can approve them; the server derives prices and freight. Payment accepts a quote ID only. Approved quotes are immutable for 24 hours, and repeated/concurrent payment requests return the same order. This local review step does not settle the future automated production checkout. Malware scanning is a separate safety gate and does not inspect dimensions, resolution, colors, printability, or any other pre-flight property.

The existing client-side previews/analysis remain unchanged and advisory. No new automatic print pre-flight or artwork validation runs on upload; malware scanning is the separate gate described above. Original downloads and manually approved final files are separately labeled. Never print the local text fixture.

Customer accounts, corrections, and cart recovery

Open Minha conta on the Site, or /portal.html. Create a local account with a test CNPJ, phone, email and password of at least 12 characters. Passwords use the current stronger scrypt format; legacy local hashes are verified and upgraded on successful login. Sessions are stored in PostgreSQL, expire after seven days, and are revoked on logout. Login/registration attempts are limited. No email service is used: email verification and password recovery remain absent.

Registration associates only the current guest session's uploads, quotes and orders with the new account. Signing in from another browser restores access to that account's orders. Matching an email or CNPJ never grants access to an order. Guests can still order and track within their current session.

In the portal, Ver detalhes e arquivos displays status history, correction reasons, files and expiry dates. When Kanban requests Correção, upload corrected files against the relevant item and add a note. The order stays in correction until the operator reviews it. The old final set is invalidated; the operator must upload/approve another complete set before re-entering the queue. Concurrent or repeated submissions with a stale order version are rejected.

The browser saves unfinished cart items and their File blobs in IndexedDB for 24 hours. Reloading restores them as cart rows; remove/replace a row to alter its artwork settings, or add more products. Existing calculations are preserved and delivery must be requoted. Payment clears the saved cart. Cart recovery is local to the browser, not a cross-device artwork library; very large files can exceed browser storage capacity, which is reported visibly. Expired browser entries are removed the next time the Site opens. Server retention does not erase downloaded files or copies stored by the browser/operating system. Logout also clears local checkout keys and IndexedDB File blobs across open Site tabs. Operator authentication uses an expiring, revocable HttpOnly session; browser storage never retains the operator password or a Basic-auth credential.

Automated checks

Pricing parity requires Python 3.10+ and Node 22+ on the host, no package install:

python3 -m unittest local.test_pricing -v

This executes the real pricing constants/functions extracted from web/site-config.js and compares all four modes, 101 grades, and 11 lengths (4,444 cases) to backend pricing, plus assembly and invalid-input checks.

With the stack healthy, run the integration test (Python standard library only):

python3 -m tests.smoke_test
python3 -m tests.workflow_test

It checks direct two-part upload/resume, missing parts, session isolation, private downloads and byte identity, customer validation, tampered totals, operator corrections, all four products, freight, immutable quotes, concurrent payment retries, state rules, history, and durable fake integration receipts. Each run leaves one clearly labeled test order and an approximately 8 MiB object. The workflow test also checks guest-to-account migration, cross-session login, revoked sessions, ownership isolation, correction submissions, final revision invalidation, secure downloads, and mandatory final-file approval before queueing.

Security and malware regressions are separate so an authorized harmless EICAR test is unmistakable:

python3 -m tests.security_test
python3 -m tests.scanning_test
docker compose exec -T api python3 -m tests.runtime_security_test
docker compose exec -T api python3 -m tests.retention_test

local.scanning_test stores an EICAR fixture as SECURITY-EICAR.cdr; ClamAV must reject it, and quote/download gates must remain closed. The rejected fixture is retained for at most three days, so it temporarily appears in alert summaries.

For an automated real-browser walkthrough, install Chrome and use Node 22+:

node tests/browser_test.mjs

Set CHROME_BIN if Chrome is not at /usr/bin/google-chrome-stable. The test uses a separate temporary browser profile, walks through the actual Site and Kanban controls, and saves screenshots to output/local/. It leaves its local test order for inspection. Both integration scripts read .env automatically.

Configuration and storage

.env.example lists local ports, database/MinIO values, operator login, adapter selection, mock freight amount, maximum file size (5 GiB), and multipart size (8 MiB by default). The malware scanner releases only files up to 128 MiB by default (SCAN_MAX_BYTES); larger uploads remain blocked even though the multipart transport supports 5 GiB. S3_ENDPOINT=http://storage:9000, database hostname db, and the internal service ports are fixed Compose wiring. The public S3 endpoint must resolve from the browser; keep http://localhost:9000 for this stack. Parts use 15-minute presigned URLs and uploads must finish within one day.

Only fake integration adapters and s3-local storage are accepted. Startup fails if a production adapter/environment or nonlocal S3 endpoint is selected. The API, worker, and database run on an internal Docker network; web gateways and MinIO also join a network that permits loopback port publishing.

Objects are private, use UUID keys rather than filenames, and persist in a named volume. MinIO lifecycle rules expire objects after 30 days and abandon incomplete multipart uploads after one day; the API also blocks expired downloads. Order history remains in PostgreSQL. Completed files start in pending; unknown, scanner-error, over-limit, encrypted/unsafe, and malware results fail closed. Only clean files can cross quote, payment, download, final-approval, and queue gates. Rejected/error objects expire within three days. The worker deletes original bytes within seven days of manual final artwork approval, and final/correction bytes by 30 days from the order's first upload. Expired files remain visible as metadata but cannot be downloaded. Retention runs once a minute; MinIO lifecycle is a backstop.

Upload retries resume completed parts. The saved cart retains files when browser storage permits; otherwise reselect them. Saved quote IDs also survive reloads. Clearing cookies loses a guest session, while registered customers can sign in again. The operator can still inspect order records.

Local backup and restore check

python3 -m app.backup create-and-verify

This writes a private four-file bundle in backups/ (ignored by Git and Docker builds): a PostgreSQL custom-format dump, a gzip object archive, a JSON manifest, and the manifest's SHA-256 sidecar. The object archive includes only complete, unexpired files that ClamAV has marked clean; pending, rejected, errored, expired, and purged objects remain excluded. Run the command while uploads and retention are idle so the database dump and subsequent object selection describe the same local state.

Verification restores the dump into a newly created UUID-named database and the object bytes under a UUID-named originals/restore-verification/ MinIO prefix. It checks database counts, bundle hashes, every archived object's hash, and the bytes downloaded after restore, then removes only the temporary database and objects. It never restores over active data. Keep every bundle file private: it contains customer data, password hashes, and customer artwork. To verify it again, run python3 -m app.backup verify followed by the printed backups/...manifest.json path. Legacy database-only .dump backups remain verifiable. Scheduling, offsite copies, and a production restore runbook remain unfinished.

After building the current image, test retention with synthetic files:

docker compose exec -T api python3 -m tests.retention_test

This checks that expired bytes are removed while unexpired files survive. It cleans up its own synthetic object bytes and retains their metadata.

Operations and troubleshooting

docker compose logs --tail=100 api worker scanner
docker compose restart api worker
docker compose ps
docker compose down

down stops the stack and preserves named database/storage volumes. Restart with docker compose up -d --wait. Do not add --volumes unless you intend to permanently erase all local orders and artwork. No reset is required for tests.

Seven long-running services have Docker health checks; the database and storage initialization jobs must exit successfully. API health queries PostgreSQL and MinIO; worker health fails if its database-processing loop, scanning thread, or live ClamAV PING is unavailable. Both web gateways expose /health. MinIO exposes /minio/health/ready.

Run the redacted local alert summary inside the API network namespace:

docker compose exec -T api python3 -m app.security_status

Exit status 1 means attention is required. Review blocked artwork, rate limits, failed logins, scanner availability, and signature age. Immediately after the security tests, expected synthetic login failures/rate limits and SECURITY-EICAR.cdr rejections will trigger the alert; confirm names and test timing before classifying them as expected. Unexpected failures, scanner errors, stale signatures (more than seven days), or non-test rejected files require investigation. Events are retained for 30 days. See SECURITY_REPORT.md.

If a host port is occupied, change SITE_PORT, KANBAN_PORT, or API_PORT in .env and recreate the stack. MinIO currently reserves 9000/9001. If Docker says permission denied, fix Docker access or use your OS-approved Docker workflow. If image/package download fails, check development internet access. A warning about missing Buildx can fall back to Docker's classic builder; install your platform's Buildx plugin for the supported modern build path.

If API is unhealthy, inspect its logs and check local adapter settings. Database and MinIO credentials initialize persistent volumes only once; changing values later also requires updating the existing database/storage account deliberately. If an upload fails, keep the session and click checkout again. If parts expired, reselect the file to start another upload. For a stale Kanban move, refresh; the backend rejects stale versions instead of overwriting another operator.

The Site retains its existing remote visual assets (fonts/logo) and optional PDF.js 3.11.174 CDN dependency. Its known eval-based advisory is mitigated in the current call with isEvalSupported:false; the old CDN dependency is still an upgrade/vendor-pinning limitation, not a claim that it is current. The local API, manual fixture flow, storage, payment, freight, Kanban, and mock integrations work without these external services. Offline PDF previews may fall back to the prototype's manual/table-price path.

Dependency lock

The API, worker, and database initializer install every Python dependency from infra/requirements.lock with --require-hashes. infra/requirements.txt remains the human-maintained direct dependency list. After deliberately changing a direct pin, regenerate the lock in the same Python 3.12 environment and rebuild:

./infra/lock_dependencies.sh
docker compose up --build -d --wait

The generator downloads public package metadata in a disposable container and does not modify the host Python environment. Review the resolved versions and run the exact-runtime audit plus regression suite before accepting an updated lock.

Staging readiness gate

compose.staging.yaml is intentionally not a staging deployment. It runs only a network-disabled validator for non-secret decisions. After completing PRODUCTION_INPUTS.md, copy staging/staging.env.example to the ignored staging/staging.env, enter non-secret metadata, and run:

docker compose -f compose.staging.yaml run --rm readiness

A pass does not authorize deployment or prove provider access. The real staging composition and external secret injection still require approved provider contracts and owners. See staging/README.md.

Next production connection step

Keep this stack local. Before connecting real services, confirm the production checkout trust workflow, complete PRODUCTION_INPUTS.md, and pass the isolated staging-readiness gate. Then implement the actual staging composition using the adapter contracts in app/adapters.py: start with private S3 staging storage, CORS, signed multipart contract tests and scoped credentials injected outside Git. Add provider sandbox adapters one at a time. Mercado Pago requires authenticated, signed, idempotent webhook handling before real payment is allowed; Tiny/Olist and WhatsApp need confirmed contracts and provider idempotency/delivery handling. Do not simply change the local fake flags or point this Compose file at production.

Production delivery package

The repository now includes a separate Docker Swarm and Gitea Actions delivery package in deploy/ and .gitea/workflows/. It is not used by this localhost Compose stack and does not alter its volumes. Production images run as unprivileged users and install the hash-locked Python runtime. Gitea publishes latest for the normal Portainer webhook plus the full commit SHA for rollback. The stack expects pre-provisioned external Swarm secrets and an external PostgreSQL volume. Only Site and Kanban ports are published for the existing TLS reverse proxy; API, database, worker, and scanner remain private.

Run the source-only gate without credentials:

python3 deploy/production_preflight.py --source-only

It must remain blocked while app/ supports only local fake adapters and does not load Docker secret *_FILE settings. Do not bypass or delete this check. After approved production implementations and inputs exist, follow PORTAINER.md and deploy/PRODUCTION_CHECKLIST.md; release and deployment still require the Gitea scan/test gates and explicit approvals.