feat: accept sheets of up to 5 GB end to end
Some checks failed
Build and deploy / Validate source (push) Successful in 12s
Build and deploy / Integration suite on a real stack (push) Failing after 2m36s
Build and deploy / Secret scan and release gate (push) Successful in 7s
Build and deploy / Publish images (push) Has been skipped

Sheets of several GB are the normal order. The upload limit is now 5 GB.
ClamAV scans files up to 2 GB; a larger file is released only when its
first bytes match the format its name claims, and a disguised file is
refused. The Site grades a sheet over 150 MB from the pixel size in its
PNG, JPEG or WebP header without decoding it, and reads large PDFs in
ranges. The worker never opens a source over 300 MB: a finished sheet
placed whole becomes its own print file, which the Kanban offers to approve
as the final, and anything else goes to hand preparation. Files start
uploading as they enter the cart, with progress in the summary, and each
part renews the reservation so slow uploads do not expire. Quotas grow to
50 GB per customer and 500 GB in total; the Swarm config for ClamAV is
renamed because a deployed config cannot change in place.

Verified locally with a 386 MB and a 1.8 GB PNG (scanned, paid, original
as print file), a 2.3 GB PNG (format check) and a disguised 2.3 GB file
(refused).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cauê Faleiros
2026-09-29 13:18:18 -03:00
parent 4437232d27
commit 5f2be7ea20
21 changed files with 329 additions and 54 deletions

View File

@@ -1,4 +1,13 @@
"""Local ClamAV boundary. Unknown/error/over-limit results NEVER release artwork."""
"""Releasing artwork: ClamAV up to its size limit, a format check above it.
Unknown or error results NEVER release artwork. ClamAV scans files up to
scan_limit_bytes() (2 GB). Sheets of several GB are the normal order and
ClamAV cannot take them, so a larger file is released only if its first bytes
are those of the format its name claims (a PNG that really is a PNG, not a
program renamed .png). That is the check the client chose for large files; it
does not look for malware inside a valid file.
"""
import re
import socket
import struct
import time
@@ -30,10 +39,9 @@ class ClamAV:
return self.command(b'VERSION').decode('utf-8','replace')
def scan(self, stream, size):
if size > scan_limit_bytes():
return 'rejected', 'File exceeds the malware scan limit'
with socket.create_connection(('scanner',3310),timeout=10) as sock:
sock.settimeout(150)
# A 2 GB file takes minutes to stream and scan.
sock.settimeout(900)
sock.sendall(b'zINSTREAM\0')
sent=0
for chunk in stream.iter_chunks(chunk_size=65536):
@@ -52,6 +60,37 @@ class ClamAV:
if result.endswith(b' FOUND'):return 'rejected','Malware or unsafe scan condition detected'
return 'error','Scanner could not verify this file'
# What each accepted extension must start with. AI files are PDF or PostScript;
# CDR and WebP are RIFF containers with their own form type.
TIFF = (b'II*\x00', b'MM\x00*', b'II+\x00', b'MM\x00+')
SIGNATURES = {
'png': (b'\x89PNG\r\n\x1a\n',),
'jpg': (b'\xff\xd8\xff',), 'jpeg': (b'\xff\xd8\xff',),
'tif': TIFF, 'tiff': TIFF,
'pdf': (b'%PDF-',), 'ai': (b'%PDF-', b'%!PS'),
'psd': (b'8BPS',), 'psb': (b'8BPS',),
}
RIFF_FORMS = {'cdr': re.compile(rb'^RIFF....CDR', re.S), 'webp': re.compile(rb'^RIFF....WEBP', re.S)}
HEAD_BYTES = 64
def format_matches(name, head):
"""Whether a file's first bytes are those of the format its name claims."""
ext = name.rsplit('.', 1)[-1].lower() if '.' in name else ''
if ext in RIFF_FORMS:
return bool(RIFF_FORMS[ext].match(head))
return any(head.startswith(sig) for sig in SIGNATURES.get(ext, ()))
def check_large(storage, row):
"""Release decision for a file above the antivirus limit."""
head = storage.client.get_object(Bucket=storage.bucket, Key=row['object_key'],
Range=f'bytes=0-{HEAD_BYTES - 1}')['Body'].read()
if format_matches(row['name'], head):
return 'clean', 'Acima do limite do antivírus; formato do arquivo conferido'
return 'rejected', 'O conteúdo do arquivo não corresponde ao formato do nome'
def scan_one(storage, scanner=None):
scanner=scanner or ClamAV()
with connect() as c:
@@ -60,9 +99,12 @@ def scan_one(storage, scanner=None):
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1''').fetchone()
if not row:return False
try:
stream=storage.client.get_object(Bucket=storage.bucket,Key=row['object_key'])['Body']
try:state,reason=scanner.scan(stream,row['size'])
finally:stream.close()
if row['size'] > scan_limit_bytes():
state,reason=check_large(storage,row)
else:
stream=storage.client.get_object(Bucket=storage.bucket,Key=row['object_key'])['Body']
try:state,reason=scanner.scan(stream,row['size'])
finally:stream.close()
except Exception:
state,reason='error','Malware scanner unavailable; file remains blocked'
c.execute("""UPDATE dtf_local.uploads SET scan_state=%s,scan_reason=%s,scanned_at=now(),