feat: accept sheets of up to 5 GB end to end
Some checks failed
Build and deploy / Validate source (push) Successful in 12s
Build and deploy / Integration suite on a real stack (push) Failing after 2m36s
Build and deploy / Secret scan and release gate (push) Successful in 7s
Build and deploy / Publish images (push) Has been skipped
Some checks failed
Build and deploy / Validate source (push) Successful in 12s
Build and deploy / Integration suite on a real stack (push) Failing after 2m36s
Build and deploy / Secret scan and release gate (push) Successful in 7s
Build and deploy / Publish images (push) Has been skipped
Sheets of several GB are the normal order. The upload limit is now 5 GB. ClamAV scans files up to 2 GB; a larger file is released only when its first bytes match the format its name claims, and a disguised file is refused. The Site grades a sheet over 150 MB from the pixel size in its PNG, JPEG or WebP header without decoding it, and reads large PDFs in ranges. The worker never opens a source over 300 MB: a finished sheet placed whole becomes its own print file, which the Kanban offers to approve as the final, and anything else goes to hand preparation. Files start uploading as they enter the cart, with progress in the summary, and each part renews the reservation so slow uploads do not expire. Quotas grow to 50 GB per customer and 500 GB in total; the Swarm config for ClamAV is renamed because a deployed config cannot change in place. Verified locally with a 386 MB and a 1.8 GB PNG (scanned, paid, original as print file), a 2.3 GB PNG (format check) and a disguised 2.3 GB file (refused). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,4 +1,13 @@
|
||||
"""Local ClamAV boundary. Unknown/error/over-limit results NEVER release artwork."""
|
||||
"""Releasing artwork: ClamAV up to its size limit, a format check above it.
|
||||
|
||||
Unknown or error results NEVER release artwork. ClamAV scans files up to
|
||||
scan_limit_bytes() (2 GB). Sheets of several GB are the normal order and
|
||||
ClamAV cannot take them, so a larger file is released only if its first bytes
|
||||
are those of the format its name claims (a PNG that really is a PNG, not a
|
||||
program renamed .png). That is the check the client chose for large files; it
|
||||
does not look for malware inside a valid file.
|
||||
"""
|
||||
import re
|
||||
import socket
|
||||
import struct
|
||||
import time
|
||||
@@ -30,10 +39,9 @@ class ClamAV:
|
||||
return self.command(b'VERSION').decode('utf-8','replace')
|
||||
|
||||
def scan(self, stream, size):
|
||||
if size > scan_limit_bytes():
|
||||
return 'rejected', 'File exceeds the malware scan limit'
|
||||
with socket.create_connection(('scanner',3310),timeout=10) as sock:
|
||||
sock.settimeout(150)
|
||||
# A 2 GB file takes minutes to stream and scan.
|
||||
sock.settimeout(900)
|
||||
sock.sendall(b'zINSTREAM\0')
|
||||
sent=0
|
||||
for chunk in stream.iter_chunks(chunk_size=65536):
|
||||
@@ -52,6 +60,37 @@ class ClamAV:
|
||||
if result.endswith(b' FOUND'):return 'rejected','Malware or unsafe scan condition detected'
|
||||
return 'error','Scanner could not verify this file'
|
||||
|
||||
# What each accepted extension must start with. AI files are PDF or PostScript;
|
||||
# CDR and WebP are RIFF containers with their own form type.
|
||||
TIFF = (b'II*\x00', b'MM\x00*', b'II+\x00', b'MM\x00+')
|
||||
SIGNATURES = {
|
||||
'png': (b'\x89PNG\r\n\x1a\n',),
|
||||
'jpg': (b'\xff\xd8\xff',), 'jpeg': (b'\xff\xd8\xff',),
|
||||
'tif': TIFF, 'tiff': TIFF,
|
||||
'pdf': (b'%PDF-',), 'ai': (b'%PDF-', b'%!PS'),
|
||||
'psd': (b'8BPS',), 'psb': (b'8BPS',),
|
||||
}
|
||||
RIFF_FORMS = {'cdr': re.compile(rb'^RIFF....CDR', re.S), 'webp': re.compile(rb'^RIFF....WEBP', re.S)}
|
||||
HEAD_BYTES = 64
|
||||
|
||||
|
||||
def format_matches(name, head):
|
||||
"""Whether a file's first bytes are those of the format its name claims."""
|
||||
ext = name.rsplit('.', 1)[-1].lower() if '.' in name else ''
|
||||
if ext in RIFF_FORMS:
|
||||
return bool(RIFF_FORMS[ext].match(head))
|
||||
return any(head.startswith(sig) for sig in SIGNATURES.get(ext, ()))
|
||||
|
||||
|
||||
def check_large(storage, row):
|
||||
"""Release decision for a file above the antivirus limit."""
|
||||
head = storage.client.get_object(Bucket=storage.bucket, Key=row['object_key'],
|
||||
Range=f'bytes=0-{HEAD_BYTES - 1}')['Body'].read()
|
||||
if format_matches(row['name'], head):
|
||||
return 'clean', 'Acima do limite do antivírus; formato do arquivo conferido'
|
||||
return 'rejected', 'O conteúdo do arquivo não corresponde ao formato do nome'
|
||||
|
||||
|
||||
def scan_one(storage, scanner=None):
|
||||
scanner=scanner or ClamAV()
|
||||
with connect() as c:
|
||||
@@ -60,9 +99,12 @@ def scan_one(storage, scanner=None):
|
||||
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1''').fetchone()
|
||||
if not row:return False
|
||||
try:
|
||||
stream=storage.client.get_object(Bucket=storage.bucket,Key=row['object_key'])['Body']
|
||||
try:state,reason=scanner.scan(stream,row['size'])
|
||||
finally:stream.close()
|
||||
if row['size'] > scan_limit_bytes():
|
||||
state,reason=check_large(storage,row)
|
||||
else:
|
||||
stream=storage.client.get_object(Bucket=storage.bucket,Key=row['object_key'])['Body']
|
||||
try:state,reason=scanner.scan(stream,row['size'])
|
||||
finally:stream.close()
|
||||
except Exception:
|
||||
state,reason='error','Malware scanner unavailable; file remains blocked'
|
||||
c.execute("""UPDATE dtf_local.uploads SET scan_state=%s,scan_reason=%s,scanned_at=now(),
|
||||
|
||||
Reference in New Issue
Block a user