mirror of
https://github.com/dreamhunter2333/cloudflare_temp_email.git
synced 2026-07-31 02:38:08 +08:00
feat(mail): support gzip compressed email storage via ENABLE_MAIL_GZIP (#933)
* feat(mail): support gzip compressed email storage in D1 raw_blob column Add ENABLE_MAIL_GZIP env var to optionally gzip-compress incoming emails into a new raw_blob BLOB column, saving D1 storage space. Reading is backward-compatible: prioritizes raw_blob (decompress) with fallback to plaintext raw field. Includes DB migration v0.0.7, docs, and changelogs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: gzip fallback on missing column + decouple resolve from handleListQuery - email/index.ts: gzip INSERT failure now falls back to plaintext INSERT instead of silently losing the email (P1: data loss prevention) - common.ts: add handleMailListQuery for raw_mails-specific list queries with resolveRawEmailList, keeping handleListQuery generic - Replace handleListQuery → handleMailListQuery in mails_api, admin_mail_api, user_mail_api (only raw_mails callers) - Add e2e test infrastructure: worker-gzip service, wrangler.toml.e2e.gzip, api-gzip playwright project, mail-gzip.spec.ts with 4 test cases Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address CodeRabbit review feedback for gzip feature - Use destructuring in resolveRawEmailRow to truly remove raw_blob key - Narrow fallback scope: only fallback to plaintext on compression failure or missing raw_blob column, re-throw other DB errors - Clean unused imports in e2e gzip test Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add try-catch in resolveRawEmail to prevent single corrupt blob from failing entire list A corrupted raw_blob would cause decompressBlob to throw, which with Promise.all in resolveRawEmailList would reject the entire batch query. Now catches decompression errors and falls back to row.raw field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(mail): align sendAdminInternalMail with gzip storage path sendAdminInternalMail now respects ENABLE_MAIL_GZIP: compresses to raw_blob when enabled, with fallback to plaintext on failure. Added e2e test verifying admin internal mail is readable under gzip. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(e2e): match admin internal mail by body content instead of encoded subject mimetext base64-encodes the Subject header, so the raw MIME string does not contain the literal subject text. Match on body content (balance: 99) which is plaintext. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(e2e): add WORKER_GZIP_URL guard and length assertions in gzip tests Address CodeRabbit feedback: - Skip gzip tests when WORKER_GZIP_URL is not set to prevent false positives - Assert results array length before accessing [0] for clearer error messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(mail): narrow gzip fallback scope and fix webhook query compatibility - sendAdminInternalMail: separate compress vs DB error handling, only fallback to plaintext on compression failure or missing raw_blob column, rethrow other DB errors (aligns with email/index.ts) - Webhook test endpoints: use SELECT * instead of explicit raw_blob column reference, so pre-migration databases don't 500 - Docs/changelog: clarify that db_migration must run before enabling ENABLE_MAIL_GZIP Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(telegram): use generic Record type for raw_mails query result Align with other query sites — avoid hardcoding raw_blob in the TypeScript type annotation so the query works with or without the column after migration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(models): add RawMailRow type and unify raw_mails query typing Add RawMailRow type to models with raw_blob as optional field, replacing ad-hoc Record<string, unknown> and inline type annotations across webhook test endpoints, telegram API, and gzip utilities. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Dream Hunter
@@ -1,6 +1,6 @@
|
||||
import { Context } from "hono";
|
||||
|
||||
import { getJsonSetting } from "../utils";
|
||||
import { getBooleanValue, getJsonSetting } from "../utils";
|
||||
import { sendMailToTelegram } from "../telegram_api";
|
||||
import { auto_reply } from "./auto_reply";
|
||||
import { isBlocked } from "./black_list";
|
||||
@@ -11,6 +11,7 @@ import { extractEmailInfo } from "./ai_extract";
|
||||
import { forwardEmail } from "./forward";
|
||||
import { EmailRuleSettings } from "../models";
|
||||
import { CONSTANTS } from "../constants";
|
||||
import { compressText } from "../gzip";
|
||||
|
||||
|
||||
async function email(message: ForwardableEmailMessage, env: Bindings, ctx: ExecutionContext) {
|
||||
@@ -65,11 +66,49 @@ async function email(message: ForwardableEmailMessage, env: Bindings, ctx: Execu
|
||||
const message_id = message.headers.get("Message-ID");
|
||||
// save email
|
||||
try {
|
||||
const { success } = await env.DB.prepare(
|
||||
`INSERT INTO raw_mails (source, address, raw, message_id) VALUES (?, ?, ?, ?)`
|
||||
).bind(
|
||||
message.from, message.to, parsedEmailContext.rawEmail, message_id
|
||||
).run();
|
||||
let success = false;
|
||||
if (getBooleanValue(env.ENABLE_MAIL_GZIP)) {
|
||||
let compressed: ArrayBuffer | null = null;
|
||||
try {
|
||||
compressed = await compressText(parsedEmailContext.rawEmail);
|
||||
} catch (gzipError) {
|
||||
console.error("gzip compression failed, falling back to plaintext", gzipError);
|
||||
}
|
||||
if (compressed) {
|
||||
try {
|
||||
({ success } = await env.DB.prepare(
|
||||
`INSERT INTO raw_mails (source, address, raw_blob, message_id) VALUES (?, ?, ?, ?)`
|
||||
).bind(
|
||||
message.from, message.to, compressed, message_id
|
||||
).run());
|
||||
} catch (dbError) {
|
||||
// Fallback to plaintext only if raw_blob column is missing (migration not applied)
|
||||
const errMsg = String(dbError);
|
||||
if (errMsg.includes('raw_blob') || errMsg.includes('no such column')) {
|
||||
console.error("raw_blob column missing, falling back to plaintext", dbError);
|
||||
({ success } = await env.DB.prepare(
|
||||
`INSERT INTO raw_mails (source, address, raw, message_id) VALUES (?, ?, ?, ?)`
|
||||
).bind(
|
||||
message.from, message.to, parsedEmailContext.rawEmail, message_id
|
||||
).run());
|
||||
} else {
|
||||
throw dbError;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
({ success } = await env.DB.prepare(
|
||||
`INSERT INTO raw_mails (source, address, raw, message_id) VALUES (?, ?, ?, ?)`
|
||||
).bind(
|
||||
message.from, message.to, parsedEmailContext.rawEmail, message_id
|
||||
).run());
|
||||
}
|
||||
} else {
|
||||
({ success } = await env.DB.prepare(
|
||||
`INSERT INTO raw_mails (source, address, raw, message_id) VALUES (?, ?, ?, ?)`
|
||||
).bind(
|
||||
message.from, message.to, parsedEmailContext.rawEmail, message_id
|
||||
).run());
|
||||
}
|
||||
if (!success) {
|
||||
message.setReject(`Failed save message to ${message.to}`);
|
||||
console.error(`Failed save message from ${message.from} to ${message.to}`);
|
||||
|
||||
Reference in New Issue
Block a user