Files
cloudflare_temp_email/worker/src/admin_api/db_api.ts
Dream Hunter 7c6d0d7c8a feat(mail): support gzip compressed email storage via ENABLE_MAIL_GZIP (#933)
* feat(mail): support gzip compressed email storage in D1 raw_blob column

Add ENABLE_MAIL_GZIP env var to optionally gzip-compress incoming emails
into a new raw_blob BLOB column, saving D1 storage space. Reading is
backward-compatible: prioritizes raw_blob (decompress) with fallback to
plaintext raw field. Includes DB migration v0.0.7, docs, and changelogs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: gzip fallback on missing column + decouple resolve from handleListQuery

- email/index.ts: gzip INSERT failure now falls back to plaintext INSERT
  instead of silently losing the email (P1: data loss prevention)
- common.ts: add handleMailListQuery for raw_mails-specific list queries
  with resolveRawEmailList, keeping handleListQuery generic
- Replace handleListQuery → handleMailListQuery in mails_api, admin_mail_api,
  user_mail_api (only raw_mails callers)
- Add e2e test infrastructure: worker-gzip service, wrangler.toml.e2e.gzip,
  api-gzip playwright project, mail-gzip.spec.ts with 4 test cases

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address CodeRabbit review feedback for gzip feature

- Use destructuring in resolveRawEmailRow to truly remove raw_blob key
- Narrow fallback scope: only fallback to plaintext on compression failure
  or missing raw_blob column, re-throw other DB errors
- Clean unused imports in e2e gzip test

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add try-catch in resolveRawEmail to prevent single corrupt blob from failing entire list

A corrupted raw_blob would cause decompressBlob to throw, which with
Promise.all in resolveRawEmailList would reject the entire batch query.
Now catches decompression errors and falls back to row.raw field.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(mail): align sendAdminInternalMail with gzip storage path

sendAdminInternalMail now respects ENABLE_MAIL_GZIP: compresses to
raw_blob when enabled, with fallback to plaintext on failure.
Added e2e test verifying admin internal mail is readable under gzip.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(e2e): match admin internal mail by body content instead of encoded subject

mimetext base64-encodes the Subject header, so the raw MIME string
does not contain the literal subject text. Match on body content
(balance: 99) which is plaintext.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(e2e): add WORKER_GZIP_URL guard and length assertions in gzip tests

Address CodeRabbit feedback:
- Skip gzip tests when WORKER_GZIP_URL is not set to prevent false positives
- Assert results array length before accessing [0] for clearer error messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(mail): narrow gzip fallback scope and fix webhook query compatibility

- sendAdminInternalMail: separate compress vs DB error handling, only
  fallback to plaintext on compression failure or missing raw_blob
  column, rethrow other DB errors (aligns with email/index.ts)
- Webhook test endpoints: use SELECT * instead of explicit raw_blob
  column reference, so pre-migration databases don't 500
- Docs/changelog: clarify that db_migration must run before enabling
  ENABLE_MAIL_GZIP

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(telegram): use generic Record type for raw_mails query result

Align with other query sites — avoid hardcoding raw_blob in the
TypeScript type annotation so the query works with or without the
column after migration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(models): add RawMailRow type and unify raw_mails query typing

Add RawMailRow type to models with raw_blob as optional field, replacing
ad-hoc Record<string, unknown> and inline type annotations across
webhook test endpoints, telegram API, and gzip utilities.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 18:46:39 +08:00

230 lines
7.9 KiB
TypeScript

import { Context } from "hono";
import { CONSTANTS } from "../constants";
import utils from "../utils";
const DB_INIT_QUERIES = `
CREATE TABLE IF NOT EXISTS raw_mails (
id INTEGER PRIMARY KEY,
message_id TEXT,
source TEXT,
address TEXT,
raw TEXT,
raw_blob BLOB,
metadata TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_raw_mails_address ON raw_mails(address);
CREATE INDEX IF NOT EXISTS idx_raw_mails_created_at ON raw_mails(created_at);
CREATE INDEX IF NOT EXISTS idx_raw_mails_message_id ON raw_mails(message_id);
CREATE TABLE IF NOT EXISTS address (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT UNIQUE,
password TEXT,
source_meta TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_address_name ON address(name);
CREATE INDEX IF NOT EXISTS idx_address_created_at ON address(created_at);
CREATE INDEX IF NOT EXISTS idx_address_updated_at ON address(updated_at);
CREATE INDEX IF NOT EXISTS idx_address_source_meta ON address(source_meta);
CREATE TABLE IF NOT EXISTS auto_reply_mails (
id INTEGER PRIMARY KEY,
source_prefix TEXT,
name TEXT,
address TEXT UNIQUE,
subject TEXT,
message TEXT,
enabled INTEGER DEFAULT 1,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_auto_reply_mails_address ON auto_reply_mails(address);
CREATE TABLE IF NOT EXISTS address_sender (
id INTEGER PRIMARY KEY,
address TEXT UNIQUE,
balance INTEGER DEFAULT 0,
enabled INTEGER DEFAULT 1,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_address_sender_address ON address_sender(address);
CREATE TABLE IF NOT EXISTS sendbox (
id INTEGER PRIMARY KEY,
address TEXT,
raw TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_sendbox_address ON sendbox(address);
CREATE INDEX IF NOT EXISTS idx_sendbox_created_at ON sendbox(created_at);
CREATE TABLE IF NOT EXISTS settings (
key TEXT PRIMARY KEY,
value TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY,
user_email TEXT UNIQUE NOT NULL,
password TEXT NOT NULL,
user_info TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_users_user_email ON users(user_email);
CREATE TABLE IF NOT EXISTS users_address (
id INTEGER PRIMARY KEY,
user_id INTEGER,
address_id INTEGER UNIQUE,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_users_address_user_id ON users_address(user_id);
CREATE INDEX IF NOT EXISTS idx_users_address_address_id ON users_address(address_id);
CREATE TABLE IF NOT EXISTS user_roles (
id INTEGER PRIMARY KEY,
user_id INTEGER UNIQUE NOT NULL,
role_text TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_user_roles_user_id ON user_roles(user_id);
CREATE TABLE IF NOT EXISTS user_passkeys (
id INTEGER PRIMARY KEY,
user_id INTEGER NOT NULL,
passkey_name TEXT NOT NULL,
passkey_id TEXT NOT NULL,
passkey TEXT NOT NULL,
counter INTEGER DEFAULT 0,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_user_passkeys_user_id ON user_passkeys(user_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_passkeys_user_id_passkey_id ON user_passkeys(user_id, passkey_id);
`
export default {
initialize: async (c: Context<HonoCustomType>) => {
// remove all \r and \n characters from the query string
// split by ; and join with a ;\n
const query = DB_INIT_QUERIES.replace(/[\r\n]/g, "")
.split(";")
.map((query) => query.trim())
.join(";\n");
await c.env.DB.exec(query);
const version = await utils.getSetting(c, CONSTANTS.DB_VERSION_KEY);
if (version) {
return c.json({ message: "Database already initialized" });
}
await utils.saveSetting(c, CONSTANTS.DB_VERSION_KEY, CONSTANTS.DB_VERSION);
return c.json({ message: "Database initialized" });
},
migrate: async (c: Context<HonoCustomType>) => {
const version = await utils.getSetting(c, CONSTANTS.DB_VERSION_KEY);
if (version && version <= "v0.0.2") {
// migration to v0.0.3: add password column
const tableInfo = await c.env.DB.prepare(
`PRAGMA table_info(address)`
).all();
const hasPassword = tableInfo.results?.some(
(col: any) => col.name === 'password'
);
if (!hasPassword) {
await c.env.DB.exec(`ALTER TABLE address ADD COLUMN password TEXT;`);
}
}
if (version && version <= "v0.0.3") {
// migration to v0.0.4: add metadata column
const tableInfo = await c.env.DB.prepare(
`PRAGMA table_info(raw_mails)`
).all();
const hasMetadata = tableInfo.results?.some(
(col: any) => col.name === 'metadata'
);
if (!hasMetadata) {
await c.env.DB.exec(`ALTER TABLE raw_mails ADD COLUMN metadata TEXT;`);
}
}
if (version && version <= "v0.0.4") {
// migration to v0.0.5: add source_meta column
const tableInfo = await c.env.DB.prepare(
`PRAGMA table_info(address)`
).all();
const hasSourceMeta = tableInfo.results?.some(
(col: any) => col.name === 'source_meta'
);
if (!hasSourceMeta) {
await c.env.DB.exec(`ALTER TABLE address ADD COLUMN source_meta TEXT;`);
await c.env.DB.exec(`CREATE INDEX IF NOT EXISTS idx_address_source_meta ON address(source_meta);`);
}
}
if (version && version <= "v0.0.5") {
// migration to v0.0.6: add message_id index on raw_mails
await c.env.DB.exec(`CREATE INDEX IF NOT EXISTS idx_raw_mails_message_id ON raw_mails(message_id);`);
}
if (version && version <= "v0.0.6") {
// migration to v0.0.7: add raw_blob column for gzip compressed email storage
const tableInfo = await c.env.DB.prepare(
`PRAGMA table_info(raw_mails)`
).all();
const hasRawBlob = tableInfo.results?.some(
(col: any) => col.name === 'raw_blob'
);
if (!hasRawBlob) {
await c.env.DB.exec(`ALTER TABLE raw_mails ADD COLUMN raw_blob BLOB;`);
}
}
if (version != CONSTANTS.DB_VERSION) {
// remove all \r and \n characters from the query string
// split by ; and join with a ;\n
const query = DB_INIT_QUERIES.replace(/[\r\n]/g, "")
.split(";")
.map((query) => query.trim())
.join(";\n");
await c.env.DB.exec(query);
// Update the version in the settings table
await utils.saveSetting(c, CONSTANTS.DB_VERSION_KEY, CONSTANTS.DB_VERSION);
return c.json({
success: true,
message: "Database migrated"
});
}
return c.json({
success: true,
message: "Database does not need migration"
});
},
getVersion: async (c: Context<HonoCustomType>) => {
const version = await utils.getSetting(c, CONSTANTS.DB_VERSION_KEY);
return c.json({
need_initialization: !version,
need_migration: version && version != CONSTANTS.DB_VERSION,
current_db_version: version,
code_db_version: CONSTANTS.DB_VERSION
});
},
}