Many managed key-value and NoSQL databases - such as Amazon DynamoDB, Azure Cosmos DB, and Google Cloud Firestore - enforce strict maximum item sizes (e.g., 400 KB in DynamoDB). This constraint imposes significant architectural challenges for applications requiring low-latency, multi-region access to objects that exceed these limits. The standard industry recommendation is to offload payloads to object storage (e.g., Amazon S3) while retaining a pointer in the database. While cost-efficient, this "pointer pattern" introduces network overhead and exposes applications to non-deterministic replication lag between the database and the object store, creating race conditions in active-active architectures. This paper presents a "chunked-object" pattern that persists large logical entities as sets of ordered chunks within the database itself. We precisely define the pattern and provide a reference implementation using Amazon DynamoDB Global Tables. The design generalizes to any key-value store with per-item size limits and multi-region replication. We evaluate the approach using telemetry from a production system processing over 200,000 transactions per hour. Results demonstrate that the chunked-object pattern eliminates cross-system replication lag hazards and reduces p99 cross-region time-to-consistency for 1 MB payloads by keeping data and metadata within a single consistency domain.
翻译:许多托管键值对和NoSQL数据库——如Amazon DynamoDB、Azure Cosmos DB和Google Cloud Firestore——强制执行严格的最大条目大小限制(例如DynamoDB中的400 KB)。这一约束对需要低延迟、多区域访问超出此类限制对象的应用程序带来了显著的架构挑战。行业标准建议是将负载卸载至对象存储(如Amazon S3),同时在数据库中保留指针。尽管这种'指针模式'具有成本效益,但它引入了网络开销,并使应用程序暴露于数据库与对象存储之间非确定性的复制延迟,在主动-主动架构中引发竞态条件。本文提出一种'分块对象'模式,将大型逻辑实体作为有序块集合持久化在数据库内部。我们精确定义了该模式,并提供了使用Amazon DynamoDB全局表的参考实现。该设计可推广至任何具有单条目大小限制和多区域复制功能的键值存储系统。我们通过处理每小时超过20万笔交易的生产系统遥测数据对该方法进行评估。结果表明:分块对象模式通过将数据和元数据保持在单一一致性域内,消除了跨系统复制延迟风险,并将1 MB负载的p99跨区域一致达成时间降低了。