Secure Document Storage
Some off-chain data needs to be stored for a period of time, or indefinitely, rather than just passing in-flight between two applications. The individual blobs of data can be large – megabytes or even gigabytes. Documents, images, tabular data, etc.
In many cases on-chain transactions need to be able to reference these documents, even though the data itself is either too large, or too sensitive, to be placed on the chain. This is where a hash can be used, to reference the overall document uniquely without needing to have a complete copy of it. Hashing algorithms such as SHA256 are a proven way to summarize an arbitrarily large package of data, into a fixed length set of bytes with an extremely low likelihood that two documents will generate the same hash (collision resistance), and with confidence that even a single bit change to the document will result in an entirely different hash. In many systems the hash is combined with other information about how to access the information, such as the technology used to store it, in a Context Identifier (CID) or a URI.
There are two approaches to hash-based document storage commonly used in decentralized applications.
Distributed file store
Applying the same p2p network connectivity approaches used by the blockchain itself to distribute transactions between nodes, distributed file stores such as IPFS split down files into small distributable chunks and distribute them for resiliency across the network – a process known as sharding. The whole original document can also be referred to by its hash/CID, and any member of the business network with their own file store node can request a complete copy of the file to be re-assembled in their local node for download. The concept is very similar to distributed file sharing technologies on the internet (like BitTorrent).
Because the data is distributed across all nodes in the permissioned network, and any participant can download a copy of the original document, it is common for payloads to be encrypted prior to being shared on the distributed file store. With strong symmetric encryption and decryption keys shared via secure app-to-app messaging, you can ensure that only those parties who should have access to the data can decrypt the shared contents.
Private document store and forward
Strong controls often exist on the storage, distribution, and deletion of documents. Often in these cases it is most efficient, and easy to prove regulatory compliance, if documents are only distributed to those parties that are allowed access to the data.
Requirements for such a document store include:
- Reliable and elastic storage of documents
- APIs for managing the lifecycle of the documents
- Ability to refer to the documents from on-chain logic using standard hash/URI semantics
- Secure forwarding capability to individual permissioned parties in the business network
The Kaleido Document Store couples reliable storage, with the end-to-end encrypted messaging of our App 2 App Messaging service. You can let Kaleido manage the storage for you, or use your own off-platform storage to back the document store – in your own AWS S3 or Azure Blob storage account.