Architecture

This article walks through the underlying messaging and cryptographic mechanics as a document flows over the Kaleido network’s Kafka backbone into a recipient’s configured storage utility.

Document Transfer

This article walks through the underlying messaging and cryptographic mechanics as a document flows over the Kaleido network's Kafka backbone into a recipient's configured storage utility.

Send

  • Verify information:
    • Sender destination is ready and owned by the service
    • Recipient destination is ready
    • Document to be transferred exists
  • Generate random unique transfer ID
  • Calculate document hash (SHA256)
  • Sign document using sender destination private key (RSA-SHA256)
  • Compress document (GZip)
  • Generate random symmetric key
  • Encrypt compressed document using symmetric key
  • Split (compressed+encrypted) document into shards (256KB)
  • Dispatch shards via Kafka as they become available:
    • Message key: shard-< transfer ID >-< shard number (0 index) >
    • Message body: (binary) shard content
  • Build transfer metadata containing:
    • Sender destination
    • Recipient destination
    • Document Hash
    • Document name
    • Document Signature
    • Shard count
    • Transfer timestamp
  • Encrypt metadata using symmetric key
  • Obtain recipient destination x509 certificate from ID Registry
  • Obtain recipient public key from destination x509 certificate
  • Encrypt symmetric key with recipient public key
  • Dispatch message with encrypted key + encrypted metadata via Kafka:
    • Message key: data-< transfer ID >
    • Message body:
    • bytes 0 to cert key length (default 2k) / 8: encrypted symmetric key
    • Remaining bytes: encrypted transfer metadata
  • Log transfer with status "sent"
  • Send out push notification

Receive

  • Save shards to temporary storage as they arrive
  • Load encrypted metadata message into memory when it arrives
  • Obtain symmetric key:
    • decrypt bytes 0 to cert key length (default: 2k) / 8 of in-memory metadata using recipient destination private key
  • Decrypt remaining metadata bytes using symmetric key
  • Verify metadata information:
    • Recipient destination is valid and owned by the service
  • Reassemble document:
    • Read shards from temporary storage
    • Decrypt using symmetric key
    • Uncompress
    • Append to single file in temporary storage
    • Calculate hash (SHA256)
  • Verify reassembled document hash matches hash in metadata
  • Retrieve sender destination X509 certificate from ID Registry
  • Verify document signature in metadata against sender certificate and document hash
  • Move assembled document to permanent storage
  • Clean all shards and assembled file from temporary storage
  • Determine transfer status: "received" if successful or "failed" if errors were encountered
  • Sign transfer ID using recipient destination private key
  • Build acknowledgement metadata containing:
    • Transfer timestamp (from metadata)
    • Transfer ID signature
    • Transfer status
  • Generate new random symmetric key
  • Encrypt acknowledgement using new symmetric key
  • Encrypt new symmetric key using sender destination certificate (obtained in step 8)
  • Dispatch acknowledgement via Kafka:
    • Message key: ack-< transfer ID >
    • Message body::
    • bytes 0 to cert key length (default 2k) / 8: encrypted new symmetric key
    • Remaining bytes: encrypted acknowledgement metadata
  • Log transfer with status determined in step 12
  • Send out push notification

Acknowledgement

  • Obtain symmetric key:
    • decrypt bytes 0 to cert key length (default: 2k) / 8 of in-memory metadata using sender destination private key
  • Decrypt remaining bytes using symmetric key
  • Update transfer log with status from acknowledgement metadata
  • Send out push notifications