Off-Chain Private Messaging

Much of the data that needs to be exchanged to coordinate a business-to-business (B2B) transaction is simply too sensitive, too high volume, or too large to put on-chain. In enterprise use cases, it is common to use dozens of off-chain communications exchanges for each transaction that touches the chain.

Physical networking, identity and permissioning are just as critical to these off-chain exchanges as the on-chain ones. The exchanges need to be secure, reliable, scalable, and coupled to the business network.

It is a decentralized problem, even if it's not a blockchain problem... but it's not a new problem.

Queue-based messaging came into existence more than 25 years ago to allow systems maintained and operated by different teams/enterprises, to be loosely coupled together with reliable delivery. Almost every core transactional system that the world depends on today has these reliable delivery asynchronous messaging tiers at their core. The technology has evolved a lot into the modern event-bus implementations in microservice architectures, and the latest generation of the technology such as Apache Kafka is built for resiliency at massive scale.

The key difference between this asynchronous messaging and web services (like SOAP and REST), is that the two applications do not need to depend directly on each others' availability to transact. Just like the event-driven programming paradigm of blockchain smart contracts, the application that generates an event or message can carry on doing other work before the message has been processed by the other application. The transport is reliable and ensures the message is received by the application. If one or more responses are sent, they can be correlated with the request using details included in the reply. If the target application falls behind temporarily in its processing, no problem. The messages are buffered/queued in the middle until the application is ready to process them.

In modern applications the reliability requirement is at-least-once delivery. The transport ensures that the messages are always delivered, and if a termination occurs in the target application before it completed processing of the message, it will deliver the message again. You might also have heard of exactly-once delivery, which is a system using two-phase transactions (coordinated usually via the XA protocol) to atomically coordinate source and target state/database updates with the delivery of messages. This XA transaction approach has lost favor in recent years. Instead there is strong adoption of idempotent interfaces, where the receiver uses some kind of unique key in the message to determine if it is a duplicate, and discard it if it has already been processed. At-least-once delivery combined with idempotent processing is a vastly simpler approach to implement in modern microservice application stacks.

Kaleido provides an application to application messaging system built into the platform:

• Best of breed Apache Kafka for the reliable transport layer, with at-least-once-delivery
• End-to-end encryption of payloads using industry standard PKCS#7 payload envelopes
• Address book stored in on-chain registry, with identities pinned to existing PKI infrastructure
• Developer friendly messaging API, using modern socket.io/Websockets connectivity over HTTPS