AWS CloudHSM

AWS CloudHSM provides hardware security modules (HSM) that are maintained in the AWS Cloud. As with any HSM, you can use the AWS CloudHSM to generate and store keys and also perform an array of operations including importing/exporting keys, encrytion/decrytion, signing, calculating message digests etc. In this document we will explain in detail about AWS CloudHSM support by Kaleido CloudHSM service and how you can use it to generate an ethereum signing key, and use it to sign and submit transactions.

To give an overview of the architecture, the Kaleido CloudHSM service communicates with the AWS CloudHSM to get a transaction signed by a signing account managed by the AWS CloudHSM. Under no circumstances would Kaleido be able to retrive the keys from the AWS Cloud, but instead uses the respective key handle and send the payload to be signed to the cloud, which gets signed and returned.

AWS CloudHSM Cluster

AWS offers clustered HSMs as a service which is FIPS 140-2 Level 3 certified. The cluster is provisioned under your AWS account and will manage your signing keys inside the secure HSM devices.

AWS CloudHSM Client

The communication between the Kaleido CloudHSM service and AWS CloudHSM cluster is established with the help of a component called AWS CloudHSM Client which runs locally in the Kaleido CloudHSM service. This is responsible for maintaining secure end-to-end connection with the AWS CloudHSM(s). More information on the client daemon can be found at https://docs.aws.amazon.com/cloudhsm/latest/userguide/client-tools-and-libraries.html.

PKCS #11 Interface

AWS provides a software library implementing the "Cryptoki" API compliant with the PKCS #11 specification. The library is supported on Linux compatible OS's. This PKCS #11 interface is used by the Kaleido CloudHSM service to communicate with the CloudHSMs in AWS Cloud.

Create AWS CloudHSM Cluster

To create an AWS CloudHSM cluster, follow the steps in https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html. Start with setting up a Virtual Private Cloud (VPC), private subnet and the cluster as mentioned in the Getting Started guide. The limits supported by AWS CloudHSM are mentioned in https://docs.aws.amazon.com/cloudhsm/latest/userguide/limits.html.

When initializing the cluster, you will be provided the opportunity to establish its identity that can be verified when your application communicates to it. This is done by you signing a Certificate Signing Request from the cluster and issuing a certificate for it to use. You will need to create a self-signed certificate as described in the instructions at https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html as the signing certificate. Make sure to save the signing certificate customerCA.crt, as it will be used to configure the Kaleido CloudHSM service to verify server certificates when connecting to your AWS CloudHSM.

Create Crypto User and secp256k1 Keys

Once the HSM is set-up use the EC2 client that you created in the previous step to create the Crypto User and secp256k1 signing keys to use by the blockchain application. The following section contains the commands for a Ubuntu 16.04 LTS client. Equivalent commands for other OS can be found in https://docs.aws.amazon.com/cloudhsm/latest/userguide/command-line-tools.html.

Create Crypto User

We need to create new Crypto User in the HSM, which we will be used by the Kaleido CloudHSM to communicate with the AWS CloudHSM. Each Crypto User can access the keys created only by that user. To create a new user, we will use the cloudhsm_mgt_util command line. Follow the instructions at https://docs.aws.amazon.com/cloudhsm/latest/userguide/cloudhsm_mgmt_util-getting-started.html in section Prepare to run cloudhsm_mgt_util.

Follow the below steps to create the Crypto User:

Start the cloudhsm_mgt_util

/opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg

Enable end-to-end encryption
```
enable_e2e
```
Login to the Cryptgraphic Officer (CO) account you created when activating the cluster at https://docs.aws.amazon.com/cloudhsm/latest/userguide/activate-cluster.html.
```
loginHSM CO admin <password>
```
Use the below command to see the list of available users.
```
listUsers
```
Create a Crypto User(CU) and specify a password. Make sure to note down the username and password of the CU as you will need to provide them when creating the configuration of Kaleido CloudHSM.
```
createUser CU <username> <password>
```
Now you will be able to see the new user in the list of users.
Logout from the CO account and quit
```
logoutHSM
quit
```

Create Ethereum Signing Keys

Follow the below steps to create secp256k1 keys:

First make sure the cloudhsm-client is running. Else run below command to start the client:
```
sudo service cloudhsm-client start
```
Start the key_mgmt_util
```
/opt/cloudhsm/bin/key_mgmt_util
```

loginHSM -u CU -s <username> -p <password>

Create a secp256k1 key
```
genECCKeyPair -i 16 -l <key label>
```
The argument -i specifies the EC curve. Value 16 is for the secp256k1 curve used by Ethereum. The argument -l provides a user-defined string as the key label, which can be used to lookup the key. You can also use the argument -id to specify a user-defined string as the key ID.
You can use the below command to see the key generated
```
findKey -l <key label>
```
This key will be used for signing the transactions. You can create more keys in the HSM.

Set up Network Load Balancer

Setting up Network Load Balancer enables client connectivity from outside the AWS VPC. In our case, the AWS CloudHSM client daemon will be running in the Kaleido CloudHSM service and will connect to the AWS CloudHSM cluster through the NLB. To setup the NLB for enabling aws client connectivity follow the below steps:

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, under LOAD BALANCING, choose Load Balancers.
Choose Create Load Balancer.
For Network Load Balancer, choose Create.
For Name, type a name for your load balancer.
For Scheme, choose internet-facing
In Listeners, Choose to Add TCP listener to port 2223, 2224, 2225.
For Availability Zones, select the VPC that you used for your CloudHSM. For each Availability Zone that you used to launch your CloudHSM instances, select an Availability Zone. Then select Choose an Elastic IP and assign an elastic IP. This IP will be used to communicate with your CloudHSM instance.
Choose Next: Configure Security Settings
Choose Next: Configure Routing
For Target group, create a target group for 2223 using the following steps:
1. Keep the default, New target group.
2. For Name, type a name for the target group, eg, group2223
3. For Target type, choose IP.
4. For Protocol, choose TCP.
5. Set Port to corresponding port 2223.
6. For Health checks, keep the default health check settings.
7. Choose Next: Register Targets.
8. For Network, select the VPC of the CloudHSM(s)
9. For IP, type the IP address of the CloudHSM(s) or specify a range that contains the IP address of the CloudHSM(s)
10. In port choose 2223
11. Choose Add to list
choose Next: Review
choose Create
After creating the Load balancer, choose Target Groups under the Load Balancing pane and choose Create Target Group. Similar to step 11 to create 2 more target groups for 2224 and 2225. Note: For VPC choose the VPC of the CloudHSM.
In the load balancer page, select your load balancer and do the following for each of 2224, 2225:
1. In Listeners, choose Add Listener
2. For Protocol:Port choose TCP: 2224 (or) 2225
3. For Action, choose Forward To and choose the respective target group
4. On the navigation pane, under LOAD BALANCING, choose Target Groups.
5. Select the newly created target group and add the listener.
Choose Targets and verify that your instances are ready. If the status of an instance is initial, it's probably because the instance is still in the process of being registered, or it has not passed the minimum number of health checks to be considered healthy. After the status of at least one instance is healthy, you can test your load balancer.

Provision the Service

The next steps are for the Kaleido CloudHSM services inside the Kaleido console. If you are using the Kaleido Asset Platform return to the wallet configuration documentation.

Once the AWS CloudHSM and NLB are setup and the Ethereum signing keys have been generated, we can create the Kaleido service.

Make sure you have the follwoing details on hand, before provisioning the service:

Field
Crypto User Username
Crypto User Password
Network Load Balancer IP
Self signed certificate customerCA.crt

Create CloudHSM Configuration:

The CloudHSM configuration for AWS has the following mandatory parameters:

Field	Usage
`type`	Must be `cloudhsm`
`name`	Optional name for the configuration. e.g. `hsm-config-aws`
`membership_id`	Membership id of the consortium that owns this configuration
`details`
-- `provider`	Provider for backend. Must be `aws`
-- `user_id`	Crypto User Username
-- `user_secret`	Crypto User password
-- `target_address`	Network Load Balancer IP
-- `tls`
---- `ca_certs`	String content of the customerCA.crt content

The following is a sample POST request to create an AWS CloudHSM configuration:

https://console.kaleido.io/api/v1/consortia/:consortia_id/environments/:environment_is/configurations

{
  "type": "cloudhsm",
  "name": "hsm-config-aws",
  "membership_id": "zzjhz5h6qq",
  "details": {
    "provider":"aws",
    "user_id":"CU_user1",
    "user_secret":"CU_password1",
    "target_address":"3.126.158.92",
    "tls": {
      "ca_certs": "-----BEGIN CERTIFICATE-----\nMIIDdDCCAlwCCQDJT0Y+LNE8eTANBgkqhkiG9w0BAQsFADB8MQswCQYDVQQGEwJV\nUzELMAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB0thbGVp\nZG8xDDAKBgNVBAsMA0VuZzEQMA4GA1UEAwwHS2FsZWlkbzEcMBoGCSqGSIb3DQEJ\nARYNaXRAa2FsZWlkDAyMjcyMTQ5MTNaFw0zMDAyMjYyMTQ5MTNa\nMHwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQ\nMA4GA1UECgwHS2FsZWlkbzEMMAoGA1UECwwDRW5nMRAwDgYDVQQDDAdLYWxlaWRv\nMRwwGgYJKoZIhvcNAQkBFg1pdEBrYWxlaWRvLmlvMIIBIjANBgkqhkiG9w0BAQEF\nAAOCAQ8AMIIBCgKCAQEApglp8Eu6N2Jw7ylPcv/SzhccB5n12D5E+7ImpNnDuHK5\nmELdLCGGxsRf9+m/Q3+wK70OAndXOjGKcB20/eTIqysrHv7Wd6T5Aj5tK9MQoMWE\nrZeoJJJIHr0agaaxnJ25L1hALiXQ8nylPjX5AlQRBjVENbF6ul9nmDVl\nvE7BhJ1D4UEo5pMiZkmcV3kbzVM9SRytDKNud9A+gwVue1wYo3ogM6r0ez8ADX/r\nUwu1pKhh57GOwbRgLL6cJDzTKpItqWX+Q4P7IEmHNP7feW8wTA4GSDOd2Dp43RHa\nYj7ToTGidorADHbEfae8CCX1nld2vQVwohZ5Hdbg0QIDAQABMA0GCSqGSIb3DQEB\nCwUAA4IBAQAJR+5JSOyL7nBIeULDMoAhVuB3ZJZXko+1M+kuUwHNE9VT3f31IVFM\nzmxhJug9FsnWuSzjoxmA7j2++i25+Q8tdqbLHXZUWb3yo22Tknyk1wh3WBxQ6UEY\nd71obcy/7yZYxFEQRy4hFRndf/oVnxZSUYfI8oaqrPVWf3AMm8wv/o33cVyhvV6m\nfN+MHRCFaPEr/z6gz29RrLFVEMsPnRTdeK3GPuMs6IlpQK2eC7iMMvcE96PdNF13\nBU8hjIW1SBxE1mCmBha6lXUUGxqlm565qUCZAJcDnMvJUfWKP48Eb7tm3Pz\nJCvPsenTn78DV4IBTalWY4dJMjBfpQEW\n-----END CERTIFICATE-----\n"
    }
  }
}

The following is a sample response:

{
  "type": "cloudhsm",
  "name": "hsm-config-aws",
  "membership_id": "zzjhz5h6qq",
  "details": {
    "provider": "aws",
    "user_id": "pkcs11user",
    "target_address": "3.126.158.92",
    "tls": {
      "ca_certs": "-----BEGIN CERTIFICATE-----\nMIIDdDCCAlwCCQDJT0Y+LNE8eTANBgkqhkiG9w0BAQsFADB8MQswCQYDVQQGEwJV\nUzELMAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB0thbGVp\nZG8xDDAKBgNVBAsMA0VuZzEQMA4GA1UEAwwHS2FsZWlkbzEcMBoGCSqGSIb3DQEJ\nARYNaXRAa2FsZWlkDAyMjcyMTQ5MTNaFw0zMDAyMjYyMTQ5MTNa\nMHwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQ\nMA4GA1UECgwHS2FsZWlkbzEMMAoGA1UECwwDRW5nMRAwDgYDVQQDDAdLYWxlaWRv\nMRwwGgYJKoZIhvcNAQkBFg1pdEBrYWxlaWRvLmlvMIIBIjANBgkqhkiG9w0BAQEF\nAAOCAQ8AMIIBCgKCAQEApglp8Eu6N2Jw7ylPcv/SzhccB5n12D5E+7ImpNnDuHK5\nmELdLCGGxsRf9+m/Q3+wK70OAndXOjGKcB20/eTIqysrHv7Wd6T5Aj5tK9MQoMWE\nrZeoJJJIHr0agaaxnJ25L1hALiXQ8nylPjX5AlQRBjVENbF6ul9nmDVl\nvE7BhJ1D4UEo5pMiZkmcV3kbzVM9SRytDKNud9A+gwVue1wYo3ogM6r0ez8ADX/r\nUwu1pKhh57GOwbRgLL6cJDzTKpItqWX+Q4P7IEmHNP7feW8wTA4GSDOd2Dp43RHa\nYj7ToTGidorADHbEfae8CCX1nld2vQVwohZ5Hdbg0QIDAQABMA0GCSqGSIb3DQEB\nCwUAA4IBAQAJR+5JSOyL7nBIeULDMoAhVuB3ZJZXko+1M+kuUwHNE9VT3f31IVFM\nzmxhJug9FsnWuSzjoxmA7j2++i25+Q8tdqbLHXZUWb3yo22Tknyk1wh3WBxQ6UEY\nd71obcy/7yZYxFEQRy4hFRndf/oVnxZSUYfI8oaqrPVWf3AMm8wv/o33cVyhvV6m\nfN+MHRCFaPEr/z6gz29RrLFVEMsPnRTdeK3GPuMs6IlpQK2eC7iMMvcE96PdNF13\nBU8hjIW1SBxE1mCmBha6lXUUGxqlm565qUCZAJcDnMvJUfWKP48Eb7tm3Pz\nJCvPsenTn78DV4IBTalWY4dJMjBfpQEW\n-----END CERTIFICATE-----\n"
    }
  },
  "_id": "zzpdg2zzxq",
  "_revision": "0",
  "created_at": "2020-04-09T13:33:40.025Z",
  "environment_id": "zztqmsszmp"
}

Create CloudHSM Service:

Once the configuration is created, the CloudHSM service is ready to be provisioned. The following are the mandatory fields to create the service:

Field	Usage
`name`	Name for your service, e.g. `cloudhsm-1`
`membership_id`	Membership id of the consortia this service will belong to
`service`	Type of service. Must be `cloudhsm`
`details`
-- `cloudhsm_id`	The configuation id created in the previous step

The following is a sample POST request to create an AWS cloudhsm service:

https://console.kaleido.io/api/v1/consortia/:consortia_id/environments/:environment_id/services

{
  "name": "cloudhsm-1",
  "membership_id": "zzjhz5h6qq",
  "service": "cloudhsm",
  "details": {
    "cloudhsm_id": "zzpdg2zzxq"
  }
}

The follwoing is a sample response:

{
  "name": "cloudhsm-1",
  "membership_id": "zzjhz5h6qq",
  "service": "cloudhsm",
  "details": {
    "cloudhsm_id": "zzpdg2zzxq",
    "node_id": "zznnf9muy3"
  },
  "service_guid": "3b92e16f-95b3-419f-8091-201f79624f99",
  "service_type": "member",
  "zone_id": "zziik3y2hb",
  "state": "provisioning",
  "_id": "zzprkw067k",
  "_revision": "0",
  "created_at": "2020-04-09T13:34:41.903Z",
  "environment_id": "zztqmsszmp"
}

The CloudHSM service pod starts along with the AWS CloudHSM client and the AWS PKCS #11 library. For now, the service does not support PKCS #11 library with Redis. The connectivity between the AWS CloudHSM and the Kaleido service is established using a secure end to end connection which is managed by the cloudhsm-client. More details on this are at https://docs.aws.amazon.com/cloudhsm/latest/userguide/client-tools-and-libraries.html#client.

Send transactions using Kaleido CloudHSM Service:

Now that the CloudHSM service has been provisioned, you can use the keys stored in AWS Cloudhsm to sign and submit transactions. The service supports the full Ethereum JSON-RPC API, such that you can send eth_sendTransaction requests to it using a Web3 client, or using the Kaleido REST API Gateway.

Get the CloudHSM service status to get a list of the API urls:

GET https://console.kaleido.io/api/v1/consortia/:consotia_id/environments/:environment_id/services/:service_id/status

A sample response is as follows:

{
  "urls": {
    "http": "https://zztqmsszmp-zzprkw067k-cloudhsm.us0-aws.kaleido.io",
    "rpc-http": "https://zztqmsszmp-zzprkw067k-rpc.us0-aws.kaleido.io",
    "rpc-ws": "wss://zztqmsszmp-zzprkw067k-wss.us0-aws.kaleido.io"
  },
  "status": "live",
  "release": "0.1.12189",
  "awsCloudHsm": "live"
}

The rpc-http and rpc-ws URLs are used to send the transactions. The format of the url is as follows:

https://<username>:<password>@<rpc-http url>

The username and password are generated through app creds. An example RPC url is:

https://zzurruiha5:k2p4PP2XKwe3r1MwBiV5rAAfeIquBxmgjXcyA8ZyZ_w@zztqmsszmp-zzprkw067k-rpc.us0-aws.kaleido.io

If more signing keys are created in the AWS CloudHSM, you need to call the Kaleido CloudHSM service to refresh the list of known signing accounts. You can invoke the POST /refresh api on the cloudhsm service to do this. You need to use the http url of the service status obtained earlier.

curl -X POST -d '{}' https://<username>:<password>@<http url>/api/v1/accounts/refresh

The output will contain a list of account(s) available in the AWS CloudHSM backend. An example account list is:

[
  "0x1d125C7D2205b299384521C82826EFe1fc83f6dA",
  "0x927F1A2F052b94E649B65dBf5C3678249960a29e"
]

Submitting transactions using `kaleido-go`

Kaleido provides handy CLI for submitting transactions. Download the CLI executable using instructions in https://github.com/kaleido-io/kaleido-go and you can send a transaction as follows:

./kaleido-go -f examples/simplestorage.sol -u https://<username>:<password>@<rpc-http url> -m set -x <value> -a <account>