Knowledge Base Home

Actionable Insights Through Realtime Log Streaming

The Ethereum logs for each node serve as a rich datasource for devops-centric tasks such as application troubleshooting and operational health monitoring. While the Kaleido /logs API provides a convenient endpoint to retrieve these logs, it requires a manual configuration against the consortia/environment/node resource IDs and must be constantly polled to ensure up-to-date streams.

As an alternative to this configuration-intensive approach, Kaleido offers the option to integrate nodes with AWS’ monitoring and management service, Cloudwatch, and directly stream realtime logs. A fully-encompassed monitoring service provides the ability to visualize Kaleido logs alongside existing resources and processes, and surfaces an aggregated trove of data that can lead to greater insights and application optimization. For example, your Cloudwatch service could be customized to trigger metric-based alarms and issue automated actions based on certain inflections. This centralized view of the core application and business processes helps lead to more informed decisions and increased efficiency.

Similar to KMS-enablement, log streams must be configured prior to creating the node. Refer to the Creating a Node with AWS Integrations section for instructions on enabling log streams via the user interface. Refer to the Understanding the Kaleido API documentation for instructions on enabling log streams via the Kaleido API. Note that if you elect to use the raw API, you must first create an opsmetric configuration object which is passed as a required parameter on the node creation call. The instructions for creating a Cloudwatch Logging Group and an AWS user with the appropriate Cloudwatch Logs permissions are outlined below in the Configuring Cloudwatch Logging Permissions section. The AWS user and logging group must be properly provisioned in order to successfully create a Log Streaming configuration.

Configuring Cloudwatch Logging Permissions

Follow the below steps to configure log streaming into an AWS Cloudwatch Group:

  • Log into the AWS console and navigate to the IAM Service
  • Click the Users tab in the IAM navigation panel to see an overview of your existing AWS users. Kaleido requires certain permissions in order to successfully propagate logs to a Cloudwatch Group. These permissions will need to applied as a custom policy against a user within your account. You can apply these permissions to an existing user or create a new user for Cloudwatch purposes. Once your users are ready, click the Policies tab in the IAM navigation panel.
  • Click Create Policy at the top of the screen and choose Cloudwatch Logs as the targeted service for the policy
  • Click the List dropdown and apply DescribeLogGroups and DescribeLogStreams access
  • Click the Write dropdown and apply CreateLogGroupCreateLogStream and PutLogEvents
  • Create a name for the policy and review it, ensuring that you have provided access to the five actions above. Click Create policy to finish.
  • Next, click the Users tab in the navigation panel to attach the newly created custom policy to an existing user.
  • Click the Add Permissions button and select the Attach existing policies directly option
  • Enter you custom policy name into the search bar and select it. Click Next: Review
  • Click the custom policy hyperlink to see the JSON output of the applied permissions. Make sure you see the five access permissions defined in the JSON object. The object should resemble the following:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        },
}
  • Next, click the Services tab at the top of the console and select Cloudwatch under the Management Tools column
  • Click the Logs tab in the Cloudwatch navigation panel
  • Click the Actions dropdown and select Create log group
  • Enter a name for the logging group, e.g. “kaleidologs”, and create the group
  • Take note of the AWS region in the URL, e.g. us-east-2. You will need this information when you configure your Ops Metrics resource on Kaleido
Prev Heightened Security through a Key Management Store Next On Demand Backups to an S3 Bucket