Kaleido Open Menu
Close Menu
Log In Try It Free

Download Logs via API

Web-based systems for browsing and searching logs are great, but sometimes you just want a straight forward .txtfile that you can open in your favorite editor... or sharpen your awk sed and grep skills on.

So we provide an API for reading/downloading your node logs (Geth, Constellation, etc.) that can be used to stream the original content down into a local file.

Take a look at the API 101 tutorial if you are new to our REST API.

The /logs API

By default the logs API will query backwards from the end of the log, so if you want the last 50 lines of the geth log:

/api/v1/consortia/$CONSORTIUM/environments/$ENVIRONMENT/nodes/$NODE/logs/geth?maxlines=50

This will return a JSON array, so if you're using curl on the command line you can use the handy jq tool to process the output:

curl ... | jq -r '.[]'

If you want the beginning of the file, then use frompos:

/api/v1/consortia/$CONSORTIUM/environments/$ENVIRONMENT/nodes/$NODE/logs/geth?maxlines=50?frompos=0

You can increase the number of lines you return with maxlines, but eventually you will request more data than it makes sense to query on a single API call.

So each request will return either x-kaleido-startpos (for queries from the end of the file) or x-kaleido-endpos (for queries from the start of the start).

A little tool to make life easy

So you don't need to do lots of scripting, here is a handy python script to output the logs from a node.

  • On Mac install Python with brew install python3
  • On Ubuntu/Debian Linux install Python with apt-get install python3

Run it like so to interactively choose the node you want to get logs from:

# first export the variables; be sure to replace the placeholder value with
# your API key
export APIURL=https://console.kaleido.io/api/v1
export APIKEY="XXXXciyyv77k-DdYYpRPBEa/aeONTn3najyMpoxncqfFSwuYF+nDquWc="

To kick off the script

python logviewer.py

You will iteratively pass values to the Consortium, Environment, Member, Node and Log Type fields in order to generate the corresponding values.

Example output: 

1) api101 [yu6ugcni]
Consortium: 0
Using yu6ugcni
0) Auto-generated Environment [mflg1hd7]
Environment: 0
Using mflg1hd7
0) Default Organization [ddnye6il]
1) Jenkins [ctt686fq]
2) Concourse [bfnm0cby]
Member: 1
Using ctt686fq
0) JenkinsNode1 [yjt76yyd]
Node: 0
Using yjt76yyd
0) geth
1) constellation
Log type: 0
DEBUG[03-18|17:32:38] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
DEBUG[03-18|17:32:58] Recalculated downloader QoS values       rtt=20s confidence=1.000 ttl=1m0s
````

Once you've run it interactively to see the IDs, you can supply these as arguments:

```sh
python logviewer.py -c yu6ugcni -e mflg1hd7 -m ctt686fq -n yjt76yyd -t geth -l 50</code></pre>

If you want to download the whole of a file, try:

python logviewer.py -c yu6ugcni -e mflg1hd7 -m ctt686fq -n yjt76yyd -t constellation -l 0 &gt; /tmp/constellation.log

The script itself

#!/usr/local/bin/python3
import urllib.request, json, os, re, argparse
log_types = ['geth','constellation']
parser = argparse.ArgumentParser(description='Get logs.')
parser.add_argument('--apikey', '-k', type=str,
 required=(not 'APIKEY' in os.environ),
 help='the API Key (or APIKEY env var)')
parser.add_argument('--apiurl', '-u', type=str,
 required=(not 'APIURL' in os.environ),
 help='the API URL (or APIURL env var)')
parser.add_argument('--consortium', '-c', type=str, nargs='?',
 help='the consortium ID')
parser.add_argument('--environment', '-e', type=str, nargs='?',
 help='the environment ID')
parser.add_argument('--member', '-m', type=str, nargs='?',
 help='the membership ID')
parser.add_argument('--node', '-n', type=str, nargs='?',
 help='the node ID')
parser.add_argument('--type', '-t', type=str, nargs='?',
 help='the log type', choices=log_types)
parser.add_argument('--lines', '-l', type=str, nargs='?',
 help='the number of lines, or 0 for whole file',
default=50)
args = parser.parse_args()
if (args.apikey is None):
 args.apikey = os.environ['APIKEY']
if (args.apiurl is None):
 args.apiurl = os.environ['APIURL']
# Get the auth token
url = '{0}/authtoken'.format(args.apiurl)
req = urllib.request.Request(url, \
 json.dumps({ 'apikey': args.apikey }).encode(), \
 {'Content-Type': 'application/json'})
res = urllib.request.urlopen(req)
token = json.load(res)[u'token']
res.close()
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer
{0}'.format(token) }
def pick_from_list(arr, descriptions, prompt):
 idx = ''
 for (i, desc) in enumerate(descriptions):
 print('{0}) {1}'.format(i, desc))
 while not (re.match("^\d+$", idx) and int(idx) &lt; len(arr)): idx = input('{0}: '.format(prompt)) return arr[int(idx)] def choose_from_api_json(path, prompt, name_prop=u'name', id_prop=u'_id') : url = '{0}/{1}'.format(args.apiurl, path) res = urllib.request.urlopen(urllib.request.Request(url, headers=header s)) json_arr = json.load(res) res.close() options = [] for json_entry in json_arr: options.append('{0} [{1}]'.format(json_entry[name_prop], json_entry[i d_prop])) chosen_id = pick_from_list(json_arr, options, prompt)[u'_id'] print('Using {0}'.format(chosen_id)) return chosen_id if (args.consortium is None): args.consortium = choose_from_api_json('/consortia', 'Consorium') if (args.environment is None): args.environment = choose_from_api_json('/consortia/{0}/environments'\ 
  .format(args.consortium), 'Environment') if (args.member is None): args.member = choose_from_api_json('/consortia/{0}/memberships'\ 
  .format(args.consortium), 'Member', name_prop=u'org_name') if (args.node is None): args.node = choose_from_api_json('/consortia/{0}/environments/{1}/nodes ?membership_id={2}'\ 
  .format(args.consortium, args.environment, args.member), 'Node') if (args.type is None): args.type = pick_from_list(log_types, log_types, 'Log type') if (args.lines == '0'): morelines = True frompos = 0 while (morelines): url = '{0}/consortia/{1}/environments/{2}/nodes/{3}/logs/{4}?frompos= {5}&amp;maxlines=50'\ 
  .format(args.apiurl, args.consortium, args.environment, args.node, args.type, frompos) res = urllib.request.urlopen(urllib.request.Request(url, headers=head ers)) jres = json.load(res) res.close() for line in jres: print(line) frompos = res.info().getheader('x-kaleido-endpos') morelines = (len(jres) &gt; 0)
else:
 url = '{0}/consortia/{1}/environments/{2}/nodes/{3}/logs/{4}?maxlines={
5}'\
 .format(args.apiurl, args.consortium, args.environment, args.node, ar
gs.type, args.lines)
 res = urllib.request.urlopen(urllib.request.Request(url, headers=header
s))
 jres = json.load(res)
 res.close()
 for line in jres:
 print(line)