Sunday, August 9, 2015

linux - How to Scale graphite?

I am having a lot of trouble understanding how to scale Graphite. I have a production instance of graphite (carbon-cache + whisper + graphite-web + grafana). I am running out of disk space and I think I need to add in a 2nd node. I can't seem to find any good guides on people doing this and I am having a hard time understanding the documentation.



Can I just spin up carbon-cache + whisper on the 2nd node and configure carbon-relay to relay the information to both the 1st and 2nd node?



Will graphite-web be able to query both successfully?




I feel like I am missing something very important.



--
PS. I tried googling this but my google fu may be bad. I also searched stackoverflow and serverfault but all I can seem to find are posts about piping multiple servers metrics to graphite/statsd.



EDIT



I think I need to clarify. I can setup the relay and cache just fine (they seem fine). It is graphite-web that I have trouble with. I setup a new graphite-web on a standalone VM (nothing but graphite-web, uwsgi, and nginx installed on it). From here I tried querying it with the find below and it is always empty.




curl 'localhost:8543/metrics/find?query=*' 
[]


That said on the original server it works just fine outside of being full.



root@original_server:/etc/nginx/sites-enabled# curl -s 
'localhost:8080/metrics/find?query=*' | jq
[
{

"text": "bobstats",
"expandable": 1,
"leaf": 0,
"id": "bobstats",
"allowChildren": 1
},
...
]



Does the carbon-relay need to be with the graphite-web? Do I need to install graphite-web on each of the cache servers as well or each of the relays?

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...