I love monitoring. Well actually I don’t, but I realized there’s no sane way to live without it and so I’ve grown to love it.
One of the key components to successfully running big data infrastructure like Apache Hadoop, Apache Cassandra or Apache Zookeeper in production is monitoring the heck out of them. This is crucial in a multitude of aspects. First and foremost is the learning aspect. Looking at these monitoring charts can teach you a lot about the internal workings and behavior of these infrastructures. Second aspect is tuning configuration. For example, when you make a change to Column Family’s caching configuration you need to get insight on how your change affects performance and the overall behavior of the cluster. Third is problem identification, you want to measure the behavior of your infrastructure over time to identify performance degradations, bottlenecks etc,. Fourth is capacity planning and the ability to measure performance as the environment around your infrastructure changes (e.g., traffic growth).
So monitoring is important, very important.