everything happens for good
nowadays i have been ‘realizing’ many proverbs i would say. well, i may post those experiences in a separate blog!
this is an infamous proverb we might have been coming across since our childhood during several occasions and we happened to hear from many people around as well. ‘everything happens for good‘ – this might make us feel irritated and like adding fuel to fire when we are ‘in’ the circle! yes, i mean it.
when we say it to others, it may not matter but only when we ‘experience’, it will give a totally different shape.
Today i got a call from office saying that the filesystem reached its 100% in Production. Despite being a holiday, it does not matter when you are on a production support :). You have to be available on 24*7 basis as per norms. Well, i just told the network admin (aka system admin) to move the heapdumps, javacore files (they are the files which the Java process writes into the filesystem when the running java process reaches out of memory) to the external hard drive so that we can get some space. but even after that, there was no luck :(.
I came to office and i did the usual work around. I stopped the application server and cleaned up the temporary files as well by archiving the log files. And moved those archived files to the external hard drive. still no luck :(. It was stubborn and showing 99% :).
Had been thinking what would be the cause and somehow started into the tuning, troubleshooting documents. In the middle, thought of bringing the server up. But to my surprise, the server was not getting started and it was showing some errors in the linux console. After some attempts, thought of killing the process as usual and while doing so, by mistake i killed a different process named ‘EmailService‘ which was on top when i searched for the ‘java’ processes running at present in the system. Usually there won’t be any other services than our regular application server process and the corresponding node agent. Till now i have not seen this EmailService and it was not attracting me too. So i went ahead in killing the process and then i found out the mistake when verifying after deletion.
In the middle, i just checked for the memory usage, it went again upto 100%. Wondering what might be happening!
Thought i have done a mistake and have to get it corrected, searched for the process or its corresponding scheduler (cronjob in linux which is similar to the batch file in windows to automate tasks) for starting the EmailService. There we found so many core dumps generated in the EmailScheduler :). Lucky are we! After taking backup of those core dumps and deleting them, we got 24% free space and the disk usage was 76% !!
When asked the colleague about the issue during the startup of EmailService, he told one more infomation that this email service would be generating some pdfs as well. After some usual searches, we found out in a subdirectory there were some 7600+ pdfs silently sitting and occupying minimum of 1.1 MB each. Few of them had around 2MB size and very few had 20 MBs.. 🙂 So altogether we got some 700 MB+ space in disk.
Had the ‘EmailService’ not been deleted accidentally, we would have ended up somewhere else. Atleast we would have wasted some more time when dealing with the ‘Production issue’.
Overall, we should not panic about the issues rather we should welcome! 🙂 — is that what i feel now! wat say?