[Nagiosplug-help] check_nrpe plugins work fine for a while then start throwing "NRPE: unable to read output"

[Philip Marcus]

Hey Kyle,

I'm just throwing out some random ideas off the top of my head...
To test, I just enabled debug in one of my nrpe configs, restarted the
process and did see more output from the daemon.log file in my /var/log
dir. (running ubuntu)
Do you see any output from nrpe? you should see a startup and stop message.
If not, check the logging facility, and Stop/Start the nrpe daemon again.

Regarding the problem where after a while the remote check fails, thats a
difficult one to pin down. It sounds like something is timing out. Without
knowing anything about your environment or what the actual check is
doing..., but it sounds like maybe some local disk access is timing out
(I've seen this with kerberized NFS.), maybe some other local environment
setting or keyring or something is expiring. When you login (as which ever
user), you get a fresh "cert" and the command works, but the cert the
existing process is running under has timed out. Like I said, just a stab in
the dark.

Good luck,

Philip


On Wed, May 20, 2009 at 3:56 PM, Kyle Smith <kyles at apple.com> wrote:

> Hey Nagios Folks,
> We're seeing some strange behavior with a few of our nrpe plugins and am
> wondering if anyone has any ideas on how to work on the problem.
>
> We are running OS X 10.5.6 with NRPE 2.12 and NRPE plugins 1.4.13.
>
> We have some plugins written in Ruby (boss loves it, I am learning it :-))
> that work fine with a freshly started nrpe daemon. Start up nrpe on the
> client, force checks from the gui, see a green nagios console, nice. But
> after a while in the console under "Status Information" we start seeing
> "NRPE: unable to read output" rather than our normal success messages.
> Bounce the nrpe on the client, force the check through the gui, and things
> start working again for a while.
>
> While seeing the errors I can still run the plugin on the client no problem
> but when I try to run it from the server via command line
> (/usr/local/nagios/libexec/check_nrpe -H IP.of.client -c command) I see the
> error "NRPE: unable to read output".
>
> Another data point - We have one system where things were working fine but
> we needed to start passing params to nrpe. I recompiled after running
> ./configure --enable-command-args and setting dont_blame_nrpe=1 in the
> nrpe.cfg. Start up a new nrpe and it works fine, can pass params from the
> server no problem...but the a check that was working fine for weeks starts
> throwing "NRPE: unable to read output" after 3 or 4 successful
> invocations. Arg!! I had to rollback to the previous nrpe and check_nrpe.
>
> I am trying to set nrpe debug on in nrpe.cfg on the client and then bounce
> nrpe but i don't see any more messages in /var/log/system.log as compared to
> when I run with nrpe debug turned off. It's like turning on debug has no
> effect.
>
> I am curious about 2 issues - 1) why no extra messages in the system log
> file with debug enabled? and 2) Anyone seen anything like this happening
> where nrpe plugins work fine for a while then start throwing the error?
>
> TIA
>
> kyle
>
>
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals.
> Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090520/dcb25c84/attachment.html>