[Nagiosplug-help] check_nrpe plugins work fine for a while then start throwing "NRPE: unable to read output"

[Kyle Smith]

Hey folks,

Just an update to anyone listening, I got nrpe debug logging working  
for OSX. I checked into the source and noticed that any messages sent  
by the call to syslog with LOG_NOTICE, LOG_WARNING, or LOG_ERR as the  
first parameter were working fine but if the first param to syslog was  
LOG_DEBUG I never saw that message. So I changed all calls to syslog  
with LOG_DEBUG as first param to use LOG_WARNING as first param  
instead. Re-compiled nrpe and moved the binary in place (/usr/local/ 
nagios/bin) and bounced nrpe and now I'm seeing all those lovely log  
messages from nrpe.

w00t!

kyle

On May 22, 2009, at 6:36 AM, Kyle Smith wrote:

> Hey Philip,
>
> Thanks for sending the log messages you see with debug enabled. I  
> wish I was seeing the same but all I see are the startup and  
> shutdown msgs with debug enabled (already pasted into this thread).  
> It's frustrating since we have a few nrpe issues here and if I could  
> get nrpe to tell me what it is doing I would have more of a foothold  
> to resoluion. As it is I am bouncing nrpe weekly on one machine and  
> daily on another just to keep things running, arg.
>
> kyle
>
> On May 21, 2009, at 11:07 AM, Philip Marcus wrote:
>
>> Hey Kyle,
>>
>> top-posting for laziness. :)
>>
>> Here's an example of the output I am seeing in my log file after  
>> enabling debug logging in the NRPE config.
>>
>> May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX  
>> port 31457
>> May 20 18:08:08 host1 nrpe[12854]: Host address is in allowed_hosts
>> May 20 18:08:08 host1 nrpe[12854]: Handling the connection...
>> May 20 18:08:08 host1 nrpe[12854]: Host is asking for command  
>> 'check_swap' to be run...
>> May 20 18:08:08 host1 nrpe[12854]: Running command: /usr/local/ 
>> nagios/libexec/check_swap -w 30 -c 20
>> May 20 18:08:08 host1 nrpe[12854]: Command completed with return  
>> code 0 and output: SWAP OK - 100% free (15257 MB out of 15257 MB) | 
>> swap=15257MB;0;0;0;15257
>> May 20 18:08:08 host1 nrpe[12854]: Return Code: 0, Output: SWAP OK  
>> - 100% free (15257 MB out of 15257 MB) |swap=15257MB;0;0;0;15257
>> May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX  
>> closed.
>>
>> When debug logging is disabled I don't see any of this.
>> You should definitely see those messages on a successful attempt.
>>
>> On the unsuccessful attempts, if you don't see any connection  
>> attempts, it may be a network issue.
>> You can run tcpdump on each host and see where the breakdown in  
>> communication may be.
>>
>> Good luck,
>>
>> Philip
>>
>>
>>
>> On Thu, May 21, 2009 at 7:59 AM, Kyle Smith <kyles at apple.com> wrote:
>> Hey Philip,
>>
>> Thanks for the response, see my comments inline...
>> On May 20, 2009, at 6:09 PM, Philip Marcus wrote:
>>
>>> Hey Kyle,
>>>
>>> I'm just throwing out some random ideas off the top of my head...
>>> To test, I just enabled debug in one of my nrpe configs, restarted  
>>> the process and did see more output from the daemon.log file in  
>>> my /var/log  dir. (running ubuntu)
>>> Do you see any output from nrpe? you should see a startup and stop  
>>> message. If not, check the logging facility, and Stop/Start the  
>>> nrpe daemon again.
>>
>> Yes I see nrpe messages for startup and shtudown of the daemon in  
>> my system log file (I'm on OSX so logfile is /var/log/system.log).  
>> My issue is that I don't see any more messages with debug enabled  
>> than I do with debug disabled. This is hampering my ability to work  
>> this issue since i can't tell what nrpe is doing internally. Here's  
>> a snippet of the logging I see for a shutdown and restart of nrpe  
>> on my host manta. As part of the startup logging it tells me I have  
>> configrued nrpe to accept command args which is accurate...
>>
>> May 21 07:55:30 manta nrpe[10200]: Caught SIGTERM - shutting down...
>> May 21 07:55:30 manta nrpe[10200]: Cannot remove pidfile '/var/run/ 
>> nrpe.pid' - check your privileges.
>> May 21 07:55:30 manta nrpe[10200]: Daemon shutdown
>> May 21 07:55:33 manta sudo[11969]: kylesmith : TTY=ttys000 ; PWD=/ 
>> Users/kylesmith ; USER=root ; COMMAND=/usr/local/nagios/etc/init.d/ 
>> init-script start
>> May 21 07:55:33 manta nrpe[11971]: Starting up daemon
>> May 21 07:55:33 manta nrpe[11971]: Warning: Daemon is configured to  
>> accept command arguments from clients!
>>
>>>
>>>
>>> Regarding the problem where after a while the remote check fails,  
>>> thats a difficult one to pin down. It sounds like something is  
>>> timing out. Without knowing anything about your environment or  
>>> what the actual check is doing..., but it sounds like maybe some  
>>> local disk access is timing out (I've seen this with kerberized  
>>> NFS.), maybe some other local environment setting or keyring or  
>>> something is expiring. When you login (as which ever user), you  
>>> get a fresh "cert" and the command works, but the cert the  
>>> existing process is running under has timed out. Like I said, just  
>>> a stab in the dark.
>>
>> I understand what you are getting at and I will investigate such  
>> possibilities.
>>>
>>>
>>> Good luck,
>>
>> Thanks again! >bows deeply<
>>
>> kyle
>>
>>>
>>>
>>> Philip
>>>
>>>
>>> On Wed, May 20, 2009 at 3:56 PM, Kyle Smith <kyles at apple.com> wrote:
>>> Hey Nagios Folks,
>>>
>>> We're seeing some strange behavior with a few of our nrpe plugins  
>>> and am wondering if anyone has any ideas on how to work on the  
>>> problem.
>>>
>>> We are running OS X 10.5.6 with NRPE 2.12 and NRPE plugins 1.4.13.
>>>
>>> We have some plugins written in Ruby (boss loves it, I am learning  
>>> it :-)) that work fine with a freshly started nrpe daemon. Start  
>>> up nrpe on the client, force checks from the gui, see a green  
>>> nagios console, nice. But after a while in the console under  
>>> "Status Information" we start seeing "NRPE: unable to read output"  
>>> rather than our normal success messages. Bounce the nrpe on the  
>>> client, force the check through the gui, and things start working  
>>> again for a while.
>>>
>>> While seeing the errors I can still run the plugin on the client  
>>> no problem but when I try to run it from the server via command  
>>> line (/usr/local/nagios/libexec/check_nrpe -H IP.of.client -c  
>>> command) I see the error "NRPE: unable to read output".
>>>
>>> Another data point - We have one system where things were working  
>>> fine but we needed to start passing params to nrpe. I recompiled  
>>> after running ./configure --enable-command-args and setting  
>>> dont_blame_nrpe=1 in the nrpe.cfg. Start up a new nrpe and it  
>>> works fine, can pass params from the server no problem...but the a  
>>> check that was working fine for weeks starts throwing "NRPE:  
>>> unable to read output" after 3 or 4 successful invocations. Arg!!  
>>> I had to rollback to the previous nrpe and check_nrpe.
>>>
>>> I am trying to set nrpe debug on in nrpe.cfg on the client and  
>>> then bounce nrpe but i don't see any more messages in /var/log/ 
>>> system.log as compared to when I run with nrpe debug turned off.  
>>> It's like turning on debug has no effect.
>>>
>>> I am curious about 2 issues - 1) why no extra messages in the  
>>> system log file with debug enabled? and 2) Anyone seen anything  
>>> like this happening where nrpe plugins work fine for a while then  
>>> start throwing the error?
>>>
>>> TIA
>>>
>>> kyle
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>>> is a gathering of tech-side developers & brand creativity  
>>> professionals. Meet
>>> the minds behind Google Creative Lab, Visual Complexity,  
>>> Processing, &
>>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>>> Barbarian
>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>>> _______________________________________________
>>> Nagiosplug-help mailing list
>>> Nagiosplug-help at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>>> ::: Please include plugins version (-v) and OS when reporting any  
>>> issue.
>>> ::: Messages without supporting info will risk being sent to /dev/ 
>>> null
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>>> is a gathering of tech-side developers & brand creativity  
>>> professionals. Meet
>>> the minds behind Google Creative Lab, Visual Complexity,  
>>> Processing, &
>>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>>> Barbarian
>>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
>>> _______________________________________________
>>> Nagiosplug-help mailing list
>>> Nagiosplug-help at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>>> ::: Please include plugins version (-v) and OS when reporting any  
>>> issue.
>>> ::: Messages without supporting info will risk being sent to /dev/ 
>>> null
>>
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity  
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>> _______________________________________________
>> Nagiosplug-help mailing list
>> Nagiosplug-help at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>> ::: Please include plugins version (-v) and OS when reporting any  
>> issue.
>> ::: Messages without supporting info will risk being sent to /dev/ 
>> null
>>
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity  
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
>> _______________________________________________
>> Nagiosplug-help mailing list
>> Nagiosplug-help at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>> ::: Please include plugins version (-v) and OS when reporting any  
>> issue.
>> ::: Messages without supporting info will risk being sent to /dev/ 
>> null
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any  
> issue.
> ::: Messages without supporting info will risk being sent to /dev/null

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090523/ffccb573/attachment.html>