[Nagiosplug-help] check_nrpe plugins work fine for a while then start throwing "NRPE: unable to read output"

[Kyle Smith]

Hey Philip,

Thanks for sending the log messages you see with debug enabled. I wish  
I was seeing the same but all I see are the startup and shutdown msgs  
with debug enabled (already pasted into this thread). It's frustrating  
since we have a few nrpe issues here and if I could get nrpe to tell  
me what it is doing I would have more of a foothold to resoluion. As  
it is I am bouncing nrpe weekly on one machine and daily on another  
just to keep things running, arg.

kyle

On May 21, 2009, at 11:07 AM, Philip Marcus wrote:

> Hey Kyle,
>
> top-posting for laziness. :)
>
> Here's an example of the output I am seeing in my log file after  
> enabling debug logging in the NRPE config.
>
> May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX  
> port 31457
> May 20 18:08:08 host1 nrpe[12854]: Host address is in allowed_hosts
> May 20 18:08:08 host1 nrpe[12854]: Handling the connection...
> May 20 18:08:08 host1 nrpe[12854]: Host is asking for command  
> 'check_swap' to be run...
> May 20 18:08:08 host1 nrpe[12854]: Running command: /usr/local/ 
> nagios/libexec/check_swap -w 30 -c 20
> May 20 18:08:08 host1 nrpe[12854]: Command completed with return  
> code 0 and output: SWAP OK - 100% free (15257 MB out of 15257 MB) | 
> swap=15257MB;0;0;0;15257
> May 20 18:08:08 host1 nrpe[12854]: Return Code: 0, Output: SWAP OK -  
> 100% free (15257 MB out of 15257 MB) |swap=15257MB;0;0;0;15257
> May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX  
> closed.
>
> When debug logging is disabled I don't see any of this.
> You should definitely see those messages on a successful attempt.
>
> On the unsuccessful attempts, if you don't see any connection  
> attempts, it may be a network issue.
> You can run tcpdump on each host and see where the breakdown in  
> communication may be.
>
> Good luck,
>
> Philip
>
>
>
> On Thu, May 21, 2009 at 7:59 AM, Kyle Smith <kyles at apple.com> wrote:
> Hey Philip,
>
> Thanks for the response, see my comments inline...
> On May 20, 2009, at 6:09 PM, Philip Marcus wrote:
>
>> Hey Kyle,
>>
>> I'm just throwing out some random ideas off the top of my head...
>> To test, I just enabled debug in one of my nrpe configs, restarted  
>> the process and did see more output from the daemon.log file in my / 
>> var/log  dir. (running ubuntu)
>> Do you see any output from nrpe? you should see a startup and stop  
>> message. If not, check the logging facility, and Stop/Start the  
>> nrpe daemon again.
>
> Yes I see nrpe messages for startup and shtudown of the daemon in my  
> system log file (I'm on OSX so logfile is /var/log/system.log). My  
> issue is that I don't see any more messages with debug enabled than  
> I do with debug disabled. This is hampering my ability to work this  
> issue since i can't tell what nrpe is doing internally. Here's a  
> snippet of the logging I see for a shutdown and restart of nrpe on  
> my host manta. As part of the startup logging it tells me I have  
> configrued nrpe to accept command args which is accurate...
>
> May 21 07:55:30 manta nrpe[10200]: Caught SIGTERM - shutting down...
> May 21 07:55:30 manta nrpe[10200]: Cannot remove pidfile '/var/run/ 
> nrpe.pid' - check your privileges.
> May 21 07:55:30 manta nrpe[10200]: Daemon shutdown
> May 21 07:55:33 manta sudo[11969]: kylesmith : TTY=ttys000 ; PWD=/ 
> Users/kylesmith ; USER=root ; COMMAND=/usr/local/nagios/etc/init.d/ 
> init-script start
> May 21 07:55:33 manta nrpe[11971]: Starting up daemon
> May 21 07:55:33 manta nrpe[11971]: Warning: Daemon is configured to  
> accept command arguments from clients!
>
>>
>>
>> Regarding the problem where after a while the remote check fails,  
>> thats a difficult one to pin down. It sounds like something is  
>> timing out. Without knowing anything about your environment or what  
>> the actual check is doing..., but it sounds like maybe some local  
>> disk access is timing out (I've seen this with kerberized NFS.),  
>> maybe some other local environment setting or keyring or something  
>> is expiring. When you login (as which ever user), you get a fresh  
>> "cert" and the command works, but the cert the existing process is  
>> running under has timed out. Like I said, just a stab in the dark.
>
> I understand what you are getting at and I will investigate such  
> possibilities.
>>
>>
>> Good luck,
>
> Thanks again! >bows deeply<
>
> kyle
>
>>
>>
>> Philip
>>
>>
>> On Wed, May 20, 2009 at 3:56 PM, Kyle Smith <kyles at apple.com> wrote:
>> Hey Nagios Folks,
>>
>> We're seeing some strange behavior with a few of our nrpe plugins  
>> and am wondering if anyone has any ideas on how to work on the  
>> problem.
>>
>> We are running OS X 10.5.6 with NRPE 2.12 and NRPE plugins 1.4.13.
>>
>> We have some plugins written in Ruby (boss loves it, I am learning  
>> it :-)) that work fine with a freshly started nrpe daemon. Start up  
>> nrpe on the client, force checks from the gui, see a green nagios  
>> console, nice. But after a while in the console under "Status  
>> Information" we start seeing "NRPE: unable to read output" rather  
>> than our normal success messages. Bounce the nrpe on the client,  
>> force the check through the gui, and things start working again for  
>> a while.
>>
>> While seeing the errors I can still run the plugin on the client no  
>> problem but when I try to run it from the server via command line (/ 
>> usr/local/nagios/libexec/check_nrpe -H IP.of.client -c command) I  
>> see the error "NRPE: unable to read output".
>>
>> Another data point - We have one system where things were working  
>> fine but we needed to start passing params to nrpe. I recompiled  
>> after running ./configure --enable-command-args and setting  
>> dont_blame_nrpe=1 in the nrpe.cfg. Start up a new nrpe and it works  
>> fine, can pass params from the server no problem...but the a check  
>> that was working fine for weeks starts throwing "NRPE: unable to  
>> read output" after 3 or 4 successful invocations. Arg!! I had to  
>> rollback to the previous nrpe and check_nrpe.
>>
>> I am trying to set nrpe debug on in nrpe.cfg on the client and then  
>> bounce nrpe but i don't see any more messages in /var/log/ 
>> system.log as compared to when I run with nrpe debug turned off.  
>> It's like turning on debug has no effect.
>>
>> I am curious about 2 issues - 1) why no extra messages in the  
>> system log file with debug enabled? and 2) Anyone seen anything  
>> like this happening where nrpe plugins work fine for a while then  
>> start throwing the error?
>>
>> TIA
>>
>> kyle
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity  
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>> _______________________________________________
>> Nagiosplug-help mailing list
>> Nagiosplug-help at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>> ::: Please include plugins version (-v) and OS when reporting any  
>> issue.
>> ::: Messages without supporting info will risk being sent to /dev/ 
>> null
>>
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity  
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like  
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
>> _______________________________________________
>> Nagiosplug-help mailing list
>> Nagiosplug-help at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>> ::: Please include plugins version (-v) and OS when reporting any  
>> issue.
>> ::: Messages without supporting info will risk being sent to /dev/ 
>> null
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any  
> issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any  
> issue.
> ::: Messages without supporting info will risk being sent to /dev/null

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090522/43ef7d80/attachment.html>