[Nagiosplug-help] check_nrpe plugins work fine for a while then start throwing "NRPE: unable to read output"

[Philip Marcus]

Hey Kyle,

top-posting for laziness. :)

Here's an example of the output I am seeing in my log file after enabling
debug logging in the NRPE config.

May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX port
31457
May 20 18:08:08 host1 nrpe[12854]: Host address is in allowed_hosts
May 20 18:08:08 host1 nrpe[12854]: Handling the connection...
May 20 18:08:08 host1 nrpe[12854]: Host is asking for command 'check_swap'
to be run...
May 20 18:08:08 host1 nrpe[12854]: Running command:
/usr/local/nagios/libexec/check_swap -w 30 -c 20
May 20 18:08:08 host1 nrpe[12854]: Command completed with return code 0 and
output: SWAP OK - 100% free (15257 MB out of 15257 MB)
|swap=15257MB;0;0;0;15257
May 20 18:08:08 host1 nrpe[12854]: Return Code: 0, Output: SWAP OK - 100%
free (15257 MB out of 15257 MB) |swap=15257MB;0;0;0;15257
May 20 18:08:08 host1 nrpe[12854]: Connection from XXX.XXX.XXX.XXX closed.

When debug logging is disabled I don't see any of this.
You should definitely see those messages on a successful attempt.

On the unsuccessful attempts, if you don't see any connection attempts, it
may be a network issue.
You can run tcpdump on each host and see where the breakdown in
communication may be.

Good luck,

Philip



On Thu, May 21, 2009 at 7:59 AM, Kyle Smith <kyles at apple.com> wrote:

> Hey Philip,
> Thanks for the response, see my comments inline...
> On May 20, 2009, at 6:09 PM, Philip Marcus wrote:
>
> Hey Kyle,
>
> I'm just throwing out some random ideas off the top of my head...
> To test, I just enabled debug in one of my nrpe configs, restarted the
> process and did see more output from the daemon.log file in my /var/log
> dir. (running ubuntu)
> Do you see any output from nrpe? you should see a startup and stop message.
> If not, check the logging facility, and Stop/Start the nrpe daemon again.
>
>
> Yes I see nrpe messages for startup and shtudown of the daemon in my system
> log file (I'm on OSX so logfile is /var/log/system.log). My issue is that I
> don't see any more messages with debug enabled than I do with debug
> disabled. This is hampering my ability to work this issue since i can't tell
> what nrpe is doing internally. Here's a snippet of the logging I see for a
> shutdown and restart of nrpe on my host manta. As part of the startup
> logging it tells me I have configrued nrpe to accept command args which is
> accurate...
>
> May 21 07:55:30 manta nrpe[10200]: Caught SIGTERM - shutting down...
> May 21 07:55:30 manta nrpe[10200]: Cannot remove pidfile
> '/var/run/nrpe.pid' - check your privileges.
> May 21 07:55:30 manta nrpe[10200]: Daemon shutdown
> May 21 07:55:33 manta sudo[11969]: kylesmith : TTY=ttys000 ;
> PWD=/Users/kylesmith ; USER=root ;
> COMMAND=/usr/local/nagios/etc/init.d/init-script start
> May 21 07:55:33 manta nrpe[11971]: Starting up daemon
> May 21 07:55:33 manta nrpe[11971]: Warning: Daemon is configured to accept
> command arguments from clients!
>
>
>
>
> Regarding the problem where after a while the remote check fails, thats a
> difficult one to pin down. It sounds like something is timing out. Without
> knowing anything about your environment or what the actual check is
> doing..., but it sounds like maybe some local disk access is timing out
> (I've seen this with kerberized NFS.), maybe some other local environment
> setting or keyring or something is expiring. When you login (as which ever
> user), you get a fresh "cert" and the command works, but the cert the
> existing process is running under has timed out. Like I said, just a stab in
> the dark.
>
>
> I understand what you are getting at and I will investigate such
> possibilities.
>
>
>
> Good luck,
>
>
> Thanks again! >bows deeply<
>
> kyle
>
>
>
> Philip
>
>
> On Wed, May 20, 2009 at 3:56 PM, Kyle Smith <kyles at apple.com> wrote:
>
>> Hey Nagios Folks,
>> We're seeing some strange behavior with a few of our nrpe plugins and am
>> wondering if anyone has any ideas on how to work on the problem.
>>
>> We are running OS X 10.5.6 with NRPE 2.12 and NRPE plugins 1.4.13.
>>
>> We have some plugins written in Ruby (boss loves it, I am learning it :-))
>> that work fine with a freshly started nrpe daemon. Start up nrpe on the
>> client, force checks from the gui, see a green nagios console, nice. But
>> after a while in the console under "Status Information" we start seeing
>> "NRPE: unable to read output" rather than our normal success messages.
>> Bounce the nrpe on the client, force the check through the gui, and things
>> start working again for a while.
>>
>> While seeing the errors I can still run the plugin on the client no
>> problem but when I try to run it from the server via command line
>> (/usr/local/nagios/libexec/check_nrpe -H IP.of.client -c command) I see the
>> error "NRPE: unable to read output".
>>
>> Another data point - We have one system where things were working fine but
>> we needed to start passing params to nrpe. I recompiled after running
>> ./configure --enable-command-args and setting dont_blame_nrpe=1 in the
>> nrpe.cfg. Start up a new nrpe and it works fine, can pass params from the
>> server no problem...but the a check that was working fine for weeks starts
>> throwing "NRPE: unable to read output" after 3 or 4 successful
>> invocations. Arg!! I had to rollback to the previous nrpe and check_nrpe.
>>
>> I am trying to set nrpe debug on in nrpe.cfg on the client and then bounce
>> nrpe but i don't see any more messages in /var/log/system.log as compared to
>> when I run with nrpe debug turned off. It's like turning on debug has no
>> effect.
>>
>> I am curious about 2 issues - 1) why no extra messages in the system log
>> file with debug enabled? and 2) Anyone seen anything like this happening
>> where nrpe plugins work fine for a while then start throwing the error?
>>
>> TIA
>>
>> kyle
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity professionals.
>> Meet
>> the minds behind Google Creative Lab, Visual Complexity, Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>> _______________________________________________
>> Nagiosplug-help mailing list
>> Nagiosplug-help at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
>> ::: Please include plugins version (-v) and OS when reporting any issue.
>> ::: Messages without supporting info will risk being sent to /dev/null
>>
>>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals.
> Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com_______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
>
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals.
> Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> Nagiosplug-help mailing list
> Nagiosplug-help at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagiosplug-help
> ::: Please include plugins version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-plugins.org/archive/help/attachments/20090521/6a756632/attachment.html>