Wednesday, December 10, 2014

Error: "Message 279 not found" while starting Listener

One of my customers has Oracle 11.2.0.4 on windows 2008 R2 (x64). After reboot of this database server, listener did not come up automatically and even manually starting the listener was returning following error at the command prompt.

###################
Starting tnslsnr: please wait...

Failed to start service, error 0.
TNS-12560: TNS:protocol adapter error
###################

Listener.log file was showing following messages related to the failure of listener.
###################
Tue Nov 25 19:29:04 2014
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL

Tue Nov 25 19:31:22 2014
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL
###################

Same server also had the application running on it and this application was using an Oracle client installation in a different home. While listener was down, TNSPING executed from the client home was showing error "ORA-12541: TNS:no listener " which was clearly showing that listener is not running.

Reason of Issue and Solution
Since production system was down, it was really a panic situation for my customer and as a DBA, it was also a challenge to find out the root cause and resolve the solution. While "googling" it, I saw a lot of different discussions and also some MOS notes and almost all of them were related to UNIX based systems. On these systems, the reason of the problem were because of 1)Wrongly set permissions on “lsnrctl” executable. 2) Wrongly set ORACLE_HOME and PATH environment variables settings. 3) Wrongly set TNS_ADMIN environment variable.
I also made sure that all of these environment variables are set properly and made sure that local windows Administrator has all the permissions, but still my problem was not fixed. While thinking of “permissions”, I just thought to check if any recent windows updates were installed on the system and this revealed that system is set to download and install Windows updates automatically and last time some windows security updates were installed 7 days ago and this was first system reboot after the installation of those security updates. So this thing clicked me that this might be the cause of this issue.
Immediately I went to uninstall these recently installed security updates and then rebooted the server. After the reboot, I could just shout “Yahoooooooo” because listener was up and running automatically.

Moral of the story
This is really dangerous to enable automatic updates download/install on Windows based systems, I always recommend to disable automatic download and installation of the Windows updates because no one knows how would it impact the system after the installation and a reboot. System administrators for Windows based Oracle servers should make sure that all updates are tested on a testing server before these could be manually downloaded and installed on the production database server.

There should also be a contingency plan so that these updates could be rolled back in case any issue happens to the listener/database or any other Oracle executable after the updates installation.