I have a problem that occurred recently where my Grid Control 11g agents keep crashing on a daily basis.
To find out why the agent is crashing, I went to the database server and looked in $AGENT_HOME/hostname
/sysman/log in the emagent.log file. At the time of the agent crash, I found this error in the log file:
2012-10-23 23:58:28,991 Thread-1109641536 Unsuccessful Upload attempts for XML file exceeds specified limit=3, Agent will shutdown (00851)
2012-10-23 23:58:28,992 Thread-2673833696 EMAgent abnormal terminating (00704)
Ok…so it should be pretty obvious that the reason the agent keeps crashing is because it cannot complete a successful upload to the OMS. So now the question becomes, why can’t it complete an upload? I can manually perform “$AGENT_HOME/bin/emctl upload agent” without any problems. The answer lies in the $AGENT_HOME/hostname
/sysman/log/emagent.trc file where I saw this entry at the same time:
2012-10-23 23:58:25,659 Thread-1109641536 ERROR upload: Failed to upload file A0000001.xml: Fatal Error.
Response received: 500|ORA-20603: Updating timezone region of a target is not supported. Use mgmt_target.set_agent_tzrgn() API to modify the timezone region.(target name = ncpp.naucom.com)(target type = rac_database)(new tzrgn = US/Central)(old tzrgn = America/Chicago)
2012-10-23 23:58:25,659 Thread-1109641536 ERROR upload: 1 Failure(s) in a row or XML error for A0000001.xml, retcode = -6, we give up
So there is a mistmatch in the timezone information. The fix for this is pretty easy:
$AGENT_HOME/bin/emctl stop agent
$AGENT_HOME/bin/emctl resetTZ agent
$AGENT_HOME/bin/emctl start agent
Then in the OMS repository database as SYSMAN: