FAST ESP - LOGTRANSFORMER fails with "process logtransformer ports are already in use"

  • Article

Issue : LOGTRANSFORMER fails with "process logtransformer ports are already in use"

Assessment/Resolution:

Looking in to LOGTRANSFORMER.log file
**************************************************************
java.io.IOException: There is not enough space on the disk
[2013-03-03 16:25:15,066] ERROR org.quartz.core.ErrorLogger: An error occured while marking executed job complete. job= 'lql.periodLqlJob' (FastLogTransformer_Worker-0)
org.quartz.JobPersistenceException: Couldn't update job data: ERROR: could not extend relation 1663/10820/52439: No space left on device [See nested exception: org.postgresql.util.PSQLException: ERROR: could not extend relation 1663/10820/52439: No space left on device]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:1933)
at org.quartz.impl.jdbcjobstore.JobStoreTX.triggeredJobComplete(JobStoreTX.java:1279)
at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1489)
at org.quartz.core.JobRunShell.completeTriggerRetryLoop(JobRunShell.java:384)
at org.quartz.core.JobRunShell.run(JobRunShell.java:268)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
* Nested Exception (Underlying Cause) ---------------
org.postgresql.util.PSQLException: ERROR: could not extend relation 1663/10820/52439: No space left on device

[2013-03-03 17:15:33,794] ERROR com.fastsearch.esp.logtransformer.harvest.download.DownLoader: Failure during logharvesting (FastLogTransformer_Worker-0)
com.fastsearch.esp.logtransformer.harvest.download.DownloadException: Unable to obtain list of node controllers Unable to obtain list of active modules with name NodeControl from configserver XmlRpc[interface com.fastsearch.esp.logtransformer.harvest.download.qrserver.IConfigServer/com.fastsearch.esp.logtransformer.harvest.download.qrserver.ConfigServerMapping -> https://fast.vinit.com:16005/RPC2]
 at com.fastsearch.esp.logtransformer.harvest.download.qrserver.FdsBackendHelper.getAllNodeControllers(FdsBackendHelper.java:76)
 at com.fastsearch.esp.logtransformer.harvest.download.qrserver.FdsBackendHelper.getNodeControllersForQrservers(FdsBackendHelper.java:168)
 at com.fastsearch.esp.logtransformer.harvest.download.qrserver.FdsBackendHelper.getAllQrserverHostAndNodeControllers(FdsBackendHelper.java:126)
 at com.fastsearch.esp.logtransformer.harvest.download.DownLoader.runBatch(DownLoader.java:95)
 at com.fastsearch.esp.logtransformer.scheduler.HarvestJob.execute(HarvestJob.java:45)
 at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
 at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: com.fastsearch.esp.logtransformer.harvest.download.DownloadException: Unable to obtain list of active modules with name NodeControl from configserver XmlRpc[interface com.fastsearch.esp.logtransformer.harvest.download.qrserver.IConfigServer/com.fastsearch.esp.logtransformer.harvest.download.qrserver.ConfigServerMapping -> https://fast.vinit.com:16005/RPC2]

 

Resolution:
 
Stop the LOGTRANSFORMER via NCTRL and made sure all the java.exe and javaw.exe files are not running for the FASTADMIN user
Then NCTRL STOP LOGTRANSFORMER, killed all java.exe and javaw.exe running by the fastadmin
 
Start all the services with NCTRL start
Then started the LogTransformer with nctrl; It should start successfully and the logs, scraps should get generated as well.