2017-12-20 03:02:57.044 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.14/10.76.17.14:2181, unexpected error, closing socket co nnection and attempting reconnect java.io.IOException: Xid out of order. Got Xid 718 with err -101 expected Xid 717 for a packet with details: clientPath:null serverPath:null finished:false header:: 717,14 replyHeader:: 0,0, -4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ab83c response:: org.apache.zookeeper.MultiResponse@0 at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2017-12-20 03:02:57.049 ERROR (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:218) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:417) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:308) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1091) at org.apache.solr.cloud.ZkController.register(ZkController.java:894) at org.apache.solr.cloud.ZkController.register(ZkController.java:849) at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183) at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212) at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:697) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:821) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:751) at org.apache.solr.handler.admin.CoreAdminOperation$1.call(CoreAdminOperation.java:131) at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935) at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915) at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:570) at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:567) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:567) at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:201) at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:49) at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:42) at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:183)
... 40 more 2017-12-20 03:03:25.675 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There may be a better lead er candidate than us - going back into recovery 2017-12-20 03:03:25.675 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ElectionContext Canceling election /collections/hospi tal_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000002 2017-12-20 03:03:25.774 ERROR (OverseerCollectionConfigSetProcessor-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342) at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:358) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:175) at java.lang.Thread.run(Thread.java:745) 2017-12-20 03:03:25.776 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None 2017-12-20 03:03:25.776 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected 2017-12-20 03:03:26.018 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable 2017-12-20 03:03:26.020 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name :ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2017-12-20 03:03:27.327 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase No version found for e phemeral leader parent node, won't remove previous leader registration. 2017-12-20 03:03:27.329 WARN (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[hospital_resources_shard1_replica1] coreNodeName=[core_node1] 2017-12-20 03:03:27.329 INFO (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery 2017-12-20 03:03:27.329 INFO (updateExecutor-2-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_resour ces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ActionThrottle The last recovery attempt started 28638ms ago. 2017-12-20 03:03:27.331 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2017-12-20 03:03:27.331 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]] 2017-12-20 03:03:27.342 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.LeaderElector Joined leadership election with path: / collections/hospital_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000004 2017-12-20 03:03:27.344 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ActionThrottle The last leader attempt started 28639m s ago. 2017-12-20 03:03:27.345 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=true and leaderVoteWait=180000 2017-12-20 03:03:27.381 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext All replicas are ready to participate in election. 2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Checking if I should try a nd be the leader. 2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext My last published State wa s Active, it's okay to be the leader. 2017-12-20 03:03:27.382 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2017-12-20 03:03:27.382 WARN (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[hospital _resources_shard1_replica1] coreNodeName=[core_node1] 2017-12-20 03:03:27.383 INFO (OverseerStateUpdate-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { "operation":"leader", "shard":"shard1", "collection":"hospital_resources"} current state version: 0 2017-12-20 03:03:27.406 INFO (OverseerStateUpdate-171210747425456190-10.76.17.14:8983_solr-n_0000000000) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/hospital_resour ces/state.json version: 3 2017-12-20 03:03:27.439 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataC hanged path:/collections/hospital_resources/state.json] for collection [hospital_resources] has occurred - updating... (live nodes size: [1]) 2017-12-20 03:03:27.442 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating data for [hospital_resources] from [3] to [4] 2017-12-20 03:03:29.882 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://10.76.17.14:8983 /solr/hospital_resources_shard1_replica1/ 2017-12-20 03:03:29.883 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2017-12-20 03:03:29.884 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.SyncStrategy http://10.76.17.14:8983/solr/hospital_re sources_shard1_replica1/ has no replicas 2017-12-20 03:03:29.889 INFO (qtp428746855-14) [c:hospital_resources s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase Creating leader regist ration node /collections/hospital_resources/leaders/shard1/leader after winning as /collections/hospital_resources/leader_elect/shard1/election/171210747425456190-core_node1-n_0000000004 2017-12-20 03:03:30.031 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.12/10.76.17.12:2181, unexpected error, closing socket co nnection and attempting reconnect java.io.IOException: Xid out of order. Got Xid 804 with err 0 expected Xid 803 for a packet with details: clientPath:null serverPath:null finished:false header:: 803,14 replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0 at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2017-12-20 03:03:30.134 INFO (zkCallback-4-thread-31-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None 2017-12-20 03:03:30.134 INFO (zkCallback-4-thread-31-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected 2017-12-20 03:03:31.065 WARN (main-SendThread(10.76.17.11:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable 2017-12-20 03:03:31.067 INFO (zkCallback-4-thread-7-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name :ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2017-12-20 03:03:31.353 ERROR (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Error while trying to recover. core=hospital_resources_shard1_replica1:org.apache.solr.common.SolrEx ception: No registered leader was found after waiting for 4000ms , collection: hospital_resources slice: shard1 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:626) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:612) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:306) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates. 2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed 2017-12-20 03:03:31.353 INFO (recoveryExecutor-3-thread-1-processing-n:10.76.17.14:8983_solr x:hospital_resources_shard1_replica1 s:shard1 c:hospital_resources r:core_node1) [c:hospital_reso urces s:shard1 r:core_node1 x:hospital_resources_shard1_replica1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false] 2017-12-20 03:03:31.636 WARN (main-SendThread(10.76.17.11:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.11/10.76.17.11:2181, unexpected error, closing socket co nnection and attempting reconnect java.io.IOException: Xid out of order. Got Xid 807 with err 0 expected Xid 806 for a packet with details: clientPath:null serverPath:null finished:false header:: 806,14 replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0 at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2017-12-20 03:03:31.739 INFO (zkCallback-4-thread-33-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None 2017-12-20 03:03:31.740 INFO (zkCallback-4-thread-33-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected 2017-12-20 03:03:32.487 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable 2017-12-20 03:03:32.490 INFO (zkCallback-4-thread-3-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 name :ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2017-12-20 03:03:34.741 WARN (main-SendThread(10.76.17.14:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.14/10.76.17.14:2181, unexpected error, closing socket co nnection and attempting reconnect java.io.IOException: Xid out of order. Got Xid 810 with err 0 expected Xid 809 for a packet with details: clientPath:null serverPath:null finished:false header:: 809,14 replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0 at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2017-12-20 03:03:34.844 INFO (zkCallback-4-thread-32-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None 2017-12-20 03:03:34.845 INFO (zkCallback-4-thread-32-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected 2017-12-20 03:03:35.526 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable 2017-12-20 03:03:35.529 INFO (zkCallback-4-thread-43-processing-n:10.76.17.14:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ec055f5 nam e:ZooKeeperConnection Watcher:10.76.17.11:2181,10.76.17.12:2181,10.76.17.14:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2017-12-20 03:03:39.346 WARN (main-SendThread(10.76.17.12:2181)) [ ] o.a.z.ClientCnxn Session 0x26043448606003e for server 10.76.17.12/10.76.17.12:2181, unexpected error, closing socket co nnection and attempting reconnect java.io.IOException: Xid out of order. Got Xid 814 with err 0 expected Xid 813 for a packet with details: clientPath:null serverPath:null finished:false header:: 813,14 replyHeader:: 0,0,-4 request:: org.apache.zookeeper.MultiTransactionRecord@dd1ac200 response:: org.apache.zookeeper.MultiResponse@0 at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
|