[RPC][Node Explorer] Application does not reconnect to a node if node was killed and then restarted

Description

Steps to reproduce:

1) Start a node
2) Connect to the node with Node Explorer
3) Kill the node
4) Restart the node

Expected result:
Node Explorer reconnects to node and user can continue using the application, transfer cash and etc

Actual result:
Node Explorer does not reconnect to node, console shows errors

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 Gtk-Message: 14:24:14.241: Failed to load module "canberra-gtk-module" [INFO ] 14:24:24,307 [JavaFX Application Thread] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 QUASAR WARNING: Quasar Java Agent isn't running. If you're using another instrumentation method you can ignore this message; otherwise, please refer to the Getting Started section in the Quasar documentation. [INFO ] 14:24:27,577 [JavaFX Application Thread] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 2032 msec [WARN ] 14:24:53,756 [Thread-19 (ActiveMQ-client-global-threads)] (RemotingConnectionImpl.java:204) core.client.fail - AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] [WARN ] 14:24:53,778 [Thread-19 (ActiveMQ-client-global-threads)] (RPCClientProxyHandler.kt:578) internal.RPCClientProxyHandler.cleanUpOnConnectionLoss - Terminating observables. [ERROR] 14:24:53,818 [Thread-19 (ActiveMQ-client-global-threads)] (RPCClientProxyHandler.kt:571) internal.RPCClientProxyHandler.failoverHandler - Could not reconnect to the RPC server. [ERROR] 14:24:53,784 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:24:53,833 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:24:53,857 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 15 msec [INFO ] 14:24:54,865 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:24:54,875 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 5 msec [INFO ] 14:24:55,991 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:24:56,001 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 5 msec [INFO ] 14:24:57,250 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:24:57,260 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 6 msec [INFO ] 14:24:58,636 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:24:58,645 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 2 msec [INFO ] 14:25:00,171 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:00,180 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 4 msec [INFO ] 14:25:01,877 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:01,883 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 3 msec [INFO ] 14:25:03,768 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:03,778 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 7 msec [INFO ] 14:25:05,892 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:05,918 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 24 msec [INFO ] 14:25:08,248 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:08,264 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 14 msec [INFO ] 14:25:10,847 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:10,855 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 6 msec [INFO ] 14:25:13,726 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:13,730 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Failed Startup took 2 msec [INFO ] 14:25:16,921 [pool-6-thread-1] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:18,432 [pool-6-thread-1] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1507 msec [ERROR] 14:25:23,834 [pool-6-thread-4] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:25:23,847 [pool-6-thread-4] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:25,199 [pool-6-thread-4] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1344 msec [ERROR] 14:25:25,222 [pool-6-thread-3] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:25:25,233 [pool-6-thread-3] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:26,316 [pool-6-thread-3] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1081 msec [ERROR] 14:25:26,342 [pool-6-thread-2] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:25:26,357 [pool-6-thread-2] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:27,444 [pool-6-thread-2] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1083 msec [ERROR] 14:25:27,459 [pool-6-thread-3] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:25:27,462 [pool-6-thread-3] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:28,607 [pool-6-thread-3] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1142 msec [ERROR] 14:25:28,628 [pool-6-thread-4] (ReconnectingCordaRPCOps.kt:146) internal.ReconnectingCordaRPCOps.reconnectOnError - Reconnecting to [localhost:10019] due to error: Connection failure detected. [INFO ] 14:25:28,630 [pool-6-thread-4] (ReconnectingCordaRPCOps.kt:161) internal.ReconnectingCordaRPCOps.establishConnectionWithRetry - Connecting to: localhost:10019 [INFO ] 14:25:29,729 [pool-6-thread-4] (InternalUtils.kt:207) internal.RPCClient.logElapsedTime - Startup took 1098 msec

User can't transfer cash, applications shows errors

1 2 3 4 5 6 7 8 9 10 11 net.corda.client.rpc.ConnectionFailureException: Connection failure detected. at net.corda.client.rpc.internal.RPCClientProxyHandler.close(RPCClientProxyHandler.kt:414) at net.corda.client.rpc.internal.RPCClientProxyHandler.forceClose(RPCClientProxyHandler.kt:372) at net.corda.client.rpc.internal.RPCClient$start$1$1.close(RPCClient.kt:111) at net.corda.client.rpc.internal.RPCClient$start$1$1.forceClose(RPCClient.kt:121) at net.corda.client.rpc.CordaRPCConnection.forceClose(CordaRPCClient.kt:58) at net.corda.client.rpc.internal.ReconnectingCordaRPCOps$ReconnectingRPCConnection.reconnectOnError(ReconnectingCordaRPCOps.kt:149) at net.corda.client.rpc.internal.ReconnectingObservable$ReconnectingSubscriber$scheduleResubscribe$1.run(ReconnectingObservable.kt:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

 

Status

Assignee

Unassigned

Reporter

Dmitry Toropov

Priority

High

Fix versions

Ported to...

None

Feature Team

Kernel Group

Severity

Medium

Affects versions

Corda 4.3 RC01
Configure