Uploaded image for project: 'Corda'
  1. CORDA-2748

Error in HikariPool in the performance cluster

    Details

    • Type: Bug
    • Status: Done (View workflow)
    • Priority: Highest
    • Resolution: Done
    • Affects versions: Corda Enterprise 4 RC05
    • Fix versions: Corda 4.1
    • Components: None
    • Labels:
      None

      Description

      No idea how to reproduce, but the error failed a flow. We meddled with Hikari to solve a memory leak when interacting with Quasar. Maybe/hopefully that is imperfect.

      [INFO ] 2019-03-14T10:34:53,880Z [flow-worker] statemachine.StaffedFlowHospital.invoke - Flow [7bfcf9b7-da7e-401c-b9da-dde7b3c16419] has error [0]

      {actor_id=corda, actor_owning_identity=OU=Corda, O=Perf-193.70.35.191, L= London, C=GB, actor_store_id=NODE_CONFIG, fiber-id=10000298, flow-id=7bfcf9b7-da7e-401c-b9da-dde7b3c16419, invocation_id=dd44ce68-1d59-4ebc-838b-9750647fe65e, invocation_timestamp=2019-03-14T10:34:53.813Z, origin=corda, session_id=5f37e1c3-8722-40e7-86b9-15ef3044ce57, session_timestamp=2019-03-14T10:34:46.464Z, thread-id=845, tx_id=C797D72B5DF2789565545BF8B804E14D16C9D5CCDA2EB12BA134E451307CAC46}

      java.lang.ArrayIndexOutOfBoundsException: -1
      at com.zaxxer.hikari.util.FastList.remove(FastList.java:199) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:133) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:163) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83) ~[HikariCP-2.5.1.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction$connection$2.invoke(DatabaseTransaction.kt:34) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction$connection$2.invoke(DatabaseTransaction.kt:21) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at kotlin.UnsafeLazyImpl.getValue(Lazy.kt:81) ~[kotlin-stdlib-1.2.71.jar:1.2.71-release-64 (1.2.71)]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction.getConnection(DatabaseTransaction.kt) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction$sessionDelegate$1.invoke(DatabaseTransaction.kt:44) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction$sessionDelegate$1.invoke(DatabaseTransaction.kt:21) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) ~[kotlin-stdlib-1.2.71.jar:1.2.71-release-64 (1.2.71)]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction.getSession(DatabaseTransaction.kt) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransactionKt.currentDBSession(DatabaseTransaction.kt:12) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.persistence.DBCheckpointStorage.removeCheckpoint(DBCheckpointStorage.kt:60) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.ActionExecutorImpl.executeRemoveCheckpoint(ActionExecutorImpl.kt:166) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.ActionExecutorImpl.executeAction(ActionExecutorImpl.kt:64) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.MetricActionInterceptor.executeAction(MetricInterceptor.kt:31) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.TransitionExecutorImpl.executeTransition(TransitionExecutorImpl.kt:41) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.HospitalisingInterceptor.executeTransition(HospitalisingInterceptor.kt:44) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.MetricInterceptor.executeTransition(MetricInterceptor.kt:23) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.processEvent(FlowStateMachineImpl.kt:132) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:196) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:266) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:45) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at co.paralleluniverse.fibers.Fiber.run1(Fiber.java:1092) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:788) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.29.Final.jar:4.1.29.Final]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
      [ERROR] 2019-03-14T10:34:53,885Z [flow-worker] statemachine.TransitionExecutorImpl.executeTransition - Error while executing CommitTransaction, erroring state [errorCode=10fp3nw, moreInformationAt=https://errors.corda.ne
      t/ENT/5.0-SNAPSHOT/10fp3nw]

      {actor_id=corda, actor_owning_identity=OU=Corda, O=Perf-193.70.35.191, L=London, C=GB, actor_store_id=NODE_CONFIG, fiber-id=10000297, flow-id=443cee50-1f5e-4f40-b892-a7ef4ff84f79, invocation_i d=39de7617-aca7-409a-9ede-e2805aba6c45, invocation_timestamp=2019-03-14T10:34:53.812Z, origin=corda, session_id=e518e457-9d1a-45ae-aa92-628c175f5b72, session_timestamp=2019-03-14T10:34:46.462Z, thread-id=797, tx_id=91328 A0DF0F3F1B0269D2B427BCFD12F129461AC7E9CD68C1EDCFFA0286C641C}

      java.lang.ArrayIndexOutOfBoundsException: -1
      at com.zaxxer.hikari.util.FastList.add(FastList.java:87) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.util.ConcurrentBag.requite(ConcurrentBag.java:194) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.pool.HikariPool.recycle(HikariPool.java:394) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.pool.PoolEntry.recycle(PoolEntry.java:89) ~[HikariCP-2.5.1.jar:?]
      at com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:252) ~[HikariCP-2.5.1.jar:?]
      at net.corda.nodeapi.internal.persistence.DatabaseTransaction.close(DatabaseTransaction.kt:108) ~[corda-node-api-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.ActionExecutorImpl.executeCommitTransaction(ActionExecutorImpl.kt:217) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.ActionExecutorImpl.executeAction(ActionExecutorImpl.kt:73) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.MetricActionInterceptor.executeAction(MetricInterceptor.kt:31) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.TransitionExecutorImpl.executeTransition(TransitionExecutorImpl.kt:41) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.HospitalisingInterceptor.executeTransition(HospitalisingInterceptor.kt:44) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.interceptors.MetricInterceptor.executeTransition(MetricInterceptor.kt:23) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.processEvent(FlowStateMachineImpl.kt:132) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:196) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:266) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:45) ~[corda-node-5.0-SNAPSHOT.jar:?]
      at co.paralleluniverse.fibers.Fiber.run1(Fiber.java:1092) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:788) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91) ~[quasar-core-0.7.11_r3-jdk8.jar:0.7.11_r3]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_201]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_201]
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.29.Final.jar:4.1.29.Final]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

        Attachments

          Activity

            People

            • Assignee:
              Rick.Parker Rick Parker
              Reporter:
              Rick.Parker Rick Parker
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: