Uploaded image for project: 'Corda'
  1. CORDA-2646

Database connection pools leaking memory on every checkpoint

    Details

    • Type: Bug
    • Status: Done (View workflow)
    • Priority: Highest
    • Resolution: Done
    • Affects versions: Corda 4, Corda Enterprise 4
    • Fix versions: Corda 4.1
    • Components: None
    • Labels:
      None
    • Severity:
      High
    • Target Version/s:
    • Feature Team:
      Performance and Platform Sustainability
    • Sprint:

      Description

      Due to the use of ThreadLocals inside the database connection pooling and Quasar, the flow state machine leaks around 300 bytes every flow suspension/context switch.

      In discussion with Former user we concluded this is because we take the database connection from the pool in the Fiber and return it in the Thread. The connection pool has an optimisation to recycle thread local connections first. The list of returned connections in the Thread builds up (leading to the memory leak) and the Fiber always experiences a miss (of the optimisation) and has to resort to the global pool.

      The fix is to somehow wormhole the Thread's ThreadLocal into the Fiber so that experiences a hit and the Thread does not experience returns only. The underlying data structures would then be shared.

      We think this is/must be possible via reflection.

      This bug could be present in 3.x. No idea why it wouldn't be, but it didn't seem to be visible when we tested it. Maybe it was not visible due to lower transaction volumes in Azure vs. OVH. Or maybe the HikariCP dependency did not experience the same problem.

        Attachments

          Activity

            People

            • Assignee:
              Rick.Parker Rick Parker
              Reporter:
              Rick.Parker Rick Parker
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: