The messages build up in the memory of the broker/node until each RPC client has at least 1MB of unacknowledged inbound messages. So for 320 clients, it requires at least 320MB of heap, because we add a queue for each client.
The solution is to reduce the acknowledge window for RPC response queues because each client has their own queue and thus the memory footprint multiplies up. The 1MB limit is excessive (even acknowledging every message does not seem to impact performance), so we will reduce it to 16KB. In testing this allows around 10 unacknowledged responses, and keeps the memory footprint for even 1000 clients to around 16MB.