Concourse v5.2.0 : Since past few days we have started observing **fatal error: runtime: out of memory**

Hi,

Since past few days we have started observing fatal error: runtime: out of memory error intermittently. I need some assistance to figure the root cause to this. Error log:

concourse-server[32654]: fatal error: runtime: out of memory
concourse-server[32654]: runtime stack:
concourse-server[32654]: runtime.throw(0x1cf526f, 0x16)
concourse-server[32654]: #011/usr/local/go/src/runtime/panic.go:617 +0x72
concourse-server[32654]: runtime.sysMap(0xcdd8000000, 0x4000000, 0x383d678)
concourse-server[32654]: #011/usr/local/go/src/runtime/mem_linux.go:170 +0xc7
concourse-server[32654]: runtime.(*mheap).sysAlloc(0x3823f40, 0xe000, 0x3823f50, 0x7)
concourse-server[32654]: #011/usr/local/go/src/runtime/malloc.go:633 +0x1cd
concourse-server[32654]: runtime.(*mheap).grow(0x3823f40, 0x7, 0x0)
concourse-server[32654]: #011/usr/local/go/src/runtime/mheap.go:1232 +0x42
concourse-server[32654]: runtime.(*mheap).allocSpanLocked(0x3823f40, 0x7, 0x383d688, 0x7f96e25a5ab0)
concourse-server[32654]: #011/usr/local/go/src/runtime/mheap.go:1150 +0x3a7
concourse-server[32654]: runtime.(*mheap).alloc_m(0x3823f40, 0x7, 0x460067, 0x7f96e25a5ab0)
concourse-server[32654]: #011/usr/local/go/src/runtime/mheap.go:977 +0xc2
concourse-server[32654]: runtime.(*mheap).alloc.func1()
concourse-server[32654]: #011/usr/local/go/src/runtime/mheap.go:1048 +0x4c
concourse-server[32654]: runtime.systemstack(0x7f980aac9ba0)
concourse-server[32654]: #011/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
concourse-server[32654]: runtime.mstart()
concourse-server[32654]: #011/usr/local/go/src/runtime/proc.go:1153
concourse-server[32654]: goroutine 270479667 [running]:
concourse-server[32654]: runtime.systemstack_switch()
concourse-server[32654]: #011/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc55f57aa08 sp=0xc55f57aa00 pc=0x466960
concourse-server[32654]: runtime.(*mheap).alloc(0x3823f40, 0x7, 0x10067, 0xc0000b6000)
concourse-server[32654]: #011/usr/local/go/src/runtime/mheap.go:1047 +0x8a fp=0xc55f57aa58 sp=0xc55f57aa08 pc=0x43012a
concourse-server[32654]: runtime.(*mcentral).grow(0x3825c80, 0x0)
concourse-server[32654]: #011/usr/local/go/src/runtime/mcentral.go:256 +0x95 fp=0xc55f57aaa0 sp=0xc55f57aa58 pc=0x423065
concourse-server[32654]: runtime.(*mcentral).cacheSpan(0x3825c80, 0xcdd7ef29f0)
concourse-server[32654]: #011/usr/local/go/src/runtime/mcentral.go:106 +0x2ff fp=0xc55f57ab00 sp=0xc55f57aaa0 pc=0x422b6f
concourse-server[32654]: runtime.(*mcache).refill(0x7f980e6b7f90, 0x67)
concourse-server[32654]: #011/usr/local/go/src/runtime/mcache.go:135 +0x86 fp=0xc55f57ab20 sp=0xc55f57ab00 pc=0x422606
concourse-server[32654]: runtime.(*mcache).nextFree(0x7f980e6b7f90, 0xc55f57ab67, 0x0, 0x8, 0x23ec)
concourse-server[32654]: #011/usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0xc55f57ab58 sp=0xc55f57ab20 pc=0x416d88
concourse-server[32654]: runtime.mallocgc(0x2500, 0x0, 0x0, 0xc)
concourse-server[32654]: #011/usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0xc55f57abf8 sp=0xc55f57ab58 pc=0x41769e
concourse-server[32654]: runtime.rawbyteslice(0x2333, 0x0, 0x0, 0x0)
concourse-server[32654]: #011/usr/local/go/src/runtime/string.go:272 +0xa1 fp=0xc55f57ac38 sp=0xc55f57abf8 pc=0x454241
concourse-server[32654]: runtime.stringtoslicebyte(0x0, 0xcdd5985900, 0x2333, 0xcdd5983400, 0x2333, 0xcdd5985900)
concourse-server[32654]: #011/usr/local/go/src/runtime/string.go:161 +0xb5 fp=0xc55f57ac80 sp=0xc55f57ac38 pc=0x453c15
concourse-server[32654]: github.com/concourse/concourse/atc/db/encryption.NoEncryption.Decrypt(...)
concourse-server[32654]: #011/tmp/build/70f2e240/concourse/atc/db/encryption/no_encryption.go:18
concourse-server[32654]: github.com/concourse/concourse/atc/db/encryption.(*NoEncryption).Decrypt(0x383b4c8, 0xcdd5985900, 0x2333, 0x0, 0xcdd5985900, 0x2333, 0x25e7420, 0xc0000b6000, 0x7f980e6b7f90)
concourse-server[32654]: #011<autogenerated>:1 +0x9c fp=0xc55f57acc0 sp=0xc55f57ac80 pc=0x8efd7c
concourse-server[32654]: github.com/concourse/concourse/atc/db.scanJob(0xcdd574db80, 0x25ca000, 0xca53d94880, 0x0, 0x0)
concourse-server[32654]: #011/tmp/build/70f2e240/concourse/atc/db/job.go:844 +0x340 fp=0xc55f57ad70 sp=0xc55f57acc0 pc=0x9738c0
concourse-server[32654]: github.com/concourse/concourse/atc/db.scanJobs(0x263ec00, 0xc0006499a0, 0x25caaa0, 0xc000dfa940, 0xca53d94880, 0x0, 0x0, 0x0, 0x0, 0x0)
concourse-server[32654]: #011/tmp/build/70f2e240/concourse/atc/db/job.go:868 +0x14e fp=0xc55f57ade0 sp=0xc55f57ad70 pc=0x973cde
concourse-server[32654]: github.com/concourse/concourse/atc/db.(*jobFactory).VisibleJobs(0xc00002c720, 0xc01fd6b800, 0x22, 0x40,