Concourse for scientific computing


#1

Hi All,

I have recently started looking at ways to bring ideas and principles from the world of industry software development, and especially DevOps, into the world of scientific computing. I encountered Concourse while looking for ways to harness the principles of CI/CD for the benefit of computing-based research loops, and I wonder if anyone has already gone in this direction.

The idea in principle is to have researches run small models on their own local machines, perhaps while using an IDE like PyeCharm, etc., and then, when they are ready to submit computing jobs to larger facilities (HPC, HTC, Grid networks, etc.), do so but actually committing their code to an appropriate GIT repository. I guess you can see how something like Concourse may be useful for implementing this mode of working.

So if anyone can point me to some useful materials that will save me from re-inventing the wheel, I’d appreciate that very much.

Best,

Oren Shani