How to Run Jobs Sequentially one after other?


#1

Hii,
I’m new to concourse and my doubt is, i want to run all the jobs one after another with one condition that is “previous job should be successful”, the method i know is using “resources” and adding a “passed attribute” we can do that, but i don’t want to use resources and passed attribute method, because my job doesn’t need any resources, how can i achieve that ??
Here is the code:
groups:

  • name: development
    jobs:
    • stage-a
    • stage-b
  • name: deployment
    jobs:
    • stage-c
      jobs:
  • name: stage-a
    serial: true
    plan:
    • task: stage-a
      config:
      platform: linux
      image_resource:
      type: docker-image
      source: {repository: busybox}
      run:
      path: env
      args: []
      params:
      STG1_NAME: ((stage-1))
  • name: stage-b
    plan:
    • task: stage-b
      config:
      platform: linux
      image_resource:
      type: docker-image
      source: {repository: busybox}
      run:
      path: env
      args: []
      params:
      STG2_NAME: ((stage-2))
  • name: stage-c
    plan:
    • task: stage-c
      config:
      platform: linux
      image_resource:
      type: docker-image
      source: {repository: busybox}
      run:
      path: env
      args: []
      params:
      STG3_NAME: ((stage-3))

      The output will look like this (image), but in this way i need to execute the job manually, they won’t execute one after other, i want them to execute one after the other.

#2

If you want them to be jobs, then you must use a resource like S3 (or a S3-compatible blob storage like Minio) with the “passed” attribute as you mention.

If you want to do the exercise of avoiding at all cost a resource (but why???), then you can use only one job and convert the existing jobs to tasks. Tasks within a single job run by default sequentially in the order they are written.


#3

So, the only way is to convert the remaining jobs into tasks and make it as one job. Thanks. Is there any place where i can learn all the terms of concourse and there usage.
Here terms means jobs, groups, task, plan etc…


#4

hi and welcome to Concourse!

Stark & Wayne made some very good beginner tutorials:

If you still got a problem and seek help, don’t hesitate to ask here :slight_smile:

best,
D


#5

The official documentation is at https://concourse-ci.org/reference.html
As any reference documentation it can be dry, but I found that it pays to go through it, I have found hidden gems there. The link about the tutorial posted by gdenn is another good one.


#6

Hi thank you so much for reply,
i have been following the same tutorial, but i’m unable to execute “Task Scripts” part, i’m getting error while running “Task Scripts”.
I have created a new topic of this issue which i’m facing. Kindly look into it and help me out.
Here is the screenshot the error which i’m facing while running “Task Scripts” (.sh file as a part of .yml file).


#7

i have been following the same tutorial, but i’m unable to execute “Task Scripts” part, i’m getting error while

answered you in your other thread already :wink:

best,
D


#8

Another thing that took me a while to figure out:

  • A pipeline is made of jobs. You can have fan-out (parallelism) and fan-in (synchronization).
  • A job is made of tasks.
  • A task is the unit of execution. On Linux, the task runs in a container. On non-Linux, the task runs directly on the host OS.
  • A task is also the unit of scheduling, so if a job is made of 5 tasks and you have 5 workers, it might be that each task lands on a different worker. This also depends on the scheduling, that by default is influenced by the presence on the worker of the image for a given task.
  • Within a job, I/O between tasks is done without requiring a resource (as we discussed above)
  • Within a pipeline, I/O between jobs requires a resource like S3 or a compatible blob storage
  • In a Concourse deployment, also if you build only for Windows, you still need at least one Linux worker because all Concourse resources, since they are implemented as container, require a Linux worker to be executed.
  • Concourse doesn’t have a build queue: everything you throw at it will be scheduled immediately. This can be a problem sometimes. Adding workers can alleviate but not resolve this.
  • Most important thing is to monitor your Concourse deployment, see https://concourse-ci.org/metrics.html. This is not unique to Concourse, any distributed system requires monitoring.

#9

Thanks for Sharing the Info.