Copy resources build output to a remote server

Hi,

I have created a pipeline to fetch the latest angular code from git repository and create a build.
It successfully creating the build and the next step is to copy the build folder to a remote server location.
I am using linux platform and using scp to copy build output folder to remote server but for some reason it is not copying the files to remote server, also the output at build console is not showing anything. And nothing copied to remote server.
Is there any better way to do that, to copy resources through concourse pipeline to a remote server?

My build.yml contain following content.
`


image_resource:
  type: docker-image
  source:
    repository: node
   
inputs: 
  - name: repo
outputs:
  - name: build-output

run:
  path: sh
  args:
  - -exc
  - |
    set -e -u -x
    cat /etc/os-release
 
    
    echo "Moving to project"
    cd repo/angular-web
    
    echo "Installing Packages"
    npm install
    
    echo "Build Angular Code"
    npm run build-prod
          
    echo "Copying dist folder in output folder(exist on root)"
   sshpass -p MYPASSWORD scp -r ./dist MYUSERNAME@MYSERVERADDRESS:/app/www/environments/qa/test1
`

Hello, welcome to Concourse!
Could you please edit and format your post so that the YAML code is formatted as code? This forum (which is an instance of https://www.discourse.org/) supports markdown. See https://commonmark.org/help/ for a quick reference. Thanks.

I am done editing, now how can I get the quick reply for this post.

The canonical way to send data to an external source in Concourse is with resources. It’s generally considered an anti-pattern to send stuff directly from a task.

I’m not sure why your task isn’t working as written. Its curious you aren’t seeing any output with set -x.

Just to note, you don’t need both - -exc and set -e -x as they do the same thing.

Here everybody is a volunteer. You get an answer when

  1. somebody has some time to look at your question
  2. you make the effort to make your post understandable
  3. You ask politely

Thanks @crsimmons from resources, I can see that there are ftp and rsync resources that I can used. Let me try them.

@crsimmons I have following pipeline.yml, it builds but failed for ftpupload, any thought on it.


Contents of pipeline.yml

- name: deploy-qa
  jobs:
  - ilex-angular-web-qa
resources:
- name: ilex-source-code
  type: git
  source:
    branch: ilex-concourse
    private_key: |
      ((github-key.key))
    uri: git@xyz.git
- name: ftpupload
  type: ftp
  source:
    uri: ftp://userName:password@serveraddress.com:/app/www/environments/qa/test/

resource_types:
- name: pull-request
  type: docker-image
  source:
    repository: jtarchie/pr
- name: ftp
  type: docker-image
  source:
    repository: aequitas/ftp-resource

jobs:
- name: ilex-angular-web-qa
  public: true
  plan:
  - get: repo
    resource: ilex-source-code
  - task: build
    privileged: true
    file: repo/angular-web/ci/Tasks/build.yml
  - put: ftpupload
    params:
        file: build-output/dist

I assume you missed copying groups: as the first line of that pipeline when you posted.

I can’t see anything wrong with your pipeline code and haven’t seen your particular error before. I noticed in your original post your task yml didn’t have platform: linux specified and that is in your error now. If you add that line to the task does it make a difference. Also, what is the output of fly workers -d against your concourse?

@crsimmons thanks for the reply. Can you share any example of resource, where we can share our build-output folders to an external resource.

I’m not sure what you mean. Generally any resource with an out implementation can send artifacts from your pipeline outputs to somewhere else. As for sharing build outputs with those resources, all gets and task outputs are automatically mounted to put containers when they are started.

pipeline.yml

- name: deploy-qa
  jobs:
  - ilex-angular-web-qa
resources:
- name: ilex-source-code
  type: git
  source:
    branch: ilex-concourse
    private_key: |
      ((github-key.key))
    uri: git@github.xyz.com:abc-software-development/ikg.git


resource_types:
- name: pull-request
  type: docker-image
  source:
    repository: jtarchie/pr
- name: ftp
  type: docker-image
  source:
    repository: aequitas/ftp-resource

jobs:
- name: ilex-angular-web-qa
  public: true
  plan:
  - get: repo
    resource: ilex-source-code
  - task: build
    privileged: true
    file: repo/angular-web/ci/Tasks/build.yml

build.yml

platform: linux

image_resource:
  type: docker-image
  source:
    repository: node
   # repository: tenjaa/java-chrome-nodejs
inputs: 
  - name: repo
outputs:
  - name: build-output

run:
  path: sh
  args:
  - -exc
  - |
    set -e -u -x
    cat /etc/os-release
 
    
    echo "Moving to project"
    cd repo/angular-web
    
    echo "Installing Packages"
    npm install
    node --version
    npm --version
    npm install -g n   # Install n globally
    n 10.15.3          # Install and use v10.15.3 
    node --version
    npm rebuild node-sass

    echo "Build Angular Code"
    npm run build-prod
          
    echo "Copying dist folder in output folder(exist on root)"
    whoami 
    apt-get update
    apt-cache search sshpass /app/www/environments/qa/test
    apt-get install sshpass
    cp -R ./dist ../../../build-output
    echo "copied to build-output"
    pwd
    ls
    cd dist
    ls
    cd ../
    cp -R ./dist ../../../build-output

    echo "done"

This pipeline executes correctly, it create my expected files in build-output.
Now the main reason is to move those build files through pipeline to a remote server.
I have used ftp but my network team didn’t allow me to ftp on those server.
Can you suggest some other solution?

@crsimmons a soft reminder to this thread

@shehraz you are supposed to do your part: learning about Concourse. From what you ask, you are looking for a solution without doing the effort to learn anything.

1 Like

@marco-m you are right but I was running out of time. Anyway Thanks for all support.

I’m currently working with a client so I have limited time to check this forum. As I mentioned before there’s nothing functionally wrong with with your pipeline as written as far as I can tell. Having your network team tell you something can’t be done is not a Concourse problem.

Without knowing what you can do or what storage options you have available I can’t recommend anything for you beyond what I have already. Find a resource other than ftp that works for you. The whole point of resources is putting something from the Concourse container somewhere external.

You can find some resources here or you can Google for some community ones or you can make your own.