阅读背景:

在AWS Ubuntu实例中为NodeJS部署设置GitLab CI

来源:互联网 

I have an AWS Ubuntu instance which has GitLab CE configured. Now I want to configure GitLab CI to deploy my NodeJS app after each commit.I don't have any proper step by step solution for this.

我有一个配置了GitLab CE的AWS Ubuntu实例。现在我想配置GitLab CI以在每次提交后部署我的NodeJS应用程序。我没有任何适当的逐步解决方案。

My NodeJS app running in /var/www/mean/my-app on https://myapp.mydomain.com and the hosting is handled by Apache Proxy,

我的NodeJS应用程序在https://myapp.mydomain.com上的/ var / www / mean / my-app中运行,托管由Apache Proxy处理,

<VirtualHost *:80>
    ServerAdmin [email protected]
    ServerName gitlab.mydomain.com
    ServerAlias www.gitlab.mydomain.com

    ServerSignature Off

    ProxyPreserveHost On

    AllowEncodedSlashes NoDecode

    <Location />
       Require all granted
       ProxyPassReverse https://localhost:8080
       ProxyPassReverse https://gitlab.mydomain.com/
    </Location>

    RewriteEngine on

    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
    RewriteCond %{REQUEST_URI} ^/uploads/.*
    RewriteRule .* https://127.0.0.1:8080%{REQUEST_URI} [P,QSA,NE]

    DocumentRoot /home/git/gitlab/public

    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
    ErrorLog /var/log/apache2/gitlab_error.log
    CustomLog /var/log/apache2/gitlab_forwarded.log common_forwarded
    CustomLog /var/log/apache2/gitlab_access.log combined env=!dontlog
    CustomLog /var/log/apache2/gitlab.log combined
 </VirtualHost>

And the app is bootstrapped using forever module

该应用程序使用forever模块进行自举

forever start app.js

The gitlab config check sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production gives,

gitlab配置检查sudo -u git -H bundle exec rake gitlab:check RAILS_ENV = production give,

Checking GitLab Shell ...

GitLab Shell version >= 4.0.0 ? ... OK (4.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
dev / my-app ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/git/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ... 
dev / my-app ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 1

Checking GitLab ... Finished

I used to login to the instance using SSH from my system,

我以前使用SSH从我的系统登录到实例,

ssh -i API-Key.pem [email protected]

Created key using command

使用命令创建密钥

ssh-keygen -t rsa

Runner config on /etc/gitlab-runner/config.toml

/etc/gitlab-runner/config.toml上的Runner配置

concurrent = 1
check_interval = 0

[[runners]]
  name = "Production Runner"
  url = "https://gitlab.mydomain.com/ci"
  token = "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
  executor = "ssh"
  [runners.ssh]
    user = "ubuntu"
    host = "ip-XXX-XX-XX-XXX"
    identity_file = "/home/ubuntu/.ssh/id_rsa"
  [runners.cache]

Code on .gitlab-ci.yml

.gitlab-ci.yml上的代码

test_async:   
 script:    
  - npm install

Because of my bad configuration, the runner gives error,

由于配置错误,跑步者会出错,

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)
Using SSH executor...
ERROR: Preparation failed: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
Will be retried in 3s ...

My confusions are:

我的困惑是:

What should be the content of .gitlab-ci.yml file to deploy the committed code to the application location?

.gitlab-ci.yml文件的内容应该是什么,以将已提交的代码部署到应用程序位置?

And how to configure a runner for this? If I have to use an ssh runner, what should be the configurations over there?

以及如何为此配置跑步者?如果我必须使用ssh runner,那里的配置应该是什么?

Update:

更新:

After providing .pem file as identity_file, getting the following error

提供.pem文件作为identity_file后,收到以下错误

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)
Using SSH executor...
Running on ip-xxx-xx-xx-xxx via ip-xxx-xx-xx-xxx...
Cloning repository...
Cloning into 'builds/a92f1b91/0/dev/my-app'...
fatal: unable to access 'https://gitlab-ci-token:[email protected]/dev/my-app.git/': The requested URL returned error: 500
ERROR: Build failed: Process exited with: 1. Reason was:  ()

Now there is a problem, git clone with http not working but ssh cloning works.

现在有一个问题,git克隆使用http无法工作,但ssh克隆工作。

Note: Both gitlab and build environment are same host (same aws instance)

注意:gitlab和构建环境都是相同的主机(相同的aws实例)

Bug reported in GitLab as well (http clone issue).

GitLab中也报告了Bug(http克隆问题)。

1 个解决方案

#1


4  

In your /etc/gitlab-runner/config.toml

在/etc/gitlab-runner/config.toml中

concurrent = 1
check_interval = 0

[[runners]]
  name = "Production Runner"
  url = "https://gitlab.mydomain.com/ci"
  token = "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
  executor = "ssh"
  [runners.ssh]
    user = "ubuntu"
    host = "ip-XXX-XX-XX-XXX"
    identity_file = "/home/ubuntu/.ssh/id_rsa"
  [runners.cache]

You define

你定义

  • host
  • 主办
  • user
  • 用户
  • and identity file
  • 和身份文件

host should be your Build host IP (in other words where you are going to execute your build)
user should be your user on Build host. Not on gitlab host.

host应该是您的Build主机IP(换句话说,您要执行构建的位置)用户应该是Build主机上的用户。不在gitlab主机上。

You can test how your password less ssh works by

您可以测试您的密码减去ssh的工作方式

  1. Login to gitlab host as root
  2. 以root身份登录gitlab主机
  3. ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@ip-XXX-XX-XX-XXX
  4. ssh -i /home/ubuntu/.ssh/id_rsa ubuntu @ ip-XXX-XX-XX-XXX

If that works and doesn't ask you for a password - all is good.
If that breaks - means you didn't setup password less auth correctly.

如果这样做并且没有要求您输入密码 - 一切都很好。如果中断 - 意味着您没有正确设置密码减去auth。

The easiest way to setup password less public key based auth is to use command called

设置密码少于基于公钥的auth的最简单方法是使用被调用的命令

ssh-copy-id

For example I want to setup password less ssh auth between my gitlab and my build host.
My build host ip is 192.168.0.42 and host name is build.home
I already have my id_rsa and id_rsa.pub generated under /home/ubuntu/.ssh on gitlab host.

例如,我想在我的gitlab和我的构建主机之间设置密码少ssh auth。我的构建主机IP是192.168.0.42,主机名是build.home我已经在gitlab主机上的/home/ubuntu/.ssh下生成了我的id_rsa和id_rsa.pub。

Now let's push our public key from gitlab host to our build host. First time it will ask you for a password.

现在让我们将我们的公钥从gitlab主机推送到我们的构建主机。第一次它会要求您输入密码。

[root@gitlab ~]# ssh-copy-id -i /home/ubuntu/.ssh/id_rsa.pub [email protected]
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.  

Note that in above example I was pushing public key to remote host.

请注意,在上面的示例中,我将公钥推送到远程主机。

But when I will be now connecting to this remote host - I will specify my private key.

但是当我现在要连接到这个远程主机时 - 我将指定我的私钥。

[root@gitlab ~]# ssh -i /home/ubuntu/.ssh/id_rsa [email protected]
[ubuntu@build ~]$ hostname
build.home

Try testing your public key auth between gitlab host and remote host and update your question.

尝试在gitlab主机和远程主机之间测试您的公钥验证并更新您的问题。

Resources:

资源:

Edit 1:

编辑1:

Here is my config.
My gitlab host is called gitlab.home 192.168.0.41
And I have another VM called sshbuild.home 192.168.0.43

这是我的配置。我的gitlab主机名为gitlab.home 192.168.0.41我有另一个名为sshbuild.home的虚拟机192.168.0.43

Below is how I added ssh runner

以下是我添加ssh runner的方法

Step 1. Install on my gitlab.home yum install gitlab-ci-multi-runner and register my remote sshbuild.home VM as ssh runner

步骤1.在我的gitlab.home上安装yum install gitlab-ci-multi-runner并将我的远程sshbuild.home VM注册为ssh runner

I also need to make sure that password less auth works between my gitlab.home and sshbuild.home, so

我还需要确保密码少auth在我的gitlab.home和sshbuild.home之间工作,所以

[root@gitlab gitlab-runner]# ssh-copy-id 192.168.0.43
The authenticity of host '192.168.0.43 (192.168.0.43)' can't be established.
ECDSA key fingerprint is b4:6a:1b:72:d1:7d:1f:34:f7:bb:ef:ad:69:42:11:13.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.0.43'"
and check to make sure that only the key(s) you wanted were added.

[root@gitlab gitlab-runner]# ssh 192.168.0.43
Last login: Fri Nov 18 17:05:06 2016 from 192.168.0.101
[root@sshbuild ~]# exit

Then I disabled my other runner (shell runner) and made new SSH runner project specific just to make sure that when I commit - it will be executed on the ssh runner

然后我禁用了我的其他跑步者(shell跑步者)并制作了新的SSH跑步项目,以确保当我提交时 - 它将在ssh跑步者上执行

And I do commit and voila - we have our successfull test which was run on sshbuild.home host

而且我确实承诺并且瞧 - 我们的成功测试是在sshbuild.home主机上运行的

Here is several links which might help with a better understanding of this topic

这里有几个链接可能有助于更好地理解这个主题

P.S: And here is my /etc/gitlab-runner/config.toml file

P.S:这是我的/etc/gitlab-runner/config.toml文件

[root@gitlab gitlab-runner]# cat /etc/gitlab-runner/config.toml 
concurrent = 1
check_interval = 0

[[runners]]
  name = "sshbuild"
  url = "https://gitlab.home/"
  token = "2bc1825d8fbde09fd632637c60e9e7"
  executor = "ssh"
  [runners.ssh]
    user = "root"
    host = "192.168.0.43"
    port = "22"
    identity_file = "/root/.ssh/id_rsa"
  [runners.cache]

P.S: I have similar error as you if I disable HTTP for my repo under Settings in the web interface. However error is not 500 but 403.

P.S:如果我在Web界面的“设置”下禁用了我的仓库的HTTP,我会遇到类似的错误。但错误不是500而是403。

Edit 2:

编辑2:

Now I will cover .gitlab-ci.yml based on a simple HelloWorld project
In my HelloWorld I have file called server.js which when run from node - will simply create Web Server running on port 3000 and will reply with "Hello World" on GET requests.

现在我将基于一个简单的HelloWorld项目来介绍.gitlab-ci.yml在我的HelloWorld中,我有一个名为server.js的文件,当它从节点运行时 - 将只创建在端口3000上运行的Web服务器,并将回复“Hello World” GET请求。

 1  const http = require('http');
 2  
 3  const hostname = '0.0.0.0';
 4  const port = 3000;
 5  
 6  const server = http.createServer((req, res) => {
 7    res.statusCode = 200;
 8    res.setHeader('Content-Type', 'text/plain');
 9    res.end('Hello World!\n');
10  });
11  
12  server.listen(port, hostname, () => {
13    console.log(`Server running at https://${hostname}:${port}/`);
14  });

My goal is to be able to run a test case against it. In this case I will run simple

我的目标是能够针对它运行测试用例。在这种情况下,我会运行简单

curl localhost:3000 | grep "Hello World" 

But I need to put it into a separate script which will have exit status 0 on success and non zero on failure

但我需要把它放到一个单独的脚本中,成功时退出状态为0,失败时退出非零

cat -n simpletest.sh 
     1  #!/bin/bash
     2  
     3  cleanup () 
     4  {
     5   count=`netstat -anp|grep ":3000"|grep LISTEN|awk '{print $NF}'|cut -d\/ -f1|wc -l`
     6   if [ $count -ne 0 ]
     7    then 
     8     pid=`netstat -anp|grep ":3000"|grep LISTEN|awk '{print $NF}'|cut -d\/ -f1`; 
     9     echo "Need to kill PID $pid"; 
    10     kill $pid
    11   fi
    12  }
    13  
    14  echo "Running simple test"
    15  curl localhost:3000|grep "Hello World"
    16  if [ $? -eq 0 ]
    17   then
    18    echo "Test was successfull"
    19    echo "Clean up node.js process"
    20    cleanup
    21    exit 0
    22   else
    23    echo "Test failed"
    24    echo "Clean up node.js process"
    25    cleanup
    26    exit 1
    27  fi

Now let's cover my .gitlab-ci.yml

现在让我们覆盖我的.gitlab-ci.yml

cat -n .gitlab-ci.yml
     1  test:
     2  
     3    before_script:
     4    - echo "Before script"
     5    - hostname
     6    - /bin/bash cleanup.sh
     7  
     8    script: 
     9    - echo "Main Script"
    10    - node server.js &
    11    - sleep 3
    12    - /bin/bash simpletest.sh

I have a single job called test.
In before_script it runs cleanup.sh script which simply kills PID listening on port 3000 in case such found.

我有一个叫做test的工作。在before_script中,它运行cleanup.sh脚本,它会在发现的情况下直接杀死端口3000上的PID监听。

cat -n cleanup.sh 
     1  #!/bin/bash
     2  count=`netstat -anp|grep ":3000"|grep LISTEN|awk '{print $NF}'|cut -d\/ -f1|wc -l`
     3  if [ $count -ne 0 ]
     4   then 
     5    pid=`netstat -anp|grep ":3000"|grep LISTEN|awk '{print $NF}'|cut -d\/ -f1`; 
     6    echo "Need to kill PID $pid"; 
     7    kill $pid
     8  fi
     9  exit 0

And under script: it runs node with my server.js, gives it 3 seconds to start and then runs test against it.
This test will also take care of killing node PID after test is done.

在脚本下:它使用我的server.js运行节点,给它3秒钟启动,然后对它运行测试。该测试还将在测试完成后处理节点PID。

So let's commit and check status of the build

所以让我们提交并检查构建的状态

And now let's change our server.js to output not "Hello World" but "HelloWorld", so there is no space in between. I expect that my test case will fail since it expects literally "Hello World". And it does fail.

现在让我们改变我们的server.js输出不是“Hello World”而是“HelloWorld”,所以两者之间没有空格。我希望我的测试用例会失败,因为它预计字面意思是“Hello World”。它确实失败了。

This is the most simplistic CI use case I could come up with.
Now if based on the status of the test case you would like to deploy code to another environment - you would have to start using

这是我能提出的最简单的CI用例。现在,如果基于测试用例的状态,您希望将代码部署到另一个环境 - 您必须开始使用

  • stages and
  • 阶段和
  • environment
  • 环境

So your .gitlab-ci.yml would turn into something like this (real working example)

所以你的.gitlab-ci.yml会变成这样的东西(实际的工作示例)

cat -n .gitlab-ci.yml
     1  stages:
     2  - test
     3  - deploy
     4  
     5  run_test_case:
     6   stage: test
     7   before_script:
     8   - echo "Before script"
     9   - hostname
    10   - /bin/bash cleanup.sh
    11  
    12   script: 
    13   - echo "Main Script"
    14   - node server.js &
    15   - sleep 3
    16   - /bin/bash simpletest.sh
    17  
    18  deploy_to_production:
    19   stage: deploy
    20   script: 
    21   - echo "Run code here to do production deployment"
    22   environment:
    23    name: production

Which upon git push will succeed.
On line 21 I simply ran echo, but this could be replaced with a script which will do push to your remote staging or production environment.

哪个git push会成功。在第21行,我只是运行echo,但这可以用脚本替换,该脚本将推送到远程登台或生产环境。


分享到: