原创 bloodzer0 2020-04-27 20:35
记录使用SonarQube实现DevSecOps中代码扫描的经历
# 解决依赖,SonarQube7.9必须是java11才可以运行``yum install java-11-openjdk.x86_64 java-11-openjdk-devel.x86_64 -y
# 安装PG数据库,新版本已经不支持MySQL了``wget https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpmrpm -ivh pgdg-redhat-repo-latest.noarch.rpm``yum install postgresql10.x86_64 postgresql10-server.x86_64 postgresql10-libs.x86_64 postgresql10-libs.x86_64 postgresql-jdbc.noarch -y
# 初始化PG``/usr/pgsql-10/bin/postgresql-10-setup initdb
# 允许本机连接``vim /var/lib/pgsql/10/data/pg_hba.conf
# 启动PG``systemctl enable postgresql-10.service``systemctl start postgresql-10.service
# 配置PG``su - postgres``psql
# 将template1取消设置为模板``UPDATE pg_database SET datistemplate = FALSE WHERE datname = 'template1';
# 删除template1``DROP DATABASE template1;
# 重新创建template1,并设置默认编码``CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UTF8';
# 将template1重新设置为模板``UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template1';
# 创建用户``create user sonarqube with password 'sonarqube';``alter role sonarqube login;
# 创建数据库``create database sonarqube encoding='UTF8' owner sonarqube;
# 创建用户``useradd sonarqube``passwd sonarqube``su - sonarqube
# 安装SonarQube``wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.0.zipunzip sonarqube-8.0.zip && cd sonarqube-8.0
# 修改配置文件``vim /home/sonarqube/sonarqube-8.0/conf/sonar.properties
sonar.jdbc.username=sonarqube``sonar.jdbc.password=sonarqube``sonar.jdbc.url=jdbc:postgresql://127.0.0.1/sonarqube
# 配置Java版本``# 由于8.0需要Java11,但是安装PG10的时候系统会默认安装Java1.8,所以需要配置Java地址``vim /home/sonarqube/sonarqube-8.0/conf/wrapper.conf
# 我是yum安装的java11,如果大家是配置或者编译的需要使用对应的java地址``wrapper.java.command=/usr/lib/jvm/java-11-openjdk-11.0.6.10-1.el7_7.x86_64/bin/java
# 启动服务``./bin/linux-x86-64/sonar.sh start``访问:http://10.10.10.12:9000,账号密码:admin/admin。
- 报错信息如下:
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]``[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
- 解决方案如下:
vim /etc/sysctl.conf
# 添加如下内容``vm.max_map_count=655360
# 立即生效``sysctl -p
vim /etc/security/limits.conf
# 添加如下内容``* soft nofile 65536``* hard nofile 65536
# limits不需要重启,但是需要重新登录
# 下载SonarScanner``wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.2.0.1873-linux.zip
unzip sonar-scanner-cli-4.2.0.1873-linux.zip -d /home/sonarqube/
- 在SonarQube管理⻚面创建一个Token:68bec86cee8ec9a9ad8621592670c9939e6ba71e
- 执行扫描
/home/sonarqube/sonar-scanner-4.2.0.1873-linux/bin/sonar-scanner -Dsonar.host.url=http://127.0.0.1:9000 -Dsonar.login=68bec86cee8ec9a9ad8621592670c9939e6ba71e -Dsonar.projectKey=java-sec-code_scanner -Dsonar.java.binaries=/home/sonarqube/projects/java-sec-code
- 在应用市场中下载安装
- 在GitHub下载jar文件放置到SonarQube的插件目录中
mvn clean package -DskipTests
/home/sonarqube/sonar-scanner-4.2.0.1873-linux/bin/sonar-scanner -Dsonar.host.url=http://127.0.0.1:9000 -Dsonar.login=68bec86cee8ec9a9ad8621592670c9939e6ba71e -Dsonar.projectKey=java-sec-code_findbugs -Dsonar.java.binaries=/home/sonarqube/projects/java-sec-code
# 不更新漏洞库执行扫描``/home/sonarqube/dependency-check/bin/dependency-check.sh -n
# 只更新漏洞库库,不执行扫描``/home/sonarqube/dependency-check/bin/dependency-check.sh --updateonly
# 使用代理更新漏洞库``/home/sonarqube/dependency-check/bin/dependency-check.sh --updateonly --proxyserver host --proxyport port
# 编译项目``mvn clean package -DskipTests
# 生成xml报告``/home/sonarqube/dependency-check/bin/dependency-check.sh -s /home/sonarqube/projects/java-sec-code/target/java-sec-code-1.0.0.jar -f XML -o /home/sonarqube/reports/java-sec-code-report.xml
# 在5.3.2 版本中,小B遇到了一个坑就是第一项执行DependencyCheck时,当执行到 Central Analyzer 分析时非常缓慢(由于网络原因导致的),可以使用--disableCentral禁用,但是官方文档中不推荐禁用,所以只能在执行是使用代理或调整网络来解决
# 生成html报告``/home/sonarqube/dependency-check/bin/dependency-check.sh -s /home/sonarqube/projects/java-sec-code/target/java-sec-code-1.0.0.jar -f HTML -o /home/sonarqube/reports/java-sec-code-report.html
# 将报告导入SonarQube并执行SonarScanner扫描,这里我们使用的是FindBugs Security Audit审计规则``/home/sonarqube/sonar-scanner-4.2.0.1873-linux/bin/sonar-scanner -Dsonar.host.url=http://127.0.0.1:9000 -Dsonar.login=68bec86cee8ec9a9ad8621592670c9939e6ba71e -Dsonar.projectKey=java-sec-code_dependency -Dsonar.java.binaries=/home/sonarqube/projects/java-sec-code -Dsonar.dependencyCheck.reportPath=/home/sonarqube/reports/java-sec-code-report.xml -Dsonar.dependencyCheck.htmlReportPath=/home/sonarqube/reports/java-sec-code-report.html
stages: - sast
sast: stage:sast script: - mvn clean package -DskipTests # 编译项目 - mkdir $PWD/reports # 创建报告目录 - /home/gitlab-runner/dependency-check/bin/dependency-check.sh -s $PWD/ -f XML -o $PWD/reports/dependency-check-report.xml # 执行扫描,并生成xml报告 - /home/gitlab-runner/dependency-check/bin/dependency-check.sh -s $PWD/ -f HTML -o $PWD/reports/dependency-check-report.html # 执行扫描,并生成html报告 - sonar-scanner -Dsonar.host.url=http://10.10.10.12:9000 -Dsonar.login=token -Dsonar.projectKey=$CI_PROJECT_NAME:$CI_COMMIT_REF_NAME -Dsonar.java.binaries=$PWD -Dsonar.dependencyCheck.reportPath=$PWD/reports/dependency-check-report.xml -Dsonar.dependencyCheck.htmlReportPath=$PWD/reports/dependency-check-report.html # 执行扫描,并将报告传输给SonarQube tags: - security
import requests
def get_vul(project_name): url = "http://172.16.26.13:9000/api/issues/search?project=%s&types=VULNERABILITY" % project_name vulnerability_number = requests.get(url,auth=('admin','admin')).json()['paging']['total']
`for row in requests.get(url,auth=('admin','admin')).json()['issues']:` `if row['status'] == "OPEN":` `if row['severity'] == "MAJOR" or row['severity'] == "CRITICAL" or row['severity'] == "BLOCKER":` `alarm_data = ""` `else:` `pass` `else:` `pass`
`print("漏洞数量: ", vulnerability_number)`
if __name__ == '__main__': project_name = "java-sec-code_dependency" get_vul(project_name)
- 使用阿里云RDS,新版本不支持MySQL了可以使用阿里云的PostgreSQL。
sonar.jdbc.url=jdbc:mysql://address:3306/sonar?useUnicode=true&characterEncoding=utf8
ERROR: Unable to load component class org.sonar.scanner.report.MetadataPublisher``ERROR: Caused by: Unable to load component class org.sonar.scanner.rule.QualityProfiles
用CI_COMMIT_REF_NAME,但是如果使用这个变量在项目名称中包含/、#等特殊字符时,Gitlab-CI的pipeline会执行失败。这个时候我们选择CI_COMMIT_REF_SLUG(先将$CI_COMMIT_REF_NAME的值转换成小写,最大不能超过63个字节,然后把除了0-9和a-z的其他字符转换成-。在URLs和域名名称中使用。)
- SonarQube之gitlab-plugin配合gitlab-ci完成每次commit代码检测:https://blog.csdn.net/aixiao yang168/article/details/78115646
- gitlab-ci/cd Dependency依赖扫描两种方式(五):https://www.linuxea.com/1837.html