GithubBackup/httprunner
1
0
Fork 0
mirror of https://github.com/httprunner/httprunner.git synced 2026-05-13 08:59:44 +08:00
Code Issues Packages Projects Releases Wiki Activity

doc: add docs to repo

Browse Source
This commit is contained in:
debugtalk
2019-11-01 23:16:58 +08:00
parent d15d811b6f
commit 8d64fede9b
85 changed files with 5116 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

218
docs/run-tests/cli.md Normal file
View File

@@ -0,0 +1,218 @@
HttpRunner 在命令行中启动测试时，通过指定参数，可实现丰富的测试特性控制。
```text
$ hrun -h
usage: hrun [-h] [-V] [--log-level LOG_LEVEL] [--log-file LOG_FILE]
[--dot-env-path DOT_ENV_PATH] [--report-template REPORT_TEMPLATE]
[--report-dir REPORT_DIR] [--failfast] [--save-tests]
[--startproject STARTPROJECT]
[--validate [VALIDATE [VALIDATE ...]]]
[--prettify [PRETTIFY [PRETTIFY ...]]]
[testcase_paths [testcase_paths ...]]
One-stop solution for HTTP(S) testing.
positional arguments:
testcase_paths testcase file path
optional arguments:
-h, --help show this help message and exit
-V, --version show version
--log-level LOG_LEVEL
Specify logging level, default is INFO.
--log-file LOG_FILE Write logs to specified file path.
--dot-env-path DOT_ENV_PATH
Specify .env file path, which is useful for keeping
sensitive data.
--report-template REPORT_TEMPLATE
specify report template path.
--report-dir REPORT_DIR
specify report save directory.
--failfast Stop the test run on the first error or failure.
--save-tests Save loaded tests and parsed tests to JSON file.
--startproject STARTPROJECT
Specify new project name.
--validate [VALIDATE [VALIDATE ...]]
Validate JSON testcase format.
--prettify [PRETTIFY [PRETTIFY ...]]
Prettify JSON testcase format.
```
## 指定测试用例路径
使用 HttpRunner 指定测试用例路径时，支持多种方式。
使用 hrun 命令外加单个测试用例文件的路径，运行单个测试用例，并生成一个测试报告文件：
```text
$ hrun filepath/testcase.yml
```
将多个测试用例文件放置到文件夹中，指定文件夹路径可将文件夹下所有测试用例作为测试用例集进行运行，并生成一个测试报告文件：
```text
$ hrun testcases_folder_path
```
## failfast
默认情况下，HttpRunner 会运行指定用例集中的所有测试用例，并统计测试结果。
> 对于某些依赖于执行顺序的测试用例，例如需要先登录成功才能执行后续接口请求的场景，当前面的测试用例执行失败后，后续的测试用例也都必将失败，因此没有继续执行的必要了。
若希望测试用例在运行过程中，遇到失败时不再继续运行后续用例，则可通过在命令中添加`--failfast`实现。
```text
$ hrun filepath/testcase.yml --failfast
```
## 日志级别
默认情况下，HttpRunner 运行时的日志级别为`INFO`，只会包含最基本的信息，包括用例名称、请求的URL和Method、响应结果的状态码、耗时和内容大小。
```text
$ hrun docs/data/demo-quickstart-6.json
INFO Start to run testcase: testcase description
/api/get-token
INFO POST http://127.0.0.1:5000/api/get-token
INFO status_code: 200, response_time(ms): 9.08 ms, response_length: 46 bytes
.
/api/users/1548560655759
INFO POST http://127.0.0.1:5000/api/users/1548560655759
INFO status_code: 201, response_time(ms): 2.89 ms, response_length: 54 bytes
.
----------------------------------------------------------------------
Ran 2 tests in 0.019s
OK
INFO Start to render Html report ...
INFO Generated Html report: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/reports/1548560655.html
```
若需要查看到更详尽的信息，例如请求的参数和响应的详细内容，可以将日志级别设置为`DEBUG`，即在命令中添加`--log-level debug`。
```
$ hrun docs/data/demo-quickstart-6.json --log-level debug
INFO Start to run testcase: testcase description
/api/get-token
INFO POST http://127.0.0.1:5000/api/get-token
DEBUG request kwargs(raw): {'headers': {'User-Agent': 'python-requests/2.18.4', 'device_sn': 'W5ACRDytKRQJPhC', 'os_platform': 'ios', 'app_version': '2.8.6', 'Content-Type': 'application/json'}, 'json': {'sign': '2e7c3b5d560a1c8a859edcb9c8b0d3f8349abeff'}, 'verify': True}
DEBUG processed request:
> POST http://127.0.0.1:5000/api/get-token
> kwargs: {'headers': {'User-Agent': 'python-requests/2.18.4', 'device_sn': 'W5ACRDytKRQJPhC', 'os_platform': 'ios', 'app_version': '2.8.6', 'Content-Type': 'application/json'}, 'json': {'sign': '2e7c3b5d560a1c8a859edcb9c8b0d3f8349abeff'}, 'verify': True, 'timeout': 120}
DEBUG
================== request details ==================
url : 'http://127.0.0.1:5000/api/get-token'
method : 'POST'
headers : {'User-Agent': 'python-requests/2.18.4', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'device_sn': 'W5ACRDytKRQJPhC', 'os_platform': 'ios', 'app_version': '2.8.6', 'Content-Type': 'application/json', 'Content-Length': '52'}
body : b'{"sign": "2e7c3b5d560a1c8a859edcb9c8b0d3f8349abeff"}'
DEBUG
================== response details ==================
ok : True
url : 'http://127.0.0.1:5000/api/get-token'
status_code : 200
reason : 'OK'
cookies : {}
encoding : None
headers : {'Content-Type': 'application/json', 'Content-Length': '46', 'Server': 'Werkzeug/0.14.1 Python/3.6.5+', 'Date': 'Sun, 27 Jan 2019 03:45:16 GMT'}
content_type : 'application/json'
json : {'success': True, 'token': 'o6uakmubLrCbpRRS'}
INFO status_code: 200, response_time(ms): 9.28 ms, response_length: 46 bytes
DEBUG start to extract from response object.
DEBUG extract: content.token => o6uakmubLrCbpRRS
DEBUG start to validate.
DEBUG extract: status_code => 200
DEBUG validate: status_code equals 200(int) ==> pass
DEBUG extract: headers.Content-Type => application/json
DEBUG validate: headers.Content-Type equals application/json(str) ==> pass
DEBUG extract: content.success => True
DEBUG validate: content.success equals True(bool) ==> pass
.
/api/users/1548560716736
INFO POST http://127.0.0.1:5000/api/users/1548560716736
DEBUG request kwargs(raw): {'headers': {'User-Agent': 'python-requests/2.18.4', 'device_sn': 'W5ACRDytKRQJPhC', 'token': 'o6uakmubLrCbpRRS', 'Content-Type': 'application/json'}, 'json': {'name': 'user1', 'password': '123456'}, 'verify': True}
DEBUG processed request:
> POST http://127.0.0.1:5000/api/users/1548560716736
> kwargs: {'headers': {'User-Agent': 'python-requests/2.18.4', 'device_sn': 'W5ACRDytKRQJPhC', 'token': 'o6uakmubLrCbpRRS', 'Content-Type': 'application/json'}, 'json': {'name': 'user1', 'password': '123456'}, 'verify': True, 'timeout': 120}
DEBUG
================== request details ==================
url : 'http://127.0.0.1:5000/api/users/1548560716736'
method : 'POST'
headers : {'User-Agent': 'python-requests/2.18.4', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'device_sn': 'W5ACRDytKRQJPhC', 'token': 'o6uakmubLrCbpRRS', 'Content-Type': 'application/json', 'Content-Length': '39'}
body : b'{"name": "user1", "password": "123456"}'
DEBUG
================== response details ==================
ok : True
url : 'http://127.0.0.1:5000/api/users/1548560716736'
status_code : 201
reason : 'CREATED'
cookies : {}
encoding : None
headers : {'Content-Type': 'application/json', 'Content-Length': '54', 'Server': 'Werkzeug/0.14.1 Python/3.6.5+', 'Date': 'Sun, 27 Jan 2019 03:45:16 GMT'}
content_type : 'application/json'
json : {'success': True, 'msg': 'user created successfully.'}
INFO status_code: 201, response_time(ms): 2.77 ms, response_length: 54 bytes
DEBUG start to validate.
DEBUG extract: status_code => 201
DEBUG validate: status_code equals 201(int) ==> pass
DEBUG extract: headers.Content-Type => application/json
DEBUG validate: headers.Content-Type equals application/json(str) ==> pass
DEBUG extract: content.success => True
DEBUG validate: content.success equals True(bool) ==> pass
DEBUG extract: content.msg => user created successfully.
DEBUG validate: content.msg equals user created successfully.(str) ==> pass
.
----------------------------------------------------------------------
Ran 2 tests in 0.022s
OK
DEBUG No html report template specified, use default.
INFO Start to render Html report ...
INFO Generated Html report: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/reports/1548560716.html
```
## 保存详细过程数据
为了方便定位问题，运行测试时可指定 `--save-tests` 参数，即可将运行过程的中间数据保存为日志文件。
```text
$ hrun docs/data/demo-quickstart-6.json --save-tests
dump file: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/docs/data/logs/demo-quickstart-6.loaded.json
dump file: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/docs/data/logs/demo-quickstart-6.parsed.json
INFO Start to run testcase: testcase description
/api/get-token
INFO POST http://127.0.0.1:5000/api/get-token
INFO status_code: 200, response_time(ms): 11.42 ms, response_length: 46 bytes
.
/api/users/1548560768589
INFO POST http://127.0.0.1:5000/api/users/1548560768589
INFO status_code: 201, response_time(ms): 2.8 ms, response_length: 54 bytes
.
----------------------------------------------------------------------
Ran 2 tests in 0.028s
OK
dump file: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/docs/data/logs/demo-quickstart-6.summary.json
INFO Start to render Html report ...
INFO Generated Html report: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/reports/1548560768.html
```
日志文件将保存在项目根目录的 `logs` 文件夹中，生成的文件有如下三个（XXX为测试用例名称）：
- `XXX.loaded.json`：测试用例加载后的数据结构内容，加载包括测试用例文件（YAML/JSON）、debugtalk.py、.env 等所有项目文件，例如 [`demo-quickstart-6.loaded.json`](/data/logs/demo-quickstart-6.loaded.json)
- `XXX.parsed.json`：测试用例解析后的数据结构内容，解析内容包括测试用例引用（API/testcase）、变量计算和替换、base_url 拼接等，例如 [`demo-quickstart-6.parsed.json`](/data/logs/demo-quickstart-6.parsed.json)
- `XXX.summary.json`：测试报告生成前的数据结构内容，例如 [`demo-quickstart-6.summary.json`](/data/logs/demo-quickstart-6.summary.json)

157
docs/run-tests/load-test.md Normal file
View File

@@ -0,0 +1,157 @@
HttpRunner 通过复用 [`Locust`][Locust]，可以在无需对 YAML/JSON 进行任何修改的情况下，直接运行性能测试。
## 原理图
![](../images/loadtest-schematic-diagram.jpg)
## 安装依赖包
安装完成 HttpRunner 后，系统中会新增`locusts`命令，但不会同时安装 Locust。
在系统中未安装 Locust 的情况下，运行`locusts`命令时会出现如下提示。
```bash
$ locusts -V
WARNING Locust is not installed, install first and try again.
install command: pip install locustio
```
Locust 的安装方式如下：
```text
$ pip install locustio
```
安装完成后，执行 `locusts -V` 可查看到 Locust 的版本号。
```text
$ locusts -V
[2017-08-26 23:45:42,246] bogon/INFO/stdout: Locust 0.8a2
[2017-08-26 23:45:42,246] bogon/INFO/stdout:
```
执行 `locusts -h`，可查看到使用帮助文档。
```text
$ locusts -h
Usage: locust [options] [LocustClass [LocustClass2 ... ]]
Options:
-h, --help show this help message and exit
-H HOST, --host=HOST Host to load test in the following format:
http://10.21.32.33
--web-host=WEB_HOST Host to bind the web interface to. Defaults to '' (all
interfaces)
-P PORT, --port=PORT, --web-port=PORT
Port on which to run web host
-f LOCUSTFILE, --locustfile=LOCUSTFILE
Python module file to import, e.g. '../other.py'.
Default: locustfile
--csv=CSVFILEBASE, --csv-base-name=CSVFILEBASE
Store current request stats to files in CSV format.
--master Set locust to run in distributed mode with this
process as master
--slave Set locust to run in distributed mode with this
process as slave
--master-host=MASTER_HOST
Host or IP address of locust master for distributed
load testing. Only used when running with --slave.
Defaults to 127.0.0.1.
--master-port=MASTER_PORT
The port to connect to that is used by the locust
master for distributed load testing. Only used when
running with --slave. Defaults to 5557. Note that
slaves will also connect to the master node on this
port + 1.
--master-bind-host=MASTER_BIND_HOST
Interfaces (hostname, ip) that locust master should
bind to. Only used when running with --master.
Defaults to * (all available interfaces).
--master-bind-port=MASTER_BIND_PORT
Port that locust master should bind to. Only used when
running with --master. Defaults to 5557. Note that
Locust will also use this port + 1, so by default the
master node will bind to 5557 and 5558.
--expect-slaves=EXPECT_SLAVES
How many slaves master should expect to connect before
starting the test (only when --no-web used).
--no-web Disable the web interface, and instead start running
the test immediately. Requires -c and -r to be
specified.
-c NUM_CLIENTS, --clients=NUM_CLIENTS
Number of concurrent clients. Only used together with
--no-web
-r HATCH_RATE, --hatch-rate=HATCH_RATE
The rate per second in which clients are spawned. Only
used together with --no-web
-n NUM_REQUESTS, --num-request=NUM_REQUESTS
Number of requests to perform. Only used together with
--no-web
-L LOGLEVEL, --loglevel=LOGLEVEL
Choose between DEBUG/INFO/WARNING/ERROR/CRITICAL.
Default is INFO.
--logfile=LOGFILE Path to log file. If not set, log will go to
stdout/stderr
--print-stats Print stats in the console
--only-summary Only print the summary stats
--no-reset-stats Do not reset statistics once hatching has been
completed
-l, --list Show list of possible locust classes and exit
--show-task-ratio print table of the locust classes' task execution
ratio
--show-task-ratio-json
print json data of the locust classes' task execution
ratio
-V, --version show program's version number and exit
```
可以看出，`loucsts` 命令与 `locust` 命令的用法基本相同。
相比于 `locust` 命令，`loucsts`命令主要存在如下两项差异。
## 运行性能测试
在 `-f` 参数后面，`loucsts` 命令不仅可以指定 Locust 支持的 Python 文件，同时可以直接指定 YAML/JSON 格式的测试用例文件。在具体实现上，当 `-f` 指定 YAML/JSON 格式的测试用例文件时，会先将其转换为 Python 格式的 locustfile，然后再将 locustfile.py 传给 locust 命令。
```bash
$ locusts -f examples/first-testcase.yml
[2017-08-18 17:20:43,915] Leos-MacBook-Air.local/INFO/locust.main: Starting web monitor at *:8089
[2017-08-18 17:20:43,918] Leos-MacBook-Air.local/INFO/locust.main: Starting Locust 0.8a2
```
执行上述命令后，即完成了 Locust 服务的启动，后续就可以在 Locust 的 Web 管理界面中进行操作了，使用方式与 Locust 完全相同。
## 多进程运行模式
默认情况下，在 Locust 中如需使用 master-slave 模式启动多个进程（使用多核处理器的能力），只能先启动 master，然后再逐一启动若干个 slave。
```text
$ locust -f locustfile.py --master
$ locust -f locustfile.py --slave &
$ locust -f locustfile.py --slave &
$ locust -f locustfile.py --slave &
$ locust -f locustfile.py --slave &
```
在 HttpRunner 中，新增实现 `--processes` 参数，可以一次性启动 1 个 master 和多个 salve。若在 `--processes` 参数后没有指定具体的数值，则启动的 slave 个数与机器的 CPU 核数相同。
```bash
$ locusts -f examples/first-testcase.yml --processes 4
[2017-08-26 23:51:47,071] bogon/INFO/locust.main: Starting web monitor at *:8089
[2017-08-26 23:51:47,075] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,078] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,080] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,083] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,084] bogon/INFO/locust.runners: Client 'bogon_656e0af8e968a8533d379dd252422ad3' reported as ready. Currently 1 clients ready to swarm.
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_09f73850252ee4ec739ed77d3c4c6dba' reported as ready. Currently 2 clients ready to swarm.
[2017-08-26 23:51:47,084] bogon/INFO/locust.main: Starting Locust 0.8a2
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_869f7ed671b1a9952b56610f01e2006f' reported as ready. Currently 3 clients ready to swarm.
[2017-08-26 23:51:47,085] bogon/INFO/locust.runners: Client 'bogon_80a804cda36b80fac17b57fd2d5e7cdb' reported as ready. Currently 4 clients ready to swarm.
```
![](../images/locusts-full-speed.jpg)
Enjoy!
[Locust]: http://locust.io/

113
docs/run-tests/report.md Normal file
View File

@@ -0,0 +1,113 @@
使用 HttpRunner 执行完自动化测试后，会在当前路径的 reports 目录下生成一份 HTML 格式的测试报告。
## 默认情况
默认情况下，生成的测试报告文件会位于项目根目录的 reports 文件夹中，文件名称为测试开始的时间戳。
```bash
$ hrun docs/data/demo-quickstart-6.yml
INFO Start to run testcase: testcase description
/api/get-token
INFO POST http://127.0.0.1:5000/api/get-token
INFO status_code: 200, response_time(ms): 10.05 ms, response_length: 46 bytes
.
/api/users/1548561170497
INFO POST http://127.0.0.1:5000/api/users/1548561170497
INFO status_code: 201, response_time(ms): 2.88 ms, response_length: 54 bytes
.
----------------------------------------------------------------------
Ran 2 tests in 0.034s
OK
INFO Start to render Html report ...
INFO Generated Html report: /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner-docs-v2x/reports/1548561170.html
```
点击查看[测试报告](/data/reports/1548561170.html)。
## 默认报告样式
在 HttpRunner 中自带了一个 Jinja2 格式的报告模版，默认情况下，生成的报告样式均基于该模版（[httprunner/templates/default_report_template.html][default_report]）。
测试报告形式如下：
在 Summary 中，会罗列本次测试的整体信息，包括测试开始时间、总运行时长、运行的Python版本和系统环境、运行结果统计数据。
![](../images/report-demo-quickstart-1-overview.jpg)
在 Details 中，会详细展示每一条测试用例的运行结果。
点击测试用例对应的 log 按钮，会在弹出框中展示该用例执行的详细数据，包括请求的 headers 和 body、响应的 headers 和 body、校验结果、响应、响应耗时（elapsed）等信息。
![](../images/report-demo-quickstart-1-log1.jpg)
![](../images/report-demo-quickstart-1-log2.jpg)
若测试用例运行不成功（failed/error/skipped），则在该测试用例的 detail 中会出现 traceback 按钮，点击该按钮后，会在弹出框中展示失败的堆栈日志，或者 skipped 的原因。
![](../images/report-demo-quickstart-1-traceback.jpg)
点击查看[测试报告](/data/reports/1548561464.html)。
## 自定义
除了默认的报告样式，HttpRunner 还支持使用自定义的报告模板。
### 编写自定义模板（Jinja2格式）
自定义模板需要采用 [Jinja2][Jinja2] 的格式，其中可以使用的数据可参考[数据结构示例][summary_data]。
例如，我们需要在自定义模板中展示测试结果的统计数据，就可以采用如下方式进行描述：
```html
<tr>
<th>TOTAL</th>
<th>SUCCESS</th>
<th>FAILED</th>
<th>ERROR</th>
<th>SKIPPED</th>
</tr>
<tr>
<td>{{stat.testsRun}}</td>
<td>{{stat.successes}}</td>
<td>{{stat.failures}}</td>
<td>{{stat.errors}}</td>
<td>{{stat.skipped}}</td>
</tr>
```
在自定义报告模板时，可以参考 HttpRunner 的[默认报告模板][default_report]，要实现更复杂的模版功能，可参考 [Jinja2][Jinja2] 的使用文档。
### 使用自定义模板
使用自定义模版时，需要通过 `--report-template` 指定报告模板的路径，然后测试运行完成后，就会采用自定义的模板生成测试报告。
```bash
$ hrun docs/data/demo-quickstart-2.yml --report-template /path/to/custom_report_template
...
同上，省略
INFO render with html report template: /path/to/custom_report_template
INFO Start to render Html report ...
INFO Generated Html report: reports/1532078874.html
```
[Jinja2]: http://jinja.pocoo.org/docs/latest
[default_report]: https://github.com/HttpRunner/HttpRunner/blob/master/httprunner/templates/report_template.html
[summary_data]: /development/#_6
### 指定报告生成路径
默认情况下，生成的测试报告文件会位于项目根目录的 reports 文件夹中。如需指定生成报告的路径，可以使用 `--report-dir` 参数。
```bash
$ hrun docs/data/demo-quickstart-2.yml --dirreport-name /other/path/
...
同上，省略
INFO Start to render Html report ...
INFO Generated Html report: /other/path/1532078874.html
```