2016年07月03日 from—http://www.itts-union.com/2923.html

Catalog

[隐藏]

概述

linuxcurl是通过url语法在命令行下上传或下载文件的工具软件，它支持http,https,ftp,ftps,telnet等多种协议，常被用来抓取网页和监控Web服务器状态。

官网下载地址 http://curl.haxx.se/download/curl-7.46.0.tar.gz

作用1 测试某一条url的相关情况

包括是否缓存，返回字段是否正确等。相当于模拟浏览器客户端做get操作

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://app613.imgcache.qzoneapp.com/app613/frontend_swf/version.js'
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:48:13 GMT
Cache-Control: max-age=6000
Expires: Sun, 03 Jul 2016 10:28:13 GMT
Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT
Content-Type: application/x-javascript; charset=utf-8
Content-Length: 69
X-Cache-Lookup: Hit From MemCache<br></p>

作用2 验证某节点ip缓存情况

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://app613.imgcache.qzoneapp.com/app613/frontend_swf/version.js' -x 106.74.23.144:80
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:50:34 GMT
Cache-Control: max-age=6000
Expires: Sun, 03 Jul 2016 10:30:34 GMT
Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT
Content-Type: application/x-javascript; charset=utf-8
Content-Length: 69
X-Cache-Lookup: Hit From Disktank<br></p>

PS:其中的-x绑定代理服务器ip的用法，类似于wget中的：

# wget -e http-proxy=119.97.137.178 http://sdgautopatch.cloudcdn.net/test008/6wireshark-win64-1.2.9.exe

作用3 验证referer头部白名单是否生效

场景

使用-H 来添加自定义的头部，模拟特殊条件下源站/cdn节点对特性是否支持，结果返回是否符合预期。下面的这个例子，就是用来验证在控制台上添加了referer白名单后，模拟该referer头部返回是否符合预期。

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://res2.eqh5.com/' -x 106.74.23.144:80 -H"referer:http://*.yqh5.cn"
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:59:50 GMT
Cache-Control: max-age=31536000
Expires: Mon, 03 Jul 2017 08:59:50 GMT
Last-Modified: Tue, 12 Apr 2016 01:59:32 GMT
Content-Type: text/html
Content-Length: 612
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Content-Disposition: inline; filename=""
Accept-Ranges: bytes<br></p>

PS 上面的

curl -I ‘http://res2.eqh5.com/’ -x 106.74.23.144:80 -H”referer:http://*.yqh5.cn”

也可以写为：

curl -I ‘http://res2.eqh5.com/’ -x 106.74.23.144:80 -e “http://pachong.org/”

作用4 验证节点/源站对分片的支持情况

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -Iv 'http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'
* About to connect() to proxy 106.74.23.144 port 80 (#0)
*   Trying 106.74.23.144... connected
* Connected to 106.74.23.144 (106.74.23.144) port 80 (#0)
> HEAD http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: update01.skyworth-cloud.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> range:bytes=0-1000
> 
< HTTP/1.1 206 Partial Content
HTTP/1.1 206 Partial Content
< Server: nws_ocmid_hy
Server: nws_ocmid_hy
< Connection: close
Connection: close
< Date: Sun, 03 Jul 2016 09:02:24 GMT
Date: Sun, 03 Jul 2016 09:02:24 GMT
< Cache-Control: max-age=600
Cache-Control: max-age=600
< Expires: Sun, 03 Jul 2016 09:12:24 GMT
Expires: Sun, 03 Jul 2016 09:12:24 GMT
< Last-Modified: Mon, 23 May 2016 20:10:06 GMT
Last-Modified: Mon, 23 May 2016 20:10:06 GMT
< Content-Range: bytes 0-1000/437405243
Content-Range: bytes 0-1000/437405243
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 1001
Content-Length: 1001
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Disktank Upstream
X-Cache-Lookup: Hit From Disktank Upstream
< X-Cache-Lookup: Hit From Disktank
X-Cache-Lookup: Hit From Disktank
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Inner Cluster
X-Cache-Lookup: Hit From Inner Cluster
< 
* Closing connection #0<br><br></p>

备注

上面的curl -Iv ‘http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip’ -x 106.74.23.144:80 -H’range:bytes=0-1000′

也可以写成

curl -Iv ‘http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip’ -x 106.74.23.144:80 -r 0-1000

作用5 验证文件在节点和源站上大小是否一致（通过md5判断）

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -o pic1 'http://as.eqxiu.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 106.74.23.144:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  40629      0 --:--:-- --:--:-- --:--:-- 54149
[root@VM_27_201_centos ~]# 
[root@VM_27_201_centos ~]# md5sum pic1
4b7671dbe1a634541726bd9473d14dcc  pic1
[root@VM_27_201_centos ~]# 
[root@VM_27_201_centos ~]# 
[root@VM_27_201_centos ~]# curl -o pic2 'http://as.eqxiu.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 115.159.20.55:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  4629k      0 --:--:-- --:--:-- --:--:-- 9888k
[root@VM_27_201_centos ~]# 
[root@VM_27_201_centos ~]# md5sum pic2
4b7671dbe1a634541726bd9473d14dcc  pic2<br></p>

备注

上面的-o 代表把文件下载到指定的路径下，-x代表使用指定的ip或者域名作为代理。

作用6 模拟https访问验证https证书

[root@VM_117_214_centos ~]#  curl -sv 'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o /dev/null -k
* About to connect() to mccdn.qcloud.com port 443 (#0)
*   Trying 113.207.16.63... connected
* Connected to mccdn.qcloud.com (113.207.16.63) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*     subject: CN=*.qcloud.com,OU=R&D,O=Shenzhen Tencent Computer Systems Company Limited,L=shenzhen,ST=guangdong,C=CN
*     start date: May 25 00:00:00 2016 GMT
*     expire date: Jul 24 23:59:59 2019 GMT
*     common name: *.qcloud.com
*     issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /static/img/57c45bb3997a277e5996dcf13314b365/user-40.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: mccdn.qcloud.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nnws/1.7.3.8
< Date: Sat, 20 Aug 2016 12:43:55 GMT
< Content-Type: image/png
< Content-Length: 3558
< Connection: keep-alive
< Cache-Control: max-age=2592000
< Expires: Mon, 19 Sep 2016 12:43:54 GMT
< Last-Modified: Wed, 25 May 2016 08:06:29 GMT
< X-Cache-Lookup: Hit From Disktank
< 
{ [data not shown]
* Connection #0 to host mccdn.qcloud.com left intact
* Closing connection #0

作用7 curl中绑定host测试方法

等同于在windows中绑定host进行测试

[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com' http://src.com.z0.glb.qiniudns.com/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-
lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Sat, 02 Jan 2016 11:39:23 GMT
Content-Type: video/mp2t
Content-Length: 673416
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="000000.ts"
Content-Transfer-Encoding: binary
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
X-Log: mc.g;IO:1
X-Reqid: iToAAMIdW1cTmSUU
X-Qiniu-Zone: 0
[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com' http://211.161.127.147/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: NWS_Appimg_HY
Connection: keep-alive
Date: Sat, 02 Jan 2016 11:45:50 GMT
Cache-Control: max-age=31536000
Expires: Sun, 01 Jan 2017 11:45:50 GMT
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
Content-Type: video/mp2t
Content-Length: 673416
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
X-ReqId: ATUAABGu8oBVjyUU
Content-Disposition: inline; filename="000000.ts"

其中：

7u2q8y.com3.z0.glb.qiniucdn.com 为用户的域名

src.com.z0.glb.qiniudns.com 为源站或者回源host地址，也可以为ip地址形式

211.161.127.147 为域名接入cdn之后解析到的cdn节点ip地址

通过验证源站url和cdn节点ip地址后的curl反馈可以判断出到底是源站有问题，还是cdn节点有问题

作用8 使用curl抓取返回时间

判断网页访问各耗时时间

curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com
This command will output this:
[root@VM_117_214_centos ~]#  curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com/1326.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 74470    0 74470    0     0  52368      0 --:--:--  0:00:01 --:--:--  116k
Connect: 0.799 TTFB: 1.309 Total time: 1.422

其他的不常见用法，仅做了解

1、抓取页面内容到一个文件中

1	`[root@www.itts-union.com]# curl -o home.html http://www.itts-union.com`

2、模拟表单信息，模拟登录，保存cookie信息

1	`[root@www.itts-union.com]# curl -c ./cookie_c.txt -F log=aaaa -F pwd=****** http://www.itts-union.com/wp-login.php`

3、使用cookie文件

1	`[root@www.itts-union.com]# curl -b ./cookie_c.txt http://www.itts-union.com/wp-admin`

4、显示下载进度条

1 2	`[root@VM_117_214_centos` `~]# curl -o smokeping.tar.gz http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -#` `########################################################################` `100.0%`

5、通过ftp下载文件

[zhangy@www.itts-union.com ~]$ curl -u 用户名:密码 -O http://www.itts-union.com/demo/curtain/bbstudy_files/style.css
　　% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
　　Dload  Upload   Total   Spent    Left  Speed
　　101  1934  101  1934    0     0   3184      0 –:–:– –:–:– –:–:–  7136

或者用下面的方式

[zhangy@www.itts-union.com ~]$ curl -O ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/style.css

6、通过ftp上传

[zhangy@www.itts-union.com ~]$ curl -T test.sql ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/

7、如下载网页发现乱码，可以使用iconv转码：

[root@VM_117_214_centos
 ~]# curl -o smokeping.tar.gz 
http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -# | iconv 
-fgb2312
######################################################################## 100.0%

8、验证https访问

curl -sv 
'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png'
 -o/dev/null --resolve "mccdn.qcloud.com:443:221.204.60.11"

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
注：
“The fear of the Lord is the beginning of wisdom, and the knowledge of the Holy One is wisdom.”

原文由博主方胜山编辑撰写，版权归博主所有。更多相关资讯请关注微信“ebookyx”原文地址 http://www.itts-union.com/2923.html 转载请注明出处！
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@