碳基体

奋斗在产品安全第一线的安全妹子
29
07

日志全记录模式开启

背景： 为了关联分析请求体与响应体

WAF为什么不检测响应体，仅仅是性能约束吗？是不是关联检测的收益远远高于部署成本呢？所以先搭建一套环境，尝试看看效果

方案一: 选择使用nginx（openresty）+lua来搭建日志生成环境


1. nginx作为web server

用于自己构造攻击的实验环境

第一步：openresty搭建

openresty 的搭建方法参加 https://danqingdani.blog.163.com/blog/static/186094195201471083759852/


第二步：安装php+php-fpm+mysql环境

（1）安装组件

apt-get install php5 php5-cli php5-cgi php5-dev mysql-server php5-mysql php5-gd php5-mcrypt php5-curl php5-fpm

（2）配置 php-fpm 监听端口

vim /etc/php5/fpm/php-fpm.conf 

查看

pid = /var/run/php5-fpm.pid

include=/etc/php5/fpm/pool.d/tanjiti.conf

error_log = /var/log/php5-fpm.log


cp /etc/php5/fpm/pool.d/www.conf /etc/php5/fpm/pool.d/tanjiti.conf 

vim /etc/php5/fpm/pool.d/tanjiti.conf

编辑

[tanjiti]


;listen = 127.0.0.1:9000
listen = /var/run/php5-fpm.sock

listen.owner = www-data
listen.group = www-data
listen.mode = 0660

启动

/etc/init.d/php5-fpm start


（3） nginx配置

vim nginx/conf/nginx.conf

编辑

        location ~ \.php$ {
            root           html;
            #fastcgi_pass   127.0.0.1:9000;
            fastcgi_pass   unix:/var/run/php5-fpm.sock;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            include        fastcgi_params;
        }


nginx/sbin/nginx -s reload


第三步：配置日志记录格式


（1） 日志记录格式

vim nginx/conf/nginx.conf

编辑

    log_format main '$remote_addr [$time_local] $request_method "$request_uri" $status '
                    '"$query_string" "$http_user_agent" '
                    '"$http_referer" "$http_cookie" "$http_content_type" $content_length  $body_bytes_sent '
                    '"$http_x_forwarded_for" '
                    '"$request_body" "$resp_body"';

    access_log  logs/access.log  main;

（2）response body


vim nginx/conf/nginx.conf

编辑 

        set $resp_body "";
        lua_need_request_body on;
        body_filter_by_lua_file ../lualib/logResponse.lua;


vim lualib/logResponse.lua

编写

local chunk, eof = ngx.arg[1], ngx.arg[2]
local buffered = ngx.ctx.buffered
if not buffered then
   buffered = {}
   ngx.ctx.buffered = buffered
end
if  chunk ~= "" then
   buffered[#buffered + 1] = chunk
   ngx.arg[1] = nil
end
if eof then
   local whole = table.concat(buffered)
   ngx.ctx.buffered = nil
   ngx.arg[1] = whole
   ngx.var.resp_body = ngx.arg[1]
end

当然，我们可以选择有条件的记录response body部分，在下文中介绍


参考：

https://groups.google.com/forum/#!msg/openresty-en/q-dcQNxpwTA/6nZ2VKg1ouwJ 谢谢春哥，谢谢春哥，谢谢春哥，重要的事情要说三遍



第四步：测试

http localhost:8085


xxx.xxx.xxx.xxx [22/Dec/2015:15:37:52 +0800] GET "/" 200 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" "-" "_pk_id.7.ca22=490c960efd9ccf66.1434705267.0.1435055206.." "-" -  177 "-" "-" "<html><body><h1>It works!</h1>\x0A<p>This is the default web page for this server.</p>\x0A<p>The web server software is running but no content has been added, yet.</p>\x0A</body></html>\x0A"

2. nginx作为reverse proxy（云WAF的模式）

用于将真实流量导入

第一步：配置nginx为代理模式

（1） 日志格式设置

vim nginx/conf/nginx.conf

编辑

    log_format main '$remote_addr [$time_local] $request_method "$request_uri"  '
                '"$query_string" "$http_user_agent" '
                '"$http_referer" "$http_cookie" "$http_content_type" $content_length  '

'$status "$sent_http_content_type" $body_bytes_sent '
                '"$proxy_add_x_forwarded_for" "$http_via" '
                '"$upstream_http_server<||>$upstream_http_x_powered_by" '
                '"$upstream_http_location" "$upstream_addr" "$upstream_status" "$upstream_response_time" '
                '"$request_body"  $attack_type($rule_id) $ban_type($waf_duration) "$resp_body"';

其中以下变量，是用于标记请求行为来进行有条件的记录response body

$attack_type： e.g. SQLI

$rule_id e.g. 0001

$ban_type e.g. BAN

$waf_duration  检测模块花费时间，用于计算访问控制模块性能的时间花费

（2）配置代理模式

p.s. 当然这是waf简化版本，去掉了最关键的cache设置（加速）与访问控制（安全）

vim nginx/conf/nginx.conf

编辑

#访问控制，标签请求行为

        set $attack_type "NONE";

        set $rule_id "";

        set $ban_type "NONE";

        set $waf_duration "0";

        access_by_lua '  

                local waf = require "waf-runner"

                waf.execute()

            ';

    #记录response body

        set $resp_body "";

        lua_need_request_body on;

        body_filter_by_lua_file ../lualib/logResponse.lua;


        location / {

proxy_http_version 1.1;     # enable chunked
proxy_set_header  Host $host:$server_port;
proxy_set_header Accept-Encoding ""; #否则Accept-Encoding为gzip时返回是gzip编码的数据
proxy_set_header Connection "";# keep alive
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass https://127.0.0.1:8080; #upstream的地址，云waf一般会使用redis动态配置来维持地址表
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 64 16k; 
proxy_busy_buffers_size 32k; 
proxy_max_temp_file_size 10240m;
proxy_temp_file_write_size 16k;
proxy_temp_path /var/log/nginx/proxy_temp;
proxy_intercept_errors on;
proxy_ignore_client_abort on;
proxy_connect_timeout 10;
proxy_read_timeout 600; # default is 60s
proxy_send_timeout 600; # default is 60s
proxy_redirect off;
proxy_ssl_session_reuse  off;
        }


(3) 设置有条件的记录response body

vim lualib/logResponse.lua

编辑


local MAX_LOGED_LENGTH = 10485760 -- 10M
local attack_type = ngx.var.attack_type
local sent_http_content_type = ngx.var.sent_http_content_type
local chunk, eof = ngx.arg[1], ngx.arg[2]
local buffered = ngx.ctx.buffered

if string.find(sent_http_content_type, "text",1,true) and attack_type == "填写需要记录的类型"   then
    if not buffered then
        buffered = {}
        ngx.ctx.buffered = buffered
    end
    if  chunk ~= "" then
        buffered[#buffered + 1] = chunk
        ngx.arg[1] = nil
    end
    if eof then
        local whole = table.concat(buffered)
        ngx.ctx.buffered = nil
        ngx.arg[1] = whole
        if #ngx.arg[1] > MAX_LOGED_LENGTH then
            ngx.var.resp_body =  string.sub(ngx.arg[1], 1, MAX_LOGED_LENGTH)
        else
            ngx.var.resp_body = ngx.arg[1]
        end
    end

end  



第二步：测试

例如记录phpmyadmin的访问细节

http localhost:8085/phpmyadmin -v


127.0.0.1 [23/Dec/2015:13:20:53 +0800] GET "/phpmyadmin" 301 "-" "HTTPie/0.9.0-dev" "-" "-" "-" -  318 "127.0.0.1" "-" "Apache/2.2.22 (Ubuntu)<||>-" "https://localhost:8085/phpmyadmin/" "127.0.0.1:8080" "301" "0.002" "-" logbody(221) LOG(48) "<!DOCTYPE HTML PUBLIC \x22-//IETF//DTD HTML 2.0//EN\x22>\x0A<html><head>\x0A<title>301 Moved Permanently</title>\x0A</head><body>\x0A<h1>Moved Permanently</h1>\x0A<p>The document has moved <a href=\x22https://localhost:8085/phpmyadmin/\x22>here</a>.</p>\x0A<hr>\x0A<address>Apache/2.2.22 (Ubuntu) Server at localhost Port 8085</address>\x0A</body></html>\x0A"





---------------------------以下是错误的分割线--------------------------------------------------------------------------------------------------


错误处理1.

connect() to unix:/var/run/php5-fpm.sock failed (13: Permission denied) while connecting to u
pstream

修改nginx配置文件

vim nginx/conf/nginx.conf

编辑user指令

user www-data www-data;


错误处理2.

FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream

修改nginx配置文件

vim nginx/conf/nginx.conf

编辑fastcgi_param指令

fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;

参考 https://lovelace.blog.51cto.com/1028430/1314565


错误处理3：

当开启nginx proxy模式的时候，如果发送请求中采用gzip编码，则采用以上方法记录的response body不可读

127.0.0.1 [22/Dec/2015:17:18:29 +0800] GET "/index.php" 404 "-" "HTTPie/0.9.0-dev" "-" "-" "-" -  236 "-" "-" "\x1F\x8B\x08\x00\x00\x00\x00\x00\x00\x03M\x8FOK\xC40\x10\xC5\xEF\xFD\x14\xE3\x9E\xF4`&-\x0B\xEE!\x04t\xDB\xC5\x85\xBA\x16M\x0F\x1E\xB3f$\x855\xA9I\xEA\x9Foo\xDAE\x90\x07\x033\xF3~\xC3\x1BqQ?n\xD5K\xD7\xC0\xBDzh\xA1\xEB\xEF\xDA\xFD\x16V\xD7\x88\xFBF\xED\x10kU\x9F7\x15\xE3\x88\xCDa%\x0Ba\xD3\xFBI\x0AK\xDA\xE4&\x0D\xE9Dr\xCD\xD7p\xF0\x09v~rF\xE0yX\x08\x5CL\xE2\xE8\xCD\xCF\xCC\x95\xF2\x9F'w\x85\x18\xA5\xB2\x04\x81>&\x8A\x89\x0C\xF4O-\xE0\xE0\x0C}\xB3\xD1\x8E\xF0\xA5#\xB8\x8C\xBC\xCD\x08x\x07\xC9\x0E\x11\x22\x85O\x0AL\xE08\x1F\x0D\xB9hc\x02\xC5(oG\xFDj\x09+\x96U\xC1e\x7F\x9C\x5C\x9A\xAE\xE0y\x01@'(\xAB\x1B\xC6\xB3J\xE8|H\xB0\xE1\x1B.\xF0\x8F\xCEy\x97\xA49\xDB\xFCa\xF1\x0B\xC8\xD5\xF63\x1C\x01\x00\x00"


后来询问了同事，并且认真读了春哥回复的帖子 https://groups.google.com/forum/#!msg/openresty-en/q-dcQNxpwTA/6nZ2VKg1ouwJ ， 春哥早就提供很简单的解决方案了


方案二、apache+modsecurity

对于不想自己二次开发的，就直接用modsecurity吧


第一步：apache+modsecurity搭建 

参考https://danqingdani.blog.163.com/blog/static/186094195201481562831737/


第二步：开启记录response body的功能

vim /etc/modsecurity/modsecurity.conf

编辑

SecResponseBodyAccess On

SecAuditEngine RelevantOnly

SecAuditLogParts ABCDEFGHIJKZ #全记录

SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

第三步：编辑记录规则

vim /etc/modsecurity/my.conf 


编辑 (比如说记录phpmyadmin的操作，仅仅是示例）

SecRule "REQUEST_FILENAME" "@contains phpmyadmin" "phase:1,auditlog,pass,t:lowercase,log,tag:'just for test',msg:'just for test',id:0000001"



第四步：测试

http localhost:8085/phpmyadmin



--a3bef27a-B--
GET /phpmyadmin/ HTTP/1.1
Host: xxx.xxx.xx:8080
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: zh-CN,zh;q=0.8,en;q=0.6
Cookie: pma_lang=zh_CN; pma_collation_connection=utf8_general_ci; phpMyAdmin
=kegrv6adlm9oae5tgvoeg61sjcjo62mk; pma_navi_width=200; _pk_id.7.ca22=490c960efd9ccf66.1434705267.0.1435055206..
If-Modified-Since: Sat, 18 Feb 2012 12:26:45 GMT

--a3bef27a-F--
HTTP/1.1 200 OK
X-Powered-By: PHP/5.3.10-1ubuntu3.21
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Last-Modified: Sat, 18 Feb 2012 12:26:45 GMT
Set-Cookie: pmaPass-1=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/phpmyadmin/
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 2492
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8

--a3bef27a-E--
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="zh" lang="zh" dir="ltr">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <link rel="icon" href="./favicon.ico" type="image/x-icon" />
    <link rel="shortcut icon" href="./favicon.ico" type="image/x-icon" />
    <title>phpMyAdmin </title>
    <link rel="stylesheet" type="text/css" href="phpmyadmin.css.php?server=1&amp;token=4cc02ab2aad2355a5213b7f80f752bbb&amp;js_frame
=right&amp;nocache=3988383895" />

省略

--a3bef27a-H--
Message: Warning. String match "phpmyadmin" at REQUEST_FILENAME. [file "/etc/modsecurity/my.conf"] [line "1"] [id "0000001"] [msg "j
ust for test"] [tag "just for test"]
Apache-Handler: application/x-httpd-php
Stopwatch: 1450855051967581 66146 (- - -)
Stopwatch2: 1450855051967581 66146; combined=201, p1=165, p2=27, p3=1, p4=1, p5=7, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.3 (https://www.modsecurity.org/).
Server: Apache/2.2.22 (Ubuntu)

--a3bef27a-K--
SecRule "REQUEST_FILENAME" "@contains phpmyadmin" "phase:1,auditlog,pass,t:lowercase,log,tag:'just for test',msg:'just for test',id:0000001"


--a3bef27a-Z--


modsecurity 的功能非常齐全，but，但性能一直被诟病，因此在继承其安全防御思路的下用nginx来做轻量级的开发是当下比较好的选择。

来源：碳基体

● nginx记录response
热度(1)  

评论

热度(1)

© 碳基体 | Powered by LOFTER