随意吧: 2013

2013年12月18日星期三

使用 goaccess 分析 nginx access.log

不需要很强大，只需要够用。

安装，今天刚好赶上一个新版本的发布：

$ yum -y install glib2 glib2-devel ncurses ncurses-devel GeoIP GeoIP-devel

$ wget http://downloads.sourceforge.net/project/goaccess/0.7/goaccess-0.7.tar.gz
$ tar -xzvf goaccess-0.7.tar.gz
$ cd goaccess-0.7/
$ ./configure --enable-geoip --enable-utf8 
$ make
# make install

因为access 的format 不一样，所以需要修改一下，我这里默认的 NCSA 也对不上，看看我的nginx 的 access log format：

    log_format  main  '$remote_addr $http_host $upstream_addr [$time_local] $request '
                      '"$status" $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$request_time"';

眼睛看了老半天，终于按空格对上号了（部分字段由于中间有空格，但是没有引号，所以又被多算了一列）， vim ~/.goaccessrc：

date_format %d/%b/%Y
log_format %h %^ %^ %^[%d:%^] %^ %r %^ "%s" %b "%R" "%u" "%^" "%D"

上面这个保存为文件，就不用每次去按照下面的提示按 C 去修改了。（至于每个字段的含义，可以去官方查询，另外发现有些是没有，

也有可能是我没发现，比如 $upstream_addr, $http_x_forwarded_for, 没有暂时用 %^ 啰）

然后就是执行命令

$goaccess -f access.log

可以重定向输出为 csv 格式或者 html 格式，罗列的各种 top 分析比较清晰。

ps: 不足之处，缺少 hourly 的支持，比如我的想要对比相邻几个小时内的访问情况，来分析异常请求，这个在作者 github 上有个 issue，

希望能够早点实现。

https://github.com/allinurl/goaccess/issues/8

haproxy 之 mysql 权限漏洞

+-------------+ +-------------+ +------------+

| ANY WHERE | | VIP 10.% |+-------->| 10.% |

+-------------+ +-------------+ +------------+

通过haproxy 后，授权都是走内网。

mysql 之权限漏洞，-h 使用的是代理服务器的ip， haproxy 使用的最简单的配置。仅仅只是代理，未配置任何其它规则。

so，使用代理的 ip 连库，mysql 数据库服务器认为是代理来连接本机。而没有 real source，grant ip 无法去限制外部连接。

这是我从任意一个外部机器上连接看到的效果：

$ mysql -hxxxx -P49711 -uxxxx -pxxxx  
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 308044655
Server version: 5.5.23-log Source distribution

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

思考一下，解决方法三种：

1. 透明代理，配置 haproxy 为透明代理，直接可用 mysql 的权限控制，但是，需要修改 haproxy 的iptables

规则，需要修改 db 服务器的网关。－－> 不靠谱，要维护的点多，一出问题就悲剧。iptables 可能影响其它服务。

2. 使用 iptables，accept 固定 ip 列表。－－> 无法使用单个文件控制，iprange 太多，不方便维护。给个例子:

＃iptables -A INPUT -p tcp --dport 49711 -m iprange --src-range 10.0.0.0/24 -j ACCEPT
 
＃iptables -A INPUT -p tcp --dport 49711:49713 -s 122.226.199.49 -j DROP

3. 使用 haproxy 的 acl 规则，使用 whitelist，只需要修改一下配置，ip 维护单独存放到文件。－－> 可行。

白名单 list 从 /etc/hosts 文件中过滤出来，保留为文件 /etc/haproxy/whitelist

10.0.0.0/8
59.32.213.0/24
61.142.208.0/24
120.31.133.0/24
113.106.26.0/24
113.105.245.0/24
211.151.181.0/24
113.107.234.109

haproxy 配置增加 tcp-request 一行：

listen vdb 180.153.21.159:49711
        tcp-request content reject if !{ src -f /etc/haproxy/whitelist }
        mode tcp
        server vdb 10.11.80.128:49710 check inter 1000 rise 2 fall 3 weight 2

上面那一行也可以写成如下两行：

        tcp-request content accept if { src -f /etc/haproxy/whitelist }
        tcp-request content reject

ps：要是像http 协议一样带一个 header，可以直接使用 forwardfor 搞定。

2013年11月26日星期二

mysql kill processlist

mysql> select concat('KILL ',id,';') from information_schema.processlist where user='root';

+------------------------+

| concat('KILL ',id,';') |

+------------------------+

| KILL 3101; |

| KILL 2946; |

+------------------------+

2 rows in set (0.00 sec)

mysql> select concat('KILL ',id,';') from information_schema.processlist where user='root' into outfile '/tmp/a.txt';

Query OK, 2 rows affected (0.00 sec)

mysql> source /tmp/a.txt;

Query OK, 0 rows affected (0.00 sec)

lua cpu 100% 故障经过总结

之所以没有昨天发wiki，是原本想再次确认一下和前天一样能够在tcpcopy 压测一段时间后看到效果，但是

昨天的测试持续好几个小时，问题不会重现；同样，我对报错的请求进行循环测试，依然没有问题。也就是

从昨天开始，缓存建立到一定程度后，这个bug 如果没有确认的话，估计就又隐藏起来了。

故障描述

从 10 月底到 11 月 22 号之前，info.v 使用 lua 取代 php 后，从数据监控图上看，性能提高至少一倍以上。

由于11 月 22 号一次清缓存，加节点的调整，导致nginx worker cpu 一段时间 100% 后无响应（即便无访问

情况，cpu 仍然100%），web 请求随之也无响应。

出现问题后的想法：

1. 缓存大的变动，导致缓存读库后响应慢引起；

2. 代码是否有bug，进程死循环了；

初步做法如下：

1. 为了先临时解决问题，通过脚本监控重启nginx，在 worker 都死完之前重启保证服务影响时间尽量短；

2. 和昊哥一再确认服务器配置和代码的修改都没有问题；验证缓存和库的访问是正常。

3. 检查access log 访问量，和前一天相差不大，error log 无变化，但仍在对可疑error 进行逐步修复。

Result： cpu 100% 问题依然复发。

商讨之后第二步做法：

1. 再次检查代码逻辑，是否有逻辑bug 问题，调整相关连接超时；

2. 和php 不同的是，没有相关slow log 可以查看，只能去找有效的工具，查相关的错误/或者瓶颈所在。

Result：代码上线了好几个版本，修复了一些错误，增加退出的逻辑，没有任何效果；

查找相关工具，openresty systemtap tools 能有效的查看信息，但需要内核相关软件支持，

systemtap 2.1 kerneldebuginfo ，由于线上服务器内核版本都是非rpm 包，想使用该版本，需要

集团那边协助才能安装。

23 号晚上，昊继续修复 nginx 中的 error；我在空闲测试机上搭建systemap 环境，使用tcpcopy 独立测试；

使用系统命令工具：

1. lsof －p pid 100% cpu的 nginx worker，和正常的worker 对比，没有想像的存在很多文件未释放情况；

2. strace -p pid 100% cpu 的 nginx worker，系统相关调用无任何输出，是不干活了；

3. pstack 和 gdb pid 100% cpu 的 nginx worker，有了以下发现：

[root@APP ~]# pstack 18523

#0 0x00002ae5d9be1208 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#1 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#2 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#3 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#4 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#5 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#6 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#7 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#8 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#9 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#10 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#11 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#12 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#13 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#14 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#15 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#16 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#17 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#18 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#19 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#20 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#21 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#22 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#23 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#24 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#25 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#26 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#27 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#28 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#29 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#30 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#31 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#32 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#33 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#34 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#35 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#36 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#37 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#38 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#39 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#40 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#41 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#42 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#43 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#44 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#45 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#46 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#47 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#48 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#49 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#50 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#51 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#52 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#53 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#54 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#55 0x00002ae5d9be1392 in match () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#56 0x00002ae5d9be1858 in lj_cf_string_gsub () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#57 0x00002ae5d9b9cd3b in lj_BC_FUNCC () from /home/openresty//luajit/lib/libluajit-5.1.so.2

#58 0x00000000004a0ca2 in ngx_http_lua_run_thread ()

#59 0x00000000004a8634 in ngx_http_lua_socket_tcp_resume ()

#60 0x00000000004a335b in ngx_http_lua_content_wev_handler ()

#61 0x00000000004a7735 in ngx_http_lua_socket_handle_success ()

#62 0x00000000004a9959 in ngx_http_lua_socket_tcp_read ()

#63 0x00000000004aa14e in ngx_http_lua_socket_read_handler ()

#64 0x00000000004a887a in ngx_http_lua_socket_tcp_handler ()

#65 0x000000000042d14a in ngx_event_process_posted ()

#66 0x000000000042d00f in ngx_process_events_and_timers ()

#67 0x0000000000432d28 in ngx_worker_process_cycle ()

#68 0x0000000000431687 in ngx_spawn_process ()

#69 0x000000000043240e in ngx_start_worker_processes ()

#70 0x00000000004332ab in ngx_master_process_cycle ()

#71 0x00000000004181cd in main ()

(gdb) attach 18523

Attaching to process 18523

Reading symbols from /home/openresty/nginx/sbin/nginx...done.

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypt.so.1

Reading symbols from /usr/lib64/libdrizzle.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libdrizzle.so.0

Reading symbols from /home/openresty//luajit/lib/libluajit-5.1.so.2...done.

Loaded symbols for /home/openresty//luajit/lib/libluajit-5.1.so.2

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libssl.so.6

Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypto.so.6

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libz.so.1

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgcc_s.so.1

Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libgssapi_krb5.so.2

Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libkrb5.so.3

Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcom_err.so.2

Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libk5crypto.so.3

Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libkrb5support.so.0

Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkeyutils.so.1

Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libresolv.so.2

Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libselinux.so.1

Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libsepol.so.1

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Reading symbols from /home/openresty//lualib/cjson.so...done.

Loaded symbols for /home/openresty//lualib/cjson.so

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffcb582000

0x00002ae5d9be11c0 in match (ms=0x2d,

s=0x4270e44f "\r\n\r\n\343\200\220\345\244\247\345\236\213\346\274\224\345\207\272\347\273\217\345\216\206\343\200\221\357\274\232\r\n2005-12 \350\276\275\345\256\201\347\224\265\350\247\206\345\217\260\346\230\245\350\212\202\346\231\232\344\274\232\350\241\250\346\274\224\345\230\211\345\256\276\r\n2006-11 CCTV4\347\224\265\350\247\206\350\266\205\344\272\272\350\212\202\347\233\256\347\211\271\347\272\246\345\230\211\345\256\276\r\n2006-12 \345\217\202\345\212\240\346\271\226\345\215\227\345\215\253\350\247\206<\345\277\253\344\271\220\345\244\247\346\234\254\350\220\245>\350\212\202\347\233\256\345\275\225\345\210\266,\344\270\223\350\256\277\345\230\211\345\256\276\r\n200"..., p=<value optimized out>) at lib_string.c:451

451 }

(gdb) bt

Cannot access memory at address 0x7fffcb56b388

这个 worker 是陷于 match 函数里了，为进一步验证，我们对另外几个死掉的worker 同样做了查看，情况基本都相同，以此定位到问题出根本原因；

再回过头查看相关的nginx error log 对应 match 相关调用：

/diska/htdocs/openresty/vinfo.lua:217: in function </diska/htdocs/openresty/vinfo.lua:1>, client: 10.11.80.54, server: info.v.56.com, request: "GET /?ids=51594989,52711230,47843719,52546854,29120020,52614818,51764257,51747382,52419156,52078753,52181976,52723946,51480892,52214031,39500819,39500976,46742475,47681770,47831882,47710651,52680032,52666705,51171938,51584136,52420233,52419871,52420185,52419754,52419307,52406085,52405507,32986833,19141870,52092021,47540485,47451641,52400473,52377987,52550667,51659238,51113427,26454976,51711240,14369633&fli=id,totaltime,times,comment,public,chk_yn,public_time,user_id HTTP/1.0", host: "info.v.56.com"

一个正则表达式匹配失败报出来的error；这种由于字符escape 失败导致的一个兼容性错误之前也是有的。

这个故障也要求深入底层的c 看问题了，要不然从表面看到的，只是一个黑盒子。

2013年10月30日星期三

linux 系统安全 25 个基本提示（转）

Securing a system in a production from the hands of hackers and crackers is a challenging task for a System Administrator. This is our first article related to “How to Secure Linux box” or “Hardening a Linux Box“. In this post We’ll explain 25 useful tips & tricks to secure your Linux system. Hope, below tips & tricks will help you some extend to secure your system.

1. Physical System Security

Configure the BIOS to disable booting from CD/DVD, External Devices, Floppy Drive inBIOS. Next, enable BIOS password & also protect GRUB with password to restrict physical access of your system.

Set GRUB Password to Protect Linux Servers

2. Disk Partitions

It’s important to have different partitions to obtain higher data security in case if any disaster happens. By creating different partitions, data can be separated and grouped. When an unexpected accident occurs, only data of that partition will be damaged, while the data on other partitions survived. Make sure you must have following separate partitions and sure that third party applications should be installed on separate file systems under /opt.

/
/boot
/usr
/var
/home
/tmp
/opt

3. Minimize Packages to Minimize Vulnerability

Do you really want all sort of services installed?. It’s recommended to avoid installing useless packages to avoid vulnerabilities in packages. This may minimize risk that compromise of one service may lead to compromise of other services. Find and remove or disable unwanted services from the server to minimize vulnerability. Use the ‘chkconfig‘ command to find out services which are running on runlevel 3.

# /sbin/chkconfig --list |grep '3:on'

Once you’ve find out any unwanted service are running, disable them using the following command.

# chkconfig serviceName off

Use the RPM package manager such as “yum” or “apt-get” tools to list all installed packages on a system and remove them using the following command.

# yum -y remove package-name

# sudo apt-get remove package-name

4. Check Listening Network Ports

With the help of ‘netstat‘ networking command you can view all open ports and associated programs. As I said above use ‘chkconfig‘ command to disable all unwanted network services from the system.

# netstat -tulpn

20 Netstat Commands for Network Management in Linux

5. Use Secure Shell(SSH)

Telnet and rlogin protocols uses plain text, not encrypted format which is the security breaches.SSH is a secure protocol that use encryption technology during communication with server.

Never login directly as root unless necessary. Use “sudo” to execute commands. sudo are specified in /etc/sudoers file also can be edited with the “visudo” utility which opens in VIeditor.

It’s also recommended to change default SSH 22 port number with some other higher level port number. Open the main SSH configuration file and make some following parameters to restrict users to access.

# vi /etc/ssh/sshd_config

Disable root Login

PermitRootLogin no

Only allow Specific Users

AllowUsers username

Use SSH Protocol 2 Version

Protocol 2

5 Best Practices to Secure and Protect SSH Server

6. Keep System updated

Always keep system updated with latest releases patches, security fixes and kernel when it’s available.

# yum updates
# yum check-update

7. Lockdown Cronjobs

Cron has it’s own built in feature, where it allows to specify who may, and who may not want to run jobs. This is controlled by the use of files called /etc/cron.allow and /etc/cron.deny. To lock a user using cron, simply add user names in cron.deny and to allow a user to run cron add incron.allow file. If you would like to disable all users from using cron, add the ‘ALL‘ line tocron.deny file.

# echo ALL >>/etc/cron.deny

11 Cron Scheduling Examples in Linux

8. Disable USB stick to Detect

Many times it happens that we want to restrict users from using USB stick in systems to protect and secure data from stealing. Create a file ‘/etc/modprobe.d/no-usb‘ and adding below line will not detect USB storage.

install usb-storage /bin/true

9. Turn on SELinux

Security-Enhanced Linux (SELinux) is a compulsory access control security mechanism provided in the kernel. Disabling SELinux means removing security mechanism from the system. Think twice carefully before removing, if your system is attached to internet and accessed by the public, then think some more on it.

SELinux provides three basic modes of operation and they are.

Enforcing: This is default mode which enable and enforce the SELinux security policy on the machine.
Permissive: In this mode, SELinux will not enforce the security policy on the system, only warn and log actions. This mode is very useful in term of troubleshooting SELinux related issues.
Disabled: SELinux is turned off.

You can view current status of SELinux mode from the command line using ‘system-config-selinux‘, ‘getenforce‘ or ‘sestatus‘ commands.

# sestatus

If it is disabled, enable SELinux using the following command.

# setenforce enforcing

It also can be managed from ‘/etc/selinux/config‘ file, where you can enable or disable it.

10. Remove KDE/GNOME Desktops

There is no need to run X Window desktops like KDE or GNOME on your dedicated LAMPserver. You can remove or disable them to increase security of server and performance. To disable simple open the file ‘/etc/inittab‘ and set run level to 3. If you wish to remove it completely from the system use the below command.

# yum groupremove "X Window System"

11. Turn Off IPv6

If you’re not using a IPv6 protocol, then you should disable it because most of the applications or policies not required IPv6 protocol and currently it doesn’t required on the server. Go to network configuration file and add followings lines to disable it.

# vi /etc/sysconfig/network

NETWORKING_IPV6=no
IPV6INIT=no

12. Restrict Users to Use Old Passwords

This is very useful if you want to disallow users to use same old passwords. The old password file is located at /etc/security/opasswd. This can be achieved by using PAM module.

Open ‘/etc/pam.d/system-auth‘ file under RHEL / CentOS / Fedora.

# vi /etc/pam.d/system-auth

Open ‘/etc/pam.d/common-password‘ file under Ubuntu/Debian/Linux Mint.

# vi /etc/pam.d/common-password

Add the following line to ‘auth‘ section.

auth        sufficient    pam_unix.so likeauth nullok

Add the following line to ‘password‘ section to disallow a user from re-using last 5 password of his or her.

password   sufficient    pam_unix.so nullok use_authtok md5 shadow remember=5

Only last 5 passwords are remember by server. If you tried to use any of last 5 old passwords, you will get an error like.

Password has been already used. Choose another.

13. How to Check Password Expiration of User

In Linux, user’s passwords are stored in ‘/etc/shadow‘ file in encrypted format. To check password expiration of user’s, you need to use ‘chage‘ command. It displays information of password expiration details along with last password change date. These details are used by system to decide when a user must change his/her password.

To view any existing user’s aging information such as expiry date and time, use the following command.

#chage -l username

To change password aging of any user, use the following command.

#chage -M 60 username
#chage -M 60 -m 7 -W 7 userName

Parameters

-M Set maximum number of days
-m Set minimum number of days
-W Set the number of days of warning

14. Lock and Unlock Account Manually

The lock and unlock features are very useful, instead of removing an account from the system, you can lock it for an week or a month. To lock a specific user, you can use the follow command.

# passwd -l accountName

Note : The locked user is still available for root user only. The locking is performed by replacing encrypted password with an (!) string. If someone trying to access the system using this account, he will get an error similar to below.

# su - accountName
This account is currently not available.

To unlock or enable access to an locked account, use the command as. This will remove (!) string with encrypted password.

# passwd -u accountName

15. Enforcing Stronger Passwords

A number of users use soft or weak passwords and their password might be hacked with adictionary based or brute-force attacks. The ‘pam_cracklib‘ module is available in PAM(Pluggable Authentication Modules) module stack which will force user to set strong passwords. Open the following file with an editor.

16. Enable Iptables (Firewall)

It’s highly recommended to enable Linux firewall to secure unauthorised access of your servers. Apply rules in iptables to filters incoming, outgoing and forwarding packets. We can specify the source and destination address to allow and deny in specific udp/tcp port number.

Basic IPTables Guide and Tips

17. Disable Ctrl+Alt+Delete in Inittab

In most Linux distributions, pressing ‘CTRL-ALT-DELETE’ will takes your system to reboot process. So, it’s not a good idea to have this option enabled at least on production servers, if someone by mistakenly does this.

This is defined in ‘/etc/inittab‘ file, if you look closely in that file you will see a line similar to below. By default line is not commented out. We have to comment it out. This particular key sequence signalling will shut-down a system.

# Trap CTRL-ALT-DELETE
#ca::ctrlaltdel:/sbin/shutdown -t3 -r now

18. Checking Accounts for Empty Passwords

Any account having an empty password means its opened for unauthorized access to anyone on the web and it’s a part of security within a Linux server. So, you must make sure all accounts have strong passwords and no one has any authorized access. Empty password accounts are security risks and that can be easily hackable. To check if there were any accounts with empty password, use the following command.

# cat /etc/shadow | awk -F: '($2==""){print $1}'

19. Display SSH Banner Before Login

It’s always a better idea to have an legal banner or security banners with some security warnings before SSH authentication. To set such banners read the following article.

Display SSH Warning Message to Users

20. Monitor User Activities

If you are dealing with lots of users, then its important to collect the information of each user activities and processes consumed by them and analyse them at a later time or in case if any kind of performance, security issues. But how we can monitor and collect user activities information.

There are two useful tools called ‘psacct‘ and ‘acct‘ are used for monitoring user activities and processes on a system. These tools runs in a system background and continuously tracks each user activity on a system and resources consumed by services such as Apache, MySQL, SSH, FTP, etc. For more information about installation, configuration and usage, visit the below url.

Monitor User Activity with psacct or acct Commands

21. Review Logs Regularly

Move logs in dedicated log server, this may prevents intruders to easily modify local logs. Below are the Common Linux default log files name and their usage:

/var/log/message – Where whole system logs or current activity logs are available.
/var/log/auth.log – Authentication logs.
/var/log/kern.log – Kernel logs.
/var/log/cron.log – Crond logs (cron job).
/var/log/maillog – Mail server logs.
/var/log/boot.log – System boot log.
/var/log/mysqld.log – MySQL database server log file.
/var/log/secure – Authentication log.
/var/log/utmp or /var/log/wtmp : Login records file.
/var/log/yum.log: Yum log files.

22. Important file Backup

In a production system, it is necessary to take important files backup and keep them in safety vault, remote site or offsite for Disasters recovery.

23. NIC Bonding

There are two types of mode in NIC bonding, need to mention in bonding interface.

mode=0 – Round Robin
mode=1 – Active and Backup

NIC Bonding helps us to avoid single point of failure. In NIC bonding, we bond two or moreNetwork Ethernet Cards together and make one single virtual Interface where we can assignIP address to talk with other servers. Our network will be available in case of one NIC Card is down or unavailable due to any reason.

Read Also : Create NIC Channel Bonding in Linux

24. Keep /boot as read-only

Linux kernel and its related files are in /boot directory which is by default as read-write. Changing it to read-only reduces the risk of unauthorized modification of critical boot files. To do this, open “/etc/fstab” file.

# vi /etc/fstab

Add the following line at the bottom, save and close it.

LABEL=/boot     /boot     ext2     defaults,ro     1 2

Please note that you need to reset the change to read-write if you need to upgrade the kernel in future.

25. Ignore ICMP or Broadcast Request

Add following line in “/etc/sysctl.conf” file to ignore ping or broadcast request.

Ignore ICMP request:
net.ipv4.icmp_echo_ignore_all = 1

Ignore Broadcast request:
net.ipv4.icmp_echo_ignore_broadcasts = 1

Load new settings or changes, by running following command

#sysctl -p

If you’ve missed any important security or hardening tip in the above list, or you’ve any other tip that needs to be included in the list. Please drop your comments in our comment box. TecMint is always interested in receiving comments, suggestions as well as discussion for improvement.

订阅：博文 (Atom)