分类目录归档:Linux

Linux

Linux 文件恢复的原理

inode 和 block

首先简单介绍一下 Linux 文件系统的最基本单元:inode。inode 译成中文就是索引节点,每个存储设备(例如硬盘)或存储设备的分区被格式化为文件系统后,应该有两部份,一部份是 inode,另一部份是 block,block 是用来存储数据用的。而 inode 呢,就是用来存储这些数据的信息,这些信息包括文件大小、属主、归属的用户组、读写权限等。inode 为每个文件进行信息索引,所以就有了 inode 的数值。linux 操作系统下可以使用 ls –id 命令来查看文件或者目录的 inode 值,一般”root”目录的 inode 值为 2,一个分区挂载到一个目录下时,这个”root”目录的 inode 值为 2

# mount /dev/sdb2 /tmp
# ls -id /tmp2 /tmp

文件恢复的原理

本文要介绍的命令是通过文件系统的 inode 值(一般是 2 )来获取文件系统信息。在 ext3 和 ext4 文件系统中,每个文件都是通过 inode 来描述其数据存放的具体位置,当文件被删除以后,inode 的数据指针部分被清零,文件目录区没有太多变化。文件的读写都是通过 inode 来实现,当 inode 数据指针被清零以后,即便文件内容还在,也没有办法把文件内容组合出来。当 ext3 和 ext4 文件系统中的元数据 metadata 发生变化时,相应的元数据 metadata 在日志文件会有一份拷贝。比如一个文件被删除了,它的 inode 信息会在日志文件中先保存一份,然后把要删除文件 inode 相关信息清零。这个日志文件是循环使用的,当操作过多时,删除的文件的 inode 日志记录会被新的数据替换,这就彻底丧失了根据 inode 找回数据的机会了。如果是大量文件的删除,这个日志文件会被反复循环利用多次,只留给最后删除的那些文件的恢复机会。

继续阅读

Postfix+Amavisd-new+SpamAssassin+Clamav

一、安装Amavisd-new+SpamAssassin

说明:amavisd-new是介于MTA与邮件分析软件(如sa、clama)之间的一个接口;可以将位于队列的邮件取出来,调用ClamAV对邮件进行病毒扫描,调用SpamAssassin对邮件内容进行过滤 。

官方网站:

http://www.ijs.si/software/amavisd/

安装参考:

http://www.shisaa.jp/postset/mailserver-3.html

http://www.postfixvirtual.net/postfixantivirus.html#amavisdnew

继续阅读

How to setup an SFTP server on CentOS

This tutorial explains how to setup and use an SFTP server on CentOS. Before I start, let me explain what actually SFTP represents and what it is used for. Currently, most people know that we can use normal FTP for transferring, downloading or uploading data from a server to client or client to server. But this protocol is getting hacked easily (if TLS is not used) by anonymous intruders as it the ports are wide open to anyone. Therefore, SFTP has been introduced to as another alternative to meet the main purpose to strengthen the security level.

SFTP stands for SSH File Transfer Protocol or Secure File Transfer Protocol. It uses a separate protocol packaged with SSH to provide a secure connection.

1. Preliminary Note

For this tutorial, I am using CentOS 7 in the 64bit version. The same steps will work on CentOS 6 as well. The tutorial result will show how a client can be provided with access to the SFTP server but unable to login to the server itself by SSH.

2. SFTP Installation

Unlike normal FTP, there’s no need to install additional packages in order to use SFTP. We just require the prebuild SSHd package that got already installed during installation on the server. Therefore, just check to confirm if you already have the required SSH package. Below are the steps:

Run:

rpm -qa|grep ssh

The output should be similar to this:

[root@localhost ~]# rpm -qa|grep sshlibssh2-1.4.3-10.el7_2.1.x86_64openssh-7.4p1-13.el7_4.x86_64openssh-server-7.4p1-13.el7_4.x86_64openssh-clients-7.4p1-13.el7_4.x86_64

That’s all, now we’ll go on how to make the SFTP configuration.

继续阅读

How to set DNS in CentOS/RHEL 7 & prevent NetworkManager from overwriting /etc/resolv.conf?

This guide shows you how to set custom DNS entries for CentOS 7 / RedHat 7 and ensure that the settings are persistent even after a reboot.

What you need

  • A CentOS 7 or a Red Hat Enterprise Linux (RHEL) 7 server
  • A couple of minutes

Overview

In CentOS and Red Hat Enterprise Linux (RHEL) 7, any custom DNS entries are stored in the file /etc/resolv.conf. However, if we simply go ahead and add our nameservers to this file, we’ll notice that after a reboot or a restart of the network.service, the file is overwritten by NetworkManager.

In this guide, we will first configure NetworkManager to not overwrite this file. Then, we will go ahead and actually add our custom nameservers to /etc/resolv.conf.

Step 1

The NetworkManager configuration is located here: /etc/NetworkManager/NetworkManager.conf Open this file using vim or your favorite text editor.

Search for the [main] section in this file. It should look something like this:

...
[main]
#plugins=ifcfg-rh,ibft
...

Add dns=none just after the [main] tag like this:

...
[main]
dns=none
#plugins=ifcfg-rh,ibft
...

Go ahead and save the file.

Step 2

Let’s restart the NetworkManager.service service so that it picks up the changes we made to the configuration.

sudo systemctl restart NetworkManager.service

Note that the service name NetworkManager.service is case-sensitive.

Step 3

Now, let’s add our nameservers to /etc/resolv.conf

Open this file in you favorite text editor and specify the name servers as follows:

# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 8.8.4.4

That’s it! You’re done. The nameservers added to /etc/resolv.conf will now persist even after a reboot. NetworkManager will not longer overwrite this file.

Centos curl ssl 替换 NSS 为 OpenSSL

系统版本是 Centos 6/7 64位。

1、先安装常用的开发环境。

yum groupinstall Development tools

2、编译OpenSSL

1.下载 OpenSSL:

wget https://www.openssl.org/source/openssl-1.0.2l.tar.gz

2.解压 OpenSSL:

tar -xzvf openssl-1.0.2l.tar.gz

3.进入 OpenSSL目录:

cd openssl-1.0.2l

4.配置并编译 OpenSSL:

./config –shared
make && make install

继续阅读

The Mega Guide To Harden and Secure CentOS 7

This tutorial only covers general security tips for CentOS 7 which can be used to harden the system. The checklist tips are intended to be used mostly on various types of bare-metal servers or on machines (physical or virtual) that provides network services.

Security and Hardening of CentOS 7

Security and Hardening of CentOS 7

However, some of tips can be successfully applied on general purpose machines too, such as Desktops, Laptops and card-sized single-board computers (Raspberry Pi).

继续阅读

Enforce Password Complexity Policy On CentOS 7/RHEL Derivatives

In this guide, we are going to learn how to enforce password complexity policy on CentOS 7/RHEL based derivatives. Our previous guide covered the enforcement of password complexity on Ubuntu 18.04. You can check the same by following the link below;

Enforce Password Complexity Policy On CentOS 7

Similar to our previous guide, we are going to use PAM pwquality modules to enforce password complexity policy on CentOS 7/RHEL based derivatives.

In Ubuntu or Debian based derivatives, we modified the, /etc/pam.d/common-password configuration file. For CentOS 7 or similar derivatives, the /etc/security/pwquality.conf or /etc/pam.d/system-auth configuration file is used.

As our normalcy, make a backup of the configuration file before making changes just in case things go south.

继续阅读

Linux下进程/程序网络带宽占用情况查看工具 — NetHogs

{“type”:”block”,”srcClientIds”:[“9592bfc1-75c1-490c-ae11-999e86836b85″],”srcRootClientId”:””}nethogs命令

有很多适用于Linux系统的开源网络监视工具。比如说,你可以用命令iftop来检查带宽使用情况。netstat用来查看接口统计报告,还有top监控系统当前运行进程。但是如果你想要找一个能够按进程实时统计网络带宽利用率的工具,那么NetHogs值得一看。

NetHogs是一个开源的命令行工具(类似于Linux的top命令),用来按进程或程序实时统计网络带宽使用率。

继续阅读

How to Install and Configure ‘Cache Only DNS Server’ with ‘Unbound’ in RHEL/CentOS 7

Version:1.0 StartHTML:000000222 EndHTML:000042418 StartFragment:000012983 EndFragment:000042312 StartSelection:000012983 EndSelection:000042308 SourceURL:https://www.tecmint.com/setup-dns-cache-server-in-centos-7/ How to Install and Configure ‘Cache Only DNS Server’ with ‘Unbound’ in RHEL/CentOS 7

Caching name servers using ‘Unbound‘ ( is a validating, recursive, and caching DNS server software ), back in RHEL/CentOS 6.x (where x is version number), we used bind software to configure DNS servers.

Here in this article, we are going to use ‘unbound‘ caching software to install and configure a DNS Server in RHEL/CentOS 7 systems.

Setup Cahing DNS Server in RHEL/CentOS 7

DNS cache servers are used to resolve any DNS query they receive. If the server caches the query and in future the same queries requested by any clients the request will be delivered from DNS ‘unbound‘ cache, this can be done in milliseconds than the first time it resolved.

Caching will only act as a agent to resolve the query of client from any one of the forwarders. Using caching server, will reduce the loading time of webpages by keeping the cache database in unbound server.

My Server and Client Setup

For demonstration purpose, I will be using two systems. The first system will act as a Master (Primary) DNS server and the second system will act as a local DNS client.

Master DNS Server
Operating System   :    CentOS Linux release 7.0.1406 (Core)
IP Address	   :	192.168.0.50
Host-name	   :	ns.tecmintlocal.com
Client Machine
Operating System   :	CentOS 6
IP Address	   :	192.168.0.100
Host-name	   :	client.tecmintlocal.com

Step 1: Check System Hostname and IP

1. Before setting up a caching DNS server, make sure that you’ve added correct hostname and configured correct static IP address for your system, if not set the system static IP address.

2. After, setting correct hostname and static IP address, you can verify them with the help of following commands.

# hostnamectl
# ip addr show | grep inet

Check IP Address

Step 2: Installing and Configuring Unbound

3. Before installing ‘Unbound’ package, we must update the our system to latest version, after that we can install the unbound package.

# yum update -y
# yum install unbound -y

4. After package has been installed, make a copy of the unbound configuration file before making any changes to original file.

# cp /etc/unbound/unbound.conf /etc/unbound/unbound.conf.original

5. Next, use any of your favorite text editor to open and edit ‘unbound.conf‘ configuration file.

# vim /etc/unbound/unbound.conf
Copy Unbound DNS Configuration

Once the file is opened for editing, make the following changes:

Interfaces

Search for Interface and enable the interface which we going to use or if our server have multiple interfaces we have to enable the interface 0.0.0.0.

Here Our server IP was 192.168.0.50, So, i’am going to use unbound in this interface.

Interface 192.168.0.50
Enable IPv4 and Protocol Supports

Search for the following string and make it ‘Yes‘.

do-ip4: yes
do-udp: yes
do-tcp: yes
Enable the logging

To enable the log, add the variable as below, it will log every unbound activities.

logfile: /var/log/unbound
Hide Identity and Version

Enable following parameter to hide id.server and hostname.bind queries.

hide-identity: yes

Enable following parameter to hide version.server and version.bind queries.

hide-version: yes
Access Control

Then search for access-control to allow. This is to allow which clients are allowed to query this unbound server.

Here I have used 0.0.0.0, that means anyone send query to this server. If we need to refuse query for some range of network we can define which network need to be refuse from unbound queries.

access-control: 0.0.0.0/0 allow

Note: Instead of allow, we can replace it with allow_snoop this will enable some additional parameters like dig and it support both recursive and non recursive.

Domain Insecure

Then search for domain-insecure. If our domain is works with DNS sec keys, we need to define our server available for domain-insecure. Here our domain will be treated as insecure.

domain-insecure: "tecmintlocal.com
Forward Zones

Then change the forwarders for our requested query not fulfilled by this server it will forward to root domain (. ) and resolve the query.

forward-zone:
        name: "."
        forward-addr: 8.8.8.8
        forward-addr: 8.8.4.4

Finally, save and quit the configuration file using wq!.

6. After making the above configuration, now verify the unbound.conf file for any errors using the following command.

# unbound-checkconf /etc/unbound/unbound.conf

Check Unbound DNS Configuration

7. After file verification over without any errors, you can safely restart the ‘unbound’ service and enable it at system startup.

# systemctl start unbound.service
# sudo systemctl enable unbound.service

Start Unbound DNS Service

Step 3: Test DNS Cache Locally

8. Now it’s time to check our DNS cache, by doing a ‘drill’ (query) one ‘india.com‘ domain. At first the ‘drill‘ command results for ‘india.com‘ domain will take some milliseconds, and then do a second drill and have a note on Query time it takes for both drills.

drill india.com @192.168.0.50

Check DNS Cache Locally

Did you see in the above output, the first query taken almost 262 msec to resolve and the second query takes 0 msec to resolve domain (india.com).

That means, the first query gets cached in our DNS Cache, so when we run ‘drill’ second time the query served from our local DNS cache, this way we can improve loading speed of websites.

Step 4: Flush Iptables and Add Firewalld Rules

9. We can’t use both iptables and firewalld at same time on same machine, if we do both will conflict with each other, thus removing ipables rules will be a good idea. To remove or flush the iptables, use the following command.

# iptables -F

10. After removing iptables rules permanently, now add the DNS service to firewalld list permanently.

# firewall-cmd --add-service=dns
# firewall-cmd --add-service=dns --permanent

11. After adding DNS service rules, list the rules and confirm.

# firewall-cmd --list-all

Add DNS to Firewalld

Step 5: Managing and Troubleshooting Unbound

12. To get the current server status, use the following command.

# unbound-control status

Check Unbound DNS Status

Dumping DNS Cache

13. If in-case you would like to have a dump of a DNS cache information in a text file, you can redirect it to some file using below command for future use.

 # unbound-control dump_cache > /tmp/DNS_cache.txt

Backup DNS Cache

14. To restore or import the cache from the dumped file, you can use following command.

# unbound-control dump_cache < /tmp/DNS_cache.txt

Restore DNS Cache

Flushing DNS Records

15. To check whether the specific address was resolved by our forwarders in unbound cache Server, use the below command.

# unbound-control lookup google.com

Check DNS Lookup

16. Some times if our DNS cache server will not reply our query, in mean time we can use to flush the cache to remove information such as A, AAA, NS, SO, CNAME, MX, PTR etc.. records from DNS cache. We can remove all information using flush_zone this will remove all informations.

# unbound-control flush www.digitalocean.com
# unbound-control flush_zone tecmintlocal.com

17. To check which forwards are currently used to resolve.

# unbound-control list_forwards

Check Current DNS Forwards

Step 6: Client Side DNS Configuration

18. Here I’ve used a CentOS 6 server as my client machine, IP for this machine is 192.168.0.100 and I’m going to use my unbound DNS server IP (i.e Primary DNS) in it’s interface configuration.

Log-into the Client machine and set the Primary DNS server IP to our unbound server’s IP.

Run the setup command and choose network configuration from TUI network manager.

Then choose DNS configuration, insert the unbound DNS server’s IP as Primary DNS, but here i have used both in Primary and Secondary because I don’t have any other DNS server.

Primary DNS	: 192.168.0.50
Secondary DNS	: 192.168.0.50

Select Network Configuration

Select DNS Configuration

Enter DNS IP Address

Click OK –> Save&Quit –> Quit.

19. After adding Primary and Secondary DNS IP addresses, now it’s time to restart the network using following command.

# /etc/init.d/network restart

Restart Network

20. Now time to access any one of the website from client machine and check for the cache in unbound DNS server.

# elinks aol.com
# dig aol.com

Check Website

Query Website

Conclusion

Earlier we were used to setup DNS cache server using bind package in RHEL and CentOS systems. Now, we have seen how to setup a DNS cache server using unbound package. Hope this will resolve your query request quicker than the bind pacakge.

Linux 磁盘IO性能测试工具:FIO

1、FIO安装

CentOS下:

$ yum install fio

2、FIO简介

FIO是测试IOPS的非常好的工具,用来对硬件进行压力测试和验证。
项目主页:https://github.com/axboe/fio
FIO输出结果详细解释可参见:https://tobert.github.io/post/2014-04-17-fio-output-explained.html
注:fio用于测试磁盘性能,不是文件系统,测试之前需要先把要测试的磁盘卸载掉,测试完后需格式化一遍再挂载上去。相关命令如下:

# 卸载磁盘
umount /mnt/datadir1
# 格式化磁盘
mkfs.ext4 /dev/sdd
# 挂载磁盘
mount -t ext4 /dev/sdd /mnt/datadir1

# 自动挂载,在里面添加该盘
vim /etc/fstab

3、FIO参数

例如:

$ fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=mytest1

说明:
filename=/dev/sdd 测试文件名称,通常选择需要测试的盘的data目录。
direct=1 测试过程绕过机器自带的buffer。使测试结果更真实。
rw=randwrite 测试模式
=read 顺序读
=write 顺序写
=randwrite 随机写
=randread 随机读
=rw,readwrite 顺序混合读写
=randrw 随机混合读写

bs=4k 单次io的块文件大小为4k
bsrange=512-2048 同上,提定数据块的大小范围
size=5G 本次的测试文件大小为5g,以每次4k的io进行测试。
numjobs=30 本次的测试线程为30个
runtime=1000 测试时间为1000秒,如果不写则一直将5g文件分4k每次写完为止
iodepth 队列深度,只有使用libaio时才有意义。这是一个可以影响IOPS的参数。
ioengine=psync io引擎使用psync方式
=libaio Linux专有的异步IO

rwmixwrite=30 在混合读写的模式下,写占30%
group_reporting 关于显示结果的,汇总每个进程的信息。
lockmem=1G 只使用1g内存进行测试。
zero_buffers 用0初始化系统buffer。
nrfiles=8 每个进程生成文件的数量。

点击查看更多详细参数说明

4、FIO测试示例

fio工具使用特别注意:

  1. 请不要在系统盘上进行 fio 测试,避免损坏系统重要文件
  2. fio测试建议在空闲的、未保存重要数据的硬盘上进行,并在测试完后重新制作文件系统。请不要在业务数据硬盘上测试,避免底层文件系统元数据损坏导致数据损坏
  3. 测试硬盘性能时,推荐直接测试裸盘(如 vdb);测试文件系统性能时,推荐指定具体文件测试(如 /data/file)

4.1 顺序读测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=read -filename=/dev/sdd -name=”BS 4KB read test” -iodepth=16 -runtime=601

4.2 顺序写测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=write -filename=/dev/sdd -name=”BS 4KB write test” -iodepth=16 -runtime=601

4.3 随机读测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randread -filename=/dev/sdd -name=”BS 4KB randread test” -iodepth=16 -runtime=601

4.4 随机写测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -filename=/dev/sdd -name=”BS 4KB randwrite test” -iodepth=16 -runtime=601

4.5 随机读写混合测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randrw -rwmixread=70 -filename=/dev/sdd -name=”BS 4KB randrw 70 test” -iodepth=16 -runtime=601

5、FIO输出结果

例如:

BS 4KB randread test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 thread

BS 4KB randread test: (groupid=0, jobs=1): err= 0: pid=26408: Mon Feb 27 16:43:07 2017
read : io=19908KB, bw=339689B/s, iops=82, runt= 60013msec
# fio做了19908KB的IO,bandwidth速率为339689B/s,总IOPS为82,运行时间为60013毫秒

slat (usec): min=18, max=97, avg=23.60, stdev= 2.79
# slat (submission latency):代表 盘需要多久将IO提交到kernel做处理
# usec 微秒

clat (msec): min=1, max=23, avg=12.03, stdev= 3.79
# clat (completion latency):命令提交到kernel到IO做完之间的时间,不包括submission latency

 lat (msec): min=1, max=23, avg=12.05, stdev= 3.79
# 从IO结构体创建时刻开始,直到紧接着clat完成的时间

clat percentiles (usec):
 |  1.00th=[ 4080],  5.00th=[ 5984], 10.00th=[ 7136], 20.00th=[ 8640],
 | 30.00th=[ 9920], 40.00th=[10944], 50.00th=[11840], 60.00th=[12992],
 | 70.00th=[14016], 80.00th=[15296], 90.00th=[17024], 95.00th=[18560],
 | 99.00th=[20864], 99.50th=[21632], 99.90th=[22400], 99.95th=[22912],
 | 99.99th=[23168]
# Completion latency百分数

bw (KB  /s): min=  276, max=  373, per=100.00%, avg=331.54, stdev=18.83
# 带宽(bandwidth)

lat (msec) : 2=0.02%, 4=0.82%, 10=30.32%, 20=66.75%, 50=2.09%
# 30.32%的request延迟在4~10毫秒,66.75%的request延迟在10~20毫秒,以此类推

cpu : usr=0.10%, sys=0.25%, ctx=5057, majf=0, minf=8
# 用户/系统CPU占用率,进程上下文切换(context switch)次数,主要和次要(major and minor)页面错误数量(page faults)。由于测试是配置成使用直接IO,page faults数量应该极少。

IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
# iodepth设置,用来控制同一时刻发送给OS多少个IO。这完全是纯应用层面的行为,和盘的IO queue不是一回事

 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 #submit和complete代表同一时间段内fio发送上去和已完成的IO数量

 issued    : total=r=4977/w=0/d=0, short=r=0/w=0/d=0
 # 发送的IO数量

 latency   : target=0, window=0, percentile=100.00%, depth=1
 # Fio可以配置一个延迟目标值,这个值可以调节吞吐量直到达到预设的延迟目标

Run status group 0 (all jobs):

Fio支持把不同的测试聚合

READ: io=19908KB, aggrb=331KB/s, minb=331KB/s, maxb=331KB/s, mint=60013msec, maxt=60013msec
# 汇总输出吞吐量和时间。
# io=表示总共完成的IO数量。在基于时间的测试中这是一个变量,在基于容量的测试中,这个值能匹配size参数。
# aggrb是所有进程/设备的汇总带宽。
# minb/maxb表示测量到的最小/最大带宽。
# mint/maxt表示测试的最短和最长耗时。和io=参数类似,时间值对于基于时间的测试应该能匹配runtime参数,对于基于容量的测试是一个变量。

Disk stats (read/write):
sdd: ios=4969/0, merge=0/0, ticks=59703/0, in_queue=59702, util=99.51%