PowerShell提速和多线程

本文目录

PowerShell性能优化系列文章

  1. PowerShell优化和性能测试
  2. 让你的PowerShell For循环提速四倍
  3. 优化PowerShell的性能和内存消耗
  4. PowerShell提速和多线程
  5. 优化PowerShell脚本的几个小技巧
  6. 轻量级的PowerShell性能测试

本篇文章本是源自PowerShell.Com上的一个教程视频,讲师为Dr. Tobias Weltner。有时间的朋友可以直接去看英文视频。我英文水平有限,还没到达单靠纯听力就能给视频加中文字幕的能力,所以就把原视频中的观点与例子分享出来。

概述

我们平时写脚本时,经常会提醒自己要多使用管道,要多使用流模式,少占内存,少占CPU。但是这篇文章会反其道而行之,少用管道,通过内存和CPU的占用来提高效率,也就是我们通常算法上说的用空间来换取时间。机器配置高,有的用,而不用就是浪费。

比如下面的场景:

  • 写一个大文件可能需要3.6分钟,提高性能后,只需3秒钟
  • 读一个大文件可能需要77秒钟,提高性能后,只需2秒钟
  • 检查250台机器的是否在线,需要23.2分钟,提高性能后,只需26秒钟
PowerShell性能提高前后对比图

PowerShell性能提高前后对比图

这一切性能的提升都是有偿的,需要你额外的投资。

投资更多内存

在PowerShell中推崇的管道主要是为了限制内存的使用量,让管道中的元素像流水线中的零件或者半成品一样,从车间一个一个穿过。但是管道并不是最快的,且看下面的随机数的例子。

管道流模式可以节省内存

管道流模式可以节省内存

随机数的例子

12345678910#这个很快PS> 1..100 |Get-Random90 #这个稍有延迟,可以容忍PS> 1..10000 |Get-Random4868 #这个慢的受不了,所以直接Ctrl-C取消。PS> 1..1000000 |Get-Random

是不是就意味着1000000这么大的一个数组Get-Random天生就这么慢,非也。换个方法:

123#这样做,快的一塌糊涂啊PS> Get-Random -InputObject (1..1000000)317486

原因是前者使用了管道,产生一条数据,流过一条数据。而后者是直接一次性产生全部数据,然后交给Get-Random,所以快。

写文件的例子

123$all = @('Some Test' * 20)*1000000$file "$Env:TEMP\testfile.txt"$all Out-File $file

上面的脚本执行大概需要215 秒钟(3.6分钟)
换个方式,使用inputObject,仅用了101秒钟(1.8分钟)

1Out-File $file -InputObject $all

是不是优化就止于此了呢,不,下面的结果会亮瞎我的眼睛啊,只需要2.5秒钟

1[Io.file]::WriteAllLines( $file$all,[text.encoding]::Unicode)

真真没想到直接调用.NET方法差别会这么大,但是我用的是PowerShell,你给我整.NET方法,用键盘敲起来未免坑爹啊,那就试试Set-Content吧,只需要3.1秒钟

1Set-Content $file -Value $all -Encoding Unicode

与WriteAllLine相比,稍慢,也可以忍受。

通过这个写文件的例子对比不难发现,3.6分钟/3.1秒钟=70,,速度几乎提高了70倍啊。

总结一下,写文件时注意两点即可:

  1. 不要使用管道。
  2. 不要使用神马的Out-这样的命令(因为它会做格式化)

读文件的例子

对于我们刚才创建的文本文件,一般读取时,我们习惯使用Get-Content,需要77秒钟(1.3分钟)

1Get-Content $file

而如果使用.NET方法,只需1.8秒钟

1[io.file]::ReadAllLines($file)

Get-Content为什么会这么慢,因为它要一行一行来读,并把读取的数据存储成数组,所以慢。但是Get-Content有一个参数ReadCount,把值设为0,一次性全部读取,只需2.2秒钟

1Get-Content $file -ReadCount 0

把文件读出来一般是需要处理的,如果这样肯定不是你预期的,因为只会输出一个X:

1Get-Content $file -ReadCount 0 | foreach {"X" }

这样就对了,只需要2.7秒钟

12$text Get-Content $file -ReadCount 0foreach ($line in $text) {'X'}

千万不要多次一举,引入管道,得68秒钟啊:

12$text Get-Content $file -ReadCount 0 | foreach {"X" }$text ForEach-Object {'X'}

警告

  • cmdlet明显非常慢
  • .NET一些底层的方法相对较快

解药

  • 不要轻易引入管道。
  • 尽量使用传统的For或者foreach循环

如果你还不相信,请继续看例子。

多用循环,少用管道的例子

11..1000000  | ForEach-Object "looping for the $_ Time"}

上面使用管道,执行时间为6.9秒钟。如果换成简单的For循环,只需要0.5秒钟,速度提高了14倍。

1234For $x=1; $x -le 100000;$x++){"Looping for the $x. time"}

再看一个抑制输出的例子,三个写法效果一样,速度相差几十倍。

12345678#耗时0.1毫秒'Hello' |out-null #耗时 0.002毫秒,速度提高了56倍$null'Hello' #耗时0.0025毫秒,速度也很快[void] 'Hello'

投资更多CPU

PowerShell默认是单线程的执行的,只能一行命令接着一行命令来执行。我们可以使用PowerShell的后台Job来提高效率:对于批量任务,启用多个后台任务去处理,使用wait-job等待所有的任务结束。

多个后台任务批处理

多个后台任务批处理

先看一个顺序执行的例子。

12345678910111213$start Get-Date$code1 = { Start-Sleep -Seconds 5; 'A' }$code2 = { Start-Sleep -Seconds 6; 'B'}$code3 = { Start-Sleep -Seconds 7; 'C'} $result1,$result2,$result3= (& $code1),(& $code2),(& $code3) $end =Get-Date$timespan$end $start$seconds $timespan.TotalSeconds Write-Host "总耗时 $seconds 秒."Write-Host "三个脚本块总共延时 18 秒"

输出为(耗时18秒钟):

总耗时 18.0240865 秒.
三个脚本块总共延时 18 秒

同样的任务,使用后台Job多线程执行:

123456789101112131415161718$start Get-Date$code1 = { Start-Sleep -Seconds 5; 'A' }$code2 = { Start-Sleep -Seconds 6; 'B'}$code3 = { Start-Sleep -Seconds 7; 'C'} $job1 Start-Job -ScriptBlock $code1$job2 Start-Job -ScriptBlock $code2$job3 Start-Job -ScriptBlock $code3 $alljobs =  Wait-Job $job1,$job2,$job3$result1,$result2,$result3 Receive-Job $alljobs $end =Get-Date $timespan$end $start$seconds $timespan.TotalSecondsWrite-Host "总耗时 $seconds 秒."Write-Host "三个脚本块总共延时 18 秒"

输出为(耗时10秒钟):

总耗时 10.3778469 秒.
三个脚本块总共延时 18 秒

效率提升很明显。

使用后台Job的开销

  1. 每一个新的任务执行时都会使用一个新PowerShell进程。(所以所谓的多线程并不是真正的多线程,而是工作在进程级别上)
  2. 每一个任务的结果需要序列化后,跨进程传递给调度的主进程。
  3. 没有节流机制(所以要注意控制后台Job的数量)。
计算后台Job的开销

使用这段示例脚本:

1234567891011121314151617181920212223242526272829303132333435363738# (C) 2012 Dr. Tobias Weltner# you may freely use this code for commercial or non-commercial purposes at your own risk# as long as you credit its original author and keep this comment block.# For PowerShell training or PowerShell support, feel free to contact tobias.weltner@email.de $code = {$begin Get-Date$result Get-Process$end Get-Date $begin$end# play here by reducing the returned data,# i.e. use select-object to pick specific properties:$result} $start Get-Date $job Start-Job -ScriptBlock $code$null Wait-Job $job$completed Get-Date $result Receive-Job $job$received Get-Date $spinup $result[0]$exit $result[1] $timeToLaunch = ($spinup $start).TotalMilliseconds$timeToExit = ($completed $exit).TotalMilliseconds$timeToRunCommand = ($exit $spinup).TotalMilliseconds$timeToReceive = ($received $completed).TotalMilliseconds '{0,-30} : {1,10:#,##0.00} ms' -f 'Time to set up background job'$timeToLaunch'{0,-30} : {1,10:#,##0.00} ms' -f 'Time to run code'$timeToRunCommand'{0,-30} : {1,10:#,##0.00} ms' -f 'Time to exit background job'$timeToExit'{0,-30} : {1,10:#,##0.00} ms' -f 'Time to receive results'$timeToReceive

脚本会在后台的Job的启动,运行,结束和接受数据每个阶段设置时间戳,然后计算各个阶段耗费的时间。

第一次运行时,输出结果为:

Time to set up background job  :     270.01 ms
Time to run code               :      10.00 ms
Time to exit background job    :   1,550.06 ms
Time to receive results        :      10.00 ms

主要的延迟在Job退出时,因为上面的Job返回了大量的数据,假如我们注释掉上面示例代码的第15行,再执行一遍:

Time to set up background job  :     350.01 ms
Time to run code               :      10.00 ms
Time to exit background job    :      10.00 ms
Time to receive results        :       0.00 ms

执行效率明显提高,但是绝大多数的后台Job应当都会返回数据的,哪怕返回一点,所以我们把上面脚本的第15改成这样,再执行一遍:

1$result select-object Name,CPU
Time to set up background job  :     420.02 ms
Time to run code               :      10.00 ms
Time to exit background job    :     100.01 ms
Time to receive results        :       0.00 ms

稍有延迟,可以忍受。

结论

通过这个例子主要是告诉大家,影响后台任务的关键因素是返回的数据量,如果没有特别的需求,尽量在后台Job中不返回数据,或者少返回数据。

从进程间迁移到进程内

正如后台Job为用户所诟病的那样,它不是真正的多线程,而是多进程。所以下面我们开始从进程间迁移到进程内,使用实至名归的PowerShell多线程,因为它会在PowerShel.exe内部创建一个新的线程。

使用进程内多线程的优点

  • 不需要新的宿主进程
  • 不需要序列化结果
  • 线程内通信方便
  • 运行空间池提供了自动内存节流
开启一个线程

先看一个简单的在PowerShell中开启一个同步线程的例子

PS>  # Running New Thread Synchronously:
PS> $code = { Start-Sleep -Seconds 2; "Hello" }
PS> $newPowerShell = [PowerShell]::Create().AddScript($code)
PS> $newPowerShell.Invoke()
Hello
让线程异步运行

稍加改动,使用BeginInvoke()异步执行,使用EndInvoke()返回线程的数据:

123456789101112$code = {Start-Sleep -Seconds 2; "Hello"} $newPowerShell [PowerShell]::Create().AddScript($code)$handle $newPowerShell.BeginInvoke() while ($handle.IsCompleted -eq $false) {Write-Host '.' -NoNewlineStart-Sleep -Milliseconds 500} Write-Host ''$newPowerShell.EndInvoke($handle)

输出示例,先有原点的进度条

PS>
.....
Hello
演示一个进度提示器
123456789101112131415161718192021function Start-Progress {param([ScriptBlock]$code) $newPowerShell [PowerShell]::Create().AddScript($code)$handle $newPowerShell.BeginInvoke() while ($handle.IsCompleted -eq $false) {Write-Host '.' -NoNewlineStart-Sleep -Milliseconds 500} Write-Host '' $newPowerShell.EndInvoke($handle) $newPowerShell.Runspace.Close()$newPowerShell.Dispose()}

记得要在运行空间使用结束后,调用Close和Dispose方法释放资源。

先显示进度信息,然后返回结果。

PS> Start-Progress -code {Get-HotFix}
..

Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
ETS-V-TEST-01 Update           KB2899189_... NT AUTHORITY\SYSTEM  5/14/2014 12:00:00 AM
ETS-V-TEST-01 Update           KB2919355     ETS-V-TEST-01\Adm... 3/18/2014 12:00:00 AM
ETS-V-TEST-01 Update           KB2919442     ETS-V-TEST-01\Adm... 3/18/2014 12:00:00 AM
ETS-V-TEST-01 Security Update  KB2920189     NT AUTHORITY\SYSTEM  5/14/2014 12:00:00 AM
ETS-V-TEST-01 Security Update  KB2926765     NT AUTHORITY\SYSTEM  5/15/2014 12:00:00 AM
ETS-V-TEST-01 Security Update  KB2931366     NT AUTHORITY\SYSTEM  5/14/2014 12:00:00 AM

这个执行起来太快了,换一个慢一点的命令,效果更明显。

PS> Start-Progress -code {Get-WmiObject -Class Win32_product}
................................................
IdentifyingNumber : {90150000-0015-0409-0000-0000000FF1CE}
Name              : Microsoft Access MUI (English) 2013
Vendor            : Microsoft Corporation
Version           : 15.0.4569.1506
Caption           : Microsoft Access MUI (English) 2013

IdentifyingNumber : {90150000-0115-0409-0000-0000000FF1CE}
Name              : Microsoft Office Shared Setup Metadata MUI (English) 2013
Vendor            : Microsoft Corporation
Version           : 15.0.4569.1506
Caption           : Microsoft Office Shared Setup Metadata MUI (English) 2013
演示***
123456789101112131415161718192021222324252627function Start-Timebomb {param([Int32]$Seconds, [ScriptBlock]$Action = { Stop-Process -Id $PID }) $Wait "Start-Sleep -seconds $seconds"$script:newPowerShell [PowerShell]::Create().AddScript($Wait).AddScript($Action)$handle $newPowerShell.BeginInvoke()Write-Warning "Timebomb is active and will go off in $Seconds seconds unless you call Stop-Timebomb before."} function Stop-Timebomb {if $script:newPowerShell -ne $null) {Write-Host 'Trying to stop timebomb...' -NoNewline$script:newPowerShell.Stop()$script:newPowerShell.Runspace.Close()$script:newPowerShell.Dispose()Remove-Variable newPowerShell -Scope scriptWrite-Host 'Done!'else {Write-Warning 'No timebomb found.'}}

在控制台上开启了***后,如果没有及时停止,倒计时结束后,控制台会自动关闭。

PS> Start-Timebomb -Seconds 10
WARNING: Timebomb is active and will go off in 10 seconds unless you call Stop-Timebomb before.
监控脚本的执行时间

如果你想让倒计时的信息显示在控制台的标题栏,只需要修改上面的脚本第10行,修改成:

1$Wait "1..$seconds | foreach-object {start-sleep -seconds 1;  [console]::Title=""`$($Seconds-`$_) seconds remaining`"}"
监控脚本的运行内存

如果一个脚本运行时,占用的内存超过了限制,就自动终结掉这个进程。

123456789101112131415161718192021222324252627282930313233343536373839404142function Start-TimebombMemory {param([Int32]$MemoryMB=30, [ScriptBlock]$Action = { Stop-Process -Id $PID }) $Wait '$initial = (Get-Process -Id $PID).WorkingSet$threshold = (XXX * 1MB)do  {$memory = ((Get-Process -Id $PID).WorkingSet - $initial)Start-Sleep -Seconds 1[system.Console]::Title = ("Current Memory Load: {0:0.00} MB. Threshold: XXX MB" -f ($memory/1MB))} while ($memory -lt $threshold) $message1 = "Shell is using {0:0.0} MB which is exceeding the threshold by {1:0.0} MB." -f ($memory/1MB), (($memory-$threshold)/1MB)$message2 = "Shell will be aborted in 5 seconds. There is nothing you can do about it, sorry."[System.Console]::WriteLine($message1)[System.Console]::WriteLine($message2)Start-Sleep -Seconds 5' -replace 'XXX'$MemoryMB $script:newPowerShellMB [PowerShell]::Create().AddScript($Wait).AddScript($Action)$handle $newPowerShellMB.BeginInvoke()Write-Warning "Timebomb is active and will go off when the shell uses more than $memoryMB MB -  unless you call Stop-Timebomb before."} function Stop-TimebombMemory {if $script:newPowerShellMB -ne $null) {Write-Host 'Trying to stop timebomb...' -NoNewline$script:newPowerShellMB.Stop()$script:newPowerShellMB.Runspace.Close()$script:newPowerShellMB.Dispose()Remove-Variable newPowerShellMB -Scope scriptWrite-Host 'Done!'else {Write-Warning 'No timebomb found.'}}

执行了Start-TimebombMemory后,会在PowerShell的控制台动态显示当前PowerShell进程占用的内存和阈值,如果内存占用超标,打印信息提示用户,并在5秒钟后自动关闭当前进程。

动态监控脚本的运行内存

动态监控脚本的运行内存

创建一个STA模式的线程

你写了一个函数,调用winform的OpenFileDialog来打开文件选择对话框。很不幸如果当前的控制台运行在MTA模式下,则对话框不能显示。所以为了增强兼容性,给你的函数单独指定一个线程运行,因为在运行空间中可以指定Apartment State。具体看下面的代码:

1234567891011121314151617181920212223242526272829303132333435363738function Show-OpenFileDialog {param([string]$Title='Select a file',[string]$Path=$home,[string]$Filter "All Files (*.*)|*.*")  $code = {param([string]$Title,[string]$Path,[string]$Filter "All Files (*.*)|*.*") Add-Type -AssemblyName System.Windows.Forms $DialogOpen New-Object System.Windows.Forms.OpenFileDialog$DialogOpen.InitialDirectory = $Path$DialogOpen.Filter $Filter$DialogOpen.Title = $Title$Result $DialogOpen.ShowDialog()if ($Result -eq "OK"){$DialogOpen.FileName}} $newRunspace [RunSpaceFactory]::CreateRunspace()$newRunspace.ApartmentState = 'MTA'$newRunspace.Open()$newPowerShell [PowerShell]::Create()$newPowerShell.Runspace = $newRunspace[void]$newPowerShell.AddScript($code).AddArgument($Title).AddArgument($Path).AddArgument($Filter)$newPowerShell.Invoke()$newPowerShell.Runspace.Close()$newPowerShell.Dispose()}
多线程中的关键组件
  • PowerShell:代表线程
  • RunSpace:代表Powershell会话
  • BeginInvoke():返回等待的句柄
  • EndInvoke():返回结果对象
  • MTA和STA模式可以完全控制
  • 每次执行完毕后,记得释放RunSpace,销毁线程。
启用节流
  1. 创建一个RunSpace 池
  2. 使用RunSpace池代替Runspace
  3. 在池中控制活动的RunSpace个数
演示简单的运行空间池

限定活动的线程最多为5,这样当尝试开启40个线程时,并不是一下子开启,而是排队等候空闲的线程池,每次最多只能有5个活动的线程池。

12345678910111213141516$throttleLimit = 5$iss [system.management.automation.runspaces.initialsessionstate]::CreateDefault()$Pool [runspacefactory]::CreateRunspacePool(1, $throttleLimit$iss$Host)$Pool.Open() $ScriptBlock = {param($id)Start-Sleep -Seconds 2[System.Console]::WriteLine("Done processing ID $id")} for ($x = 1; $x -le 40; $x++) {$powershell ::Create().AddScript($ScriptBlock).AddArgument($x)$powershell.RunspacePool = $Pool$handle $powershell.BeginInvoke()}
从多线程中接受数据
1234567891011121314151617181920212223242526272829303132333435363738$throttleLimit = 4$SessionState [system.management.automation.runspaces.initialsessionstate]::CreateDefault()$Pool [runspacefactory]::CreateRunspacePool(1, $throttleLimit$SessionState$Host)$Pool.Open() $ScriptBlock = {param($id) Start-Sleep -Seconds 2"Done processing ID $id"} $threads = @() $handles for ($x = 1; $x -le 40; $x++) {$powershell ::Create().AddScript($ScriptBlock).AddArgument($x)$powershell.RunspacePool = $Pool$powershell.BeginInvoke()$threads += $powershell} do {$i = 0$done $trueforeach ($handle in $handles) {if ($handle -ne $null) {if ($handle.IsCompleted) {$threads[$i].EndInvoke($handle)$threads[$i].Dispose()$handles[$i] = $nullelse {$done $false}}$i++}if (-not $done) { Start-Sleep -Milliseconds 500 }until ($done)

声明:本文所有观点,图片,示例脚本引用自 Dr. Tobias Weltner的视频教程。感谢 Dr. Tobias Weltner!
示例脚本出处demofiles_multithreading
视频出处Speeding Up PowerShell (网盘:密码: 7w21 )

iCACLS使用说明

iCACLS.exe (2003 sp2, Vista+)

Change file and folder permissions – display or modify Access Control Lists (ACLs) for files and folders.
iCACLS resolves various issues that occur when using the older CACLS & XCACLS

Syntax
   Add or remove permissions:
      ICACLS Name
         [/grant[:r] User:Permission[...]]
            [/deny User:Permission[...]]
               [/remove[:g|:d]] User[...]]
                  [/inheritance:e|d|r ]
                     [/setintegritylevel Level[...]]
                        [/T] [/C] [/L] [/Q]

   Store ACLs for one or more directories matching name into aclfile for later use with /restore:
      ICACLS name /save aclfile [/T] [/C] [/L] [/Q]
    
   Restore ACLs to all files in directory:
      ICACLS directory [/substitute SidOld SidNew [...]]
          /restore aclfile [/C] [/L] [/Q]

   Change Owner:
      ICACLS name /setowner user [/T] [/C] [/L] [/Q]

   Find items with an ACL that mentions a specific SID:
      ICACLS name /findsid Sid [/T] [/C] [/L] [/Q]

   Find files whose ACL is not in canonical form or with a length inconsistent with the ACE count:
      ICACLS name /verify [/T] [/C] [/L] [/Q]
 
   Replace ACL with default inherited acls for all matching files:
      ICACLS name /reset [/T] [/C] [/L] [/Q]
   This is equivalent to “Replace all child permission entries with inheritable permission from this object” in the GUI.

Key
   name  The File(s) or folder(s) the permissions will apply to.

   /T  Traverse all subfolders to match files/directories. This will apply permission changes to
       all subfolders whether or not they are set to inherit permissions from the parent. On very large
       directory structures this may take some time as the command has to traverse the entire tree.
   
   /C  Continue on file errors (access denied) Error messages are still displayed.
  
   /L  Perform the operation on a symbolic link itself, not its target.

   /Q  Quiet - supress success messages.

   /grant :r user:permission
       Grant access rights, with :r, the permissions
       will replace any previouly granted explicit permissions (for the given user).
       Otherwise the permissions are added.

   /deny user:permission
       Explicitly deny the specified user access rights.
       This will also remove any explicit grant of the 
       same permissions to the same user.

   /remove[:[g|d]] User 
       Remove all occurrences of User from the acl. 
       :g remove all granted rights to that User/Sid.
       :d remove all denied rights to that User/Sid.

   /inheritance:e|d|r
          e - Enable inheritance
          d - Disable inheritance and copy the ACEs 
          r - Remove all inherited ACEs

   /setintegritylevel [(CI)(OI)]Level 
       Add an integrity ACE to all matching files. 
       level is one of L,M,H (Low Medium or High)

             Mandatory Label\Low Mandatory Level    = Low.
             Mandatory Label\Medium Mandatory Level = Medium/Standard.
             Mandatory Label\High Mandatory Level   = Elevated.

             If No mandatory label is displayed in the output, it is Medium by default. 

       A Directory Inheritance option for the integrity ACE can precede the level
       and is applied only to directories:

   user   A user account, Group or a SID

   /restore  Apply the acls stored in ACLfile to the files in directory

   permission is a permission mask and can be specified in one of two forms:
        a sequence of simple rights:
                D - Delete access
                F - Full access (Edit_Permissions+Create+Delete+Read+Write)
                N - No access
                M - Modify access (Create+Delete+Read+Write)
                RX - Read and eXecute access
                R - Read-only access
                W - Write-only access
        a comma-separated list in parentheses of specific rights:
                DE - Delete
                RC - read control
                WDAC - write DAC
                WO - write owner
                S - synchronize
                AS - access system security
                MA - maximum allowed
                GR - generic read
                GW - generic write
                GE - generic execute
                GA - generic all
                RD - read data/list directory
                WD - write data/add file
                AD - append data/add subdirectory
                REA - read extended attributes
                WEA - write extended attributes
                X - execute/traverse
                DC - delete child
                RA - read attributes
                WA - write attributes
        inheritance rights can precede either form and are applied
        only to directories:
                (OI) - object inherit
                (CI) - container inherit
                (IO) - inherit only
                (NP) - don’t propagate inherit
                (I)  - Permission inherited from parent container

Unlike earlier command-line tools, iCACLS correctly preserves the canonical ordering of ACE entries:

  1. Explicit Deny
  2. Explicit Grant
  3. Inherited Deny
  4. Inherited Grant

Access Control Lists apply only to files stored on an NTFS formatted drive, each ACL determines which users (or groups of users) can read or edit the file. When a new file is created it normally inherits ACL’s from the folder where it was created.

An access control list (ACL) is a list of access control entries (ACE). When backing up or restoring an ACL with iCACLS, you must do so for an entire directory (using /save and /restore) even if you are only interested in the ACEs for a few individual files. In practice most permissions are set at the per-directory level.

Multiple /Grant /Deny /Remove clauses can be included in a single icacls command, on a large directory tree this has the advantage that the tree only has to be traversed once, rather than multiple times if you were to issue several consecutive icacls commands instead.

Modify vs Full control

  • To edit a file you must have the “Modify/Change” ACL (or be the file’s owner)
  • To use the iCACLS command to change the permissions of a file requires “FULL Control” (or be the file’s owner)
  • File “Ownership” will always override all ACL’s – you always have Full Control over files that you create.

Inheritance

Inherited folder permissions are given as:

 OI - Object inherit    - This folder and files. (no inheritance to subfolders)
 CI - Container inherit - This folder and subfolders.
 IO - Inherit only      - The ACE does not apply to the current file/directory

These can also be combined as follows:
 (OI)(CI)      This folder, subfolders, and files.
 (OI)(CI)(IO)  Subfolders and files only.
     (CI)(IO)  Subfolders only.
 (OI)    (IO)  Files only. 

So BUILTIN\Administrators:(OI)(CI)F means that both files and Subdirectories will inherit ‘F’ (Full control)
similarly (CI)R means Directories will inherit ‘R’ (Read folders only = List permission)

It is worth spending some time working out which permissions can be inherited and which need to be applied directly. On large/complex directory structures, minimising the number of ACLs can improve fileserver performance.

If inheritance is combined with /T (traverse subfolders) the change will apply to all folders, not just the top level.
for example:
icacls “C:\demo\example” /inheritance:e /T
Will traverse all subfolders below”C:\demo\example” and enable the inheritance for every one, this will replace any inheritance permissions that have been removed.

If no inheritance is specified, inheritance rules will not be changed but existing inherited permissions will be re-applied to existing objects in the specified location for the specified users/groups.
for example:
icacls “C:\demo\example” /grant administrators:(F) /T
This is similar to applying /reset to the child items of “C:\demo\example” but only resets the administrators group.

Built-In Groups

A command which addresses a built-in group by name like ICACLS foldername /GRANT Everyone:F /T
will only work when the system language is English.

To make this language independent, use an asterisk followed by the well-known SID for the group, see Q243330 for a list.

For example, to grant full control to Everyone on a folder: ICACLS foldername /GRANT *S-1-1-0:F /T

Running icacls under PowerShell

The options for icacls do not always run easily under PowerShell, but they can be made to work by setting a few variables and then executing with Invoke-Expression to expand all the variables:

#set PS variables for each of the icacls options
$Path = "c:\demo"   #The path must be the first thing passed to icacls
$Grant = "/grant:r"
$Remove = "/remove"
$replaceInherit = "/inheritance:r"
$permission = ":(OI)(CI)(F)"
$useraccount1 = "ss64dom\simon"
$useraccount2 = "administrators"
 
#run icacls using invoke Expression
Invoke-Expression -Command ('icacls $Path $replaceInherit $Grant "${useraccount1}${permission }"')

Examples:

Change the NTFS permissions on C:\demo\example\, remove all existing inherited permissions and replace with Full control for the Administrators group and Change/Modify permission for jsmith.
Apply the new permissions to the folder and inherit down to subfolders and files (OI)(CI):

icacls “C:\demo\example” /inheritance:r /grant:r Administrators:(OI)(CI)F
icacls “C:\demo\example” /grant:r Administrators:(OI)(CI)F /T
icacls “C:\demo\example” /grant:r ss64Dom\jsmith:(OI)(CI)M /T

or you can combine grants like:

icacls "C:\demo\example" /grant:r Administrators:(OI)(CI)F /T /grant:r ss64Dom\jsmith:(OI)(CI)M /T

View the permissions currently set on a folder:

icacls “C:\demo\example”

Grant the group FileAdmins ‘Delete’ and ‘Write DAC‘ permissions to C:\demo\example:

icacls “C:\demo\example” /grant:r FileAdmins:(D,WDAC)

Reset permissions on all child items below C:\demo\example\, note the use of \*, without that, the permissions would be reset to those of C:\demo\

icacls “C:\demo\example\*” /c /t /reset

Propagate a new permission to all files and subfolders of C:\demo\example\, without using inheritance:
(so if any of the subfolders contain specific permissions, those won’t be overwritten)

icacls “C:\demo\example” /grant:r accountName:(NP)(RX) /T

Backup the ACLs of every file in the current directory:

icacls * /save Myacl_backup.txt

Restore ACLS using a previously saved acl file:

icacls /restore Myacl_backup.txt

Change the Integrity Level (IL) of a file to High:

icacls MyReport.doc /setintegritylevel H

Remove all inheritance on the ‘Demo’ folder and grant access to the domain user ‘Volta’, in this command the /t will traverse existing subfolders and files, and the (CI) will ensure that new folders/files added in future will inherit these permissions:

icacls C:\demo\example /inheritance:r /grant SS64dom\Volta:(CI)F /t

Grant the user jdoe rights to create, edit and delete files in the folder C:\demo\example\, but prevent deletion of the folder itself:

:: First remove inheritance and grant admins Full control to the top folder
icacls “C:\demo\example” /inheritance:r /grant:r administrators:(OI)(CI)(F)

:: Grant Modify + Delete Child to subfolders and files only
icacls “C:\demo\example” /grant:r ss64Dom\jdoe:(OI)(CI)(IO)(M,DC) /T

:: Grant Read/Execute, Write and Append to the top level folder
icacls “C:\demo\example” /grant:r ss64Dom\jdoe:(RX,WD,AD)

:: if any pre-existing subfolders Grant admins Full control
icacls “C:\demo\example” /grant:r administrators:(OI)(CI)(F) /T

The above does set the correct permissions, but an undesired bug/side effect is that within the top level folder the Windows Explorer right click option to create New files will be empty – only New Folder is shown:

Right Click - New is empty

Linux下进程/程序网络带宽占用情况查看工具 — NetHogs

nethogs命令

有很多适用于Linux系统的开源网络监视工具。比如说,你可以用命令iftop来检查带宽使用情况。netstat用来查看接口统计报告,还有top监控系统当前运行进程。但是如果你想要找一个能够按进程实时统计网络带宽利用率的工具,那么NetHogs值得一看。

NetHogs是一个开源的命令行工具(类似于Linux的top命令),用来按进程或程序实时统计网络带宽使用率。

来自NetHogs项目网站:

NetHogs是一个小型的net top工具,不像大多数工具那样拖慢每个协议或者是每个子网的速度而是按照进程进行带宽分组。NetHogs不需要依赖载入某个特殊的内核模块。如果发生了网络阻塞你可以启动NetHogs立即看到哪个PID造成的这种状况。这样就很容易找出哪个程序跑飞了然后突然占用你的带宽。

本文为你介绍如何在Unix/Linux操作系统下如何安装和使用NetHogs按进程监控网络带宽使用率。

在RHEL、CentOS和Fedora下安装NetHogs

要安装NetHogs,你必须要启用你所用Linux下的EPEL源。然后运行下面的yum命令下载安装NetHogs包。
# yum install nethogs

在Ubuntu、Linux mint和Debian下安装NetHogs

键入apt-get命令安装NetHogs包:
$ sudo apt-get install nethogs

NetHogs用法

在基于RedHat系统下键入如下命令启动NetHogs工具。
# nethogs

在Debian/Ubuntu/Linux Mint下要执行NetHogs你必须拥有root权限:
$ sudo nethogs

nethogs

Ubuntu 12.10 下的NetHogs预览

正如上图所示,send列和received列显示的是按照每个进程的流量统计。总的收发数据带宽在最下方,而且可以用交互命令控制排序,下面将要讨论这些交互命令。

NetHogs 命令行参数

以下就是NetHogs命令行的参数,用-d来添加刷新频率参数,device name 用来检测给定的某个或者某些设备的带宽(默认是eth0)。例如:设置5秒钟的刷新频率,键入如下命令即可:
# nethogs -d 5

$ sudo nethogs -d 5

如果只用来监视设备(eth0)的网络带宽可以使用如下命令:
# nethogs eth0

$ sudo nethogs eth0

如果要同时监视eth0和eth1接口,使用以下命令即可:
# nethogs eth0 eth1

$ sudo nethogs eth0 eth1

其他参数和用法
-d : 刷新间隔
-h : 帮助
-p : promiscious 模式
-t : trace模式
-V : 版本

交互命令

以下是NetHogs的一些交互命令(键盘快捷键)
• m : 修改单位
• r : 按流量排序
• s : 按发送流量排序
• q : 退出命令提示符

关于NetHogs命令行工具的完整参数列表,可以参考NetHogs的手册,使用方法是在终端里输入man nethogs或者sudo man nethogs,更多信息请参考NetHogs项目主页

之前VPS侦探曾经介绍过流量带宽相关的工具如:iftop、vnstat,这几个都是统计和监控网卡流量的。但是当我们的服务器或 VPS的带宽被大量占用或占满,却没找不到称心的工具或程序来查看到底是哪个程序或进程占有率多少带宽。虽然在Windows上查看进程占用带宽情况的软件很多,像某3**、某Q家的电脑管家、IP雷达等。但是Linux下这一类软件很少,今天我们介绍的就是Linux的一款查看进程带宽网络占用的软件:NetHogs。

安装
Debian/Ubuntu下安装很简单,执行:apt-get install nethogs 就可以安装。

CentOS/RHEL下建议先安装上EPEL,再执行:yum install libpcap nethogs 进行安装。

具体使用参数说明:

[root@vpser ~]# nethogs -h
usage: nethogs [-V] [-b] [-d seconds] [-t] [-p] [device [device [device …]]] //nethogs可以使用的参数
-V : prints version.//打印版本信息
-d : delay for update refresh rate in seconds. default is 1. //延迟刷新时间,单位秒,默认1秒
-t : tracemode. //跟踪模式
-b : bughunt mode – implies tracemode. //bughunt模式
-p : sniff in promiscious mode (not recommended). //混合模式下嗅探,不推荐
device : device(s) to monitor. default is eth0 //监听的设备,默认是eth0,也就是网卡设备名称,如果是openvz的vps一般都是venet0,具体可以ifconfig进行查看,lo为本地回环,用不到。多个网卡可以一块写上,空格隔开。

When nethogs is running, press: //nethogs运行是可以使用以下按键进行操作
q: quit //运行时,按 q 键退出
m: switch between total and kb/s mode //按 m 键,切换单位或显示进程占用速度或已统计使用的流量。切换顺序是KB/sec->KB->B->MB
r : Sort by received. //按received进行排序
s : Sort by sent. //按send进行排序

使用例子:nethogs venet0

如上图,PID一列就是进程的PID,PROGRAM就是显示进程或连接双方的端口号,前面红框是服务器的IP:端口,后面红框是客户端的IP:端口,如图根据端口可以判断,目前有7700端口,如果不知道端口对应的进程可以通过lsof来进行查看。

lsof -i:被占用的端口号 如: lsof -i:80

ps aux|grep PID

DEV列显示设备名,SEND是服务器发送的流量,RECEIVED服务器接收的流量。 // 按 m 可以切换统计方式 : 流量速度KB/sec 或 总数{KB /B /m}

How to Install and Configure ‘Cache Only DNS Server’ with ‘Unbound’ in RHEL/CentOS 7

Version:1.0 StartHTML:000000222 EndHTML:000042418 StartFragment:000012983 EndFragment:000042312 StartSelection:000012983 EndSelection:000042308 SourceURL:https://www.tecmint.com/setup-dns-cache-server-in-centos-7/ How to Install and Configure ‘Cache Only DNS Server’ with ‘Unbound’ in RHEL/CentOS 7

Caching name servers using ‘Unbound‘ ( is a validating, recursive, and caching DNS server software ), back in RHEL/CentOS 6.x (where x is version number), we used bind software to configure DNS servers.

Here in this article, we are going to use ‘unbound‘ caching software to install and configure a DNS Server in RHEL/CentOS 7 systems.

Setup Cahing DNS Server in RHEL/CentOS 7

DNS cache servers are used to resolve any DNS query they receive. If the server caches the query and in future the same queries requested by any clients the request will be delivered from DNS ‘unbound‘ cache, this can be done in milliseconds than the first time it resolved.

Caching will only act as a agent to resolve the query of client from any one of the forwarders. Using caching server, will reduce the loading time of webpages by keeping the cache database in unbound server.

My Server and Client Setup

For demonstration purpose, I will be using two systems. The first system will act as a Master (Primary) DNS server and the second system will act as a local DNS client.

Master DNS Server
Operating System   :    CentOS Linux release 7.0.1406 (Core)
IP Address	   :	192.168.0.50
Host-name	   :	ns.tecmintlocal.com
Client Machine
Operating System   :	CentOS 6
IP Address	   :	192.168.0.100
Host-name	   :	client.tecmintlocal.com

Step 1: Check System Hostname and IP

1. Before setting up a caching DNS server, make sure that you’ve added correct hostname and configured correct static IP address for your system, if not set the system static IP address.

2. After, setting correct hostname and static IP address, you can verify them with the help of following commands.

# hostnamectl
# ip addr show | grep inet

Check IP Address

Step 2: Installing and Configuring Unbound

3. Before installing ‘Unbound’ package, we must update the our system to latest version, after that we can install the unbound package.

# yum update -y
# yum install unbound -y

4. After package has been installed, make a copy of the unbound configuration file before making any changes to original file.

# cp /etc/unbound/unbound.conf /etc/unbound/unbound.conf.original

5. Next, use any of your favorite text editor to open and edit ‘unbound.conf‘ configuration file.

# vim /etc/unbound/unbound.conf
Copy Unbound DNS Configuration

Once the file is opened for editing, make the following changes:

Interfaces

Search for Interface and enable the interface which we going to use or if our server have multiple interfaces we have to enable the interface 0.0.0.0.

Here Our server IP was 192.168.0.50, So, i’am going to use unbound in this interface.

Interface 192.168.0.50
Enable IPv4 and Protocol Supports

Search for the following string and make it ‘Yes‘.

do-ip4: yes
do-udp: yes
do-tcp: yes
Enable the logging

To enable the log, add the variable as below, it will log every unbound activities.

logfile: /var/log/unbound
Hide Identity and Version

Enable following parameter to hide id.server and hostname.bind queries.

hide-identity: yes

Enable following parameter to hide version.server and version.bind queries.

hide-version: yes
Access Control

Then search for access-control to allow. This is to allow which clients are allowed to query this unbound server.

Here I have used 0.0.0.0, that means anyone send query to this server. If we need to refuse query for some range of network we can define which network need to be refuse from unbound queries.

access-control: 0.0.0.0/0 allow

Note: Instead of allow, we can replace it with allow_snoop this will enable some additional parameters like dig and it support both recursive and non recursive.

Domain Insecure

Then search for domain-insecure. If our domain is works with DNS sec keys, we need to define our server available for domain-insecure. Here our domain will be treated as insecure.

domain-insecure: "tecmintlocal.com
Forward Zones

Then change the forwarders for our requested query not fulfilled by this server it will forward to root domain (. ) and resolve the query.

forward-zone:
        name: "."
        forward-addr: 8.8.8.8
        forward-addr: 8.8.4.4

Finally, save and quit the configuration file using wq!.

6. After making the above configuration, now verify the unbound.conf file for any errors using the following command.

# unbound-checkconf /etc/unbound/unbound.conf

Check Unbound DNS Configuration

7. After file verification over without any errors, you can safely restart the ‘unbound’ service and enable it at system startup.

# systemctl start unbound.service
# sudo systemctl enable unbound.service

Start Unbound DNS Service

Step 3: Test DNS Cache Locally

8. Now it’s time to check our DNS cache, by doing a ‘drill’ (query) one ‘india.com‘ domain. At first the ‘drill‘ command results for ‘india.com‘ domain will take some milliseconds, and then do a second drill and have a note on Query time it takes for both drills.

drill india.com @192.168.0.50

Check DNS Cache Locally

Did you see in the above output, the first query taken almost 262 msec to resolve and the second query takes 0 msec to resolve domain (india.com).

That means, the first query gets cached in our DNS Cache, so when we run ‘drill’ second time the query served from our local DNS cache, this way we can improve loading speed of websites.

Step 4: Flush Iptables and Add Firewalld Rules

9. We can’t use both iptables and firewalld at same time on same machine, if we do both will conflict with each other, thus removing ipables rules will be a good idea. To remove or flush the iptables, use the following command.

# iptables -F

10. After removing iptables rules permanently, now add the DNS service to firewalld list permanently.

# firewall-cmd --add-service=dns
# firewall-cmd --add-service=dns --permanent

11. After adding DNS service rules, list the rules and confirm.

# firewall-cmd --list-all

Add DNS to Firewalld

Step 5: Managing and Troubleshooting Unbound

12. To get the current server status, use the following command.

# unbound-control status

Check Unbound DNS Status

Dumping DNS Cache

13. If in-case you would like to have a dump of a DNS cache information in a text file, you can redirect it to some file using below command for future use.

 # unbound-control dump_cache > /tmp/DNS_cache.txt

Backup DNS Cache

14. To restore or import the cache from the dumped file, you can use following command.

# unbound-control dump_cache < /tmp/DNS_cache.txt

Restore DNS Cache

Flushing DNS Records

15. To check whether the specific address was resolved by our forwarders in unbound cache Server, use the below command.

# unbound-control lookup google.com

Check DNS Lookup

16. Some times if our DNS cache server will not reply our query, in mean time we can use to flush the cache to remove information such as A, AAA, NS, SO, CNAME, MX, PTR etc.. records from DNS cache. We can remove all information using flush_zone this will remove all informations.

# unbound-control flush www.digitalocean.com
# unbound-control flush_zone tecmintlocal.com

17. To check which forwards are currently used to resolve.

# unbound-control list_forwards

Check Current DNS Forwards

Step 6: Client Side DNS Configuration

18. Here I’ve used a CentOS 6 server as my client machine, IP for this machine is 192.168.0.100 and I’m going to use my unbound DNS server IP (i.e Primary DNS) in it’s interface configuration.

Log-into the Client machine and set the Primary DNS server IP to our unbound server’s IP.

Run the setup command and choose network configuration from TUI network manager.

Then choose DNS configuration, insert the unbound DNS server’s IP as Primary DNS, but here i have used both in Primary and Secondary because I don’t have any other DNS server.

Primary DNS	: 192.168.0.50
Secondary DNS	: 192.168.0.50

Select Network Configuration

Select DNS Configuration

Enter DNS IP Address

Click OK –> Save&Quit –> Quit.

19. After adding Primary and Secondary DNS IP addresses, now it’s time to restart the network using following command.

# /etc/init.d/network restart

Restart Network

20. Now time to access any one of the website from client machine and check for the cache in unbound DNS server.

# elinks aol.com
# dig aol.com

Check Website

Query Website

Conclusion

Earlier we were used to setup DNS cache server using bind package in RHEL and CentOS systems. Now, we have seen how to setup a DNS cache server using unbound package. Hope this will resolve your query request quicker than the bind pacakge.

WebConfig特殊字符的转义

Web.Config默认编码格式为UTF-8,对于XML文件,要用到实体转义码来替换。对应关系如下:

字符

转义码

& 符号 & &amp;
单引号 &apos;
双引号 &quot;
大于 > &gt;
小于 < &lt;
 

App.config:

 

 <?xml version="1.0" encoding="utf-8" ?>
 <configuration>
     <startup> 
         <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
     </startup>
   <connectionStrings>
     <add name="DBconnString" connectionString="Data Source=.;Initial Catalog=MyTest123456;User ID=sa;PassWord=123&456"/>
   </connectionStrings>
 </configuration>

 

由于数据库连接的密码中含有特殊字符”&”,编译时出现如下如下错误信息:

显然编译器不认识”&456″,怎么解决呢,总不能更换密码吧?

事实上App.config是xml文件,在xml文件中特殊字符要进行HTML转义。

HTML中<,>,&等有特殊含义(<,>,用于链接签,&用于转义),不能直接使用。这些符号是不显示在我们最终看到的网页里的,那如果我们希望在网页中显示这些符号,就要用到HTML转义字符串(Escape Sequence)了

 


HTML特殊转义字符列表

 

最常用的字符实体

显示     说明     实体名称     实体编号

 

         空格     &nbsp;     &#160;

<         小于     &lt;      &#60;

>      大于    &gt;      &#62;

&      &符号    &amp;     &#38;

“      双引号    &quot;     &#34;

©      版权    &copy;    &#169;

®    已注册商标    &reg;    &#174;

™    商标(美国)    ™    &#8482;

×    乘号      &times;    &#215;

÷     除号      &divide;    &#247;

 

所以只要把”&”进行转义就可以了,将PassWord改为PassWord=123&amp;456″成功通过编译。

 

Linux 磁盘IO性能测试工具:FIO

1、FIO安装

CentOS下:

$ yum install fio

2、FIO简介

FIO是测试IOPS的非常好的工具,用来对硬件进行压力测试和验证。
项目主页:https://github.com/axboe/fio
FIO输出结果详细解释可参见:https://tobert.github.io/post/2014-04-17-fio-output-explained.html
注:fio用于测试磁盘性能,不是文件系统,测试之前需要先把要测试的磁盘卸载掉,测试完后需格式化一遍再挂载上去。相关命令如下:

# 卸载磁盘
umount /mnt/datadir1
# 格式化磁盘
mkfs.ext4 /dev/sdd
# 挂载磁盘
mount -t ext4 /dev/sdd /mnt/datadir1

# 自动挂载,在里面添加该盘
vim /etc/fstab

3、FIO参数

例如:

$ fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=mytest1

说明:
filename=/dev/sdd 测试文件名称,通常选择需要测试的盘的data目录。
direct=1 测试过程绕过机器自带的buffer。使测试结果更真实。
rw=randwrite 测试模式
=read 顺序读
=write 顺序写
=randwrite 随机写
=randread 随机读
=rw,readwrite 顺序混合读写
=randrw 随机混合读写

bs=4k 单次io的块文件大小为4k
bsrange=512-2048 同上,提定数据块的大小范围
size=5G 本次的测试文件大小为5g,以每次4k的io进行测试。
numjobs=30 本次的测试线程为30个
runtime=1000 测试时间为1000秒,如果不写则一直将5g文件分4k每次写完为止
iodepth 队列深度,只有使用libaio时才有意义。这是一个可以影响IOPS的参数。
ioengine=psync io引擎使用psync方式
=libaio Linux专有的异步IO

rwmixwrite=30 在混合读写的模式下,写占30%
group_reporting 关于显示结果的,汇总每个进程的信息。
lockmem=1G 只使用1g内存进行测试。
zero_buffers 用0初始化系统buffer。
nrfiles=8 每个进程生成文件的数量。

点击查看更多详细参数说明

4、FIO测试示例

fio工具使用特别注意:

  1. 请不要在系统盘上进行 fio 测试,避免损坏系统重要文件
  2. fio测试建议在空闲的、未保存重要数据的硬盘上进行,并在测试完后重新制作文件系统。请不要在业务数据硬盘上测试,避免底层文件系统元数据损坏导致数据损坏
  3. 测试硬盘性能时,推荐直接测试裸盘(如 vdb);测试文件系统性能时,推荐指定具体文件测试(如 /data/file)

4.1 顺序读测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=read -filename=/dev/sdd -name=”BS 4KB read test” -iodepth=16 -runtime=601

4.2 顺序写测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=write -filename=/dev/sdd -name=”BS 4KB write test” -iodepth=16 -runtime=601

4.3 随机读测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randread -filename=/dev/sdd -name=”BS 4KB randread test” -iodepth=16 -runtime=601

4.4 随机写测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -filename=/dev/sdd -name=”BS 4KB randwrite test” -iodepth=16 -runtime=601

4.5 随机读写混合测试

$ fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randrw -rwmixread=70 -filename=/dev/sdd -name=”BS 4KB randrw 70 test” -iodepth=16 -runtime=601

5、FIO输出结果

例如:

BS 4KB randread test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.1.10
Starting 1 thread

BS 4KB randread test: (groupid=0, jobs=1): err= 0: pid=26408: Mon Feb 27 16:43:07 2017
read : io=19908KB, bw=339689B/s, iops=82, runt= 60013msec
# fio做了19908KB的IO,bandwidth速率为339689B/s,总IOPS为82,运行时间为60013毫秒

slat (usec): min=18, max=97, avg=23.60, stdev= 2.79
# slat (submission latency):代表 盘需要多久将IO提交到kernel做处理
# usec 微秒

clat (msec): min=1, max=23, avg=12.03, stdev= 3.79
# clat (completion latency):命令提交到kernel到IO做完之间的时间,不包括submission latency

 lat (msec): min=1, max=23, avg=12.05, stdev= 3.79
# 从IO结构体创建时刻开始,直到紧接着clat完成的时间

clat percentiles (usec):
 |  1.00th=[ 4080],  5.00th=[ 5984], 10.00th=[ 7136], 20.00th=[ 8640],
 | 30.00th=[ 9920], 40.00th=[10944], 50.00th=[11840], 60.00th=[12992],
 | 70.00th=[14016], 80.00th=[15296], 90.00th=[17024], 95.00th=[18560],
 | 99.00th=[20864], 99.50th=[21632], 99.90th=[22400], 99.95th=[22912],
 | 99.99th=[23168]
# Completion latency百分数

bw (KB  /s): min=  276, max=  373, per=100.00%, avg=331.54, stdev=18.83
# 带宽(bandwidth)

lat (msec) : 2=0.02%, 4=0.82%, 10=30.32%, 20=66.75%, 50=2.09%
# 30.32%的request延迟在4~10毫秒,66.75%的request延迟在10~20毫秒,以此类推

cpu : usr=0.10%, sys=0.25%, ctx=5057, majf=0, minf=8
# 用户/系统CPU占用率,进程上下文切换(context switch)次数,主要和次要(major and minor)页面错误数量(page faults)。由于测试是配置成使用直接IO,page faults数量应该极少。

IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
# iodepth设置,用来控制同一时刻发送给OS多少个IO。这完全是纯应用层面的行为,和盘的IO queue不是一回事

 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 #submit和complete代表同一时间段内fio发送上去和已完成的IO数量

 issued    : total=r=4977/w=0/d=0, short=r=0/w=0/d=0
 # 发送的IO数量

 latency   : target=0, window=0, percentile=100.00%, depth=1
 # Fio可以配置一个延迟目标值,这个值可以调节吞吐量直到达到预设的延迟目标

Run status group 0 (all jobs):

Fio支持把不同的测试聚合

READ: io=19908KB, aggrb=331KB/s, minb=331KB/s, maxb=331KB/s, mint=60013msec, maxt=60013msec
# 汇总输出吞吐量和时间。
# io=表示总共完成的IO数量。在基于时间的测试中这是一个变量,在基于容量的测试中,这个值能匹配size参数。
# aggrb是所有进程/设备的汇总带宽。
# minb/maxb表示测量到的最小/最大带宽。
# mint/maxt表示测试的最短和最长耗时。和io=参数类似,时间值对于基于时间的测试应该能匹配runtime参数,对于基于容量的测试是一个变量。

Disk stats (read/write):
sdd: ios=4969/0, merge=0/0, ticks=59703/0, in_queue=59702, util=99.51%

Python日志监控系统处理日志(pyinotify)

前言

最近项目中遇到一个用于监控日志文件的Python包pyinotify,结合自己的项目经验和网上的一些资料总结一下,总的原理是利用pyinotify模块监控日志文件夹,当日志到来的情况下,触发相应的函数进行处理,处理完毕后删除日志文件的过程,下面就着重介绍下pyinotify

pyinotify

Pyinotify是一个Python模块,用来监测文件系统的变化。 Pyinotify依赖于Linux内核的功能—inotify(内核2.6.13合并)。 inotify的是一个事件驱动的通知器,其通知接口通过三个系统调用从内核空间到用户空间。pyinotify结合这些系统调用,并提供一个顶级的抽象和一个通用的方式来处理这些功能。

  • pyinotify 说百了就是通过 调用系统的inotify来实现通知的
  • inotify 既可以监视文件,也可以监视目录
  • Inotify 使用系统调用而非 SIGIO 来通知文件系统事件。

Inotify 可以监视的文件系统事件包括:

Event NameIs an EventDescription
IN_ACCESSYesfile was accessed.
IN_ATTRIBYesmetadata changed.
IN_CLOSE_NOWRITEYesunwrittable file was closed.
IN_CLOSE_WRITEYeswrittable file was closed.
IN_CREATEYesfile/dir was created in watched directory.
IN_DELETEYesfile/dir was deleted in watched directory.
IN_DELETE_SELFYes自删除,即一个可执行文件在执行时删除自己
IN_DONT_FOLLOWNodon’t follow a symlink (lk 2.6.15).
IN_IGNOREDYesraised on watched item removing. Probably useless for you, prefer instead IN_DELETE*.
IN_ISDIRNoevent occurred against directory. It is always piggybacked to an event. The Event structure automatically provide this information (via .is_dir)
IN_MASK_ADDNoto update a mask without overwriting the previous value (lk 2.6.14). Useful when updating a watch.
IN_MODIFYYesfile was modified.
IN_MOVE_SELFYes自移动,即一个可执行文件在执行时移动自己
IN_MOVED_FROMYesfile/dir in a watched dir was moved from X. Can trace the full move of an item when IN_MOVED_TO is available too, in this case if the moved item is itself watched, its path will be updated (see IN_MOVE_SELF).
IN_MOVED_TOYesfile/dir was moved to Y in a watched dir (see IN_MOVE_FROM).
IN_ONLYDIRNoonly watch the path if it is a directory (lk 2.6.15). Usable when calling .add_watch.
IN_OPENYesfile was opened.
IN_Q_OVERFLOWYesevent queued overflowed. This event doesn’t belongs to any particular watch.
IN_UNMOUNTYes宿主文件系统被 umount
IN_ACCESS,即文件被访问
IN_MODIFY,文件被write
IN_ATTRIB,文件属性被修改,如chmod、chown、touch等
IN_CLOSE_WRITE,可写文件被close
IN_CLOSE_NOWRITE,不可写文件被close
IN_OPEN,文件被open
IN_MOVED_FROM,文件被移走,如mv
IN_MOVED_TO,文件被移来,如mv、cp
IN_CREATE,创建新文件
IN_DELETE,文件被删除,如rm
IN_DELETE_SELF,自删除,即一个可执行文件在执行时删除自己
IN_MOVE_SELF,自移动,即一个可执行文件在执行时移动自己
IN_UNMOUNT,宿主文件系统被umount
IN_CLOSE,文件被关闭,等同于(IN_CLOSE_WRITE | IN_CLOSE_NOWRITE)
IN_MOVE,文件被移动,等同于(IN_MOVED_FROM | IN_MOVED_TO)

pyinotify使用例子

#!/usr/bin/python
# coding:utf-8
 
import os
from pyinotify import WatchManager, Notifier,ProcessEvent,IN_DELETE, IN_CREATE,IN_MODIFY
   
class EventHandler(ProcessEvent):
 """事件处理"""
 def process_IN_CREATE(self, event):
  print  "Create file: %s " %  os.path.join(event.path,event.name)
   
 def process_IN_DELETE(self, event):
  print  "Delete file: %s " %  os.path.join(event.path,event.name)
  
 def process_IN_MODIFY(self, event):
   print  "Modify file: %s " %  os.path.join(event.path,event.name)
   
def FSMonitor(path='.'):
  wm = WatchManager() 
  mask = IN_DELETE | IN_CREATE |IN_MODIFY
  notifier = Notifier(wm, EventHandler())
  wm.add_watch(path, mask,auto_add=True,rec=True)
  print 'now starting monitor %s'%(path)
  while True:
   try:
     notifier.process_events()
     if notifier.check_events():
       notifier.read_events()
   except KeyboardInterrupt:
     notifier.stop()
     break
   
if __name__ == "__main__":
 FSMonitor('/root/softpython/apk_url')

How to Install Windows on RAID 5 with Drives Larger than 2TB

Setting Up the PERC with the DISKPART Tool for Windows Install

This type of RAID array requires a partition using a GUID (Globally Unique Identifier) Partition Table (GPT) to work properly for Windows in UEFI BIOS mode. Windows Setup, however, does not support making a GPT partition in the standard setup dialogue. Diskpart from the Command Prompt must be used to create a GPT partition to allow setup to see the RAID partition and continue setup.


Figure 1

  1. Start the install, and load the PERC H310 drivers via the OS “Load Diver” function. After the driver is loaded the RAID volume will appear in the install to device list (Figure 1).
  2. Press Shift+F10 to bring up a Command Prompt window.
  3. Type DISKPART and press Enter to enter the DISKPART tool. Enter the commands in the following steps as shown in bold and press enter.
  4. DETAIL DISK – This shows a list of volumes seen by the system. Make note of the Volume number for the RAID array.
  5. SELECT DISK=X – X will be the Volume number of the RAID shown in the detail disk report.
  6. CLEAN – Clears the partition information.
  7. CONVERT GPT – Sets the partition to GPT.
  8. EXIT – Exits DISKPART.
  9. Exit the Command Prompt window.
  10. The full RAID volume should show in the device list.
  11. Complete the remainder of the installation process normally. The RAID volume should show as “Windows Boot Manager” in UEFI.

If this was completed correctly, the system should boot normally in UEFI BIOS mode, and allow Windows to install.

Configure IIS to listen on specific IPs

By default IIS will listen for connections on port 80 for any IP bound to the server. This happens even if there are no host headers or bindings set for a specific IP. This can be a problem when trying to run multiple web servers on port 80.

To set IIS to listen on specific IPs follow the instructions below.

Windows Server 2003/IIS 6:

1. This requires the Server 2003 support tools. If this is not already installed it can be downloaded here.

2. Once installed open a command prompt and navigate to the support tools installation folder (default is C:\Program Files\Support Tools).
cd C:\Program Files\Support Tools

3. Stop http.
net stop http /y

4. Use this command to display the current list of IPs:
httpcfg query iplisten

5. By default it will listen on all IPs (0.0.0.0) so we can remove this.
httpcfg delete iplisten -i 0.0.0.0

6. Specify the IP(s) that IIS should listen on. Make sure to update 127.0.0.1 to the desired IP and run the command for each IP IIS should listen on.
httpcfg set iplisten -i 127.0.0.1

7. Start http and test out your sites.
net start http

Windows Server 2008/IIS 7:

1. Open a command prompt and type “netsh”.
netsh

2. Type “http”.
http

3. Enter the following command to display the current list of IPs to listen on. Note if no IPs are displayed like in the below image, IIS will listen on all IPs (default).
show iplisten

4. Use the command below to set IIS to listen on a specific IP. Make sure to replace 127.0.0.1 with the correct IP and run the command again for any additional addresses.
add iplisten ipaddress=127.0.0.1

5. In case you need to delete an IP from this list, use the following command.
delete iplisten ipaddress=127.0.0.1

6. Restart IIS to apply these changes.
iisreset

IIS listening 127.0.0.1 instead 0.0.0.0

I use command:

netsh http show iplisten

and saw 127.0.0.1 in listening list. (But I didn’t add it manually). So I delete it and add 0.0.0.0 instead.

netsh http delete iplisten ipaddress=127.0.0.1

netsh http add iplisten ipaddress=0.0.0.0

Then I restarted iis server.

SSL certificates on Sites with Host Headers

Source:https://blogs.iis.net/thomad/ssl-certificates-on-sites-with-host-headers

Today I got the following question:

“I have two sites (siteV1.mysite.com and sitev2.mysite.com). They listen on the same IP address and port. We generated a certificate for siteV1.mysite.com and SSL is working properly. The problem is that some of our customers use siteV2.mysite.com and they are getting certificate errors. What’s the problem?”

Here is the issue:

There are three pieces of data to uniquely identify an IIS site:

  • The IP address
  • The Port
  • The Host name which HTTP 1.1 clients send as an HTTP request header. 

This IP:Port:Hostname triplet is called a binding. The binding “192.168.1.192:80:myserver” for example represents a site that listens on IP address 192.168.1.192, port 80, host-header myserver

The very first things IIS (HTTP.SYS to be more precise) does when a request comes in is to read the site’s configuration. Connection limits and timeouts are examples of site configuration. The site binding is used to find the right site configuration. The SSL certificate seems to be another great example of site configuration – the SSL certificate is needed to decrypt the encrypted SSL data coming from the client.

And the IIS User Interface certainly makes it appear as if the SSL certificate would be site configuration, too – doesn’t it? In reality however you can’t bind a SSL certificate to a site. The IIS UI is fooling you. But why? 

It’s a chicken and egg problem: The host name is encrypted in the SSL blob that the client sends. Because the host name is part of the binding IIS needs the host name to lookup the right certificate. Without the host name IIS can’t lookup the right site because the binding is incomplete. Without the certificate IIS can’t decrypt the SSL blob that contains the host name. Game over – we are turning in circles. 

What IIS does under the covers is to ignore the host name. IIS binds the certificate to IP:Port and warns you when you try to bind a certificate to the same IP:Port combo with different host names. 

But there is a way if you need two different sites on the same IP:Port. You can accomplish this by getting a certificate that contains both common names, i.e. sitev1.mysite.com and sitev2.mysitem.com. Cert Authorities usually allow more than one so called “common names” in a certificate. By binding the certificate to one of the two sites you won’t not get certificate errors anymore. The client is happy if one of the names in the certificate matches. 

But there is another caveat: you can’t use the IIS7 User Interface to add a host header to an SSL site binding. You have to use command-line tools, do it programmatically or edit applicationhost.config directly. Here is an example and a link how you can it via command-line:

appcmd set site /site.name:”MySite V2″ /+bindings.[protocol=’https’,bindingInformation=’*:443:sitev2.mysite.com’]

And last but not least: with IIS7 you can use the following command to figure out what certificate is bound to a particular IP:Port combination:  
netsh http show sslcert

This command will show the IP:Port binding but also some other SSL settings.