分类目录归档:语言 & 开发

语言 & 开发

Python日志logging模块功能与用法详解

本文实例讲述了Python日志logging模块功能与用法。分享给大家供大家参考,具体如下:

本文内容:

  • logging模块的介绍
  • logging模块的基础使用
  • logging模块的扩展使用
  • logging中的Filter
  • 使用配置文件配置logging和logger
  • 小技巧
  • 想要了解更多?不如看看官方文档。
继续阅读

A quick md5sum equivalent in python

This post will show you how to write a function to compute md5 sum of a file using the hashlib module, the with statement and being memory efficient by not reading the whole file in memory.

from __future__ import with_statement
from hashlib import md5

def md5sum(filename, buf_size=8192):
    m = md5()
    # the with statement makes sure the file will be closed
    with open(filename, 'b') as f:
        # We read the file in small chunk until EOF
        data = f.read(buf_size)
        while data:
            # We had data to the md5 hash
            m.update(data)
            data = f.read(buf_size)
    # We return the md5 hash in hexadecimal format
    return m.hexdigest()

if __name__ == '__main__':
    import sys
    print md5sum(sys.argv[1])

Now let’s see how quick it is against the real md5sum using a test file of 10Go!

The real md5sum:

$ time md5sum /data/testfile
b215f7bf5b09fa3e9848a6a66f3f3172  /data/testfile

real    0m31.148s
user    0m27.738s
sys     0m3.408s

The python version of md5sum:

$ time python md5sum.py /data/testfile
b215f7bf5b09fa3e9848a6a66f3f3172

real    0m27.791s
user    0m24.514s
sys     0m3.276s

The python based version is almost 4 seconds quicker than the C based version!

PowerShell提速和多线程

本文目录

继续阅读

WebConfig特殊字符的转义

Web.Config默认编码格式为UTF-8,对于XML文件,要用到实体转义码来替换。对应关系如下:

字符

转义码

& 符号 & &
单引号 '
双引号 "
大于 > >
小于 < &lt;
 

App.config:

 

 <?xml version="1.0" encoding="utf-8" ?>
 <configuration>
     <startup> 
         <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
     </startup>
   <connectionStrings>
     <add name="DBconnString" connectionString="Data Source=.;Initial Catalog=MyTest123456;User ID=sa;PassWord=123&456"/>
   </connectionStrings>
 </configuration>

 

由于数据库连接的密码中含有特殊字符”&”,编译时出现如下如下错误信息:

显然编译器不认识”&456″,怎么解决呢,总不能更换密码吧?

事实上App.config是xml文件,在xml文件中特殊字符要进行HTML转义。

HTML中<,>,&等有特殊含义(<,>,用于链接签,&用于转义),不能直接使用。这些符号是不显示在我们最终看到的网页里的,那如果我们希望在网页中显示这些符号,就要用到HTML转义字符串(Escape Sequence)了

 


HTML特殊转义字符列表

 

最常用的字符实体

显示     说明     实体名称     实体编号

 

         空格     &nbsp;     &#160;

<         小于     &lt;      &#60;

>      大于    &gt;      &#62;

&      &符号    &amp;     &#38;

“      双引号    &quot;     &#34;

©      版权    &copy;    &#169;

®    已注册商标    &reg;    &#174;

™    商标(美国)    ™    &#8482;

×    乘号      &times;    &#215;

÷     除号      &divide;    &#247;

 

所以只要把”&”进行转义就可以了,将PassWord改为PassWord=123&amp;456″成功通过编译。

 

Python日志监控系统处理日志(pyinotify)

前言

最近项目中遇到一个用于监控日志文件的Python包pyinotify,结合自己的项目经验和网上的一些资料总结一下,总的原理是利用pyinotify模块监控日志文件夹,当日志到来的情况下,触发相应的函数进行处理,处理完毕后删除日志文件的过程,下面就着重介绍下pyinotify

pyinotify

Pyinotify是一个Python模块,用来监测文件系统的变化。 Pyinotify依赖于Linux内核的功能—inotify(内核2.6.13合并)。 inotify的是一个事件驱动的通知器,其通知接口通过三个系统调用从内核空间到用户空间。pyinotify结合这些系统调用,并提供一个顶级的抽象和一个通用的方式来处理这些功能。

  • pyinotify 说百了就是通过 调用系统的inotify来实现通知的
  • inotify 既可以监视文件,也可以监视目录
  • Inotify 使用系统调用而非 SIGIO 来通知文件系统事件。

Inotify 可以监视的文件系统事件包括:

Event NameIs an EventDescription
IN_ACCESSYesfile was accessed.
IN_ATTRIBYesmetadata changed.
IN_CLOSE_NOWRITEYesunwrittable file was closed.
IN_CLOSE_WRITEYeswrittable file was closed.
IN_CREATEYesfile/dir was created in watched directory.
IN_DELETEYesfile/dir was deleted in watched directory.
IN_DELETE_SELFYes自删除,即一个可执行文件在执行时删除自己
IN_DONT_FOLLOWNodon’t follow a symlink (lk 2.6.15).
IN_IGNOREDYesraised on watched item removing. Probably useless for you, prefer instead IN_DELETE*.
IN_ISDIRNoevent occurred against directory. It is always piggybacked to an event. The Event structure automatically provide this information (via .is_dir)
IN_MASK_ADDNoto update a mask without overwriting the previous value (lk 2.6.14). Useful when updating a watch.
IN_MODIFYYesfile was modified.
IN_MOVE_SELFYes自移动,即一个可执行文件在执行时移动自己
IN_MOVED_FROMYesfile/dir in a watched dir was moved from X. Can trace the full move of an item when IN_MOVED_TO is available too, in this case if the moved item is itself watched, its path will be updated (see IN_MOVE_SELF).
IN_MOVED_TOYesfile/dir was moved to Y in a watched dir (see IN_MOVE_FROM).
IN_ONLYDIRNoonly watch the path if it is a directory (lk 2.6.15). Usable when calling .add_watch.
IN_OPENYesfile was opened.
IN_Q_OVERFLOWYesevent queued overflowed. This event doesn’t belongs to any particular watch.
IN_UNMOUNTYes宿主文件系统被 umount
IN_ACCESS,即文件被访问
IN_MODIFY,文件被write
IN_ATTRIB,文件属性被修改,如chmod、chown、touch等
IN_CLOSE_WRITE,可写文件被close
IN_CLOSE_NOWRITE,不可写文件被close
IN_OPEN,文件被open
IN_MOVED_FROM,文件被移走,如mv
IN_MOVED_TO,文件被移来,如mv、cp
IN_CREATE,创建新文件
IN_DELETE,文件被删除,如rm
IN_DELETE_SELF,自删除,即一个可执行文件在执行时删除自己
IN_MOVE_SELF,自移动,即一个可执行文件在执行时移动自己
IN_UNMOUNT,宿主文件系统被umount
IN_CLOSE,文件被关闭,等同于(IN_CLOSE_WRITE | IN_CLOSE_NOWRITE)
IN_MOVE,文件被移动,等同于(IN_MOVED_FROM | IN_MOVED_TO)

pyinotify使用例子

#!/usr/bin/python
# coding:utf-8
 
import os
from pyinotify import WatchManager, Notifier,ProcessEvent,IN_DELETE, IN_CREATE,IN_MODIFY
   
class EventHandler(ProcessEvent):
 """事件处理"""
 def process_IN_CREATE(self, event):
  print  "Create file: %s " %  os.path.join(event.path,event.name)
   
 def process_IN_DELETE(self, event):
  print  "Delete file: %s " %  os.path.join(event.path,event.name)
  
 def process_IN_MODIFY(self, event):
   print  "Modify file: %s " %  os.path.join(event.path,event.name)
   
def FSMonitor(path='.'):
  wm = WatchManager() 
  mask = IN_DELETE | IN_CREATE |IN_MODIFY
  notifier = Notifier(wm, EventHandler())
  wm.add_watch(path, mask,auto_add=True,rec=True)
  print 'now starting monitor %s'%(path)
  while True:
   try:
     notifier.process_events()
     if notifier.check_events():
       notifier.read_events()
   except KeyboardInterrupt:
     notifier.stop()
     break
   
if __name__ == "__main__":
 FSMonitor('/root/softpython/apk_url')

用pip命令安装第三方包出现retrying且ssl error问题汇总

今天pip包时一直retrying且报ssl error的错误,我弄了一上午才好,网上有很多解决方案,但是没有pip安装失败的汇总情况,如有同错,请对比以下情况,希望能解决你的问题,也烦请对不足之处指出。

一、

错误:CouldnotfetchURLhttps://pypi.python.org/simple/pytest-xdist/: There wasaproblem confirmingthessl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocolversion(_ssl.c:590)

原因:python.org已经不支持TLSv1.0和TLSv1.1,需要升级pip,但是pip用不了,所以手动升级

解决方案:

1.mac或者linux操作系统:在终端下执行命令:

curl https://bootstrap.pypa.io/get-pip.py | python3。

2.windows操作系统:从https://bootstrap.pypa.io/get-pip.py下载get-pip.py文件,然后使用python运行这个文件python get-pip.py

二、

错误:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None))

after connection broken by ‘SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAI

LED] certificate verify failed (_ssl.c:661)’),)’

原因:可能用国外镜像源连接不好

解决方案:

1.mac或者linux操作系统:

修改 ~/.pip/pip.conf (没有就创建一个), 更换 index-url

[global]

index-url = http://pypi.douban.com/simple

2.windows操作系统:

直接在user目录中创建一个pip目录,如:C:\Users\xx\pip,新建文件pip.ini,内容同上

附:镜像源

阿里云 http://mirrors.aliyun.com/pypi/simple/

中国科技大学 https://pypi.mirrors.ustc.edu.cn/simple/

豆瓣 http://pypi.douban.com/simple

Python官方 https://pypi.python.org/simple/

v2ex http://pypi.v2ex.com/simple/

中国科学院 http://pypi.mirrors.opencas.cn/simple/

清华大学 https://pypi.tuna.tsinghua.edu.cn/simple/

三、

错误:Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /simple/pip/ (Caused by S

SLError(SSLError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)’),)) – skipping

原因:在最新的 pip 版本(>=7)中,使用镜像源时,会提示源地址不受信任或不安全

解决方案1:

pip install –trusted-host pypi.org –trusted-host files.pythonhosted.org 包名

解决方案2:(推荐)

在第二个解决方案中:添加一项配置

[install]
trusted-host=http://pypi.douban.com/simple

理解 Python 装饰器

讲 Python 装饰器前,我想先举个例子,虽有点污,但跟装饰器这个话题很贴切。

每个人都有的内裤主要功能是用来遮羞,但是到了冬天它没法为我们防风御寒,咋办?我们想到的一个办法就是把内裤改造一下,让它变得更厚更长,这样一来,它不仅有遮羞功能,还能提供保暖,不过有个问题,这个内裤被我们改造成了长裤后,虽然还有遮羞功能,但本质上它不再是一条真正的内裤了。于是聪明的人们发明长裤,在不影响内裤的前提下,直接把长裤套在了内裤外面,这样内裤还是内裤,有了长裤后宝宝再也不冷了。装饰器就像我们这里说的长裤,在不影响内裤作用的前提下,给我们的身子提供了保暖的功效。 继续阅读

Makefiles Tutorial

GNU Make is key utility in building applications.It is available on any operating systems. Its main purpose is to determine automatically which pieces of a program need to be recompiled, and issue the commands to build them

While working with open source project, Make is the common tool because almost any IDE can work with it. Make is a very powerful tool – in one command you can issue a series of tasks and working with it can make developers life easier. 继续阅读

Python — imaplib IMAP example with Gmail

I couldn’t find all that much information about IMAP on the web, other than the RFC3501.

The IMAP protocol document is absoutely key to understanding the commands available, but let me skip attempting to explain and just lead by example where I can point out the common gotchas I ran into.

Logging in to the inbox

1
2
3
4
5
6
import imaplib
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login('myusername@gmail.com', 'mypassword')
mail.list()
# Out: list of "folders" aka labels in gmail.
mail.select("inbox") # connect to inbox.

Getting all mail and fetching the latest

Let’s start by searching our inbox for all mail with the search function.
Use the built in keyword “ALL” to get all results (documented in RFC3501).

We’re going to extract the data we need from the response, then fetch the mail via the ID we just received.

1
2
3
4
5
6
7
8
9
10
result, data = mail.search(None, "ALL")
ids = data[0] # data is a list.
id_list = ids.split() # ids is a space separated string
latest_email_id = id_list[-1] # get the latest
result, data = mail.fetch(latest_email_id, "(RFC822)") # fetch the email body (RFC822) for the given ID
raw_email = data[0][1] # here's the body, which is raw text of the whole email
# including headers and alternate payloads

Using UIDs instead of volatile sequential ids

继续阅读

How to:Python

1、django.core.exceptions.ImproperlyConfigured: Requested setting DEBUG, but settings are not configured.

You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

>>> from django.conf import settings
>>> settings.configure()
>>> from django.test.utils import setup_test_environment
>>> setup_test_environment()