python 进程内存增长问题, 解决方法和工具，python增长,转载：http://

转载：http://drmingdrmer.github.io/tech/programming/2017/05/06/python-mem.html#pyrasite-%E8%BF%9E%E6%8E%A5%E8%BF%9B%E5%85%A5python%E7%A8%8B%E5%BA%8F

表现

运行环境:

# uname -aLinux ** 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux# python2 --versionPython 2.7.5# cat /etc/*-releaseCentOS Linux release 7.2.1511 (Core)

python程序在长时间(较大负载)运行一段时间后, python 进程的系统占用内存持续升高:

# ps aux | grep python2USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDroot     124910 10.2  0.8 5232084 290952 ?      Sl   Mar17 220:37 python2 offline.py restart#                                 ~~~~~~#                                 290M 内存占用

这里的python进程在经历大量请求处理过程中, 内存持续升高, 但最终负载压力下降之后, 内存个并没有下降.

解决方法

为了节省读者时间, 这里先给出结论, 后面再记录详细的排查步骤.

我们分几个步骤逐步定位到问题所在:

而最后的解决方法也很简单, 直接替换malloc模块为tcmalloc:

LD_PRELOAD="/usr/lib64/libtcmalloc.so" python x.py

定位问题过程

gdb-python: 搞清楚python程序在做什么

首先要确定python在做什么, 是不是有正常的大内存消耗任务在运行, 死锁等异常行为.

这方面可以用gdb来帮忙, 从gdb-7开始, gdb支持用python来实现gdb的扩展. 我们可以像调试c程序那样, 用gdb对python程序检查线程, 调用栈等.

而且可以将python代码和内部的c代码的调用栈同时打印出来.

这样对不确定是python代码问题还是其底层c代码的问题的时候, 很有帮助.

以下步骤的详细信息可以参考debug-with-gdb.

准备gdb

首先安装python的debuginfo:

# debuginfo-install python-2.7.5-39.el7_2.x86_64

如果缺少debuginfo, 运行后面的步骤gdb会提示blabla, 按照提示安装完继续就好:

Missing separate debuginfos, use: debuginfo-install python-2.7.5-39.el7_2.x86_64

接入gdb

然后我们可以直接用gdb attach到1个python进程, 来查看它的运行状态:

# gdb python 11122

attach 之后进入了gdb, 能做的事情就多了. 几个基本的检查步骤:

查看线程

(gdb) info threads  Id   Target Id         Frame  206  Thread 0x7febdbfe3700 (LWP 124916) "python2" 0x00007febe9b75413 in select () at ../sysdeps/unix/syscall-template.S:81  205  Thread 0x7febdb7e2700 (LWP 124917) "python2" 0x00007febe9b75413 in select () at ../sysdeps/unix/syscall-template.S:81  204  Thread 0x7febdafe1700 (LWP 124918) "python2" 0x00007febe9b75413 in select () at ../sysdeps/unix/syscall-template.S:81  203  Thread 0x7febda7e0700 (LWP 124919) "python2" 0x00007febe9b7369d in poll () at ../sysdeps/unix/syscall-template.S:81

一般加锁死锁差不多可以在这里看到, 会有线程卡在xx_wait之类的函数上.

之前用这个方法定位了1个python-logging模块引起的, 在多线程的进程中运行fork, 导致logging的锁被锁住后fork到新的进程, 但解锁线程没有fork到新进程而造成的死锁问题.

查看调用栈

如果发现某个线程有问题, 切换到那个线程上, 查看调用栈确定具体的执行步骤, 使用bt命令:

(gdb) bt#16 0x00007febea8500bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, [email protected]=0x0, args=<optimized out>,    argcount[email protected]=1, kws=0x38aa668, kwcount=2, defs=0x3282a88, defcount=2, closure[email protected]=0x0)    at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330...#19 PyEval_EvalFrameEx (    f[email protected]=Frame 0x38aa4d0, for file t.py, line 647, in run (part_num=2, consumer=<...

bt命令不仅可以看到c的调用栈, 还会显示出python源码的调用栈, 想上面frame-16是c的, frame-19显示出在python的源代码对应哪1行.

如果只查看python的代码的调用栈, 使用py-bt命令:

(gdb) py-bt#1 <built-in method poll of select.epoll object at remote 0x7febeacc5930>#3 Frame 0x3952450, for file /usr/lib64/python2.7/site-packages/twisted/internet/epollreactor.py, line 379, in doPoll (self=<...    l = self._poller.poll(timeout, len(self._selectables))#7 Frame 0x39502a0, for file /usr/lib64/python2.7/site-packages/twisted/internet/base.py, line 1204, in mainLoop (self=<...

py-bt显示出python源码的调用栈, 调用参数, 以及所在行的代码.

coredump

如果要进行比较长时间的跟踪, 最好将python程序的进程信息全部coredump出来, 之后对core文件进行分析, 避免影响正在运行的程序.

(gdb) generate-core-file

这条命令将当前gdb attach的程序dump到它的运行目录, 名字为core.<pid>, 然后再用gdb 加载这个core文件, 进行打印堆栈, 查看变量等分析, 无需attach到正在运行的程序:

# gdb python core.<pid>

其他命令

其他命令可以在gdb输入py<TAB><TAB>看到, 和gdb的命令对应, 例如:

(gdb) pypy-bt               py-list             py-print            pythonpy-down             py-locals           py-up               python-interactive

(gdb) help py-printLook up the given python variable name, and print it

# pip install pyrasite...# pip show pyrasiteName: pyrasiteVersion: 2.0Summary: Inject code into a running Python processHome-page: http://pyrasite.comAuthor: Luke Macken...

pyrasite-shell <pid>>>>

pip install psutil

pyrasite-shell 11122>>> import psutil, os>>> psutil.Process(os.getpid()).memory_info().rss29095232

# pip install guppyfrom guppy import hpyh = hpy()h.heap()# Partition of a set of 48477 objects. Total size = 3265516 bytes.#  Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)#      0  25773  53  1612820  49   1612820  49 str#      1  11699  24   483960  15   2096780  64 tuple#      2    174   0   241584   7   2338364  72 dict of module#      3   3478   7   222592   7   2560956  78 types.CodeType#      4   3296   7   184576   6   2745532  84 function#      5    401   1   175112   5   2920644  89 dict of class#      6    108   0    81888   3   3002532  92 dict (no owner)#      7    114   0    79632   2   3082164  94 dict of type#      8    117   0    51336   2   3133500  96 type#      9    667   1    24012   1   3157512  97 __builtin__.wrapper_descriptor# <76 more rows. Type e.g. ‘_.more‘ to view.>h.iso(1,[],{})# Partition of a set of 3 objects. Total size = 176 bytes.#  Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)#      0      1  33      136  77       136  77 dict (no owner)#      1      1  33       28  16       164  93 list#      2      1  33       12   7       176 100 int

pyrasite-shell 11122>>> import gc>>> gc.collect() # first run gc, find out uncollectable object and put them in gc.garbage                 # output number of object collected>>> gc.garbage   # print all uncollectable objects[]               # empty

from __future__ import print_functionimport gc‘‘‘This snippet shows how to create a uncollectible object:It is an object in a cycle reference chain, in which there is an objectwith __del__ defined.The simpliest is an object that refers to itself and with a __del__ defined.    > python uncollectible.py    ======= collectible object =======    *** init,     nr of referrers: 4                  garbage:         []                  created:         collectible: <__main__.One object at 0x102c01090>                  nr of referrers: 5                  delete:    *** __del__ called    *** after gc, nr of referrers: 4                  garbage:         []    ======= uncollectible object =======    *** init,     nr of referrers: 4                  garbage:         []                  created:         uncollectible: <__main__.One object at 0x102c01110>                  nr of referrers: 5                  delete:    *** after gc, nr of referrers: 5                  garbage:         [<__main__.One object at 0x102c01110>]‘‘‘def dd(*msg):    for m in msg:        print(m, end=‘‘)    print()class One(object):    def __init__(self, collectible):        if collectible:            self.typ = ‘collectible‘        else:            self.typ = ‘uncollectible‘            # Make a reference to it self, to form a reference cycle.            # A reference cycle with __del__, makes it uncollectible.            self.me = self    def __del__(self):        dd(‘*** __del__ called‘)def test_it(collectible):    dd()    dd(‘======= ‘, (‘collectible‘ if collectible else ‘uncollectible‘), ‘ object =======‘)    dd()    gc.collect()    dd(‘*** init,     nr of referrers: ‘, len(gc.get_referrers(One)))    dd(‘              garbage:         ‘, gc.garbage)    one = One(collectible)    dd(‘              created:         ‘, one.typ, ‘: ‘, one)    dd(‘              nr of referrers: ‘, len(gc.get_referrers(One)))    dd(‘              delete:‘)    del one    gc.collect()    dd(‘*** after gc, nr of referrers: ‘, len(gc.get_referrers(One)))    dd(‘              garbage:         ‘, gc.garbage)if __name__ == "__main__":    test_it(collectible=True)    test_it(collectible=False)

# pip install objgraphpyrasite-shell 11122>>> import objgraph>>> objgraph.show_refs([an_object], filename=‘sample-graph.png‘)