python grpc 应用,,1、rpc介绍2、g


1、rpc介绍

2、grpc

3、基于grpc协议文件传输

4、基于grpc协议jmeter压测获取实时结果

5、基于grcp协议获取jmeter最终压测报告,并将报告保存至client端

6、压测中途停止jmeter

grpcserver(jmeterserver)192.168.18.128rpcclient(本机)

1、rpc协议介绍

RPC(Remote Procedure Call Protocol)——远程过程调用协议,它是一种通过网络从远程计算机程序上请求服务,而不需要了解底层网络技术的协议。RPC协议假定某些传输协议的存在,如TCP或UDP,为通信程序之间携带信息数据。在OSI网络通信模型中,RPC跨越了传输层和应用层。RPC使得开发包括网络分布式多程序在内的应用程序更加容易。

RPC采用客户机/服务器模式。请求程序就是一个客户机,而服务提供程序就是一个服务器。首先,客户机调用进程发送一个有进程参数的调用信息到服务进程,然后等待应答信息。在服务器端,进程保持睡眠状态直到调用信息到达为止。当一个调用信息到达,服务器获得进程参数,计算结果,发送答复信息,然后等待下一个调用信息,最后,客户端调用进程接收答复信息,获得进程结果,然后调用执行继续进行。 -- 摘自 百度百科

2、grpc介绍

gRPC 是一款高性能、开源的 RPC 框架,产自 Google,基于 ProtoBuf 序列化协议进行开发,支持多种语言(Golang、Python、Java等),本篇只介绍 Python 的 gRPC 使用。因为 gRPC 对 HTTP/2 协议的支持使其在 Android、IOS 等客户端后端服务的开发领域具有良好的前景。gRPC 提供了一种简单的方法来定义服务,同时客户端可以充分利用 HTTP2 stream 的特性,从而有助于节省带宽、降低 TCP 的连接次数、节省CPU的使用等。

3、基于grpc协议文件传输

目标:将本地的文件(test.file)传输至rpc server (192.168.18.128)

python 版本 2.7.13

3.1、

gRPC 的安装:

[[email protected]rpc]#pip2.7installrequests[[email protected]rpc]#pip2.7installgrpcio

安装 ProtoBuf 相关的 python 依赖库:

[[email protected]rpc]#pip2.7installprotobuf

安装 python grpc 的 protobuf 编译工具:

[[email protected]rpc]#pip2.7installgrpcio-tools

定义 gRPC 接口:

[[email protected]~]#mkdirrpcrpc/__init__.py[[email protected]~]#cdrpc/[[email protected]rpc]#vimrpc.proto//grpc版本syntax="proto3";packagerpc;//定义接口serviceRPC{//相当于定义接口方法rpcsendConfFile(Content)returns(Status){}}//相当于定义类属性,此属性用于接受文本messageContent{stringtext=1;}//此属性用于return结果状态吗messageStatus{int64code=1;}

编译 protobuf:

[[email protected]rpc]#python-mgrpc_tools.protoc-I.--python_out=.--grpc_python_out=../rpc.proto

server 端代码:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportgrpcimportrpc_pb2importrpc_pb2_grpc_ONE_DAY_IN_SECONDS=60*60*24#server端文件保存的位置jmeter_config=os.path.join(os.getcwd(),r‘conf/config.jmx‘)classPerformance(rpc_pb2_grpc.RPCServicer):defsendConfFile(self,content,context):‘‘‘保存配置文件,如config.jmx‘‘‘text=content.texttry:printjmeter_configconf_handle=codecs.open(jmeter_config,‘w‘,encoding=‘utf-8‘)conf_handle.write(text)returnrpc_pb2.Status(code=0)exceptException,e:printereturnrpc_pb2.Status(code=1)defserve():server=grpc.server(futures.ThreadPoolExecutor(max_workers=10))rpc_pb2_grpc.add_RPCServicer_to_server(Performance(),server)server.add_insecure_port(‘[::]:50051‘)server.start()try:whileTrue:time.sleep(_ONE_DAY_IN_SECONDS)exceptKeyboardInterrupt:server.stop(0)if__name__==‘__main__‘:serve()

client 端代码:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportloggingimportjsonimportgrpcimportrpc_pb2importrpc_pb2_grpclogging.basicConfig(level=logging.DEBUG,format=‘%(asctime)s%(filename)s[line:%(lineno)d]%(levelname)s%(message)s‘,datefmt=‘%a,%d%b%Y%H:%M:%S‘,filename=‘rpc.log‘,filemode=‘a+‘)rpc_server=r‘192.168.18.128‘rpc_port=‘50051‘classperformance():‘‘‘性能测试的客户端接口‘‘‘def__init__(self,ip,port):‘‘‘初始化,连接RPC服务‘‘‘logging.info("performance_clientinit")conn=grpc.insecure_channel(ip+‘:‘+port)self.stub_client=rpc_pb2_grpc.RPCStub(channel=conn)defsendConfig(self,filename):file_handle=codecs.open(filename,‘r‘,encoding=‘utf-8‘)content=file_handle.read()‘‘‘向RPCserver发送测试的配置文件‘‘‘response=self.stub_client.sendConfFile(rpc_pb2.Content(text=content))printresponse.codeif__name__==‘__main__‘:client=performance(rpc_server,rpc_port)##将本地的rpc_client.py文件当做测试文件发送到server端client.sendConfig(‘rpc_client.py‘)

启动server端:

[[email protected]rpc]#pythonrpc_server.py

启动client端:

D:\xisuo\rpc>pythonrpc_client.py0

查看server端的文件:

[[email protected]rpc]#moreconf/config.jmx#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportlogging。。。略

4、基于grpc协议jmeter压测获取实时结果

4.1、server端部署jdk,jmeter

将jdk解压到/usr/local/ 配置环境变量

将jmeter解压到/usr/local

4.2、因为我们是有jmeter的no gui模式在Linux执行,故需要jmeter的jmx文件,我们可以在本地使用gui模式先生成jmx文件

最终结果文件为 test.jmx,内容如下:

<?xmlversion="1.0"encoding="UTF-8"?><jmeterTestPlanversion="1.2"properties="2.4"jmeter="2.9r1437961"><hashTree><TestPlanguiclass="TestPlanGui"testclass="TestPlan"testname="测试计划"enabled="true"><stringPropname="TestPlan.comments"></stringProp><boolPropname="TestPlan.functional_mode">false</boolProp><boolPropname="TestPlan.serialize_threadgroups">false</boolProp><elementPropname="TestPlan.user_defined_variables"elementType="Arguments"guiclass="ArgumentsPanel"testclass="Arguments"testname="用户定义的变量"enabled="true"><collectionPropname="Arguments.arguments"/></elementProp><stringPropname="TestPlan.user_define_classpath"></stringProp></TestPlan><hashTree><ThreadGroupguiclass="ThreadGroupGui"testclass="ThreadGroup"testname="测试"enabled="true"><stringPropname="TestPlan.comments">测试</stringProp><stringPropname="ThreadGroup.on_sample_error">continue</stringProp><elementPropname="ThreadGroup.main_controller"elementType="LoopController"guiclass="LoopControlPanel"testclass="LoopController"testname="循环控制器"enabled="true"><boolPropname="LoopController.continue_forever">false</boolProp><stringPropname="LoopController.loops">50</stringProp></elementProp><stringPropname="ThreadGroup.num_threads">2</stringProp><stringPropname="ThreadGroup.ramp_time">1</stringProp><longPropname="ThreadGroup.start_time">1496464278000</longProp><longPropname="ThreadGroup.end_time">1496464278000</longProp><boolPropname="ThreadGroup.scheduler">false</boolProp><stringPropname="ThreadGroup.duration"></stringProp><stringPropname="ThreadGroup.delay"></stringProp></ThreadGroup><hashTree><HTTPSamplerProxyguiclass="HttpTestSampleGui"testclass="HTTPSamplerProxy"testname="HTTP请求"enabled="true"><elementPropname="HTTPsampler.Arguments"elementType="Arguments"guiclass="HTTPArgumentsPanel"testclass="Arguments"testname="用户定义的变量"enabled="true"><collectionPropname="Arguments.arguments"/></elementProp><stringPropname="HTTPSampler.domain">lansgg.blog.51cto.com</stringProp><stringPropname="HTTPSampler.port">80</stringProp><stringPropname="HTTPSampler.connect_timeout"></stringProp><stringPropname="HTTPSampler.response_timeout"></stringProp><stringPropname="HTTPSampler.protocol"></stringProp><stringPropname="HTTPSampler.contentEncoding"></stringProp><stringPropname="HTTPSampler.path"></stringProp><stringPropname="HTTPSampler.method">GET</stringProp><boolPropname="HTTPSampler.follow_redirects">true</boolProp><boolPropname="HTTPSampler.auto_redirects">false</boolProp><boolPropname="HTTPSampler.use_keepalive">true</boolProp><boolPropname="HTTPSampler.DO_MULTIPART_POST">false</boolProp><stringPropname="HTTPSampler.implementation">HttpClient4</stringProp><boolPropname="HTTPSampler.monitor">false</boolProp><stringPropname="HTTPSampler.embedded_url_re"></stringProp><stringPropname="TestPlan.comments">HTTP请求</stringProp></HTTPSamplerProxy><hashTree/><ResultCollectorguiclass="ViewResultsFullVisualizer"testclass="ResultCollector"testname="察看结果树"enabled="true"><boolPropname="ResultCollector.error_logging">false</boolProp><objProp><name>saveConfig</name><valueclass="SampleSaveConfiguration"><time>true</time><latency>true</latency><timestamp>true</timestamp><success>true</success><label>true</label><code>true</code><message>true</message><threadName>true</threadName><dataType>true</dataType><encoding>false</encoding><assertions>true</assertions><subresults>true</subresults><responseData>false</responseData><samplerData>false</samplerData><xml>false</xml><fieldNames>false</fieldNames><responseHeaders>false</responseHeaders><requestHeaders>false</requestHeaders><responseDataOnError>false</responseDataOnError><saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage><assertionsResultsToSave>0</assertionsResultsToSave><bytes>true</bytes></value></objProp><stringPropname="TestPlan.comments">察看结果树</stringProp><stringPropname="filename"></stringProp></ResultCollector><hashTree/></hashTree></hashTree></hashTree></jmeterTestPlan>

当我们在linux终端上执行时,结果如下,

[[email protected]rpc]#/usr/local/apache-jmeter-3.2/bin/jmeter-n-ttest.jmx-ltext.jtlCreatingsummariser<summary>Createdthetreesuccessfullyusingtest.jmxStartingthetest@SatJun0320:41:25CST2017(1496493685813)WaitingforpossibleShutdown/StopTestNow/Heapdumpmessageonport4445summary+46in00:00:18=2.5/sAvg:2745Min:423Max:3135Err:0(0.00%)Active:49Started:50Finished:1summary+54in00:00:08=7.2/sAvg:2018Min:413Max:8000Err:0(0.00%)Active:0Started:50Finished:50summary=100in00:00:26=3.9/sAvg:2353Min:413Max:8000Err:0(0.00%)Tidyingup...@SatJun0320:42:07CST2017(1496493727553)...endofrun

而我们要将这些结果实时的在rpc client展示出来

接口文件:

syntax="proto3";packagerpc;serviceRPC{//sendconfigjmxrpcsendConfFile(Content)returns(Status){}//runjmetertestrpcrunJMeter(Content)returns(streamContent){}}messageContent{stringtext=1;}messageStatus{int64code=1;}

server端:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportjsonimportloggingimportgrpcimportrpc_pb2importrpc_pb2_grpclogging.basicConfig(level=logging.DEBUG,format=‘%(asctime)s%(filename)s[line:%(lineno)d]%(levelname)s%(message)s‘,datefmt=‘%a,%d%b%Y%H:%M:%S‘,filename=‘rpc_server.log‘,filemode=‘a+‘)_ONE_DAY_IN_SECONDS=60*60*24jmeter_config=os.path.join(os.getcwd(),r‘conf/config.jmx‘)jmeter_path=r‘/usr/local/apache-jmeter-3.2/bin/jmeter‘jmeter_result=os.path.join(os.getcwd(),r‘result/result.jtl‘)classPerformance(rpc_pb2_grpc.RPCServicer):defsendConfFile(self,content,context):‘‘‘保存配置文件,如config.jmx‘‘‘text=content.texttry:conf_handle=codecs.open(jmeter_config,‘w‘,encoding=‘utf-8‘)conf_handle.write(text)logging.info("sendConfFileSuccess!")returnrpc_pb2.Status(code=0)exceptException,e:printereturnrpc_pb2.Status(code=1)defrunJMeter(self,content,context):logging.info("beginrunJmeter.")‘‘‘执行测试任务,并将实时结果返回‘‘‘iplist=content.textifiplist:cmd=jmeter_path+"-n-t"+jmeter_config+"-l"+jmeter_result+"-R"+iplistelse:cmd=jmeter_path+"-n-t"+jmeter_config+"-l"+jmeter_resultlogging.info(cmd)popen=subprocess.Popen(cmd,stdout=subprocess.PIPE,universal_newlines=True,shell=True)forstdout_lineiniter(popen.stdout.readline,""):log_line=rpc_pb2.Content(text=stdout_line)yieldlog_linepopen.stdout.close()return_code=popen.wait()ifreturn_code:logging.warn(return_code)raisesubprocess.CalledProcessError(return_code,cmd)defserve():server=grpc.server(futures.ThreadPoolExecutor(max_workers=10))rpc_pb2_grpc.add_RPCServicer_to_server(Performance(),server)server.add_insecure_port(‘[::]:50051‘)server.start()try:whileTrue:time.sleep(_ONE_DAY_IN_SECONDS)exceptKeyboardInterrupt:server.stop(0)if__name__==‘__main__‘:serve()

client端:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportloggingimportjsonimportgrpcimportrpc_pb2importrpc_pb2_grpclogging.basicConfig(level=logging.DEBUG,format=‘%(asctime)s%(filename)s[line:%(lineno)d]%(levelname)s%(message)s‘,datefmt=‘%a,%d%b%Y%H:%M:%S‘,filename=‘rpc.log‘,filemode=‘a+‘)rpc_server=r‘192.168.18.128‘rpc_port=‘50051‘jmeter_config=os.path.join(os.getcwd(),r‘conf/config.jmx‘)jmeter_path=r‘/usr/local/apache-jmeter-3.2/bin/jmeter‘jmeter_result=os.path.join(os.getcwd(),r‘conf/result.jtl‘)classperformance():‘‘‘性能测试的客户端接口‘‘‘def__init__(self,ip,port):‘‘‘初始化,连接RPC服务‘‘‘logging.info("performance_clientinit")conn=grpc.insecure_channel(ip+‘:‘+port)self.stub_client=rpc_pb2_grpc.RPCStub(channel=conn)defsendConfig(self,filename):file_handle=codecs.open(filename,‘r‘,encoding=‘utf-8‘)content=file_handle.read()‘‘‘向RPCserver发送测试的配置文件‘‘‘response=self.stub_client.sendConfFile(rpc_pb2.Content(text=content))returnresponse.codedefrunJmeter(self,iplist):‘‘‘运行测试返回一个生成器,内容为测试过程中的实时输出‘‘‘content=iplistlogging.info("iplist%s"%content)forloginself.stub_client.runJMeter(rpc_pb2.Content(text=content)):yieldlogif__name__==‘__main__‘:client=performance(rpc_server,rpc_port)code=client.sendConfig(‘test.jmx‘)iplist=r‘10.1.1.1,10.1.1.2‘forreal_time_resultsinclient.runJmeter(None):print"getrealtimelogfromserver:%s"%real_time_results.text

结果:

技术分享

5、基于grcp协议获取jmeter最终压测报告,并将报告保存至client端

6、压测中途停止jmeter

接口文件

syntax="proto3";packagerpc;serviceRPC{//sendconfigjmxrpcsendConfFile(Content)returns(Status){}//runjmetertestrpcrunJMeter(Content)returns(streamContent){}//generateResultrpcgenerateResult(empty)returns(streamContent){}//getResultrpcgetResult(empty)returns(streamContent){}//getResultrpcstopJMeter(empty)returns(streamContent){}}messageContent{stringtext=1;}messageStatus{int64code=1;}messageempty{}

server端:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportjsonimportloggingimportgrpcimportrpc_pb2importrpc_pb2_grpclogging.basicConfig(level=logging.DEBUG,format=‘%(asctime)s%(filename)s[line:%(lineno)d]%(levelname)s%(message)s‘,datefmt=‘%a,%d%b%Y%H:%M:%S‘,filename=‘rpc_server.log‘,filemode=‘a+‘)_ONE_DAY_IN_SECONDS=60*60*24jmeter_config=os.path.join(os.getcwd(),r‘conf/config.jmx‘)jmeter_path=r‘/usr/local/apache-jmeter-3.2/bin/jmeter‘jmeter_result_file=os.path.join(os.getcwd(),r‘result/result.jtl‘)jmeter_result_dir=os.path.join(os.getcwd(),r‘result/summary/‘)classPerformance(rpc_pb2_grpc.RPCServicer):defsendConfFile(self,content,context):‘‘‘保存配置文件,如config.jmx‘‘‘text=content.texttry:conf_handle=codecs.open(jmeter_config,‘w‘,encoding=‘utf-8‘)conf_handle.write(text)logging.info("sendConfFileSuccess!")returnrpc_pb2.Status(code=0)exceptException,e:printereturnrpc_pb2.Status(code=1)defrunJMeter(self,content,context):logging.info("beginrunJmeter.")‘‘‘执行测试任务,并将实时结果返回‘‘‘iplist=content.textifiplist:cmd=jmeter_path+"-n-t"+jmeter_config+"-l"+jmeter_result_file+"-R"+iplistelse:cmd=jmeter_path+"-n-t"+jmeter_config+"-l"+jmeter_result_filelogging.info(cmd)popen=subprocess.Popen(cmd,stdout=subprocess.PIPE,universal_newlines=True,shell=True)forstdout_lineiniter(popen.stdout.readline,""):log_line=rpc_pb2.Content(text=stdout_line)yieldlog_linepopen.stdout.close()return_code=popen.wait()ifreturn_code:logging.warn(return_code)raisesubprocess.CalledProcessError(return_code,cmd)defgenerateResult(self,empty,context):‘‘‘调用jmeter,生成汇总测试结果‘‘‘cmd=jmeter_path+"-g"+jmeter_result_file+"-o"+jmeter_result_dirlogging.info(cmd)popen=subprocess.Popen(cmd,stdout=subprocess.PIPE,shell=True)return_code=popen.wait()returnrpc_pb2.Status(code=return_code)defgetResult(self,empty,content):‘‘‘获取汇总测试结果,返回给客户端‘‘‘summary_file=os.path.join(jmeter_result_dir,r"content/js/dashboard.js")file_handle=codecs.open(summary_file,‘r‘,encoding=‘utf-8‘)forlineinfile_handle.readlines():yieldrpc_pb2.Content(text=line)file_handle.close()defstopJMeter(self,empty,content):‘‘‘杀死正在执行的任务‘‘‘os.system("ps-ef|grepjmeter|grep-vgrep|awk‘{print$2}‘|xargskill-9")returnrpc_pb2.Status(code=1)defserve():server=grpc.server(futures.ThreadPoolExecutor(max_workers=10))rpc_pb2_grpc.add_RPCServicer_to_server(Performance(),server)server.add_insecure_port(‘[::]:50051‘)server.start()try:whileTrue:time.sleep(_ONE_DAY_IN_SECONDS)exceptKeyboardInterrupt:server.stop(0)if__name__==‘__main__‘:serve()

client端:

#-*-encoding=utf-8-*-fromconcurrentimportfuturesimporttimeimportsubprocessimportcodecsimportsysimportosimportloggingimportjsonimportgrpcimportrpc_pb2importrpc_pb2_grpclogging.basicConfig(level=logging.DEBUG,format=‘%(asctime)s%(filename)s[line:%(lineno)d]%(levelname)s%(message)s‘,datefmt=‘%a,%d%b%Y%H:%M:%S‘,filename=‘rpc.log‘,filemode=‘a+‘)rpc_server=r‘192.168.18.128‘rpc_port=‘50051‘jmeter_config=os.path.join(os.getcwd(),r‘conf/config.jmx‘)jmeter_path=r‘/usr/local/apache-jmeter-3.2/bin/jmeter‘jmeter_result=os.path.join(os.getcwd(),r‘conf/result.jtl‘)classperformance():‘‘‘性能测试的客户端接口‘‘‘def__init__(self,ip,port):‘‘‘初始化,连接RPC服务‘‘‘logging.info("performance_clientinit")conn=grpc.insecure_channel(ip+‘:‘+port)self.stub_client=rpc_pb2_grpc.RPCStub(channel=conn)defsendConfig(self,filename):file_handle=codecs.open(filename,‘r‘,encoding=‘utf-8‘)content=file_handle.read()‘‘‘向RPCserver发送测试的配置文件‘‘‘response=self.stub_client.sendConfFile(rpc_pb2.Content(text=content))returnresponse.codedefrunJmeter(self,iplist):‘‘‘运行测试返回一个生成器,内容为测试过程中的实时输出‘‘‘content=iplistlogging.info("iplist%s"%content)forloginself.stub_client.runJMeter(rpc_pb2.Content(text=content)):yieldlogdefgenerateResult(self):‘‘‘在rpcserver端生成jmeter最终报告‘‘‘response=self.stub_client.generateResult(rpc_pb2.empty())returnresponse.codedefgetResult(self,local_file):‘‘‘从RPCserver回传测试结果,保存到本地文件local_file‘‘‘file_handle=codecs.open(local_file,‘w‘,encoding=‘utf-8‘)forlineinself.stub_client.getResult(rpc_pb2.empty()):file_handle.write(line.text)file_handle.close()defstopJmeter(self):‘‘‘终止测试任务‘‘‘self.stub_client.stopJMeter(rpc_pb2.empty())if__name__==‘__main__‘:client=performance(rpc_server,rpc_port)code=client.sendConfig(‘test.jmx‘)iplist=r‘10.1.1.1,10.1.1.2‘forreal_time_resultsinclient.runJmeter(None):print"getrealtimelogfromserver:%s"%real_time_results.textprintclient.generateResult()printclient.getResult(‘2222222222222222‘)client.stopJmeter()

结果:

技术分享

当你想停止jmeter压测,调用stop即可。

本文出自 “大風” 博客,请务必保留此出处http://lansgg.blog.51cto.com/5675165/1931961

python grpc 应用

评论关闭