企业生产级百亿日PV高可用可拓展的RPC框架。理论上并发数量接近服务器带宽,客户端采用thrift协议,服务端支持netty和thrift的TThreadedSelectorServer半同步半异步线程模型,支持动态扩容,服务上下线,权重动态,可用性配置,泛化调用,页面流量统计,泛化调用等,支持trace跟踪等,天然接入cat支持数据大盘展示等
1:为什么要写这个RPC
市面上常见的RPC框架很多,grpc,motan,dubbo等,但是随着越来越多的元素加入,复杂的架构设计等因素似使得这些框架和spring一样,虽然号称是轻量级,但是用起来却是让我们很蹩脚,大量的配置,繁杂的API设计,其实,我们根本用不上这些东西!!! 我也算得上是在很多个互联网企业厮杀过,见过很多很多的内部RPC框架,有些优秀的设计让我非常赞赏,有一天我突然想着,为什么不对这些设计原型进行聚合归类,自己搞一套【轻量级】RPC框架呢,碍于工作原因,一直没有时间倒腾出空,十一期间工作闲暇,说搞就搞吧,落地不易,很多细节性问题,比如tcp中怎么解决大量的wait- time,如何做到thrift和netty的兼容等等大量细节的优化,希望源码对大家对认识RPC框架起到推进的作用。东西越写越多,有各种问题欢迎随时拍砖
2:为什么叫koalas
树袋熊英文翻译,希望考拉RPC给那些不太喜欢动手自己去造轮子的人提供可靠的RPC使用环境
3:技术栈
4:关于技术选型
5:安装教程
考拉RPC确保精简,轻量的原则,只需要zk服务器进行服务发现(后续版本服务治理可能需要Datasource),对于zookeeper的各个环境安装教程请自行google,不在本安装教程内特意说明 如果需要cat的数据大盘功能,想更方便的查看服务的调用情况,需要安装cat服务,至于cat的安装就更简单了,就是war包扔在tomcat里面运行,然后配置一些参数即可,当然你也可以不接入cat,单独的作为RPC框架来使用。 CAT接入参考:https://www.oschina.net/p/cat-dianping
1:前期准以及依赖
maven依赖
<dependency> <groupId>koalas.rpc</groupId> <artifactId>com.Koalas.rpc</artifactId> <version>Koalas-1.0-SNAPSHOT</version> </dependency>
关于私服的引用问题,记得全局文件不要把全局的依赖都代理掉,因为这么做只能从aliyun的私服上下载项目,由于koalas- rpc中的Cat依赖只在美团点评的私有仓库中存在,这么做会下载依赖失败,所以不要暴力的设置下面的代理做法。
<mirror> <id>nexus-aliyun</id> <mirrorOf>*</mirrorOf> <name>Nexus aliyun</name> <url>http://maven.aliyun.com/nexus/content/groups/public</url> </mirror>
正确的做法是将代理去掉,直接按照作者在pom.xml文件中给定的依赖仓库地址就可以了。
首先需要编写自己的thrift idl文件了,这里多说一句,在群里的小伙伴曾经说过idl文件编写不熟悉,有可能出错 这里顺带说一嘴,thrift的ldl文件和写java的请求体和service几乎没有任何区别,熟能生巧,上手之后非常简单 这里推荐几篇thrift的文章,有兴趣可以看一看 https://blog.csdn.net/lk10207160511/article/details/50450541, https://blog.csdn.net/hrn1216/article/details/51306395 下面截图为测试的thrift文件
namespace java thrift.service include 'WmCreateAccountRequest.thrift' include 'WmCreateAccountRespone.thrift' service WmCreateAccountService { WmCreateAccountRespone.WmCreateAccountRespone getRPC(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest1(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest2(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest3(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest4(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest5(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest6(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest7(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest8(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest9(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest10(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest11(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); WmCreateAccountRespone.WmCreateAccountRespone koaloasTest12(1:WmCreateAccountRequest.WmCreateAccountRequest wmCreateAccountRequest); } namespace java thrift.domain /** * 测试类 **/ struct WmCreateAccountRequest { 1:i32 source, 2:i32 accountType, 3:i64 partnerId, 4:i32 partnerType, 5:string partnerName, 6:i32 poiFlag, } namespace java thrift.domain /** * 测试类 **/ struct WmCreateAccountRespone { 1:i32 code, 2:string message, }
编译器需要大家去下载对应的版本 windows和linux下不同的编译器,下载地址http://archive.apache.org/dist/thrift/0.8.0/ 下载0.8.0版本即可,0.8.0版本是很老的版本了,但是相对稳定,后续会把thirft版本升级。如果上面地址下载不下来或者失效,可以上作者的网盘上下载zip包,上面有win版本和mac,linux版本的0.8.0的thrift编译器,链接: https://pan.baidu.com/s/1JpLqVbmokTOe30nU_TznWw提取码: ntye, 编译上面三个文件 thrift -gen java WmCreateAccountService.thrift, thrift -gen java WmCreateAccountRequest.thrift, thrift -gen java WmCreateAccountRespone.thrift 在当前目录下会生成3个java文件 这三个文件分别是请求体,返回体,和服务类,就这么简单 Ok作为开发者而言,所有的准备工作都结束了。下面就开始进入实际开发~
1. 客户端同步调用
首先在你的xml里面配置一下引用
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:koalas="http://www.koalas.com/schema/ch" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://www.koalas.com/schema/ch http://www.koalas.com/schema/ch.xsd"> <koalas:client id="wmCreateAccountService1" serviceInterface="thrift.service.WmCreateAccountService" zkPath="127.0.0.1:2181"/> </beans>
首先引用koalas的自定义schema,xmlns:koalas和xsi:schemaLocation, 其中serviceInterface为thrift自动生成的java类,zkPath为zk的服务地址,默认是同步调用,接下来就是在java里面的远程调用了。
package thrift.service; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import thrift.domain.WmCreateAccountRequest; import thrift.domain.WmCreateAccountRespone; @Service("testService") public class TestService { @Autowired WmCreateAccountService.Iface wmCreateAccountService; public void getRemoteRpc() throws TException { WmCreateAccountRequest request= new WmCreateAccountRequest ( ); //request.setSource ( 10 ); request.setAccountType ( 1 ); request.setPartnerId ( 1 ); request.setPartnerType ( 1 ); request.setPartnerName ( "你好" ); request.setPoiFlag ( 1 ); WmCreateAccountRespone respone = wmCreateAccountService.getRPC ( request); System.out.println (respone); } }
就这么简单一个高性能的RPC框架就诞生了。WmCreateAccountService是thrift自动生成的,作为使用者而言不需要做任何事情,只需要在spring bean中注入xxx.Iface即可。
2. 客户端异步调用
刚刚我们看了客户端的同步调用方式,下面我们一起来看看异步的使用方式, 首先在你的xml里面配置一下引用
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:koalas="http://www.koalas.com/schema/ch" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://www.koalas.com/schema/ch http://www.koalas.com/schema/ch.xsd"> <koalas:client id="wmCreateAccountService2" serviceInterface="thrift.service.WmCreateAccountService" zkPath="127.0.0.1:2181" async="true"/> </beans>
和同步的区别async=true,代表异步使用,接下来就是在java里面的异步远程调用了
package thrift.service; import client.async.KoalasAsyncCallBack; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import thrift.domain.WmCreateAccountRequest; import thrift.domain.WmCreateAccountRespone; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; @Service("testService") public class TestService2 { @Autowired WmCreateAccountService.AsyncIface wmCreateAccountService; public void getRemoteRpc() throws TException{ KoalasAsyncCallBack<WmCreateAccountRespone, WmCreateAccountService.AsyncClient.getRPC_call> koalasAsyncCallBack = new KoalasAsyncCallBack<> (); WmCreateAccountRequest request= new WmCreateAccountRequest ( ); request.setAccountType ( 1 ); request.setPartnerId ( 1 ); request.setPartnerType ( 1 ); request.setPartnerName ( "你好啊" ); request.setPoiFlag ( 1 ); wmCreateAccountService.getRPC ( request ,koalasAsyncCallBack); Future<WmCreateAccountRespone> future= koalasAsyncCallBack.getFuture (); try { //to get other things System.out.println (future.get ()); } catch (InterruptedException e) { e.printStackTrace (); } catch (ExecutionException e) { e.printStackTrace (); } } }
这次调用getRpc方法不会阻塞等待server同步结果了。而是可以去干一些自己的其他事情,然后在调用future.get ()来获得返回resopne,当然future.get ()支持最大等待时间的,超时之后会抛出TimeOutException,当然这仅仅是client超时而已不会影响server的执行结果。
3. 服务端实现
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:koalas="http://www.koalas.com/schema/ch" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.2.xsd http://www.koalas.com/schema/ch http://www.koalas.com/schema/ch.xsd"> <koalas:server id="WmCreateAccountService" serviceInterface="thrift.service.WmCreateAccountService" serviceImpl="wmCreateAccountServiceImpl" port="8001" zkpath="127.0.0.1:2181"/> </beans>
服务端只需要指定暴露的端口,zk服务地址和服务端实现即可。
@Service public class WmCreateAccountServiceImpl implements WmCreateAccountService.Iface { @Override public WmCreateAccountRespone getRPC(WmCreateAccountRequest wmCreateAccountRequest) throws TException { WmCreateAccountRespone wmCreateAccountRespone = new WmCreateAccountRespone (); wmCreateAccountRespone.setCode ( 1 ); wmCreateAccountRespone.setMessage ( "你好" ); if(new Random ( ).nextInt ( 5 )>100){ throw new RuntimeException ( "测试错误" ); } System.out.println ( "getRPC start ...." + wmCreateAccountRequest + "------" + atomicInteger.incrementAndGet () ); return wmCreateAccountRespone; } }
只需要实现xxxx.Iface即可
有的小伙伴会觉得配置xml有点麻烦,koalas-rpc也提供了纯注解的使用方式
1. 客户端调用
xml中的配置
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:koalas="http://www.koalas.com/schema/ch" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://www.koalas.com/schema/ch http://www.koalas.com/schema/ch.xsd"> <koalas:annotation package="thrift.annotation.client.impl"/> </beans>
一个扫描标签就行了,如果你在spring bean里想通过调用rpc远程服务,那么扫描一下就行了
java中使用
@Service("testServiceSync") public class TestServiceSync { @KoalasClient(zkPath = "127.0.0.1:2181",readTimeout = 5000*1000) WmCreateAccountService.Iface wmCreateAccountService; public void getRemoteRpc() throws TException { WmCreateAccountRequest request= new WmCreateAccountRequest ( ); //request.setSource ( 10 ); request.setAccountType ( 1 ); request.setPartnerId ( 1 ); request.setPartnerType ( 1 ); request.setPartnerName ( "你好啊-我是注解实现的" ); request.setPoiFlag ( 1 ); WmCreateAccountRespone respone = wmCreateAccountService.getRPC ( request); System.out.println (respone); } }
只需要在你想远程调用的类上加一个@KoalasClient注解就可以了,远程调用就这么简单,当然异步使用方式也类似
@Service("testServiceAsync") public class TestServiceAsync { @KoalasClient(zkPath = "127.0.0.1:2181",readTimeout = 5000*1000) WmCreateAccountService.AsyncIface wmCreateAccountService; public void getRemoteRpc() throws TException{ KoalasAsyncCallBack<WmCreateAccountRespone, WmCreateAccountService.AsyncClient.getRPC_call> koalasAsyncCallBack = new KoalasAsyncCallBack<> (); WmCreateAccountRequest request= new WmCreateAccountRequest ( ); //request.setSource ( 10 ); request.setAccountType ( 1 ); request.setPartnerId ( 1 ); request.setPartnerType ( 1 ); request.setPartnerName ( "你好啊-我是注解实现的" ); request.setPoiFlag ( 1 ); wmCreateAccountService.getRPC ( request ,koalasAsyncCallBack); Future<WmCreateAccountRespone> future= koalasAsyncCallBack.getFuture (); try { System.out.println (future.get ()); } catch (InterruptedException e) { e.printStackTrace (); } catch (ExecutionException e) { e.printStackTrace (); } } }
注意和同步调用不同的是自定义注解注入的接口是xxxx.AsyncIface,同步是xxxx.Iface。KoalasAsyncCallBack回调使用方式和上面的xml一样。有一点需要说明
<koalas:annotation package="thrift.annotation.client.impl"/>
如果package属性设置为空,那么所有的@KoalasClient都会生效,也就是说所有在spring bean中的自定义注解@KoalasClient都会自动注入。这里说另外一种用法
private WmCreateAccountService.Iface wmCreateAccountService; @KoalasClient(zkPath = "127.0.0.1:2181",readTimeout = 5000*1000) public void setWmCreateAccountService(WmCreateAccountService.Iface wmCreateAccountService){ this.wmCreateAccountService = wmCreateAccountService; }
直接注入方法的方式也是可以的。
2. 服务端实现
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:koalas="http://www.koalas.com/schema/ch" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://www.koalas.com/schema/ch http://www.koalas.com/schema/ch.xsd"> <koalas:annotation package="thrift.annotation.server.impl"/> </beans>
配置和client中一样只需要配置一个自定义标签即可,java中的使用方式如下:
package thrift.annotation.server.impl; import annotation.KoalasServer; import org.apache.thrift.TException; import thrift.domain.WmCreateAccountRequest; import thrift.domain.WmCreateAccountRespone; import thrift.service.WmCreateAccountService; import java.util.Random; import java.util.concurrent.atomic.AtomicInteger; @KoalasServer ( port = 8801,zkpath="127.0.0.1:2181") public class WmCreateAccountServiceNettyImpl implements WmCreateAccountService.Iface { private AtomicInteger atomicInteger = new AtomicInteger ( 0 ); @Override public WmCreateAccountRespone getRPC(WmCreateAccountRequest wmCreateAccountRequest) throws TException { WmCreateAccountRespone wmCreateAccountRespone = new WmCreateAccountRespone (); wmCreateAccountRespone.setCode ( 1 ); wmCreateAccountRespone.setMessage ( "你好啊" ); if(new Random ( ).nextInt ( 5 )>100){ try { Thread.sleep ( 5000 ); } catch (InterruptedException e) { e.printStackTrace (); } } System.out.println ( "getRPC start ...." + wmCreateAccountRequest + "------" + atomicInteger.incrementAndGet () ); return wmCreateAccountRespone; } }
这样服务实现就会主从注册到zookeeper中提供给client端使用了。值得说明的是被扫描到并且类上有@KoalasServer的类会被加载到spring上下文中,可以当成一个普通的spring bean来处理,还有一点如果你不指定package,配置成如下情况
<koalas:annotation package=""/>
这样配置会以spring的bean为基础实现,那么使用方式需要改成
package thrift.annotation.server.impl; import annotation.KoalasServer; import org.apache.thrift.TException; import thrift.domain.WmCreateAccountRequest; import thrift.domain.WmCreateAccountRespone; import thrift.service.WmCreateAccountService; import java.util.Random; import java.util.concurrent.atomic.AtomicInteger; @KoalasServer ( port = 8801,zkpath="127.0.0.1:2181") @Service public class WmCreateAccountServiceNettyImpl implements WmCreateAccountService.Iface { private AtomicInteger atomicInteger = new AtomicInteger ( 0 ); @Override public WmCreateAccountRespone getRPC(WmCreateAccountRequest wmCreateAccountRequest) throws TException { WmCreateAccountRespone wmCreateAccountRespone = new WmCreateAccountRespone (); wmCreateAccountRespone.setCode ( 1 ); wmCreateAccountRespone.setMessage ( "你好啊" ); if(new Random ( ).nextInt ( 5 )>100){ try { Thread.sleep ( 5000 ); } catch (InterruptedException e) { e.printStackTrace (); } } System.out.println ( "getRPC start ...." + wmCreateAccountRequest + "------" + atomicInteger.incrementAndGet () ); return wmCreateAccountRespone; } }
就这么简单即可。
3. 泛化调用
为什么需要泛化调用? 1:有一个通用压测平台,想去压测不同的server。那么现在就有一个问题了,不可能让压测平台服务端去依赖所有的下游服务,这样依赖会很繁杂,这时候如果说只配置serviceName,request模型和request请求json就可以进行远程调用,那么将大大的减少头疼的依赖。 2:假设php同事对java代码不熟悉,不可能让他们去依赖spring,一共一套简单的api来使用是很有必要的。 3:上游服务不想依赖下游服务的数据模型。
对于泛化调用来说,dubbo已经提供,soft-rpc也有提供。当然koalas-rpc也不会例外,并且支持xml,注解和java api的使用方式。下面几个例子来说明一下使用方式。更多demo去源码中查看,作者已经写好,开箱即用。
xml使用方式
<koalas:client id="wmCreateAccountService3" serviceInterface="thrift.service.WmCreateAccountService" zkPath="127.0.0.1:2181" generic="true" readTimeout="50000000"/> @Autowired @Qualifier("wmCreateAccountService3") GenericService.Iface wmGenericService; public void getGenericRpc() throws TException { GenericRequest request = new GenericRequest ( ); request.setMethodName ( "getRPC" ); request.setClassType ( new ArrayList<String> ( ){{ add ( "thrift.domain.WmCreateAccountRequest"); }} ); request.setRequestObj ( new ArrayList<String> ( ){{ add ( "{\"accountType\":1,\"partnerId\":1,\"partnerName\":\"你好\",\"partnerType\":1,\"poiFlag\":1,\"source\":0}"); }} ); String str = wmGenericService.invoke ( request ); System.out.println (str); }
简单说明一下,GenericService.Iface是通用服务,有三个参数,第一个是方法名称,第二个是请求体类型集合,第三个是请求体内容。直接调用即可,返回值是server端的json类型,使用json工具为阿里巴巴的Fast- json
注解使用方式
@KoalasClient(zkPath = "127.0.0.1:2181",readTimeout = 5000*1000,genericService = "thrift.service.WmCreateAccountService") GenericService.Iface genericService; public void getGenericRemoteRpc() throws TException { GenericRequest request = new GenericRequest ( ); request.setMethodName ( "getRPC" ); request.setClassType ( new ArrayList<String> ( ){{ add ( "thrift.domain.WmCreateAccountRequest"); }} ); request.setRequestObj ( new ArrayList<String> ( ){{ add ( "{\"accountType\":1,\"partnerId\":1,\"partnerName\":\"你好\",\"partnerType\":1,\"poiFlag\":1,\"setAccountType\":true,\"setPartnerId\":true,\"setPartnerName\":true,\"setPartnerType\":true,\"setPoiFlag\":true,\"setSource\":false,\"source\":0}"); }} ); String str = genericService.invoke ( request ); System.out.println (str); }
唯一区别的是注解要指定genericService,当genericService不为空时,默认开启泛化调用 当然,java api方式也是支持的。
KoalasClientProxy koalasClientProxy = new KoalasClientProxy(); koalasClientProxy.setServiceInterface ( "thrift.service.WmCreateAccountService" ); koalasClientProxy.setZkPath ("127.0.0.1:2181" ); koalasClientProxy.setGeneric ( true ); koalasClientProxy.setReadTimeout ( 50000000 ); koalasClientProxy.afterPropertiesSet (); GenericService.Iface genericService = (GenericService.Iface) koalasClientProxy.getObject (); GenericRequest request = new GenericRequest ( ); request.setMethodName ( "getRPC" ); request.setClassType ( new ArrayList<String> ( ){{ add ( "thrift.domain.WmCreateAccountRequest"); }} ); request.setRequestObj ( new ArrayList<String> ( ){{ add ( "{\"accountType\":1,\"partnerId\":1,\"partnerName\":\"你好\",\"partnerType\":1,\"poiFlag\":1,\"setAccountType\":true,\"setPartnerId\":true,\"setPartnerName\":true,\"setPartnerType\":true,\"setPoiFlag\":true,\"setSource\":false,\"source\":0}"); }} ); String str = genericService.invoke ( request ); System.out.println (str); koalasClientProxy.destroy ();
特别注意的是KoalasClientProxy对象非常非常重,一定要在服务关闭的时候执行koalasClientProxy.destroy ();方法,并且需要带应用程序中缓存该对象,千万不要每次使用都要创建,这样会极大的浪费资源,每个服务对应一个KoalasClientProxy,同步和异步也是不同的对象,这些使用者需要注意。