想法&实践

最近有把一个站点的构建过程记录下来的想法，于是把所需要的技术关键点全部都一一罗列出来，当做技术笔记。我在自己的私人代码库里面，做成了脚手架，方便自己。更重要的是，在这个梳理过程，让我对以前的一些系统设计有了新的体会。重新搭建的过程，其实是愉悦、流畅的。

技术栈

• 语言：java
• 框架：spring-cloud
• 组件：zuul,eureka,spring-cloud feign,spring-cloud ribbon,spring-cloud zipkin ,spring-cloud zipkin,sleuth，ctrip-apollo
• 中间件: 接入层 open-resty,消息中间件 kafka, 缓存 redis
• 文件存储：OSS
• 邮件服务：Ali Mail or Mail Gun
• API 日志：采集数据，存到Hive
• 搜索 elastic search
• 日志检索系统： elk相关的 Elasticsearch、Logstash、Kibana
• 监控：简单的监控可以自己实现(接口为粒度)，微服务监控：Prometheus，硬件监控 Zabbix：CPU、内存、硬盘、网络、端口、日志等
• CDN 内容分发
• 负载均衡: LVS + keepalive 、F5 、DNS轮询

划分&业务分层

• 公网
• 接入层
• 业务层
• 数据层
• 缓存
• CI/CD
• 自动化测试与运维
• 容器化
• 弹性扩容、缩容
• 其他

关键代码

nginx 配置

user       www www;  ## Default: nobody
worker_processes  1;  ## Default: 1
error_log  /Users/lucas/logs/error.log;
pid        /Users/lucas/logs/nginx.pid;
worker_rlimit_nofile 8192;

events {
  worker_connections  1024;  ## Default: 1024
}

http {
  include     /Users/lucas/Desktop/local-mime.types;
  #include    /etc/nginx/proxy.conf;
  #include    /etc/nginx/fastcgi.conf;
  index    index.html index.htm index.php;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  response_status=$status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"request_time=$request_time " uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"'
    '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   /Users/lucas/logs/access.log  main;
  sendfile     on;
  tcp_nopush   on;
  server_names_hash_bucket_size 128; # this seems to be required for some vhosts

  upstream spacex-fox-qiong {
    server 127.0.0.1:8082;
    server 127.0.0.1:8083;
    server 127.0.0.1:8084;
  }

  upstream spacex-fox-qiong-api {
    server 127.0.0.1:8085;
  }

  server { # simple reverse-proxy
    listen       80;
    server_name  localhost;
    access_log   /Users/lucas/logs/localhost.access.log  main;

    # serve static files
    location ~ ^/(images|javascript|js|css|flash|media|static)/  {
      root    /var/www/virtual/spacex.server.com/htdocs;
      expires 30d;
    }

    # pass requests for dynamic content to http://spacex-fox-qiong, et al
    location / {
      proxy_pass      http://spacex-fox-qiong;
    }

    # pass requests for dynamic content to http://spacex-fox-qiong-api, et al
    location /api {
      proxy_pass      http://spacex-fox-qiong-api;
    }
  }

  # 看这里
  upstream spacex-qiong-zuul-gateway {
    server 127.0.0.1:8765;
  }

  server { # spacex zuul gateway
    listen       80;
    server_name  api.spacex.com;
    access_log   /Users/lucas/logs/localhost.access.log  main;

    # serve static files
    location ~ ^/(images|javascript|js|css|flash|media|static)/  {
      root    /var/www/virtual/spacex.server.com/htdocs;
      expires 30d;
    }

    # pass requests to zuul gateway
    location / {
      proxy_pass      http://spacex-qiong-zuul-gateway;
    }
  }

  upstream spacex-server.com {
    server 127.0.0.3:8000 weight=5;
    server 127.0.0.3:8001 weight=5;
    server 192.168.0.1:8000;
    server 192.168.0.1:8001;
  }

  server { # simple load balancing
    listen          80;
    server_name     big.server.com;
    access_log      /Users/lucas/logs/spacex.server.access.log main;

    location / {
      proxy_pass      http://spacex-server.com;
    }
  }
}

修改本地hosts文件，我的电脑为例，地址是：/etc/hosts
这是修改后的配置hosts文件

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost   Lucass-MacBook-Pro.local api.spacex.com
255.255.255.255 broadcasthost
::1             localhost   Lucass-MacBook-Pro.local
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

注意这一句：127.0.0.1 localhost Lucass-MacBook-Pro.local api.spacex.com
这句把api.spacex.com 绑定到本地

curl 就可以这样调用：

curl api.spacex.com/community/12334456
curl http://api.spacex.com/community/123456

整个调用链就变成了：

api.spacex.com/community/12334456 -> nginx -> gateway -> community-service -> user-service

接入层拓展

如果nginx上面再加一层LVS+Keepalive，那么就可以实现软负载和高可用了。
VIP + keepalived的方式只有50%的利用率，怎么提高利用率呢？
答案就是双虚IP+dns轮询的方式。

阿里云的SLB服务（可以认为是一个增强版lvs+keepalived的高可用负载均衡服务）

接入层扩容

虽然现在实现了高可用，负载均衡，但是还是受限于一台LVS或者一台F5的性能。如果有更多的请求进来怎么办，答案就是用DNS轮询实现水平扩展。
到了这一步，那就是各大厂的级别，要做的事情很多，需要更多的工程师来配合。

接入层组件

• 1 nginx：一个高性能的web-server和实施反向代理的软件
• 2 lvs：Linux Virtual Server，使用集群技术，实现在linux操作系统层面的一个高性能、高可用、负载均衡服务器
• 3 keepalived：一款用来检测服务状态存活性的软件，常用来做高可用
• 4 f5：一个高性能、高可用、负载均衡的硬件设备（看上去和lvs功能差不多，软负载和硬件负载）
• 5 DNS轮询：通过在DNS-server上对一个域名设置多个ip解析，来扩充web-server性能及实施负载均衡的技术

(架构图待补充)

spring cloud相关的代码如下

spring could eureka 代码

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@SpringBootApplication
@EnableEurekaServer // Eureka server
public class SpaceXQiongEurekaApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpaceXQiongEurekaApplication.class, args);
    }
}

eureka application.properties 配置文件

# eureka config
# application unique name
spring.application.name=spacex-eureka-naming-server

# server port
server.port=8761

# don't register
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false

spring could zuul 代码

这个只是简单版

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.zuul.EnableZuulProxy;

@EnableZuulProxy
@EnableDiscoveryClient
@SpringBootApplication
public class SpaceXQiongGatewayApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpaceXQiongGatewayApplication.class, args);
    }
}

若干网关过滤器 filter

import com.netflix.zuul.ZuulFilter;
import com.netflix.zuul.context.RequestContext;
import com.netflix.zuul.exception.ZuulException;
import com.spacex.qiong.gateway.util.WebUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import javax.servlet.http.HttpServletRequest;

@Component
public class ZuulPreFilter extends ZuulFilter {

    private Logger logger = LoggerFactory.getLogger(this.getClass());

    @Override
    public String filterType() {
        return "pre";
        // filter before request is executed
        // return "post"; filter after request is executed
        // return "error"; upon request error
    }

    @Override
    public int filterOrder() {
        return 1;
    }

    @Override
    public boolean shouldFilter() {
        return true;
    }

    @Override
    public Object run() throws ZuulException {//这里可以做接口签名校验,鉴权，url重写，api原始数据采集等，这里只做简单展示
        logger.info("request is filtered!");
        HttpServletRequest httpServletRequest = RequestContext.getCurrentContext().getRequest();

        String ipAddress = WebUtil.getRemortIP(httpServletRequest);
        logger.info("request remote address:{}, request uri:{}", ipAddress, httpServletRequest.getRequestURI());
        return null;
    }
}

zuul 网关 application.properties 配置文件

spring.application.name=spacex-gateway-zuul
server.port=8765
eureka.client.service-url.default-zone=http://localhost:8761/eureka

# Zuul routers

zuul.routes.spacex-community-service.path= /community/**
zuul.routes.spacex-community-service.url= http://localhost:5000/community
zuul.routes.spacex-user-service.path= /user/**
zuul.routes.spacex-user-service.url= http://localhost:4000/user

这样的zuul网关的缺点是，每增加一个服务，都需要修改网关的配置文件，不方便。
这里是可以重写成为支持动态路由的，生产环境建议重写。

重写的关键是：重写服务路由的类DiscoveryClientRouteLocator,

public class CustomRouteLocator extends DiscoveryClientRouteLocator {
    private ZuulProperties properties;
    private DiscoveryClient discovery;

    public CustomRouteLocator(String servletPath, DiscoveryClient discovery, ZuulProperties properties,
            ServiceRouteMapper serviceRouteMapper) {
        super(servletPath, discovery, properties, serviceRouteMapper);
        this.properties = properties;
        this.discovery = discovery;
    }

    // Get All URL from DB (这里不难，略)

    // support Restful的URL (这里不难，略)

    //极其重要：刷新逻辑极其重要
    @Override
    public void refresh() {
        refreshRoute();
        super.refresh();
    }

    // 极其重要：新的路由匹配
    @Override
    public Route getMatchingRoute(String path) {
        RequestContext context = RequestContext.getCurrentContext();
        RouteDTO route = (RouteDTO) context.get(RequestContextConstants.ROUTE);
        if(route == null){
            return null;
        }
        return new Route(route.getSourcePath(), route.getRedirectPath(), route.getServiceId(), "", true, properties.getSensitiveHeaders());
    }

    // RouteDTO 是个自定义的POJO类,包含：SourcePath、RedirectPath、ServiceId、其他等（略）
}

配置类

import javax.annotation.Resource;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.web.ServerProperties;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.cloud.netflix.zuul.filters.ZuulProperties;
import org.springframework.cloud.netflix.zuul.filters.discovery.ServiceRouteMapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.spacex.gateway.route.CustomRouteLocator;

@Configuration
public class CustomZuulConfiguration {

    @Resource
    private ZuulProperties zuulProperties;

    @Autowired
    private DiscoveryClient discovery;

    @Autowired
    private ServiceRouteMapper serviceRouteMapper;

    @Autowired
    protected ServerProperties serverProperties;

    @Bean
    public RouteLocator routeLocator() {
        CustomRouteLocator routeLocator = new CustomRouteLocator(serverProperties.getServletPath(), discovery, zuulProperties,serviceRouteMapper);
        return routeLocator;
    }
}

网关最核心的重写逻辑都在这两个类里，如果需要添加其他服务，可以自定义相关的模块(如：url重写、接口签名校验、鉴权、监控、API调用日志以及历史记录:请求参数、返回状态、结果、响应时间等、多端适配(web/ios/android/小程序)、限流、黑白名单、熔断、权限控制、AB 测试等等)。

两个简单的微服务:user-service和commuinty-service

user-service是服务提供方，commuinty-service是调用方，commuinty-service调用user-service服务的接口。

user-service的关键代码

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

@EnableDiscoveryClient
@SpringBootApplication
public class SpaceXQiongUserApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpaceXQiongUserApplication.class, args);
    }
}

endpoint controller 代码

import javax.annotation.Resource;

@RestController
@RequestMapping("user")
public class UserController {

    @Resource
    private Environment environment;

    @GetMapping("/{userId}")
    public UserInfoDTO userInfo(@PathVariable("userId") Long userId) {
        String[] profiles = environment.getDefaultProfiles();
        UserInfoDTO userInfoDTO = new UserInfoDTO();
        userInfoDTO.setUid(userId);
        userInfoDTO.setName("user-" + System.currentTimeMillis());
        userInfoDTO.setEmail(userInfoDTO.getName() + "@user.com");
        userInfoDTO.setLabel(profiles[0]);

        return userInfoDTO;
    }
}

user-service服务的 application.properties配置文件

spring.application.name=spacex-user-service
server.port=4000

eureka.client.service-url.default-zone=http://localhost:8761/eureka

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.spacex</groupId>
    <artifactId>spacex-qiong-user-service</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.0.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <name>spacex-qiong-user-service-server</name>
    <description>spacex project for user service server</description>

    <properties>
        <java.version>1.8</java.version>
        <spring-cloud.version>Hoxton.SR4</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

community-service 服务

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients("com.spacex.community.remote")
public class SpaceXQiongCommunityApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpaceXQiongCommunityApplication.class, args);
    }
}

community-service 调用 user-service的接口示例：

import com.spacex.community.dto.UserInfoDTO;
import com.spacex.community.remote.UserService;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.Resource;

@RestController
public class CommunityController {

    @Resource
    private UserService userService;

    @GetMapping("community/{userId}")
    public UserInfoDTO get(@PathVariable("userId") Long userId) {
        UserInfoDTO userInfoDTO = userService.get(userId);
        return userInfoDTO;
    }
}

community-service 通过feign初始化 user-service的接口示例:

import com.spacex.community.dto.UserInfoDTO;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;

@FeignClient(name = "spacex-user-service")
public interface UserService {

    @GetMapping("user/{userId}")
    UserInfoDTO get(@PathVariable("userId") Long userId);
}

community-service application.properties配置文件

spring.application.name=spacex-community-service
server.port=5000

#eureka.client.fetchRegistry=true
eureka.client.service-url.default-zone=http://localhost:8761/eureka

有些中间件配置，暂时不罗列。

拓展&问题

• 给定相关的PV (日活DAU、MAU月活、PCU(同时最高在线人数) ),网关、应用服务等应该部署多少台
• 压测怎么做
• 数据冷热备份、核心数据的N备份(冷热备份)
• 灾备、异地多活
• 技术优化、降低成本
• 通用代码组件化
• 规划 & 技术road map

(完)