Hello. I'm Shimizu, and I develop network-related services for the private cloud at LY Corporation.
Modern large-scale web services are rarely built as a single monolithic application. Instead, microservice architectures, in which many small services work together, have become mainstream. LY Corporation, which provides a wide range of services, is no exception. Countless microservices run within the company, many of them on our internally developed private cloud.
That private cloud is Flava. Flava was created by integrating the private clouds previously operated by LINE and Yahoo! JAPAN. In addition to OpenStack-based Infrastructure as a Service (IaaS), it provides managed services on Kubernetes and supports a diverse range of workloads, including virtual machines, bare-metal servers, Pods, and applications running on Platform as a Service (PaaS).
Microservice architecture itself has already been discussed from many perspectives, so this article focuses specifically on service-to-service connectivity. We will introduce the challenges unique to Flava and explain how we are addressing them.
Security in Flava
Flava provides a logically isolated network for each tenant and controls traffic at network boundaries using Layer 4 ACLs. By default, these ACLs deny both outbound and inbound traffic, except for a limited set of internal systems. Many services also use the open-source authentication and authorization system Athenz to enforce application-level access control in addition to Layer 4 access control. In this way, Flava is designed to be secure by default.
Strong security, however, often comes at the expense of user experience. Our mission as private cloud developers is to provide better tools for service developers. We therefore need to achieve both strong security and an excellent user experience.
About Athenz
Athenz is an open-source authentication and authorization system based on service identities. It authenticates each workload using X.509 certificates and other credentials, and defines policies that specify which roles may perform which actions on which resources. For service-to-service communication, application-level access is controlled by assigning the source service account to a role that has permission to access the destination.
Challenges in service-to-service connectivity
Microservices communicate with other services as part of their day-to-day operation. In Flava, however, connecting services required a substantial amount of work, which degraded the developer experience. Users simply wanted to connect two services, but the process required to do so was complicated.
Let us look at a concrete example.
Suppose that Service A, which provides a frontend, needs to connect to TCP port 8080 on Service B, which provides a backend API.
Traditionally, establishing this connection required at least the following steps:
- Find the destination IP address of Service B
- Allow egress traffic to Service B in the VPC for Service A
- Allow ingress traffic from Service A in the VPC for Service B
- Configure an application-level authorization policy that permits the use of Service B
- Configure load balancing, timeouts, and retries as needed
Each setting involves different parties, such as the source and destination teams, network administrators, and the authentication and authorization platform team. Although the user only wants to connect A to B, the actual process requires requests and configuration changes across multiple systems.
As connectivity becomes more complex, it also becomes difficult to track which of the settings distributed across multiple systems are actually applied and why each rule is needed. As a result, unnecessary rules tend to remain after service configurations change, increasing the burden of periodically reviewing permissions and maintaining least privilege. Leaving unnecessary rules in place is itself a security risk. Security and user experience are often viewed as a trade-off, but a poor user experience can also prevent security measures from functioning as intended.
Our goal is to provide an easy service-to-service connectivity experience, along with a mechanism that continuously maintains security, without compromising safety.
Challenge 1: Service-to-service connections are difficult to manage
Communication from one service to another requires ACL changes on both the source and destination sides. If different teams manage the two sides, both teams must submit, review, and approve the changes.
As the number of services grows, the required work increases by more than just the number of services. Connections such as A to B, A to C, and B to C multiply, and every relationship must be managed individually. Teams can end up spending time coordinating connectivity before they can even implement the service itself.
This creates two equally valid requirements: developer agility and platform governance. Developers want to add connections quickly, while the platform must preserve default-deny policies and least privilege. What we need is not to abandon one in favor of the other, but a mechanism that establishes secure connections with less work.
Challenge 2: IP addresses do not match the reality of services
ACLs contain IP addresses. Developers, however, want to connect to services, not IP addresses. This difference in management units becomes a problem when the IP addresses that make up a service change.
For example, recreating a virtual machine in another region changes its IP address. Simply scaling the number of virtual machines up or down also changes the set of IP addresses that make up the service.
If ACLs are not updated after each of these changes, some instances become unreachable while rules for instances that no longer exist remain in place. Allowing a broad CIDR range can reduce the risk of missed updates, but doing so moves the system away from the principle of least privilege.
ACL rules are also expressed as combinations of IP addresses and ports. Looking at these combinations alone does not reveal which service is communicating or why. During incident investigation or permission reviews, engineers must map IP addresses back to services and the organizations responsible for them.
In the cloud, scaling and recreation continuously change the instances that make up a service and their IP addresses. Manually tracking these changes and managing services by IP address had reached its limits.
Challenge 3: Traffic control
In addition to access control, we faced challenges with the paths used for communication between services within data centers and clouds, also known as east-west traffic.
Traditionally, service-to-service traffic was aggregated through shared load balancers located in a subset of racks within the data center. Because many services traversed the same load-balancer layer, traffic became concentrated at specific points in the network fabric.
Furthermore, when load-balancer nodes handling traffic for the same virtual IP address do not share state, it is difficult to implement stateful controls such as circuit breakers, which detect consecutive failures and temporarily stop connections.
For microservices, simply being able to communicate is not enough. To prevent cascading failures, destination selection, timeouts, retries, rate limits, circuit breakers, and other controls must be configured according to the relationship between services.
A paradigm shift: Making "services", not IP addresses, the basis of control
To address these challenges, we decided to shift the basis of control from IP addresses to services.
Each service is registered under a name and associated with the following information:
- The IP addresses and ports of the compute nodes that provide the service (endpoints)
- The Athenz service account used by the service
- Traffic policies for communicating with the service, including load-balancing, timeout, and retry strategies
Users do not need to manually register or deregister endpoints. When a compute node is created, the user specifies the service to which it belongs, and its IP address is automatically registered as an endpoint. When the compute node is deleted, the endpoint is automatically deregistered in accordance with the node's lifecycle.
Users therefore only need to reference a stable service name, leaving the platform to track changing IP addresses.
By using the same service information not only for load balancing but also to generate VPC ACLs, the network rules can be updated automatically when a service's IP addresses change. This prevents unnecessary ACL rules for endpoints that no longer exist from remaining in place and reduces the risk of unintentionally permitting traffic.
Turning service-to-service connectivity into a single operation
Changing the unit of control to the service also allows us to change how connections are established.
The developer of a source service finds the service they want to use in a list of destinations and declares that they want to connect to it. In practice, the entire connection process is completed through the UI, as shown below.
After the administrator of the destination service approves the request when necessary, the platform records the connection between the services. Based on that relationship, it performs the following operations:
- Distribute destination information to the source service
- Configure the VPC ACLs required for communication between the services
- Associate the Athenz role with the service account
- Apply policies such as timeouts and retries to the sidecar
Network, authentication and authorization, and traffic-control settings that were previously managed separately are now derived from a single intent: connecting one service to another.
It is also important that the connection itself remains available as data. Because we can see which services use which other services, it becomes easier to investigate the scope of an impact and review permissions that are no longer needed.
The future of the "service-oriented cloud"
We are developing an internal platform called Service Connect to enable this service-centric connectivity model.
Service Connect centrally manages the service definitions and connection relationships described above. When an application specifies a service name, service discovery returns the currently available endpoints. Users do not need to be aware of IP addresses that change as compute nodes are added or removed.
This endpoint information is used not only for destination discovery and load balancing, but also to generate Layer 3 and Layer 4 VPC ACLs. When a connection between services is registered, the ACLs required for communication are configured automatically. The ACLs are also updated when endpoints are added or removed, allowing the network configuration to track the actual service configuration.
Service Connect handles more than service discovery and ACLs. Traffic policies such as timeouts and retries, as well as authentication and authorization through Athenz, are applied based on the same connection relationship. A sidecar proxy in the compute resource selects the destination and distributes traffic on the client side, eliminating the need for east-west traffic to pass through a central shared load balancer each time.
In the past, a service could also end up with multiple proxy layers: one sidecar proxy for traffic control and another for Athenz authorization. Service Connect integrates these responsibilities into a single sidecar. From the application's perspective, passing through the proxy once applies load balancing, traffic control, authentication, and authorization.
Some readers may feel that Service Connect is itself a service mesh. Technically, it shares common structures with a typical service mesh: a control plane distributes configuration, while sidecars and libraries control communication.
However, Service Connect is not limited to a single Kubernetes cluster or one group of applications. It handles bare-metal servers, virtual machines, Kubernetes Pods, and PaaS using the same service model. In addition to Layer 7 traffic control, it updates IP-based ACLs from service connection relationships.
We did not initially set out to build a service mesh. We explored how to simplify service-to-service connectivity across different execution environments while preserving the current level of security. The result was a mechanism that controls the entire private cloud in units of services. We view this as a "service-oriented cloud."
Conclusion
The future of service-to-service connectivity does not mean eliminating strict network isolation. Instead of requiring developers to work directly with IP addresses and individual ACLs, developers declare the services they want to connect to, and the platform translates that intent into concrete network configuration, authentication and authorization, and traffic control. This approach makes it possible to achieve both security and a good user experience.
This mechanism is still under development, but our goal is to build a platform that lets service developers focus on their core development work instead of connectivity configuration.
