1M messages per second: Applying E2EE to Apache Kafka in the LINE app

Introduction: Why end-to-end encryption for Kafka?

On the LINE app, billions of messages are exchanged every day. This vast data flows through many systems, and some of it contains highly sensitive personal information that requires strong protection. LY Corporation protects sensitive data with a multi-layered security system and continuously strengthens that protection as the threat landscape and scale of our services evolve.

Chat messages in the LINE app already use end-to-end encryption (E2EE) called "Letter Sealing". However, LY Corporation decided to apply additional E2EE between Kafka clients to provide the highest level of protection for user data. With this approach, the message payload stored on Kafka brokers remains encrypted from producer to consumer.

In this article, we share LY Corporation's experience designing, implementing, and rolling out Kafka E2EE in production. We focus on a large Kafka topic that can handle up to 1,000,000 messages per second, published by one team and consumed by many teams, and describe how we introduced encryption without downtime and what we learned.

Limitations of the existing Kafka security model

Kafka provides three core security features to protect data.

• Transport layer security / secure sockets layer (TLS/SSL): TLS and SSL are used to encrypt the communication channel between broker and client (producer, consumer). This prevents eavesdropping on data in transit.
• Authentication: The simple authentication and security layer (SASL) is used to verify a client's identity before it accesses the broker. This prevents unauthorized users from accessing the system. SASL supports various authentication methods such as username/password and Kerberos.
• Authorization: Access control lists (ACLs) are used to grant or restrict specific users or groups the right to publish to or consume from particular topics.

Despite these strong basic security features, an additional layer is desirable for highly sensitive data. TLS/SSL and authentication/authorization focus on protecting data in transit and controlling access. Each layer protects a different aspect of the system. Although broker servers are operated under strict access controls, those controls regulate who can access the broker, not the plaintext of the data stored on the broker.

For highly sensitive data such as personal information, it is desirable to add a layer that protects the data itself on top of access controls. In other words, encrypt the data at the time of production and keep it encrypted until a consumer decrypts it. That is why we adopted E2EE: an additional layer of defense-in-depth.

Core design

To build a secure and efficient Kafka E2EE system, we defined the following core requirements.

1. Confidentiality: Messages must be encrypted by producers so that only authorized consumers can decrypt them.
2. Scalability: Producers and consumers must be able to be added or removed dynamically. The system must allow new consumers to be added easily.
3. Minimal performance overhead: The computational overhead (CPU, memory) and message size increase caused by encryption and decryption must be minimized.

Record-level encryption

The first design decision was at what granularity to apply encryption.

Batch-level encryption encrypts multiple records after they are batched but before sending to the broker. Because data can be compressed before encryption, this is efficient and has lower CPU overhead. However, Kafka client extension points (such as interceptors) operate at record level, and once a batch is assembled there is no intervention point. To encrypt at batch level would require modifying Kafka client internals.

Record-level encryption encrypts each record before batching. Compared with batch-level encryption, compression efficiency drops and encryption cost increases, so message size grows slightly and CPU cost is higher. But record-level encryption allows using Kafka's standard extension points such as interceptors. Using standard APIs preserves compatibility with Kafka upgrades and lets us apply encryption without modifying existing producers and consumers. For these reasons we chose record-level encryption.

DEK-KEK structure

A single-key approach risks total compromise if that key is exposed. We adopted a dual-key structure of data-encryption key (DEK) and key-encryption key (KEK) to achieve both efficient data protection and flexible key management.

• DEK: For data protection we use a symmetric key that producers use to encrypt the message payload. We use AES-GCM to encrypt payloads. Symmetric algorithms are fast.
• KEK: For KEK we use an elliptic curve cryptography (ECC) key pair shared by authorized consumers. The key pair is managed in a key management service (KMS). Producers retrieve the public key from the KMS and encrypt the DEK; authorized consumers obtain access to the private key from the KMS to decrypt the encrypted DEK. For DEK encryption, we use the elliptic curve integrated encryption scheme (ECIES) with the secp521r1 curve.

Benefits of the DEK-KEK structure

1. Performance optimization for large Kafka environments

Encrypting large payloads with asymmetric keys alone is very expensive. We encrypt the actual payload with a fast symmetric key (DEK) and use the expensive asymmetric operation only to encrypt the short DEK. This keeps encryption overhead low in high-traffic Kafka environments.

2. Keep message size stable for many consumers

No matter how many consumers there are, the producer encrypts the payload only once. The message header contains only the short DEK encrypted with the KEK, so message size remains stable even when many consumers exist.

3. Separation of encryption and decryption privileges

Using asymmetric keys lets us separate the privilege to encrypt from the privilege to decrypt. Producers are given public key based encryption rights only, and only authorized consumers are given private key based decryption rights. This enforces the principle of least privilege and preserves message confidentiality even if the public key is leaked.

Message structure

Below is a comparison of a standard Kafka message and a message after applying E2EE.

To successfully decrypt a message, the encrypted data must be delivered with metadata needed for decryption. Using an external store such as a database or cache complicates system dependencies, so we chose to include the metadata inside the message.

After receiving a message, a consumer checks the metadata to see if it has the KEK that can decrypt the DEK. The consumer then decrypts the DEK with that KEK and finally decrypts the payload with the DEK.

• Key: We keep the existing key used to determine the message partition.
• Header: We store metadata required for decryption in the header introduced in Kafka 0.11.0. This includes the KEK ID that identifies the KEK (or key version) used to encrypt the DEK, and the DEK encrypted with the KEK.
• Body: The body contains the final payload encrypted with the DEK.

System architecture

Based on the message structure above, the actual system implements producers and consumers with the following roles.

Producer: encryption using an interceptor and DEK caching

The producer is responsible for encrypting messages before sending them to the broker.

• DEK generation: Generate a symmetric key (DEK) to encrypt the message payload.
• DEK encryption: Encrypt the DEK with the pre-distributed public key (KEK).
• Message creation: Place the encrypted DEK in the header and the DEK-encrypted payload in the body to form the final message.

All of this encryption processing is handled by the Kafka interceptor and serializer.

• Interceptor: Handle the header. Immediately before a message is sent, intercept it, generate the DEK, encrypt the DEK with the KEK, and insert this information into the header.
• Serializer: Process the payload. Implement the serializer as a wrapper around the existing data serializer. It receives the binary data serialized once and performs a second encryption using the DEK prepared by the interceptor.
• DEK delivery: Take advantage of the fact that the interceptor and serializer share the same execution thread by placing the generated DEK in ThreadLocal, then clean up resources when processing finishes.

Generating a new DEK and encrypting it with the KEK for every message wastes resources. We therefore cache DEKs for a fixed time and reuse the same DEK for messages sent during that period. This reduces the additional resources required to encrypt plaintext payloads for most messages and significantly improves producer performance.

Consumer: decryption strategy using a deserializer

The consumer receives encrypted messages from the broker and is responsible for decrypting them.

• KEK lookup: The topic owner creates the KEK and registers it in the key management service (KMS). After completing authorization, each consumer obtains the private key from the KMS to use for message decryption.
• DEK decryption: Find the encrypted DEK in the received message header and decrypt it with the consumer's KEK private key.
• Payload decryption: Use the decrypted DEK to decrypt the payload contained in the message body.

This decryption process is performed in the deserializer. The deserializer is implemented as a wrapper around the existing deserialization logic. On receiving a message, it first decrypts the DEK using the header metadata, uses that DEK to decrypt the payload, and then performs deserialization.

Multiple producers may use different DEKs. To optimize, consumers cache pairs of encrypted DEK and decrypted DEK so that when the same encrypted DEK is received again, the consumer can skip the decryption step. This strategy reduces the resources spent on DEK decryption and improves consumer efficiency.

Key management service (KMS)

Creation, distribution, and rotation of KEKs are all managed through the key management service (KMS). The topic owner registers an asymmetric key pair in the KMS; producers fetch the public key and authorized consumers fetch the private key. When a new consumer is added, it requests access to the private key from the KMS and, after the topic owner approves, gains decryption rights.

Optimization strategies for large-scale service

After finishing design and implementation, we faced three core challenges when applying this to the live LINE service: increased message size, zero-downtime migration, and key rotation in production.

Minimizing message size: introducing a shared KEK

The biggest challenge when applying E2EE in production was the problem of increased message size. Kafka topics can have many consumers that continue to grow. If each consumer is given a unique key, metadata accumulates in the message header proportional to the number of consumers, and message size grows accordingly. Increased message size causes severe system-wide performance degradation because it reduces the number of records per batch and increases network bandwidth, CPU, and memory consumption.

In the very high traffic environment of up to 1,000,000 messages per second mentioned earlier, using different KEKs per consumer causes header sizes to grow rapidly and degrades overall system performance. Adding a new consumer also appends metadata to the header, making this approach hard to sustain long term.

To solve this problem fundamentally, we adopted a model where multiple consumers share a single KEK from the start. Using a shared KEK keeps the message header to a single metadata entry regardless of the number of consumers. The shared KEK is a trade-off that favors stable message size over per-consumer key isolation, and we mitigate its risks with KMS authorization and periodic key rotation.

• Key distribution: The topic owner creates an asymmetric key pair and registers it in the KMS. Producers fetch the public key and authorized consumers fetch the private key from the KMS. Responsibility for key creation and distribution is concentrated on the topic owner.
• Access control: Consumers must complete an authorization process to fetch the private key from the KMS. This access control acts as the security boundary for encrypted data.
• Key rotation: To mitigate the risks of the shared key structure, we enforce periodic key rotation (the zero-downtime rotation mechanism is described later).

Zero-downtime migration

Plaintext fallback

The biggest risks when introducing encryption are data loss and service downtime. Because we cannot switch all producers and consumers at the same time, we must reliably support a transitional phase in which encrypted and plaintext messages coexist. To achieve this, consumers' deserializers detect whether header metadata is present and choose how to operate accordingly.

• Header present: If a header exists, treat the message as encrypted, extract the encryption information from the header, decrypt, and then deserialize.
• No header: If there is no header, treat the message as plaintext, skip the decryption process, and perform deserialization as before.

Thanks to this plaintext fallback structure, we were able to perform migration in a safe sequence as follows.

1. Deploy consumers first: Deploy consumers that include the fallback logic so they can handle any message format.
2. Enable producer encryption: Once all consumers are ready, enable the producers' encryption feature.
3. Verify and complete: Use monitoring metrics to confirm that the proportion of plaintext messages reaches 0% and then conclude the migration.

Gradual rollout

Even if consumers are ready, immediately raising the producers' encryption rate to 100% is risky. To prepare for unexpected performance degradation or encryption/decryption errors, we adopted a dynamic configuration-based gradual rollout strategy.

• Staged increase: Initially set the encryption rate to a minimum (for example, 1%) so that encryption applies to only a tiny portion of traffic.
• Metric-driven decisions: Observe real-time metrics such as CPU load, processing latency, and error rate, and increase the rate stepwise to 10%, 50%, and 100%.
• Immediate rollback: If anomalies are detected, reduce the encryption rate to 0% via configuration to roll back the system.

Operational stability: zero-downtime key rotation mechanism

Because shared KEK means multiple consumers use the same key, a key leak would allow an unauthorized party to decrypt past and future messages. To address this, we defined periodic key rotation as a security requirement. Rotation must proceed without service downtime, so we implemented a mechanism in which the old key and the new key coexist during the transition period.

Zero-downtime rotation mechanism

The producer's interceptor polls the KMS public key list at a configured interval. When a change is detected, the current DEK is encrypted with each registered public key and included in the message header. During the rotation transition, the header therefore contains the DEK encrypted with the old KEK and the DEK encrypted with the new KEK side by side.

Consumers also poll the KMS at intervals and automatically pick up new private keys. A consumer examines the KEK ID included in each header entry and selects only the entry that matches its key to decrypt. This allows consumers using the old key and consumers using the new key to process the same message during the transition period.

Rotation procedure

1. Register new KEK: Register a new KEK version in the KMS while the old version remains active.
2. Confirm producer adoption: About one minute later, confirm via metrics that producers are reflecting the new KEK. Both old and new KEKs should appear in headers. From this point, headers contain two encrypted DEKs.
3. Confirm consumer adoption: About one minute later, confirm via metrics that consumers are decrypting with the new KEK.
4. Deactivate old KEK: Only after confirming consumer adoption in step 3, deactivate the old version in the KMS. Proceeding earlier may cause consumer decryption failures, so perform deactivation only after confirmation.
5. Complete rotation monitoring: After verifying that producers use only the new KEK, complete the rotation.

Validation and deployment

Performance tests (benchmark)

Before deploying encryption, we quantitatively measured its impact on production topics instead of relying on theoretical estimates. We reproduced production traffic in a test environment and ran benchmark tests.

Test design

• Environment: We built a test environment using the same server hardware as production and created test-only topics so we would not affect existing topics.
• Load reproduction: We used load testing tools to reproduce the maximum requests per second (RPS) and maximum message sizes measured in production.
• Measurement method: We processed plaintext and encrypted messages for 10 minutes each, measured CPU usage per instance, and computed the difference as encryption overhead.
• Consumer concurrency: We matched partition and thread counts between test and production environments so concurrency scaled the same way as in production.

Test results

Across all measurements, CPU increase per instance was below 1%. Even for the highest RPS (up to 5,000 requests per second) and message sizes of tens of kilobytes, producer and consumer CPU increased by about 1%. Based on these results, we concluded that encryption would not require adding more servers.

Production deployment

After validating safety with benchmarks, we rolled out encryption gradually and raised the encryption rate to 100% over about two weeks.

• Operational alerts: During the entire rollout period, there were zero encryption/decryption-related incidents.
• Real load: Real-time CPU increase in production also stayed within the predicted 1% or less.

Using this approach, we applied E2EE across Kafka without service disruption or additional infrastructure cost.

Conclusion

This article went into how we designed and applied E2EE for Kafka.

The Kafka client E2EE system we built raised the data security level of the LINE app. By ensuring data confidentiality throughout message creation, transmission, and storage, we provided a stronger foundation for users worldwide to have greater confidence in using the LINE app. LY Corporation's technical efforts to build a safer and more reliable messaging platform will continue.