CCAAK Practice Questions

The Controller is a designated broker in a Kafka cluster responsible for managing the state of partitions and replicas. When a broker fails, the Controller detects this failure (historically via ZooKeeper, now via the KRaft consensus protocol). For every partition whose leader was on the failed broker, the Controller initiates a leader election. It selects a new leader from the set of in-sync replicas (ISRs) for that partition. This action ensures that leadership for the affected partitions is redistributed among the remaining healthy brokers, maintaining data availability.

A. Wait for a follower to take the lead.

This is incorrect because the process is not passive or follower-initiated. The Controller actively manages and triggers the election process.

C. Become the leader for the topic/partition that needs a leader, pending the broker return in the cluster.

The Controller broker does not centralize leadership for all failed partitions onto itself; its role is to manage and delegate leadership to appropriate replicas.

D. Automatically elect the least loaded broker to become the leader for every orphan's partitions.

Leader election during failure is not based on broker load. The new leader is chosen from the partition's specific list of in-sync replicas (ISRs) to guarantee no data loss.

1. Apache Kafka Documentation, Section 5.2 Replication: "One of the brokers in the cluster also functions as the controller. The controller is responsible for managing the state of partitions and replicas and for performing administrative tasks like reassigning partitions. ... When the controller notices that a broker has shut down, it performs a leader election for all the partitions for which that broker was the leader."

2. Confluent Documentation, "Apache Kafka Internals: Understand the Guts of Kafka": "The controller is a broker that is responsible for maintaining the leader/follower relationship for all the partitions in a Kafka cluster. If a node by some reason fails, the controller will ensure that another replica will take the role of a partition leader."

3. Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A Distributed Messaging System for Log Processing. Proceedings of the NetDB, 1-7. In Section 3.2, "Replication," the paper describes the role of the controller in managing broker failures: "The controller detects failures at the broker level and is responsible for electing a new leader for all affected partitions on a failed broker." (DOI not available for this specific workshop paper, but it is the foundational academic source).

The core principle to avoid a single point of failure for data in Kafka is replication. The topic's replication-factor determines how many copies of each partition's data are stored across different brokers in the cluster. Setting the replication factor to 3 (a common best practice) ensures that for each partition, there is one leader and two follower replicas on three separate brokers. If one broker fails, the data remains available on the other two, and a new leader can be elected from the in-sync followers, thus preventing data loss and service interruption.

A. Setting min.insync.replicas to 1 weakens durability. It allows a producer's write to be considered successful after only the leader broker has received it, risking data loss if that broker fails before replication occurs.

B. Partitions are for parallelism and scaling, not fault tolerance. A topic with many partitions but a replication factor of 1 still has a single point of failure for the data in each partition.

C. linger.ms is a producer-side configuration that controls batching to optimize throughput. It has no impact on how data is stored on the brokers for fault tolerance.

1. Confluent Documentation, "Kafka Topics": "Replication is at the core of Kafka's fault tolerance. The replication factor specifies how many copies of the data are made... For a replication factor of N, Kafka can tolerate up to N-1 server failures without losing any committed messages. A typical replication factor is 3."

Source: Confluent Documentation, "Manage Topics" section.

2. Apache Kafka Documentation, "3.4 Replication": "Replication is at the heart of Kafka's fault-tolerance... We recommend using a replication factor of 3, which allows for tolerating 2 broker failures."

Source: Apache Kafka 3.7 Documentation, Section 3.4 "Replication".

3. Confluent Documentation, "Producer Configs": The documentation for min.insync.replicas states, "When set to 1, the producer will get an acknowledgement as soon as the leader has written the message to its local log." This highlights the risk associated with this setting, which contradicts the goal of high availability.

Source: Confluent Documentation, "Producer Configuration Settings".

The acks (acknowledgments) configuration determines the durability guarantee for messages sent by a producer. To achieve the "utmost throughput," latency must be minimized. Setting acks=0 instructs the producer not to wait for any acknowledgment from the broker after sending a message. This "fire and forget" approach eliminates the round-trip time of waiting for a response, allowing the producer to send messages as fast as the network buffer allows. This configuration is suitable for use cases like high-volume log ingestion where losing a small fraction of messages is an acceptable trade-off for maximizing ingestion speed.

B. acks=1: This setting requires acknowledgment from the partition leader, which introduces latency and thus reduces throughput compared to acks=0.

C. acks=all: This provides the strongest durability by waiting for all in-sync replicas to acknowledge the write, but it incurs the highest latency and lowest throughput.

D. acks=undefined: This is not a valid configuration value for the acks parameter in Kafka.

1. Confluent Documentation, Producer Configs: The official documentation for the acks parameter states: "If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent." This describes the mechanism that yields the highest throughput.

Source: Confluent Documentation, "Producer Configs", section acks.

2. Narkhede, N., Shapira, G., & Palino, T. (2021). Kafka: The Definitive Guide (2nd ed.). O'Reilly Media. In Chapter 4, "Kafka Producers: Writing Messages to Kafka," the section "Configuring Producers" explains the acks setting. It explicitly notes, "If you choose acks=0, the producer will not wait for a reply from the broker... This is the fastest way to send messages, but you will lose messages if the server fails."

Source: Chapter 4, Section: "Configuring Producers", Subsection: "acks".

3. Confluent Developer Courseware, Apache Kafka 101: The course material on Producers explains the trade-offs: "acks=0... This is the fastest option but offers the weakest durability guarantees... This setting is often used for data that can be lost, such as metrics collection." This directly aligns with the high-throughput log ingestion scenario.

Source: Confluent Developer, "Apache Kafka 101", Module: "Producers", Lesson: "Producer Guarantees and Configuration".

When a broker that is a leader for one or more partitions fails, the Kafka Controller is responsible for managing the failover process. The Controller first selects a new leader from the list of in-sync replicas (ISR) for each affected partition. It then performs two critical actions: it persists this new leadership information and the updated ISR list to ZooKeeper to ensure the state change is durable. Concurrently, it sends a LeaderAndIsr request to all live brokers in the cluster, informing them of the new leader so they can update their metadata cache and route client requests correctly.

A. The Controller waits for a new leader to be nominated by ZooKeeper.

This is incorrect. The Controller is the entity that actively elects the new leader; it does not wait for ZooKeeper to perform this action.

D. The Controller sends the new leader and ISR list changes to all producers and consumers.

This is incorrect. The Controller communicates state changes to other brokers. Clients (producers and consumers) discover the new leader by sending metadata requests to any broker.

1. Confluent Documentation, Apache Kafka® Internals, "The Controller": "The controller is responsible for electing a new leader... It does so by reading the current list of in-sync replicas from ZooKeeper... The controller then writes the identity of the new leader to ZooKeeper. After this, the controller sends a LeaderAndIsr request to all brokers that contain a replica of the partition (both followers and the new leader)." This directly supports that the Controller writes to ZooKeeper (B) and sends updates to all brokers (C).

2. Apache Kafka Documentation, 3.7, Section 4.2 "Replication": "When the controller notices that a broker has left the cluster... it initiates a leader election for all the partitions that had that broker as their leader... The controller writes the new leader to ZooKeeper and sends a message to all the followers. The new leader begins serving requests from clients and the followers begin to pull from the new leader." This confirms the Controller's role in writing to ZooKeeper and notifying other brokers.

In traditional Apache Kafka deployments, which rely on ZooKeeper for cluster coordination, Access Control Lists (ACLs) are stored as metadata within ZooKeeper. ZooKeeper serves as the authoritative source of truth for the cluster's configuration, including security policies. Kafka brokers connect to ZooKeeper, read the ACLs, and cache them in memory to enforce authorization for client requests efficiently. While newer Kafka versions can run in KRaft mode without ZooKeeper, storing ACLs in ZooKeeper is the standard mechanism in the widely deployed architecture covered by the CCAAK exam.

A. Broker: In a ZooKeeper-based cluster, brokers only cache ACLs for performance; they do not persistently store them. ZooKeeper is the source of truth.

C. Schema Registry: This is a separate Confluent component for managing data schemas and has its own security model, but it does not store Kafka broker ACLs.

D. Connect: This is a framework for integrating Kafka with external systems and is not responsible for storing core cluster security metadata like ACLs.

1. Confluent Documentation, "Authorization using ACLs": "The out-of-box authorizer implementation that ships with Kafka stores ACLs in ZooKeeper. The Kafka brokers read the ACLs from ZooKeeper and cache them." (Reference: Confluent Documentation > Security > Authorization (ACLs) > Adding, Removing, and Listing ACLs section).

2. Apache Kafka Documentation, "Security - Authorization": "Kafka ships with a pluggable Authorizer. The default implementation is AclAuthorizer (kafka.security.authorizer.AclAuthorizer)... It stores ACLs in ZooKeeper." (Reference: Apache Kafka Documentation > Version 3.7 > Security > Section 3.3 Authorization).

3. Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A Distributed Messaging System for Log Processing. While this foundational paper predates the full ACL implementation, it establishes ZooKeeper's central role for "managing consumption and cluster metadata" which is the architectural pattern that storing ACLs follows. (Reference: Proceedings of the NetDB'11 Workshop, Section 2.3. ZooKeeper Coordination).

Kafka Connect is a framework included in Apache Kafka specifically designed for scalably and reliably streaming data between Kafka and other data systems. It uses pre-built or custom connectors to move large data sets. Source connectors ingest entire databases or collect metrics from application servers into Kafka topics, while sink connectors deliver data from Kafka topics into secondary indexes like Elasticsearch or batch systems such as Hadoop for offline analysis. The framework manages offset commits, scaling, and fault tolerance, simplifying the development of data integration pipelines.

B. Kafka Streams is a client library for building stream processing applications and microservices that read from and write to Kafka topics, not for integrating with external systems.

C. Kafka Schema Registry is a centralized service for managing and validating schemas for Kafka messages, ensuring data compatibility, not for data movement.

D. Kafka REST Proxy provides a RESTful (HTTP) interface to a Kafka cluster, which is an alternative to the native protocol but not the primary tool for scalable data integration.

1. Confluent Documentation: "Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka® and other systems. It is a framework that comes with Apache Kafka that makes it simple to write and run connectors that move data into and out of Kafka."

Source: Confluent Documentation, "Kafka Connect Overview", Introduction section.

2. Apache Kafka Documentation: "Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka."

Source: Apache Kafka 3.7 Documentation, Section 8.1 "Kafka Connect".

3. Gwen Shapira, Todd Palino, Rajini Sivaram (2021). Kafka: The Definitive Guide, 2nd Edition. O'Reilly Media, Inc. "Kafka Connect is a framework for building and running reusable connectors that import data from or export data to external systems."

Source: Chapter 10, "Kafka Connect", Introduction.

Confluent Schema Registry provides a centralized repository for managing and validating schemas for Kafka messages. It natively supports three schema formats as first-class citizens: Avro, Protobuf, and JSON Schema. These formats enable robust schema evolution, ensuring that producers and consumers can exchange data compatibly over time. The registry's RESTful interface allows for the storage, retrieval, and versioning of schemas defined in any of these three supported types, which are integrated with corresponding serializers and deserializers for client applications.

B. gRPC: gRPC is a remote procedure call (RPC) framework. While it uses Protobuf for defining service interfaces and message structures, gRPC itself is not a schema format stored in Schema Registry.

D. Thrift: Apache Thrift is a serialization and RPC framework similar to Protobuf. However, it is not a natively supported schema format within the Confluent Schema Registry.

1. Confluent Documentation, "Schema Registry Overview": "Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving your Avro®, JSON Schema, and Protobuf schemas."

Source: Confluent, Inc., Schema Registry Overview, Confluent Platform 7.6 Documentation. Retrieved from https://docs.confluent.io/platform/current/schema-registry/index.html, Section: "Schema Registry Overview".

2. Confluent Documentation, "Formats, Serializers, and Deserializers": "Confluent Schema Registry supports Avro, Protobuf, and JSON Schema as first-class citizens. Schema Registry enables you to store and retrieve schemas of these formats, and use them with the Avro, Protobuf, and JSON Schema serializers and deserializers."

Source: Confluent, Inc., Formats, Serializers, and Deserializers, Confluent Platform 7.6 Documentation. Retrieved from https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#formats-serializers-and-deserializers, Introduction paragraph.

Sizing a ksqlDB cluster requires evaluating the workload's demands on compute, memory, and network resources. The three primary factors are:

1. Number of Queries: Each persistent query runs as a dedicated Kafka Streams application. More queries consume more CPU and memory. The complexity of these queries (e.g., joins, aggregations) also significantly impacts resource utilization.

2. Number of Partitions: ksqlDB achieves parallelism by assigning topic partitions to processing tasks. The number of partitions in the input topics dictates the maximum level of parallelism, directly influencing how many ksqlDB nodes can share the processing load and thus determining the required cluster size for a given throughput.

3. Data Schema: The complexity and size of the message schema affect the CPU and memory needed for serialization/deserialization and processing each message. Complex, nested schemas require more resources than simple, flat ones.

D. Data Encryption: While TLS encryption adds some CPU overhead, it is a baseline operational cost and not considered a primary sizing dimension compared to the core workload characteristics like query volume or data complexity.

E. Topic Data Retention: This setting primarily affects storage on the Kafka brokers, not the ksqlDB cluster's compute resources. ksqlDB's state size is determined by query logic (e.g., window size), not topic retention.

1. Confluent Documentation, "Capacity Sizing for ksqlDB": This document explicitly outlines the key factors for sizing. It states, "The primary factors that affect the performance of your ksqlDB applications are the number of events to be processed and the complexity of the processing." This supports the importance of Data Schema (A) and Number of Queries (B).

Reference: Confluent, Inc. (2023). Capacity Sizing for ksqlDB. Confluent Documentation. Retrieved from https://docs.confluent.io/platform/current/ksql/capacity-sizing.html (See section "Factors that affect performance").

2. Confluent Documentation, "Scale Your ksqlDB Application": This guide details the relationship between scaling, servers, and partitions. It clarifies, "The maximum parallelism of a ksqlDB application is bounded by the number of partitions of its input topics." This directly supports the significance of the Number of Partitions (C) in determining how a cluster can be scaled out.

Reference: Confluent, Inc. (2023). Scale Your ksqlDB Application. Confluent Documentation. Retrieved from https://docs.confluent.io/platform/current/ksql/scale-ksqldb.html (See section "Scaling ksqlDB").

By default, Apache Kafka is configured for a "trusted network" environment, prioritizing ease of use and initial setup over security. The default network listener uses the PLAINTEXT security protocol, which transmits data without any encryption. This default configuration does not implement any mechanism to verify the identity of clients connecting to the brokers, meaning there is no authentication. Consequently, as there is no authenticated principal (user), there is no basis for performing authorization checks against access control lists (ACLs). All security features—encryption (SSL/TLS), authentication (SASL or SSL), and authorization (ACLs)—must be explicitly configured by the administrator.

B: Encryption is not enabled by default. The default security protocol is PLAINTEXT, which sends data in the clear.

C: Authentication is not enabled by default. Clients can connect anonymously without providing credentials.

D: Both encryption and authentication are disabled by default and require explicit configuration to be enabled.

1. Confluent Documentation, "Encryption and Authentication using SSL for Apache Kafka®": In the "Prerequisites" section, it states, "By default, Kafka communication is unencrypted. To encrypt communication, you need to configure SSL for encryption on all the Confluent Platform components." This confirms encryption is off by default.

Source: Confluent Documentation, Confluent Platform v7.6, Security > On-Premises Deployments > Encryption and Authentication using SSL.

2. Apache Kafka Documentation, "3. Security": The security overview section introduces encryption, authentication, and authorization as features that need to be configured. For authorization, it specifies, "By default, when no authorizer.class.name is configured, Kafka authorizes all actions for all principals." This confirms authorization is effectively off by default.

Source: Apache Kafka 3.7 Documentation, Section 3.4, "Authorization and ACLs".

3. Confluent Documentation, "Authentication using SASL for Apache Kafka®": This guide details the steps required to enable authentication. The existence of a multi-step configuration process for enabling SASL confirms that it is not a default feature. The introduction states you must "configure Kafka brokers for SASL authentication."

Source: Confluent Documentation, Confluent Platform v7.6, Security > On-Premises Deployments > Authentication using SASL.

The requirement to prevent both duplicate message processing and skipping messages is the definition of exactly-once processing semantics. In ksqlDB, which is built on Kafka Streams, this guarantee is enabled by setting the processing.guarantee property to exactlyonce. This configuration ensures that each input record is processed once and only once, even in the event of failures, by leveraging Kafka's transactional capabilities to atomically write output records and commit consumer offsets.

B. ksql.streams auto offset.reset=earliest: This property only defines where a consumer starts reading from a topic partition when no committed offset exists. It does not guarantee exactly-once processing during ongoing operations.

C. ksql.streams auto.offset.reset=latest: This setting would cause the consumer to start at the end of the log, explicitly skipping messages that arrived while the application was down, directly violating the requirement.

D. ksql.fail.on.production.error=false: This property (or its equivalent) controls error handling when writing to an output topic. Setting it to false would cause the application to continue on production errors, potentially skipping messages that failed to be written.

1. Confluent Documentation, "ksqlDB Configuration Parameter Reference": This document lists the server configuration parameters. The ksql.streams.processing.guarantee property is described as controlling the processing semantics.

Reference: Confluent Documentation > ksqlDB > Reference > Configuration Parameter Reference > ksql.streams.processing.guarantee. It states, "The processing guarantee that should be used. Possible values are atleastonce (default) and exactlyonce."

2. Confluent Documentation, "Process events with exactly-once semantics": This guide explains the concept and implementation of exactly-once in ksqlDB.

Reference: Confluent Documentation > ksqlDB > Develop > Process events with exactly-once semantics. It explicitly states: "To get exactly-once processing, you must set the processing.guarantee property in your ksqlDB server properties file to exactlyonce."

3. Apache Kafka Documentation, "Kafka Streams Configs": As ksqlDB uses Kafka Streams, the underlying stream configuration is relevant.

Reference: Apache Kafka Documentation > Kafka Streams > Developer Guide > Kafka Streams Configs > processing.guarantee. The documentation details that setting this to exactlyonce enables the exactly-once guarantee.