A customer has a use case for a ksqlDB persistent query. You need to make sure that duplicate messages are not processed and messages are not…

The requirement to prevent both duplicate message processing and skipping messages is the definition of exactly-once processing semantics. In ksqlDB, which is built on Kafka Streams, this guarantee is enabled by setting the processing.guarantee property to exactlyonce. This configuration ensures that each input record is processed once and only once, even in the event of failures, by leveraging Kafka's transactional capabilities to atomically write output records and commit consumer offsets.

B. ksql.streams auto offset.reset=earliest: This property only defines where a consumer starts reading from a topic partition when no committed offset exists. It does not guarantee exactly-once processing during ongoing operations.

C. ksql.streams auto.offset.reset=latest: This setting would cause the consumer to start at the end of the log, explicitly skipping messages that arrived while the application was down, directly violating the requirement.

D. ksql.fail.on.production.error=false: This property (or its equivalent) controls error handling when writing to an output topic. Setting it to false would cause the application to continue on production errors, potentially skipping messages that failed to be written.

1. Confluent Documentation, "ksqlDB Configuration Parameter Reference": This document lists the server configuration parameters. The ksql.streams.processing.guarantee property is described as controlling the processing semantics.

Reference: Confluent Documentation > ksqlDB > Reference > Configuration Parameter Reference > ksql.streams.processing.guarantee. It states, "The processing guarantee that should be used. Possible values are atleastonce (default) and exactlyonce."

2. Confluent Documentation, "Process events with exactly-once semantics": This guide explains the concept and implementation of exactly-once in ksqlDB.

Reference: Confluent Documentation > ksqlDB > Develop > Process events with exactly-once semantics. It explicitly states: "To get exactly-once processing, you must set the processing.guarantee property in your ksqlDB server properties file to exactlyonce."

3. Apache Kafka Documentation, "Kafka Streams Configs": As ksqlDB uses Kafka Streams, the underlying stream configuration is relevant.

Reference: Apache Kafka Documentation > Kafka Streams > Developer Guide > Kafka Streams Configs > processing.guarantee. The documentation details that setting this to exactlyonce enables the exactly-once guarantee.