ConsumerAndProducerGroup

class openmsistream.kafka_wrapper.ConsumerAndProducerGroup(config_path, consumer_topic_name, *, consumer_group_id='create_new', kafkacrypto=None, treat_undecryptable_as_plaintext=False, **kwargs)

Bases: LogOwner

Class for working with a group of Consumers and Producers sharing a single kafkacrypto.KafkaCrypto instance

Parameters:

  • config_path (pathlib.Path) – Path to the config file that should be used to define Consumers/Producers in the group

  • consumer_topic_name (str) – The name of the topic to which the Consumers should be subscribed

  • consumer_group_id (str, optional) – The ID string that should be used for each Consumer in the group. “create_new” (the defaults) will create a new UID to use.

  • kafkacrypto (OpenMSIStreamKafkaCrypto, optional) – The OpenMSIStreamKafkaCrypto object that should be used to instantiate Consumers. Only needed if a single specific OpenMSIStreamKafkaCrypto instance should be shared.

  • treat_undecryptable_as_plaintext (boolean, optional) – If True, the KafkaCrypto Deserializers will immediately return any keys/values that are not possibly decryptable as binary data. This allows faster handling of messages that will never be decryptable, such as when enabling or disabling encryption across a platform, or when unencrypted messages are mixed in a topic with encrypted messages.

property consumer_topic_name

Name of the topic to which the consumers are subscribed

property consumer_group_id

Group ID of all consumers in the group

get_new_subscribed_consumer(*, restart_at_beginning=False, **kwargs)

Return a new Consumer, subscribed to the topic and with the shared group ID. Call this function from a child thread to get thread-independent Consumers.

Note: This function just creates and subscribes the Consumer. Polling it, closing it, and everything else must be handled by whatever calls this function.

Parameters:

  • restart_at_beginning (bool, optional) – if True, the new Consumer will start reading partitions from the earliest messages available, regardless of Consumer group ID and auto.offset.reset values. Useful when re-reading messages.

  • kwargs (dict) – other keyword arguments are passed to the OpenMSIStreamConsumer constructor method

Returns:

a Consumer created using the configs set in the constructor/from kwargs, subscribed to the topic

Return type:

OpenMSIStreamConsumer

get_new_producer()

Return a new OpenMSIStreamProducer object. Call this function from a child thread to get thread-independent Producers. Note: this function just creates the Producer; closing it etc. must be handled by whatever calls this function.

Returns:

a Producer created using the config set in the constructor

Return type:

OpenMSIStreamProducer

close()

Wrapper around kafkacrypto.KafkaCrypto.close().

classmethod get_command_line_arguments()

Anything extending this class should be able to access the “treat_undecryptable_as_plaintext” flag

classmethod get_init_args_kwargs(parsed_args)

Get the list of init arguments and the dictionary of init keyword arguments for this class given a namespace of, for example, parsed arguments.

Parameters:

parsed_args (argparse.Namespace) – A namespace containing entries needed to determine the init args and kwargs for this class

Returns:

A list of init args

Returns:

A dictionary of init kwargs