High-Performance Financial Data Streaming Pipeline

Design a cloud-native streaming data pipeline for processing high-frequency financial market data using Apache Kafka, Kubernetes, and Python. Create custom Kafka producers and consumers that handle real-time market feed ingestion, implement robust error handling and exactly-once processing semantics. Develop a scalable Kubernetes deployment that can dynamically adjust consumer group partitions based on incoming data volume. Include comprehensive monitoring with distributed tracing using OpenTelemetry and implement secure, encrypted data transmission.