Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does not cover the one that keeps showing up in large enterprises: the Kafka cluster is in AWS, the compute is Azure Databricks, IAM is off the table because the identity system is a corporate PKI, and the traffic never touches the public internet. This walks through that setup end to end. Certificates from a private CA, a private network path between ...