Seth Vargo's presentation at Strange Loop 2019 has about 10 minutes of excellent security fundamentals followed by a deep dive into integrating kubernetes secrets with hashicorp vault. Seth also gave the talk at Velocity 2019 where slides for the talk are available. youtube 
pdf
YOUTUBE 7jSfJombUeY Seth Vargo, Better Kubernetes Secrets, Strange Loop 2019
Application level secrets, API keys.
Attractive target for hackers. Credentials usually have overly broad permissions.
Operate with the assumption that your credentials will be compromised.
Four ways to protect secrets in order of importance.
Audit—log every login, every access, every use of secrets; consider anomaly detection system.
Encrypt—encrypt at rest and in transit.
Rotate—need to rotate all the credentials frequently.
Isolate—separate where secrets are stored from where they are accessed.
Apply app-layer encryption at the earliest possible step in the lifecycle of the service. Choose unique encryption keys for each service. Imagine safety deposit boxes at a bank. There's one key guarding the vault, and then a pair of keys—one for the bank, another for the customer—needed to open the box.
Envelope Encryption 15m37s pdf p.18
Edit some graphviz: edotor
# Place the cursor inside "graph" to get some refactoring options digraph { rankdir=RL node [shape=box style="filled,rounded" colorscheme=paired10] Data [color=2] DEK [label="Data\nEncryption\nKey" color=6] KEK [label="Key\nEncryption\nKey" color=4] Storage [ shape=record color="1:5" label="Storage|Encrypted Data|Encrypted DEK"]; EData [label="Encrypted\nData" color=1] EDEK [label="Encrypted\nDEK" color=5] KEK -> DEK [label="encrypts"] {rank=same DEK -> EDEK [label="yields"] } DEK -> Data [label="encrypts"] {rank=same Data -> EData [label="yields"] } EDEK -> EData [label="combined" dir=back] Storage -> EDEK [label="saved in" dir=back] }