Bias in ML

One day perhaps we can teach machines to avoid bias but maybe just maybe we need to understand how to teach humans the same first.

It shouldn’t be a news flash that bias people “train” bias into computers just like we train bias into our children. We will one day realize we have no other choice but hard continuous work to eliminate bias.

Phishing from someone else’s container ship.

This is a theoretical attack abusing a compromised kubectl certificate pair and exposed K8s api to deploy a phishing site transparently on your targets infrastructure. This is a difficult attack to pull off and required existing compromised administrative access to the k8s cluster. A privileged insider, or compromised cert based authentication credential can be used.

  • Target which is one of my test domains.
  • Desired outcome detect an attempt to intercept admin login for a wordpress site we will utilize a fake email alert informing the administrator a critical update must be applied.
  • We will deploy the site hidden behind the targets existing ingress controller, this allows us utilize the customers own domain and certificates eliminating detection by domain name (typo squatting etc) and certificate transparency reporting monitoring.

Phase one: Recon

Using kubectl identify name spaces and find the ingress controller used for the site you intended to compromise. For the purposes of my poc my target used a very obvious “wordpress” namespace.

kubectl -n wordpress get ing


site 8020h

Phase two: deploy gophish

I’m not going to go into details on deploying gophish and setting up or sending the phishing emails. Thats beyond the scope of the blog post, I’m here to help the blue team so lets get on to detection.

The following manifest hides the gophish instance on a path under the main site url. Of note in this case /wplogin.cgi” is the real site while /wplogin is where we are credential harvesting.

apiVersion: extensions/v1beta1
kind: Ingress
name: “site”
namespace: wordpress
annotations: alb internet-facing Environment=dev,Team=test arn:aws:acm:us-east-1:174701313045:certificate/15d484c8-ca0c-4194-a4ef-f38a43b7b977 ‘[{“HTTP”: 80}, {“HTTPS”:443}]’ ‘{“Type”: “redirect”, “RedirectConfig”: { “Protocol”: “HTTPS”, “Port”: “443”, “StatusCode”: “HTTP_301”}}’
– host:
– path: /wplogin
serviceName: gophish
servicePort: 80
– path: /
serviceName: wordpress
servicePort: 80

Phase three: Detecting what we did.

Using the K8S events and meta data onboard using Splunk Connect for K8S we have some solid places we can monitor for abuse. Side note don’t get hung up on “gophish” this is a hammer type tool your opponent may be much more subtle

  • Modification to an ingress in my case AWS ALB, ingress records for most will not change often when they are changed an associated approved deployment should also exist.

“” sourcetype=”kube:container:alb-ingress-controller” modifying

  • New Image, New Image Repository, New Registry, maintain a list of approved images and registries alert when an image is used not on the pre-defined list, this . may be noisy in dev clusters for no prod clusters reporting may be better than alerting.

index=em_meta “spec.containers{}.image”=”*”


“Safely” Exposing Splunk S2S to the internet

Splunk has a great token based auth solution for its S2S protocol it was added several versions back. Inputs have both just worked and remained unchanged for so long many administrators have not noticed the feature. This allows you to safely expose indexers or heavy forwarders so that UFs on the internet can forward data back in without VPN. This is super critical for Splunk endpoints that don’t require a connection to the corporate network via VPN constantly

When a Splunk input is exposed to the internet there is a risk of resource exhaustion dos. A simple type of attack where junk data or “well formed but misleading” data is feed to Splunk until all CPU/memory/disk is consumed.

Once this feature is enabled all UF/HF clients must supply a token to be allowed to connect if your adding this feature to a running Splunk deployment be ready to push the outputs.conf update and inputs.conf updates in close succession to prevent a data flow break.

Update your inputs.conf as follows, note you can use multiple tokens just like HEC so you can mitigate the number of tokens that need to be replaced if a stolen token is used in a DOS attack

# Access control settings.
[splunktcptoken://<token name>]
* Use this stanza to specify forwarders from which to accept data.
* You must configure a token on the receiver, then configure the same
  token on forwarders.
* The receiver discards data from forwarders that do not have the
  token configured.
* This setting is enabled for all receiving ports.
* This setting is optional.

token = <string>
* token should match regex [A-Za-Z0-9\-]+ and with a min length of 12

Update outputs.conf  use the token value of choice from inputs.conf

token = <string>
* The access token for receiving data.
* If you configured an access token for receiving data from a forwarder,
  Splunk software populates that token here.
* If you configured a receiver with an access token and that token is not
  specified here, the receiver rejects all data sent to it.
* This setting is optional.
* No default.