Tech and more stuff

open source, politics and electronic music

    • Home
    • Acerca de este blog – About this blog.
calvarado04

calvarado04 / November 8, 2021

Deploy an Elasticsearch cluster for Kubernetes (ECK) on Google Compute Platform (GCP on GKE) with Terraform – Part I

This will be a very technical post but I think that is gonna be also quite interesting if you are working with cloud technologies.

Elasticsearch is a pretty nice technology widely used on big data stuff, analysis and so on. However, this tool is heavy and little bit difficult to deploy and maintain on healthy status.

I'm working a lot with Google Compute Platform (GCP) that's why I decided to include this part as well.

First things first

If you don't have a GCP account, is pretty straightforward to get one, even with some free usage, Google will give you 300 dollars to spend on it... by previous registration with your credit card 😉 go ahead and do it: https://console.cloud.google.com

Also download the gcloud CLI: https://cloud.google.com/sdk/docs/install

We will be using the project called GKE Terraform project as you can check below:

Get access to your gcloud project on the CLI and perform the browser steps needed to achieve it:

$ gcloud auth login

Get access to your project:

Let's create an empty VPC to simulate one environment with previous stuff deployed on it, like other instances and so on.

Well, at this point we have the very basic infrastructure to start using Terraform.

Infrastructure as Code, what does that mean?

Terraform is the leading tool to deploy infrastructure on this way, you can define a very complex set of infrastructure with code functions and treating them like objects and variables.

The GKE Terraform project is available here:

https://github.com/calvarado2004/terraform-gke

Please note that the size of the nodes is huge, you can go ahead and delete some of those pools of nodes and customize the CPU's and memory according to your needs and budget, I will do that, of course. You can check here another branch with smaller nodes: https://github.com/calvarado2004/terraform-gke/tree/resize-to-small

ECK can be deployed on a single node, but the minimal enterprise configuration should have:

  • One Kibana node
  • One Coordinator node
  • One Master node
  • Two Data nodes

This deployment is creating a pool of nodes for each type of node, in order to enable the autoresizing on further moments of the infrastructure lifecycle. That could give you an idea of the complexity that you can handle easily with Terraform.

Kubernetes have two internal layers of networking. We will be using the following three CIDRs:

  • 170.35.0.0/24 for our GCP VPC, the most external face.
  • 10.99.240.0/20 for our Kubernetes services.
  • 10.96.0.0/14 for our Kubernetes Pods.

You can install Terraform if you have Ubuntu using this way:

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install terraform

Otherwise, check how to install it on your machine:

https://www.terraform.io/downloads.html

This the content of the file gke.tf

variable "gke_username" {
  default     = ""
  description = "gke username"
}

variable "gke_password" {
  default     = ""
  description = "gke password"
}

variable "cluster_name" {
  default = "gke-cluster"
  description = "cluster name"
}

variable "zone" {
  default = "us-east1-b"
  description = "cluster zone"
}

#Your pods will have an IP address from this CIDR
variable "cluster_ipv4_cidr" {
  default = "10.96.0.0/14"
  description = "internal cidr for pods"
}

#Your Kubernetes services will have an IP from this range
variable "services_ipv4_cidr_block" {
  default = "10.99.240.0/20"
  description = "nternal range for the kubernetes services"
}

# GKE cluster
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.zone

  remove_default_node_pool = true
  initial_node_count       = 1

  network                  = google_compute_network.vpc-gke.name
  subnetwork               = google_compute_subnetwork.subnet.name
  cluster_ipv4_cidr        = var.cluster_ipv4_cidr
  services_ipv4_cidr_block = var.services_ipv4_cidr_block

  min_master_version = "1.17.13-gke.2001"	

  master_auth {
    username = var.gke_username
    password = var.gke_password

    client_certificate_config {
      issue_client_certificate = false
    }
  }

  cluster_autoscaling {
    enabled = false
  }

}

# Separately Managed Master Pool
resource "google_container_node_pool" "master-pool" {
  name       = "master-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "master_nodes"
    }
    # 6 CPUs, 12GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-6-12288"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-master"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Data Pool
resource "google_container_node_pool" "data-pool" {
  name       = "data-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 2

  autoscaling {
    min_node_count = 2
    max_node_count = 4
  }

  management {
    auto_repair = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "data_nodes"
    }

    # 14 CPUs, 41GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-14-41984"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"

    tags         = ["gke-node", "${var.cluster_name}-data"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Coordinator Pool
resource "google_container_node_pool" "coord-pool" {
  name       = "coord-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "coordinator_nodes"
    }

    # 6 CPUs, 22GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-6-22528"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-coord"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

# Separately Managed Kibana Pool
resource "google_container_node_pool" "kibana-pool" {
  name       = "kibana-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 2
  }

  management {
    auto_repair  = true
    auto_upgrade = false
  }

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
    ]

    labels = {
      es_type = "kibana_nodes"
    }

    # 4 CPUs, 13GB of RAM
    preemptible  = false
    image_type   = "ubuntu_containerd"
    machine_type = "custom-4-13312"
    local_ssd_count = 0
    disk_size_gb    = 50
    disk_type       = "pd-standard"
    tags         = ["gke-node", "${var.cluster_name}-kibana"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

output "kubernetes_cluster_name" {
  value       = google_container_cluster.primary.name
  description = "GKE Cluster Name"
}

And the content of the file vpc.tf

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# VPC
resource "google_compute_network" "vpc-gke" {
  name                    = "${var.cluster_name}-vpc"
  auto_create_subnetworks = "false"
}

# Subnet
resource "google_compute_subnetwork" "subnet" {
  name          = "${var.cluster_name}-subnet"
  region        = var.region
  network       = google_compute_network.vpc-gke.name
  ip_cidr_range = "170.35.0.0/24"

}

#Peering between OLD VMs vpc and GKE K8s vpc
resource "google_compute_network_peering" "to-vms-vpc" {
  name         = "to-vms-vpc-vpc-network"
  network      = google_compute_network.vpc-gke.id
  peer_network = "projects/sigma-scheduler-297405/global/networks/vms-vpc-network"
}

resource "google_compute_network_peering" "to-gke-cluster" {
  name         = "to-gke-cluster-vpc-network"
  network      = "projects/sigma-scheduler-297405/global/networks/vms-vpc-network"
  peer_network = google_compute_network.vpc-gke.id
}

output "region" {
  value       = var.region
  description = "region"
}

#Enable communication from GKE pods to external instances, networks and services outside the Cluster.
resource "google_compute_firewall" "gke-cluster-to-all-vms-on-network" {
  name    = "gke-cluster-k8s-to-all-vms-on-network"
  network = google_compute_network.vpc-portal.id

  allow {
    protocol = "tcp"
  }

  allow {
    protocol = "udp"
  }

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "esp"
  }

  allow {
    protocol = "ah"
  }

  allow {
    protocol = "sctp"
  }

  source_ranges = ["10.96.0.0/14"]
}

Let's deploy this GKE Cluster with Terraform!

Deploy a whole cluster is quite easy:

$ git clone https://github.com/calvarado2004/terraform-gke.git

$ git checkout resize-to-small
Switched to branch 'resize-to-small'
Your branch is up to date with 'origin/resize-to-small'.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v3.49.0...
- Installed hashicorp/google v3.49.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/google: version = "~> 3.49.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform plan -out=gke-cluster.plan

$ terraform apply "gke-cluster.plan"

calvarado04 / May 14, 2021

Deploy a GKE Cluster with Portworx

Let's deploy a nice GKE Cluster with a customized Portworx deployment using security capabilities and encrypted volumes

Get your own GCP account, download gcloud and authenticate on your laptop.

gcloud container clusters create carlos-lab01 \
    --zone us-east1-b \
    --disk-type=pd-ssd \
    --disk-size=50GB \
    --labels=portworx=gke \
    --machine-type=n1-highcpu-8 \
    --num-nodes=5 \
    --image-type ubuntu \
    --scopes compute-rw,storage-ro,cloud-platform \
    --enable-autoscaling --max-nodes=5 --min-nodes=5
    

gcloud container clusters get-credentials carlos-lab01 --zone us-east1-b --project <your-project>

gcloud services enable compute.googleapis.com

Wail until having your cluster available

Then you can install Portworx using the operator.

operator.yaml

# SOURCE: https://install.portworx.com/?comp=pxoperator
apiVersion: v1
kind: ServiceAccount
metadata:
  name: portworx-operator
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: portworx-operator
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: portworx-operator
subjects:
- kind: ServiceAccount
  name: portworx-operator
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: portworx-operator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portworx-operator
  namespace: kube-system
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  selector:
    matchLabels:
      name: portworx-operator
  template:
    metadata:
      labels:
        name: portworx-operator
    spec:
      containers:
      - name: portworx-operator
        imagePullPolicy: Always
        image: portworx/px-operator:1.5.0
        command:
        - /operator
        - --verbose
        - --driver=portworx
        - --leader-elect=true
        env:
        - name: OPERATOR_NAME
          value: portworx-operator
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "name"
                    operator: In
                    values:
                    - portworx-operator
              topologyKey: "kubernetes.io/hostname"
      serviceAccountName: portworx-operator

px-enterprisecluster.yaml

# SOURCE: https://install.portworx.com/?operator=true&mc=false&kbver=1.20.8&b=true&kd=type%3Dpd-standard%2Csize%3D150&csicd=true&mz=5&s=%22type%3Dpd-ssd%2Csize%3D150%22&j=auto&c=px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74&gke=true&stork=true&csi=true&mon=true&st=k8s&promop=true
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
  name: px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74
  namespace: kube-system
  annotations:
    portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=1.20.8&b=true&kd=type%3Dpd-standard%2Csize%3D150&csicd=true&mz=5&s=%22type%3Dpd-ssd%2Csize%3D150%22&j=auto&c=px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74&gke=true&stork=true&csi=true&mon=true&st=k8s&promop=true"
    portworx.io/is-gke: "true"
spec:
  image: portworx/oci-monitor:2.8.0
  imagePullPolicy: Always
  kvdb:
    internal: true
  cloudStorage:
    deviceSpecs:
    - type=pd-ssd,size=200
    journalDeviceSpec: auto
    kvdbDeviceSpec: type=pd-standard,size=50
    maxStorageNodesPerZone: 5
  secretsProvider: k8s
  stork:
    enabled: true
    args:
      webhook-controller: "false"
  autopilot:
    enabled: true
    providers:
    - name: default
      type: prometheus
      params:
        url: http://prometheus:9090
  monitoring:
    telemetry:
      enabled: true
    prometheus:
      enabled: true
      exportMetrics: true
  featureGates:
    CSI: "true"
kubectl create clusterrolebinding myname-cluster-admin-binding \
    --clusterrole=cluster-admin --user=`gcloud info --format='value(config.account)'`

kubectl apply -f operator.yaml

kubectl apply -f px-enterprisecluster.yaml
kubectl get all -n kube-system                                                                                                
NAME                                                           READY   STATUS    RESTARTS   AGE
pod/autopilot-7b4f7f58f4-kchs4                                 1/1     Running   0          34m
pod/event-exporter-gke-67986489c8-prn9p                        2/2     Running   0          41m
pod/fluentbit-gke-bn5nm                                        2/2     Running   0          41m
pod/fluentbit-gke-f7k2j                                        2/2     Running   0          41m
pod/fluentbit-gke-h672g                                        2/2     Running   0          41m
pod/fluentbit-gke-n9664                                        2/2     Running   0          41m
pod/fluentbit-gke-xjttt                                        2/2     Running   0          41m
pod/gke-metrics-agent-d64hw                                    1/1     Running   0          41m
pod/gke-metrics-agent-fhw8l                                    1/1     Running   0          41m
pod/gke-metrics-agent-gsfvk                                    1/1     Running   0          41m
pod/gke-metrics-agent-mqm64                                    1/1     Running   0          41m
pod/gke-metrics-agent-wwjvx                                    1/1     Running   0          41m
pod/kube-dns-6c7b8dc9f9-q8v75                                  4/4     Running   0          41m
pod/kube-dns-6c7b8dc9f9-wqthz                                  4/4     Running   0          41m
pod/kube-dns-autoscaler-844c9d9448-4fx8f                       1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-11k6     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-5lgd     1/1     Running   0          16m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-b73f     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-n5fl     1/1     Running   0          41m
pod/kube-proxy-gke-carlos-lab01-default-pool-a6362dc8-v02w     1/1     Running   0          41m
pod/l7-default-backend-56cb9644f6-xfd65                        1/1     Running   0          41m
pod/metrics-server-v0.3.6-9c5bbf784-9z6sm                      2/2     Running   0          40m
pod/pdcsi-node-4wprs                                           2/2     Running   0          41m
pod/pdcsi-node-685ht                                           2/2     Running   0          41m
pod/pdcsi-node-g42tb                                           2/2     Running   0          41m
pod/pdcsi-node-ln4tw                                           2/2     Running   0          41m
pod/pdcsi-node-ncqhl                                           2/2     Running   0          41m
pod/portworx-api-76kcn                                         1/1     Running   0          34m
pod/portworx-api-887bl                                         1/1     Running   0          34m
pod/portworx-api-br4f2                                         1/1     Running   0          34m
pod/portworx-api-hzfsn                                         1/1     Running   0          34m
pod/portworx-api-zcd4m                                         1/1     Running   0          34m
pod/portworx-kvdb-8ls5k                                        1/1     Running   0          71s
pod/portworx-kvdb-c797z                                        1/1     Running   0          13m
pod/portworx-kvdb-gmxpv                                        1/1     Running   0          13m
pod/portworx-operator-bfc87df78-schcz                          1/1     Running   0          36m
pod/portworx-pvc-controller-696959f9bc-4kj5v                   1/1     Running   0          34m
pod/portworx-pvc-controller-696959f9bc-gn2tw                   1/1     Running   0          34m
pod/portworx-pvc-controller-696959f9bc-jmsxn                   1/1     Running   0          34m
pod/prometheus-px-prometheus-0                                 3/3     Running   1          33m
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-9pvws      3/3     Running   0          74s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-hnzs8      3/3     Running   0          88s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-j7vrc      3/3     Running   1          34m
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-sgm67      3/3     Running   0          113s
pod/px-cluster-cb94f533-5006-4299-b6b4-ad8e09690b74-xvzxn      3/3     Running   0          13m
pod/px-csi-ext-5686675c58-5qfzq                                3/3     Running   0          34m
pod/px-csi-ext-5686675c58-dbmmb                                3/3     Running   0          34m
pod/px-csi-ext-5686675c58-vss9p                                3/3     Running   0          34m
pod/px-prometheus-operator-8c88487bc-jv9fd                     1/1     Running   0          34m
pod/stackdriver-metadata-agent-cluster-level-9548fb7d6-vm552   2/2     Running   0          41m
pod/stork-75dd8b896-g4qqj                                      1/1     Running   0          34m
pod/stork-75dd8b896-mb2xt                                      1/1     Running   0          34m
pod/stork-75dd8b896-zjlwm                                      1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-866bv                           1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-jhx7w                           1/1     Running   0          34m
pod/stork-scheduler-574757dd8d-mhg99                           1/1     Running   0          34m

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                               AGE
service/default-http-backend        NodePort    10.3.241.138   <none>        80:31243/TCP                          41m
service/kube-dns                    ClusterIP   10.3.240.10    <none>        53/UDP,53/TCP                         41m
service/kubelet                     ClusterIP   None           <none>        10250/TCP                             33m
service/metrics-server              ClusterIP   10.3.248.53    <none>        443/TCP                               41m
service/portworx-api                ClusterIP   10.3.242.28    <none>        9001/TCP,9020/TCP,9021/TCP            34m
service/portworx-operator-metrics   ClusterIP   10.3.247.121   <none>        8999/TCP                              35m
service/portworx-service            ClusterIP   10.3.245.123   <none>        9001/TCP,9019/TCP,9020/TCP,9021/TCP   34m
service/prometheus-operated         ClusterIP   None           <none>        9090/TCP                              33m
service/px-csi-service              ClusterIP   None           <none>        <none>                                34m
service/px-prometheus               ClusterIP   10.3.245.244   <none>        9090/TCP                              34m
service/stork-service               ClusterIP   10.3.242.128   <none>        8099/TCP,443/TCP                      34m

NAME                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE
daemonset.apps/fluentbit-gke               5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/gke-metrics-agent           5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/gke-metrics-agent-windows   0         0         0       0            0           kubernetes.io/os=windows                                             41m
daemonset.apps/kube-proxy                  0         0         0       0            0           kubernetes.io/os=linux,node.kubernetes.io/kube-proxy-ds-ready=true   41m
daemonset.apps/metadata-proxy-v0.1         0         0         0       0            0           cloud.google.com/metadata-proxy-ready=true,kubernetes.io/os=linux    41m
daemonset.apps/nvidia-gpu-device-plugin    0         0         0       0            0           <none>                                                               41m
daemonset.apps/pdcsi-node                  5         5         5       5            5           kubernetes.io/os=linux                                               41m
daemonset.apps/pdcsi-node-windows          0         0         0       0            0           kubernetes.io/os=windows                                             41m
daemonset.apps/portworx-api                5         5         5       5            5           <none>                                                               34m

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/autopilot                                  1/1     1            1           34m
deployment.apps/event-exporter-gke                         1/1     1            1           41m
deployment.apps/kube-dns                                   2/2     2            2           41m
deployment.apps/kube-dns-autoscaler                        1/1     1            1           41m
deployment.apps/l7-default-backend                         1/1     1            1           41m
deployment.apps/metrics-server-v0.3.6                      1/1     1            1           41m
deployment.apps/portworx-operator                          1/1     1            1           36m
deployment.apps/portworx-pvc-controller                    3/3     3            3           34m
deployment.apps/px-csi-ext                                 3/3     3            3           34m
deployment.apps/px-prometheus-operator                     1/1     1            1           34m
deployment.apps/stackdriver-metadata-agent-cluster-level   1/1     1            1           41m
deployment.apps/stork                                      3/3     3            3           34m
deployment.apps/stork-scheduler                            3/3     3            3           34m

NAME                                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/autopilot-7b4f7f58f4                                  1         1         1       34m
replicaset.apps/event-exporter-gke-67986489c8                         1         1         1       41m
replicaset.apps/kube-dns-6c7b8dc9f9                                   2         2         2       41m
replicaset.apps/kube-dns-autoscaler-844c9d9448                        1         1         1       41m
replicaset.apps/l7-default-backend-56cb9644f6                         1         1         1       41m
replicaset.apps/metrics-server-v0.3.6-57bc866888                      0         0         0       41m
replicaset.apps/metrics-server-v0.3.6-886d66856                       0         0         0       41m
replicaset.apps/metrics-server-v0.3.6-9c5bbf784                       1         1         1       40m
replicaset.apps/portworx-operator-bfc87df78                           1         1         1       36m
replicaset.apps/portworx-pvc-controller-696959f9bc                    3         3         3       34m
replicaset.apps/px-csi-ext-5686675c58                                 3         3         3       34m
replicaset.apps/px-prometheus-operator-8c88487bc                      1         1         1       34m
replicaset.apps/stackdriver-metadata-agent-cluster-level-546484c84b   0         0         0       41m
replicaset.apps/stackdriver-metadata-agent-cluster-level-9548fb7d6    1         1         1       41m
replicaset.apps/stork-75dd8b896                                       3         3         3       34m
replicaset.apps/stork-scheduler-574757dd8d                            3         3         3       34m

NAME                                        READY   AGE
statefulset.apps/prometheus-px-prometheus   1/1     33m

You can try to test the features of Portworx deploying one Statefulset application

kubectl get sc
NAME                             PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
premium-rwo                      pd.csi.storage.gke.io           Delete          WaitForFirstConsumer   true                   29h
px-db                            kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-cloud-snapshot             kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-cloud-snapshot-encrypted   kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-encrypted                  kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-local-snapshot             kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-db-local-snapshot-encrypted   kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-replicated                    kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-replicated-encrypted          kubernetes.io/portworx-volume   Delete          Immediate              true                   29h
px-secure-sc                     kubernetes.io/portworx-volume   Delete          Immediate              false                  28h
standard (default)               kubernetes.io/gce-pd            Delete          Immediate              true                   29h
standard-rwo                     pd.csi.storage.gke.io           Delete          WaitForFirstConsumer   true                   29h
stork-snapshot-sc                stork-snapshot                  Delete          Immediate              true                   37m

Alright then, we need to create a Cluster Wide secret key to handle our encrypted StorageClasses

YOUR_SECRET_KEY=this-is-gonna-be-your-secret-key

kubectl -n kube-system create secret generic px-vol-encryption \
  --from-literal=cluster-wide-secret-key=$YOUR_SECRET_KEY

And apply this secret to Portworx

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl secrets set-cluster-key \
  --secret cluster-wide-secret-key

Once having your cluster wide secret in place, you can enable the cluster security on your storagecluster object, you can achieve this by editing the storagecluster object:

kubectl edit storagecluster -n kube-system

...
spec:
  security:
    enabled: true

And wait for the PX pods to be redeployed. To get access into your PX Cluster after this, you have to get the tokens on your pods.

PORTWORX_ADMIN_TOKEN=$(kubectl -n kube-system get secret px-admin-token -o json \
    | jq -r '.data."auth-token"' \
    | base64 -d)
    
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN    

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[1].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN    

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[2].metadata.name}')
kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl context create admin --token=$PORTWORX_ADMIN_TOKEN 


kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl secrets k8s login

Test the cluster with a StatefulSet

kubectl create namespace cassandra

Label three of your nodes with the label app=cassandra because this StatefulSet uses this label as node affinity policy.

kubectl label nodes <node01> <node02> <node03> app=cassandra

cassandra.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  namespace: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: app
                operator: In
                values:
                - cassandra
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - cassandra
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: cassandra:3.11
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          tcpSocket:
            port: 9042
          initialDelaySeconds: 30
          timeoutSeconds: 7
        volumeMounts:
        - name: cassandra-data
          mountPath: /var/lib/cassandra
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: px-db-encrypted
      resources:
        requests:
          storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: cassandra
  namespace: cassandra
spec:
  clusterIP: None
  selector:
    app: cassandra
  ports:
    - protocol: TCP
      name: port9042k8s
      port: 9042
      targetPort: 9042

Apply this file

kubectl apply -f cassandra.yaml
kubectl get pvc -n cassandra                                
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
cassandra-data-cassandra-0   Bound    pvc-81e11ede-e78a-4fd5-ae64-1ca451d8c8f9   2Gi        RWO            px-db-encrypted   116m
cassandra-data-cassandra-1   Bound    pvc-a83b23ee-1426-4b78-ae29-f6a562701e68   2Gi        RWO            px-db-encrypted   113m
cassandra-data-cassandra-2   Bound    pvc-326a2419-cf50-41d3-93d0-63dbecffbcdd   2Gi        RWO            px-db-encrypted   111m

calvarado04 / March 7, 2021

Chess and more

I will start from the beginning.

I started with chess back in 2002, when my brother and I used to play at a small chess club, we were playing just for fun primarily, however, it was pretty exciting because there was many adult players with a solid career on their own fields, such as lawyers, physicists, doctors, and so on.

That environment was very formative for me, even more, one of those professionals told me about Linux and that was the first time that I got curious about it...

But anyway, enough of talking about the past, on this post entry, I'd like to show my path to beat these different bots available to play at Chess.com app.

I started beating a bot with 1200 ELO of chess strength, and now I'm dealing with a bot with 2100 ELO, which is pretty difficult to beat, I mean, starting with 2000 ELO is technically a chess master candidate level, which is certainly is not very easy to achieve.

But I just wanna show here the most interesting games that I played against all these bots using the pretty fancy PGN plugin available and also showing the gif version of the matches.

The first game was against the bot Xavier with 1900 ELO, I played the London System:

The second game was against Li, a bot with 2000 ELO, I played the Alapin Variation against the Sicilian Defense:

This was the game agains Fatima, another bot with 2000 ELO:

And the last game that I'd like to show here will be this wild game against Charles, another bot with 2000 ELO, it started with a London System but I switched into a some kind of 150 Attack:

calvarado04 / November 23, 2020

CI/CD example with Python, Django, Kubernetes and Okteto

As a sysadmin with experience providing tech support for enterprise applications for around a decade, all this DevOps stuff happened suddenly and little bit silently to be honest, mostly because when your main concerns are to keep the daily operations working properly and the IT infrastructure doing well, there is no much time to look for new and amazing technologies. I arrived little bit late to this wave but in Mexico nowadays (late 2020), many companies are not even understanding what's going on with all this stuff.

But fortunately for me, in 2017, my colleague César Olea told me on a nice conversation that all this DevOps, CI/CD and remarkably, Containers and Kubernetes were being a great success on the IT Industry; so, after that conversation, I started to look into all this new world on my own, first with Docker and after that with Rancher 1.6 directly, which was simply great for me because that version of Rancher was a glorified Docker-Compose application, that I felt like having my own data center working inside my laptop, with each container performing pretty much like a server. And that was pretty important for me, to be able to understand and adopt all this new technology and this new approach.

Sooner than later, I realized that all this containers are not just another fancy way to deal with applications, nope, is much more than that, is a complete set of practices to enhance and improve the entire IT department, even sometimes called as Digital Transformation, term that if your company is really being involved into that, could be fairly appropriate to use.

Everything working together makes a ton of sense

Now in 2020, everything is still moving forward quite fast, even this basic example will become irrelevant in few months, but anyway, this blog is mine and is pretty much an attempt to demonstrate for myself and for others that actually I have all these skills.

The list of technologies that I'm using is just the basic for a minimal CI/CD architecture:

  • Programming IDE, Visual Studio Code with all the relevant plugins installed.
  • Python with Django as programming language, I'm learning right now Python and Django/Flask
  • Github account and a repository to push all my code.
  • A Kubernetes cluster to deploy all my stuff and deploy Jenkins, I choose Okteto because is cheap and is nice.
  • Okteto internal registry for the container images.
  • Okteto ingress controllers to publish the application.
  • Jenkins as Continuous Integration/Continuous Deployment engine, this approach actually could be different on many companies, because many organizations just set a trigger on every new image pushed to the container registry.

Backend

  • PostgreSQL database managed by Django using Django Models.
  • Python using Django.
  • I'm generating an API Rest to be consumed by the Frontend, however, there are some parts that actually comes directly from Django because this framework can act as frontend as well as backend.
  • Nginx Load Balancer, this would not be needed on this very basic application, but I decided to include it just for the challenge and because on more complex applications Nginx is widely used.
  • Repository URL: https://github.com/calvarado2004/django-api-rest
  • Backend URL: https://nginx-api-rest.calvarado04.com/api/element

Frontend

  • Javascript with Vue app, which consumes the API Rest served by Django but not directly, is consuming the API through Nginx. I'm pretty new with Vue and Javascript, by the way.
  • Repository URL: https://github.com/calvarado2004/vuedjango
  • Frontend URL: https://vue-api-rest.calvarado04.com

Infrastructure

Kubernetes is the professional way to deploy and use containers on enterprise graded environments, change my mind hahaha.

So, that is the natural step that you must take if you are being involved with containers.

My K8s namespace looks like:

CI/CD Pipelines!

As you should know, Jenkins works with Groovy to build its pipelines, Groovy it is not my favorite language but is still usable and Jenkins have some useful help on the application itself as well as on its documentation.

Database deployment:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.


node {
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    
    stage('Deploy the PostgreSQL Database'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/db-k8s
            ${OKTETO_DIR}/okteto namespace
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest-db-deployment
            '''
            println output
        }
    }
}

Backend deployment pipeline, Django:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/backend-django'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    stage ('Build and Push Image with Okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            ${OKTETO_DIR}/okteto build -t ${CONTAINER_IMAGE}:${BUILD_TAG} .
            '''
            println output
        }
    }
    
    stage('Deploy the new image to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/backend-k8s
            ${OKTETO_DIR}/okteto namespace
            cat kubernetes.j2 | sed "s#{{ CONTAINER_IMAGE }}:{{ TAG_USED }}#${CONTAINER_IMAGE}:${BUILD_TAG}#g" > kubernetes.yaml
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest
            '''
            println output
        }
    }
}

Nginx pipeline

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/backend-django'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/django-api-rest.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            git url: "${GIT_PROJECT}"
    }
    
    
    stage('Deploy Nginx to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/nginx-k8s
            ${OKTETO_DIR}/okteto namespace
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/nginx-api-rest
            '''
            println output
        }
    }
}

Vue pipeline:

#!/usr/bin/env groovy

//Author: Carlos Alvarado
//Jenkins Pipeline to handle the Continuous Integration and Continuous Deployment on Okteto.
//Prerequisites: you should install the Custom tools plugin on Jenkins, ... 
//...get the okteto CLI and Kubectl. You also need to get your Okteto Token and save it on a Jenkins Credential


node {
    
    env.OKTETO_DIR = tool name: 'okteto', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.HOME = "${WORKSPACE}"
    env.CONTAINER_IMAGE = 'registry.cloud.okteto.net/calvarado2004/frontend-vue'
    env.KUBECTL_DIR = tool name: 'kubectl', type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
    env.GIT_PROJECT = 'https://github.com/calvarado2004/vuedjango.git'
    
    stage ('Prepare Environment with Okteto ') {
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            cleanWs deleteDirs: true
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            '''
            println output
        }
    }
    
    stage ('Download the source code from GitHub'){
            def output = sh returnStdout: true, script: '''git clone "${GIT_PROJECT}"'''
            println output
    }
    
    stage ('Build and Push Image with Okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/vuedjango
            ${OKTETO_DIR}/okteto build -t ${CONTAINER_IMAGE}:${BUILD_TAG} .
            '''
            println output
        }
    }
    
    stage('Deploy the new image to okteto'){
        withCredentials([string(credentialsId: 'okteto-token', variable: 'SECRET')]) {
            def output = sh returnStdout: true, script: '''
            ${OKTETO_DIR}/okteto login --token ${SECRET}
            cd ${HOME}/vuedjango/frontend-k8s
            ${OKTETO_DIR}/okteto namespace
            cat kubernetes.j2 | sed "s#{{ CONTAINER_IMAGE }}:{{ TAG_USED }}#${CONTAINER_IMAGE}:${BUILD_TAG}#g" > kubernetes.yaml
            ${KUBECTL_DIR}/kubectl apply -f kubernetes.yaml
            ${KUBECTL_DIR}/kubectl rollout status deployment.apps/django-api-rest
            '''
            println output
        }
    }
}

Dockerfiles

Docker is just a company that works with containers, but its Dockerfiles became the standard way to define almost all of them.

Here is the Dockerfile for the backend:

FROM python:3.8.5

COPY djangovue /djangovue
COPY requirements.txt /djangovue/requirements.txt
WORKDIR /djangovue
RUN pip install -r requirements.txt && chmod 755 /djangovue/manage.py
CMD python manage.py runserver 0.0.0.0:8000

And the frontend one:

# build environment
FROM node:12.2.0-alpine as build
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/package.json
RUN npm install --silent
RUN npm install @vue/cli@3.7.0 -g
COPY . /app
RUN npm run build

# production environment
FROM nginx:1.16.0-alpine
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Working with more than one environment

If you check this frontend, you will see two environment files, concretely a file called .env.development with the following content:

VUE_APP_DJANGO_HOST=localhost
VUE_APP_DJANGO_PORT=8000
VUE_APP_DJANGO_PROTOCOL=http

and a file called .env.production with something more interesting:

VUE_APP_DJANGO_HOST=nginx-api-rest.calvarado04.com
VUE_APP_DJANGO_PORT=443
VUE_APP_DJANGO_PROTOCOL=https

So, yes, this is the proper way to deal with more than one environment, define inside your code variables that you can check and modify later. Nevermore the developers mantra: but it works on my machine!...

Kubernetes definitions

Database:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: django-api-rest-pvc
  namespace: calvarado2004
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Secret
metadata:
  name: django-api-rest-credentials
  namespace: calvarado2004
type: Opaque
data:
  user: UG9zdGdyZXM=
  password: UG9zdGdyZXNrOHMk 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-api-rest-db-deployment
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-api-rest-db-container
  template:
    metadata:
      labels:
        app: django-api-rest-db-container
        tier: backend
    spec:
      containers:
        - name: django-api-rest-db-container
          image: postgres:12.4
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user

            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password

            - name: POSTGRES_DB
              value: djangovuedb

            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata

          ports:
            - containerPort: 5432
          volumeMounts:
            - name: django-api-rest-volume-mount
              mountPath: "/var/lib/postgresql/data"

      volumes:
        - name: django-api-rest-volume-mount
          persistentVolumeClaim:
            claimName: django-api-rest-pvc
---
kind: Service
apiVersion: v1
metadata:
  name: django-api-rest-db-service
  namespace: calvarado2004
spec:
  selector:
    app: django-api-rest-db-container
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432

Backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django-api-rest-container
  template:
    metadata:
      labels:
        app: django-api-rest-container
    spec:
      containers:
        - name: django-api-rest-container
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          ports:
            - containerPort: 8000
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password
            - name: POSTGRES_HOST
              value: django-api-rest-db-service
      initContainers:
        - name: django-api-rest-init
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          command: ['python', 'manage.py', 'migrate']
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: django-api-rest-credentials
                  key: password
            - name: POSTGRES_HOST
              value: django-api-rest-db-service
---
kind: Service
apiVersion: v1
metadata:
  name: django-api-rest
  namespace: calvarado2004
spec:
  selector:
    app: django-api-rest-container
  type: ClusterIP
  ports:
  - name: django-http
    protocol: TCP
    port: 8000
    targetPort: 8000

Nginx:

Note that I'm consuming the Django application making reference to the internal DNS that is the standard way to do it on Kubernetes {{application}}.{{namespace}}.svc.cluster.local when you want to consume a service with another service internally. This approach will not work for Vue because that application is effectively consuming the API on client side (literally is doing its duty on your browser) and because of that, it needs to be referenced to the API published to Internet (or Intranet if is an internal app).

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config-map
data:
  nginx.conf: |-
    events {
      
    }
    http {
      include /etc/nginx/conf.d/*.conf;
      upstream backend_server {
          server django-api-rest.calvarado2004.svc.cluster.local:8000;
      }
      server {
          listen 80 default_server;
          server_name nginx-api-rest.calvarado04.com;
          location / {
              proxy_pass http://backend_server;
              proxy_set_header Host $http_host;
              proxy_redirect off;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              proxy_set_header X-Forwarded-Proto $https;
              proxy_connect_timeout 360s;
              proxy_read_timeout 360s;
              proxy_hide_header Access-Control-Allow-Origin;
              proxy_hide_header Access-Control-Allow-Credentials;
              set $CORS_CREDS true;
              set $CORS_ORIGIN $http_origin;
              set $CORS_METHODS 'GET, POST, PUT, DELETE, OPTIONS';
              set $CORS_HEADERS 'Authentication-Token, Cache-Control, Cookie, If-Modified-Since, Range, User-Agent, X-Requested-With';
              set $CORS_EXPOSE_HEADERS 'Content-Disposition, Content-Length, Content-Range, Set-Cookie';
              set $CORS_PREFLIGHT_CACHE_AGE 600;
              set $X_FRAME_OPTIONS '';
              if ($request_method = 'OPTIONS') {
                add_header Access-Control-Allow-Origin $CORS_ORIGIN;
                add_header Access-Control-Allow-Methods $CORS_METHODS;
                add_header Access-Control-Allow-Headers $CORS_HEADERS;
                add_header Access-Control-Allow-Credentials $CORS_CREDS;
                add_header Access-Control-Max-Age $CORS_PREFLIGHT_CACHE_AGE;
                add_header Content-Type 'text/plain; charset=utf-8';
                add_header Content-Length 0;
                return 204;
              }
              if ($request_method != 'OPTIONS') {
                add_header Access-Control-Allow-Origin $CORS_ORIGIN;
                add_header Access-Control-Allow-Methods $CORS_METHODS;
                add_header Access-Control-Allow-Headers $CORS_HEADERS;
                add_header Access-Control-Allow-Credentials $CORS_CREDS;
                add_header Access-Control-Expose-Headers $CORS_EXPOSE_HEADERS;
                add_header X-Frame-Options $X_FRAME_OPTIONS;
              }
          }
      }
 
    }  
---      
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-api-rest-container
  template:
    metadata:
      labels:
        app: nginx-api-rest-container
    spec:
      containers:
        - name: nginx-api-rest-container
          image: nginx:latest
          volumeMounts: 
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
          ports:
            - containerPort: 443
          command: ["/bin/sh"]
          args: ["-c", "while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\""]
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config-map
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-api-rest
  namespace: calvarado2004
  annotations:
    dev.okteto.com/auto-ingress: "true"
spec:
  selector:
    app: nginx-api-rest-container
  type: ClusterIP
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80

Frontend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vue-api-rest
  namespace: calvarado2004
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vue-api-rest-container
  template:
    metadata:
      labels:
        app: vue-api-rest-container
    spec:
      containers:
        - name: vue-api-rest-container
          image: {{ CONTAINER_IMAGE }}:{{ TAG_USED }}
          ports:
            - containerPort: 80

---
kind: Service
apiVersion: v1
metadata:
  name: vue-api-rest
  namespace: calvarado2004
  annotations:
    dev.okteto.com/auto-ingress: "true"
spec:
  selector:
    app: vue-api-rest-container
  type: ClusterIP
  ports:
    - name: "vue-api-rest"
      protocol: TCP
      port: 80
      targetPort: 80

Last thoughts

As you can realize, DevOps adoption is not easy at all because implies to understand and know how to make it work together a huge range of technologies, that used to be very specialized and kind of isolated ones from each others. Developers needs to know more in deep about infrastructure, and Sysadmins, DBA's, Testers and Security teams needs to understand and make some effort to achieve a confortable way to deploy easily to production but warranting the best levels of quality at the same time.

This is the deal, but at the end of the day, it's not rocket science... 😉

calvarado04 / October 22, 2020

There was a long time without posting anything new here…

On this COVID-19 times, I just would like to say that this blog will continue alive, I'm thinking in big for my further career and I want to keep this little effort as part of my plans for the near and medium term future.

I did even a redirection from my custom domain to this blog.

Anyway, let's see what's going on.

calvarado04 / July 1, 2019

Infrastructure as Code approach, a real life example OCP with Vagrant

As is well known, the Infrastructure as Code approach is nowadays the trendy topic of IT industry in almost all tech companies, it has been for years on startups, and after passed the quality tests is becoming a reality on bigger and older companies, even on the pretty old and very conservative institutions such as banks.

Well, to be honest, my first impression of all the Kubernetes stuff it was little bit stressful, I started with Rancher 1.6 and that solution uses Cattle, its own orchestrator solution, but the important think it's that with Cattle, every old school sysadmin feels like home, because that console is just like seeing a control panel of a data center, in fact it looks and feels just like a f..ng data center on your own laptop, but instead of powerful and big server nodes, there are some little containers running on it.

But, what the heck is a container?

Good question, in short words, a container is a set of one or more processes that are isolated from the rest of the system (Red Hat, 2018). That's it.

You should not to see a container as a virtual machine because a container is pretty different, moreover, you can create containers inside virtual machines, and this post will provide you a proof of what I'm saying.

There is a very good explanation from Red Hat here.

Docker is the most common container product, however, there are more containers solutions, such as cri-o, podman, rocket, and so on.

OpenShift Container Platform (OCP) a Kubernetes based Platform as a Service (PaaS) solution

Well, Kubernetes is a solution to handle containerized systems with a complete integration, from networking to storage and security stuff.

Let me share with you a very good introduction video of what Kubernetes is:

OpenShift is the Red Hat PaaS product based on Kubernetes to provide a full and reliable infrastructure for containers solution.

Enough from introductions, let's get started

Vagrant is an Infrastructure as Code solution from HashiCorp, it provides some tools to deploy your virtual infrastructure by defining a Vagrantfile, it deploys the virtual machines with their configurations, networking, subscriptions and the product also includes a repository of VirtualBox images that you can use on your projects.

This project is using Vagrant and is deploying some VirtualBox Virtual Machines, so, your local host machine should have at least 16GB of RAM memory and enough free storage (like 60GB) to deploy the three nodes of this cluster.

So, the prerequisites are:

  • Laptop with Linux (Fedora, Debian, Ubuntu) with 16 GB of RAM and at least 60 GB of free space or a Mac with similar capabilities
  • Vagrant already installed
  • VirtualBox with tools
  • Red Hat Subscription to OCP 3.9, Ansible 2.4, RHEL 7 and RHEL extras repos enabled. (sorry guys, I can not share with you my own subscription)
  • Internet domain with DNS administration, for instance, Namecheap.
  • SSL certificates with wildcards enabled of your internet domain, if you want valid SSL certificates.

You can use my domain if you want, the only thing that will be that the SSL certificates will be self signed on your cluster and you should to be adding the exceptions on your web browser.

Configure your DNS like the following example, adding some A Records:

  • 192.168.150.101 is the master node, also the public name of the cluster (cluster.openshift) is making reference to this node.
  • 192.168.150.102 is the node01 the infrastructure node, in that node it will by deployed the router pod, that's why the wildcard domain *.openshift is configured to reach that node.
  • 192.168.150.103 is the compute node, node02.

As you can see, we are using the capabilities of DNS A records but we are making reference to local IP's so, all these addresses will not be making sense for external attackers.

Vagrant plugins

vagrant plugin install vagrant-hostmanager

vagrant plugin install vagrant-scp

The only thing that you should perform after having all these prerequisites. would be:

./oc-up.sh

And wait for a while

After that, you can navigate to your new cluster.

https://cluster.openshift.calvarado04.com

This cluster is including some NFS Persistent Volumes to be able to create a project with persistent storage out of the box.

Obviously, you should to replace my calvarado04.com domain with your own domain.

The default user is admin and the password is handhand.

Why OpenShift 3.9.78?

Just because is the official version for the Red Hat Certified Specialist in OpenShift Administration (EX280) certification.

The scripts

If you will be using your own domain, just replace any calvarado04.com with your domain on the following scripts.

Create a directory like openshift-vagrant3-9 and in there place the following scripts:

Vagrantfile

OPENSHIFT_RELEASE = "3.9"
OPENSHIFT_ANSIBLE_BRANCH = "release-#{OPENSHIFT_RELEASE}"
NETWORK_BASE = "192.168.150"
INTEGRATION_START_SEGMENT = 101

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.

$script = %{
if ! subscription-manager status; then
  sudo subscription-manager register --username=youraccount  --password=yourpassword
  sudo subscription-manager attach --pool=yourpool
  sudo subscription-manager repos --enable=rhel-7-server-extras-rpms
  sudo subscription-manager repos --enable=rhel-7-server-ansible-2.4-rpms 
  sudo subscription-manager repos --enable=rhel-7-server-ose-3.9-rpms
  sudo subscription-manager repos --enable=rhel-7-server-rpms
  sudo subscription-manager repos --enable=rhel-7-fast-datapath-rpms
  sudo rm -rf /etc/yum.repos.d/epel.repo
  sudo rm -rf /etc/yum.repos.d/epel-testing.repo 
  sudo yum install -y docker
  sudo systemctl enable docker
  sudo systemctl start docker
  sudo setsebool -P virt_sandbox_use_fusefs on
  sudo setsebool -P virt_use_fusefs on
fi
}

Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "generic/rhel7"
  config.vm.box_check_update = true
  config.vm.provision "shell", inline: $script

#  if Vagrant.has_plugin?('landrush')
#    config.landrush.enabled = true
#    config.landrush.tld = 'calvarado04.com'
#    config.landrush.guest_redirect_dns = false
#  end

  config.hostmanager.enabled = true
  config.hostmanager.manage_host = true
  config.hostmanager.ignore_private_ip = false


  config.vm.provider "virtualbox" do |vb|
    vb.memory = "3072"
    vb.cpus   = "2"
  end

  # Define nodes
  (1..2).each do |i|
    config.vm.define "node0#{i}" do |node|
      node.vm.network "private_network", ip: "#{NETWORK_BASE}.#{INTEGRATION_START_SEGMENT + i}"
      node.vm.hostname = "node0#{i}.calvarado04.com"

      if "#{i}" == "1"
        node.hostmanager.aliases = %w(lb.calvarado04.com)
      end
    end
  end

  # Define master
  config.vm.define "master", primary: true do |node|
    node.vm.network "private_network", ip: "#{NETWORK_BASE}.#{INTEGRATION_START_SEGMENT}"
    node.vm.hostname = "master.calvarado04.com"
    node.hostmanager.aliases = %w(etcd.calvarado04.com nfs.calvarado04.com)
    
    # 
    # Memory of the master node must be allocated at least 2GB in order to
    # prevent kubernetes crashed-down due to 'out of memory' and you'll end
    # up with 
    # "Unable to restart service origin-master: Job for origin-master.service 
    #  failed because a timeout was exceeded. See "systemctl status 
    #  origin-master.service" and "journalctl -xe" for details."
    #
    # See https://github.com/kubernetes/kubernetes/issues/13382#issuecomment-154891888
    # for mor details.
    #
    node.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus   = "2"
    end
    

    # Deploy private keys of each node to master
    if File.exist?(".vagrant/machines/master/virtualbox/private_key")
      node.vm.provision "master-key", type: "file", run: "never", source: ".vagrant/machines/master/virtualbox/private_key", destination: "/home/vagrant/.ssh/master.key"
    end

    if File.exist?(".vagrant/machines/node01/virtualbox/private_key")
      node.vm.provision "node01-key", type: "file", run: "never", source: ".vagrant/machines/node01/virtualbox/private_key", destination: "/home/vagrant/.ssh/node01.key"
    end

    if File.exist?(".vagrant/machines/node02/virtualbox/private_key")
      node.vm.provision "node02-key", type: "file", run: "never", source: ".vagrant/machines/node02/virtualbox/private_key", destination: "/home/vagrant/.ssh/node02.key"
    end
  end
end

oc-up.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# resolve links - $0 may be a softlink
PRG="$0"
RETCODE=0

while [ -h "$PRG" ]; do
    ls=`ls -ld "$PRG"`
    link=`expr "$ls" : '.*-> \(.*\)$'`
    if expr "$link" : '/.*' > /dev/null; then
        PRG="$link"
    else
        PRG=`dirname "$PRG"`/"$link"
    fi
done

# Get standard environment variables
PRGDIR=`dirname "$PRG"`

readonly openshift_release=`cat Vagrantfile | grep '^OPENSHIFT_RELEASE' | awk -F'=' '{print $2}' | sed 's/^[[:blank:]\"]*//;s/[[:blank:]\"]*$//'`

. "$PRGDIR/common.sh"

vagrant up

vagrant provision --provision-with master-key,node01-key,node02-key

vagrant scp ansible-hosts master:/home/vagrant/ansible-hosts

vagrant scp master.sh master:/home/vagrant/master.sh 

vagrant scp all.sh master:/home/vagrant/all.sh

vagrant scp common.sh master:/home/vagrant/common.sh

vagrant scp htpasswd master:/home/vagrant/htpasswd

vagrant scp calvarado04_com master:/home/vagrant/

vagrant scp _openshift_calvarado04_com master:/home/vagrant/


vagrant ssh master -c 'sudo mkdir /exports; sudo chmod 777 /exports'
vagrant ssh master -c 'sudo yum install -y nfs-utils rpcbind'
vagrant ssh master -c 'sudo systemctl enable nfs-server'
vagrant ssh master -c 'sudo systemctl enable rpcbind'
vagrant ssh master -c 'sudo systemctl enable nfs-lock'
vagrant ssh master -c 'sudo systemctl enable nfs-idmap'
vagrant ssh master -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh master -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh master -c 'sudo setsebool -P virt_use_fusefs on'
vagrant ssh master -c 'sudo firewall-cmd --zone=public --add-service=nfs'
vagrant ssh master -c 'sudo firewall-cmd --zone=public --add-service=nfs  --permanent'
vagrant ssh master -c 'echo "/exports *(rw,root_squash,sync,no_wdelay)" > /home/vagrant/exports; sudo mv /home/vagrant/exports /etc/exports'
vagrant ssh master -c 'sudo systemctl start nfs-server'
vagrant ssh master -c 'sudo systemctl start rpcbind'
vagrant ssh master -c 'sudo systemctl start nfs-lock'
vagrant ssh master -c 'sudo systemctl start nfs-idmap'


vagrant ssh node01 -c 'sudo yum install -y nfs-utils rpcbind'
vagrant ssh node01 -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh node01 -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh node01 -c 'sudo setsebool -P virt_use_fusefs on'
vagrant ssh node01 -c 'sudo mkdir /exports; sudo chmod 777 /exports'
vagrant ssh node01 -c 'sudo mount -t nfs -o rw,sync master.calvarado04.com:/exports /exports'

vagrant ssh node02 -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh node02 -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh node02 -c 'sudo setsebool -P virt_use_fusefs on'

vagrant ssh master -c 'sudo /bin/bash /home/vagrant/master.sh'

vagrant scp CreatePVs.sh master:/home/vagrant

vagrant ssh master -c 'ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml'

if [ $? -eq 0 ]; then 

  vagrant ssh master -c 'ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml'

  vagrant ssh master -c 'chmod 755 /home/vagrant/CreatePVs.sh; /bin/bash /home/vagrant/CreatePVs.sh'


else

  echo -e "\n The prerequisites has been failed, please check. \n"

fi

htpasswd

admin:$apr1$gfaL16Jf$c.5LAvg3xNDVQTkk6HpGB1

CreatePVs.sh

oc adm policy add-cluster-role-to-user cluster-admin admin

mkdir -p /exports/openshift/pvs/

chmod 777 /exports/openshift/pvs/


mkdir -p /home/vagrant/pvfiles


export volsize="2Gi"

for volume in pv-rwo{10..17} ; do
  mkdir -p /exports/openshift/pvs/${volume}
  chmod 777 /exports/openshift/pvs/${volume}
  cat << EOF > /home/vagrant/pvfiles/${volume}
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "${volume}"
  },
  "spec": {
    "capacity": {
        "storage": "${volsize}"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/exports/openshift/pvs/${volume}",
        "server": "master.calvarado04.com"
    },
    "persistentVolumeReclaimPolicy": "Recycle"
  }
}
EOF
  echo "Created def file for ${volume}";
done;


for volume in pv-rwm{20..22} ; do
  mkdir -p /exports/openshift/pvs/${volume}
  chmod 777 /exports/openshift/pvs/${volume}
  cat << EOF > /home/vagrant/pvfiles/${volume}
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "${volume}"
  },
  "spec": {
    "capacity": {
        "storage": "${volsize}"
    },
    "accessModes": [ "ReadWriteMany" ],
    "nfs": {
        "path": "/exports/openshift/pvs/${volume}",
        "server": "master.calvarado04.com"
    },
    "persistentVolumeReclaimPolicy": "Retain"
  }
}
EOF
  echo "Created def file for ${volume}";
done;


cat /home/vagrant/pvfiles/* | oc create -f -

common.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#===  FUNCTION  ================================================================
#         NAME:  version
#  DESCRIPTION:  Convert a version string to integer
# PARAMETER  1:  Version string
#===============================================================================
function version() {
    echo "$@" | awk -F "." '{ printf("%01d%03d\n", $1, $2); }'
}

master.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

yum -y install git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct
      
# Sourcing common functions
. /home/vagrant/common.sh

yum -y install openshift-ansible

mv /home/vagrant/ansible-hosts /etc/ansible/hosts

mkdir -p /home/vagrant/.ssh
bash -c 'echo "Host *" >> /home/vagrant/.ssh/config'
bash -c 'echo "StrictHostKeyChecking no" >> /home/vagrant/.ssh/config'
chmod 600 /home/vagrant/.ssh/config
chown -R vagrant:vagrant /home/vagrant

ansible-hosts

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
nfs

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=vagrant

# If ansible_ssh_user is not root, ansible_become must be set to true
ansible_become=true

openshift_deployment_type=openshift-enterprise
openshift_image_tag=v3.9.78
openshift_pkg_version=-3.9.78
openshift_release=3.9.78
openshift_disable_check=disk_availability,docker_storage,memory_availability


osm_cluster_network_cidr=10.1.0.0/16
openshift_portal_net=172.30.0.0/16
hostSubnetLength=9
os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'

openshift_console_install=true
openshift_console_hostname=console.openshift.calvarado04.com
openshift_enable_unsupported_configurations=true

#Add your own Red Hat credentials
oreg_auth_user=youruser
oreg_auth_password=yourpassword

#OCR configuration variables
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
#openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_registry_storage_volume_size=15Gi
openshift_hosted_registry_storage_host=master.calvarado04.com

openshift_examples_modify_imagestreams=true
os_firewall_use_firewalld=True

#Comment this if you don't have your own SSL certificates

#Master/API certificates
openshift_master_overwrite_named_certificates=true

openshift_master_named_certificates=[{'certfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.crt', 'keyfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.key', 'names':  ['cluster.openshift.calvarado04.com'], 'cafile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.ca-bundle' }]

#Router certificates
openshift_hosted_router_certificate={'cafile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.ca-bundle', 'certfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.crt', 'keyfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.key'} 


#Htpasswd
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/home/vagrant/htpasswd'}]


openshift_master_htpasswd_file=/home/vagrant/htpasswd
# Default login account: admin / handhand

openshift_disable_check=disk_availability,memory_availability,docker_storage,docker_image_availability
openshift_docker_options=" --selinux-enabled --log-driver=journald --storage-driver=overlay --registry-mirror=http://4a0fee72.m.daocloud.io "

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true','runtime=docker']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true','runtime=docker']}, {'name': 'node-config-infra-compute','labels': ['node-role.kubernetes.io/infra=true','node-role.kubernetes.io/compute=true','runtime=docker']}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true','runtime=docker'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

openshift_enable_service_catalog=true
template_service_broker_install=true

openshift_hosted_router_replicas=1

openshift_master_api_port=443
openshift_master_console_port=443
openshift_master_default_subdomain=openshift.calvarado04.com
openshift_master_cluster_public_hostname=cluster.openshift.calvarado04.com
openshift_master_cluster_hostname=master.calvarado04.com


openshift_template_service_broker_namespaces=['openshift']
ansible_service_broker_install=true
openshift_master_dynamic_provisioning_enabled=true

# host group for masters
[masters]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key"

[etcd]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key"

[nodes]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'master', 'node-role.kubernetes.io/master':'true'}"
node01.calvarado04.com openshift_ip=192.168.150.102 openshift_host=192.168.150.102 ansible_ssh_private_key_file="/home/vagrant/.ssh/node01.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'infra', 'node-role.kubernetes.io/infra':'true'}"
node02.calvarado04.com openshift_ip=192.168.150.103 openshift_host=192.168.150.103 ansible_ssh_private_key_file="/home/vagrant/.ssh/node02.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'compute', 'node-role.kubernetes.io/compute':'true'}"

[nfs]
master.calvarado04.com

Don't forget to add your SSL certificates.

Gallery

calvarado04 / April 7, 2019

Cómo lograr ser un Red Hat Certified Engineer (RHCE) sin morir en el intento

Red Hat es mundialmente conocida por sus certificaciones, ya que sus exámenes distan mucho de las tradicionales pruebas de opción múltiple o incluso de los exámenes con preguntas abiertas, pues en Red Hat, los exámenes son totalmente prácticos, te ponen delante de una máquina, con una lista de actividades que se tienen que realizar sin conexión a internet, siendo monitoreado remotamente por dos cámaras web por el aplicador y teniendo pocos descansos (sin que estos detengan el tiempo disponible para realizar el intento de certificación). Por estas y otras razones (como el costo de sus cursos e intentos de examen), el peso específico que tienen estas certificaciones en la industria de las TI es bastante alto.

Hace poco comencé a recorrer el certification path que la empresa ofrece para llegar a ser algún día un Red Hat Certified Architect (RHCA), camino que requiere dos certificaciones base que son la Red Hat Certified System Administrator (RHCSA) y la Red Hat Certified Engineer (RHCE) más otras cinco certificaciones que pueden ser de muchas tecnologías, dependiendo más de gustos y necesidades laborales; así pues, en total son siete certificaciones para lograr ser un RHCA.

Pues bien, dado que ya tengo la RHCSA, hablaré de mi plan de ataque para obtener la RHCE al primer o segundo intento, pues, esta certificación tiene fama de ser muy pesada, poca gente reporta terminar todas las actividades en el tiempo disponible (4 horas) así como su nivel de dificultad, pues como dije, uno está solo contra la máquina, pudiéndose consultar solamente la documentación del sistema operativo y además siendo monitoreado todo el tiempo.

El temario básicamente es el siguiente, obviamente todo se basa en Red Hat Enterprise Linux 7 hasta el momento, aunque ya no tarda RHEL 8:

  • Configuración de servicios con Systemd
  • Control del proceso de arranque
  • Configurar redes IPv4
  • Configurar redes IPv6
  • Configuración Link Aggregation
  • Configuración de bridges por software
  • Administración de Firewalld
  • Etiquetado de puertos con SELinux
  • Configurar servidor DNS (unbound)
  • Solucionar problemas de DNS
  • Configurar servidor de correo
  • Conceptos iSCSI
  • Configurar servidor NFS
  • Configurar servidor SMB/CIFS
  • Administración e instalación de María DB
  • Queries RDBMS MySQL/Maria DB
  • Configurar servidor HTTPD Apache
  • Hosts virtuales y aplicaciones web con Apache
  • Shell scripting con Bash
  • Shell scripting avanzado, estructuras de control y condiciones
  • Uso de Docker containers

Realmente no son tópicos del otro mundo, son cosas que uno como sysadmin ha tenido que realizar al menos una vez durante la experiencia laboral, sin embargo, es menester estar bien preparado, con los comandos y los archivos de configuración frescos en la memoria y con la capacidad de solucionar los problemas más comunes de forma rápida porque el tiempo avanza velozmente y cuando menos te das cuenta, te quedan 30 minutos para terminar de responder.

Espero en entradas próximas ir desmenuzando este temario para que sea útil a más personas.

Actualmente cuento ya con cuatro certificaciones de Red Hat, a saber:

calvarado04 / March 8, 2019

Katacoda, o cómo aprender sin aburrirse.

Sin duda, vivimos en tiempos donde la dinámica económica está cambiando en favor de los Servicios, donde la llamada Economía del Conocimiento es cada vez más palpable y está llegando a cada vez más personas en todo el mundo. Vivir en esta nueva realidad puede ser muy ventajoso, ya que por primera vez, se tiene un potencial de ascenso social no condicionado por el origen de la persona, sino por su talento y sus conocimientos.

Mucho se habla de cómo la educación formal, básica, media y superior han ayudado a millones de personas a mejorar su nivel de vida y esto es cierto, pero también hay que mencionar que esta educación no es suficiente, pues vivimos una tendencia de especialización de los puestos de trabajo, en detrimento de labores aburridas, repetitivas y poco gratificantes, lo cual, dicho sea de paso, me parece algo muy bueno.

En el mundo de la computación, he de decir, que las clases formales siempre me parecieron (y me parecen) muy aburridas, incluso las que trataban de los temas que más me gustaban, con excepción de las clases realmente teóricas (y donde la cátedra tradicional no puede ser sustituida), como estructuras de datos, cálculo, álgebra superior, matemáticas discretas, etc. Pero en el caso de las materias más prácticas, como las de programación, simplemente las encontré muy tediosas y creo que era por una simple razón: ver láminas o presentaciones con porciones de código sin realmente ejecutarlo uno mismo es como ver ruido de las televisiones de antaño, ruido que es descartado por el cerebro casi de inmediato. Claro, hasta el momento que intentas programar algo tú mismo, no sabes cómo hacerlo y recuerdas vagamente al profesor intentando enseñarte justamente eso.

Entonces,

¿Cómo adquirir estos conocimientos que ayudan a marcar la diferencia de una mejor forma?

No hay secreto: haciéndolo uno mismo.

Aquí es donde entran portales como Katacoda, que proporcionan plataformas bastante interesantes, donde uno puede ir practicando directamente en una consola y en interfaces web embebidas, de modo que ya no hay pretexto para no aprender algo nuevo y de una forma divertida (para los frikis jaja)

Página de inicio de Katacoda

En particular, esta herramienta contiene muchos escenarios que sirven para ir aprendiendo sobre las nuevas tecnologías como Docker, Kubernetes, Jenkins, OpenShift, NodeJS, Cloud Platforms y hasta un poco de machine learning. Por supuesto, una gran parte del contenido está disponible de forma gratuita, esperando a que alguien quiera aprender haciendo.

Consola interactiva
Interfaz web embebida

https://www.katacoda.com

calvarado04 / February 17, 2019

OpenShift Enterprise 3.11.69 on Intel NUC’s

Well, this is a very technical post and, in order to be useful for more people, I decided to write it in English.

Prerequisites

First of all, you should be able to have access to the OpenShift Enterprise repositories from Red Hat. Disclaimer: the current post is not related to CentOS based installations, also this post is not referring to OKD, MiniShift or even OpenShift on a single container (all-in-one), nope, this post is the technical review of OpenShift Enterprise 3.11 installed on a relatively small hardware.

Hardware

  • Intel NUC Core i7 quad core, 32 GB of RAM, 256 GB PCIe Flash, 750 GB Hard Disk.
  • Intel NUC Core i5 quad core, 16 GB of RAM, 256 GB PCIe Flash, 500 GB Hard Disk.
  • Intel NUC Core i3 quad core, 16 GB of RAM, 128 GB PCIe Flash, 500 GB Hard Disk.
  • HP MP 9, Intel Core i5 quad core, 16 GB of RAM, 256 SSD. This is the master node.
  • Lenovo Laptop with RHEL 7.6 as bastion host.
The nodes of my cluster.

Cluster components

  • 1 Master node
  • 3 Infra-compute nodes (infrastructure and computing nodes)
  • 3 GlusterFS nodes

As you can advice, there are not enough nodes to cover the proposed architecture, so, I'm proposing to share the node resources in order to deploy the infra-compute nodes and the glusterfs nodes together. Of course, this kind of deployment is not recommended by Red Hat, remember, this is only a cluster for learning purposes, never for production purposes.

Brief list of OpenShift services to deploy

  • Hawkular
  • Cassandra
  • Heapster
  • Elasticsearch
  • Fluentd
  • Kibana
  • Alert manager
  • Prometheus
  • Grafana
  • GlusterFS
  • Web Console
  • Catalog
  • Cluster console
  • Docker registry
  • OLM Operators
  • Problem detector
  • OC command line
  • Heketi
  • Master API
  • Internal Router
  • Scheduler
  • and more...

Operating system

Red Hat Enterprise Linux (RHEL) 7.6 up to date with the following repositories enabled:

rhel-7-fast-datapath-rpms
rhel-7-server-extras-rpms
rhel-7-server-optional-rpms
rhel-7-server-ansible-2.6-rpms
rhel-7-server-ose-3.11-rpms
rhel-7-server-rpms
rh-gluster-3-client-for-rhel-7-server-rpms
rh-gluster-3-for-rhel-7-server-rpms

You can enable them by performing the following commands (on RHEL 7.6):

subscription-manager repos --enable=rhel-7-fast-datapath-rpms
subscription-manager repos --enable=rhel-7-server-extras-rpms
subscription-manager repos --enable=rhel-7-server-optional-rpms
subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms 
subscription-manager repos --enable=rhel-7-server-ose-3.11-rpms
subscription-manager repos --enable=rhel-7-server-rpms
subscription-manager repos --enable=rh-gluster-3-client-for-rhel-7-server-rpms
subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms

RPM packages required

The required RPMs on every node and the bastion host should be: wget, git, net-tools, bind-utils, yum-utils, firewalld, java-1.8.0-openjdk, bridge-utils, bash-completion, kexec-tools, sos, psacct, openshift-ansible, glusterfs-fuse, docker and skopeo.

$ sudo yum -y install wget git net-tools bind-utils yum-utils firewalld java-1.8.0-openjdk \ 
bridge-utils bash-completion kexec-tools sos psacct openshift-ansible glusterfs-fuse docker skopeo

For the GlusterFS nodes (in this case, the nodei7, nodei5 and nodei3 nodes, also install Heketi Server on the master node) you should to install also the Heketi packages and the GlusterFS server packages:

$ sudo yum -y install heketi* gluster*

DNS (configuration required)

You need a fully qualified domain name service capable to handle wildcards, unfortunately dnsmasq is not enough for that purpose, you should to configure bind (named) or directly handling this by a DNS on your own domain (I did the last scenario). You can check this guide if you want to deploy Bind on your bastion host: https://access.redhat.com/documentation/en-us/openshift_enterprise/2/html/deployment_guide/sect-configuring_bind_and_dns

Adding these entries on the /etc/hosts file is not sufficient to deploy successfully the cluster due to OpenShift generates automatically internal rules on the Kubernetes pods and it takes them from the DNS configuration.

  • master.calvarado04.com
  • nodei7.calvarado04.com
  • nodei5.calvarado04.com
  • nodei3.calvarado04.com
  • *.openshift.calvarado04.com <--- Wildcard, it must be making reference to the nodes where is deployed the router service (infra nodes).
  • cluster-openshift.calvarado04.com <--- Master node public name

Don't forget to change the name of your nodes in the same way, you can perform that by:

$ sudo hostnamectl set-hostname nodei7.calvarado04.com

$ hostname --fqdn
nodei7.calvarado04.com      

SSL Certificate (wildcard capable) for the OpenShift cluster

As you can see, If you want to install your own SSL certificate, that certificate must have wildcard capabilities, due to the basic SSL certificates won't work at all on OpenShift.

Your common name on your CSR request should be like this for the router certificate:

  • *.calvarado04.com for the master
  • *.openshift.calvarado04.com for the router subdomain (all the apps)

I added my certificate files on the bastion host on the directory /root/calvarado04.com, the files needed are:

  • /root/calvarado04.com/calvarado04.pem (that is the certificate file merged with the intermediate certificate and the root certificate, the certificate content at the top of the file).
  • /root/calvarado04.com/calvarado04.key with the private key generated when I generated the CSR file to obtain the SSL certificate.
  • /root/calvarado04.com/calvarado04.ca with the root certificate.
  • And the same for the router certificates (Openshift subdomain) on /root/calvarado04.com/openshift/

SELinux

Many people just disable SELinux on almost all new installation, this is not the case, SELinux is mandatory, it must be targered and enforcing on all nodes on the /etc/selinux/config file.

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Also please add the following rules to give permissions to the containers:

[root@bastion ~]# ansible nodes -a "setsebool -P virt_sandbox_use_fusefs on"

[root@bastion ~]# ansible nodes -a "setsebool -P virt_use_fusefs on"

Firewalld

OpenShift used to work with IPtables, but this is not recommended anymore and any new deployment should be using firewalld instead. To mask and disable iptables:

$ sudo systemctl stop iptables
$ sudo systemctl mask iptables

$ sudo yum -y install firewalld

$ sudo systemctl start firewalld.service
$ sudo systemctl status firewalld.service
$ sudo systemctl enable firewalld.service

Also add the GlusterFS Firewalld rules in order to allow the Gluster traffic on your nodes:

$ sudo firewall-cmd --zone=public --add-service=glusterfs

$ sudo firewall-cmd --zone=public --add-service=glusterfs --permanent

NetworkManager

OpenShift uses the NetworkManager capabilities, so, is required to have enabled NetworkManager on the network device, for instance, check the following configuration:

USE_PEERDNS=yes and NM_CONTROLLED=yes

[root@nodei7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno1
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="eno1"
UUID="e57de406-5cc2-415a-90c7-5671a76dd3a2"
DEVICE="eno1"
ONBOOT="yes"
ZONE=public
USE_PEERDNS=yes
NM_CONTROLLED=yes

Restart the network and NetworkManager services:

$ sudo systemctl restart NetworkManager

$ sudo systemctl restart network

SSH keys

Ansible requires SSH keys to run the required tasks on each node, let's create a key and distribute it to every nodes:

[root@bastion ~]# ssh-keygen

[root@bastion ~]# for host in master.calvarado04.com \
    nodei7.calvarado04.com \ 
    nodei5.calvarado04.com \  
    nodei3.calvarado04.com; \ 
    do ssh-copy-id -i ~/.ssh/id_rsa.pub $host; \
    done

Ansible

The official OpenShift Enterprise 3.11.69 installation method is by running the OpenShift playbooks provided by the RPMs. The Ansible version must be 2.6.x not 2.7 due to issues with some tasks.

GlusterFS

This deployment will be using GlusterFS on the most basic installation using only GlusterFS without blocks and is not including the GlusterFS Registry cluster on Convergent mode. However, this method is allowing you to use dynamic provisioning of the PVC's (Persistent Volume Claims), this feature is quite fancy because is not longer needed to declare manually the PV's once the cluster is up and running.

Heketi configuration on the master node

OpenShift uses Heketi to generate the Gluster topology from the master node to the Gluster nodes and it perform that task by using a SSH connection. That's why you need to configure Heketi on the master host by adding the user and ssh key on /etc/heketi/heketi.json.

Also change any timeout entry to a numeric value (on _sshexec_comment and _kubeexec_comment), like "gluster_cli_timeout": 900.

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/root/.ssh/id_rsa",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab",
      "backup_lvm_metadata": false,
      "gluster_cli_timeout": 900
    },

Wait for Gluster pods issue

On my first attempt to deploy the cluster I was stuck on a step that were wait for Gluster pods, the counter was timed out and the playbook was marked as failure. I checked the pods on the nodes and I saw them, I checked its logs and its logs was not showing any error. Researching more I found that could be a bug on the wait_for_pods.yml deployment playbook, located on /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/tasks, concretely a cast issue due to the playbook is comparing a string with an integer.

This is the original playbook:

---
- name: Wait for GlusterFS pods
  oc_obj:
    namespace: "{{ glusterfs_namespace }}"
    kind: pod
    state: list
    selector: "glusterfs={{ glusterfs_name }}-pod"
  register: glusterfs_pods_wait
  until:
  - "glusterfs_pods_wait.results.results[0]['items'] | count > 0"
  # There must be as many pods with 'Ready' staus  True as there are nodes expecting those pods
  - "glusterfs_pods_wait.results.results[0]['items'] | lib_utils_oo_collect(attribute='status.conditions') | lib_utils_oo_collect(attribute='status', filters={'type': 'Ready'}) | map('bool') | select | list | count == l_glusterfs_count | int"
  delay: 30
  retries: "{{ (glusterfs_timeout | int / 10) | int }}"
  vars:
    l_glusterfs_count: "{{ glusterfs_count | default(glusterfs_nodes | count ) }}"

And this is the fixed playbook:

---
- name: Wait for GlusterFS pods
  oc_obj:
    namespace: "{{ glusterfs_namespace }}"
    kind: pod
    state: list
    selector: "glusterfs={{ glusterfs_name }}-pod"
  register: glusterfs_pods_wait
  until:
  - "glusterfs_pods_wait.results.results[0]['items'] | count > 0"
  # There must be as many pods with 'Ready' staus  True as there are nodes expecting those pods
  - "glusterfs_pods_wait.results.results[0]['items'] | lib_utils_oo_collect(attribute='status.conditions') | lib_utils_oo_collect(attribute='status', filters={'type': 'Ready'}) | map('bool') | select | list | count == l_glusterfs_count | int"
  delay: 30
  retries: "{{ (glusterfs_timeout | int / 10) | int }}"
  vars:
    l_glusterfs_count: "{{ glusterfs_count | default(glusterfs_nodes | count ) }} | int"

Wipe the GlusterFS disks

In order to install successfully the GlusterFS cluster, please wipe all the data from the chosen disks (in this case, the three GlusterFS nodes have the disk on /dev/sda, but that it could be different in your configuration, be careful), there must be on RAW format only. Warning: you can lose all your data if you are not careful. You can perform this by:

[root@bastion ~]# ansible glusterfs -a "wipefs -a -f /dev/sda"

HTPasswd (users and passwords for OpenShift)

This is a really simple configuration on this topic, I'm using only htpasswd, I saved my users on /root/htpasswd.openshift on the bastion host. You should to try with LDAP, it's more secure and advanced. Go ahead and learn something 😉

If you want to use the same way just:

[root@bastion ~]# htpasswd -c /root/htpasswd.openshift luke
New password: 
Re-type new password: 
Adding password for user luke
[root@broker ~]# cat /root/htpasswd.openshift
luke:$apr1$SzuTxyhH$DlH976Tv2cDBccFZqJ3zf1

If you want to add more users, just omit the -c option:

[root@bastion ~]# htpasswd /root/htpasswd.openshift leia
New password: 
Re-type new password: 
Adding password for user leia
[root@bastion ~]# cat  /root/htpasswd.openshift
luke:$apr1$SzuTxyhH$DlH976Tv2cDBccFZqJ3zf1
leia:$apr1$UK2B49gy$qD/n3lKsoWXT0eRAMNoqm.

Run the installation playbooks

Once the package openshift-ansible is already installed on your bastion host and your SSH keys are already distributed on all your nodes, you can perform the installation of your OpenShift Cluster.

The happy path is quite simple:

  • Fill your inventory file located in /etc/ansible/hosts
  • Run the prerequisites playbook
  • Run the deployment cluster playbook
  • Check your installation
[root@bastion ~]# cd /usr/share/ansible/openshift-ansible/

[root@bastion openshift-ansible]# ansible-playbook playbooks/prerequisites.yml

[root@bastion openshift-ansible]# pwd
/usr/share/ansible/openshift-ansible

[root@bastion openshift-ansible]# ansible-playbook playbooks/deploy_cluster.yml

The inventory file: /etc/ansible/hosts

The most important file is this one, every configuration will be taken from this file and every failure will be coming from here almost all the times.

This is my inventory file (without users and passwords):

[OSEv3:vars]

timeout=120
ansible_user=root
ansible_become=no

openshift_deployment_type=openshift-enterprise
openshift_disable_check="disk_availability,memory_availability,docker_image_availability,package_version"

openshift_image_tag=v3.11.69
openshift_pkg_version=-3.11.69
openshift_release=3.11.69

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true','runtime=docker']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true','runtime=docker']}, {'name': 'node-config-infra-compute','labels': ['node-role.kubernetes.io/infra=true','node-role.kubernetes.io/compute=true','runtime=docker']}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true','runtime=docker'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]


logrotate_scripts=[{"name": "syslog", "path": "/var/log/cron\n/var/log/maillog\n/var/log/messages\n/var/log/secure\n/var/log/spooler\n", "options": ["daily", "rotate 7","size 500M", "compress", "sharedscripts", "missingok"], "scripts": {"postrotate": "/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true"}}]
openshift_enable_olm=true

oreg_url=registry.redhat.io/openshift3/ose-${component}:${version}
oreg_auth_user=email@something.com
oreg_auth_password=passw0rd

# Set this line to enable NFS
#openshift_enable_unsupported_configurations=True

openshift_additional_registry_credentials=[{'host':'registry.connect.redhat.com','user':'email@something.com','password':'passw0rd','test_image':'mongodb/enterprise-operator:0.3.2'}]
openshift_examples_modify_imagestreams=true
os_firewall_use_firewalld=True

openshift_master_api_port=443
openshift_master_console_port=443
openshift_master_default_subdomain=openshift.calvarado04.com
openshift_master_cluster_public_hostname=cluster-openshift.calvarado04.com
openshift_master_cluster_hostname=master.calvarado04.com

osm_cluster_network_cidr=10.1.0.0/16
openshift_portal_net=172.30.0.0/16
hostSubnetLength=9
os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'

openshift_use_openshift_sdn=true
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_file=/root/htpasswd.openshift
openshift_cluster_monitoring_operator_install=true
openshift_enable_service_catalog=true
template_service_broker_install=true

openshift_template_service_broker_namespaces=['openshift']
ansible_service_broker_install=true
ansible_service_broker_local_registry_whitelist=['.*-apb$']
openshift_master_dynamic_provisioning_enabled=true

openshift_master_overwrite_named_certificates=true 
openshift_master_named_certificates=[{'certfile': '/root/calvarado04.com/calvarado04.pem', 'keyfile': '/root/calvarado04.com/calvarado04.key', 'names':  ['cluster-openshift.calvarado04.com'], 'cafile': '/root/calvarado04.com/calvarado04.ca' }]
openshift_hosted_router_certificate={'cafile': '/root/calvarado04.com/openshift/calvarado04.ca', 'certfile': '/root/calvarado04.com/openshift/calvarado04.pem', 'keyfile': '/root/calvarado04.com/openshift/calvarado04.key'} 

openshift_certificate_expiry_warning_days=20

openshift_hosted_router_replicas=3

openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_registry_storage_volume_size=15Gi
openshift_hosted_registry_storage_kind=glusterfs
openshift_hosted_registry_replicas=3
openshift_hosted_registry_pullthrough=true
openshift_hosted_registry_acceptschema2=true
openshift_hosted_registry_enforcequota=true
openshift_hosted_registry_storage_volume_name=registry


openshift_metrics_install_metrics=true
openshift_metrics_storage_kind=dynamic
openshift_metrics_hawkular_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_cassandra_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_heapster_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_storage_volume_size=15Gi
openshift_metrics_cassandra_pvc_storage_class_name="glusterfs-storage"
openshift_metrics_duration=1


openshift_prometheus_namespace=openshift-metrics
openshift_prometheus_node_selector={"node-role.kubernetes.io/infra":"true"}
openshift_prometheus_memory_requests=2Gi
openshift_prometheus_cpu_requests=750m
openshift_prometheus_memory_limit=2Gi
openshift_prometheus_cpu_limit=750m
openshift_prometheus_alertmanager_memory_requests=300Mi
openshift_prometheus_alertmanager_cpu_requests=200m
openshift_prometheus_alertmanager_memory_limit=300Mi
openshift_prometheus_alertmanager_cpu_limit=200m
openshift_prometheus_alertbuffer_memory_requests=300Mi
openshift_prometheus_alertbuffer_cpu_requests=200m
openshift_prometheus_alertbuffer_memory_limit=300Mi
openshift_prometheus_alertbuffer_cpu_limit=200m


openshift_logging_install_logging=true
openshift_logging_es_pvc_dynamic=true
openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_es_pvc_size=15Gi
openshift_logging_es_pvc_storage_class_name="glusterfs-storage"
openshift_logging_es_memory_limit=3Gi
openshift_logging_es_cluster_size=1
openshift_logging_curator_default_days=1


openshift_storage_glusterfs_wipe=true
openshift_storage_glusterfs_heketi_wipe=true
openshift_storage_glusterfs_namespace=app-storage
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_storageclass_default=true
openshift_storage_glusterfs_block_deploy=false   



openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7
openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7
openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7
openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs-gluster-s3-server-rhel7


openshift_storage_glusterfs_timeout=900

osm_default_node_selector='node-role.kubernetes.io/compute=true'


[OSEv3:children]
masters
etcd
nodes
glusterfs

[masters]
master.calvarado04.com

[etcd]
master.calvarado04.com


[nodes]
master.calvarado04.com openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true openshift_schedulable=True
nodei7.calvarado04.com openshift_node_group_name='node-config-infra-compute' openshift_node_problem_detector_install=true openshift_schedulable=True
nodei5.calvarado04.com openshift_node_group_name='node-config-infra-compute' openshift_node_problem_detector_install=true openshift_schedulable=True
nodei3.calvarado04.com openshift_node_group_name='node-config-infra-compute' openshift_node_problem_detector_install=true openshift_schedulable=True


[glusterfs]
nodei7.calvarado04.com glusterfs_ip=192.168.0.73 glusterfs_devices='[ "/dev/sda" ]'
nodei5.calvarado04.com glusterfs_ip=192.168.0.72 glusterfs_devices='[ "/dev/sda" ]'
nodei3.calvarado04.com glusterfs_ip=192.168.0.70 glusterfs_devices='[ "/dev/sda" ]'

Redeploy script (it's only working on my cluster)

I wrote a script to redeploy my cluster, I'm sharing with you:

[root@bastion ~]# cat redeployOpenShift.sh 
#!/bin/bash

cd /usr/share/ansible/openshift-ansible
ansible-playbook playbooks/adhoc/uninstall.yml
ansible-playbook -e "openshift_storage_glusterfs_wipe=true" playbooks/openshift-glusterfs/uninstall.yml
ansible nodes -a "rm -rf /etc/origin"
ansible nodes -a "rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd /var/log/glusterfs_devices /var/lib/gluster-block /var/log/gluster-block"
ansible glusterfs -a "yum -y reinstall gluster*"
ansible glusterfs -a "wipefs -a -f /dev/sda"
ansible nodes -a "shutdown -r now"
sleep 180
ansible glusterfs -a "systemctl restart glusterd"
ansible glusterfs -a "systemctl stop glusterd"
ansible nodes -a "setsebool -P virt_sandbox_use_fusefs on"
ansible nodes -a "setsebool -P virt_use_fusefs on"
ansible glusterfs -a "wipefs -a -f /dev/sda"
ansible-playbook playbooks/deploy_cluster.yml 
[root@bastion ~]# 

Final steps

Once the cluster will be up and running, just navigate to https://cluster-openshift.calvarado04.com the link it's only working for me 😉 and open the console.

To give cluster admin access to the desired user, enter to the master node and perform the following command:

[root@master ~]# oc whoami 
system:admin

[root@master ~]# oc adm policy add-cluster-role-to-user cluster-admin admin

[root@master ~]# oc adm policy add-cluster-role-to-user cluster-reader luke

[root@master ~]# oc get pods -o wide

[root@master ~]# oc get projects

[root@master ~]# oc rollout latest project-developer

[root@master ~]# oc logs 


Gallery

OpenShift Catalog
OpenShift Console
Grafana dashboards
Cluster Pods
Kibana
SSL Certificate

References

  • CHAPTER 4. DEPLOYING CONTAINERIZED STORAGE IN CONVERGED MODE https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/deployment_guide/chap-documentation-deploy-cns
  • Configuring OpenShift to use Custom Certificates http://v1.uncontained.io/playbooks/installation/custom_certificates.html
  • Container-native storage 3.9: Enhanced day-2 management and unified installation within OpenShift Container Platform https://redhatstorage.redhat.com/2018/03/28/container-native-storage-3-9/
  • Preparing your hosts https://docs.openshift.com/container-platform/3.11/install/host_preparation.html

calvarado04 / February 15, 2019

Primer post

Este es el primer post de mi segundo intento de WordPress. A ver cómo sale.

Les dejo un excelente track de música electrónica para empezar.

Recent Posts

  • Deploy an Elasticsearch cluster for Kubernetes (ECK) on Google Compute Platform (GCP on GKE) with Terraform – Part I
  • Deploy a GKE Cluster with Portworx
  • Chess and more
  • CI/CD example with Python, Django, Kubernetes and Okteto
  • There was a long time without posting anything new here…

Recent Comments

    Archives

    • November 2021
    • May 2021
    • March 2021
    • November 2020
    • October 2020
    • July 2019
    • April 2019
    • March 2019
    • February 2019

    Categories

    • Certifications
    • CI/CD
    • DevOps
    • Django
    • GKE
    • Groovy
    • Katacoda
    • Kubernetes
    • Okteto
    • OpenShift
    • Python
    • Red Hat
    • Uncategorized

    Meta

    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org

    © 2023 All Rights Reserved by Tech and more stuff. Theme Developed by Theme Rally. Powered by WordPress.