Tech and more stuff
open source, politics and electronic music

Infrastructure as Code approach, a real life example OCP with Vagrant

calvarado04 Uncategorized July 1, 2019

As is well known, the Infrastructure as Code approach is nowadays the trendy topic of IT industry in almost all tech companies, it has been for years on startups, and after passed the quality tests is becoming a reality on bigger and older companies, even on the pretty old and very conservative institutions such as banks.

Well, to be honest, my first impression of all the Kubernetes stuff it was little bit stressful, I started with Rancher 1.6 and that solution uses Cattle, its own orchestrator solution, but the important think it's that with Cattle, every old school sysadmin feels like home, because that console is just like seeing a control panel of a data center, in fact it looks and feels just like a f..ng data center on your own laptop, but instead of powerful and big server nodes, there are some little containers running on it.

But, what the heck is a container?

Good question, in short words, a container is a set of one or more processes that are isolated from the rest of the system (Red Hat, 2018). That's it.

You should not to see a container as a virtual machine because a container is pretty different, moreover, you can create containers inside virtual machines, and this post will provide you a proof of what I'm saying.

There is a very good explanation from Red Hat here.

Docker is the most common container product, however, there are more containers solutions, such as cri-o, podman, rocket, and so on.

OpenShift Container Platform (OCP) a Kubernetes based Platform as a Service (PaaS) solution

Well, Kubernetes is a solution to handle containerized systems with a complete integration, from networking to storage and security stuff.

Let me share with you a very good introduction video of what Kubernetes is:

OpenShift is the Red Hat PaaS product based on Kubernetes to provide a full and reliable infrastructure for containers solution.

Enough from introductions, let's get started

Vagrant is an Infrastructure as Code solution from HashiCorp, it provides some tools to deploy your virtual infrastructure by defining a Vagrantfile, it deploys the virtual machines with their configurations, networking, subscriptions and the product also includes a repository of VirtualBox images that you can use on your projects.

This project is using Vagrant and is deploying some VirtualBox Virtual Machines, so, your local host machine should have at least 16GB of RAM memory and enough free storage (like 60GB) to deploy the three nodes of this cluster.

So, the prerequisites are:

  • Laptop with Linux (Fedora, Debian, Ubuntu) with 16 GB of RAM and at least 60 GB of free space or a Mac with similar capabilities
  • Vagrant already installed
  • VirtualBox with tools
  • Red Hat Subscription to OCP 3.9, Ansible 2.4, RHEL 7 and RHEL extras repos enabled. (sorry guys, I can not share with you my own subscription)
  • Internet domain with DNS administration, for instance, Namecheap.
  • SSL certificates with wildcards enabled of your internet domain, if you want valid SSL certificates.

You can use my domain if you want, the only thing that will be that the SSL certificates will be self signed on your cluster and you should to be adding the exceptions on your web browser.

Configure your DNS like the following example, adding some A Records:

  • 192.168.150.101 is the master node, also the public name of the cluster (cluster.openshift) is making reference to this node.
  • 192.168.150.102 is the node01 the infrastructure node, in that node it will by deployed the router pod, that's why the wildcard domain *.openshift is configured to reach that node.
  • 192.168.150.103 is the compute node, node02.

As you can see, we are using the capabilities of DNS A records but we are making reference to local IP's so, all these addresses will not be making sense for external attackers.

Vagrant plugins

vagrant plugin install vagrant-hostmanager

vagrant plugin install vagrant-scp

The only thing that you should perform after having all these prerequisites. would be:

./oc-up.sh

And wait for a while

After that, you can navigate to your new cluster.

https://cluster.openshift.calvarado04.com

This cluster is including some NFS Persistent Volumes to be able to create a project with persistent storage out of the box.

Obviously, you should to replace my calvarado04.com domain with your own domain.

The default user is admin and the password is handhand.

Why OpenShift 3.9.78?

Just because is the official version for the Red Hat Certified Specialist in OpenShift Administration (EX280) certification.

The scripts

If you will be using your own domain, just replace any calvarado04.com with your domain on the following scripts.

Create a directory like openshift-vagrant3-9 and in there place the following scripts:

Vagrantfile

OPENSHIFT_RELEASE = "3.9"
OPENSHIFT_ANSIBLE_BRANCH = "release-#{OPENSHIFT_RELEASE}"
NETWORK_BASE = "192.168.150"
INTEGRATION_START_SEGMENT = 101

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.

$script = %{
if ! subscription-manager status; then
  sudo subscription-manager register --username=youraccount  --password=yourpassword
  sudo subscription-manager attach --pool=yourpool
  sudo subscription-manager repos --enable=rhel-7-server-extras-rpms
  sudo subscription-manager repos --enable=rhel-7-server-ansible-2.4-rpms 
  sudo subscription-manager repos --enable=rhel-7-server-ose-3.9-rpms
  sudo subscription-manager repos --enable=rhel-7-server-rpms
  sudo subscription-manager repos --enable=rhel-7-fast-datapath-rpms
  sudo rm -rf /etc/yum.repos.d/epel.repo
  sudo rm -rf /etc/yum.repos.d/epel-testing.repo 
  sudo yum install -y docker
  sudo systemctl enable docker
  sudo systemctl start docker
  sudo setsebool -P virt_sandbox_use_fusefs on
  sudo setsebool -P virt_use_fusefs on
fi
}

Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "generic/rhel7"
  config.vm.box_check_update = true
  config.vm.provision "shell", inline: $script

#  if Vagrant.has_plugin?('landrush')
#    config.landrush.enabled = true
#    config.landrush.tld = 'calvarado04.com'
#    config.landrush.guest_redirect_dns = false
#  end

  config.hostmanager.enabled = true
  config.hostmanager.manage_host = true
  config.hostmanager.ignore_private_ip = false


  config.vm.provider "virtualbox" do |vb|
    vb.memory = "3072"
    vb.cpus   = "2"
  end

  # Define nodes
  (1..2).each do |i|
    config.vm.define "node0#{i}" do |node|
      node.vm.network "private_network", ip: "#{NETWORK_BASE}.#{INTEGRATION_START_SEGMENT + i}"
      node.vm.hostname = "node0#{i}.calvarado04.com"

      if "#{i}" == "1"
        node.hostmanager.aliases = %w(lb.calvarado04.com)
      end
    end
  end

  # Define master
  config.vm.define "master", primary: true do |node|
    node.vm.network "private_network", ip: "#{NETWORK_BASE}.#{INTEGRATION_START_SEGMENT}"
    node.vm.hostname = "master.calvarado04.com"
    node.hostmanager.aliases = %w(etcd.calvarado04.com nfs.calvarado04.com)
    
    # 
    # Memory of the master node must be allocated at least 2GB in order to
    # prevent kubernetes crashed-down due to 'out of memory' and you'll end
    # up with 
    # "Unable to restart service origin-master: Job for origin-master.service 
    #  failed because a timeout was exceeded. See "systemctl status 
    #  origin-master.service" and "journalctl -xe" for details."
    #
    # See https://github.com/kubernetes/kubernetes/issues/13382#issuecomment-154891888
    # for mor details.
    #
    node.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus   = "2"
    end
    

    # Deploy private keys of each node to master
    if File.exist?(".vagrant/machines/master/virtualbox/private_key")
      node.vm.provision "master-key", type: "file", run: "never", source: ".vagrant/machines/master/virtualbox/private_key", destination: "/home/vagrant/.ssh/master.key"
    end

    if File.exist?(".vagrant/machines/node01/virtualbox/private_key")
      node.vm.provision "node01-key", type: "file", run: "never", source: ".vagrant/machines/node01/virtualbox/private_key", destination: "/home/vagrant/.ssh/node01.key"
    end

    if File.exist?(".vagrant/machines/node02/virtualbox/private_key")
      node.vm.provision "node02-key", type: "file", run: "never", source: ".vagrant/machines/node02/virtualbox/private_key", destination: "/home/vagrant/.ssh/node02.key"
    end
  end
end

oc-up.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# resolve links - $0 may be a softlink
PRG="$0"
RETCODE=0

while [ -h "$PRG" ]; do
    ls=`ls -ld "$PRG"`
    link=`expr "$ls" : '.*-> \(.*\)$'`
    if expr "$link" : '/.*' > /dev/null; then
        PRG="$link"
    else
        PRG=`dirname "$PRG"`/"$link"
    fi
done

# Get standard environment variables
PRGDIR=`dirname "$PRG"`

readonly openshift_release=`cat Vagrantfile | grep '^OPENSHIFT_RELEASE' | awk -F'=' '{print $2}' | sed 's/^[[:blank:]\"]*//;s/[[:blank:]\"]*$//'`

. "$PRGDIR/common.sh"

vagrant up

vagrant provision --provision-with master-key,node01-key,node02-key

vagrant scp ansible-hosts master:/home/vagrant/ansible-hosts

vagrant scp master.sh master:/home/vagrant/master.sh 

vagrant scp all.sh master:/home/vagrant/all.sh

vagrant scp common.sh master:/home/vagrant/common.sh

vagrant scp htpasswd master:/home/vagrant/htpasswd

vagrant scp calvarado04_com master:/home/vagrant/

vagrant scp _openshift_calvarado04_com master:/home/vagrant/


vagrant ssh master -c 'sudo mkdir /exports; sudo chmod 777 /exports'
vagrant ssh master -c 'sudo yum install -y nfs-utils rpcbind'
vagrant ssh master -c 'sudo systemctl enable nfs-server'
vagrant ssh master -c 'sudo systemctl enable rpcbind'
vagrant ssh master -c 'sudo systemctl enable nfs-lock'
vagrant ssh master -c 'sudo systemctl enable nfs-idmap'
vagrant ssh master -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh master -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh master -c 'sudo setsebool -P virt_use_fusefs on'
vagrant ssh master -c 'sudo firewall-cmd --zone=public --add-service=nfs'
vagrant ssh master -c 'sudo firewall-cmd --zone=public --add-service=nfs  --permanent'
vagrant ssh master -c 'echo "/exports *(rw,root_squash,sync,no_wdelay)" > /home/vagrant/exports; sudo mv /home/vagrant/exports /etc/exports'
vagrant ssh master -c 'sudo systemctl start nfs-server'
vagrant ssh master -c 'sudo systemctl start rpcbind'
vagrant ssh master -c 'sudo systemctl start nfs-lock'
vagrant ssh master -c 'sudo systemctl start nfs-idmap'


vagrant ssh node01 -c 'sudo yum install -y nfs-utils rpcbind'
vagrant ssh node01 -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh node01 -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh node01 -c 'sudo setsebool -P virt_use_fusefs on'
vagrant ssh node01 -c 'sudo mkdir /exports; sudo chmod 777 /exports'
vagrant ssh node01 -c 'sudo mount -t nfs -o rw,sync master.calvarado04.com:/exports /exports'

vagrant ssh node02 -c 'sudo setsebool -P nfs_export_all_rw on'
vagrant ssh node02 -c 'sudo setsebool -P virt_sandbox_use_fusefs on'
vagrant ssh node02 -c 'sudo setsebool -P virt_use_fusefs on'

vagrant ssh master -c 'sudo /bin/bash /home/vagrant/master.sh'

vagrant scp CreatePVs.sh master:/home/vagrant

vagrant ssh master -c 'ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml'

if [ $? -eq 0 ]; then 

  vagrant ssh master -c 'ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml'

  vagrant ssh master -c 'chmod 755 /home/vagrant/CreatePVs.sh; /bin/bash /home/vagrant/CreatePVs.sh'


else

  echo -e "\n The prerequisites has been failed, please check. \n"

fi

htpasswd

admin:$apr1$gfaL16Jf$c.5LAvg3xNDVQTkk6HpGB1

CreatePVs.sh

oc adm policy add-cluster-role-to-user cluster-admin admin

mkdir -p /exports/openshift/pvs/

chmod 777 /exports/openshift/pvs/


mkdir -p /home/vagrant/pvfiles


export volsize="2Gi"

for volume in pv-rwo{10..17} ; do
  mkdir -p /exports/openshift/pvs/${volume}
  chmod 777 /exports/openshift/pvs/${volume}
  cat << EOF > /home/vagrant/pvfiles/${volume}
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "${volume}"
  },
  "spec": {
    "capacity": {
        "storage": "${volsize}"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/exports/openshift/pvs/${volume}",
        "server": "master.calvarado04.com"
    },
    "persistentVolumeReclaimPolicy": "Recycle"
  }
}
EOF
  echo "Created def file for ${volume}";
done;


for volume in pv-rwm{20..22} ; do
  mkdir -p /exports/openshift/pvs/${volume}
  chmod 777 /exports/openshift/pvs/${volume}
  cat << EOF > /home/vagrant/pvfiles/${volume}
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "${volume}"
  },
  "spec": {
    "capacity": {
        "storage": "${volsize}"
    },
    "accessModes": [ "ReadWriteMany" ],
    "nfs": {
        "path": "/exports/openshift/pvs/${volume}",
        "server": "master.calvarado04.com"
    },
    "persistentVolumeReclaimPolicy": "Retain"
  }
}
EOF
  echo "Created def file for ${volume}";
done;


cat /home/vagrant/pvfiles/* | oc create -f -

common.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#===  FUNCTION  ================================================================
#         NAME:  version
#  DESCRIPTION:  Convert a version string to integer
# PARAMETER  1:  Version string
#===============================================================================
function version() {
    echo "$@" | awk -F "." '{ printf("%01d%03d\n", $1, $2); }'
}

master.sh

#!/bin/bash
#
# Copyright 2017 Liu Hongyu
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

yum -y install git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct
      
# Sourcing common functions
. /home/vagrant/common.sh

yum -y install openshift-ansible

mv /home/vagrant/ansible-hosts /etc/ansible/hosts

mkdir -p /home/vagrant/.ssh
bash -c 'echo "Host *" >> /home/vagrant/.ssh/config'
bash -c 'echo "StrictHostKeyChecking no" >> /home/vagrant/.ssh/config'
chmod 600 /home/vagrant/.ssh/config
chown -R vagrant:vagrant /home/vagrant

ansible-hosts

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
nfs

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=vagrant

# If ansible_ssh_user is not root, ansible_become must be set to true
ansible_become=true

openshift_deployment_type=openshift-enterprise
openshift_image_tag=v3.9.78
openshift_pkg_version=-3.9.78
openshift_release=3.9.78
openshift_disable_check=disk_availability,docker_storage,memory_availability


osm_cluster_network_cidr=10.1.0.0/16
openshift_portal_net=172.30.0.0/16
hostSubnetLength=9
os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'

openshift_console_install=true
openshift_console_hostname=console.openshift.calvarado04.com
openshift_enable_unsupported_configurations=true

#Add your own Red Hat credentials
oreg_auth_user=youruser
oreg_auth_password=yourpassword

#OCR configuration variables
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
#openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_registry_storage_volume_size=15Gi
openshift_hosted_registry_storage_host=master.calvarado04.com

openshift_examples_modify_imagestreams=true
os_firewall_use_firewalld=True

#Comment this if you don't have your own SSL certificates

#Master/API certificates
openshift_master_overwrite_named_certificates=true

openshift_master_named_certificates=[{'certfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.crt', 'keyfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.key', 'names':  ['cluster.openshift.calvarado04.com'], 'cafile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.ca-bundle' }]

#Router certificates
openshift_hosted_router_certificate={'cafile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.ca-bundle', 'certfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.crt', 'keyfile': '/home/vagrant/_openshift_calvarado04_com/_openshift_calvarado04_com.key'} 


#Htpasswd
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/home/vagrant/htpasswd'}]


openshift_master_htpasswd_file=/home/vagrant/htpasswd
# Default login account: admin / handhand

openshift_disable_check=disk_availability,memory_availability,docker_storage,docker_image_availability
openshift_docker_options=" --selinux-enabled --log-driver=journald --storage-driver=overlay --registry-mirror=http://4a0fee72.m.daocloud.io "

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true','runtime=docker']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true','runtime=docker']}, {'name': 'node-config-infra-compute','labels': ['node-role.kubernetes.io/infra=true','node-role.kubernetes.io/compute=true','runtime=docker']}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true','runtime=docker'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

openshift_enable_service_catalog=true
template_service_broker_install=true

openshift_hosted_router_replicas=1

openshift_master_api_port=443
openshift_master_console_port=443
openshift_master_default_subdomain=openshift.calvarado04.com
openshift_master_cluster_public_hostname=cluster.openshift.calvarado04.com
openshift_master_cluster_hostname=master.calvarado04.com


openshift_template_service_broker_namespaces=['openshift']
ansible_service_broker_install=true
openshift_master_dynamic_provisioning_enabled=true

# host group for masters
[masters]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key"

[etcd]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key"

[nodes]
master.calvarado04.com openshift_ip=192.168.150.101 openshift_host=192.168.150.101 ansible_ssh_private_key_file="/home/vagrant/.ssh/master.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'master', 'node-role.kubernetes.io/master':'true'}"
node01.calvarado04.com openshift_ip=192.168.150.102 openshift_host=192.168.150.102 ansible_ssh_private_key_file="/home/vagrant/.ssh/node01.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'infra', 'node-role.kubernetes.io/infra':'true'}"
node02.calvarado04.com openshift_ip=192.168.150.103 openshift_host=192.168.150.103 ansible_ssh_private_key_file="/home/vagrant/.ssh/node02.key" openshift_node_problem_detector_install=true openshift_schedulable=True openshift_node_labels="{'region':'compute', 'node-role.kubernetes.io/compute':'true'}"

[nfs]
master.calvarado04.com

Don't forget to add your SSL certificates.

Gallery

Tags: , , ,

About: calvarado04