On Premise Admin Guide

NOTE Please use our more up-to-date documentation at https://www.notion.so/kubos/On-Premise-Admin-Guide-4b8d6ee9c8b54093a932cf38c0378ea7

System Requirements

Operating Systems

Please make sure that you’re using a supported operating system:

  • Ubuntu 16.04.3 - 16.04.5

  • Ubuntu 18.04 (recommended)

For legacy systems running on CentOS, please make sure that you are running one of the prior supported versions: CentOS 7.4 - 7.6. Also, you must disable SELinux and firewalld, as they are not compatible with Docker and Kubernetes. You will be prompted to do this during the installation process. 

Hardware

Expected Load

Hardware is variable because it depends on the number of systems (satellites and ground stations) you are running, and how much data they are inputting into Major Tom. It also depends on the number of remote clients (browser sessions and gateways, not running on the server) that will be simultaneously viewing/interacting with Major Tom. These requirements are formulated from testing with:

  • 3 Gateways

  • 5 Simultaneous Browser Sessions

  • 3 Systems 

  • 100 Measurements/system at 1 Hz (most impactful)

Base Requirements

  • ~2.5 GHz processor with at least 4 cores

  • 16 GB memory

  • 500 GB hard drive (highly recommend RAIDed SSD) 

File Transfer Requirements

For systems using File Transfer, we recommend increasing the hard drive space to match the expected file throughput you’re going to be experiencing. We keep files for 90 days by default (contact us to have this configured), so we recommend: 

  • Total hard drive size = (500 GB base hard drive) + (expected file size)*(expected number of files per day)*(90 days)*(1.10 factor of safety) 

Expanded Requirements

If you want to run at a higher expected load than what has been posted here, please discuss your expected load with the Major Tom team and we will estimate requirements for you. These system requirements are also estimated for a server environment, where Major Tom is the only thing running on the server, and all clients/gateways are interacting with it remotely. We do not recommend using the server to host your Major Tom Gateway, regular client browsers, or other software. 

Installation

Important Notes Before Installation

Please make sure that the Chrome browser is installed on a machine that can access the server (for verification). It will also need to be installed on all machines that will be using Major Tom. 



For any server that is going to move locations (change it’s IP), you must install Docker before starting the installation process (see https://docs.docker.com/install/). You must also enable the docker service after installing with: sudo systemctl enable docker

Airgapped Installation

Follow this guide if you are installing Major Tom on a server that will have it’s internet access revoked, or can never touch the internet.

Install Replicated

Download the install package:

1 curl -OJ 'https://s3.amazonaws.com/replicated-airgap-work/replicated__docker__kubernetes.tar.gz' 

Download link: https://s3.amazonaws.com/replicated-airgap-work/replicated__docker__kubernetes.tar.gz 

Transfer the package to the server (if you did not download it on the server), and run in the same directory:

1 tar xzvf replicated__docker__kubernetes.tar.gz
1 cat ./kubernetes-init.sh | sudo bash -s airgap disable-contour

The airgap and disable-contour options are both important.

If you are going to install this machine in a different location, you must already have docker installed and select the docker0 ip address. See the link below and the Note at the beginning.

https://help.replicated.com/community/t/fixing-the-ip-of-a-single-node-kubernetes-installation/297 

Once the script completes successfully, it should ask you to visit a URL like http://<server-address>:8800 to finish the installation. You’ll visit this when you go to install Major Tom. If it does not complete successfully, please contact Major Tom support. 

Prep for Airgapped MT Install

  1. Retrieve the Airgap application bundle download link from the Major Tom team along with the password. (Note for MT Team member: It is retrieved by visiting the customer’s management page on Replicated under the License Options section.)

  2. Retrieve the license file from the Major Tom team (Note for MT Team member: download this from the same customer management page with the link in the top right. Remember to enable Airgap on the license.)

  3. Transfer the .airgap file to the target machine and make a note of the absolute path of this file.

  4. Install Airgapped MT

    1. Visit the Replicated Admin console (<server>:8800) to begin the installation of Major Tom on the machine.

    2. Upload SSL certificate or chose default one

    3. Upload the license.

    4. Choose Airgapped install

    5. Provide the local absolute path of the .airgap file you noted earlier

  5. Start using Major Tom

    1. Visit Major Tom: 

      1. Either click the “Open” link on the admin console dashboard, or visit the URL directly

      2. Should take you to the login page

    2. Login to Major Tom

      1. Email that you provided during the installation process

      2. Revisit the Replicated settings page under the admin panel to get your generated user password

    3. That’s it! 

Networked Installation

Follow this guide if you are installing Major Tom on an internal server that will continue to have access to the internet.

  1. Install Replicated and follow the command line instructions 

    1. Run: curl -sSL https://get.replicated.com/kubernetes-init | sudo bash -s disable-contour

Please note the important disable-contour option.

Note: If you are going to install this machine in a different location, you must already have docker installed and select the docker0 ip address. See the link below and the Note at the beginning.

https://help.replicated.com/community/t/fixing-the-ip-of-a-single-node-kubernetes-installation/297 

Note: Please follow any recommendations the command line interface requests unless you understand them and have some specific reason you cannot. 

  1. Make a note of the Replicated Admin Console page it asks you to visit after the script completes, most likely: <server>:8800

  2. Install Major Tom

    1. Visit the Replicated Admin Console Page (noted earlier) 

    2. Follow the instructions provided on the console page, providing all required information. 

    3. Upload the license.

    4. Choose Online Install, if prompted.

  3. Start using Major Tom

    1. Visit Major Tom: 

      1. Either click the “Open” link on the admin console dashboard, or visit the server directly

      2. Should take you to the login page

    2. Login to Major Tom

      1. Email that you provided during the installation process

      2. Revisit the Replicated settings page under the admin panel to get your generated user password

    3. That’s it! 

Moving an installation to a new IP

Warning: If you do not perform this procedure, it could cause the database to be corrupted and force a full recovery of the machine, which requires a reinstall of both the application and the OS.

  1. Make sure you followed the notes in the original installation for using Docker’s IP rather than the default IP. 

  2. Follow the Prepare for Shutdown procedure in the Shutting Down/Restart the Server section.

  3. Move the machine to desired install location and power it up, making sure it’s routable on the network.

  4. Follow the Bringing Back Online procedure in the Shutting Down/Restart the Server section.

    1. If there are no issues with this, you’re good to go! 

    2. If there are issues with reaching the Admin panel during that procedure, the internal IP of the box probably changed. We need to make sure the hostname of the server matches what kubernetes expects: 

      1. Find the hostname that kubernetes expects:
        kubectl get nodes

And copy the NAME field. For example:

1 2 3 NAME ip-172-31-25-85.us-east-2.compute.internal
  1. Change the hostname of the server to match kubernetes. Run: 

sudo hostnamectl set-hostname {hostname}

For example:

1 sudo hostnamectl set-hostname ip-172-31-25-85.us-east-2.compute.internal
  1. Restart kubernetes: (this will take a few minutes to come back online) 

1 sudo systemctl restart kubelet
  1. Recheck the Replicated admin panel on port 8800. If it’s unavailable, you’ll have to contact Major Tom support. 

  2. Check to see that Major Tom is running on the Replicated Admin Panel (as pictured on the right below). If not, click “Start Now” (as pictured on the left below) and wait until it shows that Major Tom has “Started”. This may take a few minutes.  

                 

  1. That’s it! 

Admin Panel

How to get to it

The admin panel for Major Tom always runs on port 8800 of the server running Major Tom. 

Reset Admin Password

If you’ve misplaced/forgotten the password for the admin panel, you’ll need sudo access to the server running Major Tom. Steps to reset:



  1. Open a terminal on the server running Major Tom. 

  2. Run this command: sudo replicatedctl console-auth reset

    1. This removes the current admin password

  3. Visit this page to create a new password: https://<server>:8800/create-password

Log Rotation

To prevent Docker container logs from getting too large, we strongly recommend enabling log rotation.

Create or edit /etc/docker/daemon.json and set it to the following:

1 2 3 4 5 6 7 {   "log-driver": "json-file",   "log-opts": {     "max-size": "100m",     "max-file": "5"   } }

Restart the docker daemon: sudo systemctl restart docker

Storage Management

When you set up the Major Tom appliance, you can configure the allocated disk space for:

  • uplinked/downlinked files (Minio)

  • database storage (Postgres)

  • telemetry storage (Influx)

You cannot change these values later, so it is important to keep an eye on your disk usage. You can check how much space you've used by running df inside each pod. For example:

kubectl exec major-tom-onprem-db-0 \
        -n replicated-37af422e1fda418d68e105b7c924a578 -- df -hT
kubectl exec major-tom-onprem-grafana-db-0 \
        -n replicated-37af422e1fda418d68e105b7c924a578 -- df -hT
kubectl exec major-tom-onprem-influxdb-0 \
        -n replicated-37af422e1fda418d68e105b7c924a578 -- df -hT
kubectl exec major-tom-onprem-minio-0 \
        -n replicated-37af422e1fda418d68e105b7c924a578 -- df -hT
kubectl exec major-tom-onprem-redis-0 \
        -n replicated-37af422e1fda418d68e105b7c924a578 -- df -hT

(You may need to change the -n option to match the namespace in your own installation.)

Firewall Rules

If you wish to run a firewall, it needs to be running at the network layer, not on the host that is running Major Tom. The following ports need to be open to the computer hosting Major Tom:

80

http for Major Tom

8800

Replicated admin panel (you may wish to restrict this to certain IPs)

 

You also probably want to continue to have SSH access (port 22) to the host.

Upgrades

Upgrading Major Tom

NOTE: Major Tom will have a few minutes of downtime when upgrades are performed.

Airgapped Installation

  1. Visit the download URL (unique to your instance) and input the password given to you by the Major Tom team (you can ask for an updated one if necessary). 

  2. Download the latest image from the interface.

    1. Stable Channel: Generally the one you want to run, we’ve load tested it and verified it’s performance. Releases are infrequent. 

    2. Beta Channel: If you want the latest features, despite the occasional hiccup, you can run the beta images. We will regularly release these. 

  3. In your Replicated control panel, visit Console Settings under the gear menu, note the Airgapped Settings Update Path, and upload the new .airgap file to that location on the server.



  1. When uploaded, check for update again by hitting “Check Now” on the Dashboard.

Networked Installation

  1. When an update is available, you should see “There is an update available.” in the Replicated admin console and can click to view and install it.

    1. Stable Channel: Generally the one you want to run, we’ve load tested it and verified it’s performance. Releases are infrequent. 

    2. Beta Channel: If you want the latest features, despite the occasional hiccup, you can run the beta images. We will regularly release these. 

Upgrading Replicated

Airgapped Installation

  1. Airgap installations can be upgraded by downloading a newer version of the Replicated release, uncompressing it, and re-running the install script.



  1. Download the install package: 

    1. Command line: curl -OJ ‘https://s3.amazonaws.com/replicated-airgap-work/replicated__docker__kubernetes.tar.gz’

    2. Download link: https://s3.amazonaws.com/replicated-airgap-work/replicated__docker__kubernetes.tar.gz



  1. Transfer the package to the server (if you did not download it on the server already)



  1. Run in the same directory: 

tar xzvf replicated__docker__kubernetes.tar.gz



  1. Run: 

cat ./kubernetes-init.sh | sudo bash -s airgap disable-contour

(Please note that the airgap and disable-contour options are both important.)



  1. If an upgrade of Kubernetes is required, the script will begin the upgrade and prompt to run upgrade scripts. Major Tom will have a few minutes of downtime when Kubernetes upgrades are performed.

Networked Installation

  1. Re-run the install script to upgrade Replicated to the latest version.

curl -sSL https://get.replicated.com/kubernetes-init | sudo bash -s disable-contour



  1. If an upgrade of Kubernetes is required, the script will begin the upgrade and prompt to run upgrade scripts. Major Tom will have a few minutes of downtime when Kubernetes upgrades are performed.

User Management

Managing users logging in to Major Tom. 

Preparing for Access

All user management requires terminal access to the machine. To do any user management, we first need to record the NAMESPACE and NAME to be able to access the appropriate management script. Here are the steps to obtain these: 



  1. Open a terminal or shell on the machine hosting Major Tom

  2. Run: kubectl get pods --all-namespaces

  3. In the printout from that command, there should be a few columns with several rows of data that are returned.

  4. In the NAMESPACE column, look for something similar to: replicated-randomalphanumericstring and make a note of it. There will be several rows grouped together that all have this NAMESPACE. 

  5. In the NAME column, look for something similar to: major-tom-onprem-web-randomalphanumericstring and make a note of it.

Note

NAMESPACE and NAME can and will change frequently over time, so these steps must be followed each time.

Password Reset

If a user forgets or loses their password, you can reset their user password. After following the steps to “Preparing for Access” above, perform the following steps: 

  1. Taking NAMESPACE and NAME, run this command with those substituted in: kubectl --namespace=NAMESPACE exec -it NAME --container rails -- bundle exec rails runner tools/change_user_password.rb

  2. Follow the prompts given by the script to reset your password, and you’re done! 

Add New User

You can add new users to Major Tom as needed. Make sure you have the users First and Last names, as well as their email. After following the steps to “Preparing for Access” above, perform the following steps: 

  1. Taking NAMESPACE and NAME, run this command with those substituted in: kubectl --namespace=NAMESPACE exec -it NAME --container rails -- bundle exec rails runner tools/add_user.rb

  2. Follow the prompts given by the script to reset your password, and you’re done! 

Shutting Down/Restarting the Server

Warning: If you do not perform this procedure, it could cause the database to be corrupted, and force a full recovery of the machine, which requires a reinstall of both the application and the OS. 

Prepare for Shutdown

  1. You’ll need terminal access to the server. Open a terminal on the server to perform the following steps. 

  2. Run: replicatedctl app stop --attach

  3. Run: kubectl scale deployment replicated replicated-premkit retraced-postgres --replicas=0

  4. Wait for pods to terminate 

    1. When kubectl get deploy shows replicated and replicated-premkit as 0/0 Ready. 

  5. Shutdown the machine

Bringing Back Online

  1. Bring Replicated back online: 

    1. Run: sudo systemctl restart docker

    2. Run: kubectl scale deployment replicated replicated-premkit retraced-postgres --replicas=1

  2. To restart Major Tom, visit the Admin Console (https://<server-address>:8800) and click the “Start Now” button:

  3. Major Tom is now up and running! 

Snapshots (Beta)

When running Major Tom on prem, backups are not automatically created unless you set it up for your deployment. Backups are called Snapshots in the Admin Console. All management of these Snapshots is accomplished through the Admin Console (https://<server_address>:8800).

Creating Snapshots

Manually

Snapshots can be manually triggered through the card on the right side of the main dashboard: 

You can trigger a snapshot by hitting the “Start Snapshot” button. Please do not click this multiple times, or you will see an error. Only one snapshot can be running at a time, and the time to complete varies greatly by the quantity of data you have in the system. 

Automatically

To create Snapshots automatically, visit the Console Settings page in the Admin console:

Go to the “Snapshot & Restore” section, and check the box for “Enable Automatic Scheduled Snapshots”, choosing the appropriate frequency for your needs.

We recommend, as stated under the “Snapshot File Destination” section, that the snapshots be stored on a remote destination so they can be recovered in the event that the machine hosting Major Tom is unrecoverable. 

Viewing/Managing Snapshots

Snapshots can be found under the gear icon in the top left of the Admin panel: 

There you can view what snapshots have been taken and delete them if necessary. 

Restoring from a Snapshot

  1. To restore from a snapshot you need to create a fresh install of replicated. To do this, follow the instructions in Step 1 only of the appropriate installation section (Airgap or Network).

  2. Before running the web console at https://<server_address>:8800, place a copy of the full snapshot directory on the new host under /var/lib/replicated/snapshots, completely replacing the existing directory. The source directory is listed on the original host in the “Console Settings” page under the “Snapshot & Restore” section:

    1. Assuming everything is in default locations, you probably want to create a tar.gz file out of the source snapshots directory on the original host:

      1. tar -czvf snapshots.tar.gz -C /var/lib/replicated snapshots

    2. Then, on the new host, you want to remove the empty directory structure and replace it with the snapshot from the original host:

      1. sudo rm -rf /var/lib/replicated/snapshots/*

      2. tar -xzvf snapshots.tar.gz && sudo cp -r snapshots/* /var/lib/replicated/snapshots/

  3. Proceed through the https setup screen, but on the upload your license page, click the "restore from a snapshot" link:



  1. Choose snapshot to restore:

    1. Enter the path on the host where you have copied the snapshots folder,

    2. Click “Browse snapshots”

    3. Locate the latest version you would like to backup from and click the “Restore” button.



  1. You will be given options for restoring, downloading the volumes, or deleting from the prior install, in this case we will restore to the local host by clicking the “restore” button.



  1. Wait for the restore to complete and the dashboard to come up at: https://<server_address>:8800 

  2. Once the dashboard is up, make sure the hostname is correct in the settings. It will preserve the hostname from the previous host, and so you likely need to change it. 

  3. Make sure the app has successfully started (you may need to wait, refresh the dashboard, "Apply Changes", and refresh again). 

  4. Use the terminal, run bash -l and then replicated admin copy-restored-files and wait for it to complete.

  5. Click "Stop now" on the dashboard and wait for the app to stop.

  6. Once the app has stopped (or says "Error" and "Some containers have stopped unexpectedly."), start the app again.

  7. Once startup is complete, your backup should be restored. Verify this by logging into Major Tom. 

  8. Be sure to keep your backup snapshots until you've confirmed a complete restore. If you have any issues, please send us a Support Bundle.

SSL Certificates

Kubernetes uses SSL certificates internally to authenticate to the API. They expire after one year. As of Replicated 2.43.0, these certificates are renewed automatically before they expire. But in case you need to troubleshoot this by hand, here is some more info:

The certificates are found in /etc/kubernetes.

There are server certificate and key files in pki.

The client certificates are found in the *.conf files. The *.conf files are used by kubectl and other k8s tools. Your $KUBECONFIG should point to /etc/kubernetes/admin.conf for example. Most certs are embedded directly in these files, but some *.conf files may not embed a cert directly and instead reference a separate file. For example kubelet.conf may point to /var/lib/kubelet/pki/kubelet-client-current.pem.

You can examine a cert by saying:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

And you can see the expiration date with:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text | grep 'Not After'

Regenerating client certs


Before regenerating certs, you should stop Major Tom: 

replicatedctl app stop

You can regenerate all client certificates with these commands:

sudo rm /etc/kubernetes/*.conf
sudo kubeadm init phase kubeconfig all

Or you could regenerate just the kubelet cert (for example) like this:

sudo rm /etc/kubernetes/kubelet.conf
sudo kubeadm init phase kubeconfig kubelet

Regenerating server certs

Before regenerating certs, you should stop Major Tom:

replicatedctl app stop

You can check the expiration time of server certs by saying:

sudo kubeadm alpha certs check-expiration

And you can renew them with:

sudo kubeadm alpha certs renew all

If that doesn't work for some reason, you can also regenerate certs in /etc/kubernetes/pki similarly to client certs, but you must point kubeadm at a different kubeconfig file:

sudo rm /etc/kubernetes/pki/apiserver.*
sudo kubeadmin init phase certs apiserver --config /opt/replicated/kubeadm.conf