Running S3 compatible endpoint to host Ops Manager binaries in Kubernetes
This post is part of our ongoing series on running Ops Manager with the MongoDB Kubernetes Operator. Check out the related articles for more.
Introducing MongoDB Ops Manager in
Kubernetes
Running MongoDB Ops Manager in Kubernetes
Recently, we announced the ability to deploy
Ops Manager in Kubernetes
and provided a step-by-step guide on how to set this up in your environment. Now we’re back to show you how to use an exciting new feature we’ve released: the ability to run an S3 compatible or HTTP(s) endpoint to serve the mongoDB binaries used to deploy clusters. This allows you to set up Ops Manager on a Kubernetes cluster without access to the internet and without the need to manage mongod/mongos binaries on each pod within the Ops Manager deployment.
We’ll show you how to configure your endpoint, upload the correct binaries and reconfigure the Ops Manager deployment to use files served from this endpoint.
Prerequisites
This guide assumes you’ve already set up and run Ops Manager and the MongoDB Enterprise Kubernetes Operator on your cluster, following the steps in the guide
here
. This will give you a deployed cluster that we can modify to use our new HTTP endpoint to retrieve the binaries needed to deploy and upgrade mongod/mongos instances.
If you have an existing environment, then you will need to have the following versions installed:
Kubernetes API
v1.16 or above for MongoDB Operator
v1.17 or above for MinIO Operator
MongoDB Enterprise Kubernetes Operator 1.6.0 or above
MongoDB Ops Manager 4.4.0 or above
Choosing Where to Host Binaries
The only requirement from Ops Manager is an HTTP endpoint, so you can leverage an existing file server configuration within your organisation if required.
For this tutorial, we’re going to use
MinIO
, a popular Open Source Object Storage platform that is optimised for private cloud deployments on containerised infrastructure such as Kubernetes, used in many Fortune 500 companies. With MinIO, customers can manage their binaries with individual policies through rich programmatic and UI admin services that align with best practices for file management such as bitrot protection, encryption, and identity management.
Installing MinIO
Installing MinIo is similar to the MongoDB Kubernetes Operator. The MinIO documentation
shows you how
to deploy on your Kubernetes cluster, with many configurations to suit your organization’s requirements. In this tutorial, we will show you how to use the Operator to manage the installation of your instance.
First, let’s ensure the operator is installed and configured by running the following command on your cluster:
kubectl apply -k github.com/minio/operator
Now, let's create the MinIO instances to serve our binaries:
kubectl apply -f https://raw.githubusercontent.com/minio/operator/master/examples/tenant.yaml
This deployment will take a few moments. Let's verify that the instance successfully deploys:
kubectl get pods
NAME READY STATUS RESTARTS AGE
minio-zone-0-0 1/1 Running 0 6m14s
minio-zone-0-1 1/1 Running 0 6m14s
minio-zone-0-2 1/1 Running 0 6m14s
minio-zone-0-3 1/1 Running 0 6m14s
Once this is complete, we should also get a service name, which we will use in the Ops Manager configuration later:
kubectl get svc -l v1.min.io/tenant=minio
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio-hl ClusterIP None <none> 9000/TCP 48m
Creating Buckets With MongoDB Binaries and Database Tools
Now that we have a running instance of MinIO, we can populate it with binaries for use by the operator to deploy mongoDB instances. The MongoDB Kubernetes Operator, as of
v0.10
, uses Ubuntu 16.04 mongoDB binaries, which can be downloaded from
https://www.mongodb.org/dl/linux/x86_64-ubuntu1604
. If Ops Manager will also be managing deployments, you may also wish to populate your buckets with versions for other architectures and platforms. A full list of the currently supported binaries can be retrieved from the version manifest of Ops Manager through the
Version Manifest
endpoint.
For the purposes of this tutorial, we will just fetch versions 4.2.3 and 4.2.0 for use by the operator:
curl -O http://downloads.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1604-4.2.3.tgz
curl -O http://downloads.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1604-4.2.0.tgz
Also, we’ll need the
mongodb-database-tools
for the current version of Ops Manager. For the 4.4.0 release, this is v100.0.2.
curl -O https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu1604-x86_64-100.0.2.tgz
Now, let's put the mongod files into a bucket named ‘linux’, and the database tools in a bucket named ‘tools/db’. Both buckets will have read-only permissions for all users. We can do this with the
mc
MinIO command-line client or through the graphical UI that MinIO provides. For this tutorial, we will use the UI, but the command line would allow us to script this operation if required.
To access the UI, we will need to forward a connection to the MinIO service running on kubernetes. In a terminal, run the following command:
kubectl port-forward svc/minio-hl-svc 9000:9000
You should now be able to access the MinIO dashboard at
http://localhost:9000
The default credentials, from the public sample config we deployed earlier, are
minio/minio123
.
Once we log in, we can create a bucket named ‘linux’ through the
+
→
Create bucket
button in the bottom right of the interface.
Once this is created, it will appear in the left menu. Select this and upload the binaries you downloaded earlier, again using the
+
→
Upload file
button in the bottom right.
Once all files are uploaded, you will need to allow read-only access to the bucket from anyone who has the link. This is to allow unauthenticated
GET
and
HEAD
HTTP requests to access the resource from our Ops Manager container.
To enable read-only access, click the
...
button on the ‘linux’ bucket and select Edit Policy. On the modal dialog, select
Read Only
and click
Add
, which will enable access.
At this point, we should be able to validate the files that can be downloaded. Running the following
HEAD
request should now return a successful result:
curl --head http://localhost:9000/linux/mongodb-linux-x86_64-ubuntu1604-4.2.3.tgz
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 132149028
Content-Security-Policy: block-all-mixed-content
Content-Type: application/gzip
ETag: "397903bcabe29abd9d1258653270e76f-1"
Last-Modified: Tue, 04 Feb 2020 17:44:10 GMT
Server: MinIO/RELEASE.2020-01-03T19-12-21Z
Vary: Origin
X-Amz-Request-Id: 15F0DF60120863EB
X-Xss-Protection: 1; mode=block
Date: Thu, 06 Feb 2020 17:11:12 GMT
Our S3 buckets are now ready to serve binaries for the Ops Manager instance. Let’s now configure it to consume these resources.
Upgrade Ops Manager to 4.4.0 or Later
In the previous tutorial, we created the ops-manager.yaml configuration file, which we will now edit to include the following options:
spec
version: 4.4.0 # This must be 4.4.0 or later to support the new HTTP endpoint as a source for binaries.
configuration
automation.versions.source: remote
# This sets the endpoint that will serve mongoDB binaries and was retrieved earlier when we fetched the service name for our MinIO instance.
automation.versions.download.baseUrl: http://minio-hl:9000
The updated configuration file should match the following:
apiVersion: mongodb.com/v1
kind: MongoDBOpsManager
metadata:
name: ops-manager
namespace: mongodb
spec:
# the version of Ops Manager distro to use
version: 4.4.0
# the name of the secret containing admin user credentials.
adminCredentials: ops-manager-admin-secret
configuration:
automation.versions.source: remote
automation.versions.download.baseUrl: http://minio-hl:9000
externalConnectivity:
type: LoadBalancer
# the Replica Set backing Ops Manager.
# appDB has the SCRAM-SHA authentication mode always enabled
applicationDatabase:
members: 3
version: 4.2.2
We can now update the running instance:
kubectl apply -f ops-manager.yaml
The deployment will take a few minutes to update, as it will need to redeploy the configuration with new pods. Check the status and ensure the Ops Manager resource gets to the “Running” phase:
kubectl get om -n mongodb
NAME REPLICAS VERSION VERSION (DB) STATE STATE (DB) AGE
ops-manager 1 4.4.0 4.2.2 Running Running 13m
Now, let’s connect to our Ops Manager instance through the UI Dashboard. Get the load balancer address, as with the previous tutorial:
kubectl get svc ops-manager-svc-ext -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Create a MongoDB Replica Set
At this point, everything should be in place to let us deploy mongoDB binaries using our MinIO instance. Let’s verify this, creating a new project and replica set using the Kubernetes Operator.
First, let’s make the replica set configuration Create the file replica-set-s3-endpoint.yaml describing the new MongoDB resource, reusing the ops-manager-connection configMap from the previous tutorial:
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
name: replica-set-s3-endpoint
namespace: mongodb
spec:
members: 3
version: 4.2.3
type: ReplicaSet
opsManager:
configMapRef:
name: ops-manager-connection
credentials: ops-manager-admin-secret
Apply it to the Kubernetes cluster:
kubectl apply -f replica-set-s3-endpoint.yaml
Wait until the resource enters the Running state:
kubectl get mdb -n mongodb
NAME TYPE STATE VERSION AGE
my-replica-set-s3-endpoint ReplicaSet Running 4.2.3 12m
Now, let’s verify this has deployed correctly in Ops Manager:
Also, if we click on the modify button and click the versions dropdown, we will see the listing is limited to 4.2.3 and 4.2.2; the versions we uploaded to the MinIO instance.
If the user configured a version that isn’t stored on the MinIO instance, we instead will get a deployment failure. The MongoDB Agent logs will give a clear error about which file and location could not be found on the remote endpoint, allowing admins to quickly identify where the files should be uploaded.
To test this, let’s change the version to 4.2.1 on the replica-set-s3-endpoint.yaml and deploy using:
kubectl -f apply replica-set-s3-endpoint.yaml
In Ops Manager, a red deployment bar will be displayed for the deployment. If we click on the view agent logs link, the following will now be shown :
Would you like to know more?
Stay tuned for upcoming blog posts in this series, where we will continue to do deep dives on all things Ops Manager. Learn about how to backup Ops Manager, configure it for high availability, set up SCRAM and get an inside look at the architecture behind the MongoDB Enterprise Kubernetes Operator.
August 28, 2020
