When developing software it makes sense to be able to work on local files, while the source code should be served from a controlled environment (a container) to prevent pollution of the developer workstation.
In this article I will describe the evolution of a development workflow for deploying applications on OpenShift. The ultimate goal is to make it possible to maximize dev/prod parity, while minimizing the idle time in the change/test cycle.
OpenShift Source-To-Image makes it really easy to publish source code through container technology. Testing the resulting containers in OpenShift is easily done by using either oc cluster up or Minishift, but with the release of OpenShift 4 the first method becomes obsolete. This means that Minishift now is the go-to method for providing a local OpenShift environment for developers.
Note that I will use Python as the language of choice in this article, but a similar strategy can be followed when using other interpreted languages (Ruby, PHP, Perl). When using (byte) compiled languages (Java, GoLang), the build requirement for those makes using this strategy much harder to use.
Traditional local development
When using Python as the language, local development is typically done using
virtualenv and the Python modules being
used are then defined in requirements.txt. This happens to be the file used by
the Python autodetection for OpenShift s2i, so the example repository makes it
easy to activate a local environment with the correct Python requirements.
The following commands provide you with a local clone of the example code and a Python virtualenv using the default Python version from your workstation:
$ git clone https://github.com/pjoomen/hellopythonapp.git
Cloning into 'hellopythonapp'...
remote: Enumerating objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
Unpacking objects: 100% (32/32), done.
$ cd hellopythonapp
hellopythonapp$ virtualenv venv
New python executable in /Users/pjoomen/hellopythonapp/venv/bin/python2.7
Also creating executable in /Users/pjoomen/hellopythonapp/venv/bin/python
Installing setuptools, pip, wheel...done.
hellopythonapp$ source venv/bin/activate
(venv) hellopythonapp$ pip install -r requirements.txt
Collecting gunicorn (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Collecting Flask (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting Jinja2>=2.10 (from Flask->-r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from Flask->-r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Werkzeug>=0.14 (from Flask->-r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask->-r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->Flask->-r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/cd/52/927263d9cf66a12e05c5caef43ee203bd92355e9a321552d2b8c4aee5f1e/MarkupSafe-1.1.0-cp27-cp27m-macosx_10_6_intel.whl
Installing collected packages: gunicorn, MarkupSafe, Jinja2, itsdangerous, Werkzeug, click, Flask
Successfully installed Flask-1.0.2 Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 click-7.0 gunicorn-19.9.0 itsdangerous-1.1.0
We can now start a local (debug) web server:
(venv) hellopythonapp$ python wsgi.py
* Serving Flask app "wsgi" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 318-601-219
Run the following in a separate terminal to check the code using the curl
command (or use your preferred web-browser):
$ curl http://127.0.0.1:5000/
Hello Python World!
Using this setup, changes to the application (wsgi.py) are directly visible.
A new version of the application can be deployed to an OpenShift environment – without actually committing code – using:
hellopythonapp$ oc start-build hellopythonapp --from-dir=.
This method is easy to setup and maintain, but it has a few obvious disadvantages:
- The Python binary and the installed modules are not guaranteed to be identical as the ones being used in the containerized environments,
- The time-to-build for deployment to OpenShift is bound to become an annoying delay in the development cycle.
Adding Minishift into the mix
Minishift allows you to test your code easily against a locally running OpenShift, saving you for an (extra) external dependency. To install Minishift, follow the guidelines provided within the OKD documentation.
Now start your local Minishift installation:
$ minishift start
-- Starting profile 'minishift'
...
OpenShift server started.
The server is accessible via web console at: <!-- markdown-link-check-disable-next-line -->
https://192.168.64.6:8443/console
You are logged in as:
User: developer
Password: <any value>
To login as administrator:
oc login -u system:admin
Make sure you are logged in as developer and deploy and expose the sample application:
$ oc login -u developer
Logged into "https://192.168.64.5:8443" as "developer" using existing credentials.
You have one project on this server: "myproject"
Using project "myproject".
$ oc new-app https://github.com/pjoomen/hellopythonapp.git
--> Found image 7b379e8 (11 days old) in image stream "openshift/python" under tag "3.6" for "python"
...
--> Success
Build scheduled, use 'oc logs -f bc/hellopythonapp' to track its progress.
Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
'oc expose svc/hellopythonapp'
Run 'oc status' to view your app.
$ oc expose svc/hellopythonapp
route.route.openshift.io/hellopythonapp exposed
$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World!
Local changes can now be deployed by rebuilding the image from you working directory:
hellopythonapp$ oc start-build hellopythonapp -F --from-dir=.
Uploading directory "." as binary input for the build ...
.
Uploading finished
build.build.openshift.io/hellopythonapp-2 started
Receiving source from STDIN as archive ...
...
Push successful
This takes care of the aforementioned disadvantage of not having code parity, but the build-cycle is introducing a delay which introduces an idle-cycle on the part of the developer, and there is a limit on how often one can go for a cup of coffee.
Gunicorn and dynamic code changes
The following section is specific for Python code being served through
gunicorn, but similar practices hold true for other languages (and their
respective HTTP engines).
To enable gunicorn to reread changed code from the file-system, we need to
start gunicorn with a configuration file, containing the following:
debug = True
reload = True
Save this snippet in a file called config.py, put it into a ConfigMap and
make it available to the DeploymentConfig for your application. Then enable
gunicorn to use this configuration by adding an environment variable to the
DeploymentConfig:
$ cat <<EOF > config.py
debug = True
reload = True
EOF
$ oc create configmap gunicorn --from-file=config.py
configmap/gunicorn created
$ oc set volumes dc/hellopythonapp --add --mount-path=/etc/gunicorn --type=configmap --configmap-name=gunicorn
info: Generated volume name: volume-tkqvv
deploymentconfig.apps.openshift.io/hellopythonapp volume updated
$ oc set env dc/hellopythonapp APP_CONFIG=/etc/gunicorn/config.py
deploymentconfig.apps.openshift.io/hellopythonapp updated
We now are running gunicorn with debugging and reloading enabled. Open up a
terminal to be able to keep an eye on the logs:
$ oc logs dc/hellopythonapp -f
---> Serving application with gunicorn (wsgi) ...
[2018-11-09 15:04:31 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2018-11-09 15:04:31 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2018-11-09 15:04:31 +0000] [1] [INFO] Using worker: sync
[2018-11-09 15:04:31 +0000] [31] [INFO] Booting worker with pid: 31
[2018-11-09 15:04:31 +0000] [33] [INFO] Booting worker with pid: 33
[2018-11-09 15:04:31 +0000] [35] [INFO] Booting worker with pid: 35
[2018-11-09 15:04:31 +0000] [38] [INFO] Booting worker with pid: 38
Copying code changes into a running container
The oc binary includes a rsync verb which can copy (changes in) files from
your working directory to a running container. It can even watch for changes!
Note that there is no need to synchronize permissions between the source and the
target. There is also no need to copy the .git and venv folders.
Open up terminal and start the rsync process:
$ cd hellopythonapp
hellopythonapp$ POD=$(oc get pods -l app=hellopythonapp -o custom-columns=NAME:.metadata.name --no-headers)
hellopythonapp$ oc rsync --watch --no-perms --exclude .git --exclude venv . ${POD}:/opt/app-root/src
In your original terminal execute the following:
hellopythonapp$ sed -i 's/Hello Python World/& v2/' wsgi.py
hellopythonapp$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World v2!
The first line makes a change to the source file using sed, which is then
copied into the running container by the rsync process. The change will then
become visible in the output of the curl command.
You can of course use your favorite editor and web-browser to perform those tasks.
If you do this really quick, you might not see the change immediately as there
is a delay in the change being picked up by both the rsync and the gunicorn
processes.
Make sure you terminate the rsync process by pressing ^C in the terminal
where it is running, before proceeding with the following section. You now can
close this terminal.
Using a host-folder for local source code
Although the rsync method described above is quite convenient, there is a way
that is even more convenient, but a little bit more demanding to configure:
minishift hostfolder.
When using this method you do not have to worry about having the rsync process
running on your workstation. Once configured it is enough to have your Minishift
VM up and running! This method is for all practical purposes identical to the
way Vagrant makes local source code available using
a shared mount.
To setup the hostfolder do the following from your terminal:
hellopythonapp$ minishift hostfolder add hellopythonapp --source $(pwd) --target /srv/hellopythonapp
hellopythonapp$ minishift hostfolder mount hellopythonapp
hellopythonapp$ minishift ssh -- ls /srv/hellopythonapp
requirements.txt
venv
wsgi.py
For a container running under OpenShift to be able to read from this mount-point
we need to enable the virt_sandbox_use_fusefs SELinux boolean:
hellopythonapp$ minishift ssh -- sudo setsebool -P virt_sandbox_use_fusefs on
hellopythonapp$ minishift ssh -- getsebool virt_sandbox_use_fusefs
virt_sandbox_use_fusefs --> on
By default a container running in OpenShift is not allowed any access to the host it is running on. This can be fixed by changing the SecurityContextConstraints for the ServiceAccount running the container:
oc login -u system:admin
oc adm policy add-scc-to-user hostaccess -z default
oc login -u developer
Now the DeploymentConfig can be changed to use the mount-point within the VM as the source for the code used by the application:
$ oc set volume dc/hellopythonapp --add --type hostPath --mount-path /opt/app-root/src --path /srv/hellopythonapp
info: Generated volume name: volume-kdcgs
deploymentconfig.apps.openshift.io/hellopythonapp volume updated
Now make another change and test to see if this change propagates:
$ sed -i 's/v2/v3/' wsgi.py
$ curl $(oc get route hellopythonapp --template '{{ .spec.host }}')
Hello Python World v3!
Note that there still is a delay before the change is picked up by the
gunicorn process.
Conclusion
The above gives three alternative ways for testing your local changes before committing and pushing them:
- Internal debug web server for your environment,
rsyncverb from theocbinary,minishift hostfolder.
This allows for a more lenient code-build-test cycle and thereby preventing commit-log pollution, as well as having a more efficient workflow while working on new features, or fixing the incidental bug, for your application.