Our analytics infrastructure is hosted on Microsoft Azure and a large part of it is deployed to Azure Kubernetes Service (AKS).
I have been working on a scalable deployment of MedCAT for annotation on AKS as noted in a previous post.
It made sense to also be able to deploy MedCAT Trainer on AKS. I was unable to find a Helm chart and so have started working on one.
At the time of writing this post, this is the status:
SOLR pod works and the UI is reachable through port-forwarding
NGINX pod crashes with error messages as below. Looks like the nginx container is heavily customised and I wonder if it is just possible to use k8s nginx controller with modifications. I will explore this further when I get a chance.
MedCATTrainer pod does not pull the image successfully yet. But, seeing that the image pulled successfully to my laptop when using docker-compose, I suspect that this may simply be a timeout issue and should be easy enough to fix.
NGINX error message
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/03/20 12:20:34 [emerg] 1#1: host not found in upstream "mct_solr" in /etc/nginx/sites-enabled/medcattrainer:17
nginx: [emerg] host not found in upstream "mct_solr" in /etc/nginx/sites-enabled/medcattrainer:17
If someone has already built a working version, please let me know before I travel too far down this rabbit hole.
Else I will post updates as I go along. Contributions welcome.
The custom nginx image probably isn’t needed here, as its really just a sites-enabled and static-content mount which could be performed at runtime not build time. Hoping to get around to looking into this soon.
It took me a little while to figure out that the nginx container was acting as both ingress controller and static content server. I think I have successfully separated these functions and have moved the ingress control function to the Kubernetes nginx-ingress controller that I already have deployed in the cluster.
There is still a lot of tidying up to do including serving this from a subpath of a host domain.
@tomolopolis , MedCATtrainer uses sqlite3 behind the scenes. I have simulated 2 users annotating simultaneously (1 in incognito mode) and it seems to work. If this is the case, this is a really useful feature. Does MedCATTrainer support multiple simultaneous users out-of-the-box.
If we have several people annotating at the same time, then will be useful to leverage some of k8s autoscaling features. However, I don’t think there is an immediate use case for this yet - don’t think anyone has an army of annotators at their disposal!