MedCAT as service without docker

It’s odd that it doesn’t show a stack trace along with the terminated worker.
You could try adding --log-level debug command line argument to force gunicorn to use debug log level.

Without seeing the stack trace it’s really difficult to judge what the issue is.

You could also try to install another version of medcat to see if it works on another version. But I don’t really see why that should result in a different outcome.

Reading up on gunicorn respawning without a trace of what went wrong on Google, it is mentioned that when other (e.g. OS related) causes kill the worker process , it may be that gunicorn just notices the worker was killed and respawns a new process. I will ask the systems admins of the cluster that I am on if they have implemented some settings that do not fit our use case. I can not read all the system logs here.
Is the entire model loaded into RAM (vocab + cdb ~ 1.2 GB)?

With --log-level debug no more information at the time of the crash. Perhaps you @mart.ratas see something in the start up phase?
I will contact the system admins to see if there is anything in their log or settings.

(mdcservice) [tgwelter@node012 MedCATservice]$ ./
threads = 4
Starting up Flask app using gunicorn server ...
[2023-12-11 11:20:09 +0100] [2900978] [DEBUG] Current configuration:
  wsgi_app: None
  bind: ['']
  backlog: 2048
  workers: 1
  worker_class: sync
  threads: 4
  worker_connections: 1000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 300
  graceful_timeout: 30
  keepalive: 2
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: False
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  print_config: False
  preload_app: False
  sendfile: None
  reuse_port: False
  chdir: /trinity/home/tgwelter/MedCATservice
  daemon: False
  raw_env: []
  pidfile: None
  worker_tmp_dir: None
  user: 1165
  group: 1165
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  forwarded_allow_ips: ['']
  accesslog: -
  disable_redirect_access_to_syslog: False
  access_log_format: %(t)s [ACCESSS] %(h)s "%(r)s" %(s)s "%(f)s" "%(a)s"
  errorlog: -
  loglevel: debug
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  proc_name: None
  default_proc_name: wsgi
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x1555465782c0>
  on_reload: <function OnReload.on_reload at 0x155546578400>
  when_ready: <function WhenReady.when_ready at 0x155546578540>
  pre_fork: <function Prefork.pre_fork at 0x155546578680>
  post_fork: <function post_fork at 0x15554657a980>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x155546578900>
  worker_int: <function WorkerInt.worker_int at 0x155546578a40>
  worker_abort: <function WorkerAbort.worker_abort at 0x155546578b80>
  pre_exec: <function PreExec.pre_exec at 0x155546578cc0>
  pre_request: <function PreRequest.pre_request at 0x155546578e00>
  post_request: <function PostRequest.post_request at 0x155546578ea0>
  child_exit: <function ChildExit.child_exit at 0x155546578fe0>
  worker_exit: <function WorkerExit.worker_exit at 0x155546579120>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x155546579260>
  on_exit: <function OnExit.on_exit at 0x1555465793a0>
  proxy_protocol: False
  proxy_allow_ips: ['']
  keyfile: None
  certfile: None
  ssl_version: 2
  cert_reqs: 0
  ca_certs: None
  suppress_ragged_eofs: True
  do_handshake_on_connect: False
  ciphers: None
  raw_paste_global_conf: []
  strip_header_spaces: False
[2023-12-11 11:20:09 +0100] [2900978] [INFO] Starting gunicorn 20.1.0
[2023-12-11 11:20:09 +0100] [2900978] [DEBUG] Arbiter booted
[2023-12-11 11:20:09 +0100] [2900978] [INFO] Listening at: (2900978)
[2023-12-11 11:20:09 +0100] [2900978] [INFO] Using worker: gthread
[2023-12-11 11:20:09 +0100] [2900979] [INFO] Booting worker with pid: 2900979
[2023-12-11 11:20:09 +0100] [2900979] [INFO] Worker spawned (pid: 2900979)
[2023-12-11 11:20:09 +0100] [2900979] [INFO] APP_CUDA_DEVICE_COUNT device variables not set
[2023-12-11 11:20:09 +0100] [2900978] [DEBUG] 1 workers
[2023-12-11 11:20:33 +0100] [2900979] [DEBUG] POST /api/process
[2023-12-11 11:20:33,165] [DEBUG] MedCatProcessor: APP log level set to : DEBUG
[2023-12-11 11:20:33,165] [DEBUG] MedCatProcessor: MedCAT log level set to : DEBUG
[2023-12-11 11:20:33,165] [INFO] MedCatProcessor: Initializing MedCAT processor ...
[2023-12-11 11:20:34,430] [INFO] MedCatProcessor: Loading model pack...
[2023-12-11 11:20:48,652] [WARNING] medcat.cdb: You have MedCAT version '1.7.3' installed while the CDB was exported by MedCAT version '1.3.0',
which may or may not work. If you experience any compatibility issues, please reinstall MedCAT
or download the compatible model.
[2023-12-11 11:20:57 +0100] [2900978] [WARNING] Worker with pid 2900979 was terminated due to signal 9
[2023-12-11 11:20:57 +0100] [2901139] [INFO] Booting worker with pid: 2901139
[2023-12-11 11:20:57 +0100] [2901139] [INFO] Worker spawned (pid: 2901139)
[2023-12-11 11:20:57 +0100] [2901139] [INFO] APP_CUDA_DEVICE_COUNT device variables not set

Yes, the entire model will be loaded into memory. But that shouldn’t really be an issue with the amount of memory you’ve got, should it?

With that said, since this is a HPC cluster, they might not want you to run heavy applications on the front end (i.e without running through the workload management system).
If that’s what you’ve been doing, you could try doing the same things in an interactive job environment.

Nothing that really pops out to me.

Thanks, I contacted the sysadmins. They did not mention that my login environment was restricted in anyway but we will see. Indeed, memory should certainly not be a problem.

@mart.ratas It appears to have to do with the cluster settings because the service runs on my laptop without problem. Thanks for the help, I will let you know when it runs on the cluster.

@mart.ratas It was indeed a cluster setting that put a limit on memory usage. Increasing memory for the session resolved the issue and it is working fine now.

1 Like