I’m documenting a recent issue that had caused much angst. Thankfully it is solved
The symptoms were extreme slowdown of very simple nifi processors - merge, jolt transforms etc. The slowdowns were somewhat intermittent and platform dependent. It appeared to be related to scenarios in which processors were writing or reading avro record types that were defined in cogstack.
The slowdowns were dramatic - processors that could handle the entire test dataset in a single refresh would only handle one record per refresh.
The underlying cause of the problem was an interaction between nifi’s logging and deployment in docker containers leading to very large logs being copied whenever there was a warning/error written to logs.
The fundamental problem was sharing the nifi-app.log file between host and container via a mount. The leads (on linux) to the inability of the log system to rotate the log files correctly. The symptom is the main log file never being emptied. i.e, after a rotation, triggered by file size or time, the incoming messages end up appended to the old content rather than being written to a new file. The performance problems are caused by repeated rotations (copying large files) whenever content is logged. Once the size threshold is exceeded the copying is triggered every time logging occurs as the log file is never emptied.
The simplest temporary solution is to not share the logfile via mounts.