Incomplete job logs on Linux

fsmcore · September 4, 2021, 8:16am

Hello,

I’ve been using EI on both Windows computer and Linux computer. I used EI on windows first and all is fine but when I used it on Linux the logs are incomplete.

For example below is what I saw when using a windows computer (running model testing job)

Creating job… OK (ID: 1296187)
Generating features for Audio (MFE)…
Not generating new features: features already generated and no options or files have changed. Generating features for Audio (MFE) OK
Classifying data for Neural Network (Keras)…
Classifying data for float32 model…
Scheduling job in cluster…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Still waiting for job to be scheduled…
Job did not start within 120 seconds
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…

But below is what I saw when using Linux computer

Creating job… OK (ID: 1296187)
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…
Attached to job 1296187…

I am using Linux Ubuntu 18.04.5 LTS and Chromium browser. I’ve also tried using EI on Mozilla Firefox but the logs are also incomplete.
I’ve tried using EI from a mobile phone (Android) and it shows complete job logs just like the one on windows.
All jobs are affected except the dataset uploader. I’ve tried feature generation, model training, model testing, and deployment job. All of those have incomplete log on Linux.
The jobs actually run but the process between job creation and finish (or fail) are almost completely invisible. I can’t even see the failure reason if a job fails when using EI on Linux.

I believe this isn’t the intended behavior on Linux. Is there anything I can do as workaroud to this? Thank you!

janjongboom · September 4, 2021, 8:28am

Hi, we had an issue with routing events from jobs => Studio this morning (see also Features Generation Failure). I’ve recycled the services and this should work again.

fsmcore · September 4, 2021, 8:48am

Hello @janjongboom , thank you for the response. It seems like the jobs are working again now but the incomplete log issue I raised still persists.

I just ran model testing job on both Linux computer and my mobile phone and this is the log on Linux computer:

Creating job… OK (ID: 1296228)
Job completed

While this is the log when I ran it on my mobile phone:

Creating job… OK (ID: 1296233)
Copying features from DSP block OK
Copying features from processing blocks OK
Classifying data for float32 model…
Scheduling job in cluster… Job started
Classifying data for Neural Network (Keras) OK
Job completed

It’s not a major inconvenience on model testing or deploy job but it’s pretty annoying on feature generation or model training jobs because I can’t know for sure if the job is alive at all and both of those take a relatively longer time. It would be neat if this can be fixed.

As a note, it seems like job creation, “attached to job #”, and job completed/failed log always comes through. Is there a difference between those three and the rest of the log messages?

janjongboom · September 6, 2021, 11:35am

@fsmcore it’s interesting… Do you see anything on the browser console by any chance?

fsmcore · September 7, 2021, 8:20am

Now that I checked the browser console, there is something suspicious.

Screenshot taken from model testing page, verified that it’s happening on all project related pages including dashboard. The error counter goes up by 1 every few seconds. It’s only separated on the screenshot because I started a job.

Another screenshot taken from dashboard page:

EDIT: First screenshot wasn’t showing properly

janjongboom · September 7, 2021, 9:33am

Could this be a plug-in that blocks the websocket connection? All job messages are sent over websocket but job complete also goes over HTTP.

fsmcore · September 8, 2021, 1:38am

Hello, turns out my office proxy blocks websocket and when I disabled it EI logs becomes comprehensive. Thank you for the lead on checking browser log @janjongboom!