I once spent several hours debugging a performance issue in the Linux builds of a project.
The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux.
Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.
It turned out the problem was in the program’s main loop, which was OS-specific and called something like sleep(float seconds) (yes, I know…).
The parameter was set to 1e-3 on Windows and 10e-3 on Linux.
I stared at that line several times before I noticed the difference.
Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM.
The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.
Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded).
Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.
Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.
I once spent several hours debugging a performance issue in the Linux builds of a project. The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux. Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.
It turned out the problem was in the program’s main loop, which was OS-specific and called something like
sleep(float seconds)(yes, I know…). The parameter was set to1e-3on Windows and10e-3on Linux. I stared at that line several times before I noticed the difference.Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM. The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.
Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded). Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.
Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.