[astounding]

THIS HAS BEEN RESOLVED. THANKS TO ALL. An unneeded service was terminated and upgrading clustered servers to Quad Intel Dual-Core Xeons. That'll nicely do the trick for sure. What I learned is that even though a service can show that's it's taking almost no CPU, it can be the biggest resource killer. So play around with killing background scripts and monitor how it affects your server load. At least until you find which service has the biggest impact on your load. Then you can optimize it.


Hello,

I have a really challanging task of knowing why and host to fix the situation
where at most times the CPU is totally idle and same for IO activity, yet the
load at most times is 16-40+.

For instance at time of writing this: The load is 16 now and linux says:

avg-cpu: %user %nice %sys %iowait %idle
0.49 0.40 0.90 0.00 98.20

(That's a whopping 98.20 idle state!)


How could that possibly be? And could you wonderful folks attempt to offer
me advice on how to find out why load is so high yet almost totally idle activity.

If you need various commands for me to run for you to see more info, let
me know.

Thanks.
Andre

[benoitb]

That is astounding and rather unfortunate. I really think you should reformat.
With a load like that I would be surprised that pages would even load.

What company do you run? Is this Andre from 110mb.com?

I'm trying to trouble shoot this problem conceptually. Could you tell me if this : http://www.110mb.com/forum/index.php/topic,4874.0.html is related?

[peterz]

Possible your server could be hacked. Do you have any intrusion detection tools to check with?

[gryffin]

i posted an intrusion tool here (rkhunter) before realising that there was two topics, this tool may be useful http://www.freewht.com/showthread.php?p=17985&posted=1

[astounding]

I really don't think it's a hack of any kind because the new clustered server was only activated yesterday. Got 2K members since and already it's giving performance problems.

Yet CPU is almost 100% idle. How?!!

[benoitb]

and you say the IO is at Zero? Paste what you see when you look at the TOP command.

[jcink]

What was the un needed service out of curiosity?