Why a server may fall, case II.

This case has no clear-cut solution, just a clarification:

Problem: Server crashed.

To be able to gather information, we had asked the customer to provide us with all the available log files – WebLogic log, the Standard Output, and the JVM log, to find the connections, and possibly – be able to identify the reason.

From its first lines, a JMV log file showed that it was a Java Virtual Machine crash, not WebLogic Server running on it:

# An unexpected error has been detected by HotSpot Virtual Machine:
#
# Internal Error (434F44452255464645520E435050005E 01), pid=15067, tid=8
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_11-b06 mixed mode)

As a number of other factors showed, a client had the Sun JVM installed – something beyond Oracle’s competency. We could only check whatever is was on the surface to see, it anything could trigger the crash.

We noticed the following line:

compacting perm gen total 63232K, used 63113K [0xf1800000, 0xf55c0000, 0xf9800000)
the space 63232K, 99% used [0xf1800000, 0xf55a25e0, 0xf55a2600, 0xf55c0000)

99% looked suspicious (which is true for any occasion), so we checked further:

jvm_args: -XX:MaxPermSize=128m -Xms1024m -Xmx1024m

Although unclear, shortage of memory could had been a reason of the crash. Just to make sure, we advised the customer to increase it to 512, or at least 256 mb.

In the end, we advices the customer to contact the Sun support, since they had a better cpecialty over their own Virtual Machine.

А вы чего, детектива ждали?

Why a server may fall, case I.

1. WebLogic Server have limited, definable number of threads allowed at the same time. A request sent to the server would have to wait, if all of the threads are currently busy. Assumingly, it should not ever happen.
2. Assumingly, once again, a routine request should come, be processed, and go, leaving a thread idle for the next request.
3. Now, a separate fully functional software entity is not just a server. It is also a database, an OS, some custom-built libraries, hardware, etc. All of them are communicating with each other, and with a server, on a millisecond basis.
4. Now, one of the problem, which may arise at any end of the system, at any point of time, and without a visible notice from the operator – is an event of one particular source start to send requests to the server, each of which would use one particular thread for an infinite time period. And another thread. And then – another. And another… Eventually, that one source can use all of the available threads with its requests, and force the server to crash, or restart itself, or do whatever is supposed to do in this case.
5. The worst part is that this event can have its origins from anywhere, any part of the system – database, custom libraries, OS after an update… An operator would never know, what caused the crash.

Solution:

Check the Thread dumps, – a series of them, generated with equal intervals from one another. Compare and see, which thread are doing what during that particular period – this way, you will be able to see if any request takes suspicious and may be the cause.

Configuring T3 on cluster

Can you please advice how can we configure T3 on cluster.
I am calling the service by using proxy service.