broken java networking

If your Java networking is broken and any traffic always
results in a "java.net.SocketException: Network is unreachable" error, then here comes a quick fix:

1. edit /etc/sysctl.d/bindv6only.conf
2. change value net.ipv6.bindv6only=1 into 0
3. invoke-rc.d procps restart

The story of the secure connection on Websphere 6.0.0

Once upon a time, our web application required to establish a SSL connection. Therefore we added JavaMail 1.4.1 JARs in our WEB-INF/lib directory, deployed the web application on Websphere 6.0.0 and experienced the application server was still loading its 1.3 version.

LESSON #1: use PARENT_LAST and PARENT_FIRST to determine whether the EAR/WAR file JARs are the last or the first to be loaded. A good explanation was found here.

Once we set PARENT_LAST on the Web Module, we ran into several XML casting related errors, such as: org.apache.xerces.parsers.XML11Configuration cannot be cast to org.apache.xerces.xni.parser.XMLParserConfiguration. That was because of the presence of unneeded JARs in WEB-INF/lib, eventually conflicting with Websphere ones, and previously bypassed by the PARTENT_FIRST option. Also in this care there's a lot of documentation and explanations, such as here, here, and here. However, what it had to be done was to clean up the unused JARs and/or substitute some of them with proper others.

LESSON #2: as a web application grows with time, it arrives the moment to stop and anlayze how many trash JAR it has collected. Therefore procede per steps:

1. remove all JARs;
2. if you are using the API of some suite, make sure you are using the required JARs;
   
3. add JARs one by one, from the most important to the last important, till all compinling errors go away;
4. go round you web application until you face some run time error, then add the remaining missing JARs. (Any suggestion on better ways is highly accepted!)
   

Now that the application can be started without errors and properly use JavaMail 1.4, we face a new error caused by:
javax.net.ssl.SSLException: Unsupported record version Unknown 111.109
at com.ibm.jsse2.a.b(Unknown Source)
at com.ibm.jsse2.a.a(Unknown Source)

Which seems to be due our current Websphere version (alas, still 6.0.0, while RAD 6.0.2 has it fixed). A fix is available here, even though I found it impossible to download APAR PK32916 by Fix Central. Given the fact it's impossible for us to upgrade WAS as of now, we have to find a workaround, but before that:

LESSON #3: should be needless to say, however better make sure your development team is using locally the same version of application server which is running on the test and production environments.

So, if it is impossible to use IBM JSSE API because they are old (as for JavaMail) and buggy, why not give it a try with SUN JSSE. I know it's dirty, however I extracted jsse.jar from Sun Java SDK 1.4 and added it to WEB-INF/lib. It resulted in a new error, which goes like:
java.net.SocketException: Export restriction: this JSSE implementation is non-pluggable.
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(DashoA12275)

Apparently the walkaround for this issue is reported here, and it says to patch org.apache.axis.components.net.JSSESocketFactory, and shows how. The reason is reported on Java 1.4 release notes, and goes like: "The JSSE implementation provided in this release includes strong cipher suites. However, due to U.S. export control restrictions, this release does not allow alternate "pluggable" SSL/TLS implementations to be used. For more information, please see the JSSE Reference Guide."

LESSON #4: you have to know when to stop even if you would like to carry on with the challenge. There are probably more critical things to deal with, so leave your ego apart.

We'll wait till we upgrade Websphere.

log4j causing OutOfMemory error

OutOfMemory Error on application server.

Thread dump analysis reported 15 threads active out of 650. The others were in wait. Inactive threads stack trace reported the following:

at java.lang.Thread.sleep(Native Method) at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java(Compiled Code))

In here http://www.theserverside.com/discussions/thread.tss?thread_id=27315 I found the following:

I found from the thread dump that lot of threads are created for log4j and not getting recycled.(see the thread dump below). I masked the log4j logging from my code. These log4j threads still get generated. Any ideas???

Thread Dump
"Thread-10" daemon prio=5 tid=0x113015d0 nid=0x838c waiting on monitor [0x133bf000..0x133bfdbc]
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)

SOLUTION [in short: log4j configureAndWatch() spreads lots of threads without deallocating them, eventually causing OOM exceptions]:

I have not worked on JBoss. I am having Weblogic Server. But I am pretty much sure that JBoss might let you do similar stuff.

1. After you have launched the server, run your application in a loop.
2. At the point where the server says that it cannot create new native threads, get a thread dump. In weblogic I did it by running the command :

java weblogic.Admin -url hostname:7001 -username admin -password mypassword THREAD_DUMP

3. Take a look at all the threads that are waiting on monitor. Chances are that 90% of the threads will be spawned by same process. In my case, it was

"Thread-14" daemon prio=5 tid=0x11999e90 nid=0x3388 waiting on monitor [0x135bf000..0x135bfdbc]
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)

I knew that this has something to do with log4j framework that I was using for logging.

4. So I started masking the logging lines, to see where this was being generated from. Eventually I found that the line
DOMConfigurator.configureAndWatch(fileFileName, logFileWatchTime)
was generating the threads that were not getting recycled. So I just went ahead and masked them.

Now everything works fine. It is going to be a different day I will be spending trouble shooting why this is happening. I was using this, because I liked the idea that by doing so, you could dynamically change log4j properties file and not restart the server.

But for now, I can leave with the idea of restarting the server if I change any configuration in log4j properties file. ;->

What i would suggest is that instead of looping 1000 times, just loop 20 times. Get a thread dump before you run your client and after you run youyr client. See what threads are created new, and if most of them are for same process, you need to see where they care created. It might be a bug in your code or third party library that you are using.