Dinosaur Technology and Trading

1. Mark
   Member

   Hey, Karl,

   I’ve been successfully running your socket client for data for several weeks, and a separate instance of your socket client for order processing successfully for over a week. However, at 1:24 a.m. Eastern time today the order processing client submitted an order which caused the order processing client to crash. The error was “1162: Error processing FIX msg” -- I’ve never seen this error before, and I’ve used the ActiveX control for order processing for several years (the ActiveX control uses the same error handlers as the socket).

   Here’s the relevant listing from the TWS log. I’m sending this to you to see if you know how I can handle this type of error so that the socket client for order processing doesn’t crash. I have implemented the Error event handler, but that handler didn’t pick up any messages at that time. I also have the OrderStatus event handler, but that also did not pick up any events before the crash.
   _____

   SN 01:24:03:825 JTS-EServerSocket-53: [10:31:39:1:0:0:0:ERR] Socket connection for client{10} has closed.
   SN 01:24:03:825 JTS-EServerSocket-53: Connection reset
   SN 01:24:03:825 JTS-EServerSocket-53: ArEServer: Removing 31187401 with id 10
   SN 01:24:03:839 JTS-CCPDispatcher-1162: Error processing FIX msg
   SN java.lang.NullPointerException
   at trader.order.v.a(v.java:932)
   at trader.order.v.a(v.java:870)
   at jclient.mf.b(mf.java:3473)
   at jclient.mf.a(mf.java:3366)
   at jclient.mf.a(mf.java:3353)
   at jclient.mf.a(mf.java:3322)
   at jclient.mf.a(mf.java:490)
   at jclient.wf.a(wf.java:650)
   at jclient.fo.a(fo.java:207)
   at jclient.fo.a(fo.java:166)
   at jclient.cm.a(cm.java:327)
   at jclient.mf.d(mf.java:3258)
   at jconnection.ab.a(ab.java:82)
   at jconnection.y.a(y.java:124)
   at jconnection.y.a(y.java:155)
   at jconnection.y.a(y.java:77)
   at jconnection.y.d(y.java:33)
   at jconnection.x.run(x.java:47)

   SN 01:24:03:843 JTS-EServerSocket-53: [10:31:39:1:0:0:0:SYS] Ending conversation with client{10} at localhost/127.0.0.1

   _____

   The order was received by TWS because the order line showed up, but TWS did not execute the order.

   I notice that error code 1162 is not listed on the API Message Codes on the IB website. Also, I notice that IB has a new event, winError, that might handle errors not handled by the error event. However, I don’t know why this error would cause the socket client for order processing to crash, and I have no debug trace in a case like this to tell me which event caused the crash. I would think the error event handler would just report the error, even though it’s not listed.

   I’d appreciate any ideas you have on how to handle this type of error.

   Thanks very much again for your great work.

   Mark

   Posted 6 months ago #

2. Karl
   Key Master

   Hey Mark,

   What version of TWS are you running with this? I too have been running this in data collection for months, and for placing and tracking orders (though only for 12 hours at a time), and have never seen this.

   The only thing I can think of is that TWS hiccuped - namely, TWS uses a varient of the FIX protocol to communicate with the IB servers, and since TWS received your order, but failed to send it, it then threw this exception. I am not sure why it disconnected the socket though. Have you submitted this to IB's help desk?

   Let me know about the versions, and I will think about if there is anything in the code that could catch this,

   -Karl

   Posted 6 months ago #

3. Mark
   Member

   Hey, Karl,

   Thanks for the reply.

   I’m using Build 881.7 (Jan 24 2008). I agree that it looks like a hiccup, but I’d love to know how to handle it if it ever occurs again. I’ll see if the IB API group has any ideas, and let you know.

   You may want to consider including the winError event in your next release. Here’s their description:

   virtual void winError(const CString &str, int lastError)
   Parameters
   str - this is the error message text
   lastError - the error code returned by GetLastError()
   Notes
   This event is called when there is an error on the client side.

   It’s not clear from that description, but it may be the best thing to handle errors not handled by any other event.

   Mark

   Posted 6 months ago #

4. Karl
   Key Master

   Hello Mark,

   The winError function is just an exception handler for local errors. When you were running - what exception was thrown that shutdown the client?

   -Karl

   Posted 6 months ago #

5. Mark
   Member

   Hi, Karl,

   No exception was thrown, not even an ASP.NET error. None of the logs picked up any events. I suspect this was on the order of a general system error. In addition, here's the reply I received from IB API support:

   "In analyzing the log file, before receiving the error, there was a serious deadlock messages. TWS was freezing up before receiving the error message. It is possible that this is something environmental in your system. Are you using another application that uses abundant memory or bandwidth resources? You received the error message, “1162: Error processing FIX msg” because TWS couldn’t parse or translate the fix messages from IB’s dedicated servers. If the TWS was freezing, it is possible that any incoming data weren’t recognized."

   My answer to him is that there was nothing unusual in my system at that time. Given all of these circumstances, this looks like one of those very rare errors caused by something like a bit error; ECC memory would help in this type of situation, but the computer in question does not have ECC memory.

   Since nobody can figure out what it is, I'll consider it a rare case.

   Thanks very much for your help. At least we've all been reminded to expect the unexpected!

   Mark

   Posted 6 months ago #

6. garachen
   Member

   Hey Mark,

   Sorry to bring bad news, but this is not such a rare occasion. It happens to me about once every 3 weeks - but I probably see it more because I do several hundred trades a day. Good/bad news is that I don't think it's a problem with Karl's code. You'll notice when this happens that TWS will show the execution as some bright pink color and that the line never goes away and you can't delete it until you close TWS and re-open. On one occasion, I've gotten the same message by submitting an order directly through TWS.

   Of course, I've mailed the log and screenshots to IB several times but never get a response.

   It seems to happen more frequently if TWS runs over midnight and is still writing to the prior day's log file. I don't know why this is - but after having it happen 3 days in a row I now always close down after midnight and then log on again.

   Gary

   Posted 6 months ago #

7. Mark
   Member

   Hi, Gary,

   Thanks for the info.

   When you close down after midnight and the log on again, do you use Richard King’s TWS utility to close down and re-start TWS automatically -- or do you do it manually? Also, I had trouble disconnecting the socket control from TWS and then reconnecting under program control (see my thread “Reconnect under program control”), and I haven’t had any other need to solve this problem. Do you handle that under program control when you shut down and re-start TWS at midnight?

   Thanks again.

   Mark

   Posted 6 months ago #

8. garachen
   Member

   I've given up on a lot of things. One of which is having my IB strategies run unattended. It takes a smarter person than I to hack stability into an unstable system.

   Posted 6 months ago #

9. Karl
   Key Master

   I have been using Richard King's TWS utiltiy to control TWS for nearly a year now, and have been running a collection program (program that just records TWS ticks), reliably the entire time.

   From what I gather, the reliability issues are often exchange dependant, and certainly IB has some strange errors as Gary points out, whose frequency increases with the number of orders you place.

   The amount of automation you can use, depends largely on your strategy. Low frequency, low margin strategies that just depend on finding patterns and correlations can run quite safely, provided you build a good framework for handling these errors. High frequency, high leverage applications leave you at such great risk in the case of these errors, that you can never really trust IB's glamorous framework...

   Posted 6 months ago #

10. Craig
    Member

    Where does one find the 'Richard King TWS utility'?

    Posted 5 months ago #

RSS feed for this topic

Reply

You must log in to post.