Re: Strong Typing, Dynamic Languages, What to do?

[Chris Uzdavinis <chris@atdesk.com>]

Jerzy Karczmarczuk <karczma@info.unicaen.fr> writes:

> > I like Scheme very much, but there is no possible way that such a
> > language could possibly keep up with the volume of data that my C++
> > code handles so elegantly.
> > 
> > (In all honesty, I'd be thrilled to be able to have written these
> > programs in Scheme, but it's just too slow to be practical.)
> 
> 
> Look, I don't know you, and I don't want to raise any questions about 
> your competence, seriously, with full respect. But I have seen such
> statements hundreds times. Practically *ALWAYS* people who said that, never
> tried.

I have no ego problems, so if I'm wrong I'd be happy to know about
it!  :)

Here is my situation:

We read hundreds of millions of stock quotes daily, in real time.
(Basically every quote that exists in the market on every symbol.)
The bottleneck in Scheme is mostly in the data parsing that it reads
off a network.  The data format is line based, comma-separated text
messages in name-value pairs.  Scheme's string parsing doesn't seem to
be efficient (or at least my ability to code in Scheme isn't
efficient) compared to what I can do in C++.  (I know this claim
sounds like a broken record to the FP sector, but so far it is the
truth for me.)

If it gets backed up, the data quickly becomes invalid and we lose
money, and so that's not permissable.  We make stock trades (or I
should say, our computers do) based on the data it sees... and wrong
data produces wrong decisions, and those usually end up being losing
trades.  (The GC kicking in is problematic too if it causes runtime
delays.)

Right now, I'm considering (but haven't tried) writing a C++ class
that has a nearly identical interface to the C++ STL list class,
except the implementation of the class will actually be using the
mzscheme data types, and the physical list will be maintained in the
native mzScheme list format.  Then I can have my C++ code do the
parsing and book keeping, but I can allow Scheme to do some of the
data interpretation.  I can embed Scheme into the application it will
have direct use of this data structure without having to do any kind
of transformation to "move" the data from C++ to scheme.

If you have suggestions for fast string parsing techniques in Scheme,
I'd very much appreciate hearing them.  So far I haven't found a pure
Scheme solution that comes even close, though.  It may be my naivity
with Scheme, or a lack of familiarity with some special
mzscheme-specific routine that could do this job efficiently.

Do you have any suggestions?

-- 
Chris