bits19

Hi,

While searching google for info on IPC I ran across this site and have read some of the IPC informations. I need some advice.

I have an application that needs IPC. It is being coded in VB 6 and need run across all Win platforms (95 - XP).

The application will have one, perhaps two server applications. One dealing with Internet Data mining, the other (should I so choose to do things this way) would be a database server.

Client applications to these may have many instances and manipulate retrieved data in varying ways. These client apps need communicate to the Data Mining App exactly what to gather and where. In turn the Data Mining App need send back the resultant data to the specific client app that issued it the calling data. Obviously if the Data Mining App has no work to do, it need sit idle. The amount of data going across IPC can be of significant size and I would like it mapped as an object, actually an object with several object's that are embedded within (an array of objects within the object) and the size of the array is dynamic not fixed.

Now from what I have read, There are various mechinisms available to me.

Named Pipes appear to be out as I have read 95/98 support isnt there.

Windows Messaging, it appears one needs a window. Optimally the server objects have no Window short of perhaps a configuration dialog or report dialog. I read elsewhere that there are also restrictions on the amount of data one can send in a message?

Memory mapped files. From what I keep reading everyplace is saying, "By god man dont do it!"

This leaves Sockets. Thus I would have to create code that uses a port to open a data mining request. Send back a free port to communicate through etc etc. as well as apparently needing to treat my objects as a stream. Thus the client applications needing to effectively rebuild the objects from the data stream.

Sockets being a serialized mechinism means speed takes a hit, not that it is an imperative.

Again, this same form of process may well be necessary for database storage and retrieval possible against one master database, possibly against one master DB and many smaller DB's that are associated direct with the client applications which again, might be many instances of.

What am I making an air traffic control system? No.

It is a commerce tool I'll leave it at that.

lebb

I don't have specific advice to provide for your particular needs, but just wanted to suggest that you read the IPC discussions/tutorials in our Knowledge Base by both MathImagics and Thinker.

dilettante

Win9x will have issues hosting a service of any sort though. There is no equivalent of the NT Service paradigm, and a DCOM Server on 9x must be explicitly initiated after a system start (via a Run registry key if nothing else).


For VB6 the native IPC mechanism is COM/DCOM. This works fine on Win9x as clients, and also as servers - with the appropriate DCOM95/98 add-in package installed.

As far as Named Pipes go, Win9x can't be a server though it can be a client - subject to Pipe naming convention restrictions. As long as you observe the naming limitations (I believe that each "name part" in the Pipe name is limited to 8 characters) your application can be accessed from Win9x. Only the server needs to be NT based.

You also have Mailslots, another Windows Networking IPC option. The "second class delivery" option works somewhat like UDP in that it is based on short (length < 425 byte?) datagrams and broadcasts are supported. Win9x can act as the server for Mailslots. NET SEND works over a well known Mailslot, for example.

MSMQ is another higher-level form of IPC. Again there are Win9x restrictions but those machines can readily participate on a client level and to a limited extent as a server.

Also NetDDE, though this is becoming obsolete.

There is also Windows RPC, but this is probably impractical to use from VB programs. It is however the underpinning beneath DCOM, Named Pipes, Mailslots, etc.

When you say "sockets" I assume you mean TCP/IP Winsock sockets. That's not saying much either really though. Like RPC, it's a low-level transport, and one that you normally layer something over (actually RPC can ride on top of TCP/IP, NetBEUI, IPX/SPX, etc.)

The "something" can vary from some ad hoc protocol you build atop UDP or TCP to something else more standard. If not Windows RPC and the protocols above it, then perhaps HTTP (or SOAP or XMLRPC over HTTP).


The IPC you want... is it between boxes or strictly among programs on one system?

Within one box I'd just write the server(s) as ActiveX EXEs using COM. Between boxes I'd go with the same over DCOM. Keeping an ActiveX EXE's main thread up and running requires some tweaks and breaks some rules from a component design best practices standpoint but works fine.

Nothing wrong with Winsock either, you just lose the object model based API and have to reinvent a lot of stuff. Named Pipes will probably be too limited for you on 9x as you discovered. Anonymous Pipes work fine, but probably don't fit what you want to do - they require that tasks be part of the same process family.

bits19

Part of the keyword here is "generic", the software needs to run effectively on all Win platforms.

Mailslots simply wont suit. We are speaking of significant amounts of data needing to be transported to the applications. DDE is not an option as again, the software needs to perform against all Win platforms.

--

Actually both. That is to say the data mining program can run on the users PC or on a webserver (ASP) is the goal. The various client softwares run on the users local PC. Why? If a user has a broadband connection to the web the data mining will be nice and speedy. If they however are on dial up we'd like at some point to have the data miner application also running on a ASP server. That way the server can take advantage of high speed connectivity, extact relevent data, compress it, send it off to the client app(s) on the users PC.


Anonymous pipes wont do the trick. My issue with using TCP/IP sockets is the possibility of overruns in buffering. While unlikely it is a chance that exists.

Again in describing the application. A application does nothing but mine data off the web. It is told what to mine via client applications, these app's may also be running concurrently, could be say 20 running at once. Some doing "like things" on data and others performing numerous other tasks on the data.

At some point in time it'd be "sweet" to have the data miner application (effectively a server app to these client apps) also be able to be running on a Windows Server.

IP/sockets seems the natual way to go. Alternatives would be things such as drop files. At some stage (since this is all coded in VB 6) we'd considered using RealBasic to make a port of applications to the Mac as well.

See... The application is broad based and we'd not be surprised a bit if the customers of it set up dedicated machines. We want it to range all platforms, Win 95 forward.

---

Now if I understand you correctly your saying that we could use an ActiveX component (EXE) as a go-between basically. Its only purpose in life to move the data coming from the data mining to the proper client app. I had considered something like this but have not thought it all the way through.

I'd also considered using TCP/IP (sockets) to simply convey addressing data. In other words the client apps send params to the data miner App via winsock, no biggie, not TONS of data there. The data miner puts the info into a temporary Ram buffer. It then conveys via winsock where to retrieve the info from. Client gets the info, sends back a command to the Data Miner to release the buffer. This would be snappier. Of course, in such a scenario then having the data miner also being able to function on a true Win Server would need take into account that it is indeed on a server and could not use such functionality, irregardless in that instance, it must go via IP or other Internet related format, again, such as drop files which used to be quite common on older (older) mini/mainframe machines. Files created by one app to be passed unto another, so for example with a true win server based Data Miner it creates compressed drop files, the clients FTP these off the server.

Clearly there are varied ways of performing all this. I basically dont want code up a mish mosh and then find out, "Oh... Did this wrong!" LOL.

The fact that as you say I can loose the object model due to IP is an issue and perhaps not an issue. That is to say reconstruction of said objects would not be too awful painful. At the sametime it might be quite possible to put some form of wrapper around the data. I saw some code that basically established a Server Socket Bank. In other words, the code had one listening socket. If a connection is made it instantiates a server socket. It extended base Winsock to allow for a variant to be passed and or received back to the client app. It did this from the little I glanced at it via not using an event or asyncronous communication back to the client. Instead of using "DataArrival" in other words at the client it also allows for a syncronous socket.retreive.

Normally this would be "eh?" karma since part of the advantage to Async is the ability to work on data such as XML as it streams in. With this application however this is a non-issue as all data mined must be gathered prior to any processing. It is due to the fact that the mining itself is limited by the source site(s). That is to say, they wont give back "full results" in one call to their API. Instead it one call, maximum returned data will be 200 data sets per call. If 2000 data sets exist to be mined, then its going to take 10 calls to their API. Then this data must all be parsed, built out into the appropriate objects and sent off to the client(s) to process.

dilettante

Getting your head around a high-level architecture will be an important first step, and will probably limit your IPC choices somewhat. This means deciding things like what will run where, what needs to be "real time" and what can be batched or queued, and what your scalability requirements are.

While there are several multiplatform technologies you might choose, your mention of RealBasic almost dictates starting from dirt (TCP most likely) and building your own structure upon it. I admit I've only looked at RealBasic 5.5 for Windows myself, but the documentation never mentions the Windows COM/DCOM technologies. There is support for "standard" (pre-COM) DLLs though which would make access to Windows APIs possible, and thus perhaps some other Windows IPC mechanisms could be used. This locks you back into Windows though and would result in an interoperability problem, probably defeating your reasons for using RealBasic in the first place.

RealBasic does offer HTTPSocket and HTTPSecureSocket classes. This may make a SOAP, XMLRPC, or other "web service" style client viable in RealBasic. To go that route you'd probably want to piggyback on a standard web server though to avoid reinventing such a large wheel and trying to properly secure it.

If multiplatform operation is that important I suspect raw TCP may be your answer. This leaves you to invent message framing and message formats, design interaction patterns, build your own queuing, and develop some sort of security model. You'll spend a lot of time constructing and debugging this "plumbing" but you are right: you should be able to encapsulate a lot of it in some sort of class hierarchy.

With COM/DCOM you simply construct application class hierarchies and the object serialization, communication layer, and security come "for free." You don't write any communication logic at all. The Queued Components feature even offers you a form of loosely-coupled processing with little impact on application logic.

I'm living with several applications based on drop directories, batch processed files, and FTP to sling things around. It's all pretty miserable and takes a lot of hands-on operational fiddling. Some days are great and then on others the wheels come off. FTP is not meant as a peer-to-peer transport between hunks of software, it relies on a person at one end to handle exceptions.

This is why application development moved to Message Queuing Middleware (MOM) for loose coupling and recoverability. Perhaps the most common examples are Microsoft's MSMQ and IBM's Websphere MQSeries - though others exist as well. I'm unconvinced either are viable on a Mac however.


So you may be back to considering TCP sockets and doing a lot of grunt work. This isn't so bad, lots of people do it. One of the reasons XML got to be so popular is that it makes for a flexible message formatting framework, and there are enough parser/generators out there now that it saves you some of the heavy lifting of writing and debugging your own message serializing/deserializing logic. You might consider something like XMLRPC messages over raw TCP or TCP with some lightweight framing layered on it.