VB6 COM DLL Complilation - Help. (ByRef Param Error)

I have fully read the 1st section of COM DLL tutorial here: (skipped the source and the window GUI in dlls) --> (Yes it took weeks to understand and document properly, and now I am p¡ssed it doesn't work.)
http://www.xtremevbtalk.com/showthread.php?t=282796


When I try to compile my code I get the following error:
Title = "Microsoft Visual Basic"
Icon = Exclamation
Text = "Comiple Error" + NewLine(2) + _
"ByRef argument type mismatch"

and highlights "pIID" in the call to DllGetClassObject in Sub InitVBDll



Notes:
1. This is in an ActiveX DLL Project, with a reference to my .tlb (a compiled type library using an .odl source file)
2. The .tlb was compiled with a .lnk (shortcut) that is "C:\Program Files\Microsoft Visual Studio\VC98\Bin\MKTYPLIB.EXE" /nocpp
3. The project has 1 module (Module1.bas) and 1 dummy class (clsNothing.cls)
4. The project is set for "Binary Compatibility" from Properties (dlg) --> "Component" (tab) --> "Version Compatibility" (grp) --> BC (opt)
5. The linker is installed and compiles fine.





Module1 Code:


Option Explicit

'Declare Function DllGetClassObject Lib "CodeSample1.dll" (REFCLSID As Long, REFIID As IID, PPV As Long) As Long
'Declare Function vbaS

Type IID
data1 As Long
data2 As Integer
data3 As Integer
data4(7) As Byte
End Type

____________________________________________________

Function DllMain(ByVal hInstance As Long, _
ByVal lReason As Long, _
ByVal lReserved As Long) As Long
DllMain = 1 ' this function should always return 1
Call vbaS(hInstance, lReason, lReserved)
If lReason = 1 Then Call InitVBdll ' 1 means first call
End Function

Sub InitVBdll() ' COM initialiser
Dim pDummy As Long
Dim pIID As IID ' IID_IClassFactory
' {00000001-0000-0000-C000-000000000046}
pIID.data1 = 1
pIID.data4(0) = &HC0
pIID.data4(7) = &H46
Call DllGetClassObject(pDummy, pIID, pDummy)

End Sub

Public Sub TestEvent()
MsgBox "Hello", vbInformation, "It works"
End Sub





.odl Code - used to make and reference the .tlb: (Directly from the pdf tutorial, so I dont understand why it wont work)


[ uuid(AABBCCDD-0000-0000-0000-000000000000),
helpstring("MathImagics VB6 DLL Self-initialiser"),
lcid(0x0), version(1.0)]
library vbLibraryHelper {
typedef struct {
long Data1;
short Data2;
short Data3;
unsigned char Data4[8];} IID;
[dllname("vbLibraryHelper_mathimagics")]
module ThisDLL {
[entry("DllGetClassObject")] Long DllGetClassObject(
[in] long *pClsId, [in] IID *riid, [in] long *ppv);
[entry("__vbaS")] Long vbaS(
[in] long hInst, [in] long lReason, [in] long lRsrvd);
}
}




Basically, the VB comipler doesn't want to accept the variable pIID which points to the IID structure (type)


According to the DllGetClassObject on MSDN:
http://msdn2.microsoft.com/en-us/library/ms680760.aspx

The IID is an riid meaning a "REFIID" which is a reference to the IID of IID_IClassFactory (AKA IClassFactory as defined in OLE Headers as the interface identifier)


So it is obviously something with the way it is being passed from vb to the .tlb so please let me know if you have any help and know of a solution.

Use [out] instead of [in] for your IID in the tlb declare.

I swear I tried that, but perhaps I forgot to recomiple or something. I really hope is it that simple of a fix, even though I will feel quite dumb for not changing that one line.

Ill test then when I get home. I hope it works.

Well I tried it and it still gives the same error.

Are you able to provide an example that works for you, because this obviously doesn't want to comiple =[.

[out] StructName* paramName is the correct form for ByRef paramName As StructName.

After you recompiled the ODL, did you copy the TLB over the top of previous version? Simply referencing a new TLB with the same GUID will not work.

I'm assuming you also have the non-TLB declare statement remmed out. If not, it would override any TLB declare.

[out] StructName* paramName is the correct form for ByRef paramName As StructName.

After you recompiled the ODL, did you copy the TLB over the top of previous version? Simply referencing a new TLB with the same GUID will not work.

I'm assuming you also have the non-TLB declare statement remmed out. If not, it would override any TLB declare.


Maybe I should try [out] IID* riid instead of [out] IID *riid.

isn't * the C++ syntax for a "a pointer to"
And while we are on that subject I am curious about these 3 things: * & and ->.

My guess:
* = send variables pointer (memory address) ?
& = ByRef = Reference to something = send it byref (which is the pointer) ?
-> = pointer to variable ?

So in other words for vb6
* = VarPtr or StrPtr
& = ByRef
-> = CopyMemory iVariableNumber, ByVal lPointer, 4

This has been nagging me for a while so jw.


Ok so back on topic:
Yes I fist deleted the .tlb in the comiple directory, (from the odl) then deleted the copy of the old one I put in C:\, and then I deleted it in the registry. under HKCR\TypeLib\...
I then recomipled it, put it in C:\, and referenced the new one.

I guess just to be safe Ill try to find any other traces of the GUID in the registry other than under TypeLib, change the location to something like C:\1 and then change the GUID (hopfully to something not used and that works)

Anyway. This will be something to try hopfully this weekend. Still, feel free to provide any quick exmples of a vb active-x dll that properly references a typelib similar to this to change its entry address and makes a call to the COM enabler.

Thanks.

Maybe I should try [out] IID* riid instead of [out] IID *riid.

IID* riid, IID * riid, IID *riid are all identical



isn't * the C++ syntax for a "a pointer to"
And while we are on that subject I am curious about these 3 things: * & and ->.

My guess:
* = send variables pointer (memory address) ?
& = ByRef = Reference to something = send it byref (which is the pointer) ?
-> = pointer to variable ?

So in other words for vb6
* = VarPtr or StrPtr
& = ByRef
-> = CopyMemory iVariableNumber, ByVal lPointer, 4


IID* is a pointer to an IID
IID& is a reference to an IID (btw, the definition is REFIID which is IID&)
For VB purposes & and * will both by ByRef, which passes a pointer
-> is a shortcut for dereference and dot operator (*foo).bar or foo->bar

I think from what you've said you have gotten rid of the previous TLB. All that is necessary, assuming you used the same GUID in the ODL, is to overwrite the TLB referenced with the newly compiled version. And obviously VB has to be closed and reopened. (or remove reference - Save project - readd reference) I normally just hit File - New then MRU back to the original project.

Simply referencing a new TLB in a different location will not work. VB sees the same GUID and will not comply. Even though there is no error message. Btw, I have a TLB management app posted in the Code Library.

I might get some time this weekend to work through the example. I've done it before, but it's on a different box.

IID* riid, IID * riid, IID *riid are all identical




IID* is a pointer to an IID
IID& is a reference to an IID (btw, the definition is REFIID which is IID&)
For VB purposes & and * will both by ByRef, which passes a pointer
-> is a shortcut for dereference and dot operator (*foo).bar or foo->bar

I think from what you've said you have gotten rid of the previous TLB. All that is necessary, assuming you used the same GUID in the ODL, is to overwrite the TLB referenced with the newly compiled version. And obviously VB has to be closed and reopened. (or remove reference - Save project - readd reference) I normally just hit File - New then MRU back to the original project.

Simply referencing a new TLB in a different location will not work. VB sees the same GUID and will not comply. Even though there is no error message. Btw, I have a TLB management app posted in the Code Library.

I might get some time this weekend to work through the example. I've done it before, but it's on a different box.



Hmm, so is this correct ? (What I think it does is commented)



int var = 30; // Set variable - var = 30
int ptr = &var; // Set ptr equal to var's address (Gets a pointer) - ptr = 0x5df68d;
*ptr = 10; // var = 10 AKA - ptr's variable (var) is now 10.

int new = 20;
*ptr = new; // var = 20

*ptr = &var; // var = 0x5df68d (var now equals its own address (pointer))




Although I am confused about getting the pointer because either I didn't understand this correctly, or there are 2 ways to do this:

Here http://www.cplusplus.com/doc/tutorial/pointers.html is gives an example like:



int var = 25 // var = 25
int ptr = &var // ptr = 0x5df68d

ptr is now a pointer to var


but here: http://library.thinkquest.org/C0111571/manual.php?tid=60


int var = 25 // var = 25
int *ptr = &var; // ptr = 0x5df68d

ptr is now a pointer to var

this doesn't seem right, because I think that this would set the dereference of ptr (var) equal to (=) the pointer of var (ptr) or in other words "var = &var" or "var = ptr"
Doesn't it?

although they call this "indirection"
Maybe it is contrary to what some think by 'in' = "not" AKA "not directly implied" / opposite of what is ment?


I get what (*) does now, it get the pointers value - but I am very confused now about it being 2 different things of indirection and dereferencing at the same time.



1 Other question:
what would happen if var = *var (is that even possible to dereference a variables value)


Anyway
As for the DLL, maybe I will just try it over on a different project.
I guess if it wont compile I might just have to learn how to program in C++ before creating a standard win32 DLL (which has COM enabled and allow for exports)

Well I just dont understand it:

I searched the registry for the GUID {AABBCCDD-0000-0000-0000-000000000000} and it only finds it in one place, HKEY_CLASSES_ROOT\TypeLib\ and i delete it everytime, and I dont see why this still does not what to comiple.


[ uuid(AABBCCDD-0000-0000-0000-000000000000),
helpstring("MathImagics VB6 DLL Self-initialiser"),
lcid(0x0), version(1.0)]
library vbLibraryHelper {
typedef struct {
long Data1;
short Data2;
short Data3;
unsigned char Data4[8];} IID;
[dllname("vbLibraryHelper_mathimagics")]
module ThisDLL {
[entry("DllGetClassObject")] Long DllGetClassObject(
[in] long *pClsId, [out] IID *riid, [in] long *ppv);
[entry("__vbaS")] Long vbaS(
[in] long hInst, [in] long lReason, [in] long lRsrvd);
}
}


Infact I just checked the object browser and all i need to do is replace it for it to updateA (just unreferene it then delete, no need to even close vb to delete it), because in the Object Browser F2, for a test I changed the Data4[9] and it updated in it properly displays 0-8 instead of 0-7 (when it is set to 8 in the TL)

Ok well I just did an experiment and changed the riid line to [in] long *riid and passed it 0. I remember reading it is not required that its params are correct b/c it cant check them until it initalizes COM. but i am unsure if this is ok to do.

It compiled finally, and the window came up confirming the new entrypoint, and my custom export.

Tested with a diff vb app and worked without crashing, but how do I know COM is initalized? Would it crash without?

Maybe regardless of this article a dll really doesn't need COM and works fine without? After all why would this article be created if it didn't work? http://www.windowsdevcenter.com/pub/a/windows/2005/04/26/create_dll.html?page=last

The most recent comment is what let me to this guys code "Jim White"

Is there a way to tell if a dll has COM or not?

I sucessfully compiled the ODL with mktyplib, added a reference and compiled an EXE with simply:


Option Explicit

Private Sub Form_Load()
Dim pDummy As Long
Dim pIID As IID ' IID_IClassFactory

' {00000001-0000-0000-C000-000000000046}
pIID.Data1 = 1
pIID.Data4(0) = &HC0
pIID.Data4(7) = &H46
Call DllGetClassObject(pDummy, pIID, pDummy)
End Sub


I realize you are creating a DLL, but this simple test shows the compiler (under more normal circumstances) doesn't mind the TLB definition. Can you repeat this simple test with an EXE project and TLB to verify it works?

If it does, then you might do a text search for a DllGetClassObject declare, which would override any TLB declare. Also, remove the Type IID declaration from the module, it isn't necessary and might even be causing a problem.

Thanks to OnErr0r for dealing with this - I'm in the middle of relocating and currently my home system is packed away, so I'm unable to do much to help.

BTW - that guy, Jim White, c'est moi! ;)

Have either of you been able to load this DLL into C++ applications?

I am not able to.
Well actually the dll does load, but as soon as I make an API call the Dll will execute it once, and then crash.

I am loading it into the client (The C++ App) using the standard LoadLibrary call.
I have tested this with CreateProcess and MessageBox, both work but right after that, the dll unloads from the process address space.

It makes me think that COM is not enabled properly or something. The odd thing is that when calling LoadLibrary from a VB6 application the Dll stays puts and executes the commands fine. (Opens cmd & calc.)

Here is an example of how I am testing this.

VB6 Dll New Entrypoint (Opens cmd & calc). Problem = only works when calling LoadLibrary from VB6 app, and not from a C++ app.

Private Declare Function CreateProcessA Lib "kernel32.dll" ( _
ByVal lpApplicationName As String, ByVal lpCommandLine As String, lpProcessAttributes As Any, _
lpThreadAttributes As Any, ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, _
ByVal lpEnvironment As String, ByVal lpCurrentDirectory As String, lpStartupInfo As Any, _
lpProcessInformation As Any) As Long


Public Function DllMain(ByVal hInstance As Long, ByVal lReason As Long, ByVal lReserved As Long) As Long
Call vbaS(hInstance, lReason, lReserved)

If lReason = 1 Then
Dim si As STARTUPINFO
Dim pi As PROCESS_INFORMATION

si.lpDesktop = "Winsta0\default"
si.wShowWindow = 5

Call CreateProcessA(vbNullString, "cmd.exe", ByVal 0&, ByVal 0&, 0, 0, vbNullString, vbNullString, si, pi)
Call CreateProcessA(vbNullString, "calc", ByVal 0&, ByVal 0&, 0, 0, vbNullString, vbNullString, si, pi)
End If

DllMain = 1
End Function


Here is my COM enabler:

Public Sub InitVBdll()
Dim pDummy As Long
Dim pIID As Long
Call DllGetClassObject(pDummy, pIID, pDummy)
End Sub


And here is my TypeLib ODL:

[ uuid(AABBCCDD-0000-0000-0000-000000000000),
helpstring("MathImagics VB6 DLL Self-initialiser"),
lcid(0x0), version(1.0)]
library vbLibraryHelper {
typedef struct {
long Data1;
short Data2;
short Data3;
unsigned char Data4[8];} IID;
[dllname("vbLibraryHelper_mathimagics")]
module ThisDLL {
[entry("DllGetClassObject")] Long DllGetClassObject(
[in] long *pClsId, [in] long *riid, [in] long *ppv);
[entry("__vbaS")] Long vbaS(
[in] long hInst, [in] long lReason, [in] long lRsrvd);
}
}



It compiles, links, exports fine, the only flaw is when it is loaded into a process not made with VB6. I think it has something to do with me enabling COM improperly but I am really not sure at this point as I have been working on this for months and still have not solved it,

Hopefully if anyone gets a moment that has had experience with this, can take a look for me. (A good tool to help is Process Explorer www.sysinternals.com to view loaded modules (DLLs and Exes, within the process address space)
Maybe test it out to see if it works for you when calling LoadLibrary from a VC++6 application? because in truth I dont understand why it is not for me. What I find most odd before is the result in that, cmd opens but calc does that (that is because the DLL unloads after any API call is made, and therefore the 2nd is never reached)

Help would be great because I would love to experiment with DLLs (I have not been due to my lack of experience in C++ coding. VB seems so much easier except when it comes to stuff like this where COM and multithreading have some strange behavior in VB. Grrr.

We have to remember that the method used to "initialise COM", by calling DllGetClassObject, is essentially a hack. What it really does is not really known, but it's enough to allow the VB6 dll to function.

We do know that the call can't be truly successful, that is it can't possibly create an object instance since we have no idea what the correct CLSID is.

Any attempt to use COM in the DLL, or perhaps even the client, might not be possible because of the uncertain COM state. Anecdotal evidence tends to support this - I've had a few people report trouble with using objects in the DLL.

Perhaps the method can be refined - if we knew the CLSID of our VB6 dll, it might be possible to make the auto-initialisation make a successful call to DllGetClassObject.

What I have in mind is this - the DLL project properties are set so that it doesn't change CLSID's on recompilation, then we compile and register it, then find the CLSID from the registry ... then we fix the dllMain code ...

... then maybe it could work

By the way, I can't read this thread (and probably you can't either) because one of the CODE insertions has lines that are way, way, way too long so the thread display is impossibly wide

You can fix it by editing the relevant posts

Hey again. Sorry about that post, I used firefox, so I did not see it until I opened this thread using IE7. I would edit it, although it is not letting me due to the elapsed time that passed since I posted it (or perhaps it is due to posts being posted after it)

Anyway.
It kind of sucks because I thought I was just doing something wrong, but in truth this project was never ment to work for VB apps only?

I feel like this could full work since:

1. This code (your solution) allows only 1 API call before the DLL crashes in the remote process.

2. Alex Ion created a solution here - http://pscode.com/vb/scripts/ShowCode.asp?txtCodeId=54682&lngWId=1 although this application to "fix" the VB6 dll is not free, nor does it talk about any fix other than using his solution

3. The VBAdvance works great, just not for specific applications that were created by Microsoft. What I mean by that is that these DLLs that are loaded into processes like regedit, winlogon, notepad, cmd will not work. What I found in common is the Image's (Exe's) Base Address loaded into the process address space.
What I noticed is that any process with the Exe having that address of 0x1000000, then the DLL will not load into the process =[.
I tried changing the DLL's address but it still does not work. Perhaps it is something specifically on how VBAdvance's "fix" app changes the DLL to work properly, which it conflicts with 0x1000000.
Odd enough though, is that I've examined the DLL using Process Explorer and it loads at the address where i set it before compile. So I hae no idea why it doesn't work with this.


So these 3 solutions all seem "broken" so perhaps the only way to fix this is just to learn C++ and create the DLL in there which can comunicate with the GUI writen in VB6?

Thanks, the thread view is indeed quite sane with Firefox instead of IE ...

I remember Alex Ionescu well, he is very bright, but tends to hyperbole (eg "world first"), and not to deal with criticism very well

Anyhow, I'm going to try my CLSID idea and will report back when I know more

The idea of using the correct CLSID in the DllGetClassObject call works, in the sense that the function then returns "success" rather than failure.

Unfortunately I cannot determine what, if any, difference this makes - all of the examples I've tried so far behave the same whether I use the correct CLSID or not.

Anyway, this might be a red herring as far as AltF4 is concerned. I've just noticed that the DllMain code posted above does not actually call InitVBDll. :-\

This certainly needs to be done immediately after the line "If lReason = 1 Then" or it's doomed!

PS: Ancient history now, but my apologies for using Long's instead of IID's in the original ODL, which was the cause of the original compilation error! It made the DllGetClassObject declaration non-standard, which was probably sloppy work.

In fact I had to make a new one with the correct data types to test the "correct CLSID" method! :rolleyes:

Yes, I guess I forgot to include InitVBdll in, but it even if it is called or not, there seems to be no difference.

A C++ App will execute 1 API function, and then the DLL appears to just "unload."

Process Monitor Shows that the dll and the VB Runtime library are loaded successfully in the process, so I have no idea what is causing them both to be unloaded right after the API call is made. (It appears like they almost "crash" because Process Monitor does not document an Unload of the Runtime library or the dll.)

Arg!

If you have a C++ app make a LoadLibrary call to your VB6 DLL you will see what I mean about it only working for 1 API call in the dll. =[

OK, I think I know what the problem is.

When the client is not VB, the dll can't make Declare-style API calls until fully loaded - that is, not in the DllMain processing.

While loading, it can only make TLB-declared API calls.

Hmmm, well didn't think of that.
I will have to test that in C++ by calling an exported event (Although I need to look up how. I dont have much experience with C++ Coding, haha)

Is there a way for the DLL to tell when it, itself is full loaded? I tried putting the API calls under a condition of If DLL_PROCESS_ATTACH... although the same results occur where after the first API call from the DLL, it will crash.

(It appears that the Runtime Library and the DLL both crash at the same time)

The solution is simple - declare any external functions that are to be called during DllMain (process-attach case) processing in the TLB, not the BAS module.

The affected functions can still be called elsewhere, as the TLB method is always an alternative to the Declare method, it's just that in this particular case (DLL loading), the TLB's the only way.

Hmm, ok I will give it a shot. That kind of sucks that we cant do it another way, but then again VB6 wasn't made for standard DLLs haha.

Oh and BTW, I figured out the problem that I was talking about before with the vbAdvance DLL and Alex Ion's code injector. It was DEP shutting down the process. I should have known for the fact that is says DEP will now close this application.. but I just didn't fully understand that only SOME windows exes are protected (such as winlogon, calc, regedit. But not IE)

Anyway, thanks for the help and I will try the whole TLB declare thing.

Hey I just wanted to say it works, and thank you for your help.

The only issue I have now is using the VB6 string functions. (They either result in crashing the C++ app or causing a DEP error, which crashes it anyway)

Some that do not work:
Left, (String comparison ex: If S1 = S2 Then), String

Some that work:
Len, Chr, Space

I do a work around by using the calling some APIs, but it is just troublesome.
Perhaps it has to do with the vb6 runtime. It is loading, and in the same base address so I am not sure why these do not work.

Anyway. I've successfully implemented hooks with the dll now =]
Always wanted to test out a global (system-wide) CBT hook but never knew enough about C/C++ DLL dev to do so. The TLB seems to be the best hack to bypass the Declare API call exceptions.

One other thing, calling vbaS (which calls the VB6 dll's real initializer, and InitVBdll (which calls DllGetClassObject) seems to do NOTHING. It works with or without it in either case.

Can you briefly explain the purpose of what calling these are again, because they provide no effect that I can see?
I know COM is apparently enabled in the dll, but does that really help ANYTHING?
(Note that the VB Runtime MSVBVM60.DLL is still loaded in the process, even when both of these functions are not called)

Are you passing a BSTR from the C++ app to the VB DLL? VB doesn't know how to do a char* string comparison. An alternative would be to convert the char* to BSTR using StrConv and then do the comparison.

I have tested both Left and string comparison and both work fine, but only of course if I have initialised COM (following most string operations VB checks the Err object, and this requires COM initialisation)

Regarding the general effect of COM initialisation, it is certainly possible to do without it, but only if the DLL code is very simple - you are virtually restricted to using simple arithmetic data types, and can only make external calls via a TLB.

Because COM is so intrinsic to VB, it is not easy to have a COM-free VB DLL. It might work fine for some clients and then crash others because it tries to access Err.

It would seem to make good sense to include it in all cases, wouldn't you agree! ;)

Well it seems to be working well.

Although the COM still seems to not work. (Well well Im not sure if COM is enabling or not, but string comparison still ejects the DLL)
Ex:
(In DLL Main)
Call vbaS(hInstance, lReason, lReserved)
If lReason = 1 Then Call InitVBdll

Dim s1 As String
Dim s2 As String

Call MBox2(ByVal 0, ByVal "Before compare", ByVal "", ByVal 0)
Call MBox2(ByVal 0, ByVal "Before compare2", ByVal "", ByVal 0)
If s1 = s2 Then Call MBox2(ByVal 0, ByVal "Yes", ByVal "", ByVal 0)
Call MBox2(ByVal 0, ByVal "After", ByVal "", ByVal 0)

Public Sub InitVBdll()
Dim pDummy As Long
Dim pIID As Long
Call DllGetClassObject(pDummy, pIID, pDummy)
End Sub

In any case, it isn't too big of a deal since a work around is lStrCmpIA, it it would be nice to have. (Maybe it has something to do with the process not being VB code. I am usually testing it by injecting the DLL into calc or notepad)


Other question:
Do you happen to know if there is a way to make the .odl pass a param ByVal? Reason because once the .tlb is generated, VB6 sees every single param as ByRef and I have to continuously type in ByVal for almost every param to an API call.


I just tested it with subclassing: Disabled Ctrl+Alt+Del and Ctrl+Shift+Esc by injecting the DLL into winlogon.exe and having it change the WinProc for its SAS (Secure Attention Sequence) window, and it works great. Something I always wanted to try to do in VB only.
I had my doubts that it work work, and luckily it does, because that opens up some serious functionality. Perhaps in the future I will look into some ASM and attempt some IAT hooks :P

In any case, I am very glad API calls work in the modded Dlls because that can make any app truely powerful.


Edit: I did test the string comparison in a VB app now, and it does work there, although I am assuming perhaps it does not work for C apps (such as calc, regedit, notepad etc), regardless for whether or not COM is enabled?

You never answered my question about BSTRs.

A TLB declare allows you to pass ByVal or ByRef. All foo* are pointers and are ByRef in VB. void* is ByRef As Any. To pass ByVal use a value type that is not a pointer


[in] int foo // ByVal foo As Long
[out] int* foo // ByRef foo As Long
[out] void* foo // ByRef foo As Any

In regards to your question:
"Are you passing a BSTR from the C++ app to the VB DLL?"

No, I am just performing string manipulation within this VB6 DLL which has been injected into a C/C++ app.

When I inject the DLL into a VB6 app it works fine, but when injected into one coded in a different language it will crash on things like Str1 & Str2 or If Str1 = Str2. (The DLL will crash and unload, not the process.)


If we were to access the char* (ByVal Str) as a BSTR wouldn't we just need to do: ByVal VarPtr(Str), instead of StrConv ? (Or if we were to access the actual data of the BSTR, which is represented in unicode format all we should need to do is ByVal StrPtr(Str) correct?)




For the TLB:
If [in] int foo is passed ByVal, then why does VB6's IntelliSense show the parameter as being something like this:
"foo As Long"

Meaning ByRef. Perhaps this is a glitch in the IntelliSense since it might only display "ByVal foo As Long" if it is declared in VB6 using its own Declare statement?

If we were to access the char* (ByVal Str) as a BSTR wouldn't we just need to do: ByVal VarPtr(Str), instead of StrConv ? (Or if we were to access the actual data of the BSTR, which is represented in unicode format all we should need to do is ByVal StrPtr(Str) correct?)

No, you can't just cast and expect VB functions to work on char*'s. You have to convert from 1 byte per character to 2 bytes per character (BSTR).



For the TLB:
If [in] int foo is passed ByVal, then why does VB6's IntelliSense show the parameter as being something like this:
"foo As Long"

Meaning ByRef. Perhaps this is a glitch in the IntelliSense since it might only display "ByVal foo As Long" if it is declared in VB6 using its own Declare statement?

It's not so much a "glitch" in the OB, but a shortcoming. At one time I considered writing my own for that very reason.

Okay so use unicode. (UTF-16) Thanks.


Also, yes I see what you mean, that is not very nice of IntelliSense :P

Here was my test:

[entry("Test")] long ParamOutTest([out] long* tOutPointer, [out] void* tOutPointerVoid);
[entry("Test2")] long ParamInTest([in] long tIn, [in] long* tInPointer, [in] void* tInPointerVoid);

The only thing that really changes is the As Any (As you stated), which is good.
Although that is too bad that it doesn't show use ByVal or atleast ByRef.



Do you think I should pre/post fix variable names passed ByRef with some kind of identifier to remember that it is/abide by some kind of standard?

Ex: Add R (Reference) to end:
[out] int fooR

Good standard perhaps, or just dumb idea?



In any case, thank you for all your help. You and Jim White (Mathimagics).
BTW, how did you know all this info, perhaps any good books you have for reference, or has it just come with experience/experimentation?
You can both feel free to answer that one if you like =]

Another thing that is bothering me a lot, and Im about to give up after a few days of non stop trial and error with RegisterClass and CreateWindowEx:


Have either of you come up with any creative method to have the DLL be able to communicate with an outside application?

Reading a bunch of tutorials over at CodeProject.com it appears a good solution is by creating Shared Sections and communicating through that method, although it is obvious that we will not be able to do that kind of hack on these Dlls.
The solution I like is by having the Dll creating a window in the remote process and listening for specific WM_APP messages, but this has ended in only frustration! The problem appears to be that because the DLL is doing CreateWindowEx (and RegisterClass) in its DllMain, then when the Injection thread ends (A thread created using CreateRemoteThread, which calls LoadLibrary), the window is removed/destroyed and it cannot be stopped =[.

It is messed up though because if I pause the execution of the DllMain using a MessageBox, the window is created, but as soon as the thread is destroyed, so is my window.
Perhaps the solution would be to have the Dll Spwan other threads using CreateThread, but I recall threads in VB6 does not play nice at all will almost always end with some kind of crash (and I really dont want a Dll crashing that is injected into various remote processes)

I was going to do something with SetWindowsHookEx with a Thread-Specific hook, although reading over the documentation I think Thread-Specific ones only work for code that is within a process's executable (and not a Dll loaded into its address space).

So at this point it seems that Shared sections are out of the question and CreateWindowEx is not working, so I have no idea what else could be done to communicate with the Dll from a different controller process.

If you have and suggestions or recommendations it would be very helpful.

To get anything done in the DLL, we need to use messages. To use messages we need to subclass.

To avoid accidental interference with the application, it's a good idea to use unique msg codes. Both the DLL and the controller can call RegisterWindowMessage, specifying the same string parameter, and so obtain a unique message code (WM_APP + something). We'll call this WM_APP_DLL. (We can create more than one, of course, if needed).

The DLL sublasses the app's main window, installing a message handler that reacts to WM_APP_DLL messages, but passes all other messages through to the handler it replaced.

The controller can now initiate activity with WM_APP_DLL messages, using wParam and lParam to specify "opcodes", etc.

Conversely, it's a good idea for the controller to have a ListBox in order to receive messages from the DLL. The handle can be passed in the very first WM_APP_DLL message that the controller sends. ListBox's are particularly useful because the other process can send strings without having to mess about with shared process memory (LB messages are automatically "marshalled" by the system).

Another advantage of a ListBox is that you don't have to subclass the ListBox if you'd rather not - if all DLL messages are issued with LB_ADDSTRING, you can just run a Timer in the controller that watches the ListCount value. This means you can simplify things when running in the IDE, say ...

Thank very much for that information.

I think it would have been a lot easier to have the DLL create its own window, but since that seems to not be working I will go with the method you described.

Here is some pseudocode I just created (and I think this will work) but please let me know if u have and suggestions or find any flaws:


User specifies the window to modify [via subclassing] (using a hotkey registered by the controller app)
Get a window’s owner process (from its TID)

If DLL is already injected into the process (scan all the modules)
START:

Controller sends a message to the subclassed DLL listening window (Specifies which window to modify)
DLL Subclasses the window to modify and waits for specific messages to filter

Else
Controller Calls RegisterWindowMessage to generate a unique message
Controller injects the DLL into window’s owner process
DLL calls RegisterWindowMessage to generate a unique message (same as before)
DLL subclasses a random window within the process (To listen for messages)
DLL sends a message to the controller which states which window handle it is listening on.
If window no longer exists, DLL searches for another window and repeats last 2 steps
When controller receives the listening window handle, GOTO start
End If

It's been ages since I played with this stuff, but I'm sure that I was able to create a window in DllMain of the injected DLL.

OK, I see now that it IS true after all - windows are in fact thread-tied, which is why you had trouble ... my old samples didn't have this problem, but they weren't using CreateRemoteThread for the injection, they were using a temporary window hook.

CreateRemoteThread is a neater injection method but we have to create the window outside DllMain. So the way I got it to work was for the DLL to subclass its own main window in DllMain. The controller injects the DLL, then sends a "startup" message to the target's main window - the message handler creates the new window, removes the subclass from the main window, and returns the new window handle, which the controller uses for all subsequent messages ...

I found it easier to have the controller to nominate the target application window that the DLL will subclass - this simplifies the DLL considerably.

The controller registers some mesage codes, including say WM_DLL_START, then identifies the target process, TPID, and its main window, TWND. It allocates a shared memory area (via CreateFileMapping, MapViewOfFile) using a process-specific mapname, eg "Shared." & TPID. It copies TWND into this area, then injects the DLL.

The DLL's process-attach code gets a pointer to the shared memory area, obtains the TWND value, and subclasses that window. When it gets a WM_DLL_START message, it removes the TWND subclassing, creates a LISTBOX window, say MWND, and subclasses that (we can use the same MsgProc). The DLL can return the window handle as the message return value.

The controller, after injecting the DLL, then sends a WM_DLL_START message to TWND, gets the returned window handle MWND, and uses MWND for all subsequent messages.

We could, of course, simply keep the subclassing on TWND and not create a separate window. If we do create an MWND window, and we also want the controller to be able to unload the DLL, we'll need a WM_DLL_STOP message that tells the DLL to destroy MWND, to avoid crashing when we attempt to remotely unload the DLL. If we don't need a remote-Unload facility this doesn't matter.

Skeleton code for DllMain


DlMain = 1

Case DLL_PROCESS_ATTACH
PID = GetCurrentProcessId
MapName = "Shared." & PID
mHandle = CreateFileMapping(-1, 0, PAGE_READWRITE, 0, 1024, MapName)
pShared = MapViewOfFile(mHandle, FILE_MAP_WRITE, 0, 0, 0)
CopyMemory AppWindow, ByVal pShared, 4
mWndProc = SetWindowLong(AppWindow, GWL_WNDPROC, AddressOf(msgWndProc))
If mWndProc = 0 Then DllMain = 0 ' failure

Case DLL_PROCESS_DETACH
UnmapViewOfFile pShared
CloseHandle mHandle
End Select

Interesting ideas!
I was unaware that Memory-Mapped Files were able to be shared accross mulitple processes. I think it is time to read more of Ritcher's Windows via C/C++ which explains a lot of this in a pretty indepth level.

I wonder if creating a listbox window will actually work, or if the same effects will occur where the window is destroyed after the Injection thread terminates (which occurs when DllMain returns).
What function should be used for the listbox creation? (If it is CreateWindowEx, then I have have a feeling that will be the case. BTW is your user32.dll also missing CreateWindow?)
Also, have you tested this in the past on whether or not this listbox creation is possible?
Although that is good news that window creation is not a MUST by the Dll, since (And as a result: to have the controller unload the Dll, which isn't a big deal since we just use the CreateRemoteThread method for calling FreeLibrary)


Also, are you using the same method for inital Dll injection into the target process as I? (Which is ritchers method of using CreateRemoteThread to calls LoadLibrary.) Just checking to ensure that we are on the same page.
Obviously if there was a way to make a process' main thread execute the LoadLibrary, then the CreateWindowEx wouldn't be a problem because that thread wouldn't be destroyed, in which in window would remain.
I probably really shouldn't say "if" because there actually is a way I have seen which forces other processes to execute any API function (By use of Assembly Opcodes, which might I add is VERY hard to understand) but the problem is DEP will detect it and shutdown the process (Which isn't very good news for apps like calc, regedit, explorer, notepad)

It is obvious that Math isn't the only programming attribute that you excell at (Obviously due to this whole idea of VB DLL creations, your linker hack to modify the file's export table, and intimate knowledge of process/thread-specific functions)



BTW, if you are curious about the application I am making: It basically is a security app to hide windows and keep them hidden (Hiding all taskbar windows is the easy part, but keeping them hidden is the hard part which requires each window to be subclassed in the system, therefore requiring a Dll be injected into every process' which owns the thread(s) owning the window(s) to be subclassed)

Now you might be thinking: Wow this is quite a complicated task for something that could easily be done with Hooks. Well here is the problem: After studying and doing trial and error with hooks for MONTHS I came to the conclusion that the only hook which could do this is: CallWndProc. So whats the problem: MS decided to take away its ability to modify the messages being sent to the windows :(
The closest I came was uing a CBTProc, but all results have shown that it is only able to prevent window creation and discard only some (but not all) of the SW_* messages.

You see, the goal here is: To prevent a window from becoming visible BEFORE it actually does. I have found a solution with hours upon hours of testing and found that the message of WM_WINDOWPOSCHANGING can be modified inorder to discard the SWP_SHOWWINDOW and SWP_NOACTIVATE, and works for ALL SW_* and SetWindowPos commands.
But now the time has come to implement this into a Dll which can communicate with the controller (application which decided which windows will be the target, based on those shown in the taskbar), and it is a big task indeed.




So why am I going through all this trouble?
Well there are many times that I like to have quick solution to having a clear desktop before anybody (usually friends) use the computer inorder to browse for something or perform a specific task. The issue is when a window is hidden, nothing prevents it from itself, or some other application making the window visible again. It has happened many times before, I am so tired of that occuring that I want to put a stop to it once and for all!
Ex: I have had notpad or word windows up with data I do not want to close yet (such as passwords or other information), I have hid them but there were a few times those have become visible and some of my friends decided to play some jokes with some accounts...
The easy way out could be to have a timer that runs in the background and update every few milliseconds to check the window's visibility state, if it is visible (Has WS_VISBILE) then call either the ShowWindow[Async] or SetWindowPos, but there is still flickering and focus change (not a very *Solid* method, unlike what this Dll can do by preventing it from ever happening.)

Other times have been where I have had windows up that others might get mad over (such as when I have finished some class work and going to some forum which a teacher has become angry that I was "off task". Obviously I hid it, but I recall once where the teacher unminimized a window and a firefox window instantly reappeared and I was totally thinking W-T-F! The teacher was dumb anyway and though Firefox was some kind of animal racing game and actually became quite mad. To clarify, this was more back in HS and now in college that really isn't too much of a problem now.)

The main reason: I just want reassurance that if I hide a window, then I can rest assure that it will not be shown for any reason again. Hopefully you dont think of this as a hack app, because if anything it is an application that will provide a useful example for future reference or even to help others with similar tasks for this "Multi-Process subclassing." My main motivation is really just to see if it is possible (Obvisouly it is, but I am a visual learner and sometimes can only feel like I truely believe something when I see the action occur)



EDIT: So get this, I just tested CreateWindowEx in a C Dll, and the same behavior occurs (window is removed when the injection thread (LoadLibrary) ends. That is actually good because that means it isn't just something in VB that is causing that behavior and somewhat proves that indeed when the thread ends, the window goes with it.

I wonder if bad stuff will happen if we try to spawn another thread using CreateThread (I recall the results being bad in the past)

In brief:
Yes, I'm using the CreateRemoteThread(LoadLibrary) injection method

Yes, creating window useless in DllMain, as we lose it on return, since we are running in a temporary thread. This is an endemic problem, not VB-specific

CreateThread and VB simply don't get along
When I said before that I had a working model along the lines of code I showed above, this was only partly true. The code was based on real code, but in a PowerBasic DLL, where it worked just fine. I did have lots of trouble with the VB version. :-\

Fortunately that was one of those hard-to-find but easy to fix problems, of which more below :cool:

If we want the controller to be able to unload the library with CRT, then we definitely need the DLL to destroy any windows we created, before we unload, otherwise we'll crash the target, since the window will persist but its WndProc will be invalid. And, just as we can't create in DllMain, neither can we destroy, so we need a controller msg to initiate this


The problems I had implementing the VB version of the DLL turned out to be due to a rather peculiar thing that happens when the VB dll gets loaded - any global data items that get set in DllMain unfortunately are wiped when the DLL is next called. (I seem to remember this problem from long ago!)

This is catastrophic when DllMain initiates the subclassing of a target window, storing as a global item the window's real WndProc.

When a msg does arrive we find that we've forgotten the forwarding address! The shared memory handles we obtained in DllMain have also gone. Very bad ....

DLlMain needs to write any needed items into the shared memory, and then put a test in the WndProc like "If pShared = 0 Then ResetGlobals" - that routine repeats the mem mapping code to get a fresh pointer and loads the saved values from there.

Hope this makes sense!

By the way, in the end I decided that a simpler way of controller-DLL communication is probably one without window creation or subclassing. We can use CRT to call any function in the DLL, so long as takes a single argument. The DLL can return info via shared memory and/or sending text messages to a ListBox in the controller. Nothing is really lost by the function running in a separate thread

One pitfall to avoid is a potential deadlock if the Controller invokes a DLL function via CRT, and the DLL function sends a message to a window in the controller.

If the controller issues the CRT and then waits for the thread to complete using WaitForSingleObject(hThread, -1), then we get a "deadly embrace" - the DLL is waiting for the SendMessage call to complete, but the controller is waiting on the thread and so cannot process messages.

The controller should use a timed wait like this:

rThread = CreateRemoteThread(rHandle, 0&, 0&, Address, Param, 0&, 0&)

Do
DoEvents
Loop Until WaitForSingleObject(rThread, 300) = 0
GetExitCodeThread rThread, result
CloseHandle rThread

Great info Mathimagics.

Sorry I never replied to this sooner. Thought I did a while back, but must have just read it and moved on with college crap.


In any case, here is something for either of you to hopefully answer:
I probably would have PM-ed on of you, but I figured I would post it so others can know if they happen to stumble upon this thread full of this quite valuable info.


1. In a .tlb (well .odl being the source) is there any difference between:
[in] long*
[out] long*

where "long" can be any variable's data type (like void for "As Any")
For anyone curious here is all the data types you can use in a odl:
http://msdn.microsoft.com/en-us/library/ms221332.aspx

I ask this, because it appears doing some testing VB uses these as the EXACT same thing.

So really perhaps my question should be, why is there even an [out] modifier?
My theory is so that is explicitly define in the .odl that the parameter returns some value, even though it does the same thing as [in]

It is funny, [in] is more "flexible" than [out] because it doesn't require that the parameter data type be a pointer "*". By this I mean, with
[out] - it must have a "*" ex: [out] long* or the .odl will not compile into a .tlb




2. This is more of a finding that I would like to say in case others are seeking similar info, but feel free to correct me if this is not 100% correct.

It appears that anything declared as a pointer (whether it be [in] Var* or [out] Var* ) tells VB that it will ONLY accept the parameter as ByRef.
So basically if you try to do ByVal 0, VB will give an error right in the IDE.

This can be a problem if there is an API parameter that can be optional and you declared it as a pointer (where you would need to have the calling parameter say "ByVal 0" meaning a NULL)

It seems that the only way to have VB accept either ByVal or ByRef, is to have the datatype as void*, meaning "ByRef ___ As Any"
For anyone not familiar with VB6 and APIs, kust be careful: When "As Any" you have to force VB to pass the 0 as a long using the "old variable declaration style" by using a "&"
Ex: ByVal 0&




3. My final question is to both of you:
Do either of you, use any kind of identifiers on your variable names to remember that they are declared as ByRef? Ex: "R" as some pre or post script maybe?

I ask this because you to seem to be pretty savvy users of.tlbs and I am curious if you have found it easier to just remember/lookup how it is declared in the .odl, or to embed it into the var name by use of some identifier to help assist with VB6's IntelliSense downfall.

As you say, [out] requires a pointer, and is simply a reinforcement of that. Sure, can be used with a pointer, but if you forget the asterisk, then the odl compiler won't pick that error up.

Personally, I don't tag variable names (ref or non-ref) - if I'm using an API routine via a TLB (or in any case) then I usually know exactly what that API does and what the arguments are - at least, I [i]should know! ;)

Thanks you very much for the input =]