Subclassing modeless UserForms

This is probably one for Timbo, though of course free for all... ;)

Is it generally impossible to subclass Excel UserForms that are shown modeless? If I try to do that, the form freezes as soon as the mouse pointer enters the form's title bar, so e.g. moving or closing the form is impossible while it's subclassed. Usually, one can unfreeze it by switching to a different thread and then back to Excel, thought sometimes not. I'm pretty sure the problem is due to an infinite message loop, but have no idea what to do about it.

Applying exactly the same subclassing code to a modal form works peachy... well, let's say as well as one could expect IDE subclassing to work. As long as one doesn't do anything silly, it's as stable as one could hope.

I'm attaching a sample workbook with a subclassable userform. The buttons should be self-explanatory, the window proc displays the message name in the label, just to give some feedback.

ps: I'm getting the form's hWnd using GetForegroundWindow, which I at first thought might be related to the problem. But it isn't, I tried all other ways I could think of (FindWindow, EnumThreadWindows, navigating down from Application.hWnd or up from a child window's hWnd) and the problem is unaffected.

pps: Tried google, obviously, couldn't find anywhere discussing this.

Thanks for taking a look. :)

Update: Commented extracts from message logs for different cases. Legend: "message -> return value"

modal/form/move (= form is modal/subclassed the form/moved the form)

'move cursor to title bar, then click and hold on title bar, then drag
[...]
WM_NCHITTEST -> 2
WM_NCHITTEST -> 2
WM_NCHITTEST -> 2
WM_SETCURSOR -> 0
WM_CAPTURECHANGED -> 0
WM_GETMINMAXINFO -> 1
WM_WINDOWPOSCHANGING -> 0
WM_ENTERSIZEMOVE -> 0
WM_MOVING -> 0
WM_WINDOWPOSCHANGING -> 0
WM_MOVE -> 0
WM_WINDOWPOSCHANGED -> 0
WM_MOVING -> 0
WM_WINDOWPOSCHANGING -> 0
WM_MOVE -> 0
WM_WINDOWPOSCHANGED -> 0
WM_MOVING -> 0
WM_WINDOWPOSCHANGING -> 0
WM_MOVE -> 0
WM_WINDOWPOSCHANGED -> 0
[...]


modal/client/move (= form is modal/subclassed the client area/moved the form)

'move cursor to title bar, then click and hold on title bar, then drag
[...]
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_SETCURSOR -> 1
WM_MOUSEFIRST -> 0
WM_WINDOWPOSCHANGING -> 0
WM_PAINT -> 0
WM_WINDOWPOSCHANGING -> 0
WM_PAINT -> 0
WM_WINDOWPOSCHANGING -> 0
WM_PAINT -> 0
[...]


modal/form/close

'move cursor to [x] button, then click and release
[...]
WM_NCHITTEST -> 20
WM_NCHITTEST -> 20
WM_NCHITTEST -> 20
WM_SETCURSOR -> 0
WM_CAPTURECHANGED -> 0
WM_SHOWWINDOW -> 0
WM_WINDOWPOSCHANGING -> 0
WM_WINDOWPOSCHANGED -> 0
WM_NCACTIVATE -> 1
WM_ACTIVATE -> 1
WM_WINDOWPOSCHANGING -> 0
WM_PARENTNOTIFY -> 0
'end subclass


modal/client/close

'move cursor to [x] button, then click and release
[...]
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_SETCURSOR -> 1
WM_MOUSEFIRST -> 0
WM_KILLFOCUS -> 0
WM_WINDOWPOSCHANGING -> 0
WM_CHILDACTIVATE -> 0
WM_WINDOWPOSCHANGED -> 0
'end subclass


modeless/form/click title bar

'move cursor to title bar
[...]
WM_SETCURSOR -> 0
WM_SETCURSOR -> 0
WM_SETCURSOR -> 0
'click in title bar -> freeze
WM_NCHITTEST -> 2
WM_NCHITTEST -> 2
WM_NCHITTEST -> 2
[...]
'frozen


modeless/client/click client area

'move cursor to client area
[...]
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
'click in client area ->freeze
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
WM_NCHITTEST -> 1
[...]
'frozen

So, in brief, a click in the interactive area of a subclassed window (client area for client area window, title bar for form window) that is part of a modeless form results in an infinite NonClientHitTest message loop. Why?

I'll have to experiment with altering the return value, I guess...

I've been down this same road myself Cas, and it took literally months to get to the bottom of it. Finally some kind VB6 expert on another forum took pity on me and suggested I detroy the subclass, then recreate it on every callback. It's been a while since I went over this, so apologies I can't recall why this is necessary, but it should solve the issue :)


ps. I do recommend that you use FindWindowEx to return the hWnd however, since clicking on the worksheet makes that the foreground window and you'll find yourself subclassing that instead!

I've been down this same road myself Cas
Lucky me! :)
detroy the subclass, then recreate it on every callback.
:eek: I don't get it. Do you mean I should SetWindowLong to the window classes WndProc and then back to my WndProc inside my WndProc? Before or after or instead of CallWindowProc?
I do recommend that you use FindWindowEx to return the hWnd however, since clicking on the worksheet makes that the foreground window and you'll find yourself subclassing that instead!
The "safest" thing I could come up with is to change the form's caption to something unique temporarily and then use FindWindow on that (no Ex, UserForms are top level). Either way, I would only have to do that once, not on every destruction/recreation cycle, right?

Thanks a bunch! :cool:

Do you mean I should SetWindowLong to the window classes WndProc and then back to my WndProc inside my WndProc?
You can just call the "StopProcessing" & "StartProcessing" subs at the end of your WinProc :)
no Ex, UserForms are top level
True, but they are still children of an XLMAIN instance, and catching the right one is important ;)

True, but they are still children of an XLMAIN instance, and catching the right one is important ;)
:confused: My UserForms are children of the Desktop. They're owned by an XLMAIN instance, but FindWindowEx doesn't care about that, does it?

XLMAIN is also the Parent, are you using SPY++ or similar?

No, I was just using GetParent and GetAncestor APIs, the latter of which takes a GA_PARENT/GA_ROOT/GA_ROOTOWNER argument.

The first thing I tried was FindWindowEx with Application.hWnd, which didn't retrieve the UserForm. But what you said earlier sounded like that might have been the wrong window, so I'll try that again with each of the XLMAIN-class windows. :)

Cas, I owe you an apology - I've been leading you on a wild goose-chase :whoops:

You were right in your previous assessment about the use of FindWindow vs FindWindowEx with the UserForm, so please ignore everything I said to the contrary! Am I too young to blame this on dementia? :rolleyes:

Cas, I owe you an apology
Considering that I'd probably never ever have figured this
destroy the subclass, then recreate it on every callback
out on my own, I'm somewhat inclined to forgive you... ;)

Seriously, I'm extremely pleased with your answer, I hoped you'd have come across this before but never expected a suggestion as simple and to the point as that!!

*goes off to test if it actually works*

ps:
Am I too young to blame this on dementia?
There's always the "lobotomy" excuse... :p

pps:
I've been leading you on a wild goose-chase
You mean "I could be pursuing an untamed ornithoid without cause"? (Anyone recognize the quote? :))

It doesn't work! :p

Seriously, it doesn't, by which I mean it behaves just as before. I'm hoping I misunderstood your instructions... here's how I implemented it:
Private hWndProcessed As Long
Private lpPrevWndFunc As Long

Public Sub startProcessing(Optional whichHWnd As Long = -1)

If (whichHWnd = -1) Then
Call SetWindowLong(hWndProcessed, GWL_WNDPROC, AddressOf processMessage)
Else
hWndProcessed = whichHWnd
lpPrevWndFunc = SetWindowLong(hWndProcessed, GWL_WNDPROC, AddressOf processMessage)
End If

End Sub

Public Sub stopProcessing()
Call SetWindowLong(hWndProcessed, GWL_WNDPROC, lpPrevWndFunc)
End Sub

Public Function processMessage(ByVal hWnd As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long

formTest.lblMessage.Caption = messageName(uMsg)

'processMessage = CallWindowProc(lpPrevWndFunc, hWnd, uMsg, wParam, lParam)
Call stopProcessing
'processMessage = CallWindowProc(lpPrevWndFunc, hWnd, uMsg, wParam, lParam)
If Not ((uMsg = WMList.WM_DESTROY) Or (uMsg = WMList.WM_NCDESTROY)) Then Call startProcessing
'processMessage = CallWindowProc(lpPrevWndFunc, hWnd, uMsg, wParam, lParam)

End Function

I call startProcessing with the form's hWnd from btnStart, stopProcessing from btnStop. I wasn't sure where to put the CallWindowProc call, so I uncommented each of the three in turn - no joy.

Please tell me I misunderstood you...

Odd, because this time did actually test my suggestion first to make sure it worked for me. Have you traced the literal execution path through? I just used the bog standard "start" & "stop" procedures you had earlier at the very end of the WinProc (minus your new condition etc) and it worked fine :-\

I'll have a play again on Tuesday if you've not cracked it sooner :)

Could you post your version?

Here ya go Cas :)

Cheers. I really want to get to the bottom of this. :)

Either there's something really really obvious that I'm overlooking, or we're actually seeing different behaviours. Running your example exactly as-is, this is what I'm doing/getting:

Start form.
Click on Modeless.
Click on (sometimes even move to) titlebar -> form freezes.
Steal focus by switching to a different process.
Return to Excel.
Form behaves normally now.

Making a few changes to get status info (debug.prints and using the label), the behavious stays the same and I see that the reason the form behaves normally in the end is that it isn't subclassed any more.
Now, if I understood you correcly, your form doesn't freeze at all, right? If you comment out clearing the label in the Stop proc, can you check if that's due to not being subclassed at all, since that's what I observe after stealing focus?

Not that that suggestion makes any sense, I'm just fresh out of ideas otherwise... :confused:

Woohoohoohoo,

I got it to work:

stopProcessing
DoEvents
startProcessing

Even makes sense, for a change... :cool:

I'd still be interested to know why it works for you without. Probably something about the Excel version and OS/SP we're using, right?

I'm using Excel 2002 SP3, but now you mention it, I did use DoEvents in my much earlier foray into Excel subclassing. Glad you figured it out though! :)

I'm using Excel 2002 SP3
Ah, I didn't even know there were SPs for Office. :o I'll download SP3 and see if that makes a difference.
I did use DoEvents in my much earlier foray into Excel subclassing
:cool: Do you remember ever having had problems with missing events, in that case? That's the one possible side-effect that occured to me after a bit.
Glad you figured it out though!
Yeah, same here. It's not like I've been working on it more than half an hour every other day or so, but it was starting to get a little frustrating.

Thanks again for the basic solution, as I said there's no way I would have thought of that myself. :)
I did think of two other potential ways to circumnavigate the problem, though, but both are rather involved and probably not very stable. I might try them out sometime, anyway...

Do you remember ever having had problems with missing events,
Do you mean missing messages? There's such a flood of them anyway I think I would have missed anything that was missing! :D

I mean both. :)
There's the background noise of HitTest and MouseMove messages, but there's also the event-y kind like a click, a keypress or an activate. And it's one of the latter ones that would be noticable if missed - if the user has to press the mouse button several times to get the button to click, for example.

Mind you, I'm not sure that this could ever actually become a problem, it may well be that the only situation in which "our" hWnd misses messages during the DoEvents interval is the very one we want it for, the one that would make the window freeze up otherwise.

I don't think DoEvents would have that effect though. All messages are places in a que and that que always gets sent to the target windows proc. AFAIK, DoEvents just allows other windows to "catch up".

Yes - but no! :)

I'm not definite on this either, but I imagine it like this:

Known situation: game loop running, user clicks on button, botton code is "inserted" into execution order:

1: Do
2: DoEvents
3: Loop

4: Sub btn_Click()
5: MsgBox "Click"
6: End Sub

Ignoring everything outside the process, the execution order is like this:
2,3,2,3,2,3,2, 4,5,6, 3,2,3,2,3,2,3
At every 2, the message pipeline is checked and if there are pending messages, they're dispatched to their window procs and thence distributed to their addressees.

Now, we have a similar situation with the class window proc and subclass window proc:

1: Function wndProc(msg)
2: If msg = 42 Then wndProc = drawOnTheCeiling Else wndProc = clsWndProc(msg)
3: SetWindowLong(clsWndProc)
4: DoEvents
5: SetWindowLong(wndProc)
6: End Function

7: Function clsWndProc(msg)
8: clsWndProc = DefWndProc(msg)
9: End Function

The wndProc replaces the loop as framework. As long as the messages arrive slower than the wndProc can handle them, it just runs through it again and again and all the DoEvents does in-process is check the pipeline and find it empty. If it doesn't find it empty, though, because the messages arrive faster than wndProc handles them, DoEvents should have the effect of emptying the pipeline into the current window proc, which is clsWndProc. And if a 42 happens to be among those messages, it would be missed - not by the window, but by our code whose point is to wait for 42s and then paint the ceiling. Instead, the 42 is e.g. just passed through to the default window proc which might do something quite different with it. :eek:

Does that roughly match how you imagine this to work?

I just had a look at Merrion's DoEvents replacement (http://www.xtremevbtalk.com/showthread.php?t=213459&highlight=doevents) in the KB, and am wondering if that might be a solution to this potential pitfall. By dispatching the messages ourselves, we could check them for 42s one by one. But by rights that should re-freeze the window, methinks. :confused:

I should probably just stop worrying about this until the problem actually occurs. :)

I should probably just stop worrying about this until the problem actually occurs.
lol - I concur!

However I'll stick my neck out here and say that while you appear to be suggesting that windows messages are asynchronous, I think I'm right in saying that despite appearances, Windows is actually synchronous in behaviour. That is to say, all messages are sent in one long que, however Windows cleverly cycles through each *process* and executes a few messages from each in order to keep the appearance of multi-tasking.

Thus DoEvents simply instructs Windows to stop "concentrating" on one process, and to break & run through all processes, allowing them to all "catch up"!

Don't you just love my tech-jargon? :D

For reference of anyone who comes across this thread in the future, I implemented the solution we were talking about, expanded the earlier test project a little, cleaned it up and put in some classes just to prove that subclassing and OO aren't mutually exclusive. ;)

I find it pretty cute... even though it doesn't really do anything, other than put a message log on your desktop. :rolleyes:

I've been following up on my idea of replacing DoEvents by message APIs, and have come to realize that it's in fact possible to replace subclassing by message APIs altogether - which might be a much better way to go in VBA! :)

The MSDN article "About Messages and Message Queues (http://msdn.microsoft.com/en-us/library/ms644927(VS.85).aspx)" is surprisingly well-written; worth a read.
The way it works is like this: The system maintains one message queue per thread, and places messages for that thread into the queue as they arrive. The thread runs a message loop which removes the messages from the queue and dispatches them to the individual window procedures associated with the thread. And since we have full access to the thread queue, we can go as far as entirely replacing Excel's message loop with our own if we want to. Meaning full access to messages to all windows without any subclassing. :cool:
The problem is that Excel's message loop does actual work, and by dispatching messages ourselves we're removing functionality. But it should be possible to intercept only those messages we want and yield control to Excel's loop for the remainder, it's only a matter of fine-tuning. The two extremes are these:

Let Excel do everything (classic game loop)
Do
DoEvents
Loop

Do everything ourselves (with interesting but harmless effects when trying to use Excel)
Dim thisMessage As ApiMessage
Do
Call apiGetMessage(thisMessage, 0, 0, 0) 'get all messages for all windows
Call apiDispatchMessage(thisMessage) 'and dispatch them
Loop

The second approach can easily be extended to do things that are traditionally done with subclassing, e.g.

Do everything ourselves and scroll the window caption on mouse wheel :cool:
Dim thisMessage As ApiMessage
Do
Call apiGetMessage(thisMessage, 0, 0, 0) 'get all messages for all windows
With thisMessage
If (.hWnd = hWnd) And (.nMessage = WMList.WM_MOUSEWHEEL) Then
Me.Caption = VBA.Right$(Me.Caption, 1) & VBA.Left$(Me.Caption, VBA.Len(Me.Caption) - 1)
End If
End With
Call apiDispatchMessage(thisMessage) 'and dispatch them
Loop

Will report back with progress. Hopefully. :p

Booyaa, I got it to work. And how.

For Forms, this is really way superior to subclassing when running in the IDE. Not only is it solid as rock, but it also gives one access to the entire message queue rather than just those messages that are posted to one measly little window, so it makes a dozen issues that require work-arounds in a subclassing approach disappear.
I'm not sure yet if it can be adopted to work for Worksheets, because as I said above the Excel message loop is doing some of the work in providing user input functionality, such as putting a selected cell in edit mode when one starts to type. It should in principle be possible to duplicate all of this, but it'd take some effort and tons of testing until one can be sure that nothing has been missed.

Anyway, I'm mainly interested in Forms at this time, so it doesn't worry me much. I was hoping to have a demo workbook ready to post today, but adding more stuff kept getting in the way of cleaning up the old, so it's not really in any shape for public consumption yet. Tomorrow, definitely. :cool:

This is the outline of the final structure of the message loop, pseudocode and VBA excerpt:

do
if the form is active

if there are messages in the queue
if the next message is for me
process it myself if it's interesting, otherwise dispatch (to my window procedure)
else
dispatch to whatever window it's for
end if
else
wait for new messages
end if

else
defer to excel's message loop
end if
loop

Do
If apiGetActiveWindow = myHWnd Then

If apiPeekMessage(thisMessage, 0, 0, 0, PM_REMOVE) Then
If thisMessage.hwnd = myHWnd Then
If Not processMessage(thisMessage) Then Call dispatchMessage(thisMessage)
Else
Call dispatchMessage(thisMessage)
End If
Else
Call apiWaitMessage
End If

Else
DoEvents
End If

Loop Until myDoEndLoop

Definitely very interesting stuff Cas!

It appears you've researched this approach in depth, are there any resources that make it clear why this isn't as "mainstream" as subclassing?

Well, the first thing one has to realize is that any non-RAD application, e.g. everything written in C++, needs to do both of these things anyway, since the wrapper layers that e.g. VB gives us are non-existant. So, a typical pure-API application will almost invariably have a main procedure that looks like this:

procedure WinMain
register custom window classes (i.e. forms and user controls in VB parlance)
instantiate and show top-level window
message loop
end procedure

The message loop runs until the application ends. That's really all there's to it, basically.

Now, there are three main channels of communication:

System to WinMain - the system creates a message queue for each thread, and posts messages to it that concern the application. The message loop can use the GetMessage and PeekMessage APIs to retrieve these messages without having to worry about the inner workings of the queue. Each message consists of six parameters, the four we are used to from the WindowProc parameter list (i.e. a message identifier, a window handle, and two numerical parameters with no fixed meaning), plus an event signature for all user-input type messages (a POINTAPI for the cursor position and the tick count). Typically, the message loop will only examine the hWnd parameter at first. If it's zero, the message is directed to the thread itself, so the message loop needs to process it directly.


WinMain to WindowProc - If the hWnd is non-zero, the message loop will use the DispatchMessage API to request that the system send it on to the applicable WindowProc. It's actually quite possible to do this directly, i.e. one can use GetWindowLong to retrieve the address of the WindowProc from the handle and then call it directly, but the system does some intermediate processing to some messages that make this inadvisable in some cases.


System to WindowProc - The system does also send messages to individual windows directly, however. Mainly, this concerns messages that must be processed directly, rather than just put into the queue which is more or less last-in last-out. "More or less" meaning that the message loop is free to retrieve messages in any order, but it generally chooses not to do so, for obvious reasons. Using the SendMessage API uses the same communication mode as this, it goes directly to the WindowProc associated with the specified handle.

So, all that being said, it makes very little difference for an application like this whether it executes its custom "event handlers" for posted (queued) messages from the message loop or from the WindowProc, since it needs to implement both of these itself anyway. This is the crucial difference from VB, where we don't need to do either - the compiler does that for us by implementing function templates.
It seems to me that the main reason for implementing code that concerns a given window from the WindowProc rather than the message loop is merely simplicity of design here - you get all the messages (posted and queued) that concern the window in one place, and you get none of the messages that don't concern the window. Just as, in VB, we'd rather implement code that should be executed when the user clicks on a button in the button's click handler rather than run a game loop in the form and keep track of the mouse button states, say. :)

In conclusion, the obvious disadvantage of my approach is that it isn't as powerful as subclassing in that there are messages that pass it by. As long as we're interested in user input, though, we're just fine. The second disadvantage, which I already mentioned, is that we're interfering with the pre-implemented message loop that we're running under, be it Excel's or the VB-runtime's. If that loop does more than just call DispatchMessage, which in Excel's case it evidently does, we either need to duplicate this aspect or yield to the main loop as appropriate. Furthermore, VB's .Show vbModal technique is a bit of a brute to deal with in this approach, as by its nature it interrupts all code running outside of the form's scope.
The obvious advantages are those I've already stated - many less workarounds required once one gets the loop up and running, as it's sitting at the source, and no stability issues in VBA because no usage of callback functions at all.

It's more powerful in some ways and less so in others, and it's a trade-off of added complexity and added stability. I think the main reason that subclassing is so much more prevalent in VB is that the stability issue mostly applies to VBA, and not many people go to these lengths in VBA. Still, I found quite a few threads during my research over the last week that would certainly have profited from using this technique in VB proper as well, such as several by Dr. Memory about popup-menu manipulation.

Sorry for the ramble, but I'm still quite excited about this whole discovery. Oh, and I should mention that I've never developed a GUI app in C++; the above is just the understanding I gained from reading a large portion of the MSDN sections on Messages and Windowing. So I may have got a few of the details wrong, but the fundamentals seemed quite straightforward. :)

Keep on digging Cas, I wish I had the time to pick up a spade with you - but deadlines loom... :(

But on the face of it, this sounds perfect for VBA encapsulation (single classes) since as you say there's no requirement for callbacks built into a separate module :)

But on the face of it, this sounds perfect for VBA encapsulation (single classes) since as you say there's no requirement for callbacks built into a separate module :)
True. My demo project is using a dozen classes by now rather than a single one, but that's just a matter of design choices. I'm using two modules, one for global singletons and one for type declarations, but both of these are not strictly necessary.

I've now taken the last self-imposed hurdle, namely figuring out from within a form if it's being shown modal or modeless. Can't believe how difficult that was - the Help claims that there's a .ShowModal property which is read-only at runtime, but in fact it's design-time only, period. I found one of your old threads where you asked the same question, although it was focussed on changing the modality state rather than reading it.
Now, I want to expand the timer feature (also very simple in this approach - no callback, no separate loop, just listening for WM_TIMER) built into my framework and have a last clean-up go at the code, and then it's ready for publication. Should be this afternoon unless anything unforeseen comes up.

D'uh, I just jinxed it, didn't I. :eek:

*wipes sweat off forehead*

Done. Finally. As happened several times before, I thought I had found a solution because a technique (in this case, detecting modality) worked with one form, so naturally it would also work with many forms open at the same time. Not.

I'll write up a description now and post it in the tech forum (http://www.xtremevbtalk.com/showthread.php?t=297736). Have a look and tell me what you think. Even if you're not into the whole subclassing problem, have a look anyway, the demo itself is pretty cool. If I may say so myself. :cool:

Aha, it seems my earlier suspicion about missing messages when using the DoEvents anti-freeze workaround were well-founded. I've started playing with menus now, since a dynamic menu system would seem the natural first useful implementation of the message loop technique, and found that the subclassing approach wouldn't work at all for this.

The message sequence for selecting an item from a first-level submenu is the following (extracted from the message log for a subclassed modal form without the workaround):

WM_ENTERMENULOOP -> 0
WM_INITMENU -> 0
WM_MENUSELECT -> 0 'select the menu bar item
WM_INITMENUPOPUP -> 0 'show the menu
WM_MENUSELECT -> 0 'select the menu item
WM_INITMENUPOPUP -> 0 'show the submenu
WM_MENUSELECT -> 0 'select the submenu item
WM_UNINITMENUPOPUP -> 0 'hide the submenu
WM_UNINITMENUPOPUP -> 0 'hide the menu
WM_EXITMENULOOP -> 0
WM_COMMAND -> 0 'identify the clicked-on item

With the workaround... we get zilch. The menu is shown modally itself, so naturally the windowproc will hang in the DoEvents while it's shown and miss all the internal menu messages, but it doesn't even get the WM_COMMAND, which is the real prize.

Good thing I changed approaches entirely. :cool: