IPC - Semaphores Primer

Semaphores

Introduction

cf: 125706 for a general discussion of IPC

Semaphores are the simplest IPC objects and are commonly used in virtually all Operating Systems for synchronisation.

A semaphore is, as its name suggests, a flag, just like an integer or boolean variable in your problem, but unlike variables in your application programs it can be shared across the system (i.e. between co-operating processes).

It has other special properties, too, that distinguish it from your program variables.

Firstly, it can never hold a negative value. It's always 0, or 1, or 2, etc, up to a maximum value specified when it is initialsed.

Secondly, changing the semaphore's value can only be done in two ways, by adding 1, or subtracting 1.

Basic Operations on Semaphores

A semaphore has only two basic operations, "increment" and "decrement".

The "increment" operation is quite conventional, in that it simply adds 1 to the semaphore value. If the semaphore is already at maximum value, the operation is simply ignored.

The "decrement" operation is the one that is special, and makes the semaphore a useful IPC object. It behaves like this:

if the semaphore value is greater than 0, decrement it and return immediately
if the semaphore value is 0, then WAIT (i.e. suspend execution) until it becomes greater than 0


Clearly, if the value is 0, then the semaphore is effectively locked (aka "not signalled", "not available", "not notified") until some other process unlocks ("signals", "releases", "notifies") it by adding one to the count.

The "increment" operation is commonly called Notify, Signal, or Release (the Win32 API function for this operation is called ReleaseSemaphore, for example).

One final point to note is that the semaphore can be initialised to any valid value when it is created. So it can begin in a "locked" state, or a "signalled" state.

Using Semaphores

Semaphores as Locks

A semaphore with a maximum value of 1 has only 2 possible values, 0 and 1. It can be used, then, as a global boolean flag that can indicate whether some resource is "available" (1) or "not available" (0). Notify(semaphore) makes it available, i.e. unlocked, while Wait(semaphore), when it succeeds, marks it unavailable or busy, in other words, locked!

The operation Wait(semaphore) when used on a lock is called "getting the lock", "obtaining the lock", even "procuring the lock"!


Semaphores as Signals, or Events

A lock and and event are the same thing, just a different set of terms. A lock is either "unlocked" or "locked". An event is either "happened" or "not happened" (aka "signalled/not signalled").

The act that Locks and Events are so similar, and have so many different terms that describe the basic semaphore functions is a common cause of confusion.

Even the term "Event" has two totally different meanings. In the sense of our discussion we are talking about "asynchronous events", that is, things that can be regarded as either "happened" or "not happened", or simply "on" and "off".

Thus the term "signalling an event" simply means setting a boolean flag to true!

But many people nowadays think of Events as things that automatically cause things to happen, like the execution of a "Command1_Click" function when the user clicks a button!

This operation might be more accurately called "pre-emptive interrupt signalling". It's a combination of Events and Interrupts. When a program gets a runtime error, this event is detected by the system, which then forcibly interrupts your program to signal the error.


Options in the Wait function

The semaphore Wait function actually has 3 flavours. These affect what happens if the semaphore is locked (i.e. 0).

Wait for it to be available (as described above)

return after a given timeout value has expired if still locked

return immediately, with "success" or "fail" indicator


The additional options provide useful alternatives to the unconditional wait operation, allowing an interactive program to continue if the lock could not be obtained.


Capacity Semaphores

Because a semaphore can have a maximum value greater than 1, it can be used to control the maximum number of copies, or instances, of some resource.

A good example is limiting the number of copies of an application that can run simultaneously. You know that the App.PrevInstance property can be sued to ensure only 1 copy is running, but what if we want to allow up to 3 copies, but no more?

Here we'd use a semaphore with a maximum value of 3 and an initial value of 3. During startup, our application would begin by issuing a Wait request with immediate return.

If the "Wait" suceeded the semaphore's value will have been decremented and the application proceeds. But the 4th copy will fail on the Wait, as the semaphore value will be 0, so it can simply unload itself.

The application must, of course Notify the semaphore when closing, or the available number of "free" copies is reduced.

It's handy to know here that when ALL the applications have terminated, regardless of whether they actually performed a Notify, then the semaphore is automatically destroyed, and will be recreated and correctly initialised when next the application is started.

API functions for Semaphores

Open and Close

There are two functions for "opening" a Semaphore, and one for "closing". "Opening" and "Closing" refer to the getting of a handle, and releasing that handle. It's like a window handle, that you store as a 32-bit Long.

CreateSemaphore

OpenSemaphore

CloseHandlee

As the names suggest, one is used to create a semaphore, the second is used to get a handle to an existing semaphore, and the third is for closing (all kernel object handles are closed with this function).

Semaphore's are, like files, identified by name. When you design applications that will use semaphores as locks or events, or resource counters, you assign each one a Name (just like the Caption property of a button).

Just like files, these names are global, every program that opens a semaphore called "MyLock" will access the same object!


CreateSemaphore

Some process must create the semaphore initially. [api]CreateSemaphore[/i] has the parameters that are needed to initialise a semaphore:

Name
Initial value
Maximum value


There is another parameter called the "inheritance" flag. This is only relevant when applications are using the CreateProcess API, and for most VB applications (and all examples here) will always be passed as 0.
[list]
the NAME attribute is the global identifier for the semaphore.
the Maximum value must be 1 or more.
the initial value can be any value from 0 to Maximum.

The return value of the function is a handle to the semaphore - like a window handle, this is used to identify the semaphore in all calls to the other semaphore functions.


OpenSemaphore

This the alternative Open method, which just specifies the name, and only returns a handle if the semaphore already exists.


Who should create the semaphores?

If the semaphore already exists, CreateSemaphore still returna a handle to it, ignoring the Initial and Maximum value arguments. Thus it behaves in this case just like OpenSemaphore.

So which form should you use? When would you use OpenSemaphore? This depends on your IPC model.

You may wish that any participating application be able to create the semaphore. This is like a Peer-to-Peer model.

All applications use the same CreateSemaphore call, the first one activated will actually create it, while the others will also get a handle to the same semaphore.

Alternatively, you might have a Client-Server model, where one application is a single-instance server, and other applications are its clients. Although you could still allow clients to create the semaphores, it's probably better to have the Server create the objects.

For one thing, this gives the clients a simple test of whether the Server is actually running. If OpenSemaphore fails, then there's no Server, and the client can report that, perhaps it can initiate the Server automatically. This can be a useful feature.

API functions for Semaphores (ctd)

NotifySemaphore

The actual API name is ReleaseSemaphore, but we'll use this alias as it's more convenient for the discussion, and I use it in all code samples.

This function is called to increment a semaphore, it is passed the semaphore handle and an increment value, which we mostly want to be 1.

It increments the semaphore's value, regardless of its state, and returns immediately. If the semaphore is already at Maximum value, this call is ignored.


WaitSemaphore

The actual API name is WaitForSingleObject, which is a generic Wait function for various IPC objects. This is where it all happens!

This function requests that the semaphore's value be decremented.

We pass in the semaphore handle, and an option that specifies 1 of 3 possible wait methods:

Indefinite wait: the process is suspended if, and until, the semaphore value is greater than zero

Timed wait: the process is suspended only until the time given (in mSecs) has elapsed. On return the caller is informed which case

Test-and-Set: the process is never suspended, the call returns immediately. On return the caller is informed whether the call was successful.


We now have enough background (possibly more than enough :p ) to construct a simple demo program

VB Demo #1 - A Simple Lock

Introduction

Let's say we have several different applications that we wish to be able to log messages into a common file, so we get a sequential record of all messages logged.

Obviously we could use normal VB file operations to do this, using the Locked option on the file Open statement, and error trapping to detect when the file is being used, and to loop until it becomes available.

But using a semaphore is a good example for our topic, and also may surprise how much simpler the coding in our file-contention-check code is (in fact there is just one line!).

Here's an outline of a general-purpose function we'll implement:

Sub LogMessage(message as String)
Wait(LogfileLock)
Write the message to end of log file
Notify(LogfileLock)
End Sub


The IPCapi module

Here's our simple re-usable module for the whole operation. It has a single exported function Logmessage.

It will automatically initialise the lock on the first call. The lock name and logfile path are hardcoded, and the optional variable APPID can be set to insert a common prefix to all messages. Messages have a timestamp included in any case.

Option Explicit
'
' Dr Memory's lock demo Dec 2003
'
Const LOCKID = "LogfileLock" ' Semaphore name for demo
Public APPID As String ' all mesages marked with timestamp + this stub

Dim hLock As Long ' Lock handle
Dim lFile As Integer ' file# of Log File
'
' there are 5 Semaphore-related API functions Create/Open, Wait/Notify and Close
'
Private Declare Function CreateSemaphore Lib "kernel32" Alias "CreateSemaphoreA" _
(ByVal lpInherit As Long, ByVal lInitialCount As Long, ByVal lMaximumCount As Long, _
ByVal lpName As String) As Long

Private Declare Function OpenSemaphore Lib "kernel32" Alias "OpenSemaphoreA" _
(ByVal dwDesiredAccess As Long, ByVal bInheritHandle As Long, _
ByVal lpName As String) As Long

Private Declare Function WaitSemaphore Lib "kernel32" Alias "WaitForSingleObject" _
(ByVal hSemaphore As Long, ByVal wTime As Long) As Long

Private Declare Function NotifySemaphore Lib "kernel32" Alias "ReleaseSemaphore" _
(ByVal hSemaphore As Long, ByVal lReleaseCount As Long, lpPreviousCount As Long) As Long

Private Declare Function CloseSemaphore Lib "kernel32" Alias "Closehandle" _
(ByVal hSemaphore As Long) As Long
'
' special values for wait-time parameter
'
Const WAIT_FOREVER = -1 ' wait until signalled
Const DONT_WAIT = 0& ' test-and-set


This is our LogMessage function:Sub LogMessage(ByVal Message As String)
If lFile = 0 Then Call LogInitialise
WaitSemaphore hLock, WAIT_FOREVER
Open "C:\MathImagics_IPClog.txt" For Append As lFile
Print #lFile, Format(Now, "hh:mm:ss") & APPID & Message
Close lFile
NotifySemaphore hLock, 1, ByVal 0
End Sub

And here's LogInitialise:Private Sub LogInitialise()
'
' Get the Lock handle, specify Max = Init = 1
' to apply if we create it
'
hLock = CreateSemaphore(0&, 1&, 1&, LOCKID)
lFile = FreeFile
Open "C:\MathImagics_IPClog.txt" For Binary As lFile
Close lFile
End Sub

We can now share log files between any sets of processes by using the same lock name and log file.

VB Demo #1 (ctd) - A Simple Application

By pasting a button and a textbox onto a form, and including the IPCapi module, we have a very simple test application to show our IPC lock works.

When the button is pressed, the text in the textbox is logged into the file.
Option Explicit

Private Sub Command1_Click()
Command1.Caption = "waiting"
Me.Enabled = False
LogMessage Text1
Me.Enabled = True
Command1.Caption = "LOG MESSAGE"
End Sub

Private Sub Form_Load()
APPID = " (" & Me.hWnd & ") " ' handy stub for id-ing messages
LogMessage "Started on " & Format(Date, "dd mmm yyyy")
End Sub


Notice that each message inserted in the log file will identify the window of the calling application.


Testing the Lock

Adjust the logfile name if you want, then make an exe, and start that up.

Now start the IDE version running. When it's started, set a break point on the line that says Print in LogMessage. Now press the button and have the program stall at that breakpoint. It now has the lock!

Now press the button in the EXE. It will, of course, hang completely on the WaitSemaphore call, until you allow the IDE version to continue.


Where's the Cleanup Code? :huh:

You'll notice that there's no tidy-up code for the semaphore. Not sloppy programming, there's no need (in this simple case!)

Semaphore objects are automatically closed when the program terminates, and removed altogether when the last handle to the current instance is closed.

The only way to crash this demo is to terminate it while it's got the lock.

Any other copy would then hang if you pressed their button. But all you have to do is NOT press the button, simply close them all and you'll be back to a clean slate.

Of course, accidents WILL happen ... :p

Crash recovery

What if the application terminates or crashes crashes while it has the lock?

This can be simulated easily. Start a copy of the EXE. In the IDE insert "Debug.Print 1 / 0" inside Logmessage, and choose END when the error is signalled.

The exe will hang when you press the button. You can use Task manager to kill it, or you can restart your IDE, set the breakpoint on the WaitSemaphore call, and when it gets there, go directly to the NotifySemaphore call.

The EXE should make a miraculous recovery!

Otherwise, the only way to recover from hung applications is to use Task Manager to kill the process.

Fortunately, even this method of termination will close the semaphore handles properly. Once all users of the lock have been disposed of, the next user will get a brand new lock.

Here is a very simple event signalling model. We have an application that runs in the background, perhaps we start it up at boot time, in any case we want it to do something only when SIGNALLED to do so by another application.

This is the simplest of all client/server models - the server "sleeps", and only does its thing (whatver that is) when it is "woken up" (signalled) by a client.

In semaphore terms, the server sits in a loop like this:

Do
WaitForSignal EventId
PerformTask
Loop

and any client app that knows the correct "EventId" can get the server to "performTask" by making a "wakeup call" - i.e. by signalling the semaphore.


The SERVER for this demo will be a simple Form-less EXE, with a loop in Sub Main. Whenever a client makes a "wakeup" call, it will display a MsgBox.

The CLIENT will just be a single Form with a button - press the button and it will make the "wakeup" call, and a MsgBox will be displayed by the Server.

The server here is a simple module (set the project's startup object to "Sub Main"):

Option Explicit
Const EVENTID = "EventDemo" ' Semaphore name for demo
Dim hEVENT As Long ' Semaphore handle

Private Declare Function CreateSemaphore Lib "kernel32" Alias "CreateSemaphoreA" _
(ByVal lpInherit As Long, ByVal lInitialCount As Long, ByVal lMaximumCount As Long, _
ByVal lpName As String) As Long

Private Declare Function WaitSemaphore Lib "kernel32" Alias "WaitForSingleObject" _
(ByVal hSemaphore As Long, ByVal wTime As Long) As Long

Const WAIT_FOREVER = -1 ' wait until signalled
Const DONT_WAIT = 0& ' test-and-set

Private Sub Main()
'
' Create the semaphore
'
hEVENT = CreateSemaphore(0, 0, 1, EVENTID)
MsgBox hEVENT

'
' Loop: Wait for signal, display a message when signalled
'
Do
WaitSemaphore hEVENT, WAIT_FOREVER
MsgBox Time$ & vbLf & "Client signalled"
Loop
End Sub


Note that in this simple coding, it's a "run forever" loop. You can add a button-check option into the MsgBox call to allow the loop to exit and terminate the program.

A simple client Form, it gets a handle to the server's semaphore in Form_Load, and signals the semaphore when you press Command1
Option Explicit
Const EVENTID = "EventDemo" ' Semaphore name for demo
Dim hEVENT As Long ' Semaphore handle

Private Declare Function OpenSemaphoreA Lib "kernel32" _
(ByVal dwDesiredAccess As Long, ByVal bInheritHandle As Long, _
ByVal lpName As String) As Long

Private Declare Function SignalEvent Lib "kernel32" Alias "ReleaseSemaphore" _
(ByVal hSemaphore As Long, ByVal lReleaseCount As Long, lpPreviousCount As Long) As Long

Private Const SEMAPHORE_ALL_ACCESS = &H1F0003


Private Sub Command1_Click()
SignalEvent hEVENT, 1, ByVal 0
End Sub


Private Sub Form_Load()
hEVENT = OpenSemaphoreA(SEMAPHORE_ALL_ACCESS, 0, EVENTID)

If hEVENT = 0 Then
MsgBox "Server not running", vbExclamation
Unload Me
End If
End Sub



Testing: compile the server as an EXE and run it, then you can run the client from the IDE. You can make the server easy to terminate by changing the MsgBox call to this:

If MsgBox(Time$ & vbLf & "Client signalled" _
& vbLf & vbLf & "Continue?", vbYesNo) = vbNo Then Exit Do

The example above (Demo #3) uses WAIT_FOREVER in the server loop. That means exactly that, no further processing is possible until the semaphore is notified.

That's fine if the server's task is purely data processing - if the server has a GUI of any description (i.e. shows any non-modal Form), then that Form is also frozen as it can't receive any windows messages, and can't be refreshed.

If we want to show a Form then we need to to change to other wait option, which is "DONT_WAIT", and include a DoEvents in the loop. DONT_WAIT returns immediately and we can tell from the return value whether or not the semaphore was signalled. DONT_WAIT is like a "Test for signal" request.

Only if the function returns zero has the semaphore been signalled.

For this demo, we use the same Client. Add a Form to the Server project, and put a ListBox on the Form.

When the client signals the event the server will log the current time into the ListBox.

A "DoEvents" call is included in the loop that services the Form by allowing outstanding messages to be processed.

A "Sleep 0" api call is placed at the end of the loop, that minimises the amount of CPU that the server uses by surrendering the current thread timeslice.

For convenience, I've added a QuitFlag to allow the Server to be easily terminate. If you add "QuitFlag = True" to the form's Unload event, then closing the form will close the server.


Option Explicit
'
' Server with GUI
'
Const EVENTID = "EventDemo" ' Semaphore name for demo
Dim hEVENT As Long ' Semaphore handle

Private Declare Function CreateSemaphore Lib "kernel32" Alias "CreateSemaphoreA" _
(ByVal lpInherit As Long, ByVal lInitialCount As Long, ByVal lMaximumCount As Long, _
ByVal lpName As String) As Long

Private Declare Function WaitSemaphore Lib "kernel32" Alias "WaitForSingleObject" _
(ByVal hSemaphore As Long, ByVal wTime As Long) As Long


Const WAIT_FOREVER = -1 ' wait until signalled
Const DONT_WAIT = 0& ' test-and-set

Private Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
Public QuitFlag As Boolean

Private Sub Main()
'
' Create the semaphore
'
hEVENT = CreateSemaphore(0, 0, 1, EVENTID)
Form1.Show
'
' Loop: Wait for signal, log Time in listbox when signalled
'
Do
DoEvents
If QuitFlag Then Exit Do
If WaitSemaphore(hEVENT, DONT_WAIT) = 0 Then
Form1.List1.AddItem Time$
End If
Sleep 0 ' surrender cpu to other processes
Loop
End Sub

In the Client code for Demos 3 and 4, I have omitted a cleanup call that should be placed in Form_Unload:
CloseSemaphore hEvent

If you are running the client in the IDE, then once a Server copy has created the semaphore, it will not be destroyed properly until you exit the IDE session. So subsequent runs will still think the Server is running, whether it is or not.

Closing the semaphore handle (as I should have done) will prevent this problem. ;)