This case is a more complex. Basically, you need to study the API and figure out how you should report about errors in your function. You can not use arbitrary nor default way - because API is already established by someone. It's not you who develop API. You only develop a DLL.

Note: when you write EurekaLog-enabled DLL to be used in non EurekaLog-enabled host (such as Explorer, Internet Explorer, Microsoft Office, etc.) - you have to compile your DLL with "Standalone DLL" profile. See also.

Let's consider a little example. Suppose that you want to write a global system hook - the one that is installed via SetWindowsHookEx function. Global hook requires you to place your handler code inside DLL, so that DLL can be injected in all running programs (which makes the hook a global one).

Naturally, API (i.e. communication rules between OS and your code) is already established - it's defined by Microsoft (as a developer of hooking functions). Therefore, the first thing that you should do is to study documentation for the functions. You pass a pointer to your handler's code via second argument in SetWindowsHookEx function (lpfn param). Prototype of the handler depends on what kind of hook do you want to use. Let's use WH_GETMESSAGE hook for this example. This means that we must study description of GetMsgProc callback.

The important part for error handling looks like this:

If code is less than zero, the hook procedure must return the value returned by CallNextHookEx.

If code is greater than or equal to zero, it is highly recommended that you call CallNextHookEx and return the value it returns; otherwise, other applications that have installed WH_GETMESSAGE hooks will not receive hook notifications and may behave incorrectly as a result. If the hook procedure does not call CallNextHookEx, the return value should be zero.

In other words, your code could not report any failure reason. All that you can do is either return 0 or return whatever CallNextHookEx returns.

Therefore, your DLL code must looks at least like this:

library Project2;
 
uses
  Windows;
 
function MyHook(Code: Integer; _wParam: WPARAM; _lParam: LPARAM): LRESULT; stdcall;
begin
  try
    if Code >= 0 then
    begin
      // <- your code
    end;
  except
    // There is no way to report errors, 

    // so we must handle all exceptions
  end;
  Result := CallNextHookEx(Code, _wParam, _lParam);
end;  

Usually it's not a good idea to silently hide all exceptions. If API doesn't allow you to report about errors - then you should at least implement some kind of logging, so you can store information about exception in the log. Some possible solutions for except block are examined in this article.

Let's see another example. Suppose you're writing a control panel applet without using any framework. This means that you must write and register DLL. DLL must export CPlApplet function. This function will be used for all communication between OS and your code. Description of CPlApplet says:

The return value depends on the message.

For more information, see the descriptions of the individual Control Panel messages.

This means that you also must study each message from the system that you want to process. Luckily, most messages require you to handle errors in the same way:

If the CPlApplet function processes this message successfully, the return value is zero; otherwise, it is nonzero.

So, you should write your DLL at least like this:

library Project2;
 
uses
  Windows;
 
function CPlApplet(hwndCPl: HWND; uMsg: UINT; lParam1, lParam2: LPARAM): LongInt; stdcall;
begin
  try
    case uMsg of
      ...
    end;
 
    Result := 0;
  except
    Result := 1;
  end;
end;
 
exports
  CPlApplet;
 
end.

Since you can't report what is actual report is - a good idea would be to report error to the user. We can safely do this because control panel applet is a single interactive GUI application. Showing error as dialog box is not a good idea for non-interactive applications (such as services) or code that may be used multiple times (such as global hook).

library Project2;
 
uses
  Windows;
 
function CPlApplet(hwndCPl: HWND; uMsg: UINT; lParam1, lParam2: LPARAM): LongInt; stdcall;
begin
  try
    case uMsg of
      ...
    end;
 
    Result := 0;
  except
    on E: Exception do
    begin
      MessageBox(hwndCPl, PChar(E.Message), 

        'Error', MB_OK or MB_ICONERROR);
      Result := 1;
    end;
  end;
end;
 
exports
  CPlApplet;
 
end.

Of course, since you can show error message - you can show the entire bug report instead:

library Project2;
 
uses
  // EurekaLog units for "Standalone DLL":
  EMemLeaks,
  EResLeaks,
  EDialogWinAPIMSClassic,
  EDialogWinAPIEurekaLogDetailed,
  EDialogWinAPIStepsToReproduce,
  EDebugExports,
  ExceptionLog7,
 
  // EurekaLog units for our code
  EExceptionManager,
 
  Windows;
 
function CPlApplet(hwndCPl: HWND; uMsg: UINT; 
  lParam1, lParam2: LPARAM): LongInt; stdcall;
begin
  try
    case uMsg of
      ...
    end;
 
    Result := 0;
  except
    on E: Exception do
    begin
      ExceptionManager.Handle(E, ExceptAddr);
      Result := 1;
    end;
  end;
end;
 
exports
  CPlApplet;
 
end.

(Of course, your DLL must be compiled with EurekaLog enabled, DLL must use "Standalone DLL" profile, you should configure dialogs, bug reports, sending).

Please note that code above is just example. Not all messages to control panel applet have the same requirements. You should study description of each message that you're going to handle in your code. For example, CPL_INIT message has different requirements:

If initialization succeeds, the CPlApplet function should return nonzero. Otherwise, it should return zero.

If CPlApplet returns zero, the controlling application ends communication and releases the DLL containing the Control Panel application.

Therefore, you need to use such code to handle CPL_INIT message:

library Project2;
 
uses
  Windows;
 
function CPlApplet(hwndCPl: HWND; uMsg: UINT; 

  lParam1, lParam2: LPARAM): LongInt; stdcall;
var
  SuccessCode, FailureCode: LongInt;
begin
  // "If initialization succeeds, the CPlApplet function should return nonzero. 

  // Otherwise, it should return zero."
  if uMsg = CPL_INIT then
  begin
    SuccessCode := 1;
    FailureCode := 0;
  end
  else
  // "If the CPlApplet function processes this message successfully, 

  // the return value is zero; otherwise, it is nonzero."
  begin
    SuccessCode := 0;
    FailureCode := 1;
  end;
 
  try
    case uMsg of
      ...
    end;
 
    Result := SuccessCode;
  except
    on E: Exception do
    begin
      MessageBox(hwndCPl, PChar(E.Message), 

        'Error', MB_OK or MB_ICONERROR);
      Result := FailureCode;
    end;
  end;
end;
 
exports
  CPlApplet;
 
end.

Next example would be a Shell extension. Shell extensions are implemented as COM objects. That means that you need to write and register a DLL, which follows COM rules. A COM rule for error handling is to use HRESULT as return value of any method. There are two ways to work with HRESULT. First one is quite direct: you write a function/method that returns HRESULT and you convert each exception to HRESULT value:

...
 
function ConvertExceptionToHRESULT(const E: Exception): HRESULT;
begin
  Result := E_FAIL; // <- this is just a simple example
  // See HandleSafeCallException function from ComObj unit

  // to see more complicated example
end;
 
type
  ICopyHook = interface(IUnknown)
  ['{000214FC-0000-0000-C000-000000000046}']
    function CopyCallback(Wnd: HWND; wFunc, wFlags: UINT; 

      pszSrcFile: PWideChar; dwSrcAttribs: DWORD; 

      pszDestFile: PWideChar; dwDestAttribs: DWORD): HRESULT; stdcall;
  end;
 
  TMyHook = class(TInterfacedObject, ICopyHook)
  protected
    function CopyCallback(Wnd: HWND; wFunc, wFlags: UINT;

      pszSrcFile: PWideChar; dwSrcAttribs: DWORD; 

      pszDestFile: PWideChar; dwDestAttribs: DWORD): HRESULT; stdcall;
  end;
 
function TMyHook.CopyCallback(Wnd: HWND; wFunc, wFlags: UINT; 

      pszSrcFile: PWideChar; dwSrcAttribs: DWORD; pszDestFile: PWideChar; 

      dwDestAttribs: DWORD): HRESULT; stdcall;
begin
  try
    // your code
 
    Result := S_OK;
  except
    on E: Exception do
      Result := ConvertExceptionToHRESULT(E);
  end; 
end;
 
function DllCanUnloadNow: HRESULT; stdcall;
begin
  try
    if { it's OK to unload DLL } then
      Result := S_OK
    else
      Result := S_FALSE;
  except
    on E: Exception do
      Result := ConvertExceptionToHRESULT(E);
  end; 
end;
 
...

The second way is to use Delphi wrapper for HRESULT. Delphi compiler provides assisting for HRESULT returning methods via safecall keyword. Any function like this: 

function Funcensten1(... some arguments ...): HRESULT; stdcall;
function Funcensten2(... some arguments ...; out AResult: TSomeType): HRESULT; stdcall;

has the same protype and the same calling convention as such function:

procedure Funcensten1(... some arguments ...); safecall;
function Funcensten2(... some arguments ...): TSomeType; safecall;

In other words, the above code fragments are binary compatible with each other. So, for example, DLL may use first code block and host may use second code block - and both will work correctly.

The difference between HRESULT/stdcall and safecall headers is assisting from Delphi compiler. Each safecall function and method automatically handles all exceptions within itself. Moreover, each call to safecall function/method automatically converts HRESULT return value back to exception.

So, the second way to work with HRESULT is:

...
 
type
  ICopyHook = interface(IUnknown)
  ['{000214FC-0000-0000-C000-000000000046}']
    procedure CopyCallback(Wnd: HWND; wFunc, wFlags: UINT; 

      pszSrcFile: PWideChar; dwSrcAttribs: DWORD; pszDestFile: PWideChar; 

      dwDestAttribs: DWORD); safecall;
  end;
 
  TMyHook = class(TInterfacedObject, ICopyHook)
  protected
    procedure CopyCallback(Wnd: HWND; wFunc, wFlags: UINT; 

      pszSrcFile: PWideChar; dwSrcAttribs: DWORD; pszDestFile: PWideChar; 

      dwDestAttribs: DWORD); safecall;
  end;
 
procedure TMyHook.CopyCallback(Wnd: HWND; wFunc, wFlags: UINT; 

  pszSrcFile: PWideChar; dwSrcAttribs: DWORD; pszDestFile: PWideChar; 

  dwDestAttribs: DWORD); safecall;
begin
  // your code
end;
 
function DllCanUnloadNow: HRESULT; stdcall;
// Unfortunately, it's not possible to customize return code 

// to be S_FALSE for simple function.
// Otherwise DllCanUnloadNow could have been written like this:
//  procedure DllCanUnloadNow; safecall; 
begin
  try
    if { it's OK to unload DLL } then
      Result := S_OK
    else
      Result := S_FALSE;
  except
    on E: Exception do
      Result := ConvertExceptionToHRESULT(E);
  end; 
end;
 
...

Converting exception to HRESULT value will be done automatically by Delphi's RTL code.

Notes:

Using framework(s) within your DLL

You may want to use frameworks inside your DLL. For example, you may want to display dialog with VCL within your DLL function. If this is the case - follow guidelines for frameworks.

See also: