Here's another technique for my anti-analysis collection! It uses an Exception Handler and an induced exception to detect debuggers. A handler is registered using the kernel32.SetUnhandledExceptionFilter API and it is triggered by a crafted exception.

This technique is used by malware families like SmokeLoader and GuLoader.

The example code below sets the ”unhandled_exception_filter” handler using SetUnhandledExceptionFilter API and then invokes it with an INT 3 exception, typically used as software breakpoints by debuggers.

// If Debug is active this handler will not get executed and INT3 will be 
// captured by the debugger as a breakpoint.
LONG unhandled_exception_filter(PEXCEPTION_POINTERS pExceptionInfo) {
    PCONTEXT ctx = pExceptionInfo->ContextRecord;
    #if defined(__x86_64__)
        ctx->Rip += 3; // Skip \xCC\xEB\x??
    #else
        ctx->Eip += 3; // Skip \xCC\xEB\x??
    #endif
    return EXCEPTION_CONTINUE_EXECUTION;
}

// https://wiki.osdev.org/Inline_Assembly
bool __is_debugged() {
    __asm__ volatile goto
    (
        "INT 3\n\t"                        // CC
        "JMP %l[being_debugged]\n\t"      // EB ??  
        :                                 // No output operands
        :                                 // No input operands
        :                                 // No clobbered registers
        : being_debugged                  // Label operands
    );

    return false;

being_debugged:
    return true;
}

You can find the full code in this here.