Uploaded image for project: 'Core ReactOS'
  1. Core ReactOS
  2. CORE-9210

Race condition between co_MsqSendMessage and co_MsqDispatchOneSentMessage causes event object use after free when running user32_apitest:GetIconInfo

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 0.4.0
    • Win32SS
    • None

    Description

      I finally found a way to consistently repro the event use-after-free situation I was seeing at the end of CORE-9173.
      Wanna have a look?

      To repro:

      I did a little debugging and here's the order of events I observed:

      • co_MsqSendMessage enqueues the event and goes into a wait state
      • co_MsqSendMessage wakes up (presumably) with STATUS_TIMEOUT
      • After waking up but before it has a chance to call UserEnterCo, co_MsqSendMessage gets unscheduled
      • co_MsqDispatchOneSentMessage gets called concurrently and removes the message from SentMessagesListHead
      • co_MsqDispatchOneSentMessage does a callout and thus leaves the user lock
      • co_MsqSendMessage gets scheduled again, does not find Message in the list so performs no action on it, (correctly) assuming that it will be freed by someone else
      • co_MsqSendMessage frees the CompletionEvent and returns (using ExFreePoolWithTag in the special pool version, it simply goes out of scope and the stack space gets reused in the unpatched version)
      • co_MsqDispatchOneSentMessage returns from its callout and tries to set Message->CompletionEvent, which was just freed

      Here's Windbg output from when the use after free occurs in co_MsqDispatchOneSentMessage:

      Running Wine Test, Module: user32, Test: GetIconInfo
      (..\..\win32ss\user\ntuser\winsta.c:937) err: WS : Granted Access 000F037F
      (..\..\win32ss\gdi\eng\pdevobj.c:447) PDEVOBJ_bSwitchMode, ppdev = E1474AF0, pSurface = E1412B88
      (..\..\win32ss\gdi\eng\pdevobj.c:500) leave, ppdev = E1474AF0, pSurface = E14FD950
      GetIconInfo.c:163: Icon default size: 32x32.
      (..\..\win32ss\user\winsrv\consrv\frontends\gui\conwnd.c:767) TODO: Destroy console caret
       
      *** Fatal System Error: 0x000000cc
                             (0xF3701FF0,0x00000000,0x804A7DDE,0x00000000)
       
      Driver at fault: 
      ***  NTOSKRNL.EXE - Address 804A7DDE base at 80400000, DateStamp 54e217a3
      .
      Break instruction exception - code 80000003 (first chance)
       
      A fatal system error has occurred.
      Debugger entered on first try; Bugcheck callbacks have not been invoked.
       
      A fatal system error has occurred.
       
      Connected to Windows Server 2003 3790 x86 compatible target at (Tue Feb 17 09:15:05.451 2015 (UTC + 1:00)), ptr64 FALSE
      Loading Kernel Symbols
      ...................................................
      Loading User Symbols
      ...................................
      *******************************************************************************
      *                                                                             *
      *                        Bugcheck Analysis                                    *
      *                                                                             *
      *******************************************************************************
       
      Use !analyze -v to get detailed debugging information.
       
      BugCheck CC, {f3701ff0, 0, 804a7dde, 0}
       
      Probably caused by : NTOSKRNL.EXE
       
      Followup: MachineOwner
      ---------
       
      nt!RtlpBreakWithStatusInstruction:
      8055ef68 cc              int     3
      kd> !analyze -v
      *******************************************************************************
      *                                                                             *
      *                        Bugcheck Analysis                                    *
      *                                                                             *
      *******************************************************************************
       
      PAGE_FAULT_IN_FREED_SPECIAL_POOL (cc)
      Memory was referenced after it was freed.
      This cannot be protected by try-except.
      When possible, the guilty driver's name (Unicode string) is printed on
      the bugcheck screen and saved in KiBugCheckDriver.
      Arguments:
      Arg1: f3701ff0, memory referenced
      Arg2: 00000000, value 0 = read operation, 1 = write operation
      Arg3: 804a7dde, if non-zero, the address which referenced memory.
      Arg4: 00000000, Mm internal code.
       
      Debugging Details:
      ------------------
       
       
      READ_ADDRESS:  f3701ff0 
       
      FAULTING_IP: 
      nt!KeSetEvent+1e [c:\ros\reactos-clean\reactos\ntoskrnl\ke\eventobj.c @ 166]
      804a7dde 0fb608          movzx   ecx,byte ptr [eax]
       
      MM_INTERNAL_CODE:  0
       
      IMAGE_NAME:  NTOSKRNL.EXE
       
      DEBUG_FLR_IMAGE_TIMESTAMP:  0
       
      MODULE_NAME: NTOSKRNL
       
      FAULTING_MODULE: 00000000 
       
      DEFAULT_BUCKET_ID:  DRIVER_FAULT
       
      BUGCHECK_STR:  0xCC
       
      PROCESS_NAME:  explorer.exe
       
      CURRENT_IRQL:  1
       
      TRAP_FRAME:  00000010 -- (.trap 0x10)
      Unable to read trap frame at 00000010
       
      LAST_CONTROL_TRANSFER:  from 804a3a46 to 8055ef68
       
      STACK_TEXT:  
      f20dd5b8 804a3a46 00000003 f20dd9a0 ffdff408 nt!RtlpBreakWithStatusInstruction
      f20dd5e8 804a4494 00000003 f20ddac0 f20dd9e4 nt!KiBugCheckDebugBreak+0x36 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 536]
      f20dd9a8 804a4c2e 00000050 f3701ff0 00000000 nt!KeBugCheckWithTf+0x5b4 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 1100]
      f20dd9c8 804ca4e2 00000050 f3701ff0 00000000 nt!KeBugCheckEx+0x1e [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 1429]
      f20ddac0 804f474e 00000000 f3701ff0 00000000 nt!MmArmAccessFault+0x822 [c:\ros\reactos-clean\reactos\ntoskrnl\mm\arm3\pagfault.c @ 1853]
      f20ddadc 80541a23 00000000 f3701ff0 00000000 nt!MmAccessFault+0xce [c:\ros\reactos-clean\reactos\ntoskrnl\mm\mmfault.c @ 243]
      f20ddb10 804036df f20ddba4 804a7dde 02580320 nt!KiTrap0EHandler+0x1c3 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\i386\traphdlr.c @ 1277]
      f20ddb10 804a7dde f20ddba4 804a7dde 02580320 nt!KiTrap0E+0x8f
      f20ddba4 f25831c5 f3701ff0 00000000 00000000 nt!KeSetEvent+0x1e [c:\ros\reactos-clean\reactos\ntoskrnl\ke\eventobj.c @ 166]
      f20ddbd8 f257729c f20ddcdc f20ddc5c cccccccc win32k!co_MsqDispatchOneSentMessage+0x385 [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\msgqueue.c @ 892]
      f20ddc38 f257770c f20ddcbc 00000000 00000000 win32k!co_IntPeekMessage+0x9c [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 828]
      f20ddc80 f257a0ae f20ddcbc 00000000 00000000 win32k!co_IntGetPeekMessage+0xfc [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 1052]
      f20ddcf4 805427d9 0012fc50 00000000 00000000 win32k!NtUserGetMessage+0x7e [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 2127]
      f20ddd14 805422ed f257a030 0012fc10 00000010 nt!KiSystemCallTrampoline+0x19 [c:\ros\reactos-clean\reactos\ntoskrnl\include\internal\i386\ke.h @ 725]
      f20ddd5c 80403e03 0012fc24 7c92fb8e badb0d00 nt!KiSystemServiceHandler+0x23d [c:\ros\reactos-clean\reactos\ntoskrnl\ke\i386\traphdlr.c @ 1717]
      f20ddd5c 7c92fb8e 0012fc24 7c92fb8e badb0d00 nt!KiFastCallEntry+0x8c
      0012fc04 77a9f8c7 77a8713a 0012fc50 00000000 ntdll!KiFastSystemCallRet
      0012fc08 77a8713a 0012fc50 00000000 00000000 user32!NtUserGetMessage+0xc
      0012fc24 00414944 0012fc50 00000000 00000000 user32!GetMessageW+0x3a [c:\ros\reactos-clean\reactos\win32ss\user\user32\windows\message.c @ 2072]
      0012fc74 0041462a 0013a710 0013a710 0012feb4 explorer!CTrayWindow::TrayMessageLoop+0x34 [c:\ros\reactos-clean\reactos\base\shell\explorer\traywnd.cpp @ 2909]
      0012fc84 00401a96 0013a734 0012ffb4 0012fef0 explorer!TrayMessageLoop+0x3a [c:\ros\reactos-clean\reactos\base\shell\explorer\traywnd.cpp @ 3154]
      0012feb4 00401850 00400000 00000001 0012fedc explorer!StartWithDesktop+0x156 [c:\ros\reactos-clean\reactos\base\shell\explorer\explorer.cpp @ 177]
      0012fec4 0041f55e 00400000 00000000 00133be6 explorer!wWinMain+0x90 [c:\ros\reactos-clean\reactos\base\shell\explorer\explorer.cpp @ 224]
      0012fedc 0041ecec 00000001 00138bc8 001333c8 explorer!wmain+0x1e [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crt0_w.c @ 26]
      0012ffb4 0041e9e6 000000ff 0012fff0 77d93909 explorer!__tmainCRTStartup+0x2ac [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crtexe.c @ 307]
      0012ffc0 77d93909 00000000 00000000 7ffd6000 explorer!wWinMainCRTStartup+0x26 [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crtexe.c @ 168]
      0012fff0 00000000 0041e9c0 00000000 ec0100ed kernel32!BaseProcessStartup+0x69 [c:\ros\reactos-clean\reactos\dll\win32\kernel32\client\proc.c @ 478]
       
       
      STACK_COMMAND:  kb
       
      FOLLOWUP_NAME:  MachineOwner
       
      FAILURE_BUCKET_ID:  0xCC_IMAGE_NTOSKRNL.EXE
       
      BUCKET_ID:  0xCC_IMAGE_NTOSKRNL.EXE
       
      Followup: MachineOwner
      ---------
       
      kd> kp
      ChildEBP RetAddr  
      f20dd5b8 804a3a46 nt!RtlpBreakWithStatusInstruction
      f20dd5e8 804a4494 nt!KiBugCheckDebugBreak(unsigned long StatusCode = 3)+0x36 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 536]
      f20dd9a8 804a4c2e nt!KeBugCheckWithTf(unsigned long BugCheckCode = 0x50, unsigned long BugCheckParameter1 = 0xf3701ff0, unsigned long BugCheckParameter2 = 0, unsigned long BugCheckParameter3 = 0xf20ddb18, unsigned long BugCheckParameter4 = 0, struct _KTRAP_FRAME * TrapFrame = 0xf20ddb18)+0x5b4 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 1100]
      f20dd9c8 804ca4e2 nt!KeBugCheckEx(unsigned long BugCheckCode = 0x50, unsigned long BugCheckParameter1 = 0xf3701ff0, unsigned long BugCheckParameter2 = 0, unsigned long BugCheckParameter3 = 0xf20ddb18, unsigned long BugCheckParameter4 = 0)+0x1e [c:\ros\reactos-clean\reactos\ntoskrnl\ke\bug.c @ 1429]
      f20ddac0 804f474e nt!MmArmAccessFault(unsigned char StoreInstruction = 0x00 '', void * Address = 0xf3701ff0, char Mode = 0n0 '', void * TrapInformation = 0xf20ddb18)+0x822 [c:\ros\reactos-clean\reactos\ntoskrnl\mm\arm3\pagfault.c @ 1853]
      f20ddadc 80541a23 nt!MmAccessFault(unsigned char StoreInstruction = 0x00 '', void * Address = 0xf3701ff0, char Mode = 0n0 '', void * TrapInformation = 0xf20ddb18)+0xce [c:\ros\reactos-clean\reactos\ntoskrnl\mm\mmfault.c @ 243]
      f20ddb10 804036df nt!KiTrap0EHandler(struct _KTRAP_FRAME * TrapFrame = 0xf20ddb18)+0x1c3 [c:\ros\reactos-clean\reactos\ntoskrnl\ke\i386\traphdlr.c @ 1277]
      f20ddb10 804a7dde nt!KiTrap0E+0x8f
      f20ddba4 f25831c5 nt!KeSetEvent(struct _KEVENT * Event = 0xf3701ff0, long Increment = 0, unsigned char Wait = 0x00 '')+0x1e [c:\ros\reactos-clean\reactos\ntoskrnl\ke\eventobj.c @ 166]
      f20ddbd8 f257729c win32k!co_MsqDispatchOneSentMessage(struct _THREADINFO * pti = 0xb49da120)+0x385 [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\msgqueue.c @ 892]
      f20ddc38 f257770c win32k!co_IntPeekMessage(struct tagMSG * Msg = 0xf20ddcbc, struct _WND * Window = 0x00000000, unsigned int MsgFilterMin = 0, unsigned int MsgFilterMax = 0, unsigned int RemoveMsg = 0x4ff0001, long * ExtraInfo = 0xf20ddc68, int bGMSG = 1)+0x9c [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 828]
      f20ddc80 f257a0ae win32k!co_IntGetPeekMessage(struct tagMSG * pMsg = 0xf20ddcbc, struct HWND__ * hWnd = 0x00000000, unsigned int MsgFilterMin = 0, unsigned int MsgFilterMax = 0, unsigned int RemoveMsg = 0x4ff0001, int bGMSG = 1)+0xfc [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 1052]
      f20ddcf4 805427d9 win32k!NtUserGetMessage(struct tagMSG * pMsg = 0x0012fc50, struct HWND__ * hWnd = 0x00000000, unsigned int MsgFilterMin = 0, unsigned int MsgFilterMax = 0)+0x7e [c:\ros\reactos-clean\reactos\win32ss\user\ntuser\message.c @ 2127]
      f20ddd14 805422ed nt!KiSystemCallTrampoline(void * Handler = 0xf257a030, void * Arguments = 0x0012fc10, unsigned long StackBytes = 0x10)+0x19 [c:\ros\reactos-clean\reactos\ntoskrnl\include\internal\i386\ke.h @ 725]
      f20ddd5c 80403e03 nt!KiSystemServiceHandler(struct _KTRAP_FRAME * TrapFrame = 0xf20ddd64, void * Arguments = 0x0012fc10)+0x23d [c:\ros\reactos-clean\reactos\ntoskrnl\ke\i386\traphdlr.c @ 1717]
      f20ddd5c 7c92fb8e nt!KiFastCallEntry+0x8c
      0012fc04 77a9f8c7 ntdll!KiFastSystemCallRet
      0012fc08 77a8713a user32!NtUserGetMessage+0xc
      0012fc24 00414944 user32!GetMessageW(struct tagMSG * lpMsg = 0x0012fc50, struct HWND__ * hWnd = 0x00000000, unsigned int wMsgFilterMin = 0, unsigned int wMsgFilterMax = 0)+0x3a [c:\ros\reactos-clean\reactos\win32ss\user\user32\windows\message.c @ 2072]
      0012fc74 0041462a explorer!CTrayWindow::TrayMessageLoop(void)+0x34 [c:\ros\reactos-clean\reactos\base\shell\explorer\traywnd.cpp @ 2909]
      0012fc84 00401a96 explorer!TrayMessageLoop(struct ITrayWindow * Tray = 0x0013a734)+0x3a [c:\ros\reactos-clean\reactos\base\shell\explorer\traywnd.cpp @ 3154]
      0012feb4 00401850 explorer!StartWithDesktop(struct HINSTANCE__ * hInstance = 0x00400000)+0x156 [c:\ros\reactos-clean\reactos\base\shell\explorer\explorer.cpp @ 177]
      0012fec4 0041f55e explorer!wWinMain(struct HINSTANCE__ * hInstance = 0x00400000, struct HINSTANCE__ * hPrevInstance = 0x00000000, wchar_t * lpCmdLine = 0x00133be6 "", int nCmdShow = 1)+0x90 [c:\ros\reactos-clean\reactos\base\shell\explorer\explorer.cpp @ 224]
      0012fedc 0041ecec explorer!wmain(int flags = 1, unsigned short ** cmdline = 0x00138bc8, unsigned short ** inst = 0x001333c8)+0x1e [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crt0_w.c @ 26]
      0012ffb4 0041e9e6 explorer!__tmainCRTStartup(void)+0x2ac [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crtexe.c @ 307]
      0012ffc0 77d93909 explorer!wWinMainCRTStartup(void)+0x26 [c:\ros\reactos-clean\reactos\lib\sdk\crt\startup\crtexe.c @ 168]
      0012fff0 00000000 kernel32!BaseProcessStartup(<function> * lpStartAddress = 0x0041e9c0)+0x69 [c:\ros\reactos-clean\reactos\dll\win32\kernel32\client\proc.c @ 478]
      kd> ?? Message
      struct _USER_SENT_MESSAGE * 0xf36e5fa8
         +0x000 ListEntry        : _LIST_ENTRY [ 0xb49da2a9 - 0xb49da2a9 ]
         +0x008 Msg              : tagMSG
         +0x024 QS_Flags         : 0x40
         +0x028 CompletionEvent  : 0xf3701ff0 _KEVENT
         +0x02c Result           : 0xf2041920  -> 1
         +0x030 lResult          : 0
         +0x034 ptiSender        : 0xb4951ce0 _THREADINFO
         +0x038 ptiReceiver      : 0xb49da120 _THREADINFO
         +0x03c CompletionCallback : (null) 
         +0x040 ptiCallBackSender : (null) 
         +0x044 CompletionCallbackContext : 0
         +0x048 DispatchingListEntry : _LIST_ENTRY [ 0xb4951d30 - 0xb4951d30 ]
         +0x050 HookMessage      : 0
         +0x054 HasPackedLParam  : 0
      kd> ?? Message->CompletionEvent
      struct _KEVENT * 0xf3701ff0
         +0x000 Header           : _DISPATCHER_HEADER
      kd> ?? Message->CompletionEvent->Header
      struct _DISPATCHER_HEADER
         +0x000 Type             : ??
         +0x001 TimerControlFlags : ??
         +0x001 Absolute         : ??
         +0x001 Coalescable      : ??
         +0x001 KeepShifting     : ??
         +0x001 EncodedTolerableDelay : ??
         +0x001 Abandoned        : ??
         +0x001 NpxIrql          : ??
         +0x001 Signalling       : ??
         +0x002 ThreadControlFlags : ??
         +0x002 CpuThrottled     : ??
         +0x002 CycleProfiling   : ??
         +0x002 CounterProfiling : ??
         +0x002 Reserved         : ??
         +0x002 Size             : ??
         +0x002 Hand             : ??
         +0x003 Inserted         : ??
         +0x003 DebugActive      : ??
         +0x003 ActiveDR7        : ??
         +0x003 Instrumented     : ??
         +0x003 Reserved2        : ??
         +0x003 UmsScheduled     : ??
         +0x003 UmsPrimary       : ??
         +0x003 DpcActive        : ??
         +0x000 Lock             : ??
         +0x004 SignalState      : ??
         +0x008 WaitListHead     : _LIST_ENTRY

      Attachments

        1. CrashTest.patch
          4 kB
          jimtabor
        2. msgqueue_1.patch
          5 kB
          jimtabor
        3. msgqueue_10.patch
          39 kB
          jimtabor
        4. msgqueue_11.patch
          40 kB
          jimtabor
        5. msgqueue_2.patch
          25 kB
          jimtabor
        6. msgqueue_5.patch
          30 kB
          jimtabor
        7. msgqueue_8.patch
          34 kB
          jimtabor
        8. msgqueue_9.patch
          35 kB
          jimtabor
        9. msgqueue.c.patch
          0.4 kB
          Timo Kreuzer
        10. msgqueue.patch
          4 kB
          jimtabor
        11. ntuser-avoid-msgqueue-race.patch
          3 kB
          ThFabba
        12. ntuser-avoid-msgqueue-race.patch
          3 kB
          ThFabba
        13. ntuser-msgqueue-race-patchbot.patch
          5 kB
          ThFabba
        14. sendmsg-alloc-event-from-special-pool.patch
          2 kB
          ThFabba
        15. test-recursive-interthread.patch
          7 kB
          ThFabba
        16. win32k-special-nolookaside.patch
          6 kB
          ThFabba

        Issue Links

          Activity

            People

              jimtabor jimtabor
              ThFabba ThFabba
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: