Using Windbg to analyze the problem of high memory occupation

Posted by Dysan on Sun, 02 Jan 2022 21:00:46 +0100

Hello, I'm Feng Hui, a researcher of Microsoft MVP laboratory in this issue. This article mainly introduces how to use Windbg to analyze the memory problems in the application process, from managed heap to unmanaged heap and to memory allocation. Let's explore it together next.

Recently, several friends have used our Magicodes.IE Feedback: the memory soared during export. Next, let's see what caused it through Windbg.

Let's take a look at the current application memory usage through address -summary.

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    581     7df8`ef0c9000 ( 125.972 TB)           98.42%
<unknown>                              1678      206`ffb9e000 (   2.027 TB)  99.99%    1.58%
Image                                   950        0`064fd000 ( 100.988 MB)   0.00%    0.00%
Heap                                     58        0`050f6000 (  80.961 MB)   0.00%    0.00%
Stack                                   156        0`04380000 (  67.500 MB)   0.00%    0.00%
Other                                    11        0`019ad000 (  25.676 MB)   0.00%    0.00%
TEB                                      52        0`00068000 ( 416.000 kB)   0.00%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED                              282      200`038a6000 (   2.000 TB)  98.64%    1.56%
MEM_PRIVATE                            1674        7`07184000 (  28.111 GB)   1.35%    0.02%
MEM_IMAGE                               950        0`064fd000 ( 100.988 MB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                581     7df8`ef0c9000 ( 125.972 TB)           98.42%
MEM_RESERVE                             295      205`f8659000 (   2.023 TB)  99.79%    1.58%
MEM_COMMIT                             2611        1`188ce000 (   4.384 GB)   0.21%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1595        1`0dc6c000 (   4.215 GB)   0.20%    0.00%
PAGE_EXECUTE_READ                       156        0`04d66000 (  77.398 MB)   0.00%    0.00%
PAGE_READONLY                           600        0`03851000 (  56.316 MB)   0.00%    0.00%
PAGE_NOACCESS                            99        0`021f2000 (  33.945 MB)   0.00%    0.00%
PAGE_EXECUTE_READWRITE                   19        0`0027b000 (   2.480 MB)   0.00%    0.00%
PAGE_WRITECOPY                           90        0`001a0000 (   1.625 MB)   0.00%    0.00%
PAGE_READWRITE | PAGE_GUARD              52        0`0009e000 ( 632.000 kB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                    189`0413c000     7c6b`01ed4000 ( 124.418 TB)
<unknown>                              7dfb`2a153000      1f9`bd2ef000 (   1.976 TB)
Image                                  7ffc`883c1000        0`009ba000 (   9.727 MB)
Heap                                    183`0e9a1000        0`00f01000 (  15.004 MB)
Stack                                    37`62980000        0`0017b000 (   1.480 MB)
Other                                   183`77707000        0`01775000 (  23.457 MB)
TEB                                      37`62600000        0`00002000 (   8.000 kB)
PEB                                      37`627dd000        0`00001000 (   4.000 kB)

MEM_COMMIT occupies 4.384G. Next, we use eeheap -gc to check the managed heap.

0:000> !eeheap -gc
GC Allocated Heap Size:    Size: 0x11ac2568 (296494440) bytes.
GC Committed Heap Size:    Size: 0x120e7000 (302936064) bytes.

According to these memories, it seems that the problem is not here. A large amount of memory still appears in unmanaged memory. Let's take a look at the Windows NT heap. In fact, most of the user heap allocators in windows are in ntdll NT heap manager API in DLL (RtlAllocateHeap/RtlFreeHeap), such as malloc/free and new/delete in C, SysAllocString in COM framework and LocalAlloc, GlobalAlloc and heaprealloc in Win32. Although these allocators will create different heaps to store their memory, they will eventually call the NT heap in ntdll.dll.

0:000> !heap -s


************************************************************************************************************************
                                              NT HEAP STATS BELOW
************************************************************************************************************************
NtGlobalFlag enables following debugging aids for new heaps:
    stack back traces
LFH Key                   : 0x7cfd4cc2db4ddb4d
Termination on corruption : ENABLED
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                            (k)     (k)    (k)     (k) length      blocks cont. heap 
-------------------------------------------------------------------------------------
0000018378fd0000 08000002   65128  15296  64928   1720   177    17    2      c   LFH
    External fragmentation  11 % (177 free blocks)
00000183775c0000 08008000      64      4     64      2     1     1    0      0      
000001837aa90000 08001002    1280    108   1080     26     3     2    0      0   LFH
000001837ad20000 08001002      60      8     60      2     1     1    0      0      
000001837aca0000 08041002      60      8     60      5     1     1    0      0      
000001887bfd0000 08001002      60     20     60      1     2     1    0      0      
000001830cf30000 08001002    3324   1364   3124     19    10     3    0      0   LFH
000001830ce30000 08001002      60      8     60      5     1     1    0      0      
-------------------------------------------------------------------------------------

The output result is as shown above. The NT heap has a small content What's the reason? Well, according to maoni It seems that there is something wrong with the verification.

On Windows, all user mode allocations ultimately obtain memory through VirtualAlloc, bitmaps or GC heap.

The difference is whether you call VirtualAlloc directly or call it in other ways. If it is unmanaged memory, GC does not care about it, and of course, it does not know their existence.

GC does not govern these memories, so there is still a problem with the code we write. Let's go back and consider another thing, "when the export is in progress, the memory will increase greatly, and the memory will decrease after the export is completed". Let's take a look at the code, as shown below. In fact, what we now understand is that these memories must have been "held" and not released during our execution.

app.MapGet("/excel", async content =>
{
    string path = Path.Combine(Directory.GetCurrentDirectory(), "test.xlsx");
    List<TestDto> list = new();
    for (int i = 0; i < 400; i++)
    {
        list.Add(new TestDto
        {
            ImageUrl = "https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fup.enterdesk.com%2Fedpic_source%2F53%2F0a%2Fda%2F530adad966630fce548cd408237ff200.jpg&refer=http%3A%2F%2Fup.enterdesk.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1641193100&t=417a589da8c9ba3103ed74c33fbd6c70"
        });
    }
    Stopwatch stopwatch = Stopwatch.StartNew();
    ExcelExporter exporter = new ExcelExporter();
    await exporter.Export(path, list);
    stopwatch.Stop();
    await content.Response.WriteAsync(stopwatch.Elapsed.TotalSeconds.ToString());
});

According to the memory performance and our theory, we continue to use Windbg to check. Now we can find that these objects are finally recovered by GC. With the theory, we continue to conceive that GC knows which objects can be terminated, right? And they call their finalizers when they become unreachable, and these finalization objects will be recorded in the GC using the finalization queue. So can we check? As shown below, let's take a look.

0:000> !finalizequeue
----------------------------------
Statistics for all finalizable objects (including all objects ready for finalization):
              MT    Count    TotalSize Class Name
00007ffc2dc23818        1           24 System.Net.Security.SafeCredentialReference
00007ffc2dac4238        1           24 System.WeakReference
00007ffc2d6eb908        1           24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d6e4120        1           24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
00007ffc2d572b68        1           24 System.WeakReference`1[[Microsoft.Extensions.DependencyInjection.ServiceProvider, Microsoft.Extensions.DependencyInjection]]
00007ffc2d429258        1           24 System.WeakReference`1[[System.IO.FileSystemWatcher, System.IO.FileSystem.Watcher]]
00007ffc2dd15c20        1           32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle
00007ffc2d6de4d8        1           32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
00007ffc2d68fa00        1           32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle
00007ffc2d3a5cc0        1           32 System.Net.Quic.Implementations.MsQuic.Internal.SafeMsQuicRegistrationHandle
00007ffc2db390c8        1           40 Interop+WinHttp+SafeWinHttpHandle
00007ffc2d69a420        1           40 Internal.Cryptography.Pal.Native.SafeCertContextHandle
00007ffc2d5bea18        1           40 System.Diagnostics.EventLog
00007ffc2dc29a38        1           48 System.Net.Security.SafeFreeCredential_SECURITY
00007ffc2d963f80        2           48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
00007ffc2d7a3750        2           48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d685e10        1           56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Char, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d44c4d0        1           56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Byte, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d96be68        1           64 CellStore`1[[System.Uri, System.Private.Uri]]
00007ffc2d96b780        1           64 FlagCellStore
00007ffc2d96af48        1           64 CellStore`1[[System.Object, System.Private.CoreLib]]
00007ffc2d96a5b8        1           64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
00007ffc2d6ddab8        2           64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle
00007ffc2d69d528        2           64 Internal.Win32.SafeHandles.SafeRegistryHandle
00007ffc2d685bc8        2           64 Microsoft.Win32.SafeHandles.SafeWaitHandle
00007ffc2d685280        3           72 System.Threading.ThreadInt64PersistentCounter+ThreadLocalNodeFinalizationHelper
00007ffc2d5f5f50        3           72 System.Runtime.InteropServices.PosixSignalRegistration
00007ffc2d4299d0        1           72 Microsoft.Win32.SafeHandles.SafeFileHandle
00007ffc2d6e40b8        1           80 System.Runtime.Loader.DefaultAssemblyLoadContext
00007ffc2dac9ed0        2           96 PageIndex
00007ffc2d96d0c8        2           96 ColumnIndex
00007ffc2d464470        3          120 System.Gen2GcCallback
00007ffc2d40a620        1          120 System.IO.FileSystemWatcher
00007ffc2d96bc18        2          128 CellStore`1[[System.Int32, System.Private.CoreLib]]
00007ffc2dac20c8        2          144 System.Reflection.Emit.DynamicResolver
00007ffc2d680f10        3          144 System.Threading.LowLevelLock
00007ffc2d683c48        3          168 System.Threading.ThreadPoolWorkQueueThreadLocals
00007ffc2d681e80        1          176 System.Threading.LowLevelLifoSemaphore
00007ffc2dc25ef0        1          184 System.Collections.Concurrent.CDSCollectionETWBCLProvider
00007ffc2db8e658        1          184 System.Net.NetEventSource
00007ffc2db8c378        1          184 System.Net.NetEventSource
00007ffc2db38f90        1          184 System.Net.NetEventSource
00007ffc2d90c658        1          184 Microsoft.IO.RecyclableMemoryStreamManager+Events
00007ffc2d689b48        1          184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
00007ffc2d66f9f8        1          184 System.Diagnostics.Tracing.FrameworkEventSource
00007ffc2d66b720        1          184 System.Net.NetEventSource
00007ffc2d44d128        1          184 System.Buffers.ArrayPoolEventSource
00007ffc2d2e2ec8        1          184 System.Diagnostics.Tracing.NativeRuntimeEventSource
00007ffc2d694e10        1          192 System.Threading.Tasks.TplEventSource
00007ffc2d572ab0        1          192 Microsoft.Extensions.DependencyInjection.DependencyInjectionEventSource
00007ffc2d505f00        1          200 Microsoft.Extensions.Logging.EventSource.LoggingEventSource
00007ffc2db8ade8        1          224 System.Net.NameResolutionTelemetry
00007ffc2d428b08        7          224 System.Threading.PreAllocatedOverlapped
00007ffc2d563c78        1          232 System.Diagnostics.DiagnosticSourceEventSource
00007ffc2d61fe88        1          240 Microsoft.AspNetCore.Hosting.HostingEventSource
00007ffc2db6b788        8          256 System.Threading.TimerQueue+AppDomainTimerSafeHandle
00007ffc2d690270        1          280 System.Net.Sockets.SocketsTelemetry
00007ffc2db6bc80        1          296 System.Net.Http.HttpTelemetry
00007ffc2d68b998        1          336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
00007ffc2dc21998        1          360 System.Net.Security.NetSecurityTelemetry
00007ffc2d2dae28        1          384 System.Diagnostics.Tracing.RuntimeEventSource
00007ffc2d66ad60       10          480 System.Net.Sockets.SafeSocketHandle
00007ffc2d2e0240       21          504 System.WeakReference`1[[System.Diagnostics.Tracing.EventSource, System.Private.CoreLib]]
00007ffc2d2b0538        9          648 System.Threading.Thread
00007ffc2d77a188        2          704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
00007ffc2d90cec0        6          960 Microsoft.IO.RecyclableMemoryStream
00007ffc2d5fc658       10         1280 System.Net.Sockets.Socket
00007ffc2d68d898        4         1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
00007ffc2d2dc778       42         4704 System.Diagnostics.Tracing.EventSource+OverrideEventProvider
00007ffc2daec058      356        14240 System.Drawing.Bitmap
Total 553 objects

WOW!!!, Look at the 356 systems above Drawing. Bitmap is waiting for recycling. It seems that this is our influencing factor. Let's check the code.

try
{
    cell.Value = string.Empty;
    Bitmap bitmap;
    if (url.IsBase64StringValid())
    {
        bitmap = url.Base64StringToBitmap();
    }
    else
    {
        bitmap = Extension.GetBitmapByUrl(url);
    }

    if (bitmap == null)
    {
        cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
    }
    else
    {
        ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap);
        AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
            colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
        CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
        pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
    }

}
catch (Exception)
{
    cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
}

Use the Bitmap object in the ExcelPicture object. For the online picture source, we will read and store it in the Bitmap, but we find that the object has not been released, so a large number of bitmaps have not been released. Let's deal with it through using.

using (ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap))
{
    AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
        colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
    CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
    pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
}

A new object with a finalizer must be added to the finalization queue. This behavior is also called "registering for finalization". Of course, I also suggest you choose to use the SOSEX extension. It provides similar content to finalization, which seems more intuitive, as shown below.

  • Download address: http://www.stevestechspot.com/default.aspx

:000> .load D:\sosex_64\sosex.dll
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !finq -stat
Generation 0:
       Count      Total Size   Type
---------------------------------------------------------
          54            2160   System.Drawing.Bitmap

54 objects, 2,160 bytes

Generation 1:
       Count      Total Size   Type
---------------------------------------------------------
           1             184   Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
           1             336   Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
           4            1536   System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
           1              32   Internal.Cryptography.Pal.Native.SafeCertStoreHandle
           1             280   System.Net.Sockets.SocketsTelemetry
           1             192   System.Threading.Tasks.TplEventSource
           1              40   Internal.Cryptography.Pal.Native.SafeCertContextHandle
           2              64   Internal.Win32.SafeHandles.SafeRegistryHandle
           2              64   Internal.Cryptography.Pal.Native.SafeChainEngineHandle
           1              32   Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
           1              80   System.Runtime.Loader.DefaultAssemblyLoadContext
           1              24   System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
           1              24   System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
           2             704   Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
           2              48   System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
           1             184   Microsoft.IO.RecyclableMemoryStreamManager+Events
           6             960   Microsoft.IO.RecyclableMemoryStream
           2              48   System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
           1              64   CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
           1              64   CellStore`1[[System.Object, System.Private.CoreLib]]
           1              64   FlagCellStore
           2             128   CellStore`1[[System.Int32, System.Private.CoreLib]]
           1              64   CellStore`1[[System.Uri, System.Private.Uri]]
           2              96   ColumnIndex
           2             144   System.Reflection.Emit.DynamicResolver
           1              24   System.WeakReference
           2              96   PageIndex
         302           12080   System.Drawing.Bitmap
           1             184   System.Net.NetEventSource
           1              40   Interop+WinHttp+SafeWinHttpHandle
           8             256   System.Threading.TimerQueue+AppDomainTimerSafeHandle
           1             296   System.Net.Http.HttpTelemetry
           1             224   System.Net.NameResolutionTelemetry
           1             184   System.Net.NetEventSource
           1             184   System.Net.NetEventSource
           1             360   System.Net.Security.NetSecurityTelemetry
           1              24   System.Net.Security.SafeCredentialReference
           1             184   System.Collections.Concurrent.CDSCollectionETWBCLProvider
           1              48   System.Net.Security.SafeFreeCredential_SECURITY
           1              32   Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle

499 objects, 30,736 bytes

Generation 2:
0 objects, 0 bytes

TOTAL: 553 objects, 32,896 bytes

Maybe everyone will have a question like me at the beginning. I saw your picture It's not that big, and it doesn't show size in Windbg. First, let's look at the quality of this picture. The pixel of the picture is 2560x1440 and the bit depth is 24. At present, we know this information. Let's calculate the size of the uncompressed picture.

2560x1440x24/8

For a picture of about 10M, the number of known pictures x10M=3G. In fact, for this problem, it does not belong to memory leakage.

summary

This article mainly introduces how to use Windbg to analyze the memory problems in the application process, explore from managed heap to unmanaged heap and memory allocation, and finally confirm the memory problems according to the memory performance and theory. Of course, for memory analysis, we don't have to love a tool. Of course, we can do it together with PerfView, which may have a better effect.

Microsoft MVP

Microsoft's most valuable expert is a global award awarded by Microsoft to third-party technology professionals. Over the past 28 years, technology community leaders around the world have won this award for sharing expertise and experience in their online and offline technology communities.

MVP is a strictly selected expert team. They represent the most skilled and intelligent people. They are experts who are enthusiastic and helpful to the community. MVP is committed to helping others through lectures, forum Q & A, creating websites, writing blogs, sharing videos, open source projects, organizing meetings, and helping users in the Microsoft technology community use Microsoft technology to the greatest extent.
For more details, please visit the official website:
https://mvp.microsoft.com/zh-cn

Topics: windbg