ASP.NET, AppDomains, and shadow-copying

Apr 29, 2012 in BLOG
advanced asp.net c#
9 min read

I answered a question on StackOverflow nearly two years ago, and I’m surprised at how few votes it has received, despite comments such as:

+1 for teaching me something new today thanks. -kobe

Being one of my favorite answers, I thought I should discuss it on my blog a little more in-depth than just posting the SO answer. I’d like to briefly discuss what ASP.NET really is (in the context of IIS), why AppDomains are needed, and lastly what shadow-copying does for an application. There is no code associated with this post, and it is driven more by contemplation than by a specific resolution to a problem. So, I apologize in advance if it seems disjointed at times.

What is an ASP.NET application?

When I first started using the .NET Framework, I was somewhat confused about how applications actually ran considering an assembly is a PE format with some CLR headers and CLR data. What this means (and it makes a lot of sense) is that the CLR is really just a COM application hosting a runtime environment (the CLR).

I usually consider programming languages like C# to be compiled into a binary, but the CLR really only operates on IL. This means C# et al. are simply abstractions to execute “just-in time”-compiled code in a compartmentalized sandbox within an instance of the CLR. The compartmentalization is further secured using AppDomains to prevent (or attempt to prevent) applications from destroying data in other AppDomains. It makes sense, but I’ve never really seen it put so simply before. Even though I knew this to be the case and understood how it all worked, my thoughts of the CLR were also compartmentalized. When I stepped back and looked at it, I thought to myself, “Wait, what? That’s it?” Why I thought the CLR was some sort of voodoo is beyond me.

An ASP.NET application is .NET code written against a framework (ASP.NET), executing in a runtime (CLR), which is in turn hosted in a (COM) process (IIS). That’s all there is to it, there’s no magic. It’s been a few years since I learned this, but I seem to still encounter developers with the same years of experience or more who don’t see how these pieces are integrated. It really only takes a little exploration on your system to get a better picture of the whole environment.

IIS

If you take a look at an IIS assembly in FileAlyzer, you might see CorBindToRuntimeEx or CLRCreateInstance listed as PE Imports. This is because IIS manages the creation of multiple CLR instances and multiple AppDomains per instance. ASP.NET is implemented as an ISAPI extension within IIS.

In fact, %WinDir%\Microsoft.NET\Framework[Version]\aspnet_regiis.exe imports RegisterAspNetEx from webengine.dll. Running

dumpbin /imports webengine.dll

will output all header information, in which you will see mscoree.dll is delay-loaded and provides the CLR creation methods (from .NET 2.0):

mscoree.dll
              00000001 Characteristics
              6A301BB8 Address of HMODULE
              6A2FE1C4 Import Address Table
              6A2FB5D4 Import Name Table
              00000000 Bound Import Name Table
              00000000 Unload Import Name Table
                     0 time date stamp

          6A2FAC25               0 CorBindToRuntimeEx
          6A2B292E               0 CorBindToRuntimeHost
          6A2B21C6               0 ClrCreateManagedInstance

Here we’ve found CorBindToRuntimeEx! This library is part of the COM application responsible for running the CLR instances that host ASP.NET AppDomains inside IIS. Cool, huh? That’s the magic.

IIS is the hosting environment responsible for running the ASP.NET ISAPI filter. Once ASP.NET is registered with IIS, webengine.dll handles the creation of CLR instances within Application Pools. Inside those Application Pools, one or more AppDomains may exist.

There is a lot that has to go on under the hood in an ASP.NET application, but I’d like to discuss the management of application resources– specifically shadow copying.

Shadow Copying

Shadow copying is a technique used by ASP.NET to allow resources to remain available continuously without interfering with the execution of code in an AppDomain. Conceptually, it is a very simple idea. Unfortunately, most developers don’t even know it is happening under the covers and I’ve seen production site get bitten by it in some weird configuration and file modification scenarios.

When shadow copying is enabled, ASP.NET copies certain resources (all assemblies) from the application directory into a temporary folder and those are the resources which remain in use during the life of an AppDomain. This cache is the same one used for downloaded assemblies and is cleaned by the CLR when the assemblies are no longer in use (akin to Garbage Collection of runtime resources). By default, the location for these cached assemblies will be something like

C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\demo1\9b9144a7\8665ac07

As you can see, the directories are hashed in what (I assume) is similar to the hashing algorithm used for the GAC. If you like, you can set the AppDomainSetup.CachePath property to shadow copy assemblies to a different location. You must also set the AppDomainSetup.ApplicationName property, otherwise CachePath will be ignored. One downside to this is that you’d have to create a new AppDomain to initialize these values and the management of assembly resolution may become cumbersome. You may also attempt setting the cache path as described here:

protected const string ApplicationAssembliesFolder = "~/Assemblies";

    protected void Application_Start(object sender, EventArgs e)
    {
        var assembliesPath = Server.MapPath(ApplicationAssembliesFolder);

        AppDomain.CurrentDomain.SetShadowCopyPath(
            AppDomain.CurrentDomain.SetupInformation.ShadowCopyDirectories + 
            Path.PathSeparator + assembliesPath);

        Assembly.LoadFrom(
            Path.Combine(assembliesPath, "Example.dll"));
    }

When changing the default location of shadow copied assemblies, be aware that ASP.NET no longer handles the clean up of the assemblies– it is up to the application to maintain these assemblies.

The shadow copying process

The shadow copying process is kicked off any time an AppDomain recycle occurs. Tess Ferrandez (who has a killer blog, you should check it out) lists some reasons why the AppDomain may be recycled in her post, ASP.NET Case Study: Lost session variables and appdomain recycles:

Machine.Config, Web.Config or Global.asax are modified
The bin directory or its contents is modified
The number of re-compilations (aspx, ascx or asax) exceeds the limit specified by the <compilation

numRecompilesBeforeAppRestart=/> setting in machine.config or web.config (by default this is set to 15)
The physical path of the virtual directory is modified
The CAS policy is modified
The web service is restarted
(2.0 only) Application Sub-Directories are deleted (see Todd’s blog http://blogs.msdn.com/toddca/archive/2006/07/17/668412.aspx for more info)

My rule of thumb is to assume that anytime something happens which could change the execution of an application, the AppDomain will be recycled.

When an AppDomain is recycled, resources are shadow copied to a new location and any connections to the old AppDomain are ‘drained’, meaning they are allowed to finish on the old AppDomain but subsequent requests are handled by the new AppDomain. This allows the framework to create a fairly seamless experience, but is also part of the reason why many developers don’t know about shadow copying.

How shadow copied files are updated

When using .NET 4, shadow copying assemblies in an application for which assemblies rarely ever change has improved. In previous versions of ASP.NET, there was often a noticeable delay in application startup time while assemblies were being shadow copied. Now, the framework checks the file date/time of an application’s assemblies and compares that with the file date/time of any shadow copied assemblies. If they are the same, the shadow copying process does not occur. This causes the shadow copying process to kick off only if an assembly has been physically modified.

Now, if these things are happening a lot and you either have many assemblies or your assemblies change often, .NET 4 is actually adding one extra step to the shadow copying process (checking file times). You can disable this extra step by adding the following configuration to your web.config:

<configuration>
   <runtime>
      <shadowCopyVerifyByTimestamp enabled="false" />
   </runtime>
</configuration>

The process itself includes the application location, a temporary location, and the shadow cache location. The process would look something like this for each assembly:

Copy assembly from application location to temporary location
Open assembly
Verify assembly name
Validate strong name
Compare update to current cached assembly
Copy to shadow copy location (if newer)
Remove assembly from temporary location

That is a pretty hefty process if your application contains a lot of assemblies. You can surely see why the file time comparison introduced in .NET 4 is beneficial. This is also why applications with many, many resources take longer to “start up”. The framework isn’t executing the code directly from the application directory.

Disabling shadow copying

Shadow copying is important if you are modifying assemblies directly in a live application. If you are working with a load balancer in which you’re able to drain connections from a server and stop IIS for deployment, you may benefit from disabling shadow copying altogether.

<hostingEnvironment shadowCopyBinAssemblies="false" />

Alternative configuration: Wait-Change Notification

One modification I would suggest if you are swapping out assemblies regularly is to set the maxWaitChangeNotification and waitChangeNotification attributes on the httpRuntime element. Both of these attributes take an Int32 value that, when combined, determine when the AppDomain recycling occurs because of a file change.

The waitChangeNotification attribute will modify how long the framework waits between file changes. In other words, if File A finishes copying and waitChangeNotification is set to 2 seconds, the framework will wait 2 seconds for another file modification to occur before spawning a new AppDomain. If File B is modified within that time, it will again wait 2 seconds for another file to be modified.

If maxWaitChangeNotification is also set, the framework no longer chains the wait time. There is an absolute time from when the first file modification starts. For instance, if maxWaitChangeNotification is set to 5 seconds, it doesn’t matter if a file is in the process of being modified, a new AppDomain will be spawned based on the File A modification (operation 1). The File B modification will begin the wait even of operation 2, after which another AppDomain will be spawned. I’ve created the image below as an attempt to clarify what these settings do. These file copy operations represent changes to the application directory. The “AppDomain restart” indicators are the points at which shadow copying begins. Files that are being written are locked and not copied to the temporary location.