Performance Issue resulted due to ContractResolver
Proc dump found high CPU is caused by lock in application code.
Specifically, we found there was a Critical Section owned by thread 190 which blocked almost a half of threads from processing further.
Detected possible blocking or leaked critical section at 0x000002b8`1126d060 owned by thread 190 in K00009G_w3wp_6952_20230808-060047.dmp
Impact of this lock
43.61% of threads blocked
(Threads 23 58 72 75 77 82 83 84 85 87 88 89 95 96 98 99 102 107 108 109 111 117 118 121 122 123 124 127 128 129 135 137 138 139 140 141 143 144 145 148 150 151 152 156 158 159 160 161 162 163 167 168 170 172 173 174 175 177 179 180 182 183 184 186 188 192 193 195 196 198 199 201 203 204 207 208 209 210 211 212 215 217 218 219 223 224 226 227 228 229 230 231 232 233 234 235 237 238 241 242 243 244 245 248 251 253 254 256 257 259 265 266 267 270 272 274 278 279 281 284 287 288 290 292 293 295 296 299 300 303 304 305 306 309 310 311 315 318 319 320)
Lock is further found caused by application code not following the best practice.
By comparing the call stack of thread 190 and sample threads blocked by thread 190, we can see they were all blocked on below methods
Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue(System.Object)+74
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContract ByRef, System.Object ByRef)+cb
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+10a
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(Newtonsoft.Json.JsonWriter, System.Collections.IEnumerable, Newtonsoft.Json.Serialization.JsonArrayContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+1fe
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+185
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+185
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+185
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+116
Newtonsoft.Json.JsonSerializer.SerializeInternal(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+433
Newtonsoft.Json.JsonConvert.SerializeObjectInternal(System.Object, System.Type, Newtonsoft.Json.JsonSerializer)+ed
Newtonsoft.Json.JsonConvert.SerializeObject(System.Object, Newtonsoft.Json.JsonSerializerSettings)+1d
abc.Services.Infrastructure.SessionService.Set[[System.__Canon, mscorlib]](abc.Entities.Infrastructure.Enums.SessionKeys, System.__Canon, System.String)+145
clr!InstantiatingMethodStubWorker+bc [f:\dd\ndp\clr\src\vm\amd64\InstantiatingStub.asm @ 132] f:\dd\ndp\clr\src\vm\amd64\InstantiatingStub.asm @ 132
abc.AppServices.IdentityManagementService.CreateUserContextForCurrentSession(System.String, abc.Entities.Api.Models.Membership, System.Guid)+143
abc.AppServices.IdentityManagementService.CreateUserSession(System.String)+26e
abc.MVC.Areas.abc.Controllers.SomeController.SOmeManagement(System.String, System.Nullable`1<System.Guid>)+54d
By decompiling application code, we can see in the implementation of abc.Services.Infrastructure.SessionService.Set method, application was creating new ContractResolver instance for every request.
public void Set<T>(SessionKeys key, T value, string additionalKey = null)
{
string text = this.ConstructSessionKey(key, additionalKey);
bool flag = bool.Parse(ConfigurationManager.AppSettings[AppSettingKeys.Common.EncryptSessionData]);
if (flag)
{
JsonSerializerSettings settings = new JsonSerializerSettings
{
TypeNameHandling = TypeNameHandling.All,
ContractResolver = new PrivateSetterContractResolver()
};
HttpContext.Current.Session[text] = this._cryptoService.EncryptWithCbc(JsonConvert.SerializeObject(value, settings), this._encryptionKey);
HttpContext.Current.Items[text] = value;
}
else
{
HttpContext.Current.Session[text] = value;
}
}
Above implementation is against best practice documented at Performance Tips (newtonsoft.com) where we should avoid recreating contracts every time you use JsonSerializer but should create the contract resolver once and reuse it. In this case, not reusing contract resolver instance triggered JIT to compile code upon every request. As JIT can’t work in parallel, one request has to wait for previous request to finish before it could get processed. In other words, JIT is locked down by one thread which also blocked other threads from moving forward.
Solution
Sample code:
// GOOD - reuse the contract resolver from a shared location
string json2 = JsonConvert.SerializeObject(person, new JsonSerializerSettings
{
Formatting = Formatting.Indented,
ContractResolver = AppSettings.SnakeCaseContractResolver
});
No comments :
Post a Comment