Thursday, July 20, 2017

IIS Log Parser and important troubleshooting queiries

Understand your IIS Log better

There are cases when even asp.net and sitecore application level logs doesn't provide much of an information.The reason behind is some request is just processed by IIS webserver but never executed at application server i.e sitecore or asp.net pipeline.

This is where HTTP.Sys , Failed Request Tracing and IIS logs comes into picture. This is not when you looking for memory leak or CPU utilization.But this is certain some of the pattern of page execution or 500 internal server exception indirectly contributes to memory leak or CPU utilization. Always visit your IIS logs post performance load test. This will help you align your performance results with incoming request made in IIS.

Long story short. Download log parser studio if you have IIS log to analyze. This is very handy tool.

https://gallery.technet.microsoft.com/office/Log-Parser-Studio-cd458765

Query#1 Http.5xxx


SELECT TO_LOCALTIME( TO_TIMESTAMP( date, time ) ),
    cs-username,
    sc-status,
    sc-substatus,
    cs-uri-stem,
    time-taken as time_taken_MS,
    DIV(time-taken,60000) as timetaken_MM,
    cs-method, 
    cs-uri-query ,
    Count(*) AS Total
    From '[LOGFILEPATH]'
    Where sc-status = 500
    Group By cs-username, sc-status, cs-uri-stem, cs-uri-query,date,time,cs-method, cs-uri-query ,time-taken,sc-substatus



Query#2:/*  Count and sort all HTTP status codes */
Remember http status code below have apended http SUB STATUS CODE

SELECT TOP 25  
    STRCAT(TO_STRING(sc-status),  
    STRCAT('.', TO_STRING(sc-substatus))) As Status,  
    COUNT(*) AS Hits  
    FROM '[LOGFILEPATH]'  
GROUP BY Status  
ORDER BY Hits DESC 

Query#3 To find DDOS Attack

A distributed denial-of-service (DDoS)attack occurs when multiple systems flood the bandwidth or resources of a targeted system, usually one or more web servers. Such an attack is often the result of multiple compromised systems (for example, a botnet) flooding the targeted system with traffic.

/* Hits on a particular page by IP address */

select c-ip, count(c-ip) as requestcount 
from '[LOGFILEPATH]' 
where cs-uri-stem like '%/%' 
group by c-ip order by count(c-ip) desc



Monday, July 10, 2017

Sitecore Pipeline HttpBeginRequest and Processor HttpRequestProcessor to Intercept requests before page get processed completely.

Problem Statement:
To intercept each request and check site configuration and take required steps before it is served and processed complete. This can be enabling intermediate maintenance page or site down page in case unfortunate events occurred such as third party service is down . This in particular is the case your content site is more dependent on upstream or downstream application to serve user specific data. Even though you aimed for zero downtime as per your architecture.

Solution:
Approach 1: To check this flag at every action call using action filter in mvc sitecore controller.
Approach 2: To intercept the call before it call action filter.

Reference: Pipeline and its Processor

Implementation:


Add processor just after ItemResolver to intercept the request to check for sitecore stored flag and take action based on values that something suits your requirement

Pipeline: httpBeginRequest: Sitecore.Pipelines.HttpRequest





Key Notes:
  • Ensure you have proper patch insert for CD and delete for CM. As per the scenario.
  • Ensure you have AbortPipeline in place to prevent -Cascading response header issue- Such as append header and results in Too many redirects.
  • Use custom processor wisely and regression test to cover different use case across the page navigation involves logout /login [For transactional Site or Personalised site]

Thursday, July 6, 2017

Maintenance Page Take App Offline! Magic with App_Offline.htm


Overview


The App_offline thing is around for quiet long time and I haven't seen enterprise level system use it. Either they do it using at load balancer end or redirect to some third party hamster page until the site is provision for up and running. Leaving the real time usage and know-how.

Headstart




Here are quick help tip if you want to enable it.


Key Note
Remember just putting App_offline.htm in the root folder of the website will just stop the further processing of request and always return =

Http Status 503:

The service is unavailable.


If you really want to show up App_Offline.htm custom page you must add this in your web.config file


Refer this for more information
https://stackoverflow.com/questions/17135109/app-offline-site-returning-the-service-is-unavailable


One can even try to automate this using FTP command and powershell or either use out of the box option in VSO Build and Release pipeline.


Take App Offline option under Deployment


Tuesday, July 4, 2017

Gotcha ! Special Character & in web.config


What is special about &?


Whenever you're dealing with ampersand or querystring url in rewrite rules beware of special character "&". There are chances you may play around with different ways to make it working. Inline rewritemap maps things bit difficult. Yes it might happen you have few rewrite and you end up with inline rewrite in web.config. Careful watch rewritemap urls as these may content special character as part of URI and redirection symantics. Sometimes # results in error and exception.


Always try this!



  • Replace & with &Make use of html encode and decode instead of URL Encode value such as "%26"
  • Good practice to keep rewritemap outside web.config . In many scenario the URL encode for ampersand as well # in querystring works.


References:







Monday, July 3, 2017

Sitecore CMS Logout not working due to conflicting Authentication Controller



Introduction

Sitecore CMS Logout will throw Javascript error due to multiple controller named 'Authentication'


Sitecore Logs
10344 02:57:52 ERROR Application error.
Exception: System.InvalidOperationException
Message: Multiple types were found that match the controller named 'Authentication'. This can happen if the route that services this request ('sitecore/shell/api/sitecore/{controller}/{action}') does not specify namespaces to search for a controller that matches the request. If this is the case, register this route by calling an overload of the 'MapRoute' method that takes a 'namespaces' parameter.

The request for 'Authentication' has found the following matching controllers:
Sitecore.Controllers.AuthenticationController
MyWebsite.MyArea.Controllers.Api.AuthenticationController
Source: System.Web.Mvc
   at System.Web.Mvc.DefaultControllerFactory.GetControllerTypeWithinNamespaces(RouteBase route, String controllerName, HashSet`1 namespaces)
   at System.Web.Mvc.DefaultControllerFactory.GetControllerType(RequestContext requestContext, String controllerName)
   at System.Web.Mvc.DefaultControllerFactory.System.Web.Mvc.IControllerFactory.GetControllerSessionBehavior(RequestContext requestContext, String controllerName)
   at System.Web.Mvc.MvcRouteHandler.GetHttpHandler(RequestContext requestContext)
   at Sitecore.Mvc.Routing.RouteHandlerWrapper.GetHttpHandler(RequestContext requestContext)
   at System.Web.Routing.UrlRoutingModule.PostResolveRequestCache(HttpContextBase context)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Resolution


  1. Either rename your custom controller from Authentication to something Account to avoid sitecore conflict.
  2. Else apply below Maproute.

public override void RegisterArea(AreaRegistrationContext context) 
       {           
           context.MapRoute(
            "SCAuth",
              "sitecore/shell/api/sitecore/{controller}/{action}",
              new { controller = "Authentication", action = "Index" },
              new[] { "Sitecore.Controllers" });
 
 
       }

Reference:
https://sitecore.stackexchange.com/questions/201/how-to-override-replace-the-default-sitecore-authentication-controller

Sitecore .Net File Change Notification Shut down issues

 

Problem Statement

Not sure what is causing file change in webapps whether it is appinsight or for that matter logging handler that keep adding debug, info and error traces, cache or something else. For some reason there is something going under the hood which causes this file change notification. This take me back to the basics of file change notification causes app pool recycle when there is change in web.config or global.asax. Now as best practice it is always mentioned the logging has to be done outside your webroot and for some reason this is not the case with webapp. I'm naïve in webapp world and file and drive structure in azure webapp paas environment. Those days are gone when you use to have full control of VMs and you can deep dive into solving and identifying issues.


KB: Sitecore stability issues when using Azure Web Apps

It talks about dynamic cache- WEBSITE_DYNAMIC_CACHE

FCN Cause and Effect
Now this file change notification itself is a problem which as ripple effect on few things. This is how it works.
Chain and sequences of events due to FCN- Filewatcher/File change notification
  1. File change notification, filewatcher notification triggers
  2. Sitecore shut down
  3. Solr or lucene Indexing lookup failed. Empty Index found.
  4. Session State, if sql , Redis or Mongo used: Failure of session state, results in session leaks and session_end events, and all other subsequent calls.
  5. XDb: Analytics current tracking failed or current tracking is null
  6. Eventually spike in CPU utilization and memory leaks.

How it goes?

FCN Cause -> Affected--> Sitecore shut down-> Session --> Indexing--> Analytics-> CPU utilization--> Memory Leaks

It is very important to understand how we approach to the problem and how we look up error logs. It is not that straight forward instead need problem occurrence chart[Fish bone technique to lay down the cause and effect] to note down each events. In fact I see this as solving mystery or identify suspects. Like above chain of sequence. As rule of thumb memory leaks and CPU utilization always leads you to bad code or DI leaks etc. I insist more lateral thinking than primitive way of identifying problem.

.Net Way Short term Workaround: Brute force Fixes



As a work-around I have added fcnMode = "Disabled" to the httpRuntime element in the web.config. The root cause remains unknown.


httpruntime fcnmode="Disabled"/

Wednesday, June 21, 2017

KeepAlive Url agent SSL Forced authentication issues and quick fix!

Table of Contents

  1. Introduction
  2. Problem Statement: Failed SSL forced Authenticatio
  3. Bit of a background
  4. Quick Fix: Sitecore SSL Keepalive UrlAgent Issue
  5. Miscellaneous Custom UrlAgent for KeepAlive

Introduction


The purpose of this article is bring forward the key issues that one can come across when dealing with keepalive url. I'll not delve deep into what is Keepalive rather I'll focus on problem statement and list of references that can handy. Also I have given my way of approaching to the solution in different scenario. Maybe the solution I have provided may not stand the best recommended practice however it atleast give some foresight to deal with issues like this. The content more or less is about using https :// keepalive url when working with .net 4.5 azure webapp or environment required forced ssl.

Problem Statement: Failed SSL forced Authentication



Bit of a background


The SSL forced authentication will only arise if you have forced https for your website and http is restricted through firewall. Now if you are using .net 6.0 or higher this will work without any issue whereas .net 4.5 and lower version will not enforce TLS protocol by default.You may have to set them explicitly to map to your ssl certificate.
  • .NET 4.6 and above. You don’t need to do any additional work to support TLS 1.2, it’s supported by default.
  • .NET 4.5. TLS 1.2 is supported, but it’s not a default protocol. You need to opt-in to use it.
Refer this http://blogs.perficient.com/microsoft/2016/04/tsl-1-2-and-net-support/

***Poodle Attack


As per my understanding reason to do this to prevent poodle attack. Refer this for more details Security bulletin TechNet Poodle Attack

***SSL Scan


You can scan your site for poodle attack. Scan your site for poodle attack SSL Scan Test

Quick Fix: Sitecore SSL Keepalive UrlAgent Issue


Add this  Global.asax startup Or application Start method for .net 4.5 web app. This explains lot https://stackoverflow.com/questions/28286086/default-securityprotocol-in-net-4-5
 System.Net.ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; 
Add this to your web config Ip restriction to allow access to O.0.0.0 azure webapps




http://www.secqa.com/index.php?qa=2247&qa_1=sitecore-azure-website-forbidden-its-access-permissions-127
https://stackoverflow.com/questions/34389510/sitecore-8-1-azure-website-forbidden-by-its-access-permissions-127-0-0-1/34869927
https://stackoverflow.com/questions/24884098/azure-website-ip-restriction

The above solution will solve your forced ssl authentication issue.

Sitecore support KB must eloborate and refine this one to reduce number of support request. https://kb.sitecore.net/articles/376036 [Check out 4th Scenario Covered]

Someone even face this for mongo integration. http://techqa.info/programming/question/39699800/Intermittent--Authentication-failed-because-the-remote-party-has-closed-the-transport-stream-----Mongo-C-

Miscellaneous Custom UrlAgent for KeepAlive


If you are facing some other issues you will always have an options to create custom URLagent like this one below. I tried to create one with adding above solution. You can even create generic version to get azure host using custom solution making use of sitecore webutil. For e.g calling out of box webtil methods. for e.g GetHostIPAddress
Making use of above method we can create custom urlagent and add it to our solution.For testing purpose we reduce the interval to 2 minutes. Sitecore.config

using Sitecore.Diagnostics;
using Sitecore.Web;
using System;
using System.Net;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using System.Text;

namespace MyProject
{
    public class SslUrlAgent
    {

        private bool _logActivity = true;
        private readonly string _url;

        public bool LogActivity
        {
            get
            {
                return this._logActivity;
            }
            set
            {
                this._logActivity = value;
            }
        }


        public SslUrlAgent(string url)
        {
            Assert.ArgumentNotNull((object)url, "url");
            this._url = url;
        }

    
        public void Run()
        {
            try
            {
                string fullUrl = WebUtil.GetFullUrl(this._url);
                this.LogInfo("Scheduling.UrlAgent started. Url: " + fullUrl);
                this.LogInfo("Scheduling.UrlAgent done (received: " + (object)ExecuteSSLWithCertificatePolicyWebPage(fullUrl).Length + " bytes)");
            }
            catch (Exception ex)
            {
                Log.Error("Exception in UrlAgent (url: " + this._url + ")", ex, (object)this);
            }
        }
     
        private bool ValidateRemoteCertificate(object sender, X509Certificate cert, X509Chain chain, SslPolicyErrors error)
        {
            // If the certificate is a valid, signed certificate, return true.
            if (error == System.Net.Security.SslPolicyErrors.None)
            {
                return true;
            }

            LogInfo("ICertificatePolicy: " + error.ToString());

            return false;
        }
        private string ExecuteSSLWithCertificatePolicyWebPage(string url)
        {
            System.Net.ServicePointManager.ServerCertificateValidationCallback += (send, certificate, chain, sslPolicyErrors) => { return true; };
            using (WebClient webClient = new WebClient())
            {
                // System.Net.ServicePointManager.ServerCertificateValidationCallback += (send, certificate, chain, sslPolicyErrors) => { return true; };
                // ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
                System.Net.ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;                          
                return Encoding.UTF8.GetString(webClient.DownloadData(url));
            }         
        }
      
        private void LogInfo(string message)
        {
            if (!this.LogActivity)
                return;
            Log.Info(message, (object)this);
        }
    }
}

 
Some useful webutil method for reference. I used dotPeek to get the implementation of these method.
    public static string GetFullUrl(string url)
    {
      Assert.ArgumentNotNull((object) url, "url");
      return WebUtil.GetFullUrl(url, WebUtil.GetServerUrl());
    }


 
    public static string GetFullUrl(string url, string serverUrl)
    {
      Assert.ArgumentNotNull((object) url, "url");
      Assert.ArgumentNotNull((object) serverUrl, "serverUrl");
      if (url.IndexOf("://", StringComparison.InvariantCulture) < 0)
        return FileUtil.MakePath(serverUrl, url, '/');
      return url;
    }


 
    public static string GetHostIPAddress()
    {
      HttpContext current = HttpContext.Current;
      try
      {
        if (current != null)
        {
          if (current.Request != null)
          {
            Uri url = current.Request.Url;
            if (url.HostNameType == UriHostNameType.Dns)
            {
              IPHostEntry hostEntry = Dns.GetHostEntry(url.Host);
              if (hostEntry.AddressList.Length >= 0)
                return hostEntry.AddressList[0].ToString();
            }
          }
        }
      }
      catch (SocketException ex)
      {
        Log.SingleWarn(string.Format("Cannot resolve the IP address of the web server with the specified hostname: {0}.", (object) current.Request.Url.Host), (object) typeof (WebUtil));
      }
      return string.Empty;
    }

 
 
    public static string GetHostName()
    {
      HttpContext current = HttpContext.Current;
      if (current != null && current.Request != null)
        return current.Request.Url.Host;
      return string.Empty;
    }

 
 
    public static string GetIISName()
    {
      HttpServerUtility server = WebUtil.GetServer();
      if (server != null)
        return SecurityUtil.ComputeHash(server.MapPath("/"));
      return string.Empty;
    }




With above implementation if you have any webproxy issue or anything you can try to resolve using custom urlagent approach as given above. Hope this article help you to some extend. Thanks.



Monday, May 1, 2017

Domain Name Service Cheat Sheet

Domain Name Service


First look for domain name to IP mapping is done through Host file and if not exist it look up in server domain cache. If the mapping is not available in host and aswell as in DNS cache it will eventually look up into DNS server.
  1. Host- This is host file C:\Windows\System32\drivers\etc\hosts
  2. DNS Cache
  3. DNS Server



Sr.No
DNS Attribute
Remark
1
Ipconfig/flushdns
Flush DNS Cache
2
Ipconfig/displaydns
Display DNS Details
3
Forward DNS Lookup
Internet domain name to find an IP address
4
Reverse DNS Lookup
Internet IP address to find a domain name.
5
Root Hint

6
Authorative DNS Response
Contains Address reference in Enterprise DNS server
7
Non Authorative DNS Response
Available either in non enterprise root dns server
8
Cache DNS Path
C:\Windows\System32\dns\cache.DNS
9
Wireshark
To intercept all network protocal
10
Addr ARPA-.in-addr.arpa (inverse address)
Translate IP address to a hostname
11
PTR-record Reverse DNS
Translaate Hostnmae to IP address
12
Cname - Canonical Name
The ALIAS record maps a name to another name, but in turns it can coexist with other records on that name
13
Load balancing
Netmasking Ordering
Identify default gateway Ip and subnet of incoming request and based on zone/Subnet it allows the access to server.
14
Round Robin
Based on availability and if server is not busy it will access and route request to the given server
15
FQDN
Fully Qualified Domain Name
16
DNS Resolution