Control lost its state (viewstate) during postback

I want to share my experience during deployment of a new custom application which was develop on top of SharePoint Server 2013. The application was an upgrade version of existing application which had been run well under SharePoint 2007. Long story short the application has been upgraded and verified. I installed this application under SharePoint Server 2013 which act as application server and WFE (server on the cloud), and I joined to another server which has much bigger memory capacity (physical server) which also act as application and WFE. That server also has Project Server 2013 installed, both server has been upgraded to SharePoint and Project Server 2013 SP1.

The next day when users started to use that application, I received many email from users that they couldn't submit theirs request because they have experienced error (looking at ULS it was nullreferenceexception). Looking at trace information that exception was come from a method that take it's value from aspx dropdownlist control. I checked the code and didn't find anything that could related to this error. I asked user to provide me with screen capture before and after the error. To my surprise all aspx web controls has been lost its value during postback. dropdownlist become empty, label become empty and all aspx web controls was lost it's state during postback.

I spent 3 days to figure out what could happen, and that error didn't happen to anyone and happened randomly not just on some button click. During that time I had also noticed that the memory server was utilized at 96-98 percent (this was 16GB server on the cloud). Since this was farm installation, I stopped distributed cache and project server app services on that server. I tested that application with this configuration and the lost control state was still happened. Then I tried accessing that application from another WFE which has twice bigger memory. It was running well without that error.

After 2 weeks of investigation I discovered that the problem came from distributed cache, It looks like aspx or SharePoint store it's viewstate on that distributed cache, and when application trying to get its state back that information got lost during transport from distributed cache server to my application. Network latency between those server hasn't been good during peak hours and using ping command it's sometimes display Request time out. SharePoint default timeout request cache information from distributed cache was set at 20ms. Which was very narrow gap, so I increased that timeout to 80ms. I've also removed distributed cache service from that server.

This blog pointing some got guide on how to troubleshoot this error.