That's all good information.  If the application isn't crashing, then something is making it take a really long time to load.  Do you have EnableDataViewState set to true for the grid?  Or are you pushing your datasource into viewstate manually?  50,000 rows of data is much too large to fit into viewstate, and it would significantly slow down processing.  But more importantly since upload speeds are usually the bottleneck, form posts (which is what happens when you click a pager link) will likely cause the application to break completely.  If you're not pushing anything up into viewstate, I'd set a breakpoint in your app code and try clicking on a pager link.  See whether the breakpoint is getting hit, and when the response is written.  That would help determine whether the problem is on the browser's form post side, or the server's response side.

"Server Not Responding" would seem to indicate the client isn't getting a response.  The fact that your session variable is being reset means the app_pool has recycled.  App pools can recycle for many reasons, including unhandled application errors, or excessive memory consumption.  If there was an unhandled exception I would think the Grid would get a response, just not the one it wanted – which makes me lean towards memory.

First I'd check the system event log (Application Log) and look for any ASP.NET errors that were logged.  They could lead you to your solution.  

Are you storing anything else in session?  If you're storing say 50,000 rows in session you could easily push the app_pool over it's max memory threshold, which would force the app_pool to recycle.