Capture

Windows Server 2012 R2 Web Application Proxy and ADFS 3.0 Azure Lab

The following diagrams are based on a lab I built on Microsoft Azure IaaS leveraging Web Application Proxy and ADFS 3.0. to demonstrate single sign-on with claims based applications.

As I come from an application development and architecture background, I learned a great deal with Azure IaaS and system administration with respect to Azure Virtual Networks, Virtual Machines, IP addressing, Azure PowerShell and the Azure management portal, domain controllers, DNS, subnets, certificates and other relevant Windows Server Roles and Features. At the present time of May 2016, I thought I share my notes to help others who may find this helpful in the manner that it is built. Note that I have built this lab in March of 2015 given the Azure’s feature and capabilities at that time.

Lab Architectural Overview

Hosting Infrastructure

  • Microsoft Azure Infrastructure-as-a-Service

Virtual Network

  • One Virtual Network with three subnets
  • Subnet-DC for the domain controller and ADFS server
  • Subnet-Web for web applications and other applications such as SharePoint Server.
  • Subnet-DMZ for the Web Application Proxy

Network Security Groups

  • I didn’t implement any NSG yet, but for proper network security you would have NSG around each subnet to allow/deny traffic based on a set of Access Control List rules.

Windows Domain

  • All servers except for the DMZ are on the same rk.com domain, except for the Web Application Proxy server. For trivial reasons of it being in the DMZ and as a proxy server to the internet.

Public domain name

  • I purchased rowo.ca domain name to be used as part of public urls to internal applications.

Certificates

  • There was a great deal of certificate dependencies between WAP and ADFS and Relying Party (web apps) and token signing. This was a challenging learning point for me and to set things up appropriately and troubleshooting. The detailed topics involved public/private key, export/import certificates, authority chain, thumbprint, certificate subject name, SSL, server authentication, expiry, revocation, browser certificate errors, etc.

screenshot1464024458932

Azure Virtual Network configuration involving address spaces and subnets

screenshot1464024922211.png

I setup ADFS and added my simple .NET claims aware web application as a relying party trust.

screenshot1464025034973.png

I conducted the following test:

Logging into the rkweb1 web server (i.e. internal to the network), I opened the browser
1.Enter the url: https://rkweb1.rk.com/ClaimApp
2.Redirected to ADFS and then authenticated
3.Redirect back to the ClaimApp with access.

screenshot1464025058988.png

Testing withing internal network:

screenshot1464025175345.png

I configured the Web Application Proxy to publish the following applications to the internet.

Internet-facing External URLs are start with https://rowo.ca/ and are mapped to backend URLs starting with https://rkweb1.rk.com for the following applications.

ClaimApp

  • .NET claims based application using Windows Identity Foundation.
  • WAP Pre-authentication is ADFS

HTMLApp

  • HTML web application with no authentication.
  • WAP Pre-authentication is Pass-through. No authentication.

TodoListService

  • REST API with windows authentication
  • WAP Pre-authentication is ADFS

Capture.JPG

Accessing ClaimApp from the internet:

screenshot1464025578290.png

Accessing a REST API via a .NET WPF desktop application from the internet. User will be prompted for credentials in a separate dialog per OAuth.

screenshot1464025704524.png

Accessing ClaimApp through iOS Sarafi browser with device registration. In AD there is a dev

screenshot1464025974358.png

In Active Directory, my iPhone mobile device has been registered for added authentication and conditional access rules to applications.

screenshot1464030919794 (1).png

In conclusion, I loved the fact that Azure has become my IT sandbox to learn and build solutions such as this remote access solution. Also, the Web Application Proxy is one of many other options in the market to publish out internal on-premises applications using ADFS to support single sign-on.

Online References that helped me build this lab

Operational

 

SharePoint 2016 Preview Large List Automatic Indexing with Deep Dive Analysis

The list view threshold (LVT) has been a pain point in some SharePoint sites that I have seen. The default setting in SharePoint 2016 Preview is still 5,000 as it is in 2013.

In cases where lists contain >5,000 items, users will eventually encounter the following message and the list is not displayed.

autoindex-1

According to Software boundaries and limits for SharePoint 2013 article the definition of the List view threshold (LVT) is:

“Specifies the maximum number of list or library items that a database operation, such as a query, can process at the same time outside the daily time window set by the administrator during which queries are unrestricted.”

To manage this constraint in SharePoint 2010 and 2013, read the article Manage lists and libraries with many items

In my opinion, many don’t quite understand what this really is and how to manage it properly. Many project stakeholders other than SharePoint SMEs understand this to be a limitation of how many items can be queried from the list. Rather, it is about the number of items or rows the SQL database has to ‘scan’ implicated by the list view’s query.

For example, let’s say we had a list of 30,000,000 items. Out of these items, we have 4,999 that have Country column value of Canada. List view threshold is set at 5,000.
There is a custom list view where a filter condition is Country = ‘Canada’.

Although it seems that this list view is doing a query for only 4,999 items, what is really happening at the SQL database table level is that all 30,000,000 items are being scanned.

A recommended solution is to index the column found in the list settings.

autoindex-2

Note that the indexing of columns is not a SQL based index such as a non-clustered index, but rather indexing through the NameValuePair_Latin1_General_CI_AS table in the respective content database.

The new Automatic Index Management setting

Now, in SharePoint 2016 Preview, there is a new list setting to automatically index found in List Settings > Advanced Settings. The default is set as ‘Yes’.

autoindex-3

The automatic indexing is supported by the ‘Large list automatic column index management job’.
Go to Central Administration > Monitoring > Review Job Definitions

sp16prev-autoIndexTimerJob

Large List Demo

Configuration

  • Central Administration > Select Web Application > General – Resource Throttling
    • List View Threshold for end users to be at 10,000
    • Auditors and administrators as 20,000
      Note: I doubled the default values just for general testing.

autoindex-5

  • Test User: Added ‘Roy Kim’ user account with only contribute permissions so that I can simulate the list view threshold without the special exceptions that a site collection administrator would have.
  • Custom list
    • Named ‘Large List’
    • Added site columns: Status, Gender, City, Province/State, Country,
    • Added 25,146 items with custom columns including Status. (via a PowerShell script)
    • Created View ‘By Not Started’ where Status equal to ‘Not Started’

autoindex-6

  • Large list automatic column index management job
    • Allow the timer job to run or manually run the job immediately.
  • Indexed Columns
    • Status column has become automatically indexed.
      autoindex-7

SharePoint 2016 Preview Install – First look

SharePoint 2016 Preview was released yesterday on Aug 24.

Download from here: https://www.microsoft.com/en-us/download/details.aspx?id=48712

Announcement: https://blogs.office.com/2015/08/24/announcing-availability-of-sharepoint-server-2016-it-preview-and-cloud-hybrid-search/

After installing, here are my comments as I walk through for noticeable changes:

  1. Similar to Office 365, there is a similar ‘App Launcher’ at the top left.

Newsfeed, OneDrive and Sites sit under your personal My Site.
http://<hostname>/my/personal/<username>/&#8230;

sp16preview-app launcher

2. Under List Settings, there is a new setting ‘Automatic Index Management’:
This may help with the list view threshold constraint where default  5,000 items for a list.

sp16preview-autoindexing

There is a new ‘Large list automatic column index management job’ timer job that may support this setting on the configured lists.

sp16prev-autoIndexTimerJob

After running this timer job, I went to the List Settings > Indexed columns and haven’t noticed any indexed columns. Perhaps the “indexing” can be seen elsewhere. I will have to continue to investigate.

3. Central Administration > Office 365 > Configure hybrid OneDrive and Site Features

SharePoint Hybrid Solutions Center: http://go.microsoft.com/fwlink/?LinkID=613711

sp16preview-O365hybridconfig

4. There are many more new timer jobs at a total of 228. In this screen shot, you may notice some that are new such as DeleteUnusedAbs, Document Changed Anti-virus Processing, DrainInlineStreams and Dump site information. Not sure what these do, but get ready to understand them and leverage accordingly.

sp16prev-newTimerJobs

To compare to a list of SharePoint 2013 timer jobs, read https://technet.microsoft.com/en-us/library/cc678870.aspx

5. “layouts/15” in URL and SharePoint Root Folders

For example, http://<hostname>/_layouts/15/start.aspx#/Shared%20Documents/Forms/AllItems.aspx

The URL contains “_layouts/15” rather than “_layouts/16” taking after the production major version number. Maybe that is why we are seeing the 2013 UI and perhaps in future there could be a change in the UI look.

Also in the the file system there is still the SharePoint 14 root folder (i.e. SharePoint 2010). Maybe this will go away in final release.

sp16prev-rootFolder

6. New Video thumbnails and playback.

I uploaded a 475mb video I grabbed from channel9.msdn.com into the Documents library. I happened to be running a PowerShell script adding 10,000 list items to a custom list and when playing the video in this thumbnail, the playback was a little choppy and slow. So even though this is a single server farm setup, one has to consider scalability and performance of playing videos. This video is surely stored in the content SQL database.

sp16prev-videoThumbNail

Preparing Developer Virtual Machine for SharePoint 2016 Preview

In anticipation for the SharePoint Server 2016 Preview release this month of August 2015, I have prepared a Hyper-V virtual machine for development and evaluation purposes.

System Requirements
http://blogs.technet.com/b/wbaer/archive/2015/05/12/what-s-new-in-sharepoint-server-2016-installation-and-deployment.aspx

Scenario Deployment type and scale Processor RAM Hard disk
Database server running a single SQL instance Development or evaluation installation with
the minimum recommended services
64-bit, 4 cores 12-16 GB 80 GB for system drive
100 GB for second drive

My setup:

Host
OS: Windows 10 Pro
Desktop
Intel Core i5 @ 2.67Ghz 4 Logical Processors
120GB Intel Solid State Drive for VM Disks only
16GB RAM

Hyper-V Virtual Machine
2 Logical processors
12GB RAM

Windows Server 2012R2 with Update
SQL Server 2014 with Service Pack 1
Visual Studio 2015
Chrome

My Software Preparation:

I recently upgraded to Windows 10 and I am enjoying it including the new Start Menu with Tiles and Cortana.

I installed the Hyper-V Manager feature
Control Panel > Programs and Features > Turn On windows features or off
Check off ‘Hyper-V’

Hyper-V

I ran Hyper-V Manager and create a new Virtual Machine and went through the wizard and allocated system resources such as 2 logical processors, location of the VM disk in my solid state drive, and 12 GB of ram. I highly recommend solid state drives for SharePoint VMs as I notice a big performance gain with much higher IOPS.

Note that according to minimum requirements of 4 processors, I will try to settle with only 2 logical processors but if needed I’ll bump up to 4 logical processors. I plan on saving VM system resources by turning off services in Central Administration ‘Services on Server’ such as SharePoint Server Search, Excel Calculation services, Access Services, Machine Translation Service until I want to use them.

I had installed the Windows Server 2012 R2 operating system through .ISO image file.

After installation is complete and booting up Windows Server, I ensured to add roles and features with
Active Directory Domain Services
Web role and app role.
Application Server role

Windows Server Add Role

Next, I installed SQL Server 2014. I just include all the features.

Next, I installed Visual Studio 2015 to develop SharePoint apps/add-ins. I noticed that the install options, you can choose Office Developer tools. Note that you shouldn’t expect it to support SharePoint 2016, but my guess it will to some extent such as simple web parts and list and library deployment.

To download Office Tools separately, visit https://www.visualstudio.com/features/office-tools-vs

Now wait for the SharePoint 2016 Preview download hopefully sometime this month of August.

SharePoint 2013 Custom AJAX filters on List View Web Parts

Requirement:
Implement ‘exact match’ filter against the list view web part on text values and drop down values using AJAX for great user performance. An alternative to filtering in a fly out menu of the list view web part’s column values.ajaxfilter1
Filter by ‘Title’
ajaxfilter2

Filter by ‘Task Status’
ajaxfilter3

Technique:

I am essentially emulating the column based filters that are out-of-the-box functionality of the SharePoint 2013 List view web parts. I do this by tracking down how the filtering is executed in the SharePoint inplview.js code. Then wrap the filtering JavaScript logic through custom text boxes.

ajaxfilter4

You can see the URL query string parameters reflect the filtering
/SitePages/Tickets.aspx#InplviewHashdba27ef6-7bff-4c0f-8ec1-b3e3e18fb0b8=FilterField1=LinkTitle-FilterValue1=Printer broken

For large lists (i.e. > 5,000 items), keep in mind to index the columns you want to filter by. You can find the setting in the List Settings page and find a link ‘Indexed Columns’

<table style="" border="0">
    <tr>
        <td>Title:</td>
        <td><input type="text" id="title" onblur="TitleFilter(); return false;" /></td>
        <td>
            <button onclick="TitleFilter(); return false;">Filter by Title</button>
        </td>
    </tr>
    <tr>
        <td>
            Task Status:
        </td>
        <td>
            <input type="text" id="taskStatus" onblur="TaskStatusFilter(); return false;" />
        </td>
        <td>
            <button onclick="TaskStatusFilter(); return false;">Filter by Task Status</button>
        </td>
    </tr>
    <tr>
        <td></td>
        <td></td>
        <td>
            <button onclick="TitleAndTaskStatusFilter(); return false;">Filter by Title AND Task Status</button>
        </td>
    </tr>
</table>
 

<script language='javascript'>
       function MyRefreshPageToEx(lvTableID, url, bForceSubmit) {

        // Hardcode reference to list view <table> Id attribute
        var tblv = document.getElementById("{BE9A90D1-BF5D-401B-8A9B-C44CCCD14241}-{DBA27EF6-7BFF-4C0F-8EC1-B3E3E18FB0B8}");
        var clvp = CLVPFromCtx(tblv);

        if (clvp != null && clvp.ctx.IsClientRendering) {
            clvp.RefreshPaging(url);
            clvp.ctx.queryString = url;
            if ((typeof clvp.ctx.operationType == "undefined" || clvp.ctx.operationType == SPListOperationType.Default) && Boolean(clvp.ctx.ListData)) {
                var fromPage = clvp.ctx.ListData.FirstRow - 1;
                var toPage = Number(GetUrlKeyValue("PageFirstRow", false, url));

                if (!isNaN(fromPage) && !isNaN(toPage) && fromPage != toPage)
                    fromPage < toPage ? (clvp.ctx.operationType = SPListOperationType.PagingRight) : (clvp.ctx.operationType = SPListOperationType.PagingLeft);
            }
        }
        else {
            SubmitFormPost(url, bForceSubmit);
        }
    }

    function TitleFilter() {
        var title = $('#title').val()
        var url;
        if (title) {
            url = "?List={BE9A90D1-BF5D-401B-8A9B-C44CCCD14241}&View={DBA27EF6-7BFF-4C0F-8EC1-B3E3E18FB0B8}&FilterField1=LinkTitle&FilterValue1=" + title
        }
        inplview.MyRefreshPage = MyRefreshPageToEx;
        inplview.MyRefreshPage(null, url, null);
    }

    function TaskStatusFilter() {
        var taskStatus = $('#taskStatus').val()
        var url;
        if (taskStatus) {
            url = "?List={BE9A90D1-BF5D-401B-8A9B-C44CCCD14241}&View={1F1B1E55-992B-4A03-98B2-9E5D34916611}&FilterField1=TaskStatus&FilterValue1=" + taskStatus
        }
        inplview.MyRefreshPage = MyRefreshPageToEx;
        inplview.MyRefreshPage(null, url, null);
    }

    function TitleAndTaskStatusFilter() {
        var title = $('#title').val()
        var taskStatus = $('#taskStatus').val()

        // HARD Code: query string to reference List GUID and View ID
        var url = "?List={BE9A90D1-BF5D-401B-8A9B-C44CCCD14241}&View={DBA27EF6-7BFF-4C0F-8EC1-B3E3E18FB0B8}&ViewCount=1&IsXslView=TRUE&IsCSR=TRUE&";
        if (title && taskStatus) {
          url = url + "FilterField1=LinkTitle&FilterValue1=" + title + "&FilterField2=TaskStatus&FilterValue2=" + taskStatus
        } else if (title && !taskStatus) {
          url = url + "FilterField1=LinkTitle&FilterValue1=" + title
        } else if (!title && taskStatus) {
          url = url + "FilterField1=TaskStatus&FilterValue1=" + taskStatus
        }
        inplview.MyRefreshPage = MyRefreshPageToEx;
        inplview.MyRefreshPage(null, url, null);
    }
</script>
 

There is some hard coding in the custom JavaScript code making references to the list view web part such as the table Id, list GUID and view GUID.

Find the table element Id as follows.
ajaxfilter5

Limitations

  • Inplview.js filtering only supports exact match of values and not partial match
  • OR operand is not supported in the inpview.js from what I can see.
  • This custom approach is a bit of “hack”. Since it is client side, there is no impact to the SharePoint farm on the server side. Nevertheless, follow any development guidelines in your team. At the least, regression testing is recommended on SharePoint product patching and upgrades for any breaking changes.

The solution fit

I think the better solution fit is where the user already knows exactly what to filter on, but doesn’t want to scroll a largest list of values through the OOTB column filters every single time. For example, filtering by exact name or a small number.

SharePoint 2013 Custom List View Web Part Refresh

In the SharePoint 2013 list view web part, the paging, filtering and sorting functionality is driven by the inplview.js file. This is supported by AJAX calls against a RESTful service inplview.aspx. I am able to extend the JavaScript inplview object to implement client side refresh as data is updated on the server.
refresh1

To configure the automatic refresh by OOTB (out-of-the-box) configuration, go to the Web Part Properties: 

refresh2

 

What I find unattractive of this approach is that it is using the older AJAX UpdatePanel introduced in ASP .NET 2.0.

  • There is at least 23kb transferred for each refresh which is a relatively heavier in bandwidth.
    refresh3
  • The http response time is longer.
  • Using IE 8 browser, there is a memory leak such that memory utilization by the browser maxes out.
  • The out of the paging, filtering and sorting with the AJAX options do not work well together. For example, the AJAX refresh option sometimes shows no rows even after repeated combination of filtering and manual OOTB AJAX refresh. To resolve, refresh the page without any query string parameters.

The applicable business scenario or requirement is when frequently added and updated list items by multiple users are to be seen updated in the browser without user requiring to manually refresh the page. 

In SharePoint 2013, when the user interacts with the list view web part for activities such as paging, column filtering and sorting, there is an AJAX request making a RESTful call to inplview.aspx.
refresh4 Compared to the OOTB refresh, the inplview.aspx REST calls are significantly lighter in bandwidth and has faster response time.

How inplview.js Works

The javascript code that supports the list view web part paging, sorting and filtering is in the inplview.js file

The sequence of calls are made as follows:

  1. Browser
    1. User clicks on list view web part to page, sort and filter
    2. invoke objects in various SharePoint .js files
    3. invoke objects inplview.js
  2. Server
    1. inplview.aspx that is a RESTful http service
    2. HTTP response back to browser with only JSON formatted row data.

Using the chrome JavaScript debugger against inplview.js, I have identified the following sequence of function calls. To support the refresh, I have “forked” the code with my own custom functions.

Original OOTB sequence Customized “forked” sequence
RestoreAllClvpsNavigation Refresh_RestoreAllClvpsNavigation
EnumCLVPs(RestoreClvpNavigation) EnumCLVPs(Refresh_RestoreClvpNavigation)
RestoreClvpNavigation(clvp) <skip>
clvp.RestoreNavigation(); <skip>
CLVPRestoreNavigation() Refresh_RestoreClvpNavigation(clvp)

The Implementation

There is no explicit data refresh support in the inplview javascript code so I have injected custom code that extends the inplview javascript object to have supporting functions to implement refresh.

// automatic refresh based on interval
function autoRefresh()
{
 window.setInterval(listViewRefresh, 2000); // 20 seconds
}

// refresh all list view web parts on the page
function listViewRefresh() {
 $('#lblMessage').text('refreshed ').fadeIn("slow").fadeOut("slow"); // debugging
 inplview.MyRestoreAllClvpsNavigation = MyRestoreAllClvpsNavigation;
 inplview.MyRestoreAllClvpsNavigation();
}

// Enumerate list view web parts
function MyRestoreAllClvpsNavigation()
{
 EnumCLVPs(MyCLVPRestoreNavigation);
}

// refresh referencing list view web part
function MyCLVPRestoreNavigation(clvp) {
 var strHash = ajaxNavigate.getParam("InplviewHash" + clvp.WebPartId());
 if (strHash == null)
 strHash = '';

 var strInpl = '?' + DecodeHashAsQueryString(strHash);
 var strShowInGrid = GetUrlKeyValue("ShowInGrid", true, strInpl);

 if (strShowInGrid == "True") {
 InitGridFromView(clvp.ctx.view, true);
 }
 else if (clvp.ctx.inGridMode) {
 ExitGrid(clvp.ctx.view, true);
 }
 clvp.strHash = strHash;
 clvp.fRestore = true;
 var curRootFolder = GetRootFolder2(this);

 if (curRootFolder != null)
 strInpl = SetUrlKeyValue("RootFolder", unescapeProperly(curRootFolder), true, strInpl);
 clvp.RefreshPagingEx(strInpl, true, null);

 }

$(document).ready(function() {
 autoRefresh();
});

refresh5

Benefits

  1. Overcome IE8 browser memory leak issue
  2. Noticeable performance improvement over OOTB Ajax option
  3. The ability to extend and customize the client side rendering of the list view web part functionality

Caveats

  1. Upgrades and patches:
    Since this is extending from product JavaScript code, in the event of upgrades or patching, there could breaking changes.
    To mitigate this risk, remember to regression test after a product upgrade and patch.

Supportability

  1. Still supports general list view paging, column filtering and sorting.
  2. Exception: column filtering and sorting is not supported on lookup columns as it is already with OOTB.
  3. Ability to refresh all list view web parts that exist on the same page.

I’ll have a another blog post to add filtering based on user input by customizing the inpvliew.js