Compare CAML and LINQ

Yesterday - I interviewed couple of candidates for a new lead developer position in my project. Unfortunately none of those candidates were able to compare CAML and LINQ properly which is a basic question. Let me share my notes here

CAML - Collaborative Application Markup Language is an XML based markup language that helps developers to both construct and display data. CAML can be used by  developers to query against SharePoint lists and views, when programming against the SharePoint API. CAML is also supported by SharePoint Web Services.

LINQ - Language-Integrated Query is relatively new feature that extends powerful query capabilities to the language syntax of C# and Visual Basic. It introduces standard, easily-learned patterns for querying and updating data, and the technology can be extended to support potentially any kind of data store including SharePoint Lists. The LINQ to SharePoint Provider is defined in the Microsoft.SharePoint.Linq namespace. It translates LINQ queries into Collaborative Application Markup Language (CAML) queries. It is no longer necessary for developers to know how to write CAML queries. LINQ queries can be used in server code.  

Disadvantages of CAML: 

• CAML query is text based so, if we are joining two lists across a lookup field there may be various problems associated with that. 
• There is no mechanism to know until run time if the query is written correctly or not. If the query is not correct, then it will simply fail at run time. Means it won't support at design time 
• When writing the query, you have no idea what CAML elements are legal in the syntax without having a reference open. 
• The query is somewhat difficult to understand. We cannot determine easily what the query is doing and what lists are being joined. 
• The data returned from the query is placed in a SPListItem collection, which does not provide strongly typed business entities.

Advantages of LINQ over CAML: 

• First advantage is, it is an object-oriented query language. 
• It can provide strongly typed objects at design time; we can create queries in code and can check that they are correct because we can the compiles the code. 
• The results are returned from queries are strongly typed objects, so the items and fields can provide compile-time checking. 

Disadvantages of Using LINQ 

• LINQ translates the LINQ queries into Collaborative Application Markup Language (CAML) queries thus adding an extra step for retrieving the items. 

Developer Dashboard in SP2010 and new 2013

Have you ever told to figure out why your SharePoint page is consuming much time? In projects where the page render SLAs are at an aggressive end - like one of my current project - we SharePoint engineers will have a real good time...

In the older versions (MOSS 2007, SPS 2003) this was too tuff - personally I used to follow a manual approach in isolating the issue by removing web parts one by one etc... I believe SharePoint product team invested a lot and came up real good feature in SP2010 named Developer Dashboard.

Developer Dashboard is designed to provide additional performance and tracing information that can be used to debug and troubleshoot issues with page rendering time.  This used often by the administrators whenever there is a need to identify / improve page performance. Enabling this great feature will help you get critical information about execution time, log correlation ID, critical events, database queries, service calls, SPRequests allocation and web part events offsets etc.

We can set the DashBoard in the following 3 levels

OnDemand. : In this mode you should see on the left hand side of the ribbon a small icon next to the login credentials. On clicking that icon - SharePoint will toggle the dashboard display beneath the page output. This toggle option will be available only to site collection admins. Looks promising right...

On. :  In this mode you will not see the small icon at the top of the page but the Developer Dashboard will be available on the bottom of your entire page always.

Off. : In this mode you neither see the dashboard nor the icon to display it.

By default the Developer Dashboard will be turned off - it can be enabled easily using object model code, STSADM commands or PowerShell script.

The following script/command will enable developer dashboard in "OnDemand" mode. To switch it on and of , use "on" and "off" parameter values respectively.

PowerShell

$sp =[Microsoft.SharePoint.Administration.SPWebService]::ContentService.DeveloperDashboardSettings;

$sp.DisplayLevel = [Microsoft.SharePoint.Administration.SPDeveloperDashboardLevel]::OnDemand;

$sp.RequiredPermissions = 'EmptyMask';

$sp.TraceEnabled = $true;

$sp.Update();

STSADM

stsadm -o setproperty -pn developer-dashboard -pv ondemand

Object Model

SPWebService sp = SPWebService.ContentService;

sp.DeveloperDashboardSettings.DisplayLevel = SPDeveloperDashboardLevel.OnDemand;

sp.DeveloperDashboardSettings.Update();

Note: DeveloperDashboardSettings has a property called RequiredPermissions.  You can assign a collection of base permissions (like EditLists, CreateGroups, ManageAlerts, or whatever you want) to it; only those people that have those permissions will be able to see the output.

Note: Developer Dashboard is a farm level setting. So If you code the above object model snippet  up in a web part and try to execute it in a non-central admin site, it will throw a security exception.

Note : code from Sandboxed component won’t be showing up in Dashboard since its running in completely different process from the page request.

What is there in Dashboard

In the Dashboard you will find information about the controls, queries and execution time that occur as part of the page rendering process. Usually it provides information from the perspective of the event pipeline, the web server and database. On the left side you can see the different events that fired in the page processing pipeline and its processing duration.  On the top right hand side you see information about the page processing as whole, including the overall execution time, the amount of memory used in the processing of the page request and the correlation ID. Correlation IDs are great info when trying to link the page render to entries in the ULS log.  Underneath the server information you will find a list of the different database calls that were made through the object model by various components in the page itself as well as the controls it hosts. There is an additional option to to see the call stack of every database call - by clicking the database call hyperlinks.

Why to use SPMonitorScope as a best practice

Developer DashBoard does not display the entire set of events instead the code you have in your override for OnInit or Render will be captured in this pipeline.  To ensure the trace of code in other places use the new object model class called the SPMonitoredScope it helps to keep track of useful usage and tracing information just like the developer dashboard uses. You may use it as shown below

using (SPMonitoredScope scpBtnGetScore = new SPMonitoredScope("BtnGetScore_Click"))

{//add your custom code here}

In my understanding this is the only option to understand an manage the custom components once you deploy it in PROD.

Whats new in SP2013

Developer Dashboard in SP2013 is using dedicated WCF service (diagnosticsdata.svc) it allows Detailed request information per page with chant view and additional detailed information included for request analyzing. There is an Interactive addition in the new Developer Dashboard with which you can now View ULS logs (under a dedicated tab) for particular request. This will definitely save developer since it avoids the dependency with admin teams to get ULS logs that they need. Now Developer Dashboard is running in separate window to avoid affecting rendering of actual page.

Don’t know why the OnDemand option is deprecated: Only ON & OFF settings are available. Will share more details soon...

Back to basics 6 - Whats SharePoint

SharePoint enables information workers who have no knowledge of website design or website administration to create, almost instantly, attractive and functioning websites. This is by the SharePoint features that allows delegated administration. This relieves IT departments from the burden of creating and administering the sites, and it empowers the Info Workers to create their own sites for teams, blogs, wikis, and other purposes. SharePoint provides a platform on which Info Workers  can create collaboration solutions that include document libraries and workspaces, workflows, wikis, blogs, and team-oriented lists, such as Events, Announcements, and Tasks. Microsoft SharePoint Workspace provides an offline experience for these collaboration solutions.

SharePoint also allows Business processes to be systematized and modeled with workflows that are triggered by associated events; for example, the addition of a document to a document library. SharePoint Server Enterprise Content Management (ECM) features include document management, records management, and web content management. The Microsoft Business Connectivity Services (BCS) features of SharePoint enable data from non-SharePoint sources, such as a SAP installation or Oracle database, to be accessed (read/write) just as if it were an ordinary SharePoint list.

SharePoint can provide an intranet system with many of the functions that an operating system provides for a computer, including storing and copying files, hosting services, starting applications, and securing data. SharePoint can also host extranet and Internet-facing solutions. SharePoint deployments make data available through a client object model, the REST-based Windows Communication Foundation (WCF) Data Services (formerly ADO.NET Data Services), and many out-of-the-box ASMX web services. In addition, the SharePoint Service Application Framework provides a platform that enables developers to build scalable middle-tier services that can provide data or processing resources to other SharePoint features.

SharePoint stores data as multicolumn lists in a Microsoft SQL Server database. You can query the data by using LINQ and also using Collaborative Application Markup Language (CAML). The data can be mirrored, backed up, restored, and, depending on the edition of SQL Server being used, you may be able to take snapshots of the data store. Besides its native UI of webpages (including special versions for mobile devices), which can contain ECMAScript (JavaScript, JScript), SharePoint also supports access from Microsoft Silverlight applications and the Microsoft SharePoint Workspace client application. With the SharePoint client object model, you can access SharePoint using Windows Presentation Foundation (WPF), Windows Forms, or any other managed code application.

Back to basics :4 Sharepoint config files

I always excited about the SharePoint product team's vision on the way they aligned multiple moving parts to achieve the business objectives of the product. One of the main things in that context is the configuration entity. SharePoint configuration settings exist at several levels. As in all ASP.NET applications that use the integrated pipeline, 

• some fundamental settings are in the machine.config file, 
• the global web.config file, and 
• the applicationhost.config file, which is the IIS configuration store. 

SharePoint makes no changes in the default versions of the first two files. It does make some changes in the IIS configuration store. Each IIS website and, therefore, each SharePoint web application, can have a web.config file in its root folder. This file is substantially customized by SharePoint whenever a SharePoint web application is created. 

SharePoint also takes advantage of the fact that web.config files can be applied to specific virtual or physical folders within a web application. For example, SharePoint puts a web.config file in the virtual directory _layouts\mobile (which is mapped to the physical directory %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\LAYOUTS\MOBILE\). Among other things, the web.config file registers a series of filters that control how a page is rendered, based on the capabilities of the mobile device that has requested the page.

Farm-wide configuration settings are stored in the SharePoint configuration database on the computer that is running SQL Server in the farm. Configuration for specific types of websites is contained in various kinds of XML files, such as the Onet.xml file, and configuration for specific instances of websites is contained in the content database. In addition, several classes in the SharePoint object model have property bags that can be used to store custom configuration information for specific objects, such as objects that represent websites, Features, alerts, and files.

Back to Basics 3 - SharePoint Security

SharePoint extends a strong security framework around its deployments to protect from unauthorized users, and errant code. SharePoint's role based security controls user access at different levels right from web app to list item. SharePoint solely depend on the authentication mechanism extended by external systems - Windows Auth or non-Windows Auth. The authorization process implemented within sharepoint assumes the user is properly authenticated.

The default authentication mechanism supported by SharePoint is "Windows Claims based auth" which is built on top of Windows Identity Foundation. In this model, user presents an identity to the Farm as a set of claims (eg: user name, email) and the external identity system is configured to give SharePoint all the 
info that it needs about the user with each request along with cryptographic assurance that the identity data come from a trusted source. SharePoint 2010 also support Windows classic authentication and ASP.Net forms based auth.

Access to all securable sharepoint objectts controlled thru a role based membership system by which users, groups are assigned rles to authorize their access to SharePoint objects. SharePoint follows a inheritance model in granting / revoking user access. A role definition will logically group a set of rights (read list item, create item etc.) that can be associated with a user , group as part of setting role assignment. SharePoint supports both Domain groups and SharePoint Groups. Domain groups will exist outside SharePoint with a less flexibility to manage (add users, remove etc) it but it can be used anywhere within the scope of the Active Directory Service. SharePoint groups are scoped to the site collection level and will be visible within the scope of that site collection. 

As part of code security , SharePoint supports app pool seperation, ability to keep code in GAC and BIN which provides different levels of acces, sandboxed solutions

Every web application runs in an IIS application pool that processes HTTP requests. The pool itself runs in a user account known as the application pool identity. In a multiserver farm, this is usually a domain user. The application pool identity is the user identity for code that runs in an IIS worker process. However, access to various SharePoint objects is determined by the permissions of the user who has made the request that is being processed. The isolation of web applications into separate application pools ensures that if one of them crashes, the others are not affected. The IIS worker process calls assemblies that may operate under their own additional restrictions. If the assembly is loaded out of the global assembly cache, it operates in full trust. However, if the assembly is loaded out of the web application's \bin directory, it is subject to the trust limitations that are defined by a CAS policy.If the request is for a sandboxed solution, the SharePoint execution manager that runs in the IIS worker process spawns a sandboxed worker process. The latter process runs within a highly restrictive CAS policy, is limited to a subset of the SharePoint server object model, and can only access resources within the site collection to which the solution was deployed.

Back to Basics - 2 SharePoint 2010 Server Object Model

Server Object Model in SharePoint is a large and critical classes helps developers to manage

1. Physical object hierarchy : classes represent physical components like servers and farms.
2. Content  hierarchy  : classes that represents publishable items of data and the nested containers of data.
3. Services hierarchy : includes classes that represent web services, Windows services, other types of services, instances of services, and CFSIs of services. 

Main classes in Physical object hierarchy are

• SPFarm : represents a sharepoint farm and its config DB.* 
• SPServer : represenets a physical server in the farm.*
• SPService : represents a farm-wide service. *
• SPSolution : represents a solution on a farm.

Main classes in Content  hierarchy  are

• SPWebApplication : represents a sharepoint web app. *
• SPContentDatabase : represents a content database that contains all the data and the customized page files that constitute the site collections that belong to the database. *
• SPWeb : represents a single website. 
• SPSite : represents a collection of websites within a SharePoint web application that are grouped together for mainly administrative reasons (Among the SharePoint elements that can be scoped to the site collection level are master pages, Web Parts, themes, lists, content types, and Features. A site collection can also be a unit of backup and restoration. It is also the level at which groups of users are created and assigned default permissions. Site collections are the largest possible scope for a search in SharePoint Foundation, although broader search scopes are possible in SharePoint Server 2010. The content of a site collection is always included within a single content database.)
• SPList : represents a sharepoint list and it  has members for programmatically adding, deleting, and retrieving list items, and also for managing metadata about the list such, as its content types and fields. 
• SPListItem : represents individual items of a list and it has members for managing the item's fields, the values of its fields, its content type, and any associated workflows. 
• SPFolder : represents Folders of a List which give a hierarchical structure. 
• SPField : represents field (column) on a list item.

Main Calsses in Services hierarchy are 

• Child class of SPService : represent sharepoint service which provides members that get info about the jobs the service is performing.
• Child class of SPServiceApplication : represents CFSI of a service  *
• Child class of SPServiceInstance : represents each instance of a service. A given CFSI can run on multiple servers, in which case each server has its own actual instance (a running process) of the CFSI. If the service has no CFSIs, it still has instances on each server on which it runs, but there can be no more than one instance of such a service on a given server. 
• SPServiceProxy & SPServiceApplicationProxy  : Applications that consume services that conform to the framework are represented by proxies. The consumer proxy for a service is represented by an SPServiceProxy object and the proxy for the CFSI is represented by an SPServiceApplicationProxy object.*

* inherits from SPPersistedObject, which means that the object that instantiates the class persists in the configuration database

Back to Basics - 1 : Right from web apps and site collections

Basically a SharePoint Web application is a logical abstraction on top of IIS. Physically, a Web Application is a "collection of one or more IIS Web sites configured to map incoming HTTP requests to a set of SharePoint sites."  A mapping of each SharePoint site to one or more content databases that are specified within Central Administration.

Ideally we should primarily utilize separate Web apps for exposure to differing environments for example have intranet and extranet Web apps using HTTP and SSL respectively. As far as scalability is concerned, Web apps support up to 300 content databases and a recommended limit of around 2000 site collections per content DB assuming your infrastructure is up to it.

Adding additional Web apps adds additional administrative and infrastructure burden and - wherever possible - I would suggest using as few as possible. The decision to go with Web apps vs. site collections should be made as part of a wider discussion around your infrastructure.

When we need more than one web application? if there a  need of

• different authentication provider
• different service applications group association
• application pool isolation (e.g. for fault tolerance or security)
• different url (with alternate access mappings)
• different web.config
• different bin folder
• differing throttling settings
• differing security policies. (e.g. set up a "finance" Web app with a locked down user policy, whereas a collaborative Web app may be less restrictive) 

What additional overheads are associated with a Web application?

• Additional RAM required per Web app (even with 0 users) > more hardware required.
• Timer jobs (around 30 per Web app, several of which run every few minutes (e.g. alerts & workflow) and many which run daily (e.g. workflow processing and gradual site deletions)
• More Content databases - per SQL instance limit, backup concerns, storage space
• IIS sites > administrative overheads (host headers, IP addresses, SSL certificates, Web configuration files, assemblies).
• Other operational overheads: DNS / request routing; SharePoint solution management 

I would suggest having different site collection for different department, because

• site collection is root level for many settings starting from permissions,  site collection features like branding, master page, css, etc.
• Also, there are features like Move-SPSite which moves one site collection to a separate content database.
• If your SP Farm grows in time, it will be more scalable if each department having its own site collection is moved in separate content db.
• With this kind of structure, you will prevent having huge databases, have better performance, etc.

Record Management capabilities of SharePoint 2010

Last year I was part of an initiative to assess SharePoint 2010 capabilities around records / document management. Spend few weeks to compare business requirements and found the following info...

Functionality	Description	Customizations Required to address majority of enterprises' need	Rating out of 5
Declare record  in-place manually	Option to mark a sharepoint document as record manually to prevent further changes   and deletion	NA	5
Declare records by manually moving it to record center	The “send to Record Center” option to manually declare records and send them to the RecordCenter is   located in Compliance Details overlay. It would be better if its available in the ribbon / pull down menu to make it more usable.	Custom command in ribbon to trigger "send to record center" action	4.5
Declare records automatically based on rule	Process to define rules that mark a sharepoint document as record  to prevent further changes and deletion	NA	5
Retain records and their associated metadata for   their retention time period	Whenever user changes the metadata of a record in Record Center,   system is not synchronizing the metadata changes to the record link which is located in the document center.	Custom component to                         i.      Prevent metadata updation of a record link                       ii.      Propagate metadata changes to record links in document center, whenever metadata values of the linked record has been changed	4
Ensure Record integrity throughout their retention time period	Only Selected metadata fields should be editable. metadata column edit   permissions should be based on the user role   / state of record matrix	Custom feature to restrict updation of critical metadata / record content by applying field level locking based on the following parameters                         i.      Permission Level of the user                       ii.      Stage of document (record, hold)	0
Dispose records in accordance with the Records Retention Schedule	Retention and disposition process of out-of-the-box SharePoint 2010 executes on individual items. The high volume of records for disposition will cause a critical usability issue of managing disposition of individual items. It is recommended to have a batch disposition option. Out-of-the-box SharePoint terminates the expiration process whenever user cancels disposition approval request. It is recommended to have a “Cancelled Records Repository” to restart “disposition of rejected records” / put them on hold / assign a different content type. A workflow to redirect records to record library / document library on an event of cancelled item metadata updation is recommended.	Custom disposition process with the following high level functionalities                         i.      Disposition schedules should run on scheduled intervals (monthly, quarterly, yearly)                       ii.      Records/documents of sub site (project or dept) should be batched together and send for disposition approval                     iii.      Record Stewards should be able to pick and choose items from the list and approve disposition.                     iv.      Record Stewards should also able to apply hold, and change metadata to items by specifying valid remarks. These disposition cancelled records will be sent to Record Administrator for approval                       v.      The Record Administrator should be able to review the disposition of cancelled records and apply the changes suggested by the Record Steward / himself. When Record Administrator approves the changes obtained through disposition cancellation, record should move to Record Center or document center.	1.5
Discover Records for hold and apply hold in batch	The following changes in eDiscovery for Hold are recommended:                         i.      Search refinement with advanced search feature to find actual record set for hold                       ii.      Pick and choose option from the search results Every legal hold search should find the documents and links to records. On applying hold to record links, it should apply hold to the linked records. The following advanced options to streamline the hold process in batch are recommended:                         i.      Apply hold to multiple records after selecting them from a list.                       ii.      Applying hold to a library / folder	Advanced Search functionality to search records to apply hold. Custom solution to apply hold only for those records chosen from the search results. Custom solution to Select multiple records from a library and apply hold. Provision to apply hold on all records/documents of a specific library or folder The eDiscovery search in Document Center should find   documents and links to record along with in-place records.	3
Apply legal Hold to a Record manually	Option to hold a record from further processing and change	NA	5
Release legal Hold from a Record manually	Option to release hold a record to allow  further processing and change	Component to apply hold to the linked record on applying hold in record link.	1.5
Release legal Hold from set of Records	“Search and Release Hold” option should be available which would enable user to release hold by selecting records from search result	Advanced Search functionality to search records on hold. Custom solution to release hold only for those items chosen from the hold search results. Component to Select multiple records from a library and release hold. Component to release hold from the linked record on releasing hold from a record link	1.5
Delete an in place record / record in record center   / record link/ record on hold	Since SharePoint considers a link to a record as an independent entity, users with “contribute” access can delete record links. This action should be restricted to Administrators to ensure accessibility.	Custom feature to prevent record link deletion	3.5
Move an in place record / record in record center / record link/ record on hold	Since SharePoint considers a link to a record as an independent entity, users with “contribute access” can move / edit a record link. This action should be limited to Record Administrators (to make it consistent with Record Center). Also, the system should prevent metadata changes of record link in Document Center. The ability to move documents on hold should be restricted to Record Administrators.	Custom feature to restrict record link and hold records movement.	1
Create a new   version of an in place record / record in record center / record link/ record on hold	SharePoint allows users with “contribute access” to edit all metadata fields. However, only selected metadata fields should be editable based on the logged-in user’s role and status of the record. Updation of metadata (like dates) should recalculate the information management processing dates.	Custom feature to restrict updation of critical metadata / record content by applying field level locking based on the following parameters                         i.      Permission Level of the user                       ii.      Stage of document (record, hold)	0
Upload a new   version of an in place record / record in record center / record link/ record on hold	SharePoint allows users with “contribute access” to replace both in-place records and record links with a new version. This should not be allowed.	Custom feature to prevent in place, record link updation with a new version of document. Custom component to                         i.      Prevent metadata updation of a record link Propagate metadata changes to record links in document center, whenever metadata values of the linked record have been changed.	0
Modify an in place record / record in record center   / record link/ record on hold metadata	SharePoint allows users with “contribute access” to edit all metadata fields. However, only selected metadata fields should be editable based on the logged-in user’s role and status of the record. Updation of metadata (like dates) should recalculate the information management processing dates.	Custom feature to restrict updation of critical metadata / record content by applying field level locking based on the following parameters                         i.      Permission Level of the user Stage of document (record, hold)	0
Email an in place record / record in record center / record link/ record on hold	When we send link to record by attaching it with an email, it should attach the actual record, not the link file.	Custom feature that attaches record with an e-mail when user tries to send a link to record from document center.	4
Apply information management policies	Allow information management policies to process metadata and perform associated actions including record declaration	NA	5
Apply information security by setting up permissions for the library	For certain record types, System must ensure item level permission setting on upload	Custom code component to apply item level permission while uploading a document.	4.5
Generate File   Plan Report	SharePoint 2010s out-of-the-box file plan report is not sufficient to address user needs. A custom implementation to display hierarchical report of taxonomy with record retention details is recommended.	Custom Hierarchical file plan report that displays:                         i.      Site information organization (Site Collections>Sites>Libraries>Folders>Content Type)                       ii.      Record retention details (Libraries>Content Types)	1
Generate Records Audit Report	Out of the box report neither satisfy specify user requirements nor compliant with data sensitivity guideline.  Requires custom code components on top of the log data to make it usable.	Custom audit log report that can perform search SharePoint audit log data and present info related to:                         i.      User actions                       ii.      Action during a period                     iii.      Specific actions (like delete)                     iv.      Failed attempts [optional]"	1
Find a record	It is recommended to use FAST 2010 for search   integration.	Custom options to distinguish records, information on hold within search results.	3.5
Find an in place record / record in record center / record link/ record on hold	Search results do not distinguish in-place records from documents. There should be some mechanism to distinguish type of documents and records.	Option to filter search results based on the type of item (document / in-place record / hold / record link).	3.5

Bottomline:  SharePoint 2010 records management policies would be suited for a small enterprise with a limited number of record types and less repositories and have a straight forward business process to manage info.