Compare CAML and LINQ

Yesterday - I interviewed couple of candidates for a new lead developer position in my project. Unfortunately none of those candidates were able to compare CAML and LINQ properly which is a basic question. Let me share my notes here

CAML - Collaborative Application Markup Language is an XML based markup language that helps developers to both construct and display data. CAML can be used by  developers to query against SharePoint lists and views, when programming against the SharePoint API. CAML is also supported by SharePoint Web Services.

LINQ - Language-Integrated Query is relatively new feature that extends powerful query capabilities to the language syntax of C# and Visual Basic. It introduces standard, easily-learned patterns for querying and updating data, and the technology can be extended to support potentially any kind of data store including SharePoint Lists. The LINQ to SharePoint Provider is defined in the Microsoft.SharePoint.Linq namespace. It translates LINQ queries into Collaborative Application Markup Language (CAML) queries. It is no longer necessary for developers to know how to write CAML queries. LINQ queries can be used in server code.  

Disadvantages of CAML: 

• CAML query is text based so, if we are joining two lists across a lookup field there may be various problems associated with that. 
• There is no mechanism to know until run time if the query is written correctly or not. If the query is not correct, then it will simply fail at run time. Means it won't support at design time 
• When writing the query, you have no idea what CAML elements are legal in the syntax without having a reference open. 
• The query is somewhat difficult to understand. We cannot determine easily what the query is doing and what lists are being joined. 
• The data returned from the query is placed in a SPListItem collection, which does not provide strongly typed business entities.

Advantages of LINQ over CAML: 

• First advantage is, it is an object-oriented query language. 
• It can provide strongly typed objects at design time; we can create queries in code and can check that they are correct because we can the compiles the code. 
• The results are returned from queries are strongly typed objects, so the items and fields can provide compile-time checking. 

Disadvantages of Using LINQ 

• LINQ translates the LINQ queries into Collaborative Application Markup Language (CAML) queries thus adding an extra step for retrieving the items. 

Developer Dashboard in SP2010 and new 2013

Have you ever told to figure out why your SharePoint page is consuming much time? In projects where the page render SLAs are at an aggressive end - like one of my current project - we SharePoint engineers will have a real good time...

In the older versions (MOSS 2007, SPS 2003) this was too tuff - personally I used to follow a manual approach in isolating the issue by removing web parts one by one etc... I believe SharePoint product team invested a lot and came up real good feature in SP2010 named Developer Dashboard.

Developer Dashboard is designed to provide additional performance and tracing information that can be used to debug and troubleshoot issues with page rendering time.  This used often by the administrators whenever there is a need to identify / improve page performance. Enabling this great feature will help you get critical information about execution time, log correlation ID, critical events, database queries, service calls, SPRequests allocation and web part events offsets etc.

We can set the DashBoard in the following 3 levels

OnDemand. : In this mode you should see on the left hand side of the ribbon a small icon next to the login credentials. On clicking that icon - SharePoint will toggle the dashboard display beneath the page output. This toggle option will be available only to site collection admins. Looks promising right...

On. :  In this mode you will not see the small icon at the top of the page but the Developer Dashboard will be available on the bottom of your entire page always.

Off. : In this mode you neither see the dashboard nor the icon to display it.

By default the Developer Dashboard will be turned off - it can be enabled easily using object model code, STSADM commands or PowerShell script.

The following script/command will enable developer dashboard in "OnDemand" mode. To switch it on and of , use "on" and "off" parameter values respectively.

PowerShell

$sp =[Microsoft.SharePoint.Administration.SPWebService]::ContentService.DeveloperDashboardSettings;

$sp.DisplayLevel = [Microsoft.SharePoint.Administration.SPDeveloperDashboardLevel]::OnDemand;

$sp.RequiredPermissions = 'EmptyMask';

$sp.TraceEnabled = $true;

$sp.Update();

STSADM

stsadm -o setproperty -pn developer-dashboard -pv ondemand

Object Model

SPWebService sp = SPWebService.ContentService;

sp.DeveloperDashboardSettings.DisplayLevel = SPDeveloperDashboardLevel.OnDemand;

sp.DeveloperDashboardSettings.Update();

Note: DeveloperDashboardSettings has a property called RequiredPermissions.  You can assign a collection of base permissions (like EditLists, CreateGroups, ManageAlerts, or whatever you want) to it; only those people that have those permissions will be able to see the output.

Note: Developer Dashboard is a farm level setting. So If you code the above object model snippet  up in a web part and try to execute it in a non-central admin site, it will throw a security exception.

Note : code from Sandboxed component won’t be showing up in Dashboard since its running in completely different process from the page request.

What is there in Dashboard

In the Dashboard you will find information about the controls, queries and execution time that occur as part of the page rendering process. Usually it provides information from the perspective of the event pipeline, the web server and database. On the left side you can see the different events that fired in the page processing pipeline and its processing duration.  On the top right hand side you see information about the page processing as whole, including the overall execution time, the amount of memory used in the processing of the page request and the correlation ID. Correlation IDs are great info when trying to link the page render to entries in the ULS log.  Underneath the server information you will find a list of the different database calls that were made through the object model by various components in the page itself as well as the controls it hosts. There is an additional option to to see the call stack of every database call - by clicking the database call hyperlinks.

Why to use SPMonitorScope as a best practice

Developer DashBoard does not display the entire set of events instead the code you have in your override for OnInit or Render will be captured in this pipeline.  To ensure the trace of code in other places use the new object model class called the SPMonitoredScope it helps to keep track of useful usage and tracing information just like the developer dashboard uses. You may use it as shown below

using (SPMonitoredScope scpBtnGetScore = new SPMonitoredScope("BtnGetScore_Click"))

{//add your custom code here}

In my understanding this is the only option to understand an manage the custom components once you deploy it in PROD.

Whats new in SP2013

Developer Dashboard in SP2013 is using dedicated WCF service (diagnosticsdata.svc) it allows Detailed request information per page with chant view and additional detailed information included for request analyzing. There is an Interactive addition in the new Developer Dashboard with which you can now View ULS logs (under a dedicated tab) for particular request. This will definitely save developer since it avoids the dependency with admin teams to get ULS logs that they need. Now Developer Dashboard is running in separate window to avoid affecting rendering of actual page.

Don’t know why the OnDemand option is deprecated: Only ON & OFF settings are available. Will share more details soon...

Back to basics 8 : Client side apps in SharePoint 2010

Traditional  server-side controls approach to building a web-based user experience include master pages, application pages, themes, cascading style sheets (CSS), Web Parts, delegate controls, navigation controls and providers, and so on. SharePoint uses these resources to construct an ASP.NET page dynamically. This is then converted into HTML, packaged into an HTTP response, and sent to the web browser on the client. Here All the logic execute on the server and proceeds with a full page postback.

Info flow is Browser -Get/Put/Post---> ASP.net page (with web controls) --> SharePoint Object Model --> ASP.net page -HTML--> Browser.

This approach has the disadvantages like limited interactivity, high response time etc. Rich Internet Application technologies like Ajax, Silverlight, Flash provide more engaging user experience since it execute some logic on browser rather than relying entirely on server side execution. RIA tech use async communication without reloading entire page - it disassembles request from response and results more responsive work and ability to perform work in parallel.

AJAX (Async JavaScript and XML) is a set of technologies that allow us to retrieve data asynchronously from the server and manipulate data in client computer without disrupting entire page. AJAX traditionally rely on Javascript XMLHttpRequest API to send async requests to web server and to return the response to the calling script, the script could then use the response to retrieve and manipulate the DOM of current page (value of a table cell etc). Popular Javascript frameworks such as JQuery (jQuery is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development. AJAX is a technique to do an XMLHttpRequest (out of band Http request) from a web page to the server and send/retrieve data to be used on the web page. AJAX stands for Asynchronous Javascript And XML. It uses javascript to construct an XMLHttpRequest, typically using different techniques on various browsers. jQuery is a javascript framework that makes working with the DOM easier by building lots of high level functionality that can be used to search and interact with the DOM. Part of the functionality of jQuery implements a high-level interface to do AJAX requests. jQuery implements this interface abstractly, shielding the developer from the complexity of multi-browser support in making the request. In nutshell Ajax is a technology / paradigm, whereas jquery is a library (which provides - besides other nice functionality - a convenient wrapper around ajax).AJAX is a way of sending information between browser and server without refreshing page. It can be done with or without library like jQuery. AJAX is the underlying technology for every website that doesn't do a refresh to perform some server-side functionailty. AJAX is provided by javascript using the XMLHttpRequest object which is the heart of AJAX. jQuery, Prototype, Dojo, and all other client side frameworks, are just providing a wrapper for the XMLHttpRequest object so that it makes it browser compatible and easy to use. This is because using the XMLHttpRequest object natively can be quite difficult and may not work across all browsers the way you intend.

Silverlight is a development platform that enables you to create rich, engaging applications that run in web browsers and on other devices. Silverlight developers use a specialized, lightweight version of the Microsoft .NET Framework to create applications. Alongside the benefits of a familiar .NET development experience, Silverlight offers capabilities in areas such as graphics, animation, and multimedia that go well beyond what you can achieve with Ajax.It's increasingly common to use a combination of Silverlight applications, Ajax elements, and traditional server-side controls together to provide a full range of functionality for SharePoint users. Silverlight applications can also interact directly with Ajax elements on a page.

Client Side application is any logic that interacts with SP data from an external computer including browser, stand alone solutions and mobile devices. Client Side logic development will provide the advantages like richer UX, option to modify data from other app like custom solution. Usually compiled Silverlight applications (XAP files) are hosted within HTML object elements, which can either be embedded in the page or generated dynamically through JavaScript. In the case of a SharePoint web page, Silverlight applications are typically hosted within a Web Part. SharePoint 2010 includes a Silverlight Web Part that is specifically designed for hosting XAPs within a SharePoint page. Just like web pages that contain Ajax-enabled components, the browser must use a traditional synchronous page load at least once in order to render the page and download associated resources, such as the Silverlight XAP and any JavaScript files. Silverlight applications can also interact directly with Ajax elements on a page.

Client Object model provide a subset of classes in server side (microsoft.sharepoint.dll) assembly and are implemented as a WCF service (/_vti_bin/client.svc)  , but they use web bindings to implement efficient request batching(Bindings are used to specify the transport, encoding, and protocol details required for clients and services to communicate with each other. WCF uses bindings to generate the underlying wire representation of the endpoint, so most of the binding details must be agreed upon by the parties that are communicating.). Commands are serialized into XML and sent to the server in a single HTTP request. For every command, a corresponding server object model call is made, and the server returns a response to the client in compacted JavaScript Object Notation (JSON) format, which the proxy parses. The client APIs provide a familiar, object-oriented interface to the WCF service, so developers are shielded from the details of the service. In particular, the client-side runtime handles all communication between the clients and server.

Three Client object model Sp2010

1. Client Object Model for Silverlight apps : object model will either be downloaded as part of the .xap file or the file Microsoft.SharePoint.Client.xap seperate.
2. Client Object Model for .Net : called only if the object model assembles installed in client computer (silvelight redistribution package) Info Flow is ...  Managed Control & Logic (Managed Client) --> Managed COM --> Proxy -XML request-> Client.svc (SharePoint server) --> Server Object Model --> Content DB --> Server Object Model --> Client.svc -JSON Response-> Managed COM --> Managed Control & Logic (Managed Client)
3. Client Object Model for JavaScript : the constituent .js files downloaded automatically into users computer when a page references the object model is accessed. Info Flow is ...  Javascript Control & Logic (Browser) --> Javascrict COM --> Proxy -XML request-> Client.svc (SharePoint server) --> Server Object Model --> Content DB --> Server Object Model --> Client.svc -JSON Response-> Javascript COM --> Javascript Control & Logic (Browser) 

In older SharePoint versions the options to access data from client apps were limited to ASP.Net web Services. The lack of a strongly typed object model and the need to construct complex CAML queries to perform simple data operations made the development of SharePoint client apps challenging and somewhat limited. SP2010 has new data access mechanisms that make it easier to build RIA that consume and manipulate SharePoint data. 

There are 3 approaches to access SharePoint data from client apps.

1. Client side object model : 3 seperate APIs that provide a subset of Server Object Model for client side usage. ECMA Script client object model, Silverlight client Object Model and .Net managed client object model
2. REST (Representational State Transfer) Interface uses WCF Data Services to expose SP  Lists and ListItems as addressable resources that can be accessed through HTTP requests. REST interface maps read, create, update and delete operations to GET, POST, PUT and DELETE HTTP verbs.
3. ASP.Net web services are nothing but the ASMX web services that were available in prior sharepoint version

For future compatability, its advisable to use CSOM and REST.

Of the three principal approaches to client-side data access, using the CSOM APIs is the only approach that provides the kind of hierarchical, strongly-typed representation of SharePoint objects, such as sites and Webs, that compares to server-side development. The CSOM is the only approach for any client-side data access scenarios beyond list queries. The CSOM allows you to query SharePoint lists by creating CAML queries. This is the most efficient way to query lists, although it requires that developers revert to creating CAML queries. Specific cases where you should favor the use of the CSOM include the following:

You need to perform advanced list operations, such as complicated joins or paging. You can also perform joins through REST requests, although this is subject to various limitations.
You need to manipulate SharePoint objects, such as sites or Webs.
You need client-side access to other areas of SharePoint functionality, such as security. Although the CSOM APIs mirror the server APIs in terms of functionality, the way CSOM actually works necessitates some changes to the way in which you approach your development tasks. The CSOM uses a set of specialized Windows Communication Foundation (WCF) services to communicate with the SharePoint server. Each API is optimized to work efficiently with remote clients and with the asynchronous loading model used by the Ajax and Silverlight frameworks.

Note: All the CSOM APIs include a ClientContext class that manages the interaction between client-side application code and the SharePoint server. Before you perform any operations in client-side code, you must instantiate a ClientContext object with the URL of a SharePoint site

The client-side code uses the ClientContext class to define a series of operations to execute against a SharePoint site. In this example, the operations are the following:
Retrieve the Parts list.
Retrieve the Inventory Locations list.
Build a query for the Parts list.
Build a query for the Inventory Locations list.
Execute the query against the Parts list.
Execute the query against the Inventory Locations list.
Load the Parts query results (which causes them to be returned to the client).
Load the Inventory Locations query results.
The client code calls the ClientContext.ExecuteQueryAsync method. This instructs the CSOM to send a request containing all operations to the server.
The SharePoint server executes the series of operations in order and returns the results to the client. The CSOM notifies the client-side code of the results by invoking the callback method associated with the onQuerySucceed delegate. This request batching process helps to improve performance and reduce network traffic in two ways. First, fewer Web service calls occur between the client and the SharePoint server, which reduces the "chattiness" of the client-server interface. For example, you can perform two list queries in a single request. Second, as a set of operations occur on the server in a single request, the data being acted on doesn't need to be moved between the client and the server for the intermediate operations—only the list of operations and the final result set are passed between the client and the server.

Request batching requires a different mindset when you create queries from client-side code. First, be aware that you do not have access to any results until you call ExecuteQueryAsync (or ExecuteQuery) and receive the call back with the results. If you need to implement conditional logic in the client-side code that can't be expressed in the command list that you send to the server, you will need to execute multiple queries. Second, you should aim to group your operations to minimize the number of service calls. This means you may need to think about how you sequence your logic in order to take full advantage of request batching.

Representational State Transfer (REST) interface is a WCF data service that allows us to use construct HTTP request to query SharePoint list data.  Like all RESTful Web services, the SharePoint REST interface maps HTTP verbs to data operations, as 
GET - Retrieve
POST - Create
PUT - Update (update all fields and use default values for any undefined fields)
DELETE - Delete
MERGE - Update (update only the fields that are specified and changed from current version)
In practice, many firewalls and other network intermediaries block HTTP verbs other than GET and POST. To work around this issue, WCF Data Services (and the OData standard) support a technique known as "verb tunneling." In this technique, PUT, DELETE, and MERGE requests are submitted as a POST request, and an X-HTTP-Method header specifies the actual verb that the recipient should apply to the request.

The SharePoint REST interface is based on the REST-based Open Data protocol (OData) for Web-based data services, which extends the Atom and AtomPub syndication formats to exchange XML data over HTTP. Because OData is a platform-independent open standard, the SharePoint REST interface is a great way to access SharePoint list data from platforms on which the CSOM may be unavailable to you, such as from non-Windows–based operating systems. However, the REST interface only provides access to list data—if you need to manipulate other data on your SharePoint site, you will need to use the CSOM. The REST implementation can also return the output in JavaScript Object Notation (JSON) format as an alternative to an ATOM feed. JSON is a compact representation of the returned results that can be easily parsed by JavaScript clients.

Ref:

• http://msdn.microsoft.com/en-us/library/ee537247.aspx
• http://msdn.microsoft.com/en-us/library/ff798413.aspx
• http://msdn.microsoft.com/en-us/library/ff521587(v=office.14).aspx
• http://msdn.microsoft.com/en-us/library/ff798339.aspx
• http://msdn.microsoft.com/en-us/library/ff521587(office.14).aspx
• http://msdn.microsoft.com/en-us/library/ee535480(v=office.14).aspx
• http://msdn.microsoft.com/en-us/data/ff478141.aspx

Back to basics 7 : How Sharepoint works

A web application in SharePoint terminology is closely related to what is called a website in IIS terminology. An IIS website monitors for incoming requests through a particular port, and it checks for a particular host header or IP address, or both. Every SharePoint Foundation web application is hosted in an IIS website that has the same name as the web application. It can be helpful, especially when you are trying to see the relation between SharePoint and IIS from a high and broad perspective, to think of the SharePoint web application and its corresponding IIS website as a single entity. For one thing, although there is usually a one-to-one relation between SharePoint web applications and IIS websites, this is not always the case. It is possible to extend a SharePoint web application to multiple IIS websites, although that is not a common design. Just as in any application that is built on the ASP.NET-IIS integrated pipeline, when a front-end web server receives a request from a client for a page or other resource in a SharePoint site, the request is passed through a pipeline of units that process the request. This processing includes authenticating the user, verifying the user’s authorization, building the response, sending the response, and finally, logging the request.

The response to any request is produced by an HTTP handler object. Requests are assigned to one or another HTTP handler object (or to a handler factory class that creates HTTP handler objects) depending on the resource requested and the HTTP verb in the request. The assignment is determined by a stack of configuration files named applicationhost.config, machine.config, and web.config.

The request pipeline also contains HTTP modules. Modules are assemblies that typically contain one or more event handlers or define new events that other modules can handle. An HTTP module can register for one or more events in the life cycle of the request. They are often used to preprocess requests or postprocess the response. The result of a module’s preprocessing is stored in the HttpContext object. For example, the value of the User property is produced by an authentication module.

The processing of a request by HTTP modules and an HTTP handler is governed by an HttpApplication object or an object derived from that class. SharePoint installs a global.asax file in the root of each web application (IIS website) that identifies SPHttpApplication as the class from which an HTTP application object is created.

Back to basics 6 - Whats SharePoint

SharePoint enables information workers who have no knowledge of website design or website administration to create, almost instantly, attractive and functioning websites. This is by the SharePoint features that allows delegated administration. This relieves IT departments from the burden of creating and administering the sites, and it empowers the Info Workers to create their own sites for teams, blogs, wikis, and other purposes. SharePoint provides a platform on which Info Workers  can create collaboration solutions that include document libraries and workspaces, workflows, wikis, blogs, and team-oriented lists, such as Events, Announcements, and Tasks. Microsoft SharePoint Workspace provides an offline experience for these collaboration solutions.

SharePoint also allows Business processes to be systematized and modeled with workflows that are triggered by associated events; for example, the addition of a document to a document library. SharePoint Server Enterprise Content Management (ECM) features include document management, records management, and web content management. The Microsoft Business Connectivity Services (BCS) features of SharePoint enable data from non-SharePoint sources, such as a SAP installation or Oracle database, to be accessed (read/write) just as if it were an ordinary SharePoint list.

SharePoint can provide an intranet system with many of the functions that an operating system provides for a computer, including storing and copying files, hosting services, starting applications, and securing data. SharePoint can also host extranet and Internet-facing solutions. SharePoint deployments make data available through a client object model, the REST-based Windows Communication Foundation (WCF) Data Services (formerly ADO.NET Data Services), and many out-of-the-box ASMX web services. In addition, the SharePoint Service Application Framework provides a platform that enables developers to build scalable middle-tier services that can provide data or processing resources to other SharePoint features.

SharePoint stores data as multicolumn lists in a Microsoft SQL Server database. You can query the data by using LINQ and also using Collaborative Application Markup Language (CAML). The data can be mirrored, backed up, restored, and, depending on the edition of SQL Server being used, you may be able to take snapshots of the data store. Besides its native UI of webpages (including special versions for mobile devices), which can contain ECMAScript (JavaScript, JScript), SharePoint also supports access from Microsoft Silverlight applications and the Microsoft SharePoint Workspace client application. With the SharePoint client object model, you can access SharePoint using Windows Presentation Foundation (WPF), Windows Forms, or any other managed code application.

Back to basics : 5 All about onet xml

The global Onet.xml file defines list templates for hidden lists, list base types, a default definition configuration, and modules that apply globally to the deployment. Each Onet.xml file in a subdirectory of the %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\SiteTemplates directory can define navigational areas, list templates, document templates, configurations, modules, components, and server email footers that are used in the site definition to which it corresponds.

When Microsoft SharePoint Foundation is installed, several Onet.xml files are installed—one in %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\GLOBAL\XML that applies globally to the deployment, and several in different folders within %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\SiteTemplates. Each file in the latter group corresponds to a site definition that is included with SharePoint Foundation.

Depending on where an Onet.xml file is located and whether it is part of a site definition or a web template, the markup in the file does some or all of the following:

• Specifies the web-scoped and site collection-scoped Features that are built-in to websites that are created from the site definition or web template.
• Specifies the list types, pages, files, and Web Parts that are built-in to websites that are created from the site definition or web template.
• Defines the top and side navigation areas that appear on the home page and in list views for a site definition.
• Specifies the list definitions that are used in each site definition and whether they are available for creating lists in the user interface (UI).
• Specifies document templates that are available in the site definition for creating document library lists in the UI, and specifies the files that are used in the document templates.
• Defines the base list types from which default SharePoint Foundation lists are derived. (Only the global Onet.xml file serves this function. You cannot define new base list types.)
• Specifies SharePoint Foundation components.
• Defines the footer section used in server email.

You can perform the following kinds of tasks in a custom Onet.xml file that is used for either a custom site definition or a custom web template:

• Specify an alternative cascading style sheet (CSS) file, JavaScript file, or ASPX header file for a site definition.
• Modify navigation areas for the home page and list pages.
• Add a new list definition as an option in the UI.
• Define one configuration for the site definition or web template, specifying the lists, modules, files, and Web Parts that are included when the configuration is instantiated.
• Specify Features to be included automatically with websites that are created from the site definition or web template

You can perform the following kinds of tasks in a custom Onet.xml file that is used for a custom site definition, but not in one that is used for a custom web template:

• Add a document template for creating document libraries
• Define more than one configuration for a site definition, specifying the lists, modules, files, and Web Parts that are included when the configuration is instantiated.
• Define a custom footer for email messages that are sent from websites that are based on the site definition.
• Define custom components, such as a file dialog box post processor, for websites that are based on the site definition.

Back to basics :4 Sharepoint config files

I always excited about the SharePoint product team's vision on the way they aligned multiple moving parts to achieve the business objectives of the product. One of the main things in that context is the configuration entity. SharePoint configuration settings exist at several levels. As in all ASP.NET applications that use the integrated pipeline, 

• some fundamental settings are in the machine.config file, 
• the global web.config file, and 
• the applicationhost.config file, which is the IIS configuration store. 

SharePoint makes no changes in the default versions of the first two files. It does make some changes in the IIS configuration store. Each IIS website and, therefore, each SharePoint web application, can have a web.config file in its root folder. This file is substantially customized by SharePoint whenever a SharePoint web application is created. 

SharePoint also takes advantage of the fact that web.config files can be applied to specific virtual or physical folders within a web application. For example, SharePoint puts a web.config file in the virtual directory _layouts\mobile (which is mapped to the physical directory %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\LAYOUTS\MOBILE\). Among other things, the web.config file registers a series of filters that control how a page is rendered, based on the capabilities of the mobile device that has requested the page.

Farm-wide configuration settings are stored in the SharePoint configuration database on the computer that is running SQL Server in the farm. Configuration for specific types of websites is contained in various kinds of XML files, such as the Onet.xml file, and configuration for specific instances of websites is contained in the content database. In addition, several classes in the SharePoint object model have property bags that can be used to store custom configuration information for specific objects, such as objects that represent websites, Features, alerts, and files.

Back to basics - 3 Sharepoint Solutions

SharePoint solutions are generally packaged in SharePoint solution package files . SharePoint solution package files can contain several kinds of elements, including assemblies, user controls, custom ASP.NET pages, XML configuration files, resource files, images, list definitions, Web Parts, Features, and others. JavaScript files and Silverlight .xap files can also be deployed to the servers by using SharePoint Solution Packages.

There are 2 kinds of solutions

Farm solutions that are deployed by farm administrators to the farm's solution store, which is in the farm's configuration database. This is done by using either the SharePoint Management Shell or the object model. Solution Deployment step requires a farm administrator and can be done by using either Central Administration or SharePoint Management Shell, or the object model. Some examples of how elements are deployed: user control files (.ascx) are copied to %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\ControlTemplates and assemblies are deployed to the global assembly cache or to a web application's \bin folder. If there are SharePoint Features in the farm solution, they are in this step copied to a subfolder of %ProgramFiles%\Common Files\Microsoft Shared\web server extensions\14\TEMPLATE\FEATURES.

Sandboxed solutions that are deployed to a specific site collection's Solution Gallery by a site collection administrator. The gallery is a special SharePoint list, so it is in the content database. Solution deployment to the gallery is done through the Site Actions UI of the site collection or by using SharePoint Management Shell. The deployment step is taken by a site collection administrator. There are limits on what kinds of elements can be in sandboxed solutions and where they can be deployed. For example, nothing can be deployed to the file system of the servers from a sandboxed solution. Features in sandboxed solutions are deployed to the content database.

Installation of a SharePoint solution package is a multi-step process:
1. Adding: A solution package file is added to one of two stores based on the solution type(farm, sandboxed).
2. Deploying: The solution package is unpacked and its elements are deployed to their appropriate places. It is also in this second step that any Features in the solution are installed in the Feature Gallery of the farm, web application, site collection, or website, depending on the scope of the Feature.

Within a SharePoint solution package, there can be an additional level of encapsulation because a solution can have one or more SharePoint Features. A Feature can be installed at the scope of the farm, the web application, the site collection, or the website. After it is installed, a Feature must be activated by owners of any website within the scope; so Activating becomes a third step of installation for Features. Features can contain content types, controls, custom actions, custom fields, files, workflows, list instances, list templates, event receivers, and document converters; although some of these cannot be included in certain scopes. (Features that are deployed in sandboxed solutions can be scoped only to a site collection or website. For site collection-scoped features in sandboxed solutions, the second and third steps are combined. The Features are activated when the solution is deployed.)

Some SharePoint solutions target one of the client object models, either exclusively or in addition to targeting the server object model: the JavaScript, Silverlight, and Microsoft .NET Framework client object models. The script files that define the JavaScript object model are downloaded to a client computer when a page that references them is opened. Similarly, a Silverlight .xap executable is downloaded to the client computer when a page that is hosting it is accessed. The SharePoint assemblies that contain the Silverlight client object model are encased in the .xap file. Those assemblies are Microsoft.SharePoint.Client.Silverlight.dll and Microsoft.SharePoint.Client.Silverlight.Runtime.dll. 
Stand-alone .NET Framework applications (such as a WPF application) that target the SharePoint client object model are installed just as any other client applications: using MSI files or ClickOnce. The assemblies that contain this client object model, Microsoft.SharePoint.Client.dll and 
Microsoft.SharePoint.Client.Runtime.dll, must be distributed as part of the solution.

Back to Basics 3 - SharePoint Security

SharePoint extends a strong security framework around its deployments to protect from unauthorized users, and errant code. SharePoint's role based security controls user access at different levels right from web app to list item. SharePoint solely depend on the authentication mechanism extended by external systems - Windows Auth or non-Windows Auth. The authorization process implemented within sharepoint assumes the user is properly authenticated.

The default authentication mechanism supported by SharePoint is "Windows Claims based auth" which is built on top of Windows Identity Foundation. In this model, user presents an identity to the Farm as a set of claims (eg: user name, email) and the external identity system is configured to give SharePoint all the 
info that it needs about the user with each request along with cryptographic assurance that the identity data come from a trusted source. SharePoint 2010 also support Windows classic authentication and ASP.Net forms based auth.

Access to all securable sharepoint objectts controlled thru a role based membership system by which users, groups are assigned rles to authorize their access to SharePoint objects. SharePoint follows a inheritance model in granting / revoking user access. A role definition will logically group a set of rights (read list item, create item etc.) that can be associated with a user , group as part of setting role assignment. SharePoint supports both Domain groups and SharePoint Groups. Domain groups will exist outside SharePoint with a less flexibility to manage (add users, remove etc) it but it can be used anywhere within the scope of the Active Directory Service. SharePoint groups are scoped to the site collection level and will be visible within the scope of that site collection. 

As part of code security , SharePoint supports app pool seperation, ability to keep code in GAC and BIN which provides different levels of acces, sandboxed solutions

Every web application runs in an IIS application pool that processes HTTP requests. The pool itself runs in a user account known as the application pool identity. In a multiserver farm, this is usually a domain user. The application pool identity is the user identity for code that runs in an IIS worker process. However, access to various SharePoint objects is determined by the permissions of the user who has made the request that is being processed. The isolation of web applications into separate application pools ensures that if one of them crashes, the others are not affected. The IIS worker process calls assemblies that may operate under their own additional restrictions. If the assembly is loaded out of the global assembly cache, it operates in full trust. However, if the assembly is loaded out of the web application's \bin directory, it is subject to the trust limitations that are defined by a CAS policy.If the request is for a sandboxed solution, the SharePoint execution manager that runs in the IIS worker process spawns a sandboxed worker process. The latter process runs within a highly restrictive CAS policy, is limited to a subset of the SharePoint server object model, and can only access resources within the site collection to which the solution was deployed.

Back to Basics - 2 SharePoint 2010 Server Object Model

Server Object Model in SharePoint is a large and critical classes helps developers to manage

1. Physical object hierarchy : classes represent physical components like servers and farms.
2. Content  hierarchy  : classes that represents publishable items of data and the nested containers of data.
3. Services hierarchy : includes classes that represent web services, Windows services, other types of services, instances of services, and CFSIs of services. 

Main classes in Physical object hierarchy are

• SPFarm : represents a sharepoint farm and its config DB.* 
• SPServer : represenets a physical server in the farm.*
• SPService : represents a farm-wide service. *
• SPSolution : represents a solution on a farm.

Main classes in Content  hierarchy  are

• SPWebApplication : represents a sharepoint web app. *
• SPContentDatabase : represents a content database that contains all the data and the customized page files that constitute the site collections that belong to the database. *
• SPWeb : represents a single website. 
• SPSite : represents a collection of websites within a SharePoint web application that are grouped together for mainly administrative reasons (Among the SharePoint elements that can be scoped to the site collection level are master pages, Web Parts, themes, lists, content types, and Features. A site collection can also be a unit of backup and restoration. It is also the level at which groups of users are created and assigned default permissions. Site collections are the largest possible scope for a search in SharePoint Foundation, although broader search scopes are possible in SharePoint Server 2010. The content of a site collection is always included within a single content database.)
• SPList : represents a sharepoint list and it  has members for programmatically adding, deleting, and retrieving list items, and also for managing metadata about the list such, as its content types and fields. 
• SPListItem : represents individual items of a list and it has members for managing the item's fields, the values of its fields, its content type, and any associated workflows. 
• SPFolder : represents Folders of a List which give a hierarchical structure. 
• SPField : represents field (column) on a list item.

Main Calsses in Services hierarchy are 

• Child class of SPService : represent sharepoint service which provides members that get info about the jobs the service is performing.
• Child class of SPServiceApplication : represents CFSI of a service  *
• Child class of SPServiceInstance : represents each instance of a service. A given CFSI can run on multiple servers, in which case each server has its own actual instance (a running process) of the CFSI. If the service has no CFSIs, it still has instances on each server on which it runs, but there can be no more than one instance of such a service on a given server. 
• SPServiceProxy & SPServiceApplicationProxy  : Applications that consume services that conform to the framework are represented by proxies. The consumer proxy for a service is represented by an SPServiceProxy object and the proxy for the CFSI is represented by an SPServiceApplicationProxy object.*

* inherits from SPPersistedObject, which means that the object that instantiates the class persists in the configuration database

Notes after first Upgrade from SP2010 to 2013 preview

Ohh... as and when a new SharePoint version releases - techies like us will be asked to think and act on upgrade. Now its from SP2010 to SP2013 - I m just done with the first candidate upgrade from SP2010 to Sp2013. As most of us guessed, Database attach is the only method supported by SharePoint team right now. As we all aware this is a costly option where we will upgrade the data and sites on a separate farm from the source. 

Before the upgrade, it is recommended to gather info from the source farm (SP2010) - settings like AAM, Auth Providers and Auth Modes, Quota Templates, Managed Paths, Se;f Service Site Mgmt, email settings, custom components (WSPs etc) Certificates. environment info like - sites list, DB details and number of users. It is also recomended to do a clean up to eliminate poential upgrade errors, major clean up activities are to check and repair all DB consistency errors and turning off Web Analytics service application.

To proceed with database attach upgrade, a new farm should be available and the usual SP2013 setup steps should be employed to setup that. Let me refresh the steps in high level

1. Install software - DB server (SQL Server 2008 R2 or SQL Server 2012), Web / App Server - SP2013 prerequisites and SP2013 product
2. Run config wizard in   Web / App Servers
3. Do not use config wizard to configure setup BCS, MMS, PPS, Search, Secure Store, UPS. We willl do it as part of upgrade

Once the environment is setup properly, copy the following DBs to new environment

• Site Content DBs
• BCS DB
• MMS DB
• PPS DB
• Secure Store DB
• Search DB
• UPS DBs (profile, social, sync)

Note: You may set the DBs read only if you wish to restrict changes in source DBs

Next step is to upgrade service applications using the existing DBs and creating service app proxies using Powershell cmdlets like
New-SPProfileServiceApplication (Adds a User Profile Service application to a farm.)
New-SPProfileServiceApplicationProxy (Creates a User Profile Service application proxy on the local farm.)

Note: Please refer MSDN for full set of cmdlets @ technet

Now we can create and configure web applications, deploy customizations like wsps, site defs, CSS, web parts, web services, features, assemblies and web.config changes, form templates et al. No ned to create site collections since it will be created automatically when we upgrade content DB.

Make sure you are running Test-SpContentDatabase cmdlet before proceed with the next ste which is the Content DB upgrade. This step will verify the new environment has all of the components we need before we actually upgrade the DB.

Use Mount-ContentDatabase cmdlet to upgrade the DB. Note that this wont upgrade UI for the sites contained ion the DB.

Once the DBs are upgraded, site coll admins can upgrade their sites. I recomemnd the following steps for this

1. Run site collection health  checks  to identify and address potential issues in their site collections.
2. Optionally create an upgrade evaluation site collection. This will allow us to preview the new UI and address issues before actual upgrade.
3. Finally the Actual upgrade

You are all set to use the first upgraded SP2013 site!!!!

Right SharePoint 2013 APIs for your needs – A ready reckoner

SharePoint products team amazes me every time they introduce a product version release. There are several options available when it comes to access SharePoint 2013 using APIs. Per my understanding the following will be the factors that decide which option to use

• Type of customization like web parts, Silverlight App, SharePoint App Page, timer job and more. All these customization are different in most aspects and choosing right API option would eventually decide its eventual success
• Enterprise governance rules will be the key differentiator in deciding the API option.
• Development skill level is the other factor. Expertize in a the technologies like .NET Framework, Silverlight, PowerShell, JavaScript, ASP.NET, REST/OData will ensure the best development practices are applied in design / development of custom components
• Devices we are targeting like farm server, client computer, mobile devices etc. The customization targeting different device should use appropriate API option.

Let me share a high level mapping based on the current info ...

API Option	Custom Component
Server Object Model	PowerShell script for farm / web app maintenance, tracking etc
	Timer job to perform scheduled actions on web app
	Web Parts to be displayed in sites
Silverlight client object model	Web Parts to be displayed in sites
	SharePoint  App
	Silverlight Application that integrate with SharePoint site data
	ASP.Net web application that integrate SharePoint site data
REST/Odata Endpoints	ASP.Net web application that integrate SharePoint site data
	SharePoint  App
	HTML/Javascript application that consume SharePoint site data
	App for office
	LAMP web application that integrate SharePoint site data
	iOS/Android app for mobile devices that access SharePoint site data
Javascript APIs	Web Parts to be displayed in sites
	ASP.Net web application that integrate SharePoint site data
	SharePoint  App
	App for office
	HTML/Javascript application that consume SharePoint site data
.Net Client Object Model	.Net application that consume SharePoint site data
Mobile Silverlight API	Silverlight app for windows phone
	XNA app for Windows Phone

more to follow... 

Back to Basics - 1 : Right from web apps and site collections

Basically a SharePoint Web application is a logical abstraction on top of IIS. Physically, a Web Application is a "collection of one or more IIS Web sites configured to map incoming HTTP requests to a set of SharePoint sites."  A mapping of each SharePoint site to one or more content databases that are specified within Central Administration.

Ideally we should primarily utilize separate Web apps for exposure to differing environments for example have intranet and extranet Web apps using HTTP and SSL respectively. As far as scalability is concerned, Web apps support up to 300 content databases and a recommended limit of around 2000 site collections per content DB assuming your infrastructure is up to it.

Adding additional Web apps adds additional administrative and infrastructure burden and - wherever possible - I would suggest using as few as possible. The decision to go with Web apps vs. site collections should be made as part of a wider discussion around your infrastructure.

When we need more than one web application? if there a  need of

• different authentication provider
• different service applications group association
• application pool isolation (e.g. for fault tolerance or security)
• different url (with alternate access mappings)
• different web.config
• different bin folder
• differing throttling settings
• differing security policies. (e.g. set up a "finance" Web app with a locked down user policy, whereas a collaborative Web app may be less restrictive) 

What additional overheads are associated with a Web application?

• Additional RAM required per Web app (even with 0 users) > more hardware required.
• Timer jobs (around 30 per Web app, several of which run every few minutes (e.g. alerts & workflow) and many which run daily (e.g. workflow processing and gradual site deletions)
• More Content databases - per SQL instance limit, backup concerns, storage space
• IIS sites > administrative overheads (host headers, IP addresses, SSL certificates, Web configuration files, assemblies).
• Other operational overheads: DNS / request routing; SharePoint solution management 

I would suggest having different site collection for different department, because

• site collection is root level for many settings starting from permissions,  site collection features like branding, master page, css, etc.
• Also, there are features like Move-SPSite which moves one site collection to a separate content database.
• If your SP Farm grows in time, it will be more scalable if each department having its own site collection is moved in separate content db.
• With this kind of structure, you will prevent having huge databases, have better performance, etc.