Implementing ECTs in SPD using Stored Procedures

If you plan to use Stored Procedures, you will need a separate stored procedure for each CRUD operation. In addition, you will need separate stored procedures for any associations you might need. It is important to note that each Read List, Read Item, and Association stored procedures need to return all the fields that will be required by any other stored procedure defined by that Content Type. In other words, the Read List, Read Item, and Association stored procedures need to return the same exact fields. If they don’t, you will get runtime errors.

Since most examples center on tables, you will often not see a detailed discussion of fields that are required for all the operations as tables always return to you all the fields of that table. So to avoid unintended runtime errors with your ECTs always make sure that your stored procedures return to you all the fields that you think you might need even if you expect not to need them in a particular ECT operation definition. SPD then allows you to define which of these fields should be included in the ECT definition.

The following is a list of field issues that you should be aware of:

  • Unique Identifiers: Each stored procedure needs to provide a unique identifier of type integer. SPD will allow you to have other types of unique identifiers, but you will run into runtime errors if you try to perform any association, create, update, or delete operations. You need these identifiers to avoid issues even if they are completely meaningless to your solution.
  • Limit filters (Read List operations): If it is possible that your data will return more than two thousand records, this will become big problem down the line. BCS by default has a 2000 item throttling limit. This limit can be changed, see BCS PowerShell: Introduction and Throttle Management. You can go without limit filters in development and not see any issue even if your database has hundreds of thousands of records as External lists will by default implement paging. Just understand that if you are using the object model (BCS Runtime or Client object models) to access your data, all records will be returned to you. This can be a major cause of performance degradation and you will not likely see it till you are on a production environment where there are greater latency issues (such as distributed servers, zones, and SSL implementations that you are likely not to have in development). One important thing to note is that a limit filter on its own will just limit the items returned; this means that without another filter type you can only access a subset of your data. For example if you want to limit the amount of books returned by a query to 100, you would add a limit filter and add another such as a Wildcard Filter (say for example a book’s partial title or publish date), this will mean you will get a maximum of 100 books which match the Wildcard filter returned. So in order to implement limit filtering on Read List operation, your Read List stored procedure needs to have an input parameter to use for performing an additional filter criteria.
  • Nullable field types: SPD will give you warnings if it finds fields that are nullable, but it can handle them just fine. Be careful with this as External lists will try to return empty strings to these fields if the fields are not required. This can be a real problem if the field is not of a CHAR, VARCHAR, or some other string type. This will give you runtime errors. If you are using these fields via the object model (BCS Runtime or Client object models), then you can handle this by returning nulls for these field types.

How to develop and deploy the ECTs/BDC Models to multiple environments

Up to this point I have discussed creating ECTs and BDC Models and Resources in SPD, VS, notepad, and Central Admin. So which should we be really using? I will first discuss each of these platforms:

SharePoint Designer
is free tool from Microsoft that can provide no-code solution to creating BDC Models and ECTs. It will provide for rapid development without writing any code or markup, but has two major drawbacks. First, the solution will not be searchable as stated in Issue 3. Second, the solution will be specific to a SharePoint site that you are required to identify before developing your solution and you will not be able to deploy your solution to a different site.

If neither of the drawbacks is an issue for you, then SPD is a wonderful tool for your BCS development. It will even allow you to create External Lists (among other things) based on your ECTs with a push of a button.

Visual Studio (2010)
Another tool from Microsoft that will allow you to create BDC Models, Resources, and ECTs with code or XML markup. Although it is not free, no SharePoint developer should develop without it. It has no limitations on your BCS solution and will allow you to package and deploy your BCS Solution to multiple sites. It’s only true drawback is that you will either need to write code or XML markup for your solution and can therefore not really be recognized as a rapid development platform.

Notepad (or your favorite text editor)
Since you can develop BCS Solutions using only XML markup, there is nothing stopping you from writing your XML in your favorite text editor. This however is no small feat as you will need to be very familiar with all the necessary BCS tags and all their necessary attributes. In addition, this approach is very error prone.

Central Admin
Although Central Admin is not a development platform; it will however allow you to do certain things that will make your development easier. Central Admin will allow you to import and export both .bdcm and .bdcr files. This means that if you do develop your solution in either Notepad or SPD (or even VS for that matter), Central Admin will allow you to import those files, strip out or add resources, and then allow you to export the new model or resources. This is incredibly useful if you want to take a solution built in SPD for example which is targeted to a specific site, strip out all site specific information, and then export out a true BDC Model that can then be implemented on any SharePoint site.

There are also third-party platforms that provide much of the capabilities of SPD without some of the limitations in creating and implementing BCS Solutions that are worth investigating.

So which of the above would you think is the best way to go?

Well, the answer turns out to be a combination of the platforms above, taking advantage of each of the above platform’s capabilities and using another platform to overcome the drawbacks of the first. Below is the process I used to rapidly develop a solution that is both searchable and deployable to multiple environments:

  1. Create all the required ECTs for a given BDC Model in SPD
    1. Include at least a Finder and a Specific Finder method for each ECT
    2. Include any other necessary operations for each ECT
    3. Define any associations for each ECT
    4. Identify a field to mark as a Title field for each ECT
  2. Select all the ECTs and export them out as a single BDC Model
  3. Save that model as a .bdcm file on your local system
  4. Import the .bdcm file into (the BCS Service Application on) CA
  5. Export the model out of CA as a different .bdcm file then was imported into it
    1. Make sure to uncheck any resource checkboxes prior to exporting as a BDC Model (this strips out any resources out of your BDC Model)
  6. Open VS and create a new (or open an existing) Blank SharePoint 2010 project
  7. Add a new BDC Model item
  8. Delete the generated Entity1.cs and the Entity1Service.cs files
  9. Open the .bdcm file (using an XML editor within VS)
    1. Right-Click the .bdcm file and choose Open With…
  10. Delete all content in the .bdcm file in VS
  11. Open the .bdcm file exported from CA with Notepad (or your favorite simple text editor)
  12. Copy all content and paste it into the .bdcm file in VS
  13. Make all the necessary changes to make the model searchable (see Issue 3)
  14. If deploying from VS, then open the newly created Feature.Template.xml file and add the following markup:
    <?xmlversion=”1.0″encoding=”utf-8″ ?>
    <Feature xmlns=””&gt;
    <Property Key=’SiteUrl’ Value=’http://YourDevSite‘/>

    1. Note that for each BDC Model, there is a SharePoint feature (you cannot have more than one model per feature)
    2. The feature will be scoped to the Farm level (as all BCS Solutions are defined at the Farm level in CA)
    3. You should only deploy from VS for development purposes, for Integration, QA, or production deployments, you should use PowerShell
  15. Right-Click on your project and choose to package your solution
  16. Go to your Bin folder in your project from your file system and copy the WSP file that is generated when you performed step 15
  17. Save the WSP onto the SP Application Server that you want your solution deployed to
  18. Deploy and Activate your WSP package to the target SP Application Server
  19. Go to the BCS Service Application in CA and select the BDC Model that was just deployed
  20. 20. Right-Click the model and choose to Set Permissions
  21. Add all the accounts that will need the appropriate permissions to your BDC model and click OK

And you are done…Only 21 steps, not a lot ;).

BDC Models, Resource files, and making Content Types Searchable

As stated earlier, SharePoint Content Types (including ECTs) represent entities and encapsulate all the necessary metadata about those entities. BDC Models represent a collection of ECTs and any relationships between those ECTs. So if you wanted to represent publishers and books and associate all the books published by each publisher, you would need to define the publishers, books, and their association within one BDC Model. BDC Models can be implemented in code via Visual Studio (VS) or can be implemented using XML via VS, SPD, or notepad. If they are implemented using XML, then the resulting file is referred to as the BDC Model (.bdcm).

BDC Models could include other information such as permissions for the ECTs, system or line of business properties, localized names, or proxies, but these types of information are considered resources and are best either implemented in Central Admin or placed in BCS Resource files (.bdcr). These can also be implemented via code in VS, but it is not usually a good practice.

Now although you can create a BDC Model, that doesn’t mean that you can search that model in SharePoint Enterprise (or FAST) Search just yet. You will have to mark the Model and each ECT as searchable. You will also need to create a profile page for at least each model and may create a profile page for each ECT. It is important to note that none of the steps necessary to make your solution searchable can be implemented directly in SPD, you will either have to open your model in notepad or VS and make XML changes for this to work. I will go into further detail about exactly how to do this in a future blog.

Defining and developing your BCS entities into ECTs

There are many examples online and in books on how to create ECTs using SPD. Most of them focus on building ECTs against SQL Server database tables and show you how to create all CRUD operations without differentiating between the different operational requirements for each CRUD operation. Here I will focus on these differences and what they mean.

A SharePoint Content Type is a way for SharePoint to represent an entity. So if you have external data that represents books and publishers, then you will need to create two External Content Types, one representing publishers and another representing books. All ECTs require two operations as a minimum; Read List and Read Item operations. This is because in order to create an external list, you need to be able to read a list of items (books or publishers) and be able to view a particular item (a book or publisher). Each of these operations is a separate request to the database. It is important to note that the Read Item operation is of particular importance as any Update operations will also require a call to the Read Item operation prior to performing the Update.

An Association operation is how the BCS will allow you to get parent-child data. For example, if you have a publisher’s content type and a books content type, then you can use an Association operation to get all the books that belong to a particular publisher. Essentially you need to implement a Stored Procedure that has a publisher identifier as an input parameter and returns book entity fields. In order to do this consistently in the books content type, you should have the Publisher identifier (your foreign key) as one of the fields of the books content type. That field is then used to map the Find Item and Association methods for your ECT. It is important to note that this foreign field has to be a different field than the field that is used as the unique identifier for your child ECT and it has to be unique on the parent ECT. SPD will allow you to use non-integer fields for this, but to avoid runtime errors, make sure the foreign field is of integer type.

Authenticating a BCS Solution to an External System

The first questions you should ask yourself is how do you want to access the data in SQL Server and what accounts you want to get the data with. Most examples you can find will use Pass through, so essentially you are passing the logged-on user’s credentials to SQL Server. This is a problem when you have thousands of users. Do you really want to give customers direct access to the database? Ok, what are our other options then?

We could use Revert To Self. This means that we would use the identity of the application pool to get our data. This is a viable option if we treat all users to our application the same. Unfortunately my client wanted customers to be able to perform CRUD operations on their own data, but no one else. If we used Revert To Self, the database would not know if the current request is for a user that should or should not be able to update the requested information. So that leaves us with one final option; Impersonation.

Impersonation is implemented using the Secure Store Service (SSS) in SharePoint 2010. The idea is that the SSS will detect the current logged-on user’s identity and based on permission rules that we create in the Secure Store, the request to SQL Server will be permitted or denied. If permitted, the SSS will use an impersonated identity defined in the Secure Store to make the request to SQL Server. This approach is ideal if you are going to deploy the solution to multiple environments as users and users’ permissions could be different between the environments and it pushes security as a configuration step, abstracting it from the solution itself. Another benefit to this approach is that the client wanted to authenticate customers via Claims authentication, but wanted their staff to login using AD. The SSS allows us to use Claims groups, as well as AD groups, and give us the capability to assign Claims users permissions to the database that are different from the AD users permissions.

This is the approach we took for authenticating to the SQL Server and I will go into further detail about this in a future blog. For now, you can find more information on Authenticating to Your External System on the BCS Team Blog.

Implementing BCS Solutions using Stored Procedures

Recently I had a client request a BCS solution to provide custom forms to enable customers to manage their profiles and allow employees and other customers to be able to view those profiles. They wanted a BCS solution instead of a traditional web application with a database backend in order to take advantage of SharePoint Enterprise Search. This approach would allow users to be able to search against any customer information that is stored in SQL Server. In addition, they wanted a SharePoint solution package so that they can deploy/redeploy to multiple environments (Integration, QA, and Production).

Customer’s profile information was in tables but not in a form that was directly meaningful to end users (for example they have different types of contact information as different records in the same table and used as lookups). In addition, some of this data was imported from other systems by timer jobs and triggers. The solution was to abstract the data for the BCS by using stored procedures. So our challenge was to develop a BCS solution package(s) based SQL Server stored procedures.

Sounds reasonable…

If you have read any of the marketing materials on the BCS, you would think that the BCS is most ideal way to achieve the client’s goals. SharePoint articles and books would then state that you can create External Content Types (ECTs) and External lists in SharePoint Designer (SPD) and you can be done in an hour or two. To access this data, you can create web parts that use the SharePoint API to access the External lists just like any other SharePoint list.

Easy, right? Well, let’s explore some of the issues that are common to all BCS Solutions and then take a further look at the issues that this particular approach brings up.  For each issue, I will explain our design decisions and the final solution.

Issue 1: Authenticating a BCS Solution to an External System

Issue 2: Defining and developing your BCS entities into ECTs

Issue 3: BDC Models, Resource files, and making Content Types Searchable

Issue 4: How to develop and deploy the ECTs/BDC Models to multiple environments

Issue 5: Implementing ECTs in SPD using Stored Procedures

Why choose a BCS solution?

Surfacing external data in SharePoint enables users to build composite applications that give access to critical information and make their interactions with that information more convenient.  Business Connectivity Services (BCS) is the SharePoint service which allows surfacing of external data from SQL Server, Web Services, or.NET Assembly Connector.  SharePoint even provides no-code BCS solutions to surface the external data via SharePoint Designer to allow for rapid development and provides External Lists to quickly interacting with that data.  You can also secure the data by setting permissions on who can create, read, update, and delete (CRUD) the data.  You can even crawl that data using SharePoint Enterprise Search and set a profile page for rendering the search results in a meaningful way.  You can rapidly developing a feature rich front end for your external data.

Search enabling your BCS solution will however require more than SharePoint Designer to develop.  This is also true if you want to deploy your solution to multiple environments.  This does not mean that you have to write code, but it does mean that you will find yourself in Visual Studio modifying declarative markup that SharePoint Designer can produce and packaging that markup into WSP solution packages.  This approach will allow you to develop external data solutions that you can quickly develop and deploy to multiple environments consistently and will provide your users the ability to immediately search and render meaningful results.

Now if you do determine that you need to integrate external data with SharePoint, but want to still have custom forms or any kind of richer user experience to interact with that data, you could write code using the SharePoint API against external lists.  This approach is often touted as one of the great things about external lists in that you can treat them as any other SharePoint lists.  This approach will work for small datasets, but for large datasets and anytime you are concerned about performance of your forms, you will really want to write your code to go against the BCS Runtime or Client Object Models.  This is a very powerful approach as you can now develop a very rich custom user interface to interact with your external data within the SharePoint context.  You can even access your external data from other applications via the BCS Client Object Model.

As I have discussed, SharePoint provides for rapid development of external data and allows you to apply security and search on your external data.  It also provides the capability to build very powerful and very rich custom user experiences for your external data.  Implementing these solutions to different environments and making your application production ready presents some challenges that are not well documented.  In my next few blog posts, I will go through the process of making an external data solution from beginning to end and show you techniques that will make your solution more stable and production ready.