Tuesday, April 22, 2014

Sentinet–Service Virtualization Part 5 - BizTalk Server

In the previous post I have discussed the benefits of the service virtualization concept for BizTalk applications that have to provide many alternative endpoints configured with different security requirements. I have demonstrated how Sentinet provides easy means to achieve this goal where I configured Sentinet virtual service with two endpoints that route messages to the same single BizTalk Receive Location. Both endpoints provide secure HTTPS access to the BizTalk application with one endpoint configured with anonymous access, and the second configured with the username/password authentication and access control based on specific username/password credentials.

In this post I would like to continue where I left off in the previous post, and demonstrate how to set up another endpoint in the virtual service. This time I want my BizTalk application to make use of a different security mechanism based on the client X.509 certificate. Digital certificates are often used in an application-to-application communication scenarios that require the strongest security tokens based on the digital certificates PKI infrastructure. As in all cases with Sentinet, I will not have to change anything in my BizTalk application. All I have to do is to remotely configure Sentinet virtual service with an additional endpoint.

First I will use Sentinet console to create a new virtual endpoint that is configured with the WsHttpBinding (Mutual Certificates) policy:

clip_image001

The WsHttpBinding (Mutual Certificates) policy requires client X.509 certificate, so I described it with the following WCF binding:

<bindings>
<wsHttpBinding>
<binding name="defaultBinding">
<security mode="Message">
<message clientCredentialType="Certificate" negotiateServiceCredential="false" establishSecurityContext="false" />
</security>
</binding>
</wsHttpBinding>
</bindings>

Note, that this binding configuration uses the flavor of standard WCF wsHttpBinding that requires consumer of the virtual service to present its client X.509 certificate, while the service will also authenticate itself to the client with its own X.509 certificate - hence the mutual X.509 certificates message-level security over HTTP.

Managing X.509 certificates can be a tedious task. I could use the makecert.exe command line tool to create client X.509 certificate, however Sentinet comes here with an additional feature. Sentinet offers optional X.509 certificates management infrastructure that allows remote and secure issuance and management of certificates issued by Sentinet as if it is a Certificate Authority. Therefore, I will request Sentinet to generate a client X509 certificate for me.

clip_image003

A window will pop-up where you can specify the settings for the certificate.

clip_image004

When you click OK you can download complete certificate (with its Private Key) as a PFX password protected file. You can also download only the public part of the certificate as a CER file.

clip_image005

PFX file will be used by the client application as its X.509 identity (client needs complete certificate with its Private Key). At the same time, public part of the certificate in CER file format can be used in the Sentinet Console to create an Access Rule for the virtual service. In my case the access rule will require this particular client certificate to be present as an identity of the service caller. In other words, no other client certificate (even if it is valid and trusted) will be allowed to use the virtual service (hence, I will have authorization).

You can now go ahead and create an Access Rule, which can be applied to the new virtual service endpoint. In Access Rules designer I create a new access rule, MyCertificate Access Rule.

clip_image006

Select Add Access Rule, drag and drop X509 certificate. Click Browse button and navigate to the CER certificate file you just saved.

clip_image007

You can now apply newly created access rule to the RunningX509 virtual endpoint. Go to the virtual service and select Access Control tab, click Modify button and drag and drop the access rule on the desired endpoint. The endpoint is now secured with an access rule that requires very specific client X.509 certificate for authorization. Note that the access rule can be modified to allow many different client certificates, not just one.

clip_image009

At this point I added to my existing virtual service a new endpoint that requires client authentication based on its X.509 certificate. I also secured endpoint with Access Rule that requires very specific client certificate for authorization. This all looks pretty straight forward. The whole process takes only few minutes of time with easy to navigate drag-and-drop Sentinet user interface and no coding.

Now I want to build consumer application and call my virtual service through the new endpoint. For this post I will use Microsoft WcfTestClient utility that will act as a consumer of the virtual service. I start the utility and import the WSDL of the virtual service using URL to the WSDL provided to me by the Sentinet console. I select my GetResultsRaceNameNumber operation that the WcfTestClient utility shows under IVirtualInterface(RunningX509) element.

clip_image011

I fill in sample request data, click Invoke button and get the following error:

clip_image013

WcfTestClient was automatically configured (from the WSDL provided by Sentinet) to use security that requires client certificate, but the certificate was not configured with this client application. So I need to do some additional configuration in the WcfTestClient to provide its client endpoint with the client certificate.

Right click on the Config File entry and select Edit with SvcConfigEditor menu option. In the SvcConfigEditor tool expand Advanced element, right click Endpoint Behaviors element and select New Endpoint Behavior Configuration. This will create NewBehavior0 endpoint behavior. Add clientCredentials, select clientCertificate element and then paste the thumbprint of the client certificate in FindValue field, select the correct store location and its name.

clip_image014

Finally you navigate to the RunninX509 endpoint and select NewBehavior0 from the BehaviorConfiguration drop down.

clip_image015

Click Save in the SvcConfigEditor, and click OK in the WcfTestClient tool when it asks to reload its configuration. Now the NewBehavior0 is attached to the RunningX509 endpoint where behavior specifies client certificate. Select the GetResultsRaceNameNumber operation, fill in request data and Invoke button. Now you will see the result of a successful call to my virtual service.

clip_image017

In Sentinet you can examine this call from the Monitoring screen with all the tracking details where you can see messages coming in and out of the virtual service.

clip_image019

I can also review details of the messages recordings where request and responses can be shown in original wire representation (encrypted) form and in clear text (decrypted). Below is an example of a client request message recorded by the virtual service once request’s body is decrypted and shown in clear text.

clip_image020

Below is the recording of the response message that comes back from the virtual service to the client application. The virtual service’s response message shows the content of the original response generated by my BizTalk application. Recorded response message is shown here in clear text just before it is encrypted for the wire transmission. Note that I can also review recorded requests and responses in encrypted form exactly how they are transmitted over the wire, as well as the same message exchange between the virtual service and my BizTalk service (in addition to described here messages exchange between the WcfTestClient application and the virtual service).

clip_image022

In this blog post I demonstrated how I can secure my internal BizTalk services application with X.509 certificates security (that might represents hypothetical B2B application-to-application scenario). For that, I did not have to change anything in the way my internal BizTalk application is deployed or configured. I simply virtualized BizTalk service through a Sentinet virtual endpoint that requires X.509 certificate security. Sentinet also provided additional managed access control (authorization) and messages exchange monitoring.

In the next post I will discuss and demonstrate even more interesting use case where I will add yet another endpoint to the virtual service. New virtual endpoint will require a SAML token (Federated Security scenario). Effectively, the objective of this use case will be to enable my BizTalk service to be a “claims-aware” application with no coding or any changes in how my existing BizTalk application is deployed and configured.

Cheers,

Steef-Jan

Wednesday, April 09, 2014

Sentinet – Service Virtualization Part 4 - BizTalk Server

In the previous post I have discussed the protocol mediation between SOAP and REST services by the Sentinet product. I also demonstrated how Sentinet provides monitoring capabilities for service and API message exchanges. In this post I would like to continue my exploration of the Sentinet and share my experiences of using Sentinet together with the BizTalk Server.

In this scenario I want to expose the data of my LOB systems through the BizTalk Server. The BizTalk Adapter Pack includes WCF LOB Adapters that will give me access to LOB systems like SAP, Oracle eBusiness suite, and databases like SQL Server or Oracle. After BizTalk Send Port is configured with the WCF LOB Adapter to interface with LOB system, I can provide BizTalk with inbound endpoints that will expose my LOB data as a SOAP or REST service. I can expose BizTalk service endpoints using regular BizTalk WCF Adapters with bindings for HTTP or SOAP protocols. The access to these endpoints must be secured with authentication and authorization – and that can be challenging! In case when BizTalk Server resides behind a DMZ but access to the LOB data is meant to be provided externally from the outside (which is a very typical case), you will need to think about using a reverse proxy.

You could mitigate these challenges by using service virtualization concept implemented in the Sentinet. By placing a Sentinet Node in front of the BizTalk WCF service endpoints you can delegate most (if not all) of the authentication/authorization logic to a virtual service hosted in the Sentinet Node. By using Sentinet you can also make use of the other service virtualization features it offers, like protocol mediation, routing, and monitoring.

In the diagram below you see a Sentinet Node in front a BizTalk hosted endpoint that accepts messages and routes them a send port that communicates with a LOB system (in this case just a SQL Database).

clip_image002

You can setup security of the service endpoint hosted in BizTalk by configuring WCF Adapter. In this scenario you can delegate external security (authentication/authorization of the end users) to the virtual service hosted in the Sentinet Node and keep BizTalk endpoint configured with any WCF Adapter you want regardless of the required external security (for example, you can use WCF-WSHttp adapter with internal windows integrated security). By means of using virtual service you can now leverage the diverse security capabilities offered by Sentinet, where security of the virtual service can be controlled dynamically and remotely via Sentinet browser-based console. You can also make use of multiple identity types such as Username/Password, Kerberos/NTLM, X.509, SAML, binary security tokens and custom identity types over message-level or transport-level security models. Authorization can be enforced by using Sentinet Access Rules designer.

When you have to change BizTalk endpoint security, you have to reconfigure it with a different WCF Adapter, or at minimum you have to reconfigure adapter itself. Sentinet provides BizTalk services with far more flexibility by offering many different authentication/authorization schemes without any need to reconfigure BizTalk artifacts. Besides that, you can configure multiple virtual endpoints with different bindings to provide access to a single BizTalk endpoint, while BizTalk endpoint itself provides access to an LOB system or a database. Basically, Sentinet Node acts as a special gateway that can host any number of diverse virtual services. Once you have exposed your data and/or process through a BizTalk endpoint you will never have to change it to provide access to it for different consumers. The Sentinet will provide that for you through its configuration without any code. You can also create multiple virtual services with multiple endpoints supporting various bindings, authentication and authorization methods exceeding the capabilities of a single BizTalk adapter.

In the previous posts I have demonstrated how to create a Sentinet virtual service. To provide end users with more ways to access LOB data through a virtual service, you need to create multiple inbound virtual endpoints like indicated by the Sentinet screenshot below. Each inbound virtual endpoint will be configured with its own security, while they all route messages to the same BizTalk endpoint.

clip_image004

Now I will walk through creating multiple endpoints with various bindings and security mechanisms to demonstrate the capabilities Sentinet offers. In this and the upcoming posts I will create five different virtual endpoints configured with different bindings and security mechanism. All five virtual endpoints will route messages to the same single BizTalk endpoint.

In this post I will start with a simple (unsecured) endpoint configured with SOAP 1.2 over SSL and no user credentials. Then I will add a secure endpoint configured with SOAP 1.1 over SSL that requires username/password client credentials. In the next posts I will continue with the endpoint that requires client X509 certificate, SAML Token (Claims), and yet another endpoint configured with the Microsoft Azure Service Bus. Note, that while I will be complicating use cases by adding and changing security scenarios, I will not be “touching” my BizTalk Server configuration that will remain configured with the same single endpoint and the same single BizTalk Adapter. In other words, my BizTalk application will be given agility to adapt for continued changes, and it will be provided with new capabilities (such as to be a "claims-aware” application) with no configuration or code changes.

clip_image002[6]

The first endpoint will be an endpoint that uses transport security (https) with SOAP 1.2 binding.

clip_image007

I named this endpoint SearchTimes and I applied to this endpoint the policy that I named Secure HTTPS. Policy configuration is visualized by another Sentinet screenshot below.

clip_image008

The second endpoint will be configured with basicHttp and transport-level security (https) that requires username/password client credentials.

clip_image009

I named this endpoint RunningResults and I applied to this endpoint the policy that I named Steef. Policy configuration is visualized by another Sentinet screenshot below.

clip_image010

After both virtual endpoints are created, I add an Access Control that will manage authorization of the messages sent to these endpoints. Access Control in the Sentinet is driven by the Access Rules that are created in the Sentinet Access Rules Designer. I will use Everyone Access Rule that will provide access to anyone who sends messages over HTTPS to SearchTimes endpoint, while SimpleAuth access rule will require specific username/password. The latter can be created by adding a new access rule in the Sentinet Repository, then using drag and drop of the Username/Password element from the Identity conditions of the Sentinet Access Rules Designer. Provide specific username and password and save your access rule.

Note that Sentinet Access Rule can be configured with many username/password combinations, where specific username/password identities may be stored in the Sentinet Repository, or in Active Directory, or in your own custom database, or in any other external identity system.

clip_image011

After you created Access Rules you assign them to the virtual service Access Control. You simply drag and drop the Access Rule on the virtual endpoint and hit the Save button. You will now see Access Rules applied to virtual endpoints SearchTimes and RunningRules like it is shown in the screenshot below.

clip_image013

To test both virtual endpoints I will use SoapUI. You export the WSDL of your virtual service and save it on disk. Open SoapUI and create a new SOAP Project, specify project name and navigate to the WSDL file saved on disk. Click Ok and the project will create operations for each web service endpoint. Double click the SearchTimes endpoint, click Request 1 and fill in the parameters.

clip_image014

Hit the green play button and the request will be sent to the endpoint of the virtual service. Request will be sent through the virtual service to the BizTalk endpoint (receive location).

clip_image016

You can see all the details of this message exchange through the Sentinet Monitoring and optional messages recording. You will see request message sent by the SoapUI and recorded by the Sentinet virtual service. You can also see complete response that is returned by the virtual service to the SoapUI client application.

The SearchTimes endpoint is using HTTPS transport to encrypt and digitally sign messages on the wire. This does not really mean the use cases represents secure communication because the client identity is not required for this message exchange and anybody can send a message to the SearchTimes endpoint. By adding the RunningRules virtual endpoint I make it a secure endpoint by forcing the user of the service endpoint to provide credentials. In SoapUI navigate to the RunningRules endpoint, double click the endpoint, click Request 1 and fill in the parameters. Hit play and you will see SOAP Fault coming back to SoapUI client.

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<s:Fault>
<faultcode xmlns:a="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">a:InvalidSecurity</faultcode>
<faultstring xml:lang="en-US">An error occurred when verifying security for the message.</faultstring>
</s:Fault>
</s:Body>
</s:Envelope>


In the monitoring of Sentinet I will see recorded request and response. Moreover, you will be able to see that virtual service did not even try to forward request to the BizTalk endpoint, rather it immediately generated SOAP Fault and returned it to SoapUI. The reason for that is clear - I did not provide username/password in the SoapUI. As a result of that, Sentinet virtual service rejected my request with the SOAP Fault, and it did not even “bother” my BizTalk endpoint.

clip_image018

So I have to provide credentials when making my request to the virtual service. This can be specified in the SoapUI behind the Auth tab.

clip_image020

If I hit the Play button now I will receive a response with the data I want. In the Sentinet monitoring I can see the message (request) sent to the virtual service endpoint.

clip_image022

What’s interesting is in the Usernametoken element:

<wsse:UsernameToken wsu:Id="UsernameToken-9">
<wsse:Username>user</wsse:Username>
<wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">test</wsse:Password>
<wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">O11AOMFb+wfUqTn9nB7CBQ==</wsse:Nonce>
<wsu:Created>2014-03-26T09:58:28.550Z</wsu:Created>
</wsse:UsernameToken>

So far I have demonstrated two different endpoints of the virtual service that sits in front of the BizTalk application. As you can see I did not create any code. Everything is done through Sentinet configuration. In the next part I will continue with X509 virtual endpoint followed by parts on SAML/Claims and Service Bus virtual endpoints.

Cheers,

Steef-Jan

Saturday, April 05, 2014

Windows Azure BizTalk Service REST API

The BizTalk Service in Windows Azure can be managed through a REST API. The API provides programmatic access to much of the functionality available through the Management Portal. With the REST API for managing BizTalk Services you can perform the following operations:
  • Register BizTalk Service
  • Create or Update a BizTalk Service
  • Get Cloud Service
  • Get Cloud Services
  • Get BizTalk Service Properties
  • Suspend BizTalk Service
  • Resume BizTalk Service
  • Restart BizTalk Service
  • Poll on an Async Operation on BizTalk Service
  • Sync Access Control Keys
  • Delete BizTalk Service
  • Backup BizTalk Service
  • Restore Azure BizTalk Service from Backup
In this post I like to show how you can leverage the REST API to manage BizTalk Service by means of a windows form application.

Security


To send request messages through the operations of the API you will need to have authentication in place. The REST API like the Windows Azure Service Management API uses mutual authentication of management certificates over SSL to ensure that a request made to the service is secure. Note that anonymous requests to the API are not allowed. Therefor requests made to the REST API for managing WABS requires that a management certificate is associated with your subscription. This certificate can be self-signed using the makecert command line tool or a commercial one obtained through VeriSign or Go Daddy. See the MSDN article Create and Upload a Management Certificate for Windows Azure.


REST API for BizTalk Services


The REST API for BizTalk Services is documented on MSDN. Each operation mentioned in the introduction is documented. The Request URI, URI Parameters, Request Header, Request Message, Request Elements, Response, Status Code, Response Header, and Response Body of each operation is described.

A common operation to perform before any other operation is the Get Cloud Services. This general operation will provide you with the details of your BizTalk Service(s). The Get Cloud Services will get all BizTalk Services within all cloud services. With the data provided by these operations you can perform any other operations as it will provide you with a common piece information the cloud-service-name. This is the unique name for the cloud service that hosts your BizTalk Service.


Get Cloud Services operation



To perform the Get Cloud Services operation from the REST API in a Windows Form you need to provide the subscription id, which is your Azure subscription ID. You can find this id by going to the settings page of your Windows Azure Portal.




The request will be as follows:



The response will look like:



As you see the response provides a list of resources i.e. BizTalk Services and its details like State, Edition, and tracking details.

Get BizTalk Service Properties

Sending a request to the Get BizTalk Service Properties operation will provide you the properties of a specific Windows Azure BizTalk Service. You specify in your request the subscription id of your Azure Account, Cloud Service Name (which can be obtained through the Get Cloud Services operation) and the resource name (name of one of your BizTalk Service(s)).

The request will have the following format:
  • https://management.core.windows.net/subscription-id/cloudservices/CloudServiceName/resources/biztalkservices/~/biztalk/resource-name/properties


The response might look like:




You see all the details of the request BizTalk Service in the response message.

Delete BizTalk Service

Sending a request to the Delete BizTalk Service operation will result in the deletion of the BizTalk Service. You specify in your request the subscription id of your Azure Account, Cloud Service Name (which can be obtained through the Get Cloud Services operation) and the resource name (name of one of your BizTalk Service(s)).

The request will have the following format:

  • https://management.core.windows.net/{subscription-id}/cloudservices/{cloud-service-name}/resources/biztalkservices/biztalk/{resource-name}


The response might look like:



The status code is 202, which means the operation was successful. You can call to check the asynchronous result through GET operation with URL provided Request URI.



If you check within the Windows Azure Portal under BizTalk Services you will see that the specified BizTalk Service is no longer present.

This post I demonstrated how you can leverage the REST API to manage the BizTalk Services. As an example a windows form application implements a few of the REST API operations for WABS. Three operations were discussed yet you can implement any of the other operations in a similar way using .NET code.

There are different ways you can leverage the API other than using a Windows Form Application. You could choose to use ASP.NET or a Mobile application. The REST API for BizTalk Service is created by Microsoft to enable you to perform various operations on the BizTalk Services. Microsoft offers various other REST API’s to manage other services offered in Windows Azure.

The code of the WABS Manager can be obtained through MSDN Code Gallery: Windows Azure BizTalk Services Manager

Note that not all the operations have been implemented in the tool. You reuse the code to create your own full fledged WABS Manager.

Cheers,

Steef-Jan

Friday, April 04, 2014

BizTalk Developer Productivity with BTSG NoS AddIn

A dear friend of mine has been working hard on a developer productivity add in for BizTalk projects in Visual Studio 2012. He name is Nino Crudele a Microsoft Integration MVP from Italy.

DSCN2418

Steef-Jan, Sandro, and Nino Palazzo Gotico in Piacenza, October 2013.

The add in is called BTSG NoS. Nino started with the add in creation started last year after he thought it was time someone cared about developer productivity in BizTalk. Since there has been not any enhancements the last years in the BizTalk Server product regarding productivity Nino took matters in his own hands.

The add in is almost ready for a public beta. At the moment the add in is vigorously tested by fellow Microsoft Integration MVP Sandro Perreira. And he recently created a few post about the add in:

More posts/documentation on the add in will follow. I myself had the opportunity to look at a stable version of the add in and I must say it the user experience is amazing and intuitive. The Installation is straightforward and it is easy to add in Visual Studio through the Add-in Manager.
There is a good set of features available in the add in that will make live easier for a BizTalk developer. You can for instance select your BizTalk project in your solution and choose reflector.

image

It will then show you a dialog with information of the solution.

image

This is just an example of the many feature the add in contains. All of them are target to increase the productivity of the developer.

Nino has done an outstanding job creating this add in and make it available for community with next couple of weeks.So keep a watch on Nino blog. A video of his presentation during the London BizTalk Summit 2014 will be available soon. See BizTalk360 BizTalk Summit 2014, London – Videos.

Cheers,

Steef-Jan

Wednesday, April 02, 2014

First book on WABS is there!

Windows Azure BizTalk Services was released November 2013 and now within six months’ time there is a book about available. It has been written by Karthik Bharathy a Lead Program Manager in the BizTalk product group and longtime BizTalk veteran and Microsoft Integration MVP Jon Fancey.

image

I have had the pleasure of being one the technical reviewers of this book and I must say the authors have done an excellent job. Is this a biased opinion. No, because:
  • I have been working extensively with WABS and its predecessors EAI/Labs (beta). So I was able to put the thumbscrews on both of them, which resulted in valuable feedback for the book
  • of the Microsoft BizTalk Product Group involvement, and they are very committed to it,
  • Scott Guthrie endorses it and wrote a forward.
WABS provides EAI or B2B services through the cloud and both are covered in the book. It is a promising technology with the increase demand for integration. The landscape we know today has changed with the rise of the cloud (internet), devices and on premise growth of the IT-systems (including mainframe, servers, applications, and services). Integration of it is key and so is having an integration service in the cloud to support new hybrid scenarios.
You will learn the following while reading it from cover to cover:
  • Use the EAI and B2B features of BizTalk Services
  • Connect with Line-Of-Business systems in your datacenter on-premises
  • Create bridges and configure them to process and route messages
  • Design transforms
  • Using custom code
  • Address and diagnose problems
  • Operations using the API
  • Migrate from BizTalk Server to BizTalk Services

If you are new to WABS or want to learn more than I definitely recommended it. Even I learned quite a few thing during the review of the chapters. You can download a sample chapter in case you are interested. It is about bridges. You can buy the book from Packt, Amazon, or Barnes & Noble.

In case you are interested in more content than you can visit the TechNet Wiki article: Windows Azure BizTalk Services Resources on the TechNet Wiki.

Cheers,

Steef-Jan

Monday, March 24, 2014

Windows Azure BizTalk Services: Pulling Messages from a Service Bus Queue

Windows Azure BizTalk Services (WABS) provides capabilities for EAI and B2B in the cloud. This relative new service was made available for customers in November 2013. WABS is a cloud integration platform or integration platform as a service or IPaaS. Characteristic of IPAAS is that you build your integration on premise, deploy in the cloud, where it is hosted in a service (set of dedicated hardware). The service is offered through a subscription model for operating the service in the cloud (for BizTalk Services this means Windows Azure hence the name Windows Azure BizTalk Services).

Currently there are quite some players offering that service like Dell, Attunity, MuleSoft, TIBCO, and so on. Microsoft is currently pushing the service hard to catch up with some of these players and try to quicken/hasten the maturity of the product. Therefore, Microsoft has committed to have a release cadence for new features every three months. The recent release was in February 2014, which added new features for WABS. These new features are:
  • EDIFACT Protocol Support and X12 Schema Updates
  • Pulling Messages from Service Bus Queues and Topics
  • Service Bus Shared Access Signatures (SAS) support with Service Bus Queues and Topics
  • BizTalk Adapter Services No Longer Needs SQL On Premises
  • Backup and Restore Support
  • Operation Log Support
This post will dive into the new feature of pulling messages from a Service Bus Queue. When a developer designs and build a bridge he/she can now choose two new sources i.e. Service Bus Queues and Topics. A Bridge with WABS is a workflow that processes a message pulled from a source and delivered to a destination (see picture below).



Benefits of pulling messages from Service Bus queues and topics are:
  • Durable messaging
  • Leveraging the support for multiple protocols
Durable messaging
The ability to pull messages from Service Bus queues and topics increases the durability of a bridge solution. You can leverage the pub-sub mechanism of the topics and subscriptions. It is even possible to have multiple bridges pulling messages from different subscriptions and have them send to multiple destinations. A single bridge will not enable you to do that.
Leveraging multiple protocols
Another benefit is that you as a developer will have more input channels for the bridge besides FTP, and HTTP (REST). Indirectly through the Service Bus Queues you could have support for AMQP.

Scenario

To explore the new capability of pulling message from the Service Bus we will explore the following scenario. A payroll company wants to offer a service in Windows Azure BizTalk Services for organizations to send their payroll data for pay rolling. Organizations that like to outsource their pay rolling can connect to this service in Azure. Basically the organizations can send their payroll data to the service in any kind of format and structure they want. The payroll company will provide a bridge to provide connectivity to the source the organization pushes its data to. This can be FTP(S), HTTP or the Service Bus Queue or Topic (subscription). The data will be picked up from the source and processed to one unified format to be routed to a LOB system or database for a later pay roll run that the payroll company will do. Below you will find a diagram detailing the scenario.



I have used this scenario also in my talk during the BizTalk Summit 2014 in London discussing the manageability of WABS.

Building the solution

The solution will be a bridge for an organization that requires to push out its payroll data to the payroll company. In this case the data is pushed to a Service Bus Queue and the Bridge will pull the data from the queue, transform it to a format that enables a table operation in SQL Server on premise. To build this solution the following parts need to build, configured and/or created:
  • a LOB Target that needs to be configured for access to SQL Server through a LOB Relay endpoint,
  • generation of a schema for the table operation (insert operation),
  • a mapping,
  • a bridge design,
  • connection of the source to the bridge and to the destination,
  • configuration of the source, the bridge and the destination. 
When solution is build it needs to be deployed to the BizTalk Service in the cloud.

Create a BizTalk Service Project


To build a WABS solution you need to create a BizTalk Service project: Open Visual Studio and on the File menu, point to New, and then click Project. Use the details in the following table to create the project and then click Ok:

Installed Templates:  Click BizTalk Services, then click BizTalk Service.
Name:  Specify a name.
Location:  Set this to a location on your computer.
Create directory for solution: Select this if you want this solution to have a separate folder in Windows Explorer.

LOB Target/Relay

A part of the Windows Azure BizTalk Services is on-premise infrastructure. By downloading the WABS SDK you can set up that infrastructure. By infrastructure in this context is addition of WABS templates in Visual Studio 2012 and installation/configuration of the BizTalk Adapter Service. This service is basically a IIS wrapper around the adapter pack that provides access to on-premise LOB systems. The BizTalk Adapter Services can be viewed as a gateway to premise LOB systems that can be securely accessed indirectly through WABS. The concept of BizTalk Adapter Service is to ability to create kind of endpoints to operation on your LOB systems. You create a LOB Target, which is your on-premise Line-of-Business (LOB) system and the operations (like SELECT or INSERT) exposed to your client applications (a Bridge). During the configuration of a LOB Target you create a LOB Relay, which is a URL that provides a connection to the cloud using Service Bus Relays.
For installation of the BizTalk Adapter Service see TechNet Wiki article BizTalk Adapter Service Installation.
Creating a LOB Target/Relay
A LOB Target is created through Visual Studio. You have to perform the following steps to create the LOB Target:
  1. In the BizTalk Service project, open Server Explorer, right-click BizTalk Adapter Service and expand.
  2. You will need to authenticate to further expand to LOB Types.
  3. Provide credentials to access the BizTalk Adapter Service.



  4. Click Ok. You can now expand LOB Types, right-click SQL, and select Add SQL Target.
  5. You will see a Wizard pop-up and the first screen will be a welcome screen.



  6. Click Next and configuration process of LOB target will start.
  7. In the Connection parameters tab you specify the connection details of SQL Server and the credentials.



  8. Click Next.
  9. In the Operations tab you can select object, operations and add them to the selected operations.



  10. Click Next.
  11. In the Runtime Security tab you specify the way you like the authentication to LOB target i.e. SQL Server.



  12. Click Next.
  13. In the Deployment tab you specify LOB Relay. You can select an existing relay or specify a new one.



  14. To create a new LOB Relay, enter the following details: Service Bus Namespace
    Specify the Service Bus namespace on which the LOB relay endpoint is created.
    Service Bus
    Issuer Name

    Specify the issuer name for the Service Bus namespace
    Service Bus
    Issuer Secret

    Specify the issuer secret for the Service Bus namespace
    LOB Relay Path
    Enter a name for the relay.
    LOB Target Sub-path
    Enter a sub-path to make this target unique. 
    Target runtime URL
    This read-only property displays the URL where the relay is deployed on Service Bus. This is the path where you could send a message to be inserted into the on-premises SQL Server. In the payroll scenario, this is where the bridge routes the message.
  15. Click Next.
  16. The summary tab will appear and you review your LOB Target/Relay setup.



  17. By click Create the LOB target/relay will be created. What basically will happen is that LOB target will be created in the BizTalk Adapter Service and is created as an application in IIS.

    Generate schema for the insert operation
    When a LOB Target is created you can generate the schema(s) for specified operation(s). The following steps describe how to generate the schema for the insert operation:
    1. In the BizTalk Service project, in the Server Explorer, right click the LOB target you created, and then click Add Schemas To Payrollservice....
    2. A dialog will appear, where you specify the credentials, folder name, ...


    3. Click Ok.
    4. The schema(s) will be added to the BizTalk Service project.


      Generate schema for request

      In BizTalk Service project you can create the schema for the incoming request message containing the payroll data. The data is delivered per employee. The schema can be created as visualized by screenshot below.


      Create Mapping from source schema to destination schema (XML Transform)

      The following steps show how to transform the incoming request message (source) to a request message for LOB Target (destination):
      1. In Visual Studio, right click the your project, point to Add, and then click New Item.
      2. In the Add New Item dialog box, select Map, specify the map name as ".trfm", and then click OK.
      3. In the Solution Explorer, double click the ".trfm" file to open the Transform. On the Transform surface, select the source schema to Request.xsd and the destination schema to xxxxx_PAYROLLSERVICE_payrolldata_TableOperation.dbo.Employee.xsd.
      4. Drag lines between the various elements in the source and destination schema like below:



        Configure a one-way Bridge

        The following steps describe how to configure the one-way bridge with a source and destination. You select the MessageFlowItinerary.bcs and:
        1. Drag and drop the LOB Target Entity from the BizTalk Adapter Service onto the design surface (destination).
        2. Drag and drop XML one-way bridge onto the design surface.
        3. Finally drag and drop Service Bus Queue source to the design surface (source).
        4. Double click the XML One-Way Bridge on the itinerary designer.
        5. On the XML One-Way Bridge design surface, within the Message Types box, click the add icon [+] to open the Message Type Picker dialog box.
        6. In the Message Type Picker dialog box, from the Available message types box, select the Request (PayRolldata) message type, click the right arrow icon to associate the request schema with the XML One-Way Bridge, and then click OK
        7. Configure the bridge to use the Transform created earlier. Within the Transform stage, select the Xml Transform activity, and then from the Properties window click the ellipsis button (…) against the Maps property to open the Map Selection dialog box.
        8. Save the bridge configuration and go back to the Bridge Configuration designer surface
        Configure the Service Bus Queue source
        The source of the bridge needs to be configured (connectivity) to enable the bridge to receive or pull a message from. For the service bus queue you need to specify the connection string in the following format:
        Endpoint=sb://<your service bus name>.servicebus.windows.net/;SharedSecretIssuer=<issuer name>;SharedSecretValue=<issuer secret>
        Beside the connection string you specify the name (entity) of the source and name of the queue.


        Configure the LOB Target Destination
        Once you have dragged a LOB Target on the design surface you will see a <your lob target>.config appear. You can double click this file in the solution explorer. This file resembles a kind of binding file where you need to specify the credentials of your LOB Target Relay. These are service bus credentials.  After specifying those in the file you can save it.


        Connect the components on the bridge design surface
        After configuration of the source, bridge and destination it is time to connect the source to the bridge and the bridge to the destination. The following steps describe how to connect the components and connection configuration from the XML One-way bridge to the LOB Target:
        1. From the Toolbox, select Connection, and drag-drop the mouse pointer from the Payroll Queue component (source) to the bridge component to connect the two.
        2. Select a Connection again from the Toolbox, and drag-drop the mouse pointer from the bridge component to the LOB Target/Relay component to connect the two. 
        3. Set the filter condition on the connection between the bridge and the LOB Relay entity.
        4. Click the connection between XML One-Way Bridge and the LOB Relay entity.
        5. In the Properties window, click the ellipsis (…) button for Filter Condition.
        6. In the Route Filter Configuration dialog box, set the filter condition to Match All. This ensures that all the messages that are processed through the bridge are routed to the LOB entity.
        7. Click Ok.
        8. Set the Route action so that the outgoing message to the LOB application has the appropriate SOAP action header.
        9. Open Server Explorer and navigate to the SQL LOB Relay created earlier. Right click the relay, click Properties, and for the Operations property, copy the value of the first operation (insert). This value denotes the value of the SOAP action header that must be set on the message that is routed to the LOB Relay.
        10. On the Bridge Configuration surface, click the connection between XML One-Way Bridge and the LOB Relay entity.
        11. In the Properties window, click the ellipsis (…) button for Route Action. In the Route Actions dialog box, click Add to open the Add Route Action dialog box. In the Add Route Action dialog box, do the following:
        12. Under Property (Read From) section, select Expression, and then paste the value that you copied earlier.
        13. Under Destination (Write To) section, set the Type to SOAP and the Identifier to Action. The dialog box resembles the following:



        14. Click OK in the Add Route Action dialog box to add the route action. Click OK in the Route Actions dialog box and then click Save to save changes to an BizTalk Service project. The end-to-end message flow resembles the following:



        15. Save change to the project.

        Build, deploy and Test the solution

        You have finished creating the application. In this steps, you will build and deploy the application under your BizTalk Services subscription. Subsequently you will use the Service Bus Explorer to test your solution.

        To build and deploy the solution

        To build and deploy your solution you can perform the following steps:
        1. In Visual Studio, right click your bridge solution, and then click Build Solution.
        2. Once the build succeeds, right click your solution, and then click Deploy Solution.
        3. In the deployment window, the Deployment Endpoint is a read-only property and the value is derived from the BizTalk Service URL/Namespace set in the message flow surface. However, you must provide the ACS Namespace for BizTalk Services, Issuer Name, and Shared Secret.



        4. Click Deploy. The Visual Studio Output pane displays the deployment progress and result. The URL where the bridge is deployed is also displayed in the Output pane. For this scenario, the bridge is deployed at http://sltn2014.biztalk.windows.net.

        Test the solution

        To test the solution you can use the Windows Azure Service Bus Explorer. This tool can be used to sent a message to a service bus queue. The following steps will describe the way to test the solution:
        1. Open the Service Bus Explorer.
        2. Connect to the appropriate namespace within the service bus where the queue resides that the bridge will pull the messages from.
        3. Right click the queue and select send messages.
        4. Paste/Load a message payload in the Message Text pane.



        5. Click Start.
        6. Message will be sent to the queue.
        7. You can see the log displaying the result.



        8. In SQL Server you can view the Employee table to verify if the data is present in the table.















        The addition of queues and topics as sources to WABS has extended the flexibility and ability to create durable cloud solutions. However, WABS will need more adapters (sources/destinations) to be able to compete with some of the other IPaaS vendors like MuleSoft, who offers a great deal of adapters. Hence more versatile integration capabilities.

        This post demonstrated the use of pulling messages from a Service Bus Queue. This ability can add durability to your cloud integration solution using WABS. The walk through of the payroll scenario demonstrated the ease of creating a bridge solution. Most of the configuration is done through Visual Studio a main component currently to build, deploy and manage a bridge solution.

        The dependency of Visual Studio for building WABS can be considered undesirable. Changes to the configuration of sources and destinations for a bridge has to be done in Visual Studio accordingly and then redeployed to the BizTalk Service hosting the bridge. Therefore, managing some parts of a bridge solution like the configuration of sources and destinations is preferable better off in a different tool like a management console/web application or within Windows Azure itself. Since this is still the first release after the go-live of WABS last year in November you can expect improvement of the service in many areas including the manageability.

        Cheers,

        Steef-Jan