Wednesday, 21 January 2015

Splitting flat file message

Hello again,

After a long time, almost after a year.

In this blog, I will share with you some of my findings while handling Flat File Schema.

Whenever any message goes through the receive pipeline, Disassembler gets executed. This is the second step in pipeline execution. In case of XML disassembler, it can split the incoming message, based on envelope property and XPath. But flat file disassembler do not provide any of such feature.

So, how we can split the incoming bunch of data in a flat file.

The answer is very simple. Do follow below steps to split the incoming flat file.

1) Create input flat file schema. I have created the schema like below:

2) Go to Properties of schema. Set the property "Allow Message Breakup At Infix" to "Yes".

3) Create a new Receive Pipeline -- I named it "receiveFFBunch".

4) Set the "Document Schema" property of the flat file disassembler with the proper incoming schema.

Thats, all we need to do. Now go ahead and deply the poject and test it.

I tried with sample input file as below:

For above mentioned input, I received three output files as below:

That's it.. :-)

Tuesday, 11 March 2014

Filters on Receive Shape -- Biztalk Server 2010

We all know, we can have filters on the Send ports. Those filters are used to make subscription on send port. Filter on the sent port are used to handle/manipulate the outbound flow.

But then what about inbound flow? Is there any way by which my orchestration will pick only those messages which fulfil the condition and ignore the rest of messages?

To do so we have some options available such as to use decide shape, to implement Buisness Rule Engine but those options handles the messages in the orchestration. But, We want something by which messages which satisfy our condition, only those should have reached to orchestration.

Solution is "Filters on Receive Shape"

What is receive shape?
Receive shape is one of the shape provided in Biztalk Orchestration toolbox. In almost all of the orchestration, this is the start of process. It initiates the orchestration. We have to bind the receive shape with expected message type, so that it will subscribes to specified messages only.

In message box, whenever any message comes in, Messaging agent checks for the subscribers for that message. Messaging agent does that by matching the messagetype(target namespace#rootnode) of message with all the current available subscribers. If it finds any match, agent sends copy of message to that subscriber and delete the message from messagebox once it is delivered to subscriber.

So because of this, all the messages with required message type get passed to orchestration.

Suppose we have to create solution for below scenario:

Problem Description :
Suppose we having a process in which we receive orders from say customer ABC. We can process only those orders which are having Price less than 1000. If order price is greater than 1000 then we need to send it to Management dept which will do the further processing on it.

As I said earlier we can achieve this in number of ways, we will see how to do this with the help of filter on receive shape.

Will show you how to create solution for above problem.

Lets start..
Create a biztalk project in Visual Studio 2010.Name it OrderProcess_Filter.
Create two schemas OrderRequest.xsd and PO.xsd

Note : You need to Promote the OrderRequest/Price. We need to do that because we want to set condition in Receive shape based on that field. When you promote the field, Biztalk will create property schema automatically.

Then, create a map which will convert OrderRequest To PO. Name it OrderToPO.btm.

Create a orchestration name it POFilter_process. Add shapes in the orchestration as shown below:

Create below two messages:
msg_inputOrder  --- Message Type OrderProcess_Filter.OrderRequest
msg_outputPO    --- Message Type OrderProcess_Filter.PO

Configure the shapes in Orchestration as suggested below:
Receive_Order(Receive shape) is configured with the message msg_inputOrder. It will activate the orchestartion (Activate property is True)

ConstructPO(ConstructMessage shape) will have transform shape in it. It will transform msg_inputOrder to msg_outputPO. It will use the map OrderToPO.btm,  which we have already created.

Send_PO (Send Shape) is configures to send message of type msg_outputPO to the send port.

Configure Receive Port_ReceiveOrder and port_SendPO.

Now, go to Receive_Order Properties -> Filter Expression.
Click on the Ellipsis. Filter Expression Window will come up. In that select the property on which you want to set the condition/filter. In our case it is OrderProcess_Filter.PropertySchema.Price.
Select aprropriate Operator. We will use use ">"
Specify the constant value in Value column. We will use 1000.

Thats all we need to do, to add the filter on receive shape.
Now Build, deply the solution.

Create physical ports in the Biztalk Server 2010. We will need one receive port and one send port. Create/Configure it.

Ok then all configuration is done now, and we are ready to test the solution..

First we will test with below sample order
<ns0:OrderReq xmlns:ns0="http://OrderProcess_Filter.OrderRequest">

Create sample file with above data and place it in the receive location.File will get picked up, check the Destination folder. You will get file with below contents:

<?xml version="1.0" encoding="utf-8"?>
<ns0:PO xmlns:ns0="http://OrderProcess_Filter.PO">

So far so good, order with price less than 1000 is picked up by the orchestartion. Now lets try one order with price more then 1000. Use below content:

<ns0:OrderReq xmlns:ns0="http://OrderProcess_Filter.OrderRequest">

File will get picked up, check destination folder, you will not get anything over there. So this means that this file did not get picked by our orchestration. In other words message box did not found any subscriber for this message. You will see this order in Suspended State in Biztalk Server Console Admin.

We wanted to send orders with price more than 1000 to administration dept(say to folder Administration). So to do that create one more physical port which will have filter in place like below:

Now test the sample with more than 1000 price again. You will get as is message in the Administration folder. That is because it did not get process through orchestration it was sent directly to Send Port.

This is how we implement filter on Receive shape.

This is it for now.. Will back with some more next time..

Till then Enjoy..

Thursday, 6 March 2014

Call Orchestartion Shape in Biztalk

Most of the programming languages provides a way to do nested coding. By that we can define the a block of code which is generic and may be required number of times. Once we have that defined bunch then we just need to call it wherever we want.
For example:


As you can see in above example A and B are two functions which are nested. Function A call Function B. When execution of called function(B) gets started execution of calling function(A) goes on hold. Calling function resumes it execution after completing the called function.

We have same concept in Biztalk also, we can achieve this through Call Orchestration shape in the given toolbox.
In this blow we will do following things:
Understand what is Call Orchestration.
How to Implement it(with suitable example).

So lets start,

What is Call Orchestration??
We can add nested Orchestration in the Biztalk. Call Orchestration shape allows us to achieve that.
We can Call other Orchestration synchronously, which means calling Orchestration waits till  called orchestration's execution gets complete.
This feature of Biztalk helps programmers to reuse the common orchestration workflow patterns across Biztalk projects.

We can pass on parameters while calling the orchestration. Parameters can be like Message, variable, port reference, role links, correlation set.

How to Implement it(with suitable example):

I have created a demo application, which uses the Call Orchestration.
Below are the steps that I  followed:
Create a Biztalk Project in Visual Studio 2010 name it "Orc_Calling".

Add one simple schema into it.(Make Price as a distinguishable field, also make sure you specify its data type as int) name it "Outer_Schema.xsd"

Then Add couple of Orchestrations in the project
1)InnerOrch.odx -- It will be the common orchestration that we will call from other orchestration.

Short description of InnerOrch.odx --

Create two Orchestration Parameters(New Message Parameter)

This orchestration is designed to check if Price of message it receive is above 500 or not.
input_MsgPara.Price > 500

We have used Construct shape which contains MessageAssignment shape. In this we set the element Status of the message, according to the Price(decide shape branches).

Below are the statement that we have in the Message Assignment:
output_MsgPara = input_MsgPara;
xpath(output_MsgPara, "//Status") = "Expensive Product";

and in other shape we have

output_MsgPara = input_MsgPara;
xpath(output_MsgPara, "//Status") = "Non-Expensive Product";

2)OuterOrch.odx -- It will be the mail orchestration that we will call to other orchestration.

Short descrption of OuterOrch.odx --  We receive a message from Receive port, then we use Call Orchestration shape to call the innerOrch orchestration.Its result is then passed to Send port.

Create two Messages of schema Outer_Schema:

Configuration of the Call Orchestration:
Right click the Call Orchestration shape and select Configure.
In Call Orchestration Configuration, select the innerOrch orchestration, and pass the two messages we have created.

Thats All we need to implement the Call Orchestration.
Build it, deploy it and Test it(Will not cover those steps here, all these steps are same as we did for other applications).


We can even call the orchestration from other projects.
To do that, all we need is to follow couple of simple steps:
1)set the access modifier of the (to be)called orchestration to "public". By Default orchestrations are having modifier as "internal".
2)We Need to add the reference of the project having (to be)called orchestration, in the required project.

After that you can configure the Call Orchestration in the same way as we did earlier.

**Difference Between Call and Start Orchestration**
As we know, for every orchestration Biztalk assigns a thread to it.

Start and Call orchestration differs in following ways:
1) Calling Orchestration uses same thread to execute the another orchestration.
whereas Start Orchestration, Biztalk creates new thread to execute the other orchestration.

2) Call orchestration returns the control after its execution to the calling orchestration, hence it is a synchronous way.
whereas in Start Orchestration, starts the execution in non deterministic way, hence it is a synchronous way.

3) Call Orchestration Reduces the latency, as message does not go via messagebox.
whereas Start Orchestration increases the latency as message travels via message box.

Thursday, 20 February 2014

Biztalk Design Patterns -- Sequential Convoy (Uniform Message) And Aggregator

In this blog we will continue what we have started in my last blog

We have implemented Case(I) in last blog in this we will go through Case(II)
Business Scenario -- Suppose we are creating hospital administration system. In this case, throughout day we will keep on receiving patient Information forms. We have system which receives this form and converts that info into the other form/db.

Technical Aspects -- For every patient's form a separate Orchestration instance will get created.

Solution -- We can create bunch of three Forms. By this way we reduce the number of Orchestration Instances.

This is Uniform Sequential Convoy, as we are using same receive message.

While Implementing solution for above scenario we are going to implement two design patterns 1)Sequential Convoy(Uniform Message) 2) Aggregator Pattern

Aggregator Pattern -- In this pattern we group together number of input messages, to create single batch message.
Throgh the given scenario, we will have two more design pattern under our belt.. :)

Now we will start building the solution for this scenario..

Create one schema for Patient Info. Patient will fill this info when he comes to hospital. (PatientFormas_Src.xsd)

Promote the City. We will need this to create Correlation Sets. Make sure that you are selecting field which will be same for all the received msgs. This is because our correlation set will try to match the msgs with same specified field.

Now lets create one Envelope Schema. Before creating it, lets chk what is the concept of Envelope Schema
As the name suggest it works as Envelope for other schemas. We create Envelope schema in the same way as we create xml schema, both will have structure as well . The only difference is, we need to specify the "Envelope" Property of schema as "Yes". By doing so, we tells the compiler that this schema will act as a envelope. In other words, this envelope will act as a Batch of schema msgs.
We also need to set Body XPath of the Root node, this XPath will point to the node which will actually act as Envelope(Which will contain batch in it).

In our case Envelope Schema will only contain Root node and its child node as "Any Element"
After all your Envelope schema will look something like this(Form_Batch.xsd):

As we all know, One of the important task of Send Pipeline is to Assemble/Batch messages. We will make use of this feature in our orchestration to batch our required message.
For that we will need to create a Send Pipeline in out solution. Right Click Project -> Add -> New Item -> Send Pipeline -> In the created Send Pipeline's skeleton add "XML Assembler" component, at Assemble level.(SendPipeline_Uniform.btp)
Also, Go to Properties of XML Assemble,  set the property "Envelope schemas" in the specified ellipsis with the name of your envelope schema.

So lets design orchestration now, we will design it in a way that it will receive 3 input forms and then creates the batch and sends it to destination folder. Our Orchestration will looks like:(Uniform_Process.odx)

Create Messages of type PatientForms_Src, Form_Batch
Will also need some variables as below,
Var_ctr         Int32
SendPipelineVar Microsoft.XLANGs.Pipeline.SendPipelineInputMessages

You need to add reference of the SendPipelineInputMessages. Its dll is present at the location "Drive:\Program Files\Microsoft BizTalk Server 2010".

Create a Correlation Type and Correlation Set based on UID.

Explanation of shapes in the above orchestration is as follows:

Port_ReceiveForms -- This is receive port, which will receive the input Patient forms from the specified location.

Receive_Input -- Receive Shape, which will receive input and starts the orchestration. Initialise the correlation set in this receive shape.

Expression_Initializer -- It will contain following instructions:
Var_ctr = Var_ctr + 1;

We are incrementing the counter of received messages, so that we can add the condition based on that.
Along with that we are adding the received message in the Send Pipeline variable.

Scope_Batching -- This is a None Scope which is required as we are going to deal with Send Pipeline objects, which are non serializable.

Loop_Batch -- Looping required add the repetitive actions for required number of time.In this scenario I have used 3 times iterations. So the condition is Var_ctr < 3

Receive_InputLoop  --  This Receive shape will receive second onwards messages. This will follow the correlation set.

Expression_Batching -- This will contain same instartion what we had in Expression_Initializer.

ConstructMessage_Envelope -- This message will hold MessageAssignment_1 which will create Batch message from the SendPipeline variable. At this point of time variable will hold all the messages but those are not yet batch together, so that we will do in Message Assignment.

Msg_Env = null;
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(typeof(SendPipeline_Uniform), SendPipelineVar, Msg_Env);

Send_Envelope -- To send the Envelope message to send port.

So far, so good.. We done with the solution creation, Now name it, sign it, deploy it.
Configure the application in the Biztalk console with physical Receive/Send port details.

Now Lets test the application..
I have placed below same file three times in the receive folder

Sample1.xml :
<ns0:PatientForm xmlns:ns0="http://SequentialConvoy.PatientForms_Src">

(When biztalk picks the first file, it get dehydrated, meanse it is waiting for some more files for its processing)

After consuming these three sample files what I got is output is as below:

<?xml version="1.0" encoding="utf8" ?> 
 <ns0:Patient_Batch xmlns:ns0="http://SequentialConvoy.Form_Batch">
 <ns0:Patient_Batch xmlns:ns0="http://SequentialConvoy.Form_Batch">
 <ns0:PatientForm xmlns:ns0="http://SequentialConvoy.PatientForms_Src">
 <ns0:PatientForm xmlns:ns0="http://SequentialConvoy.PatientForms_Src">
 <ns0:PatientForm xmlns:ns0="http://SequentialConvoy.PatientForms_Src">

So in this way, we have seen implementaion of two Biztalk Design Patterns in this blog..

Will catch you in next blog with some more Patterns..


Wednesday, 19 February 2014

Biztalk Design Patterns -- Sequential Convoy (Non-Uniform Message)

In this blog we will have look at Biztalk Pattern -- Sequential Convoy Non-Uniform.

What is Biztalk Pattern??
Biztalk Patterns are nothing but the common scenarios which we use frequently while designing the Biztalk solution.
Depending on the requirement, one has to decide which pattern to use. We can use one or more patterns in the solution.

In this blog we will understand and Implement Sequential Convoy(Non Uniform) Pattern.

What is Sequential Convoy?
Sequential Convoy is the pattern in which we group number of receive messages together in Singleton manner.

What are the scenarios where we can implement Sequential Convoy.
Case I --

Buisness Scenario -- In most cases patient also have their Mediclaim. Now we have two documents for each patient(patient info and Mediclaim).

Technical Aspects -- Patient may or may not have mediclaim. So wait for some specific time for mediclaim form if it comes then group the mediclaim and patient info and send to send folder. If Mediclaim does not come in specified time then pass on the only patient info to send folder.

Solution -- At first, We will receive patient Info form then we will wait for mediclaim form for same customer(Unique Reference ID). If if comes in specified time then club the two message for that customer and process those.

This is Non Uniform Sequential Convoy, as we grouping two different receive messages in this sequential Convoy.

Case II --

Buisness Scenario -- Suppose we are creating hospital administration system. In this case, throughout day we will keep on receiving patient Information forms. We have system which receives this form and converts that info into the other form/db.

Technical Aspects -- For every patient's form a separate Orchestration instance will get created.

Solution -- We can create bunch of Forms such as 20 forms at a time or collect all the forms after every one hour. By this way we reduce the number of Orchestration Instances.

This is Uniform Sequential Convoy, as we are using same receive message.

Enough with the theory, Lets start with the implementation of Case I with Non Uniform Sequential Convoy...

To start with, lets create three schemas:

1)schema_patient -- This will contain patient's Info

2)Schema_Medi    -- This will contain patient's mediclaim info.

3)patientForm    -- This will contain mix atrributes of PatientForm and Mediclaim Form.

4)Property111(Property Schema) -- This schema is needed for the purpose of correlation. Here we can promote the properties so that we can use them in orchestation.
In our scenario we will promote schema_patient/AdharNo, Schema_Medi/AdharNo. In the Property Schema create a new field name it as "UniqueNo".

To promote the properties go to that field -> Promote -> Show Promotion
Following window will pop up

Click on Property Fields Tab. Browse to Property file which we have created. From left panel add the field which we want to promote.

We will also create a map file which will convert schema_patient and Schema_Medi into PatientForm.It will contain following links:

Now, we are all set to design out orchestration.  Design the orchestration as below:

Explanation :
Port_1 : This is the receive port which will accept the Patient Info forms and Mediclaim Form from same location. It has two different Operations. (Operation_1) for receiving msg of type schema_patient (Operation_2) for receiving msg type Schema_Medi.
Note - Set "Ordered Delivery" property to "True". This is to specify that Msg will get delivered in sequence.

Receive_1 : This is the Starting point of Orchestration where we will receive msg of schema_patient.Set "Activate" property to "True".

Listen_1 : This Listen shape is added because we need to wait for Mediclaim for some specific time. So it will contain two branches. One branch(Receive_2) will be waiting for msg of type Schema_Medi and other(Delay_1) will wait for some time period. I have used following expression:

new System.TimeSpan(0,1,0) 
By this Listen shape will wait for 1 minute and will start the execution if other branch does not start their execution.

Correlation Set up
Create one Correlation Type with Property Field which we have set in the Property file.
Create Correlation Set based on that Correlation Type.

For Receive_1 -- Set property "Initializing Correlation Set" with name of Correlation set you created.
For Receive_2 -- Set property "Following Correlation Set" with name of Correlation set you created.

ConstructMessage1 : This construct shape contains Transform_1 shape which is having map to convert schema_patient + Schema_Medi => patientForm.

Send_2 : Send shape to send the msg of type patientForm to Send Port.

Send_1 : Send shape to send the msg of type schema_patient to Send port.

Port_2 : This is Send Port to which we will send the msgs of type patientForm.

Port_3 : This is Send Port to which we will send the msgs of type schema_patient.

With this we are done with the creation of solution, now deploy this solution.
Configure the application in Biztalk console with receive and send port configuration.
Now, lets test what we have done till now..

Generate the instance of schema_patient place it in the specified Receive folder.
My Input sample contains:
<ns0:Pat xmlns:ns0="http://Listen_Practice.schema_patient">

You wont see any output for 1 minute. This is because we have set the delay of 1 min. So Orchestration will wait for Schema_Medi msg for 1 min.

Within this 1 minute generate the instance of Schema_Medi place it in the specified Receive folder.
My Input sample contains:
<ns0:MClaim xmlns:ns0="http://Listen_Practice.Schema_Medi">

And when I chk the Output folder it is having following contents:
<?xml version="1.0" encoding="utf-8" ?> 
<ns0:PatientForm xmlns:ns0="http://Listen_Practice.patientForm">

If you do not send Schema_Medi msg within 1 min then you will get output as:
<?xml version="1.0" encoding="utf-8" ?> 
<ns0:Pat xmlns:ns0="http://Listen_Practice.schema_patient">

This is how Non-Uniform Sequential Convoy works..

We will implement next pattern in next blog, so till then "Keep Reading, Keep Improving"..

Friday, 31 January 2014

Handling SMTP Server on Windows 7

I want to share some thoughts which I have encountered while trying my hands on Biztalk 2010.
From last few days I was thinking to note my technical experiences somewhere and I found this as one of the best way to do so.

So to start with I will take a scenario which I am trying from last couple of days.
In this scenario, we wanted to implement error handling and if error occurs then we have to send the error content to the customer’s Email address (which we get dynamically).
To send mail from Biztalk we have to use SMTP server.

There are two mechanisms through which we can send email out from Biztalk.
1) For Dynamic Email routing – Handle the SMTP configuration in Orchestration.
2) For Static Email Routing – Handle the SMTP at send port.

To do this I am using the machine with following configuration:
1) Microsoft Visual Studio 2010.
2) Microsoft Biztalk Server 2010.
3) IIS6
4) Operating System – Windows 7 Enterprise.

Ok then let’s start with creating solution for the given problem.
First question comes in the picture is can I implement all this with the given development machine?
Ans -- "No"

Reasons (In the sequence I figured out those) :

IIS6 does not provide SMTP server at all
                So for this I have installed IIS7, which provides SMTP server.

Windows 7 does not ship with any SMTP Server capabilities.
                This means even though I have install SMTP on my machine (through IIS7), I can’t send out the mails from it. The reason is Windows don’t have actual Virtual  SMTP Server which is responsible to sending the mails.

Then I tried the same scenario on the server machine with following configuration:
1) Microsoft Visual Studio 2010.
2) Microsoft Biztalk Server 2010.
3) IIS6
4) Operating System – Windows Server 2007 Enterprise.

And guess what it worked on the above mentioned machine.. In the specified error scenario Mail was getting composed and got send to the recipient.

But still I have to figure out some solution by which I can do same through my development machine. So after a lot of surfing, forum QA, senior’s input I got the solution.

Someone suggest me to download smtp4dev. This acts a virtual smtp  server. It is supported on Windows XP/Vista/7. This provides us simulation of SMTP.
When I used this virtual SMTP server, It actually accepts the message sent from biztalk and sends the mail to recipient (simulation – Actually mail does not get sent).

Generated mail by this downloaded patch is as below

You can check more details/insights about this on below link

SO this is what I have learned in last couple of days. Even though this has not solved the question but it gives me some thought to bite my nails..

Thanks.. will be back very soon..