[WSO2 ESB] [4.8.1] How to Convert XML to JSON Array

Following API Demonstrate this functionality.

Try above api with a rest client with following sample requests.

1)Multiple Stocks.
XML request: JSON response : 2) Single Stock
XML request: JSON response (As an array): This is with following message formatter and builder
        <messageformatter class="org.apache.synapse.commons.json.JsonStreamFormatter" contenttype="application/json" />

        <messageformatter class="org.apache.synapse.commons.json.JsonStreamFormatter" contenttype="application/json" />

[Active-MQ] PortOffset

1) Changing transport ports. Edit /conf/activemq.xml
and change ports numbers in following configuration.

2) Changing web console port. Edit /conf/jetty.xml
and change following port in following config.

[Active-MQ] Setting up AMQ with MySQL database.

1) Create MySQL database.

2) Download activeMQ and extract it.
3) Edit /conf/activemq.xml
4) Add following configuration.

5) Copy mysql jdbc driver (mysql-connector-java-5.1.25-bin.jar) in the directory "activemq_home/lib/optional"

6) Start ActiveMQ server using $ ./activemq start ( To stop the server use ./activemq stop )

7) Log in to activeMQ management console using http://localhost:8161/ with admin:admin credentials.

8) Create a queue and send a message to the queue with persistence enabled.

9) You can see the message in database.

Packt offers its library subscription with an additional $150 worth of free content

Packt provides full online access to over 2000 books and videos to give users the knowledge they need, when they need it. From innovative new solutions and effective learning services to cutting edge guides on emerging technologies, Packt’s extensive library has got it covered.

For a limited time only, Packt is offering 5 free eBook or Video downloads in the first month of a new annual subscription – up to $150 worth of extra content. That’s in addition to one free download a month for the rest of the year.

This special PacktLib Plus deal expires on 4th November, 2014.

Check out the deal here http://bit.ly/1wN0Rq0

Book Review : WS-BPEL 2.0 Beginners Guide

WS-BPEL 2.0 Beginners Guide: As name suggest, This is a good reading for who wants to learn BPEL concepts from beginning. Authors start book with a very simple BPEL sample. They give details instructions from setting up IDE, creating schema and wsdl, etc. They finish their first sample giving instructions to how you can test your first BPEL process using a BPEL engine. That is why I recommend this book as a true beginner guide.

BPEL specifications is so long and even BPEL 2.0 primer is very hard to understand without prior knowledge on BPEL. But authors have structured book chapters from simple to complex BPEL concepts, so that beginners can easily understand. It contains chapters for fault handling, compensation, parallel processing  asynchronous invocation etc. Best part is authors have given samples for each section including guidance how you can do it in IDE.

This book contains special chapter for Humantask with samples. That adds a true value to this book.

Even BPEL 2.0 is published in 2007, (Nearly 7 years ago) it is usage is increasing in enterprise world. It is an widely used industrial standard for Business process management (BPM). I think knowing BPEL is an added advantage for a programmer, architect and students; and this is the best book to start with.

Have a look on this book by clicking this link.

ESB: Invoking secured backend - Part 3 - Username Token with BasicAuth

This post shows, how to invoke an UsernameToken secured backend ( Hosted in WSO2 AS ), using basic auth. For this we use POXSecurityHandler, (Which comes default with WSO2 Products) to convert the HTTP basic auth information into wsse:UsernameToken.

Setting up environment : 

Setup both WSO2 AS and WSO2 ESB as mentioned in previous post.


ESB Proxy

• Create a proxy called EchoUTBasicProxy with following content. 

Testing Scenario

• Enable Soap tracer on WSO2 AS.
• Enable wire log in WSO2 ESB.
• Invoke EchoUTBasicProxy  using SOAP UI. 

You can see, there is no username token in incoming message to backend. Instead you see basic auth header in outgoing message to backend from ESB.

ESB: Invoking secured backend - Part 2 - Username Token - Dynamic username

My previous post shows how to invoke an username token secured backend using an ESB proxy. But we used static value for the username ( tom ), which is hard coded in the policy file. So each request authenticated as tom at the backend service.

But some may wants to access backend service as different users. This post discusses how you can extend it to support dynamic user name in policy file.


Setting up environment : 

Setup both WSO2 AS and WSO2 ESB as mentioned in previous post. 


ClassMediator (ESB)

• In this scenario, we set username as a property in the ESB proxy. 
• To pass username into RampartConfiguration, we use custom class mediator called, SetUserMediator.
• This custom mediator, adds username into rampartConfigCallbackProperties map and set the map into Axis2MessageContext. So later we can access these properties from Rampart ConfigCallbackHandlers. 
• We have to use customMediator, since we can't set a Map using standard ESB mediators.  
• ( Maven Project is located here. )

Rampart ConfigCallbackHandler (ESB)

• Similar to PasswordCallback handler, Rampart provides Configuration Callback handler to dynamically load Rampart configuaraion to runtime. We use this to set username dynamically. 
• ( Maven Project is located here. )

Policy for UsernameToken  ( ESB )

• Create an ESB in-line xml local entry called "UTOverTransportDynamic.xml" with following content. 

Proxy Service (ESB)

• Create a proxy called EchoUTDynamicProxy with following content. 

Testing Scenario

• Enable Soap tracer on WSO2 AS.
• Invoke EchoUTDynamicProxy  using SOAP UI. 

You can see Username token in request message as follows.

ESB: Invoking secured backend - Part 1 - Username Token

Scenario 

1. Backend service is secured using Username token. 
2. Client invokes ESB proxy using http. ( no security between client and ESB) 
3. At the ESB, proxy adds username token to outgoing message and invokes secured backend.
4. ESB sends back echo service's response back to client. 

Setting up environment 

Backend ( WSO2 Application server 5.2.1)

1. Start WSO2 AS 5.2.1 using ( Unix: sh wso2server.sh / Windows: wso2server.bat ) 
2. Log in to management console. ( https://localhost:9443/carbon/ ) 
3. Create two user called tom and bom
• Goto Configure -> Users and Roles -> Users
• Create an user called tom with password "tompass". 
• Create another user called bob with password "bobpass"
• Assign both users to "admin" role.
4. Secure Echo service with Username token. 
• Goto Main -> Services -> List 
• Click on "echo" service. This will open up "Service Dashboard (echo)" page.
• Under "Quality of Service Configuration", Select "security".
• In "Security for the service" page, Select Enable security.
• Under Security scenarios, select "Username token"  ( First security policy) and click next. 
• In next page, select "admin" under user group. 
• Click Finish. 

ESB ( WSO2 ESB 4.8.1 )

1. Start WSO2 ESB with port offset =1 ( Unix: sh wso2server.sh -DportOffset=1 / Windows: wso2server.bat --DportOffset=1) 

Rampart configuration for UsernameToken  ( ESB )

• Create an ESB in-line xml local entry called "UTOverTransport.xml" with following content. 

Password callback Implementation

• Create a jar with following class, and drop it to /repository/components/lib/
• Then restart ESB server. 
• ( Maven Project is located here. )

Some useful References on Rampart password callback handler:  

1. http://wso2.com/library/3733/
2. http://wso2.com/library/240/

ESB Proxy

• Create a proxy called EchoUTProxy with following content. 

Testing Scenario

• Enable Soap tracer on WSO2 AS.
• Invoke EchoUTProxy  using SOAP UI. 

You can see Username token in request message as follows. 

Java4BPEL - Java Class extension activity for WSO2 BPS

java4bpel

java4bpel is an extension for Apache ODE and WSO2 BPS. java4bpel introduces new BPEL extension activity called invokeClass, which can be used to invoke a JAVA class.

Project home page

 http://hasithaa.github.io/java4bpel/

Project git location

https://github.com/hasithaa/java4bpel

User Guide 

https://github.com/hasithaa/java4bpel/wiki
 

Clone project from 

https://github.com/hasithaa/java4bpel.git

Tip: Find and Replace a string in multiple files at onces by avoiding LTS (Leaning toothpick syndrome)

Linux Command:

$ grep -lR "oldString" -r | xargs sed -i 's/oldString/newString/g'

How to avoid LST.

If your oldString or newString contains character "/" (the delimiter), then Delimiter collision occurs, which cause to LST.  To avoid that, select another delimiter such as # or ? in sed command.

examples:

sed -i 's#oldString#newString#g'
sed -i 's?oldString?newString?g'

WSO2 BPS 3.2.0 - Performance Tuning - Know When, Why and How

• Following instructions are only applied to WSO2 BPS 3.2.0 . 
• All configuration files are located under $WSO2BPS-3.2.0_HOME/repository/conf/ directory

JDBC connections.

When ?

• When your application requires high throughput  

Why ?

• BPS server has two engines; Apache ODE BPEL processor and HumanTask engine. These two engines are tightly coupled with database layer. They persist every instance data into database. Thus to function BPS properly, you need to allocate enough database connections for BPS datasource.
• Both these engine share same BPS datasource and db connections. Thus generally we are recommending to allocate db connections 50-50 manner for a BPEL + HumanTask application.
• For example if you have total 100 db connections, for a BPEL + HumanTask application, you can use upto 50 db connections for ODE engine and leave the rest of the db connections for HumanTask operations.
• If you have only BPEL usecase, you can allocate many db connections ( see next topic) for ODE engine. 
• Also note that, even you have allocated higher number of db connections for BPS datasource, performance may not increase as excepted. One reason would be, there are no enough db sessions from database side. If that is the case, you need to increase number of db session from database side.  

How ?

• Configure BPS data source using datasources.properties file.
• eg: see highlighted section. 

 ...
 synapse.datasources.bpsds.validationQuery=SELECT 1 FROM DUAL
 synapse.datasources.bpsds.dsName=bpsds
 synapse.datasources.bpsds.maxActive=100
 synapse.datasources.bpsds.maxIdle=20
 synapse.datasources.bpsds.maxWait=10000

 

ODE Scheduler threads

When ?

•  When your application requires high throughput  

Why ?

• In ODE engine, every scheduler thread is associated with a db connection. So the rule of thumb is, number of ODE scheduler threads should be less than or equal to number of db connections allocated for ODE engine. Otherwise some threads may not work properly, because they can't acquire a db connections to work. 
• For example In BPEL + HumanTask scenario, if you have total 100 db connections, your can allocate 50 threads for ODE scheduler. This will grantee that at a given time, only 50 db connections are acquired by ODE engine.    

How ?

• Configure this via bps.xml.
• eg: 

    <tns:odeschedulerthreadpoolsize>50</tns:odeschedulerthreadpoolsize>

HTTP connections

When ?

• When your have lot of service invocations.(external or internal) 

Why ?

• When your BPEL processes do service invocations, they use http connections. By default this http connections are limited. Because of this, thread have to waits for http connections. 
• To avoid this, you can increase this value using Multithreaded Http Connection Manager Configuration. 
• If your processes do lot of service invocation to localhost ( or particular host), then it is required to increase maxConnectionsPerHost configuration as well.

How ?

•  enable this configuration using bps.xml

    <tns:MultithreadedHttpConnectionManagerConfig>
        <tns:maxconnectionsperhost value="350">
        <tns:maxtotalconnections value="400">
    </tns:maxtotalconnections>

 

TimeOuts 

When ?

•  When partner services take more time (slow) to response.

Why ?

• When partner services are slow or take more time to response, callee BPEL process's invoke activity fails due to message exchange timeout. By increasing time will avoid these kind of failures. 
• Also note that, slow partner services will slow entire BPEL process. This will cause to timeout the client application. ( callee of the BPEL process.). Thus it is required increase timeout interval for client application.      

How ?

• Via bps.xml and axis2.xml

Read for more information.
http://nandikajayawardana.blogspot.com/2012/10/how-to-increase-external-invocation.html

 

HumanTask Caching 

When ?

• When you have to deal with large user store.

Why ?

• HumanTasks are tightly coupled with users and user roles/groups. Because of this, BPS does lot of user store lookups for HumanTask operations. These user store calls can take considerable amount of time, if user store is large or located remotely. This causes to degrade performance of the entire HumanTask engine. Caching user and role lookup data at BPS side will reduce those remote user store calls and improve overall HumanTask engine's performance.  

How ?

• Enable HumanTask caching in humantask.xml
• eg: 

    <cacheconfiguration>
        <enablecaching>true</enablecaching>
    </cacheconfiguration>

 

Number of HumanTask Scheduler threads.

When ?

• When you are not using humantask deadline/escalation.

Why ?

• HumanTask deadline and escalation are scheduled tasks, which are executed by HumanTask scheduler. By default 50 threads are allocated for HumanTask scheduler. If you are not using deadline/escalations, you can configure this value to lower value such as 5. This will utilize idle threads in BPS server.
• Note that, you can't set this to 0, because humantask engine has several internal scheduled tasks to run.

How ?

• Configure this value in humantask.xml,
• eg:

    <schedulerconfig>
        <maxthreadpoolsize>5</maxthreadpoolsize>
    </schedulerconfig>

An Async BPEL Process and A Human task ?

In Async process ( in other words a long running process ), after the invocation of a partner process or service, the BPEL process continues to carry on with its execution process while that partner service completes performing its operation. This partner service may take couple of seconds, minutes, hours or even days to complete. The Async BPEL process then receives the result of the partner service via a callback service, when the partner service is completed.

HumanTask (Web service) is one type of such a long running partner service which returns the task result when user completes the task. In this case, BPEL use B4P extension, which stops the BPEL execution flow which contains the B4P extension activity and waits till HumanTask output to come.

Change maven local repo in one command - Ubuntu/Linux

This will be useful for the developers who works with multiple maven local repositories.

Requirement:  set M2_HOME environment variable, before you start. 

Configuration: 
Add following  bash function to .bashrc in your home directory. Change M2_LOCATION if you need.



Then reload the .bashrc using command,

$ source ~/.bashrc

How to use: 

Type changeM2 in the terminal and give the name of the m2 repo (folder) you wish to change.

Recovering BPEL Activity Failures - WSO2 BPS

Read more about ODE activity failure and recovery from.

http://ode.apache.org/extensions/activity-failure-and-recovery.html

 

Method 1: Via WSO2 BPS Management Console 

In WSO2 BPS, BPEL failed activity recovery can be done using WSO2 BPS Management console.

To do this, it is required to enable ODE activityLifecycle events for a BPEL process. To do that modify process-event configuration in deploy.xml as follow.
(refer http://ode.apache.org/ode-execution-events.html for to see how you can enable ODE events for a BPEL process.)

Users can view activity failures for a BPEL process instance in the instance view page. 

1) Goto Instances -> click on of the instance ID to open the instance view for that instance.

 2) In the instance view, you can find failed activity/activities under activity informations. (See image). Also you can retry/cancel (ignore a failed activity) a failed activity using two buttons listed under Action column.

Method 2 - Via InstanceManagementService admin API

Also users can recover failed activities by using BPS InstanceManagementService. Unfortunately there is no UI functionality for this when ode events are disabled. (BPS 3.2.0 & older)


These are the steps, to retry activities using InstanceManagementService admin service.

1) Execute following SQL query on BPS database.

This will return the failed activities with corresponding BPEL process instance id. You will require following information to retry a failed activity.

• INSTANCE_ID
• ACTIVITY_ID

2) Set HideAdminServiceWSDLs configuration to false in /repository/conf/carbon.xml file.

eg:

3) Then start the BPS server.

4) Now create a SOAP-UI project using https://localhost:9443/services/InstanceManagementService?wsdl

5) Create a new request under recoverActivity operation. A sample request will be like this.


6) Then Authenticate soap-ui request by configuring authentication and security related settings.
For Basic Auth select Authorization type as preemptive, and give admin user name and password.

7) Then for each failed activity (results in step 1), send a recoverActivity request. Use "retry" action to retry activity, and use "cancel" action to cancel the activity retry and continue instance execution.

Running HumanTask Cleanup Job - WSO2 BPS

HumanTask engine allows you to configure periodic cleanup tasks from the WSO2 BPS persistence storage based on task status. To enable Task clean up task, uncomment TaskCleanupConfig element in HumanTask.xml


HumanTask engine uses quartz-scheduler for cron jobs. Refer [1] to declare cron expressions format. In above example, Cron scheduler will trigger TaskCleanup task every Four Hours, to remove COMPLETED, OBSOLETE, EXITED from the database.

[1] -http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger

Generating a random unique number in a SOAP UI request

In the request use,

${=System.currentTimeMillis() + ((int)(Math.random()*10000))}

example :  

Note : Here I am generating this number by adding currant milliseconds as a prefix. So this will generate almost unique number.


Update: 20th June 2014. 

Another simple way to do this. 

${=java.util.UUID.randomUUID()}