WebServices dot org

Making the Common Thing Easy – Part II

What is the Common Thing?

In Part I , I talked about the complexity of the landscape for Web services, the fact that the standards were proliferating faster than they could be absorbed, and the fact that we needed more Web services solutions that focus on supplying the common components that users require when they embark on an integration project.

Today, I would like to take the time to present the common things that I have found most customers need in such a project:

1. Expose controlled services over Web-based protocols

2. Encapsulate data and logic from multiple systems as a single service

Expose Controlled Services Over Web-based Protocols

This is quite a mouthful, so many customers will tell you they just want to "Web-enable their applications." However, the word "Service" is a powerful concept when it comes to integration. The user does not want to expose existing systems; rather they are looking for ways to expose something that is an aggregate of what one or more of their back-end systems does, along with some tasks that had previously been performed manually. In general, the "thing" (i.e. service) being exposed is not something that already exists as a whole; rather it is something new that leverages the old and creates new value for both the enterprise exposing it, and that enterprise’s customers and trading partners.

Take the example of a manufacturer that fulfilled massive orders by farming them out to multiple factories around the world. They were faced with the problem of trying to provide order status information to customers in a timely manner when none of their backend systems at the different factories were compatible with each other or accessible from the head office. As a result, customer service representatives (CSR) had to pull up the details (on the head office system) that indicated where the components of the order were being manufactured. Then they had to phone or email all the factories for status information. Due to time-zones and the inefficiencies of a highly manual process, it sometimes took days to determine status of a single order and the cost of this process was prohibitive. The manufacturer quickly saw that it needed to provide an automated system that would allow their CSR’s to query and access order status information via the Web to provide a better quality of service to their customers and reduce costs. The data was all there, albeit spread across continents in multiple disparate formats – it just needed to be aggregated and some "glue" (workflow) was required to consolidate the data and present it in a concise format. They created a new service (on their intranet) that leveraged their existing backend systems and automated the CSR’s tasks, increasing productivity by 70%.

There will often be cases where a service may need to suspend activity until triggered to resume by some event. For example, in order to conform to a business rule, a sales manager may have to manually approve an increase in a customer’s credit limit. In order to do this, you need a mechanism that allows you to support long-running transactions. Introducing business rules like this requires a flexible workflow engine.

Another consideration is how these services will be accessed. You may want to be able to access the same services using a form post from a Web page, or a post of an XML document or as a SOAP method.

The common factor here is that you need the ability to send/receive messages over HTTP; you probably will need HTTP over a secure socket, and you may need client certificate support for security purposes.

Next in line after HTTP is the ability to access data that may be dropped into a folder by another application – e.g. FTP, an email server, a fax/OCR server, or some other system.

And the need to process non-XML formats, like relational databases, comma delimited files, EDI, MS Office documents etc and transform these to a canonical XML format is also frequently required.

SOAP Adoption Slow

Despite the obvious value of SOAP and SOA, I’ve found, much to my surprise, that many of our customers only want to explore the complexities (and uncertainties) of SOAP technology if they are required to support it by a partner or customer. In the manufacturing case I mentioned above, shortly after exposing the "order status" service to their customer service representatives, the manufacturer made this information available as a secure Web service to some of their larger customers so they could access their status information directly. The same business logic is now accessible through Web services and through a form post from a Web page.

Our customers are nervous of jumping on the SOAP bandwagon for a number of reasons, the main ones boiling down to:

• Security – a perception that SOAP may be less secure than a simpler solution.

• Confusion over standards – it isn’t clear which standards are needed and which are ready for prime-time.

• SOAP is more complex than, say XML over HTTP.

Other than the last point, it is our opinion that the first two tend to evaporate as reasons not to proceed as users become more comfortable with the slice of SOAP that they really need to implement their solutions.

In fact, we recommend that these users start with the simpler solution, but show them how we can take, for instance, an XML over HTTP solution and turn it into a SOAP method, complete with WSDL, with a single point-and-click step. Users who see this type of an evolution are generally comforted. They get their simplicity now, but get to migrate to SOAP as the complexity and size of their systems increases.

Encapsulating Data and Logic from Multiple Systems

This is the second thing that a user commonly needs to do. Very few companies have a single back-end line of business system. Invariably, you find yourself pulling in data from multiple sources in multiple formats.

Most databases can be accessed through standards-based mechanisms, such as JDBC, ODBC, OLE DB and others. There is no excuse for having to hand-code this type of access – integration systems should come with these interfaces, out of the box. There are also a number of common formats such as comma delimited, EDI and others that should be accessible without having to write custom code. However in the case of very old legacy systems that can only be accessed through proprietary interfaces, you may need to connect to them through Java, COM, .NET or C++ – your integration server should facilitate this.

You should also be able to easily access other Web services, servers, and Web-based resources, especially as more and more of your customers, suppliers or partners publish these.

Legacy systems can often be slow, so the ability to access them concurrently is necessary in time critical applications. For example, if you need to access three legacy systems and two of them take more than 30 seconds to give you a connection, you might risk timing out a browser-based client connection if you access them serially. You need parallel execution or multi-threading capabilities in your server.

As your applications reach out across both the Web and within your enterprise, your application needs to be able to react to unexpected errors such as legacy systems going down or remote servers being temporarily unavailable. You need a mechanism to put a transaction on hold, then resume it when the service is again available, or fall-back to a disconnected mode of operation.

General Considerations

One of the strong benefits of XML, that has caused developers to so readily adopt it, is the visually intuitive nature of the nested structures that are possible with it. The ability to see an invoice as a whole, yet programmatically access it as an object, makes it a compelling way to model, manipulate and pass data between systems. One could argue that XML has already become the Lingua Franca of developers. A development environment that transforms all data into XML, then lets you visually manipulate and reference this data is a powerful paradigm for developers to work with.

Whenever possible, you should leverage standards such as SQL, JDBC/ODBC, XSLT, XPath without having to write complex code in Java or C++ . Your development environment should provide builders and wizards that allow you to point and click to build applications that use these technologies.

Conclusions

When I apply the Pareto’s Principle (sometimes called the 80-20 rule - http://management.about.com/cs/generalmanagement/a/Pareto081202.htm ) to Web services, I find that there is a fairly common set of functionality that most companies will need in an integration product:

• Communicate over HTTP

• Support for secure HTTP and client certificates

• Directory Polling

• Transform data formats to XML

• Transform between different XML formats

• Workflow to control integration-level business logic

• Easy access to common structured and unstructured data sources

• Easy access to custom code for non-standard applications

• Easy access to other Web-based resources

• Easy way to extend access to less common sources

• Concurrent access to resource-bound legacy applications and databases

• Easy advanced error handling

All combined in an easy-to-use IDE that generates code for all the common tasks you need to perform, but still allows you to create your own code for the less common, more complex tasks you want to implement.