More and more analysts, project and product managers come to the IT industry from marketing, consulting, sales, and other non-technical industries, or even without any work experience. I wasn’t an exception. There is a problem that in the first years you get a lot of technical terms and concepts: you try to figure them out, but you don’t understand to what level of detail you need to dig for each of them. Do you need to understand what Web Service Definition Language (WSDL) means and what type of asynchronous request handling is needed, or is it enough to know that a web service just has a request and a response?
I aim to solve this problem for one of the most common features — web service usage for exchanging data, performing targeted actions in other systems, and other integrations.
You can easily find many posts on what web services are, why they are needed, and what are the interesting implementation cases. I did not find anything comprehensive about how to interact with them in practice, especially in complex corporate systems. Experience has shown that the devil is in the details — most of those in the article were obtained from my real cases.
Imagine that you need the following feature: depending on the response from a web service that guesses a person’s salary, display a button of a different color to the user. For example, if the salary is up to $20,000, the button is red. It’s yellow if the salary is from $20,000 to $100,000, and green if it is more than $100,000. Seems like nothing could be easier! However, to describe this task for engineers and figure out everything yourself, it’s important to pay attention to 15 different questions, and if the feature is insanely important, then to 8 others.
Level 1. Must-haves:
Find or ask for pages with a web service description and tasks where someone described the integration with it
Check the presence of all the points below, so as not to annoy the person by asking him again :)
Prod and test (QA) URLs - endpoints
The web service is available at these URLs. The test endpoint will be useful for running the service without making changes or loading the prod. Endpoints are very similar to regular website URLs, often they will contain the word “api”. You can try opening them in a browser, and sometimes the response can be displayed.
Authentication
How do we log in to the web service to get the needed data? There are many different options: using an API key, token, login and password in the request body, or some other authentication methods.
Get WSDL if it’s a SOAP web service
The two most popular types of web services are SOAP and REST, which is usually specified at the very top of the service description page. REST can theoretically change the response format at any time, while SOAP responds only in the format that is in the XML Schema Definition (XSD). These response schema and endpoints are contained in the WSDL file that you pass with the request. It means for SOAP services you can handle without endpoints if you already have a WSDL file. Accordingly, WSDLs are different in the prod and test environments, because they contain different endpoints. Moreover, for SOAP, you have to make requests in some programs, often a free SOAP UI is used. SOAP uses .XML files for requests and responses, and REST uses .JSON, .XML, .TXT.
To get examples of requests
What shall you pass to the web service to get the needed data for different cases? Do you have all these parameters?
To get examples of relevant successful responses
For REST services, Swagger can be used to see an example of what a successful response may look like. Once you locate the Swagger URL, there is a nice interface where you can send a request, get an answer, and log in to see what services are adjacent to yours on PROD and QA.
What do we do while waiting for a response from the web service?
It’s especially relevant if we are dealing with services with a long response time or one that makes live users wait, as in my example with a multi-colored button. The user may not understand why nothing is happening but maybe it’s a good time to think about adding a loading screen.
Timeout (maximum waiting time for a response) and reaction to it
After the timeout, we stop waiting for a response and perform certain actions. For example, send a request to another service, or handle a timeout error. Why do we stop waiting for an answer? Either because we can’t wait any longer and we have to react now, or if there’s nowhere to rush, but we are fairly sure that the service will not answer. It will be nice if you can get the distribution of the service response time so you can set a timeout, for example, as the 99th percentile. But more often it will be already described or you can ask the service developers.
Characteristics of fields: field type, required or optional, and string length
- Field type. We must be sure that we pass fields in the request in the same format that the service expects to receive. The most common cases here: we pass a number as a string of text (“2007” instead of 2007) and the different date and time formats (“2007–01–01” instead of DateTime “2007–01–01 11:00:00”). When we receive a response from the service, it is also better to make sure that the data from it comes in the same type that we expect. Otherwise, we’ll need to convert it into the desired type.
- Required and optional fields. There are fields without which the request will not run and there are fields that can be omitted sometimes. If you know that the field is optional, but you need to pass it according to the received request example, then you can discuss with the web service developers what will happen if you do not pass it. Perhaps it is possible and you don’t need to get the field from somewhere to pass it in the request. For SOAP, you can see the required fields in the WSDL or in the sample request (look for the green text <! — Optional: ->). For REST, this can be viewed in Swagger, denoted by a red asterisk.
- Text string length. It is better to check that the text strings we pass in the request will not be truncated. For SOAP, it is possible to look at WSDL for the minLength / maxLength fields. For REST, the lengths of the strings are defined on the service side and it is usually assumed that the strings will not be truncated. However, if you are doing an important feature or passing a string of a paragraph length in a field that has to be short (Name, Phone, Address), it is better to check this up with service developers.
Action of your system in case of errors received from the service (exception flow)
You need to think about how to respond to errors from the service. For example, you can make it so: the service is requested every 20 minutes, up to 10 times in total, then we stop and display errors.
Action of your system if important fields in the response come empty / with incorrect values
In practice this happens quite often, for example, the service starts returning unvalidated data from new sources.
Run and check the integration on the QA / Prod environment, if possible Please note there are often mocks or stubs on the QA environment when the service returns the same hard coded value in the response instead of normal handling of the request. If the web service has to do something on another system, first check that this target action has been performed, then show the requests to the owner of the service. In practice, problems are often encountered at this stage.
Load
Discuss with web service developers the frequency of requests, the change in the load during the day/week, as well as its potential changes in the future.
Shall you log the request and response
There are many advantages: you can track bugs, fraud, and/or see how many service requests were in total.
Service response caching
If it’s possible that a service call with the same request parameters will be repeated and the response will not change, then this response can be saved to not call the service again. It is especially important if the service call is not free and fast.
Level 2. If your feature must be properly integrated, check these points:
Service availability depending on time
If a service has “rush hours” during which it becomes less available, you should take this into account when planning the integration. Warn users in advance, and carefully consider the action in case of timeouts and errors in responses.
Alerts
If a successful response from the service is critical, you can set up alerts for errors, availability, or a sharp change in the average value in the response or share of certain text values.
Is it sensible to implement the response processing logic on the side of the web service?
Let’s look at my example with a colored button depending on the salary received in the response. You can make this logic in your system (often the simplest option); however, if the boundaries of $20,000 and $100,000 are supposed to be changed frequently and the red/yellow/green classification will be used by other systems, it makes sense to modify the service and send the color of the button in the response. Otherwise, desynchronization can easily take place and people with the same salary will have different buttons in different interfaces.
Does the structure of the response change depending on the request?
For example, the request contains the “strategy” parameter and its different values lead to different parameters in the response. Our system must expect such behavior.
Response processing logic
If you are changing an existing integration with a web service, do not forget that there may be logic on your system that needs to be updated. For example, you read only certain fields from a response so you need to add new ones.
Level 3. Most important project of the year, no bugs possible!
Examples and meanings of exceptions
On some occasions, different scripts and alerts for various errors will be required. This is how errors are described for Google Calendar API.
Not performing the target action while returning successful response codes There are cases when the service returns 200 OK when the target action is not performed. For example, the web service sends a request to another service after our request but does not wait for its response and immediately sends us 200 OK. The target action might not happen and there should be a response with the 201 Created code.
Synchronous / Asynchronous request handling
When a request is handled synchronously, it is processed immediately upon receipt. With asynchronous handling, requests are queued on the service side and the target action is not immediately performed. From service owners or its users, you can find out for how long the request stays in the queue and how much more time it takes to process the request. Success codes 201 and 202 mostly indicate asynchronous handling. If you received one of them, it is better to check that the target action will be finally fulfilled.
Thanks for your attention! Please feel free to share your cases and feedback in the comments!
