Note: NetTalk Web Services requires xFiles

• Introduction
• Example
• Documentation
• Method Data
  • Fields In Returning VIEWs
• Tables
• Advanced: Extended Attributes  NEW
• Advanced: Extended Pictures     NEW
• Advanced: Multi-Level Views      NEW
• Field Settings
• Nesting
• Advanced: Nested Queues          NEW
• Nested Queues Template Settings
• Wizard
• Calling Methods from a Client Program
• Your code in the Method
• Results
• Errors
• SOAP versus REST
• Return Format Rules
• Overriding XML or JSON methods
  • Formatting Fields in the Response
• Field Prefixes
• Authentication
  • Basic Authentication
  • Digest Authentication
  • Sessions

There are a variety of common features that programs can offer that collectively are known as Web Services. NetTalk 8 introduced two new Procedure templates, called NetWebService and NetWebServiceMethod to assist with creating web services.

A NetWebService is the name for a collection of service methods.

In that application tree you can have multiple NetWebServiceMethods for a single NetWebService. The NetWebService is just a container, and does not include much more than just a list of the methods.

Your application can contain multiple NetWebService procedures (each one with one or more methods.)

All methods support a variety of encodings, for input and for output. Possible input encodings are XML (using xFiles), JSON (using jFiles) or Form-URL-Encoded. Possible output encodings are XML or JSON.

Supported Web Service standards include WSDL, SOAP and REST.

NetTalk 14

In NetTalk 7 and earlier a hand-coded approach to web services was demonstrated in an example called SOAPServer (42). This example is included for reference purposes.

However a new example, WebService (77) is provided from NetTalk 8 and as it makes use of the new templates, is a better example for developing future WebServices.

The service, and each method is self documenting. From any browser you can see the documentation for the service and for each method. The documentation includes as much information as possible so that other developers can easily make use of your service.

To see the documentation in your browser navigate to one of the following links;

www.whatever.com/servicename
www.whatever.com/servicename?help
www.whatever.com/methodname?help
www.whatever.com/servicename?methodname

A WSDL file is a formal, computer-readable way of documenting a web service. Using this file, other programmers can write programs that make use of your service, and the existence of this file makes their lives much easier.

NetTalk automatically (and dynamically) generates this file for you when it is requested. A WSDL file for your service can be retrieved by using one of these links;

www.whatever.com/servicename?wsdl or
www.whatever.com/methodname?wsdl

The method template has a place where you can set the incoming data fields, and the returning data fields. These fields need to be declared in your application either as tables, table fields, global data or local data.

Parameters

Returns

Fields in Returning VIEWs

Probably the most common use for a WebServiceMethod is to read and/or write database records. Since this is a common use-case, the templates are aware of this situation, and creating these sorts of methods is very straightforward.

1. Create a NetWebServiceMethod procedure. Add it to a NetWebService as normal.
2. Create an incoming parameter of type TABLE, and select a Table. Tick on the database actions you want to allow (Inserts, Updates etc)
3. If the method allows the user to read the table then create a return value, of type VIEW. Give the view a name, tick on GenerateViewStructure and select the Table to view. If you leave the Fields list blank then all the fields from the table will be exported. Alternatively you can select a subset of fields to send to the client. Also be sure to enter an appropriate filter here.

The service will look for an additional parameter, called TableName_Action. This parameter is a string and should contain one of insert, update, get or delete.

NetTalk 14 makes use of jFiles 3 (and later), xFiles 4 (and later) and CapeSoft Reflection in order to create a consistent API interface. These tools allow data structures (Queues, Groups, Tables) to self-document, and they make things like nested structures possible.

If you are unfamiliar with them then please read the background on ClarionHub, as well as the Reflection Documentation which goes into them in some detail.

You may also want to review the jFiles and xFiles documentation for notes on the supported attributes. A key to unlocking the power of API's lies in being familir with Extended Attributes.

One of the extended attributes that you can make use of is a formatting picture, which can be used to format outgoing data, and deformat incoming data. Clarion has a solid set of pictures which allow for a wide range of formatting options. In addition to the normal Clarion pictures, NetTalk also supports Extended Pictures, as described (and provided) by StringTheory.

Clarion allows complex VIEW's (aka Client-Side-Views) to be created in your code. You can hand-code VIEW declarations in the Data Section embed point, and then use the Label of this View as a Return Value. (If you hand-code the declaration then you will leave the Geneate View Structure OFF.)

Here's an example of a complex, hand-coded VIEW for exporting Invoices;

InvoiceView  View(Invoice) 
               Project(INV:ID)   
               Project(INV:Date)
               Join(CUS:Key,INV:Customer)  
                 Project(CUS:Name)
                 Project(CUS:Address)
                   Project(CUS:Line1)
                   Project(CUS:Line2)
                   Project(CUS:PostalCode)
                 Project(CUS:Notes)
               End
               Join(LIN:InvKey,INV:ID)
                 Project(LIN:Price)
                 Join(PRO:Key,LIN:Product)
                   Project(PRO:Name)
                   Project(PRO:CostPrice)
                 End
               End
             End  

This view is rooted in the Invoice table, but includes product and line-item information. It also includes Product information for each line item.

Apart from the View declaration, no other hand-code is required. However there are template settings, specific to Views (generated or hand-coded) which you will want to set.

On the template settings, go to the Returns Tab, and add the VIEW there. Notice that when you set the Return Type to VIEW then a Field Settings and a Nesting Tab appear.

Field Settings

Nesting

Both JSON and XML are nested text structures. They allow levels of data, with one set of data inside another. Clarion uses QUEUEs for lists of data, and indirectly allows for Queues-Inside-Queues - by allowing Queue pointers to be used inside queue structures.

jFiles 3 and xFiles 4 have both been updated to make using nested queues easy, and this functionality therefore flows into NetTalk. Nested queues (as well as Queues-In-Groups) can therefore be used as Parameters or Return Values.

While it's possible to declare these data structures in the data pad, for this documentation they will be created in embed code. It's possible to copy them from the embed code to the Data pad if you prefer it there. Consider the following declaration;

CompanyQueue         Queue ! Using 'Companies' as Field Boundary for parameter and return Template settings.
Registered             String(100),name('Registered')
Date                   Long,name('Date | date | @d6')
Online                 byte,name('Online | boolean')
Age                    long,name('Age')
Branches               &BranchQueueType,name('Branches | queue | rowname(Branch)')
                     end

BranchQueueType      Queue,type
Name                   String(100),name('Name')
Departments            &DepartmentQueueType,name('Departments | queue | rowname(Department)')
StaffLevel             long,name('StaffLevel')
Manager                string(100),name('Manager')
                     End             

DepartmentQueueType  Queue,Type
Name                   string(100),name('Name')
Head                   String(100),name('Head')
StaffLevel             Long,name('StaffLevel')
                     End

The important think to note here is that there is a primary queue structure (CompanyQueue) which contains a reference variable (Branches) which
is a reference to an internal queue (of the BranchQueueType.) The BranchQueueType in turn has a field (Departments) which is of the DepartmentQueueType.

NetTalk is able to work with these nested structures because of the extended attributes. Some interesting attributes to note are on the reference-to-a-queue lines. You'll see there the attribute queue is used, as well as the attribute rowname. The rowname attribute is used by XML as a wrapper around each row. Setting this to blank suppresses the row wrapper. JSON doesn't use this field as it has no row-wrapper anyway.

You can also see the use of the date picture on the Date field - this is used to format the output, and deformat the input.

Now for the tricky bit;

Nested Queues Template Settings

The wizard can now generate a web service and a web service method for each table that you select.

The above calls are typically made from a browser, and the goal is to determine what the service can do, and how to use it. To actually make use of the service though you need to call one of the methods.

A service is a collection of one or more methods. You can create your own methods using the NetWebServiceMethod procedure type. You can add as many methods as you like to a service, and your application can contain as many services (ie collections of methods) as you like.

Methods have a name, a list of incoming parameters, and a list of returned data. While a Clarion procedure only returns a single piece of data, a web method can return any amount of data, including GROUP and QUEUE structures.

The NetWebServiceMethod procedure type supports a number of calling techniques;

1. Normal HTTP GET. The URL is of the form /service/method. Incoming parameters are passed as part of the URL or as cookies. The returned result is a simple XML structure. For example;
   
   GET /School/GetSchoolTeacher?FromTeacherID=value&ToTeacherId=value&Authenticate=value
   
   It's also possible to call the method using the PUT or DELETE commands instead of GET. For example;
   
   PUT /School/GetSchoolTeacher?TeacherID=value&Teacher
   Name=value&
   
   In the method itself you can determine which Verb was used by checking p_web.RequestMethodType. Typical possible values are one of NetWebServer_GET, NetWebServer_PUT, NetWebServer_POST and NetWebServer_DELETE.
   
2. Normal HTTP POST. The URL is of the form /service/method. Incoming parameters are passed as Post Data, but in addition to the URL and Cookies, data can be passed in as plain "POST Data". For example;
   
   POST /School/GetSchoolTeacher HTTP/1.1
   Content-Type: application/x-www-form-urlencoded
   Content-Length: length
   
   FromTeacherID=value&ToTeacherId=value&Authenticate=value
   
   
3. SOAP 1.1. The URL is of the form /service. In this case a POST is used, but the incoming Post Data is formatted as XML, and (optionally) wrapped in a SOAP envelope. A HTTP Header called SOAPAction: is also set to the method name.
   
   POST /School HTTP/1.1
   Host: somehost
   Content-Type: text/xml
   Content-Length: length
   SOAPAction: /GetSchoolTeacher
   
   <?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
     <soap:Body>
       <GetSchoolTeacher xmlns="https://capesoft.com">
         <FromTeacherID>value</FromTeacherID>
         <ToTeacherId>value</ToTeacherId>
         <Authenticate>
           <User>value</User>
           <Password>value</Password>
         </Authenticate>
         <AsAtDate>value</AsAtDate>
         <AsAtTime>value</AsAtTime>
       </GetSchoolTeacher>
     </soap:Body>
   </soap:Envelope>
   
   
4. SOAP 1.2. The URL is of the form /service. the request is very similar to a SOAP 1.1 request, although the SOAPAction header is not included. the content-type of the SOAP 1.1 request is text/xml whereas the content-type for a SOAP 1.2 request is application/soap+xml. For example;
   
   POST /School HTTP/1.1
   Host: somehost
   Content-Type: application/soap+xml
   Content-Length: length
   
   <?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
     <soap:Body>
       <GetSchoolTeacher xmlns="https://capesoft.com">
         <FromTeacherID>value</FromTeacherID>
         <ToTeacherId>value</ToTeacherId>
         <Authenticate>
           <User>value</User>
           <Password>value</Password>
         </Authenticate>
         <AsAtDate>value</AsAtDate>
         <AsAtTime>value</AsAtTime>
       </GetSchoolTeacher>
     </soap:Body>
   </soap:Envelope>
   

CORS

Services can include automatically generated methods (more on that in a moment) but a NetWebServiceMethod procedure doesn't actually do anything unless you add the necessary code.

In this sense a method can do "anything" that you code it to do. The template will parse the incoming request, and place it in the Parameter data structures. It will also format the return structures into XML or JSON, and return them to the caller. Your job is to write the code that populates the return structures.

You add your code to a routine called ServiceMethod. By the time this routine is called the parameter structures have been primed. Once this code is completed another (generated for you) routine will turn the result data structures into XML (or JSON) - you don't need to worry about that, you just write the code to populate the return structures with the correct value. Your code should not need to care about the technique used to call the method.

You add normal Clarion code here, opening tables, reading data and performing calculations, just like you would in any Clarion procedure.

You have access to the SessionQueue here for fetching and storing information using the normal p_web.GetSessionValue and p_web.SetSessionValue methods. However the incoming request will be bound to the session ONLY if the sessionID cookie is set in the request. Since the web client accessing this method is usually not a browser, the cookie may not automatically be set.

Aside: If you are using a Clarion program as a client, and you are using the NetWebClient class, then you can set the NetWebClient.OptionAutoCookie property so that multiple requests will preserve the cookies, and hence the session ID. If you are using a different tool for the client, then you will need to research that tool to determine how to send the cookie.

If your code does not interact with the session queue, then you don't need to worry about this.

Your code can (and should) generate Errors when things go wrong. See the next section for more on adding errors to your code.

NetTalk contains a generic Results queue (p_web.ServiceResultQueue) which is used to pass information back to the caller. This queue is populated when the caller is adding, editing or removing records in the database, using the generated template code.

The Queue contains four fields;

• Action
• TableName
• RecordID
• Description

You can add your own information to the queue if you wish. The method to call is

p_web.AddServiceResult (action, tablename, recordId, description)

A method may fail for any number of reasons. Services should return detailed, meaningful error information to the client wherever possible. To this end NetTalk Service Methods include a built-in, automatic, always consistent queue of errors.

Errors can be generated automatically by the validation template settings, or they can be generated in your method code. To Generate a method simply call;

p_web.AddServiceError (Number, Position, RecordID, Description, Recommendation)

You are free to pass whatever you like to the AddServiceError method, but the more information passed to the client the better.

If any errors are added to the queue in this way, then only the errors list will be returned to the caller. None of the other return values will be included in the reply.

The ServiceErrorQueue is automatically added as a possible reply to the generated WSDL file for all methods in the service.

As you can see from the above your method will happily accept incoming requests formatted as a SOAP XML packet. It is equally happy though to receive the request as a simple GET, PUT, POST or DELETE command. This is sometimes known as a REST request.

Your embed code does not change greatly between creating a SOAP service, or a REST service. REST is basically the same as SOAP without all the SOAP wrapping. Typically a REST client will also use the different HTTP verbs (GET, POST, PUT and DELETE) to match up to regular file activities. A SOAP service on the other hand will typically use a parameter to determine the file action to take.

The template generated code can easily handle both, for example in your embed code you might have some code like this;

(In this example a parameter called ACTION is assumed. If the parameter is sent, then loc:act is set from that, if the parameter is omitted, or set to 0, then the HTTP verb is checked and the loc:act based on that.)

loc:act = action
if loc:act = 0
    case p_web.RequestMethodType
    of 'GET'
        loc:act = Net:ViewRecord
    of 'DELETE'
        loc:act = Net:DeleteRecord
    of 'PUT'
        loc:act = Net:ChangeRecord
    of 'POST'
        loc:act = Net:InsertRecord
    end
end
   
To be a truly RESTful method you should not need to access the SessionQueue in order for the method to work. Ideally the client should pass you all the information you need in order for the method to work.

NetTalk API's can return results in either XML or JSON format. NetTalk applies a number of rules to decide which format to use. As soon as a rule is met then the decision is made, and no further rules are tested. The rules are as follows;

1. If JSON is not supported by the method then return XML.
2. If XML is not supported by the method then return JSON.
3. If the incoming ACCEPT header specifies XML then return XML.
4. If the incoming ACCEPT header specifies JSON then return JSON.
5. If the incoming request type is SOAP 1.1 or SOAP 1.2 then return XML.
6. If the incoming CONTENT-TYPE header is XML then return XML.
7. If the incoming CONTENT-TYPE header is JSON then return JSON.
8. If the local template setting is not DEFAULT then use local template setting
9. Use global template setting

The NetWebServiceMethod will generate two objects[1] in the procedure. The one (xml) supports XML structures, the other (json) supports JSON structures.

xml  xFileXML
json JSONClass

These are simple object declarations based on the xFiles xFileXML and the jFiles JSONClass classes respectively. These objects are used internally to parse incoming requests, and to create outgoing answers.

There may be times when it is desirable to override one or more methods in one, or both, of these objects. For example when parsing xml input it may be necessary to add embed code to the xFileXML.AssignField method (as described in the xFiles documentation.) To expose the embed points it is necessary for the object to rather be generated by the appropriate extension template.

For XML, go to the Extensions tab and add the xFiles IncludexFilesObject extension. Set the object name to xml. (This is important, it MUST have this name.) Then go to the settings for the method (Properties / Actions), to the General tab, and turn off the Generate XML Object option there. Doing these two steps means the xml object declaration is generated by the xFiles template, and not generated by the NetTalk template. Once this is done all the embed points for the xml object will be defined and you can embed code in them.

For JSON you follow the same process as for XML, but add the jFiles IncludejFilesObject extension and set the object name to json. Then on the General tab turn off the Generate JSON Object setting.

[1] The JSON object is only generated if the jFiles global extension has been added to the app.

Formatting Fields in the Response

Field prefixes are a thorny problem for a couple of reasons. Firstly, they're a foreign concept in most languages, so they are often not desirable in a public API at all. Secondly they make use of a colon separator, which is not a valid character in a variable name in most languages.

For this reason you have the option to include, or exclude prefixes from parameters and return values. In most cases having the prefix OFF seems the better option.

For builds prior to 9.17 the default for prefixes was ON. For parameters and values added using 9.17 or later the default value is OFF. the setting for existing parameters and returns are not changed by the update, however support for the prefix support in the system has been overhauled so definitely check that your methods are still working the way you expect.

When writing a web app, colons are translated into a double underscore to make the names compatible with HTML. For ServiceMethods though, colons are translated into a single underscore. In builds 9.16 and earlier this was slightly inconsistent as simple GET and POST calls still made use of the double underscore, where XML and JSON used a single underscore. From 9.17 this has been made consistent so that all requests, and all responses use a single underscore if prefixes are ON.

It is probable that many of the services you are providing will require that the user authenticate themselves in order for the action to complete.

As with normal Web apps, the method of authentication is left largely under your control - you can determine the best approach that suits your situation. Some approaches include (but are not limited to);

1. Pass the Login and Password as fields on all incoming packets. These fields are then first checked in your code to verify the request before any other action is taken. The login and password could be passed as data in the request, or in the  HTTP Authentication header. If using headers then be sure to add code to your WebHandler procedure in the Authenticate method.
2. Allow the user to log in with one request and set the session as logged in (as you would in a normal web app). Then further requests from the client can use the SessionID cookie with further requests. Note however that normal session timeout rules apply here - if no traffic from the service is received for a pre-determined period of time, then the session will timeout.

Bear in mind that because the service will typically be used by some service other than a browser more complicated authentication schemes are possible.

Basic Authentication

Digest Authentication

Sessions