Support for Applications Rest API

[deruelle]

Add Support for https://www.twilio.com/docs/api/rest/applications

[ivelin]

This feature has to be coordinated with the Visual App Designer work.

[deruelle]

From IRC chat :

where you able to check with the PlayMyBand WebRTC hackers
how they could potentially package their app to RAS
even if they didn't use RVD
to push it still to RAS ?
I'm thinking we will probably have other use cases of pure WebRTC apps
not involving RVD
or pure RCML apps
where this could be potentially useful

[ghjansen]

Thanks @deruelle , i'll proceed with this adjustment on the current branch.
Btw, today i've looked for possible integrations between Applications API and RVD/AdminUI, but i'm not sure if we'll really need some. For instance: the applications created using RVD should use the API to store data in some way, or the API will be more used for non-RVD apps?

[ghjansen]

The info visibility control between different accounts is in my last commit, now while trying to create a client or app using the same name as an existing one, a HTTP 409 is sent with a explanation message, when the account used to request is different than existent client/app account. I'm not sure if 409 is the best choice, I can change if there is a better one.

Also, after some words with @gvagenas and @otsakir some ideas came up about using Applications API on RVD and AdminUI, so i'm testing a few things now and i will share the news as i progress.

[otsakir]

DID and Application entities

These are the conclusions of a constructive discussion on the issue with @gvagenas

• An Application contains a single RCML-returning url.
• DID contains references to one one or more Applications using Application SIDs. Thus, all rcml url fields should be replaced by the respective application SID fields.
• A DID belongs to an account through the respective foreign key.
• An Application belongs to an account through the respective foreign key.

So when a call arrives, the DID is processed, the respective Application SID is retrieved and rcmlURL is determined and used for getting the actual RCML.

• An application could be accessed by multiple sub-accounts if it belongs to their parent account. Thoughts?
• There is nothing wrong for an Application entity that has an empty rcmlURL. It's just useless.

Syncing Application entities with RVD projects

These are the conclusiong of a constructive discussion on the issue with @ghjansen . We thought of three ways of implementing syncing Restcomm Application entities with RVD. Operations that are at stake are:

• Project listing
• Project creation/deletion
• Getting project RCML

1. RVD-UI uses the existing RVD endpoints for project management. RVD-server directly accesses restcomm DB using a copy of the dao.jar to manage applications and takes into account the filesystem state while doing that.
2. RVD-UI uses the existing RVD endpoints for project management. RVD-server uses the newly created Application REST api for application management and takes into account the filesystem state while doing that.
3. RVD-UI directly uses the Application REST api to manage application records. The client (RVD-UI) makes sure that project creation/removal operations are reflected to the filesystem too using the RVD-server api.

thoughts? @deruelle ? @gvagenas ? @ghjansen ?

[deruelle]

• DID and Application entities : I think this is in Context of Issue Multi-tenancy support #87 and as discussed in IRC today, @ghjansen please check how Twilio works in that regard so we have best compatibility.
• Syncing Application entities with RVD projects: I'm thinking we should probably use 3. the new Application REST API. I'm wondering if it would make sense to store the RVD Application in the DB somehow to ensure easy sharing of apps within a cluster of RestComm nodes and avoid filesystem state synching ?

[ghjansen]

There is a new topic about Multi-tenancy on Google Groups to gather the information about our discussions: https://groups.google.com/forum/#!topic/restcomm/NygtnHpgeXE. Also, the document: https://docs.google.com/document/d/1zjZts9a86oL9xV2wlADJ5m2BA96flvPJ8gQ2qRWErJE.

I'm moving forward on strategies to sync Applications entities.

[ivelin]

Excellent discussion. Very important to get this right. It has implications on app publishing, distribution, scaling, usage reporting, billing,

A couple of comments

@ghjansen wrote:
"DID contains references to one one or more Applications using Application SIDs."

• When would a DID need to have reference to more than one apps?

@deruelle wrote:
"Syncing Application entities with RVD projects..."

• We should detach RVD from app runtime. RVD is only scoped to creating and publishing apps. As noted earlier in this thread, we will eventually allow developers to package and publish apps outside RVD.
• Once an app is installed on a Restcomm deployment (which could be a large geo distributed cluster) the config settings for the app for a particular user account controlling the app should be seen equally from any restcomm runtime instance in the deployment cluster.
• The runtime state of an app can be local to the hosting Restcomm runtime instance where it executes. Let's keep the runtime model simple. Each executing app instance will run in its own memory space working mostly with variables local to the call context which triggered creating of the app instance. If multiple calls arrive at the same time, multiple runtime instances of the app will be created, one for each call context. This is a model well proven to work for JSLEE apps.
• Sharing context between instances should be explicit and rare. We have to think this through how to implement so it performs consistently in a scaled deployment. External services are the interim answer.

@ghjansen Lets not forget that an app can have multiple entry points. One for Voice/Video Call, another for message/SMS/MMS, and yet another for Web Trigger (click to call).

[ghjansen]

@ivelin actually, "DID contains references to one one or more Applications using Application SIDs." was a mention from @otsakir. Maybe this is about Voice/SMS/USSD request and fallback options, where is possible to configure different apps between these options. @otsakir you mind to evolve on that?

@ivelin considering your mention on the environmental aspects:

• Would be correct to assume that multiple instances on a Restcomm deployment will share the same database, so instances will be able to equally provide the config settings about a app along the cluster?
• Considering a single instance, if 2 calls arrives at the same time for the same app, this instance will have 2 isolated call contexts, each one running its own app instance?

Im considering to create a diagram about this so we can have a visual of the environment. Waiting for your confirmations/corrections.

[otsakir]

Regarding RVD internals and state, application execution is stateless. All state is transferred on RCML action urls.

Btw, does it make sense to still use the filesystem for persisting RVD projects in the new distributed cloud setup or is it a blocker? If it's a blocker we should consider porting the project persistence layer to DB.

[ghjansen]

Runtime
Once Restcomm check for RCML urls before execute Applications, its good to mention that besides the url assigned to a DID or SIP Client, the current logic is already testing if the IncomingPhoneNumber has the applicationSid instead of a Voice/SMS url. So, replace RCML url fields by applicationSid as mentioned before seems the right thing to start the Applications using database info. We can also keep RCML fields and allow both urls and Applications, like Twilio does. (see new images on Google Docs)

Synchronisation and storage
Considering the information and ideas we have until now, i've created the following diagram to try to represent the current Applications life cycle so we can better visualise and plan modifications. The red line across the diagram represents the synchronisation idea.

Publish/Build
There is also the new idea regarded to detach RVD from app runtime. The goal is to keep a "application snapshot" used by Restcomm for runtime, while RVD is free to do modification over a "application project". So by manually invoking a "Publish/Build" event, a new application snapshot is created based on current application project, and runtime start using the latest version of this app.

Suggestions?

wdyt @otsakir @gvagenas @deruelle @ivelin ?

[ivelin]

@ghjansen you are doing an outstanding work leading the design discussion on this issue. Let me digest it and get back to you with thoughts.

[ivelin]

@otsakir
"Regarding RVD internals and state, application execution is stateless. All state is transferred on RCML action urls."

How about application level variables? They are supposed to be stored somewhere, right? Or are they passed from module to module throughout the workflow and never stored in the RVD runtime environment?

"Btw, does it make sense to still use the filesystem for persisting RVD projects in the new distributed cloud setup or is it a blocker? If it's a blocker we should consider porting the project persistence layer to DB."

Good point. Apps should go in the shared DB. All Restcomm/RVD runtime instances in the same deployment should see the exact same persistent data view. Including app configuration and user created apps.

The runtime state of an executing RVD app thread doesn't have to persist and replicate in the cluster...for now. Eventually we would want redundancy for some apps. Such as emergency, financial, etc.

[ivelin]

@ghjansen Comments on your design and diagram. Its excellent material.
"Synchronisation and storage": Lets add the workflow steps for managing apps from within Restcomm Admin UI. Looking up apps in RAS, installing, removing.

"Publish/Build":
Great idea. Its a natural extension to having apps download from RAS and configured vs executing. This will become very important for things like rolling upgrades of apps. If an app is running, installing a new version should not terminate it, but rather prepare and wait. The old version of the app should temporarily stop receiving new calls but wait until existing calls complete or timeout. When all executing threads of the all version finish, then the new version is activated to take on new calls. Its a concept similar to gracefully upgrading J2EE and JSLEE apps.

[otsakir]

@ivelin
"How about application level variables? They are supposed to be stored somewhere, right? Or are they passed from module to module throughout the workflow and never stored in the RVD runtime environment?"

They too are transferred from module to module through the action urls. So, no state there either regarding the runtime.

Btw, the only place that contains information that can change at runtime by the user are the configuration options of the application (if defined).

@ghjansen
The snapshot concept and the two step Build/Publish idea sound great.

Regarding the filesystem VS DB discussion the following need to be considered:

• What happens with .wav files ? Are they stored as BLOBS? It doesn't seem nice. It will add too much overhead to the DB and processing for operations that are pretty elementary in principle (read a file and stream it as a response). There are no additional fields required, special queries to be done on using the .wav properties so, having a BLOB for it seems like an overkill. I was thinking of having a different storage layer for media but i really have no decent alternative to suggest.
• What happens with support of new kinds of applications? Having a specification that is based on filesystem with definitions of directories and filenames will be done one way or the other since an application need to be submitted in the form of a .zip package (right?). Then we need to define another spec for mapping the application to the database. What fields would be normalized etc.
• Having projects in the fs allows a much looser rule set of requirements for them. For example, older versions can be supported with a different project loader/interpreter in place. All RVD has to do is check the project version and adapt its behavior accordingly. If the project needs to be stored in the DB then all projects should be converted to latest version on loading and stored afterwards. Only latest version supported as there seems to be no way to store different kinds of projects in the same table. On the other hand, that's probably a good thing.
• We should be carefull with what we normalize! There is no need to normalize every little detail of every single step/element. I suggest a mixed model of pretty fixed table properties and JSON content with step details. Here are some reasons for that:
  • There is no need to search by the 'phrase' of a or the 'Service URL' of an ES. Why keep different fields for that.
  • Having table properties for every single detail will cause a chaos in maintenance. Imagine upgrades in the database schema with new fields in RVD etc. It won't work.

Overall, porting to DB seems a pretty heavy and time-consuming task with clear contracts to be defined, upgrade process etc.

[ghjansen]

Considering all the discussions, designs and development on this issue, now we have RVD Projects sync through Applications API. So every time that a RVD Project is created, imported, renamed or removed using RVD UI, a request is made to Restcomm to create/rename/remove on database.

No changes were made until now on FS. The current reference between FS Project and DB Application is the name choosed to create the RVD Project. After some chat with @otsakir we've agreed that a new RVD Project could be created on FS using the applicationSid (obtained after create a new Application) instead of the project name, thus forming a strongest reference between FS and DB.

Change the name that RVD uses to store projects on FS may imply in some migration process, to deal with existent projects from production environments, but it can potentially simplify reverse sync (from DB to FS), then by creating/renaming/removing over Applications API we could have the correspondent events on RVD. Anyway, the synchronisation of events from Applications API to RVD Project was another discussion that i had with @otsakir and we agreed that maybe is not so crucial for now.

Guys, let me know if i can go further in something for this issue and thank you for all support!