User Tools

Site Tools


developerguide

KLets Plugins - Developer Guide

Overview

KLets provides an easy to use plugin mechanism to allow other applications to provide their own actions in order to make them available through KLets itself.

A plugin must provide the list of actions that it want to handle, providing for each action a list of phrases that the user must say in order to activate them. KLets will then execute the relevant action once an activation phrase belonging to it has been recognized.

Once executed, an action can influence KLets behavior, for example to tell the result of the command to the user, to ask for some sort of data, or other.

KLets works also as a translator between what user is saying and what type of data an action needs to perform its work, handling intrinsic problems and inaccuracies of the voice recognition technology.

To do that, for every data needed by an action, a parameter must be declared inside the action definition itself, specifying to which type that parameter belong, among types supported by KLets. Plugins can also declare their own parameter types, using the well know Providers system of Android.

When an action tells to KLets to ask for a parameter, KLets will then ask for it to the user, and once the user replied, it will perform all the logic needed to translate what the user said to data usable by the action, handling possible ambiguities.

It is also possible to catch any data directly from an activation phrase, in order to allows users to perform action quickly.

Technical Documentation

KLets interact with plugins through BroadcastReceivers, Intents, Providers and Manifest meta-data.

Each plugin must export one BroadcastReceiver that handle the following Intent:

It must also declare an application meta-data named “voiceControlActions” pointing to an XML containing static actions information.

If needed by the plugin, it can also export a Provider for each custom data type handled by the plugin itself. In this case, the plugin must also declare an application meta-data named “voiceControlTypes” pointing to an XML containing static custom types information.

The comunication protocol between KLets and a plugin is STATELESS (like the http protocol), so, every time a plugin action is executed, KLets will pass to it the complete set of data collected so far from the user, including any additional data requested by previous plugin executions.

Actions

Actions Meta-Data

The meta-data must be declared inside the AndroidManifest.xml and must point to the xml resource containing actions information:

- AndroidManifest.xml
<manifest>
	<application>
		<meta-data android:name="voiceControlActions" android:resource="@xml/action_definitions"/>
	</application>
</manifest>

The pointed xml file (action_definitions.xml in the sample above) can contain one or more actions, and it is built using the following schema:

- res/xml/action_definitions.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- pluginSpecVersion attribute must be set to the plugin protocol version. Actually only "1.0" is supported -->
<actions xmlns:vc="http://api.voicecontrolapp.com/actions" vc:pluginSpecVersion="1.0">
	<!-- Each plugin can have one or more actions -->
	<action vc:id="action id" 
		vc:name="action name" 
		vc:icon="(optional) action icon"
		vc:help="action help"
		vc:hint="(optional) action hint"
		vc:commands="action patterns"
		vc:languages="list of supported languages in ISO-639 format and separated by commas">
 
		<!-- Each action can have zero or more parameters -->
		<param vc:id="id of this parameter" 
			vc:type="parameter type" 
			vc:help="information about this parameter"
			vc:sample="example of what can be said for this parameter"
			vc:request="text spoken by KLets when asking for this parameter"
			vc:notValid="text spoken by KLets when it cannot find a value for this parameter with the text said by the user" />
	</action>
</actions>

Every “action” element has these attributes:

Attribute Mandatory XML Type Android resource type Description
id Yes String N/A It is used for intent EXECUTE_ACTION in order to identify the action to handle
name Yes String Resource of type String It is used inside options and help screens to help users to identify the action
icon No N/A Resource of type Drawable If present, it is displayed inside the main screen
help Yes String Resource of type String It is displayed inside the main screen
hint Yes String Resource of type String It is used when the user asks for help and is also displayed on the screen as quick-help
commands Yes N/A Resource of type String Array It must be a reference to a string array containing the list of command templates needed to activate the action.
configIntent No String N/A If present, it must be an Intent handled by a plugin Activity. It will be fired when the user choose to open configuration options of this action
languages Yes String N/A It list language codes supported by the action separated by commas. Codes are the lower-case, two-letter codes as defined by ISO-639

When an action has variable parts inside its command templates or it needs additional parameters that could be asked to the user, the plugin must declare them as inner “param” xml elements with these attributes:

Attribute Mandatory XML Type Android resource type Description
id Yes String N/A Used to identify this parameter as token inside command templates and when the plugin must ask for it on execution time
type Yes String N/A The id of the data type of the parameter, among supported types or plugin custom types
help Yes String Resource of type String A quick help explaining what the parameter could contain
sample Yes String Resource of type String A small example of what could be said for the parameter
request Yes String Resource of type String Text said by KLets when it must asks the parameter to the user
notValid No String Resource of type String Text said by KLets when it can't found a valid valued for this parameter. It contains a formatting variable with index 1, that will be set to the text said by the user

Action command templates

A command template consist in a text with zero or more variable parts as defined by “param” elements inside the action definition.

These variable parts are marked by tokens in square brackets containing the parameter id in uppercase (like this [PARAMETER_ONE]). Each token MUST be placed only one time inside a command template and MUST be separated from other tokens AT LEAST by a single character, otherwise KLets will ignore the command template and will not use it to activate the action.

Commands template doesn't require to contains tokens for parameters declared in an action, as they could be asked directly by the plugin at execution time, but to use a token it is required to declare the relative parameter in the action definition, otherwise the token will be handled as simple text of the command template.

Command templates, also, don't have to respect a specific case, because KLets will convert them in lowercase at runtime (except for tokens).

Parameter types

Parameter types provided by KLets

Parameter types provided by KLets out-of-the-box are:

Type ID Accepted value Value passed to the action
TEXT Any textual value A String with the value said by the user
NUMBER Any textual value that can be translated in a sequence of number A String of numbers
PHONE_NUMBER Any textual value that can be translated in a valid phone number A String containing only phone number characters
CONTACT Any text that could be mapped to contacts inside the phone contact list A List of Bundle with the data defined by ContactData class constants.
CONTACT_PHONE Any text that could be mapped to contacts with one or more phone numbers A List of Bundle with the data defined by ContactData class constants.
CONTACT_MAIL Any text that could be mapped to contacts with one or more email fields A List of Bundle with the data defined by ContactData class constants.
CONTACT_ADDRESS Any text that could be mapped to contacts with an address set A List of Bundle with the data defined by ContactData class constants.
CONTACT_SKYPE Any text that could be mapped to contacts with a Skype account linked to him A List of Bundle with the data defined by ContactData class constants.
AUDIO_MEDIA Any text that could be mapped to a song, an artist name, an album or a playlist on the phone. A List of Bundle with the data defined by MediaData class constants.
AUDIO_MEDIA_ARTIST Any text that could be mapped to a song, an artist name, an album or a playlist on the phone. A List of Bundle with the data defined by MediaData class constants.
AUDIO_MEDIA_SONG Any text that could be mapped to a song on the phone. A List of Bundle with the data defined by MediaData class constants.
AUDIO_MEDIA_ALBUM Any text that could be mapped to an album on the phone. A List of Bundle with the data defined by MediaData class constants.
AUDIO_MEDIA_PLAYLIST Any text that could be mapped to a playlist on the phone. A List of Bundle with the data defined by MediaData class constants.
APPLICATION Any text that could be mapped to a launch icon name on the phone. A single Bundle with the data defined by AppData class constants.
DATE_TIME Any text that could be mapped to a date with optional time. A Date with the resolved absolute time.
TIME_ABS_OR_REL Any text that could be mapped to a time, said in absolute form (like “at 8 and 20 minutes), or in a relative form (like “in 86 minutes”). A Date with the resolved absolute time.
TASKER Any text that could be mapped to a Tasker's task name A String with the Tasker's task name

Custom parameter types Meta-Data

If a plugin needs a custom parameter type not supported by KLets, then it need to implement and export a Provider, and to configure it inside the custom parameter types Meta-Data.

The meta-data must be declared inside the AndroidManifest.xml and must point to the xml resource containing custom types information:

- AndroidManifest.xml
<manifest>
	<application>
		<meta-data android:name="voiceControlTypes" android:resource="@xml/type_definitions"/>
	</application>
</manifest>

The pointed xml file (type_definitions.xml in the sample above) can contains one or more custom types, and it is built using the following schema:

- res/xml/type_definitions.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- pluginSpecVersion attribute must be set to the plugin protocol version. Actually only "1.0" is supported -->
<dataTypes xmlns:vc="http://api.voicecontrolapp.com/data_types" vc:pluginSpecVersion="1.0">
	<dataType vc:id="data type id" 
	    vc:providerAuthority="provider autority that provides data for this type"
	    vc:valueField="the field containing the value to pass to the action (IT MUST BE A String TYPE FIELD)"
	    vc:textField="the field containing the text to check against what the user said (IT MUST BE A String TYPE FIELD)"/>
</dataTypes>

When KLets will need to get available values for this type, due an action activation phrase, or an explicit request by the plugin, it will query the configured provider autority and will select the value with the text that best match what the user said, setting its value in the map of the extra EXTRA_PARAMETERS when invoking the action.

Action execution

KLets will fire an intent for INTENT_EXECUTE_ACTION every time an action must be executed, and it must be handled by a BroadcastReceiver.

This intent is fired in the following situations:

  • When the action has been successfully identified by one of its templates
  • When the action has made a request for a parameter and it has been said by the user
  • When the action has made a request for a user choice and it has been performed

The intent contains all extras needed by the action to perform its logic:

  • EXTRA_ACTION_ID: The action id for wich KLets is asking the execution, as configured inside the action definition. Useful when a plugin provide more than a single action.
  • EXTRA_PARAMETERS: A Bundle with a map with a key/value pair for every parameter handled so far (both parameters parsed from the activation phrase and parameters explicitly asked by the action itself).
  • EXTRA_CHOICES: A Bundle containing a map with a key/value pair for every choice performed by the user so far.
  • EXTRA_ASYNC_PENDING_INTENT: A pending intent that must be used for asynchronous tasks completion.

Once executed, the action must set a result code from the BroadcastReceiver handling the INTENT_EXECUTE_ACTION Intent, in order to tell to KLets what should be done.

Accepted result codes are:

Result ACTION_RESULT_OK

The result ACTION_RESULT_OK must be used when the action has completed its work.

Optionally, the BroadcastReceiver can set the result extras to a bundle with the following extras:

  • EXTRA_MESSAGE: Optional; Can be set to a message that must be said before KLets quits.
  • EXTRA_PENDING_INTENT: Optional; Can be set to a PentingIntent that must be fired just before KLets quits and after that any bluetooth and audio connections have been closed.

Once received this result code, KLets will quits terminating the vocal session.

Result ACTION_RESULT_ASK_PARAM

The result ACTION_RESULT_ASK_PARAM must be used when the action needs to ask the user for a parameter.

When setting this result code, the BroadcastReceiver must also set the result extras to a bundle with the extra EXTRA_PARAMETER_ID set to the id of the parameter to ask.

Once the requested parameter has been said by the user and looked-up by KLets, KLets will fire a new INTENT_EXECUTE_ACTION intent, with the map inside the extra EXTRA_PARAMETERS updated to contains the entry, identified by the parameter id, set to the value said by the user accordy to the parameter type.

Result ACTION_RESULT_ASK_CHOICE

The result ACTION_RESULT_ASK_CHOICE must be used when the action needs to ask to the user to select a value from a list of elements.

When setting this result code, the BroadcastReceiver must set the result extras to a bundle with following extras:

  • EXTRA_MESSAGE: must be set to a message that must be said to the user.
  • EXTRA_CHOICE_LABELS: must be set to an array of String with labels to use for the choice list.
  • EXTRA_CHOICE_VALUES: must be set to an array of Parcelable object to use for the choice list. The element selected by the user will be set on the extra EXTRA_CHOICES.
  • EXTRA_CHOICE_ID: must be set to an identifier that the broadcast receiver can use to identify the choice performed by the user on the next EXECUTE_ACTION intent fired.

Once the requested choice has been performed by the user, KLets will fire a new EXECUTE_ACTION intent, with the map inside the extra EXTRA_CHOICES updated to contains the entry, identified by the previously set EXTRA_CHOICE_ID, set to the value selected by the user.

Result ACTION_RESULT_ASYNC

The result ACTION_RESULT_ASYNC must be used when the action execution is asynchronous or cannot be completed inside a broadcast receiver, due resources constraints, timing constraints or other reasons.

In this case, once the action completed its asynchronous logic, it must fire the PendingIntent originally passed to the BroadcastReceiver in the extra EXTRA_ASYNC_PENDING_INTENT, using the PendingIntent.html.send(Context, int, Intent) Android API, setting:

  • the context parameter to the current plugin context.
  • the code parameter to the same value that would be set using the “BroadcastReceiver.setResultCode(int)”, with the exception of the ACTION_RESULT_ASYNC value that isn't supported in this mode. (this parameter is supported starting from KLets 2.4.0, on previous versions, set this parameter to 0 and put the resultCode value in an additional EXTRA_RESULT_CODE extra on the Intent passed to the intent parameter)
  • the intent parameter to an intent instance with extras that would be set using the “setResultBundle()” of the the action BroadcastReceiver

Ensure to always execute the PendingIntent in case of error, otherwhise KLets will remain in a “working in progress” state infinitely.

Sample plugin

This is a sample plugin that will repeat what the user said.

Lets defining which are the activation texts of this plugin.

We can say that the action will be called “parrot” and that it could be activated using “tell”, “tell to me” or “say” phrases. We will also setup two quick activation phrases, that will allows the user to say the text to repead within the activation phrase itslef.

So lets add them to the string.xml resource file, with other resources needed for the action.

- res/values/strings.xml
<?xml version="1.0" encoding="utf-8"?>
<resources>
	<!-- Parrot action resources -->
	<string name="action_name">Parrot</string>
	<string name="action_hint">tell, tell to me or say [words to repeat]</string>
	<string name="action_help">This action repeats what you say.\n
\n
Examples:\n
\"<b>tell</b> Hi! I\'m an Android phone!\"\n
\"<b>tell to me</b> You are the best!\"\n
\"<b>say</b> I\'m the best voice command app of the world!\"</string>
 
	<string-array name="activation_texts">
		<item>tell</item>
		<item>tell to me</item>
		<item>say</item>
		<item>tell to me [TEXT_TO_REPEAT]</item>
		<item>say [TEXT_TO_REPEAT]</item>
	</string-array>
 
	<string name="action_param_text_help">The text to repeat</string>
	<string name="action_param_text_sample">You are the best!</string>
	<string name="action_param_text_request">Please, tell me what I need to say.</string>
</resources>

Then, we define the actio metadata in an XML file called parrot_action_definition.xml. We will also define the action parameter TEXT_TO_REPEAT, used inside two of the activation phrases, and that will be asked by the plugin when missing.

- res/xml/parrot_action_definition.xml
<?xml version="1.0" encoding="UTF-8"?>
<actions xmlns:vc="http://api.voicecontrolapp.com/actions" vc:pluginSpecVersion="1.0">
	<action vc:id="parrotAction" 
			vc:name="@string/action_name" 
			vc:help="@string/action_help"
			vc:hint="@string/action_hint"
			vc:commands="@array/activation_texts"
			vc:languages="en,it">
		<param vc:id="TEXT_TO_REPEAT" 
				vc:type="TEXT" 
				vc:help="@string/action_param_text_help"
				vc:sample="@string/action_param_text_sample"
				vc:request="@string/action_param_text_request" />
	</action>
</actions>

After, we will write the BroadcastReceiver that will handle the action execution. As you can see, the action will check if there is the text to repeat, and if it is missing, the action will ask for it.

- src/com/example/testandroid/ExecuteActionBroadcastReceiver.java
package com.example.testandroid;
 
import com.voicecontrolapp.klets.api.KletsPluginApi;
 
import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.os.Bundle;
 
public class PluginActionReceiver extends BroadcastReceiver {
	@Override
	public void onReceive(Context context, Intent intent) {
		Bundle resultExtras = getResultExtras(true);
 
		Bundle parameters = intent.getBundleExtra(KletsPluginApi.EXTRA_PARAMETERS);
 
		// Lets check if we have something to repeat.
		if (!parameters.containsKey("TEXT_TO_REPEAT")) {
			// The user didn't said the text to repeat. We tell to KLets to ask for it.
			setResultCode(KletsPluginApi.ACTION_RESULT_ASK_PARAM);
 
 
			resultExtras.putString(KletsPluginApi.EXTRA_PARAMETER_ID, "TEXT_TO_REPEAT");
 
			// Quit and wait for the next invocation, that will have the needed parameter.
			return;
		}
 
		String textToRepeat = parameters.getString("TEXT_TO_REPEAT");
 
		// Setting the result Bundle and result code for this BroadcastReceiver execution.
		setResultCode(KletsPluginApi.ACTION_RESULT_OK);
 
		// We just tell to KLets to say back what the user said, setting it to the message in the result bundle.
		resultExtras.putString(KletsPluginApi.EXTRA_MESSAGE, textToRepeat);
	}
}

As latest stepwe need to wire everyting up inside the Application Manifest:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.testandroid"
	android:versionCode="1" android:versionName="1.0">
 
	<uses-sdk android:minSdkVersion="8" android:targetSdkVersion="16" />
 
	<application android:allowBackup="true" android:icon="@drawable/ic_launcher" android:label="@string/app_name">
 
		<receiver android:name="com.example.testandroid.PluginActionReceiver" android:exported="true">
			<intent-filter>
				<action android:name="com.voicecontrolapp.klets.api.INTENT_EXECUTE_ACTION" />
			</intent-filter>
		</receiver>
 
		<meta-data android:name="voiceControlActions" android:resource="@xml/parrot_action_definition" />
	</application>
 
</manifest>

That's it. Now the action will be usable from KLets, with every feature of standard actions (like activation texts customization, confirmation requests and so on).

For a complete sample plugin project, containing also an example for custom data types, config intents atc, download it from the topic Downloads, JavaDocs and Support.

Security

The current API tries to provide a protection against malicious applications that could try to use KLets plugin system as a “backdoor” to trigger plugin actions: to do that, KLets provides and holds two permissions with a “signature” protection level (in order to deny other application to hold them), PERMISSION_EXECUTE_TASK for plugin execution and PERMISSION_QUERY_TYPE for plugin data access. Plugins could use this permissions to protect their own receivers and providers, in order to allow only KLets to access them.

Now, there is a problem on how permissions provided by applications are handled inside Android: they can't be trusted, because another application, if installed before KLets, can provide the same permissions, in order to hold them and get access to plugins receivers and providers. However, plugins can protect themselves from this sort of permission-hijacking using a simple application signature check: Android allows only one application at time to provide a specific permission, so, plugins can check if the public signature of the application that is providing a KLets permissions match with the KLets public signature. This way, a plugin can be reasonably confident that who is calling it is KLets and not a malicious app.

To check the signature, use the utility method KletsPluginApi.isPermissionAuthentic(...). It can be used to check both PERMISSION_EXECUTE_TASK and PERMISSION_QUERY_TYPE permissions.

Changelog

1.1.0

  • Refactored API classes base package from com.lukasoft.android.voicecontrol to com.voicecontrolapp.klets
  • Renamed class com.lukasoft.android.voicecontrol.api.VoiceControlPluginApi to com.voicecontrolapp.klets.KletsPluginApi
  • Added two new INTENT usable by plugins to monitor KLets vocal session starts and shutdowns: INTENT_SESSION_START and INTENT_SESSION_STOP

1.0.0

  • Initial release

Downloads, JavaDocs and Support

Plugin API JavaDocs can be found here: Api JavaDocs

Plugin API jar can be downloaded here: voicecontrol-plugin-api.jar

Plugin API sources can be downloaded here: voicecontrol-plugin-api-sources.zip

(OUTDATED: refers API version 1.0.0) A sample plugin project can be downloaded here: voicecontrol-plugin-sample-project.zip

Plugin Development support forums can be found here: Plugin Development forums

developerguide.txt · Last modified: 2013/10/20 20:18 by lukakama