BlackBerry PushRegistry How-To

Suppose you’d like to create a BlackBerry application that will launch when you send an SMS message to the appropriate port.  It’s possible, but there are a number of non-obvious (and not-necessarily-documented) tricks.  I spent a few too many hours on experimenting to figure out precisely what works, and I’m posting my notes here to save you some time:

1. Apparently your application has to be a MIDlet, not a “CLDC application” (RIMlet).  I was a little surprised when I read this in the BlackBerry Java Dev forum, since it would undoubtedly be trivial for RIM to allow either type of application to register with the PushRegistry and get launched.  So I did a few tests (with a “MIDlet-Push-1” Jad property analogous to the way the RIMlet uses the “MIDlet-1” property), and, indeed, the PushRegistry doesn’t seem to work for RIMlets, but it works fine for MIDlets. (Note: it’s still theoretically possible that there’s a way to make it work for RIMlets, but I couldn’t find the trick.)
2. Adding a custom property to your Jad file is really easy if you know the trick. 😉 The trick is that (in the BlackBerry JDE) you can add the Jad file to the project (in the same way you add source files and images).  Once you’ve added the Jad file to the project, you can add properties and then they won’t get overwritten when you rebuild.  But before you add it, you should modify the project properties (setting the application type to MIDlet) and then build the project once, so that it will create for you a correct initial Jad file to use.
3. Before your application closes, you should dynamically re-register with the PushRegistry.  This was the trickiest one to guess!  If your MIDlet is launched by the PushRegistry — and you open the corresponding connection to read the SMS message that launched your MIDlet — this automatically unregisters your MIDlet from the PushRegistry.  With other handsets, once you’re registered with the PushRegistry (either statically in the Jad or dynamically from within the application), you stay that way until you actively unregister it.  But not BlackBerry.  Or at least not with the BlackBerry simulator that comes with the BlackBerry JDE 4.2.0.  I assume the behavior is the same in other versions and on the device, but I’m not certain…
4. How do you simulate the SMS to test the application launch with the PushRegistry?  I explained that here.

SMS message exchange between BlackBerry simulator instances: a very simple demo

If you’re writing a BlackBerry Java application that sends and/or receives SMS text messages, naturally you need to be able to test the application on the simulator.  And naturally the simulator doesn’t really send SMS messages, so you need to know how to get the simulator to exchange simulated SMS messages with another instance of the simulator.

This is very easy to do.  You can do it with the SmsDemo that comes in the BlackBerry JDE’s sample applications bundle — even though the SmsDemo’s JavaDoc seems to imply that you can’t.  If you look in the code of SmsDemo.java (usually found at C:\Program Files\Research In Motion\BlackBerry JDE <version>\samples\com\rim\samples\device\smsdemo), the JavaDoc says the following:

This program requires the associated server component found in the com.rim.samples.server.smsdemo package under the samples directory in your JDE installation.

That’s not precisely true.  It doesn’t require the server component — it can actually send demo messages from one instance of the simulator to another, as long as you set the sms-source-port of the one to be the sms-destination-port of the other, and vice-versa.  There are just a few quick steps and notes for launching the two instances:

Bluetooth Checkers!!!

My bluetooth checkers game is ready to go!!!

If your MIDP 2 handset supports JSR 82 (Bluetooth API), you can download and install it (free) from my usual checkers wml page or my games html page!!!

I promise this is the last Checkers version (for a long time anyway 😉 ), and next I’ll move on to some graphics discussions. But I wanted to take one standard game and program it using a few different protocols to compare and contrast how each one works in a multiplayer game.

I think Bluetooth is a little more challenging and complicated to program than SMS. Part of the reason for that is that SMS uses the operators’ standard way of routing communications to a handset. You find the right destination by typing in the right phone number. And it’s the same phone number you’d use to route other comunications to that device. So you don’t have to reinvent the wheel to find, identify, and authenticate the device you’re looking for.

Bluetooth allows you to find all active, bluetooth-enabled devices within range and see what kind of services they offer. Each server offering a bluetooth service publishes a set of detailed service records describing the services available. So you have a fair amount of work on the server side creating useful descriptions and on the client side reading the descriptions and choosing the right service on the right device. Add to that the fact that you have the option of choosing a plain socket type connection or discrete packets, different authentication and/or encryption schemes, and even object-oriented communications using Java OBEX, you have practically unlimited potential to customize (and complicate 😉 ) your application according to your needs.

In my case, I decided I wanted to make my game’s communications as simple as possible. So I started with the assumption — since bluetooth is a protocol for local communications — that you and your friend are in the same room together and have verbally agreed to start up a game of checkers on your two handsets. My checkers game is fundamentally a peer-to-peer type game, but bluetooth is a client-server protocol, so I had my game start with a screen where the user can choose whether his handset will be acting as the client or as the server (the first player selects server and the second client).

Then there was the question of writing the code so that the two devices will find each other instead of finding other random bluetooth devices and services. Fortunately — with a simple services like this one — all you really need is a UUID. You can create your own UUID (in both Linux and Windows) using the uuidgen command. Then on the server side, all you need to do is construct your bluetooth URL using your UUID and open the connection and wait for clients to call you up. My game’s server URL looks like this:

btspp://localhost:2BBC2D287C8C11DBA1500040F45842EF;name=Checkers;authorize=false

The “btspp” indicates that I’d like to use the bluetooth streaming (socket-like) protocol, the “localhost” makes me a server; then I add on my UUID (what the “uuidgen -t” with the hyphens removed), then some attributes (the application name and the fact that I don’t want to require authorization). Opening this connection creates the corresponding service record, which you can get a handle to and add attributes to if you like. The WTK’s BluetoothDemo shows a good example of how to add additional descriptive attributes. The BluetoothDemo has a server that makes images available for bluetooth clients to download, and there you can see the programmer has added a custom attribute to give the name of the image being offered. As you can see in the ServiceRecord JavaDoc, there are a number of common standard attributes that you can use as well. The attributes aren’t too complicated to use — you just need to keep in mind that different attributes values can be of different types so the values are wrapped in DataElements that allow you to handle all of the different types of values in a consistent way.

Once your server is up and running, the next thing is to program the client to find it and connect to it. Basically, all you need to do is grab a handle to the local DiscoveryAgent and send it off in search of devices and services. It toodles around on its own thread looking for them, then it calls the assigned DiscoveryListener when it’s done.

Discovery is a two-step procedure: First you gather a list of devices, then you search the devices for the service you want. (Since I didn’t bother with any special service record attributes on the server side, when I started searching my device list for services I sent a null attrSet to the searchServices method.) Note that if you’re writing a program where the client frequently reconnects to the same set of known servers, you can optimize discovery by caching a list of known devices.

It might seem like you could simplify the discovery procedure by calling the method DiscoveryAgent.selectService() which finds all of the available connections that provide the desired service (without first getting all the devices and searching each one). Or that you could optimize by searching each device for services as you find it rather than first finding all devices and then searching them one by one. However, over on developer.sun.com I found this interesting little powerpoint presentation full of bluetooth programming best practices, and this little slideshow specifically says not to do either one of those things. Because they seem like good ideas, but really the lead to nothing but badness. And I figure that if anyone would know the people over at developer.sun.com would, so I decided to follow their advice.

Once the client side of the program and the server side find each other, the rest is a breeze! Whew! From then on it’s just standard Java socket programming. The server’s acceptAndOpen command stops blocking and returns a handle to the connection. Both the client and the server open the InputStream and OutputStream of their respective connections, and start sending each other data. All you have to do is send the right data down the stream and read it and interpret it correctly on the other side. What could be easier? 😉

For the bluetooth version of the checkers game, I eliminated the “taunt” feature I’d added for SMS Checkers. With SMS checkers, you can play against your friend wherever he may be. When you start up the game and enter your friend’s phone number — thanks to the push registry — the invitation SMS will buzz his handset to invite him to play whether he’s on the other side of the world or in a meeting or whatever. So it’s useful to be able to send a text message along with every move. In the bluetooth version you essentially have to have already agreed together that you want to play, so the taunt feature is less useful. It would have been possible to hook up the bluetooth version to the push registry as well, but I figured it was already complicated enough as it is.

So please try it out and tell me what you think!

More about JSR 75

For today’s JSR adventure, I improved my SMS checkers game — as promised — by adding a feature to read contacts from the handset address book to choose opponents. The new version is available as CheckersPlus. Then I figured that since I’m playing with JSR 75, I should write a sample program to test the other half of JSR 75: the FileConnection API.

I like the elegance of the FileConnection API, and in particular the way it’s designed to function just like the various network connection APIs (using Connector.open with a URL, returning a Connection with an InputStream and and OutputStream just as any a network connection). It’s a very natural way of accessing files both from the Java perspective (where files have traditionally been accessed through the same streams as sockets, etc.) and in the general networking universe (where users are already accustomed to reading local files in an Internet browser using a file://-type url).

Another cool advantage to FileConnection is that you can access parts of the handset’s memory that you can’t touch from MIDP’s standard little RMS memory sandbox. I found by listing the available file roots on my Sagem my700x that I can access the images folder. So for some handsets it may be possible to have a Java MIDlet store a wallpaper or a ringtone as a wallpaper or a ringtone by storing it in the right place where the handset can use it.

But that brings us to the usual disadvantage: you have to know the particular handset very well to know which folders you have access to and how to access them. So it’s great if your project has a small set of target handsets, and not so great if you want your application to be useful on all MIDP handsets across the board. The RMS (Record Management System) has the advantage of always being supported, so it’s probably better to use the RMS if all you want is to set some simple data aside and find it again later.

Of course, even the RMS isn’t as predictible as one might like. There are handsets where RMS access is slower than FileConnection access (and some where it’s not), some that handle individual records in unexpected ways (setting aside a big space for each record even if it’s not used, or failing to free up the memory of a deleted record). Then there’s the fact that the RMS is not convenient for pooling data among multiple MIDlet suites: To share a record store between suites, you need to leave the access door open to every MIDlet on the device, plus the record store is deleted when the associated MIDlet suite is deleted even if other MIDlet suites still wanted to access the data.

So, there’s a bit of a trade-off, meaning it’s better to know both the FileConnection API as well as the RMS, and choose which one to use depending on your application.

For today’s fun, here’s a sample class to illustrate the FileConnection API, doing a few very basic, standard things like listing the available roots and files; reading, writing, creating a test file, etc. As with PIM, FileConnection access is done in its own thread:

package net.frog_parrot.test;

import java.util.Enumeration;
import java.io.*;

import javax.microedition.io.*;
import javax.microedition.io.file.*;

/**
* A simple file Connection testing utility.
*/
public class FCRunner extends Thread {

FCTest myMidlet;

FCRunner(FCTest test) {
myMidlet = test;
}

public void run() {
FileConnection rootdir = null;
try {
Enumeration items = FileSystemRegistry.listRoots();

// Now print the available roots:
while(items.hasMoreElements()) {
String rootname = (String)(items.nextElement());
myMidlet.display(“\n *** new root: ” + rootname);

// open the root directory:
// note there are three slashes before the root name
// because there is no “host” for this connection:
rootdir = (FileConnection)Connector.open(
“file:///” + rootname);
// List the current files:
Enumeration ls = rootdir.list();
while(ls.hasMoreElements()) {
String filename = (String)(ls.nextElement());
myMidlet.display(” file: ” + filename);

// print the contents of the file:
FileConnection file = null;
try {
file = (FileConnection)Connector.open(
“file:///” + rootname + “/” + filename);
if(file.canRead()) {
InputStream is = file.openInputStream();
byte[] contents = new byte[25];
int len = is.read(contents);
is.close();
myMidlet.display(” contents: ”
+ new String(contents, 0, len));
} else {
myMidlet.display(” * not readable”);
}
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
file.close();
} catch(Exception e) {}
}
}

// now try to create a file:
FileConnection newfile = null;
try {
newfile = (FileConnection)Connector.open(
“file:///” + rootname + “myNewFile”);
if(newfile.exists()) {
OutputStream os = newfile.openOutputStream();
os.write((new String(“overwriting old contents”)).getBytes());
os.close();
} else {
newfile.create();
OutputStream os = newfile.openOutputStream();
os.write((new String(“creating new contents”)).getBytes());
os.close();
}
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
newfile.close();
} catch(Exception e) {}
}
}
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
rootdir.close();
} catch(Exception e) {}
}
}

}

For completeness, here’s the MIDlet class to use to run the above FileConnection-testing class.  These two classes together make a complete MIDlet that can be easily built by creating a new project for it in the WTK:

package net.frog_parrot.test;

import javax.microedition.lcdui.Command;
import javax.microedition.lcdui.CommandListener;
import javax.microedition.lcdui.Display;
import javax.microedition.lcdui.Displayable;
import javax.microedition.lcdui.Form;
import javax.microedition.midlet.MIDlet;

/**
* A simple MIDlet to test which APIs are implemented.
*/
public class FCTest extends MIDlet implements CommandListener {

private Command myExitCommand = new Command(“Exit”, Command.EXIT, 1);
private Command myOkCommand = new Command(“OK”, Command.OK, 1);
private Form myResultScreen;

/**
* Empty constructor.
*/
public FCTest() {
}

/**
* Initialize the Displayables.
*/
public void startApp() {
myResultScreen = new Form(“Results”);
myResultScreen.addCommand(myExitCommand);
myResultScreen.addCommand(myOkCommand);
myResultScreen.setCommandListener(this);
Display.getDisplay(this).setCurrent(myResultScreen);
}

/**
* Implementation of MIDlet.
*/
public void pauseApp() {
}

/**
* Implementation of MIDlet.
*/
public void destroyApp(boolean unconditional) {
}

/**
* Respond to a button push.
*/
public void commandAction(Command command, Displayable screen) {
if(command == myExitCommand) {
destroyApp(true);
notifyDestroyed();
} else {
FCRunner runner = new FCRunner(this);
runner.start();
}
}

/**
* Append a string to the current display.
*/
public void display(String str) {
myResultScreen.append(str);
myResultScreen.append(“\n”);
}

}

SMS Checkers!!!

My SMS checkers game is done and now available!!! Try it out!!!

Just go here to download it free onto any MIDP 2 phone:

http://www.frog-parrot.net/checkers.wml

This version is a big improvement on the original. First of all, it’s a peer-to-peer game — relying entirely on exchanging SMS messages from one handset to the other — so any two people who both have the game installed can play it without worrying about whether a central server is up and running and accessible. The server wasn’t doing much more than routing the game moves from one handset to the other, so it was fairly easy to write it out of the picture.

Each move requires sending a whole SMS, yet the move data typically takes no more than five bytes. It seemed like such a shame to waste the rest of the SMS payload, so I added a “taunt” feature so that you can send your opponent a message along with your move if you like!!!

Another new feature is that the game can now recover from a pause — so you can receive a call and then return to the checkers game in progress.

Additionally I did a bunch of refactoring. The original game logic was great, but the meta-logic — the game state, whose turn it is, etc. — was too intimately tied to the communications/networking code. That makes for smaller code, but the game is already pretty small (16K). So I modularized it a bit, making separate classes for the game state logic and the communications code. Ah, with a personal project you get the luxury of taking the time to refactor the code when there’s a change in requirements! 😉

If you try this game, please feel free to post any comments about it, suggested improvements, etc. Thanks.

I’m sending out an SMS

My latest programming adventure was using SMS (short message service) — that little tiny packet of data that does big things!!!

Every mobile developer should be familiar with this protocol because it is so widely accepted and supported, and because it’s such an important part of the revenue stream, conveniently tied in with the operators’ billing systems. Despite the arrival of more evolved standards like MMS and EMS, plain old SMS is likely to be with us for some time.

The main challenge with SMS is that it fits so little data per message — only 140 bytes of payload!!! So it’s good for sending small things like a URL, a short message, or a signal that the user has paid to use a bonus feature of your application. It’s less useful for sending more data-intensive items (like a whole new board for a game) unless you really optimize. Of course you can send more data by concatenating multiple SMSes, but this has limited utility. Sending too many leads to a bad user experience (costly in time and money). It takes three just to send a tiny favicon, so it’s not terribly practical for sending whole images. It’s more useful for sending signals than for sending content.

The WTK’s “WMADemo” provides a really good basic example of how to write the code to send and receive an SMS. It shows how to register to listen for incoming messages on a Push-Port, how to recognize and read both text and binary messages, and how to open the connection to send them. Text messages are simpler to deal with and can be sent either to an application or to the user’s inbox, depending on the URL used when creating the connection. Binary messages are useful for packing the data in tightly 😉 .

I tried uploading the WMADemo as-is to my Sagem my700x, but got some sort of strange permission error that prevented it from installing. Rather than tracing down the exact culprit, I just stripped down the demo to a version that used SMS only, eliminating the other protocols described in the demo. (I think I may have had to change the port number as well to something within the application port number range: 16000 to 16999, under project > settings > “push registry” and “user defined”.) With that I was able to get my handset to successfully send a message to itself.

The WMADemo shows how to statically request a push-port. That means that the precise push-port number the MIDlet wants to listen on is written in the jad and manifest files. (If you’re looking at the “WMADemo” example, note that the relevant jad attribute is the “MIDlet-Push-i” attribute. The user-defined “SMS-Port” attribute is not standard — it was added by the programmer so the MIDlet could read the push-port from the jad without having to parse the more complicated “MIDlet-Push-i” attribute.)

The advantage to defining the push port in the jad file is that you know in advance what port your MIDlet will be listening on. The disadvantages is that if you have the very bad luck of choosing the same push-port that another installed MIDlet is already listening on (of the 1000 possible choices), then your MIDlet won’t install at all on the handset. The other option is to request the push-port dynamically once the MIDlet is launched (basically keep requesting push-ports in a loop until you find one that isn’t in use). As explained in the MIDP PushRegistry JavaDoc, the effect is the same as static push-port registration in that your MIDlet is permanently registered with the requested push-port (and messages will be routed to the right MIDlet even if it isn’t running). The disadvantage is that you don’t know in advance what the port number will be. On the other hand, you don’t necessarily know in advance the phone number of all of the handsets that have downloaded and installed your MIDlet, so one solution is to program your MIDlet to request to send an SMS back to your server once it’s installed to tell you the relevant phone number and push-port info so you can push messages to your application. Note that receiving a message rings the handset and requires the user to allow your MIDlet to be launched, so obviously it’s important to be careful to send your users only messages they’ve agreed to receive — you can get in trouble for irritating people with spam.

My first home SMS exercise (after making sure that SMS worked from Java on my handset) was to modify the WMADemo to request its push-port dynamically, and then code the port number into a binary SMS and have the recipient handset read the message and reply to the correct port. It wasn’ too difficult — see the MIDP PushRegistry JavaDoc for hints.

My next trick will be to modify the checkers game from my book so that two players can play each other via SMS. The version in the book has the two players playing each other by communicating with a central server on a plain socket. Plain-socket programming is a useful exercise for understanding the theory of how more complex protocols are built on simpler ones (and don’t knock this kind of exercise — protocols change quickly enought that it’s often more useful to learn to learn new protocols than it is to learn one individual protocol 😉 ). However in a real application, using a protocol that is supported by the target handset and operator network is a fundamental concern. As I discovered in my previous blog entry, the operator network is not always willing to route http to an unfamiliar server port number, so the network is will probably like sending plain socket transmissions to a strange port even less…

On the other hand, your SMS is sure to get through. 😀

Troubleshooting blues…

“Why won’t my MIDlet connect with my Servlet?”

It’s such a simple question, isn’t it?

Unfortunately, there are so many layers of communication and so many places to make a little (yet fatal) error, that there’s no simple answer that solves the problem every time.

In my experience, establishing the initial “hello world”-level back-and-forth communication between the client and the server is one of the most frustrating steps in client-server programming. Once you’ve established a complete and correct communication circuit, you’ve got your foundation to build on. Before that — even if you know your networking and protocols well — you’re kind of tossing your message into the black void and hoping it comes out the other side…

But troubleshooting — tracking the problem down wherever it may be — is something every good engineer should be ready and able to do. So let’s talk a little bit about strategies for narrowing down where the problem may lie in a simple case such as a MIDlet that fails to connect with a Servlet via HTTP.

1. Does it work locally, on the simulator?

Getting it running on the WTK simulator is the first step to making sure it isn’t your MIDlet or Servlet that are the problem. Unfortunately, even if it works on the simulator, it still may be that your MIDlet is wrong. Sometimes the WTK is excessively lenient about things that your handset might not like such as reading the body of the message before getting the response code.

If it’s not even working on the WTK, what can you do?

As a general rule, at this point, I would strip my example down to the simplest possible example — a servlet that returns “hello world” and a MIDlet that displays the response code and the message recieved — to reduce the number of places where things can go wrong.

Be sure you know where your server logs are, and check them for clues. Try your URL in your browser and see what the browser returns. If you’re running the client and server on the same machine, you can substitute 127.0.0.1 for the IP address to see if it’s some sort of routing issue. If none of that yields any response from the server, try your server’s “Hello World Servlet” tutorial.

On the client side, note that the WTK has a built-in profiler. You can find it under Edit > Preferences > Monitor > Enable Profiling. This will display for you the raw data exchanged between the client and the server. I’m a little wary of this profiler because it starts some processes that sometimes interfere with installing a MIDlet using Project > Run via OTA for some unexplained reason, but when you really need it, the profiler is useful.

Now, let’s suppose it’s working on the WTK simulator but not on the handset.

2. Can the handset successfully make any HTTP communications from Java?

Generally a handset has a configurable profile which gives information regarding proxy servers and whatnot that tells how to get from the operator’s network to the open Internet. Sometimes the handset has a separate profile for Java and for WAP. They’re usually careful to make sure the WAP profile is right, so if they’re separate, one thing to do is to copy the data from the WAP profile to the Java profile, if you can find the right menus to do it…

One way to check if your Java profile is correct is to try some other working MIDlet that makes a connection via HTTP. For example, I ran the MobileZoo MIDlet the other day, and it succeeded in contacting MobileZoo, so I know that my handset is correctly configured to use HTTP from Java.

3. Is the server accepting requests from the outside?

You know the old story: internal routing is different from external routing, and those lovely firewalls… Try your URL on a computer outside of your local network.

Now once you’ve done all that, it should work!!!

Unfortunately for my “Hello World” MIDlet/Servlet it doesn’t…

I think we’ve narrowed it down to the fact that I’ve set up my server to use a non-standard port for HTTP, and some step in the network chain doesn’t like it.

Hopefully it will be up and running this weekend. I’ll keep you posted!!! 😀