Developer Guide

Last updated January 19, 2004

In addition to inline documentation found in the code for the mentioned platforms, this document will serve to educate future developers of the CPET software package. Our intent is that after reading this document and spending some time with the code, a person can freely modify and enhance the software.

Elsewhere

Contents of This Document

Introduction to CPET Programming

Overview

CPET adds functionality to an existing web course management system (CMS) such as Moodle or WebCT. CPET provides receptacles (think of an electrical wall outlet/receptacle) that can be plugged-in to a particular section of the CMS. The receptacles are interfaces to co-processes that extend the basic capabilities of a CMS. Some example co-processes include a C++ compiler/linker or a Scheme interpreter. Users of CPET can communicate with these receptacles.

A Diagram of the CPET Architecture:

Diagram of the CPET architecture

 

CPET is implemented as a client/server application. The server is currently implemented in Java and executes shell scripts based on the input it receives from the CPET client. The client is implemented as a browser plug-in or extension .

Assuming the software is installed and running, here is a walkthrough of a typical example. A CS professor sets up an online quiz on Moodle and delivers it to his students; a student visits the quiz and submits responses. Here is the example in detail:

 

CPET Protocol

The CPET server is built in Java. It runs on any port (specified by a command line argument). The default port is 39801. The server, when started, listens for incoming connections from the CPET client on the specified port, and executes the command received. The following commands are supported.

Summary of Protocol

commandbrief description
get-receptacles Retrieve lists of receptacles by type
get-structure Get receptacle-specific data needed by a client
engage-receptacle Invoke a receptacle
poll Check that the server is responding
set-parameter Assign a parameter value
get-parameter Retrieve a parameter value
set-question Store information specific to a CMS question
get-question Retrieve information specific to a CMS question
process-question Invoke a receptacle for a CMS question (comparable to get-question and engage-receptacle combined)
login Provide user identification and receive a session ID
new-session-id Obtain a fresh session ID
annotate
  action="get-tree"		 Retrieve the entire tree of annotation keywords and values
annotate
  action="add"		 Add a new group of keyword lists, keyword list, or keyword/value pair
annotate
  action="update"		 Replace a value for a keyword in an annotation keyword list
annotate
  action="delete"		 Remove a keyword/value pair, keyword list, or group of keyword lists
annotate
  action="user-add"		 Add a keyword list to the user's annotation preferences
annotate
  action="user-delete"		 Remove a keyword list from the user's annotation preferences
annotate
  action="get-groups"		 Retrieve all groups of keyword lists
annotate
  action="get-lists"		 Retrieve all keyword lists
annotate
  action="get-user-lists"		 Retrieve all keyword lists in the user's annotation preferences
annotate
  action="get-user-keywords"		 Retrieve all keywords in the combined keyword lists in the user's annotation preferences
annotate
  action="get-user-values"		 Given a list of keywords, retrieve all values for those keywords in keyword lists that appear in user's annotation preferences.

Detailed description of protocol

<get-receptacles type="cmss" />

Retrieves the list of course management systems. Returns empty string if no such receptacles supported.

<get-receptacles type="services" />

Retrieves the list of supported services (e.g. Scheme, C++). Returns empty string if no such receptacles supported.

<get-structure receptacle="Moodle" />

Retrieves specific information regarding a specific CMS (this information is used when adding elements to the DOM tree, etc., described below), replace "Moodle" with the appropriate CMS name. Returns empty string if no such structure is available.

<engage-receptacle type="Scheme" code="(+ 1 4)" />

<engage-receptacle type="..." ... />

Engages the specified receptacle with parameters specified as receptacle-specific attributes. In the first example, the server will reply with output "5" from the executed command. See receptacle documentation for the attributes expected or permitted by each receptacle.

<poll />

Check that the server is responding. The server will reply with the string "ACK" (if it is running correctly).

<set-parameter type="user" id="smithx" name="timeout" value="100" />

Enter a name-value pair for the indicated type and id combination. The attributes type, id, and name have alphanumeric values. If there was no prior value for the combination (type, id, name), a new entry is created; otherwise, the prior value is overwritten. Return is the label "Prior: " followed by the prior value, or followed by "NO_VALUE" if there was no prior value.

<get-parameter type="user" id="smithx" name="style" />

Retrieve and return a value associated with the indicated name, for the indicated type and id combination. If no such value exists, the string NO_VALUE is returned.

If some or all of the attributes for get-parameter are omitted, those attributes are treated as wildcards, and all matches for the included attributes are returned. For example, if only type and id attributes are present in a get-parameter query, all available name and value combinations are returned (except NO_VALUE is returned if no such combinations were found). Returned fields are separated by tab characters, and records are terminated by a newline; a line of field labels precedes the record lines.

<set-question root-url="moodle.stolaf.edu" qid="314" receptacle="Scheme" cpet-version="1" info="before=??&after=?(car+lis)?" />

Stores a question with the specified qid and root-url. Cpet-version is "1" until incompatible features arise. The info field can be used for anything the client wishes; in this example it uses "before" and "after" to store input that will be executed before and after the student's response. If the qid is blank (i.e., the qid is unknown at the present time), the server stores the question with a temporary qid, and will be updated at a later time when the proper qid is determined.

<set-question root-url="moodle.stolaf.edu" qid="qtmp314" new-qid="715" />

Copies the values associated with a temporary question ID (e.g., qtmp314), associating an actual question ID (e.g., 715) with the copied values. Both question IDs will be recognized for a subsequent period of time (e.g., 24 hours); thereafter, the values for the temporary question ID will be deleted, and only the values for the actual question ID will remain.

<get-question root-url="moodle.stolaf.edu" qid="314" />

Retrieves the question at the specified root-url and qid. Returns empty string if no such question.

<process-question root-url="moodle.stolaf.edu" qid="314" code="(+ 1 4)" />

Checks whether the specified root-url and qid are stored in the CPET database. If so, that question's receptacle is engaged with the indicated code, and the server replies with the output (e.g., "5" if this is a CPET Scheme question); this case is equivalent to client calls of get-question and engage-receptacle. Otherwise, the server replies with the string "NON-CPET".

<login user="mueller" />

<login user="mueller" persistent="true" />

Records the indicated username and associates it with a fresh session ID. Returns that session ID, a string to be passed in subsequent protocol messages where required. The second form causes the socket to be marked as persistent.

<new-session-id old-session-id="..." />

Generates a fresh session ID and records it for the same session as the session ID provided, but with a newly calculated expiration date. Returns that new session ID. The session may be accessed with either IDs until the first one expires. Returns string "INVALID_SESSION_ID" instead if the old-session-id value is not currently recorded for a session, e.g., if it is expired.

<annotate action="get-tree" session-id="..." />

Return the entire keyword tree, including values for keywords. The tree is transmitted as a sequence of group names, list names, keywords, and values, in the order of a depth-first tree traversal using the following strings as delimiters.

  • "#@#" separates groups

  • "#!#" terminates a group name before a list name

  • "$@$" separates lists within a group

  • "$!$" terminates a list name before a keyword

  • "&@&" separates keywords within a list

  • "&!&" terminates a keyword before a value

  • "*@*" separates values for a keyword (needed for <annotate action="get-user-values" .../>).

Example: "chemistry#!#chem125$!$entropy&!&val1 #@#physics#!#phys126$!$gravity&!&val2&@& momentum&!&val3$@$phys127$!$entropy&!& val4&@&momentum&!&val5" represents the following tree.

Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="add" session-id="..." group="chemistry" list="chem125" keyword="entropy" value="HTML text" />

<annotate action="add" session-id="..." group="chemistry" list="ions" />

<annotate action="add" session-id="..." group="neuroscience" />

Add a new group, list, or keyword-value pair the annotations database. The first form adds a keyword-value pair to a keyword list: HTML text will be returned for subsequent <annotate action="get-user-values" .../> requests involving that keyword. HTML text must be encoded using CGI-style % codes for special characters. The indicated group and/or list is created if it doesn't already exist. Server replies with an error message and performs no database changes if that keyword already exists in that group and list; use <annotate action="update" .../> to change a keyword's value.

The second form adds a new keyword list to the indicated group. If that group doesn't exist, it is created also. The third form adds a new group for keyword lists. For both of these forms, no action is performed if the indicated group or list already exists.

Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID. No database changes occur if attributes are missing from one of the three forms. For example, if the value is missing in the first form, no group or list is created even if those group or list names are new.

<annotate action="update" session-id="..." group="chemistry" list="chem125" keyword="entropy" value="replacement" />

For the specified group and list, replace the value of the indicated keyword with the replacement text. Server replies with an error message and performs no database changes if that group/list/keyword combination does not already exist. Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="delete" session-id="..." group="chemistry" list="chem125" keyword="entropy" />

<annotate action="delete" session-id="..." group="chemistry" list="chem125" />

<annotate action="delete" session-id="..." group="chemistry" />

Remove a group, list, or keyword-value pair from the annotations database. The first form removes the keyword and value for the indicated group, list, and keyword. Server replies with an error message and performs no changes if that group/list/keyword combination does not already exist.

The second form removes an empty keyword list from the indicated group. Server replies with an error message and performs no changes if that group/list combination does not already exist or if that keyword list is not empty.

The third form removes an empty group for keyword lists. Server replies with an error message and performs no changes if that group does not already exist or if that group is not empty.

Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="user-add" session-id="..." group="chemistry" list="chem125" />

Add the indicated keyword list to the annotation preferences for the user associated with the session ID provided. An error message is generated and no changes are made if that keyword list already appears among the user's preferences. Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="user-delete" session-id="..." group="chemistry" list="chem125" />

Remove the indicated keyword list from the annotation preferences for the user associated with the session ID provided. An error message is returned and no database change occurs if that keyword list does not appear among that user's preferences. Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="get-groups" />

Retrieve all groups of keyword lists. Groups in the server's reply are separated by the group separator delimiter used for <annotate action="get-tree" .../>.

<annotate action="get-lists" group="chemistry" />

<annotate action="get-lists" />

Retrieve all keyword lists. In the first form, where a group is specified, only keyword lists for that group are provided, and these are separated by the list separator delimiter used for <annotate action="get-tree" .../>. In the second form, where no group is specified, all group/list combinations are returned, using delimiters for <annotate action="get-tree" .../>.

<annotate action="get-user-lists" session-id="..." />

Retrieve names of all keyword lists with their groups chosen by the user associated with the session ID provided. Group and list names in the server's reply are separated by the delimiters used for <annotate action="get-tree" .../>. Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

<annotate action="get-user-keywords" session-id="..." />

Retrieve a list of all keywords in all keyword lists chosen by the user associated with the session ID provided. Keywords in the server's reply are separated by the word separator delimiter used for <annotate action="get-tree" .../>. Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID. Note: Use actions user-add and user-delete to modify the keyword lists associated with that user.

<annotate action="get-user-values" session-id="..." keywords="keyword sequence" />

For all keywords appearing in the indicated keyword sequence, find all values associated with those keywords among the keyword lists chosen by the user associated with the session ID provided. The chosen keywords are those appearing in the value of the parameter keyword-lists for that user. Delimit the keywords in the keyword sequence by the separator for keywords within a list used for <annotate action="get-tree" .../>. The server's reply provides a list of values for each keyword in the keyword sequence, using the same delimiters as for <annotate action="get-tree" .../>.

Example: If the attribute keywords has the value "entropy&@&gravity", the server might reply with "entropy&!&value1*@*value2&@& gravity&!&value3", which supplies two values (from two different keyword lists) for the keyword entropy and one value for the keyword gravity.

Server replies with the string "INVALID SESSION ID" if no user is associated with the supplied session ID.

 

CPET for Internet Explorer

CPET for Internet Explorer is implemented in C# as a Browser Helper Object, which is a little program that hooks into the IE interface to catch events like page loads and mouse-click events. Usually Browser Helper Objects are implemented to be Spyware or Adware, but they can be benevolent as well, e.g. the Google toolbar. More information about Browser Helper Objects can be found in the Resources section at the end of this document.

One thing to note about CPET in IE is that it attempts to reuse some of the Javascript code that is used in Mozilla, so that we do not need to code the same things twice in different languages. However, IE does not allow Javascript to return a value to a C# program that has called it, so it is problematic use Javascript for all of our needs. This means that we need to do some things like parse the DOM tree in C#. This is one of the major roadblocks in developing for IE. The interfaces that need to be implemented for interacting with Internet Explorer and the Document Object Model are poorly documented and confusing at best.

The CPET client for IE is implemented with the .NET environment in Microsoft Visual Studio. The CPET client is built as one project, with the CPET Helper (used for setting user preferences) is built as another. The outputs of both are inserted into a third installer project, which packages the entire solution for deployment.

When CPET is installed, the routines in AssemblyInfo.cs registers the CPET BHO, adding values to the registry that notify IE of the new BHO. When a new Explorer window opens (IE or Windows Explorer), the window notifies the BHO of events via the IObjectWithSite interface and its GetSite and SetSite methods. CPET catches "OnDocumentComplete" events and "BeforeNavigate2" events.

When an OnDocumentComplete event occurs, the BHO passes a reference to the new document to an instance of the CpetMain class. CpetMain searches the location field of the window and decides whether the document it received is a document that should be adjusted by other CPET routines (by comparing it to preferenes set in the registry), at which point it passes the document to the method DoFrame. DoFrame scans the document for frames, and for each valid frame, passes the document in the frame to either the DoAuthor or DoQuiz methods, depending on the URL of the document.

DoAuthor handles the authoring portions of CPET, allowing quiz authors to add CPET functionality to their questions. (This is done by inserting DOM nodes for extra form fields with Javscript and a call to doc.parentWindow.execScript(...).) When this new form is submitted, we need to process the updated fields with the client, and store the appropriate question data at the CPET server. There is a bug in the .NET submit handler, so we developed a workaround that utilizes the BeforeNavigate2 event. Just before a navigation event occurs, the BeforeNavigate2 event is called, and the parameters passed to the event include the target URL, any HTTP post data (i.e., form data), a boolean "cancel", and a few other objects. We can cancel the Navigation event by setting "Cancel = true;". Then we can parse the post data for values that the user inputted into the CPET form fields, and send those off to the server in the form of a "set-question" command (see below for more information). We then re-format the post data to remove the CPET-only fields and send a new Navigate2 event.

The DoQuiz method processes the quiz page, and sets up the page to be processed later by the CPET server. DoQuiz scans the quiz page and sets some state variables in the CpetMain object that will later be recalled when the BeforeNavigate2 event for the quiz submission occurs. For each question, the client queries the server to see whether that question requires CPET processing, and sets the state variables accordingly. When the BeforeNavigate2 event occurs, these state variables are queried, and any matching questions in the PostData string of the BeforeNavigate2 event parameters are sent to the CPET server for processing. The responses from the server are incorporated into a modified PostData string, and that string is passed to a new Navigate2 event.

The CPET Helper program is an independent executable that sets user preferences in the Registry. These preferences include which host and port the CPET server is located at, and which URLs CPET should be active for (e.g., http://moodle.stolaf.edu). The CPET Helper program is implemented in C++ using Windows Forms.

 

CPET for Mozilla

How to Program for Mozilla

The CPET plugin for Mozilla is built using Javascript, XML (XUL), and CSS. Javascript performs the actions and holds the core functionality for the plugin. XML is used for layout of windows (such as the "CPET Options" window), and CSS is used for styling (e.g. font sizes and colors).

The Mozilla development environment is surprisingly easy to program with. It was designed for Rapid Application Development (RAD). In fact, the Mozilla browser itself is just built on top of the XPCOM interfaces. If you analyze the chrome/ subfolder of the Mozilla folder, you will find XML documents that describe the layout of toolbars and windows, and a bunch of Javascript that is executed to load HTML pages, etc.

Javascript

The Javascript used by Mozilla is an enhanced version of the Javascript commonly available to developers for websites. In addition to supporting basic math, I/O, and DOM manipulation functions, Mozilla's Javascript allows a programmer access to low-level operations like file read/write and networking. This is achieved by allowing Javascript to access the low-level C++ functions that Mozilla itself uses to run. The interface is called XPCOM, or Cross-Platform Component Object Model. Microsoft uses a similar COM (COM+ or .NET) to provide access to Windows functions. You can think of the COM as a set of method definitions in a header file: they tell you what you can do with classes but you don't have to worry about how any of the underlying structure works.

Some of the CPET program relies on these XPCOM interfaces, such as the network connection. But much of CPET requires extensive parsing of the HTML DOM tree. When Mozilla receives an HTML page, it first stuffs it into a "tree" data structure that can be easily traversed by Javascript functions--this is called a DOM (Document Object Model) tree. It is important to be comfortable with these Javascript functions. Most of them can be experimented with in a sample HTML document loaded into Mozilla. More information can be found at http://developer.apple.com/internet/webcontent/dom2i.html

It is also important to know about regular expressions in Javascript, since these are what CPET uses to store and parse information. Regular expressions are basically special strings that are used to match patterns in other strings. For example, the expression [0-9] will match one instance of any digit between 0 and 9 in a string: the string "car3" will be matched at '3'; the string "car43" will be matched at both characters '4' and '3'; and the string "car" will not match at all. Detailed information about the syntax and construction of regular expressions in Javascript can be found by many places on the Web.

XPCOM and XUL

The easiest way to become acquainted with Mozilla development environment is to look at existing examples. http://www.mozilla.org/docs/tutorials/tinderstatus/ provides an introductory tutorial.

A lot of information about the XPCOM interfaces is available at http://www.xulplanet.com/ . This site provides some good tutorials about building GUIs with Mozilla, in addition to also outlining in great detail the COM interfaces and how to use them. Specifically:

Another invaluable resource for developing with Mozilla is Nigel McFarlane's book Rapid Application Development With Mozilla . It is available for free download in several formats. They can be found at http://www.mozillazine.org/talkback.html?article=4600 .

If all else fails, it never hurts to post a question to the Mozilla developer forums, which are found at http://forums.mozillazine.org .

 

Setup of the CPET Client in Mozilla

At the time this document was drafted, there was no XPI installer for the CPET client. If an XPI installer is developed, it will do these steps for you. If not, you need to follow these steps:

Structure of the CPET Client in Mozilla

The CPET client is essentially a collection of functions that are associated with specific events in a web page (such as the onLoad event or the onClick event). Generic information that needs to only be loaded once per Mozilla session is stored in a centralized object I like to call the hub. The hub stores information such as how to connect to the server or which URLs should be associated with CPET (so that CPET doesn't run on every HTML page, but only on the ones we know are Moodle or WebCT pages).

Walkthrough of CPET in Mozilla

CPET is started whenever Mozilla is started. (The contents of the contents.rdf allow us to tell Mozilla to execute our CPET Javascript when Mozilla starts.) When Mozilla is first loaded, the CPET plugin queries the CPET server for information regarding the latest status of supported receptacles and Course Management Systems (this happens in common.js). For example, the server might respond with a message saying that it supports Scheme and C++ receptacles, and Moodle and WebCT. This information is stored in the CPET_hub object. The other function called when CPET is started is addEventListener . This function sets up a listener for pageLoad events, and passes a callback function called CPET_Initialize (this is called from cpet.js ). So when pageLoad event occurs, CPET_Initialize is called.

The CPET_Initialize function looks at the URL of the page being displayed and determines whether it is a site that is supported by CPET (these sites are specified by the user and stored in user prefs). There is a little tricky business here because we must also look for frames in the HTML document. Each frame in a page contains a different HTML page and has a different URL associated with it. Frames are not an issue in Moodle, but WebCT uses them extensively. CPET_Initialize , if it determines that the page loaded is indeed in a site supported by the CPET client, checks to see which type of page is being displayed. For the purposes of the code, we specify these as the authoring page (what a professor sees) or the quiz page (what a student sees). Depending on which page is found, an appropriate function is called to process that page.

In the case of the authoring page, we call CPET_OverrideAuthoringPage . (This function is located in crossp.js ). It does three things: 1) adds form elements (e.g. drop-down menus) to the page to provide a way to input information about which CPET receptacle to use with which question; 2) retrieves any previously stored information about receptacles and populates the form elements accordingly; 3) sets up submit handlers with a callback function for when the user clicks the "submit" button in the form. Some elaboration on each of these steps:

Adding form elements. This is performed by the function CPET_AddNodesToDom . It takes as arguments the current document, the HTML code to be inserted into the page (this contains form elements like drop-down menus), the position at which the HTML should be inserted, and the browser type (because IE and Mozilla have slightly different behaviors and this function attempts to be cross-platform). CPET_AddNodesToDom parses the HTML string to be inserted and makes a mini DOM tree out of it. The function then scans through the HTML page for the desired insertion location (which is specified by a specially formatted string) and appends this new mini DOM tree to the existing document's DOM. When this happens, Mozilla immediately redraws the document so that it has the new HTML elements in place.

Retrieve previously stored information. This is performed by the function CPET_GetReceptacleType . It takes two arguments, the current document and the id of the textarea containing the question text. This function looks through the characters in specified textarea to see if any CPET information was stored with this question previously (CPET information is added to the question text in the form of an HTML comment). If any information did exist, it is removed from the textarea, and it is inserted appropriately into the form elements that were added in the previous step. For example, step 1 above has added a drop-down menu to the page that asks the user which CPET receptacle should be associated with this question; the drop-down menu has two options: C++ and Scheme. The user has previously specified this question to use the Scheme receptacle, so that value has been stored with the question text. It is removed from the question text and the Scheme option in the drop-down menu is selected.

Set up submit event handlers. This is performed by the function CPET_ListenForAuthoringSubmit . It takes two arguments: the current document and the id of the textarea containing the question text. This function sets up a callback function (when a submit event happens, the callback function is called). This is a bit complicated with WebCT because it uses Javascript to perform form submission, instead of the traditional form submit button. And unfortunately, Mozilla's handling of both of these ways of submission is different. So we actually have to set up two callback functions. One of them catches Javascript submits, the other catches regular submits. When the form is submitted, the function CPET_ProcessAuthoringSubmit is called. This function stores question information (receptacle type, etc.) on the CPET server by passing a "set-question" command (see above). Then it deletes the nodes from the DOM tree that were added in step 1, so that the course management sytems don't become confused by additional form fields.

So that outlines what happens on an authoring page. Now let's look at what happens on a quiz page. The function CPET_OverrideQuizPage is called, which in turn performs the necessary operations on the quiz page. This function essentially does two things: 1) retrieves information from the question text that indicates which questions have which receptacles associated with them, etc., and adds this information in the form of HTML attributes to the answer boxes (text fields) for easy retrieval later; 2) set up form submit event handlers. Here are the steps in more detail:

Thus completes the walkthrough.

 

Other Notes

Throughout this document I have used the example of a CPET receptacle as being the one option available to professors wishing to add enhanced functionality to their online course materials. In fact, CPET currently supports two other enhancements: the ability to add code before or after a student's code. So for example, a professor wishes to setup a black-box testing program. He writes a function which is hidden from the student (in the "execute this code before the student's code" box on the authoring page). The student need only write a function call to that function, maybe passing certain values that test the boundaries of the function. Then when the CPET server processes the student's answer, it first executes the function definition specified by the professor. More support for this sort of enhancement is expected.

 

Future Work

 

Resources

Javascript

Mozilla

Internet Explorer

Department of Mathematics, Statistics and Computer Science
St. Olaf College
Northfield, Minnesota
© 2004-2005