Sketchable Interaction (SI) is a concept for a generic graphical environment that facilitates end-user customization of user interfaces and visual programming. Within an SI context, all graphical objects, including canvas, icons, and the mouse pointer, are interactive regions with arbitrary shapes. Users can draw further interactive regions with their mouse cursor, fingers or stylus, creating a sketch of a user interface. All interactive regions may contain effects which are applied to other regions once those overlap or touch. These effects can modify data, graphical representation, or both, of other regions. For example, a “Delete” region will delete all documents that are dragged onto it. Additionally, regions have properties such as position, scale, or color which can be linked to arbitrary properties of other region effects. For example, while dragging a region, the position of that region is temporarily linked to the position of the cursor’s region. These two principles allow users to iteratively construct user interfaces for a variety of different tasks, such as document management, data processing, or window management. Sketchable Interaction can also function as a visual programming language via special conveyor belt regions which transport other regions (e.g. documents) along their path. While in transit, these regions can pass through other regions which apply their effects to the document. Conveyor belts can also be used as conditional branches, allowing for simple application logic to be implemented via SI concepts. A C++ runtime handles collision and linking. All interactive regions and their behaviors are implemented via Python plugins.

We have completed a robust implementation of the runtime and are currently exploring novel application supported by combinations of simple tools.