We present the research framework for a taxonomy of online educational peer-assessment systems. This framework enables researchers in technology-supported peer assessment to under- stand the current landscape of technologies supporting student peer review and assessment, specifically, its affordances and constraints. The framework helps identify the major themes in existing and potential research and formulate an agenda for future studies. It also informs educators and system design practitioners about use cases and design options.

Peer assessment is widely used at all levels of education. Students give and receive feedback from their classmates, and thereby produce a wealth of information that can potentially be used to improve the assessment process. But thus far, each online peer- assessment system has been an entity unto itself. There has been no attempt to compare the approaches taken by such systems, for example, the rubrics or the structuring of the assessment process. Our PeerLogic project is an attempt to change that. We are constructing a data warehouse of millions of peer reviews, from at least half-a-dozen systems, that can be mined to determine how differences in the assessment processes translate into differences in peer assessments. This paper reports on some of the issues that arise in the construction of the warehouse, and how we have resolved them in a way that will work for all constituent systems. We also presented an example of comparing data coming from two systems that are based on rating and ranking.

This paper explores the use of rear projected fabric panel tangible interfaces for use in music performance, interactive sculpture, and experiential systems. This idea is explored using the piece What We Have Lost / What We Have gained as an example. This paper demonstrates how HCI can be applied to and included within art disciplines to increase engagement with the artworks by transforming viewers into performers, participants, players, and co-creators. It further argues that by including embodied interactions artworks expand their ability to convey meaning to users.

Advances in online and computer supported education afford exciting opportunities to revolutionize the classroom, while also presenting a number of new challenges not faced in traditional educational settings. Foremost among these challenges is the problem of accurately and efficiently evaluating learner work as the class size grows, which is directly related to the larger goal of providing quality, timely, and actionable formative feedback. Recently there has been a surge in interest in using peer grading methods coupled with machine learning to accurately and fairly evaluate learner work while alleviating the instructor bottleneck and grading overload. Prior work in peer grading almost exclusively focuses on numerically scored grades – either real-valued or ordinal. In this work, we consider the implications of peer ranking in which learners rank a small subset of peer work from strongest to weakest, and propose new types of computational analyses that can be applied to this ranking data. We adopt a Bayesian approach to the ranked peer grading problem and develop a novel model and method for utilizing ranked peer-grading data. We additionally develop a novel procedure for adaptively identifying which work should be ranked by particular peers in order to dynamically resolve ambiguity in the data and rapidly resolve a clearer picture of learner performance. We showcase our results on both synthetic and several real-world educational datasets.

This paper describes the Digital Culture BA degree: an engineering-arts undergraduate curriculum that combines competency-based education (CBE) and knowledge-oriented education (KOE) structures and related Pull-Push approaches. The degree has been offered for three years at Arizona State University, has 200 enrolled students and is continuing to grow. The degree embeds nine knowledge-oriented concentrations, each offered by a relevant participating department, within an interdisciplinary CBE context. The CBE part of the degree provides customized access to 40 interdisciplinary digital culture courses from 12 different academic units by connecting these courses through a set of core competencies. Access to courses is not determined by fixed prerequisites but rather by having one of several possible combinations of lower level competencies. This flexible curriculum is attractive to students, promotes integrative collaborative learning that inspires innovation, and prepares the type of engineering-arts experts and complex problem solvers that are currently needed in creative industries. This type of degree also presents several important challenges for educators and administrators. To address these challenges we developed project based assessment approaches, custom web-based software for advising a very diverse student body, as well as online tools for facilitating peer critique and feedback in large creative classrooms.

Emergent order in complex dynamic systems is a concept that is both important to a wide variety of fields of science and difficult to understand and teach. Recent advancements in interactive and tangible interfaces point towards the possibility of new tools which can help with both the analysis and simulation of complex systems, while also making the systems dynamics more approachable and easier to understand for students. This paper describes an experimental tangible interactive interface designed to make the complex and emergent behavior of ant colony foraging more understandable. We present the system design, its interactive capabilities, and discuss evaluation methods for further studies.

Children and young adults are immersed in digital culture, but most are not familiar with the computational thinking behind the latest tools and technologies. There are few opportunities in secondary school curricula for students to learn such practices, but we believe that skills such as computational thinking, creative coding, collaboration, innovation, and information literacy can be taught in a highly effective manner by using aesthetic challenges as a motivation. In other words, by engaging students in creative digital arts projects they are naturally driven to acquire the many new skills to effectively use and understand the computational tools and techniques involved in creating digital and interactive projects. In this paper, we outline a project-based digital arts curriculum through which novice middle/high school students are intrinsically motivated to learn and apply science, technology, engineering and mathematics (STEM) skills and computational thinking.

Critviz is an experimental web-based tool created to support peer critique in large classrooms. Critique sessions are an effective tool to use in the classrooms where students’ work is not so clearly “right” or “wrong” and when students need detailed critical feedback in order to clarify their direction. Face-to-face critiques work well in smaller groups where all can have their work seen and discussed. CritViz’s development has been driven by the question of whether a critique can be scaled up to larger classes of 100 students or more. Through CritViz’s countdown timers, assignment uploading, and randomized peer ranking, we can effectively scale the use of critiques to much larger classes and change the “motivational structure” of the students who use it, resulting in increases in student learning, motivation, and student reflection on work. Folk wisdom suggests that when it comes to class size, “Smaller is Better,” but our tool points toward possibilities for increasing class sizes with little negative impact, even leading to a “Bigger is Better” phenomenon in the classroom.

CritViz is an online framework for supporting real- time critique, conversation, and peer ranking of creative work in the classroom. In creative classes where student work cannot be entirely graded on objective criteria, classes often arrange critique sessions to provide direction and feedback to students, raise their level of performance, and teach them to give and receive constructive criticism. Critiques usually work best in classes small enough to sustain a single discussion. CritViz was created to scale up critiques for large classes whose size normally prohibits this activity. Through CritViz’s countdown timers, assignment uploading, and randomized peer feedback, we can now run effective critiques in much larger classes and have seen changes in overall classroom “motivational structure.” Typically with classrooms “Smaller is Better” but our work implies possibilities for increasing class sizes with little negative impact, and even leading to “Bigger is Better” in the classroom.

In the 21st century knowledge economy there is a growing need for the types of creative thinkers who can bridge the engineering mindset with the creative mindset, combining multiple types of skills. New economies will need workers who have "diagonal" skill sets, who can develop systems and content as an integrative process. This requires a new type of training and curriculum. In the newly formed "Digital Culture" undergraduate program at ASU, we attempt to support new types curricula by structuring differently the way students move through courses. With a constantly shifting and changing curriculum, structuring course enrollment using class "prerequisites" leads to fixed and rigid pathways through the curriculum. Instead, Digital Culture structures course sequences based on the students accumulation of abstract "Proficiencies" which are collected by students as they complete courses, and which act as keys to unlock access to higher level course. As a student accumulates more and more of these proficiencies, they are increasingly able to unlock new courses. This system leads to more flexible and adaptive pathways through courses while ensuring that students are prepared for entrance into more advanced classes. It is however more complicated and requires that students strategically plan their route through the curriculum. In order to support this kind of strategic planning we have designed and deployed a course planning system where students can simulate various possible paths through the curriculum. In this paper, we show our design process in coming up with our "Digital Culture Visual Planner". This design process starts with a network analysis of how all the Digital Culture courses are interrelated by, visualizing the relationships between "proficiencies" and courses. A number of possible design directions result from this analysis. Finally we select a single design and refine it to be understandable, useful and usable by new undergraduate Digital Culture majors.

Contemporary distributed enterprises present challenges in terms of demonstrating community activity awareness and coherence across individuals and teams in collaborating networks. Building with a Memory is an experiential media system that captures and represents human activity in a distributed workplace over time. The system senses and analyzes movement in two workspaces in a mixed-use building with the results rendered in an informative ambient display in the building entryway. We describe the design and development of the system, together with insights from two studies of the installation and promising future directions.

The Interaction Stations Exhibit features screen-based, interactive works that align with the new conference themes, and integrate into the physical setting of the conference center. In this paper we describe our motivation for this format, as well as the works selected and larger connections.

We present an overview of interactive art that reflects human activity in physical space. We discuss the varied approaches of these pieces and present our own work, Building with a Memory, which is situated in a workplace and builds on this prior work to provide an informative and aesthetically satisfying experience.

Building with a Memory is an experiential media system for capturing and representing dynamic human activity in a distributed workplace. Participants can perform purposefully, incidentally and unknowingly with the system, with individual and collective actions represented through a variety of ambient feedback mechanisms. Two iterations of the system are presented and observations and recommendations for performative workplace media systems described. In this paper we discuss the performative aspects of Building with a Memory, an experiential media system for capturing and presenting patterns of human activity in a transdisciplinary workplace. The indirect mapping between performance and feedback challenges how we consider performativity in the piece.

Thrii is a multimodal interactive installation that explores levels of movement similarity among its participants. Each of the three participants manipulates a large spherical object whose movement is tracked via an embedded accelerometer. An analysis engine computes the similarity of movement for each possible pair of objects, as well as self-similarity (e.g., repetition of movement over time) for each object. The extent of similarity among the movements of each object is communicated by a visualization projected on a three-sided pyramid, a non-directional audio environment, and lighting produced by the spherical objects. The installation’s focus is intended to examine notions of collaboration between participants. We have found that participants engage with Thrii through exploration of collaborative gestures.

Building with a Memory is a subtle responsive intervention that aims to provide cohesion and community awareness through the use of light and color. The installation delivers thought-provoking information by capturing, analyzing and rendering real-time and archived human activity in a workplace setting. The installation senses movement in the space through an IR camera and computer vision techniques. Two custom lighting fixtures and a video monitor render the aggregated movements. The visually simple aesthetic of the piece aims to balance active engagement and passive contribution, providing a rewarding experience for both occasional passersby and regular users of the space. This paper describes the motivations and contributions of the installation, together with insights gained from an informal evaluation and directions for future explorations.

As e-mail has become the preferred medium for communication, the inbox and mail folders become one hub for organizing activities and schedule. The combination of a natural rise in message volume and the large amounts of unsolicited bulk messages (spam) have led some to suggest that the usefulness of e-mail is at an end; users feel overwhelmed by the sheer numbers of individual messages and the effort to manage the inbox. This paper frames the issue of e-mail/message overload as a specific example of data overload and uses previous results to suggest three design concepts – cognitive buoyancy, e-mail constellations, and the intelligent subject line – for use in e-mail.

Patterns of responses to “message overload” can be seen in the ways in which people adapt messaging systems and capabilities. Blogging is an effective and increasingly popular decentralized form of group communication that is proving useful in helping people find and share what is informative. We look to blogging for clues to new solutions to the problem of data overload in the world of email. These design solutions to email overload go beyond efforts to block spam, and are based on shifting the basic unit of organization toward communication relationships that allow patterns in communications to emerge.

This paper describes how a Cognitive Task Analysis (CTA) was used to jumpstart the exploration of useful design aids to combat data overload in intelligence analysis. During a simulated analysis task, we observed how professional intelligence analysts were vulnerable to making inaccurate statements when they were under time pressure and working in a topic area outside their area of expertise. From these observations, we generated design recommendations and criteria for evaluating the usefulness of any effort aimed at reducing data overload. Then, we used CTA insights to trigger the development of modular design concepts, or “design seeds,” that leverage advances in machine processing to address vulnerabilities. Nine design seeds were integrated into a “Visual Narratives” workspace visualization concept. Feedback about the usefulness of the design seeds was obtained during an elicitation session following an animated fly-through, or “Ani-mock,” demonstration.

This report details a complete, beginning-to-end Cognitive Systems Engineering (CSE) project tackling the challenges of conducting intelligence analysis under the condition of data overload. We first reviewed and synthesized the research base on data overload from multiple complex, high-consequence settings like nuclear power generation. The product of this activity was a diagnosis of the challenges of dealing with data overload in general. Then, leveraging this research base and previous experience in conducting Cognitive Task Analysis (CTA), we conducted a study to identify the aspects of this research base that applied to intelligence analysis as well as unique challenges. We observed expert intelligence analysts conducting an analysis on a selected unclassified scenario, the 1996 Ariane 501 rocket launch failure, with a baseline set of tools that supported keyword search, browsing, and word processing in an investigator- constructed database. From this study, we identified challenging tasks in intelligence analysis that leave analysts vulnerable to making inaccurate statements in briefings when they are working in a new topic area and are under short deadline constraints. In parallel, we identified limitations of the baseline tools in addressing these vulnerabilities that pointed to ideas for new design directions. In addition, the study findings were translated into objective criteria for evaluating the usefulness of any effort aimed at reducing data overload. In the final phase of the project, we shifted from an emphasis on problem definition to an emphasis on developing modular design concepts, or “design seeds,” that could be incorporated into both ongoing and future design efforts. The design seeds were instantiated as animated fly-through mockups, or “Animocks,” so that feasibility of certain usability considerations, such as the ability to display data in parallel in an easily interpretable form, could be explored without being forced into committing to a particular design.