AR Arduino
Project Assistant

Contextual inquiry

Our team observed people working in Cornell Tech’s MakerLAB Electronics Room and conducted simple interviews, supplementing the fieldwork with our prior Arduino prototyping experience.

Beginners spent time browsing labeled bins without knowing which sensor or component suited their project. During assembly, they repeatedly shifted attention between a screen and the physical breadboard—losing their place, misreading pins, or connecting the wrong row. Components were also frequently left at workstations or returned to the wrong bin.

Key pain points

• Component uncertaintyBeginners relied on trial and error or external help.
• Context-switchingLooking between a 2D diagram and the workspace caused mistakes.
• Labeling gapsBin labels did not always distinguish similar-looking parts.
• Hands occupiedHolding components made another device inconvenient to control.
• Disorganized storageMaterials were not consistently returned to labeled locations.

Users & stakeholders

Primary users are students and makers with limited to intermediate electronics experience working on Arduino- or Raspberry Pi-based projects. Secondary stakeholders include more experienced student makers, course instructors, and lab staff who benefit when users can work more independently and return materials consistently.

Problem scenario

Emily moves between a laptop diagram and a physical breadboard while assembling a temperature-sensor circuit. One misplaced wire produces no readings, so she initially assumes the code is wrong. After finishing, she cannot confidently return unfamiliar components to the right locations. This constructed scenario illustrates the cost of context-switching and unclear inventory; its timing is narrative, not measured data.

UX goals

• Goal A

  Easy to learn, minimally disruptive. Voice and simple gestures should keep attention on hands-on work.

• Goal B

  Reduce setup time and errors. Recommendations and spatial steps should simplify finding and wiring parts.

• Goal C

  Build confidence. Clear confirmation and corrective guidance should reduce beginner anxiety.

• Goal D

  Bridge digital knowledge and physical action. Keep users’ hands free and eyes on the workspace.

• Goal E

  Maintain lab organization. Shelf guidance and return prompts should support the next user.

Functional requirements

Performance & interface direction

The intended system should respond with minimal latency, keep instructions legible in a peripheral AR view, avoid obstructing the workspace, highlight active pins and breadboard rows, confirm completed steps, present actionable errors, and show shelf-location overlays after a QR scan. The prototype did not implement every capability.

Design rationale

A static heads-up display still required users to translate flat images into physical action, so we prioritized spatial anchoring directly on the breadboard. Voice was favored over touch because users’ hands are often occupied. QR-based shelf overlays offered a more reliable inventory direction than complex room mapping.

Interaction model

• Voice commandsDescribe a project goal without interrupting physical work.
• Gestural navigationMove through discrete wiring steps with simple next and back gestures.
• Multimodal confirmationCombine high-contrast overlays with audio cues for success or errors.
• Spatial anchoringConnect instructions to physical pins and breadboard rows.
• Indirect manipulationReposition supporting panels without blocking the primary work area.

Storyboarding

The storyboard follows Emily through a temperature-sensing project—from requesting help and locating components to wiring, recovering from an error, completing the project, and returning materials. It illustrates the broader concept, including capabilities not implemented in the later pushbutton LED prototype. Images were generated using ChatGPT and manually refined by the team.

Focused spatial prototype

We built a mid-fidelity ShapesXR prototype, using VR to simulate a future AR-glasses experience inside a recreation of the MakerLAB. The focused scenario follows a pushbutton LED circuit. It demonstrates the interaction flow—not a working AI model, live inventory, computer vision, or automatic circuit detection.

Evaluation method

We conducted a lightweight comparative formative evaluation. Two participants completed two Arduino wiring tasks: an LED-and-button circuit with AR guidance and an LED-and-potentiometer circuit with 2D instructions. We counterbalanced the order and recorded completion time, wiring errors, help requests, context-switching, ease of use, confidence, mental translation burden, and guidance clarity.

Key findings