AI Agent Evaluator, Remote- Contract

Role Title: AI Agent Evaluator

Role Type: Contractor

Location: U.S based

• ChatGPT

• Muse

• AI agents

• Attention to Detail

• Following Instructions

• Screen Recording

About the role: AI agents are starting to handle real-life errands: paying bills, booking trips, doing the groceries. We want to know how well they actually do it, from the point of view of a real user. That’s you. You’ll test two AI agents side by side, using your own accounts on your phone, and tell us what worked, what didn’t, and how it felt.

• Time per task: ~1.5 hours per run (evidence and scoring included)

• Project kickoff: Monday, October 12

• Onboarding: Today (Fri), Saturday, or Sunday, so you’re ready to start Monday October 10th

• You’ll need: An active personal popular AI account (Free, Go, or Plus; not Pro), an active personal and a smartphone that can screen record

• Your tasks: Run everyday errands with two AI agents, approve key steps, screen record, and score the results

• Run real-life scenarios: bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms

• Approve or stop the agent at key steps

• Screen record every attempt

• Score each run on completion, quality, control issues, satisfaction, and time

• Share clear written feedback on the agent experience

• US-based, with a Bachelor’s degree

• Someone who already uses AI in daily life

• 2-3 years of experience, preferably in engineering, business management, or office/operations

• Detail-oriented, great at following instructions and documenting each step

• Clear written communication

• Generalists welcome, domain experts preferred

• No coding or AI background needed

• Get paid to use the AI tools you already use

• Work entirely from your phone, on your schedule

• Have a direct say in how next-generation AI agents get built and improved

Originally posted on Himalayas