QA Engineer (AI products)

Ascendix Technologies is an AI and Real Estate Technology partner headquartered in Dallas, Texas, with offices in Poland, Portugal, and Ukraine. We specialize in AI solutions, Salesforce, and custom software development, helping businesses streamline their operations through innovative technology.

We're looking for an experienced QA Engineer to join our Product Department and take ownership of quality assurance for our Salesforce-related products, starting with the Ascendix AI Agent.

In this role, you'll work with a small, collaborative team to ensure the reliability of both traditional software functionality and AI-driven features. You'll combine manual testing, test automation, and LLM evaluation techniques to validate AI-generated responses, identify potential issues, and continuously improve product quality.

If you enjoy exploring AI technologies, thinking critically about complex scenarios, and taking initiative in shaping QA processes, we'd love to hear from you.

Requirements:

• 5+ years of experience in Software Quality Assurance;

• Hands-on experience testing AI-powered applications with non-deterministic outcomes, such as LLM-based products, AI agents, or chatbots;

• Practical experience with LLM evaluation techniques, including test datasets, scoring rubrics, LLM-as-judge approaches, and regression tracking;

• Experience in test automation and maintaining automated test suites;

• Solid API testing skills using Postman or similar tools;

• Strong analytical and critical thinking skills, with the ability to evaluate not only whether an AI-generated response is incorrect, but also why;

• Ownership mindset, proactive approach to problem-solving, and willingness to improve QA processes;

• Good communication and collaboration skills, with the ability to work effectively in a small, distributed team;

• Upper-Intermediate English (B2+) or higher, both written and spoken.

• Experience with Playwright (TypeScript or JavaScript);

• Salesforce testing experience, including Lightning, managed packages, and permission sets;

• Familiarity with LLM evaluation tools such as Promptfoo, DeepEval, Ragas, or Langfuse;

• Experience using AI-assisted development tools such as Claude Code, Cursor, or similar;

• Basic prompt engineering skills;

• Experience working with CRM systems or real estate data.

Responsibilities:

• Take ownership of product quality across our Salesforce-related solutions, initially focusing on the Ascendix AI Agent;

• Perform functional, regression, and exploratory testing of both deterministic and non-deterministic functionality;

• Evaluate AI-generated responses for accuracy, hallucinations, and correct execution of actions;

• Design, maintain, and improve LLM evaluation datasets, scoring criteria, and regression tests;

• Develop and maintain automated tests using Playwright and modern AI-assisted tools;

• Collaborate with developers and the Business Analyst in a spec-driven development process, reviewing specifications and defining clear acceptance criteria;

• Establish and continuously improve QA processes, including maintaining AI agent skills, rules, and workflow documentation in Markdown files;

• Investigate and reproduce issues reported by clients, providing clear and actionable bug reports;

• Proactively identify quality risks and suggest improvements to product reliability and testing practices.

Healthcare:

• 20 paid days-off;

• 5 paid sick days.

Career and Professional Growth:

• B2B and remote cooperation;

• Career development and professional growth;

• Technical performance reviews and 360 Evaluation;

• Collaboration with a personal Direct Manager, Technical Leads, and mentors.

Learning and Development:

• Opportunity for external event compensation;

• Internal knowledge-sharing opportunities.

Corporate Spirit:

• Online and offline events and team buildings;

• Friendly and supportive atmosphere;

• Open management.

Originally posted on Himalayas