Automation QA (AI Testing)

Key Responsibilities for Automation QA (AI Testing Suite):

• Framework Development:
• Design and implement a scalable automation testing framework for AI applications using .NET technologies.
• Build reusable libraries and components that enhance the efficiency of the testing process.
• Test Script Development:
• Write, maintain, and optimize automated test scripts for functional, regression, and integration tests using frameworks like Selenium, SpecFlow, or NUnit.
• Ensure test scripts are designed to handle the unique requirements of AI applications, including validating model outputs and performance.
• AI Model Validation:
• Collaborate with data scientists and engineers to understand the architecture and outputs of AI models.
• Develop specific test cases to evaluate the accuracy, performance, and reliability of AI models, including edge case scenarios.
• Data Management:
• Create and manage diverse test datasets that accurately reflect real-world scenarios, ensuring comprehensive coverage for testing AI models.
• Implement data validation processes to ensure input data integrity and correctness.
• Integration into CI/CD:
• Integrate automated testing processes into CI/CD pipelines using tools like Azure DevOps or Jenkins to facilitate continuous integration and delivery.
• Collaborate with DevOps teams to ensure smooth deployment and execution of automated tests.
• Performance Testing:
• Conduct performance and load testing of AI applications using tools like JMeter or LoadRunner to assess scalability and responsiveness under various conditions.
• Analyze performance metrics and work with development teams to identify and resolve performance bottlenecks.
• Collaboration and Agile Practices:
• Work closely with cross-functional teams, including developers, product managers, and data scientists, to gather requirements and clarify testing objectives.
• Participate in agile ceremonies, contributing to sprint planning, daily stand-ups, and retrospectives.
• Documentation and Reporting:
• Maintain clear and comprehensive documentation of the automation framework, test cases, and testing outcomes.
• Utilize bug tracking tools like Azure DevOps, JIRA to report, track, and prioritize defects, ensuring clear communication with development teams.
• Continuous Improvement:
• Stay updated with the latest trends in automation and AI testing, evaluating new tools and techniques for potential integration into the testing process.
• Proactively suggest improvements to existing testing methodologies and frameworks to enhance efficiency and effectiveness.

Expected Technology Stack:

• AI/ML Tools:
• Familiarity with frameworks like TensorFlow or PyTorch (to understand AI model validation)
• Tools for model evaluation (e.g., Scikit-learn)
• End-to-end development cycle of LLM-based AI applications – (e.g. Promptflow)
• Understanding on Prompt Evaluation Metrics.
• Programming Languages (preferred):
• C# (primary language for .NET development)
• Python (for AI-related testing and data manipulation)
• JAVA (at least beginner level understanding)
• Protractor (Good to Have)
• Automation Frameworks:
• Selenium (for web application testing)
• SpecFlow (for behavior-driven development)
• NUnit (for unit testing)
• Continuous Integration/Continuous Deployment (CI/CD):
• Azure DevOps
• Jenkins
• GitHub Actions
• Performance Testing Tools:
• Apache JMeter
• LoadRunner
• Bug Tracking and Documentation:
• Azure DevOps / JIRA (for documentation, defect logging and tracking)
• Version Control: Git (for source control)