This study was undertaken to investigate the assessment of airline pilots’ performance when being considered for promotion to the position of airline captain. Within Australia, the assessment standards set for pilots are based heavily on the technical skills associated with flying an aircraft, and related knowledge. Although technical skills are important, there is, however, another set of pilot skills which are implicated in around 70% of airline accidents. This latter group of skills is termed nontechnical skills (NTS), and includes facets such as decision making, communication, and situational awareness. Although so critical to airline safety, these NTS are not explicitly identified in pilot performance assessments. This study comprises an investigation of the implicit and explicit criteria used by assessors, who are experienced airline captains, in making judgements about the performance of pilots who are applying for promotion to airline captain. The research included in-depth interviews with five experienced airline captains, each of whom possessed between 17 and 29 years’ experience in assessing pilots. These interviews and related analyses were conducted utilising a phenomenological methodology. Despite the overt emphasis on technical skills in current regulatory standards, the research demonstrated that an additional set of NTS (nontechnical skills) were being implicitly assessed. It was found that three main skills criteria were assessed as essential for being a captain: flying skills, situational awareness, and decision-making. There are of course other skills, such as aviation knowledge, management of crew, and communication amongst crew, which assist pilots in maintaining and improving their performance. This study concludes that the technical metrics currently used by the regulatory authority do not encompass all of the criteria actually used during assessment of airline pilots’ performance. The research has led to the development of a new model for assessing pilots’ performance (MAPP), which is based on current practice, and which makes explicit the NTS areas which are not currently identified in industry standards. Industry use of such a model will ensure that pilots are being actively assessed in the totality of skill areas that reflect current practice.