The notion of Ability-Based Design, put forth by Wobbrock et al. [80, 82] as a solution to the challenge of creating accessible technology, has been discussed in human-computer interaction research now for over a decade. However, despite being cited as influential on various projects, the concept still lacks a general characterization of how to implement its required focus on abilities. In particular, it lacks a formulation of how to perceive and model users within an articulated design process. To address this shortcoming, we rely on conceptual user modeling to examine Ability-Based Design and propose a characterization of it that is not dependent upon a specific project or research effort, but that enables the ability-based design of new technologies in a systematic manner. Our findings show that Ability-Based Design’s focus on abilities requires important changes in typical user modeling approaches that cannot be met with current techniques. Based on the challenges identified through our analysis, we propose a first modification not only of current user modeling, but of current requirements analysis approaches to address abilities and their intertwined dependencies with tasks and contexts as core elements of conceptual models in Ability-Based Design. We thereby demonstrate not only the complexity of modeling users’ abilities, but also draw out promising ideas and perspectives for future research, emphasizing the need for future evaluative work on our approach.