Interactions between humans are complex, highly adaptive to the situation at hand, and rely to a large extent on non-verbal behavioural cues, most importantly visual and physical behaviour (aka body language). Despite significant advances in affective computing and social signal processing, current human-computer systems still fail short in leveraging the full potential of these cues. We envision human-computer systems that fully exploit the large information content available in non-verbal human behaviour and thereby offer human-like perceptual and interactive capabilities. Realising this vision is challenging and requires scientific advances along three dimensions: human behaviour sensing, behaviour analysis and modelling, as well as application development. With a focus on human visual behaviour, in my talk I will provide an overview of previous and ongoing work towards our vision along these three dimensions.