During the last decade, hateful and sexist content towards women is being increasingly spread on social networks. The exposure to sexist speech has serious consequences to women’s life and limits their freedom of speech. Previous studies have focused on identifying hatred or violence towards women. However, sexism is expressed in very different forms: it includes subtle stereotypes and attitudes that, although frequently unnoticed, are extremely harmful for both women and society. In this work, we propose a new task that aims to understand and analyze how sexism, from explicit hate or violence to subtle expressions, is expressed in online conversations. To this end, we have developed and released the first dataset of sexist expressions and attitudes in Twitter in Spanish (MeTwo) and investigate the feasibility of using machine learning techniques (both traditional and novel deep learning models) for automatically detecting different types of sexist behaviours. Our results show that sexism is frequently found in many forms in social networks, that it includes a wide range of behaviours, and that it is possible to detect them using deep learning approaches. We discuss the performance of automatic classification methods to deal with different types of sexism and the generalizability of our task to other subdomains, such as misogyny.