Touch-based authentication is now a promising method for ensuring secure access on mobile devices with some researchers reporting very low EERs for authentication. However, these works have not shown experimentally the impact of contexts beyond phone orientation on touch-based authentication. In this work, we present experimental results on how touch-based authentication is impacted by users who are listening to music while swiping, users who are not listening to music while swiping, users who are swiping over images and users who are not swiping over images. We experiment with a data-set which we collected for this purpose from 34 subjects. Moreover, we provide design considerations towards a touch-based context-aware system and show how a module which considers the presence of music (which we found to have statistical significance) can be incorporated.