Let the Silence Do the Heavy Lifting