What can I do about nuisance neighbors?

I live in a double and the neighbors upstairs are constantly fighting to the point where the cops are called. The woman has been "evicted" from the apartment (she isn't actually on the lease) by the police 3 times in the last 2 months, the locks have been changed several times, there is lots of yelling when she's here, stomping around, slamming doors and loud music. Within a couple weeks, the man up there will then drop the charges and have her move back in. The old tenant moved out because of this and we did not know until after we signed the lease. If anyone has any serious advice on what to do, I don't want the landlord to think I am a trouble maker if I call and complain, but we signed the lease before we knew any of this and there is no way I can pack up and move again at this point. I've already called him about the loud noise this week. The cops were here with her yesterday to pick up the last of her things, banging on the door, and then today she moved back in. Also, if it helps, I live in NY (not the city).