You can walk the whole town in twenty minutes, so the real choice is whether to pay to sleep inside the river bend or pay less and climb a hill twice a day.
Vnitrni Mesto - inside the bend
The old town around namesti Svornosti. Cars mostly cannot get in, so expect to drag a bag over cobbles, and expect coach groups outside the window from about ten to four. It is also the only base from which the town after dark actually works.
Latran - the castle side
The strip under the castle on the far bank, two minutes from the main square and closer to the castle gate, which matters when the tower opens at nine. Usually a little cheaper than the square itself.
Plesivec - up the hill
The residential slope south of the old town, ten to fifteen minutes' walk in and noticeably cheaper. Worth it if you have a car, because you can park at the door instead of in the paid lots on the edge of the centre.
Check the season before you book
The castle gardens open only from 1 May to 31 October, the guided interior routes finish at the end of October, and the castle museum and tower carry on to 20 December before the whole complex shuts. Winter here is genuinely quiet, which is either the point or the problem.