Right after the mirador, you leave the highway and go through on a village street. There is also a grocery store in town center.
There are two sets of arrows when leaving town, and a mojon which keeps you on the camino towards Villavieja. Do not follow the arrows to Las Barrancas.
The Calle Chaos takes you out of town, and it turns to a dirt path.
A couple of kms out of town, after walking through ag fields and orchards, a mojón takes you to the left, across the national highway and an ascent begins. There is a sign for the Villavieja Albergue.
You can, however, stay on the road, bypass the castle, and go directly on the road to Borrenes. There is minimal elevation gain, but walking on the shoulder of a national highway is generally a bad idea. If you take the road, you will have another option later to take a road ascent to the castle, but the Camino route through Villavieja is far preferable.
If you take the climb, you will have great views of the castle and arrive in Villavieja.