Ditch the backpack and use a frame bag to carry the essentials. It gets the weight off your back and shoulders and helps lower your centre of gravity to make you more stable. There are off-the-shelf products out there, but the best option is to support local and get a custom one measured up for about the same price. It sets the baseline to add a seat bag and handlebar bag for the bigger trips.