We are in a time of increasing costs and shortages of skilled labor. The construction industry isn't the most "mature" in terms of supply chain. Savings are hard to come by but they "shouldn't" be. Costs vary WILDLY to get a building out of the ground - you never know what you'll hit. Costs today will be more tomorrow. Weather will cause delays. Overbooked crews will cause delays. Nothing ever gets cheaper as you get into it.
I felt like we had firm quotes for a LOT of our build. (pretty much the entire envelope, roof, windows and about $100K of retaining walls and with the exception of $20K more on the roof, we kept those costs) and had purchased almost all our appliances and fixtures.) Concrete flat work, drainage, excavation, septic, footings, all more than expected.
Oh and it's asking A LOT to get windows from a local maker. I can't imagine someone making my windows locally - triple pane PassivHaus certified aluminum clad Euro windows. Between the custom aluminum extrusions, the thermal isolation, the custom extruded seals, and the cladding, that would be something to pull off is a small custom builder.
Many have tried to do what you are doing. You see what looks to be an industry ripe for streamlining and cost reduction and you are just the person to do it. (we were those people) And many have failed. And if you are doing something "out of the ordinary" - you also limit your subcontractor pool.
Let us know when you get a price from your builder. Depending on if you are getting a construction loan or not - you may need to have a contract that has pricing that is not to exceed - as cost plus isn't favored by banks.
Best of luck to you.
Q