My rule is that I bring work in-house when the thing being outsourced is something my clients think they're buying from me. Everything else I'm happy to remediate or switch.
I've run Green Planet Cleaning Services in the Bay Area for 16 years. The clearest version of this for us was subcontracting. Early on, when demand spiked beyond what my own crews could cover, the industry-standard answer was to hand overflow jobs to independent contractors. It's how a lot of cleaning companies scale, and on paper it's elegant — you only pay for the work you get.
What actually happened was that the quality of a job stopped being something I controlled. A client doesn't experience "our subcontractor had a bad week." They experience Green Planet did a bad job. And the failure showed up on the client's side before it ever showed up on mine, which is the worst possible sequence — by the time I knew, someone was already deciding whether to keep us.
That was the trigger: the vendor's miss was invisible to me and visible to my customer. Once that pattern repeated, remediation was pointless, because the problem wasn't effort, it was that I had no ability to see quality before the client did. So we went the other direction and built a W-2 employee model. My people are hired, trained, paid, and accountable to me. It costs more per hour than contracting, without question. It also means the product my clients buy is the product I actually make.
The general rule I'd offer: outsourcing is fine where a miss is recoverable and invisible to the end customer, and dangerous where a miss becomes your reputation. Switch vendors for the first kind. In-source the second kind, and accept the margin hit as the price of controlling your own brand.
The signal I watch for is who tells me about the problem. If I'm consistently learning about failures from my clients rather than from my vendor, the relationship can't be fixed by a scorecard, because the visibility gap is the problem. That's my line, and it's saved me from long, polite improvement plans that were never going to work.