See Ignore everything except /X pattern still scans everything
I don’t think there is a way around it.