We develop a new scheme of proofs for spectral theory of the $N$-body Schr\"odinger operators, reproducing and extending a series of sharp results under minimum conditions. Our main results include Rellich's theorem, limiting absorption principle bounds, microlocal resolvent bounds, H\"older continuity of the resolvent and a microlocal Sommerfeld uniqueness result. We present a new proof of Rellich's theorem which is unified with exponential decay estimates studied previously only for $L^2$-eigenfunctions. Each pair-potential is a sum of a long-range term with first order derivatives, a short-range term without derivatives and a singular term of operator- or form-bounded type, and the setup includes hard-core interaction. Our proofs consist of a systematic use of commutators with `zeroth order' operators. In particular they do not rely on Mourre's differential inequality technique.