The scaling of decoherence rates with qubit number N is studied for a simple model of a quantum computer in the situation where N is large. The two state qubits are localized around well-separated positions via trapping potentials and vibrational centre of mass motion of the qubits occurs. Coherent one and two qubit gating processes are controlled by external classical fields and facilitated by a cavity mode ancilla. Decoherence due to qubit coupling to a bath of spontaneous modes, cavity decay modes and to the vibrational modes is treated. A non-Markovian treatment of the short time behaviour of the fidelity is presented, and expressions for the characteristic decoherence time scales obtained for the case where the qubit/cavity mode ancilla is in a pure state and the baths are in thermal states. Specific results are given for the case where the cavity mode is in the vacuum state and gating processes are absent and the qubits are in (a) the Hadamard state (b) the GHZ state.