well thats a little ghetto. Why not just swap out the SSD?
Anyway, it will not slow the computer down, but wobbling a hard disk around isn't great for it's lifespan. Alternatively, you could just get a massive SD Card if you have an SD card reader built in. Or you can get an external SSD, which won't suffer from being thrown around.