Caleb, there are many commercial products to do this type of stuff. Mercury has a pretty full suite. There are also plenty of free ware ones that are more difficult to work with. One decent one is the Java Grinder.
I've done this many times in my career, starting with building my own tools at Lycos and Amazon and running a performance team at Zillow and here at XBRLCloud.
BTW, the Zillow launch crash I had stress tested our application to 3 X projected traffic by marketing
and we got 5 X. It took 45 minutes for us to find and fix the bottleneck which was in our network.
You can get a feel by just hitting URLs in simple cases and that test is pretty quick to build. But a real test you have to have some idea what your users are going to do and to build little user scripts that mimic real use and run them as N parallel users.
You can also avoid a lot of this by simply buying hardware in the cloud. Hand test a bit and architect to scale and throw hardware at it at early stages to save money, when and if you start getting traffic.
However, in the long run an automated performance test with each build is essential. One program bug can take you down totally and keeping your CPU rental costs down can save a lot of money.