Let's call this the newbie guide to starting a website. I've always hosted my own sites, so the reason I'm doing this research is because I'm considering a move to hosting off-site. There are probably hundreds of tutorials out there, so for those of you browsing, this is purely about hosting, not about actually making a website.
This post was also inspired by a question. First thing's first, you should decide on a hosting provider based on your needs. So this tutorial will be broken up based on the levels of need.
Minimal Need - $5/month
So the first level of need is minimal. This type of site is something your mom is using to present photos of her ebay sales. It's probably something Wordpress can handle. If you think it's going to need to handle traffic, many providers also allow for PHPBB on these. All of this is called Content Management System, and you can use any host that will allow you to FTP and upload documents for your website for this. Most of the time, you won't be making changes to the site itself in this case (though it is possible). This is designed around using web-based tools to make and edit your site, not changing the server side documents.
Moderate Need - $10+/month
The second level of need is for the hardcore developer who just wants to code. You need full access to an operating system that you can remote into and edit settings so that you can do everything from trying out different technologies to hosting video game servers. This is going to require a Virtual Private Server. This is the option I'm exploring at the moment and I'm not 100% sure what level of control I get over the machine. This server is, in reality, a Virtual Machine running on someone's massive server sharing hard drive space, bandwidth, RAM, and processing power with other similar Virtual Machines. In order to allow for this, they allocate portions of each of the above (HD space, RAM, etc) to each Virtual Machine so that you don't run into issues with someone trying to do something really weird. I found a $10 VPS host, but the pricing varies greatly depending on what Operating System you want on it, and I've heard upwards of $310 per month for decent servers. It clouds your data and forces your host to deal with backups and updates, but it also ties your server to a particular operating system and can get very expensive dependent upon your needs.
Serious Need - $??????
This is the level I'm currently at. I host my own server. It's doable, but the cost is based purely on what kind of resources you need. If you need a dedicated internet connection for your server, you're going to be paying a lot for bandwidth and the initial installation. If you're running a large number of RAID drives, then you're going to be paying a lot for electricity. If I want to host off-site so your roommate doesn't trip over a cord, then you have to rent space in a warehouse. The cost of this can vary by great margins. On top of that, you're completely responsible for your own backups, your own updates, your own maintenance, and of course, your own setup. No one is going to make IIS or Apache run magically for you. You have to really know your stuff. That said, it gives you extra flexibility. Upgrading your storage space isn't an extra $20 per month, but just the flat cost of a hard drive. If you want to switch over to hosting in OSX for a while, then it's within your ability. If you then decide that you'd prefer to try hosting on Ubuntu, it's up to you to do, and no one is going to stop you. The extra flexibility is quite nice to have. It's up to you if the cost, in both time and money, is worth it.