Most subjects are not available at all colleges. Therefore, find out what colleges offer your subject and pick one of these. This is the most important aspect of choosing a college.
Attend one of the open days. Even if you know you want to go to Oxford or Cambridge and you know what college you want to go to, there's a chance that at interviews you might get asked why you choose this college or university. Therefore, it's great to be able to say that you attended an open day and liked the college or university for this or that reason.
All the colleges are good and they all have particular good points. Some students will choose and interview at one college and be offered a place at another college. In most cases, students are happy to take what they are offered and enjoy their experience at whichever college they end up at.
If you are applying as a mature student think about whether you want a mature or graduate student college only (such as Linacre and Wolfson) or whether you would be happy having younger students hanging around
Some colleges have particular reputations which may attract you. For example, Wadham College at Oxford has a reputation for being very LGBT (Lesbian, Gay, Bisexual and Transgender) friendly. The students there run the very popular 'queer bop' every year. Exeter College, Oxford has a reputation for admitting Welsh students.
Pay attention to where the college is on the map. For example, if you want to row every day, you might want to be at a college close to the river. And if you don't like walking much, you might want to chose one close to your department.
That said, application numbers to particular colleges can be influenced by how close they are to the department. And this might impact your chances of getting a place. In theory, if you do well at an interview but that college has too many applicants, you can get sent for a second interview elsewhere. However, it is preferable to be interviewed at a less popular college to maximize your chances of getting in. If you can choose a college that would not necessarily be the obvious choice for your subject, you might benefit.
Think about whether you want a small or big college. The small colleges can feel socially very friendly to some, but a little claustrophobic to others. For example, everyone will know your gossip. Larger colleges feel less friendly, but less claustrophobic.
If money is tight, find out how much the accommodation and meals cost at different colleges. Some colleges make you pay for daily college meals and don't provide cooking facilities. Some colleges have more bursaries and grants (eg book grants) available than others.
Consider how important the academic credentials of the college are. If getting a first is critical to you, then choose one with a good score on the degree tables: the norrington table and the Tompkins Table
Are you a keen team sports player, but you're not good enough to make the university squad? If so, check out whether your college has a relevant team. For some sports, some of the small colleges either won't have a team, or the team might put up a rather poor show.