I obviously have less experience in English culture, but I do know for a fact that members of the 1882 movement/supporter group of Tottenham look down on even wearing the shirt. I assumed that was representative of all other clubs, although I know the gooners tend to wear the shirt more than Spurs fans.
I'll be honest - I've never even heard of the 1882 group. I'm pretty sure they're relatively rare in their dislike of shirt-wearing in footballing culture, though. At a guess, since 1882 was founded to improve the atmosphere, their no-kits choice was most likely made to try to distance themselves from the fans who they consider "not committed enough", just turning up and wearing a shirt and thinking they're doing it right - in the same way that attending a football game here with a camera round your neck immediately denotes you as a day-tripper and not a season ticket holder. But that's a guess.
At any rate, most fans here would consider the wearing of colours to be fairly important at least for aesthetic value, so that those watching on TV and in the away fans' section see you as a block of your team's colour.
However, if you guys want to follow a casual style then go for it. Part of the interest of this club, after all, is seeing it work out its own identity from bottom up. You have to do what you feel is right.