People forget that censorship of blocks becomes easier and easier the harder it is for someone to run a full node / mine blocks. Giant blocks would take longer to download and as Gavin has explained to me over email when I was a noob on this issue, this would basically cut out any nodes that aren't in datacenters with great connections. I don't think we've found the answer to the blocksize problem yet, but making it unlimited is probably jumping the gun
True, but there is a big caveat in the treat of big blocks. Unlimited blocks will not lead to multi gigabyte blocks automatically. There needs to be demand for such transaction volume. It is very expensive to spam the network with an unlimited block size, if all a miner needs to do is include the transactions that pay enough, ignore the ones that do not, and thank you for the money.
The only risk I see, is that miners include transactions that they themselves create only to bloat the block size for others. But I think there are two sides to this story. Miners are not anonymous and this action is publicly visible. They have to be including transactions in a block that have never been seen before, or lots of zero-fee transactions and both are visible. The community could (and in my opinion should) publicly denounce such actions.
A large number of miners would have to collude for this to work. But they will not abuse that, because miners are servants to the network. Sure, they could collude to create big blocks, but they won't if all the community needs to do is demand that BitPay and exchanges etc ignore the blocks created by this bad actor and ask the other miners to create a competing chain.