Wearing a hat to the derby is tradition. It is a great occasion to look stunning! But keep in mind, you will be in the sun while at the paddock watching the horses being saddled, at the betting window, having fun in the infield and soaking up the traditions and history of horse racing i.e. all day long!
You will want to choose your derby hat before you pick your outfit. Getting a hat with a large brim will give you sun protection which will keep you looking young and forgo a nasty sunburn. Don't! wear uncomfortable high heeled shoes (because you will be walking from morning to night) or carry a heavy purse... and don't forget to enjoy a mint julep :)