If you are going in to the office after hours or on the weekend, you should dress in casual business attire such as khakis and a button down or polo shirt. Clothes should be clean and pressed just as they would be during the workday.
Do not wear the clothes you lounge around the house in such as old jeans, tennis shoes, and a sweatshirt. You may feel comfortable in them, but you never know who you might see at the office or who might see you walking in.
A professional image should always be maintained in order to positively represent yourself and the company you work for.