Agreed with @jeremy. Data Annotation is tricky, strongly domain and use case-dependent. It is one of the hardest problem with ML right now. I saw a lot of issues related to Mechanical Turk, and I would instead hire domain expert or at least semi-experts for annotation (semi-expert could be the person who is keen on learning domain characteristics and after some training will provide feasible annotations). The crucial issue is related to how do you design annotation process, what will be positive/negative examples, will it be repeatable for annotations and surely you must monitor inter-annotator agreement (inter-rater reliability) and correct annotation process if something goes wrong.
I would advise looking into book Natural Language Annotation for Machine Learning http://shop.oreilly.com/product/0636920020578.do