Building a standard dataset for Arabie sentiment analysis: Identifying potential annotation pitfalls

Sentiment Analysis (SA) is one of the hottest research fields nowadays. It is concerned with identifying the sentiment conveyed in a piece of text. The current efforts in SA require the existence of standard datasets for training/testing purposes. Such datasets already exist for some languages such as English. Unfortunately, the same cannot be said about other languages such as Arabic. Currently existing Arabic SA datasets are restricted (in their domain, size, dialects covered, etc.) and/or have limited availability. Moreover, the annotation process did not receive the proper attention it deserves. Some of the existing datasets relied on the author's point of view for annotation, while others employed annotators, but did not take into account the personal variations between the annotators and how would that affect their agreement. This study presents our efforts to build a standard Arabic dataset with the above concerns in mind. The constructed dataset is intended for generic use as it contains reviews from different domains written in Modern Standard Arabic (MSA) as well as several dialects. As for the annotation process, it is given high attention by studying the inter-annotator agreements and investigating the potential factors affecting them.