Automated development of a parallel distributed system for streaming data processing

D.S. Titov, A.Yu. Doroshenko, O.A. Yatsenko

Abstract


An automated development of a parallel distributed dynamically scalable fault-tolerant system for processing large amount of streaming data is performed. The system is based on the framework for distributed computing Hazelcast and the usage of the toolkit for generation of programs from high-level specifications of algorithms. The inspection and study of this system is performed on an example of data processing in
Twitter social network in which sentiment analysis functionality is implemented. The mechanism of the deployment of the created system on a cloud platform is examined.

Problems in programming 2016; 2-3: 96-104


Keywords


network; analysis; fault-tolerance; scaling; cluster; stream; cloud, node; program generation

References


Brandwatch. (2016) Social Listening in Practice. Market Research. [Online] Available from: https://www.brandwatch.com/guide-market-research. [Accessed: 25th January 2016]

Brandwatch. (2016) Social Listening in Practice. Social customer service. [Online] Available from: https://www.brandwatch.com/customer-service-guide. [Accessed: 25th January 2016]

Titov, D.S. & Doroshenko A.Yu. (2015) Social networks monitoring in real-time systems. In Proc. of 5th Int. Conf. "Scientific discussion: theory, practice, innovation". Kyiv, Ukraine, 27-28 March 2015. Kyiv: IOMP. p. 93-96 (in Ukrainian)

Intel. (2016) Intel Core i7-3770k Processor. [Online] Available from: http://ark.intel.com/products/65523. [Accessed: 25th January 2016]

Mamudi S. (2016) Charlie Munger: HFT is Legalized Front-Running. [Online] Available from: http://blogs.barrons.com/stockstowatchtoday/ 2013/05/03/charlie-munger-hft-is-legalized-front-running. [Accessed: 25th January 2016]

Titov, D.S. & Doroshenko A.Yu. (2015) The parallel distributed system for social media data streams analysis. Problems in programming. (4). p. 31-39. (in Ukrainian)

Hazelcast. (2016) Hazelcast. [Online] Available from: https://hazelcast.org. [Accessed: 25th January 2016]

Sentiment analysis (2016) Sentiment analysis. [Online] Available from: https://en.wikipedia.org/wiki/Sentiment_analysis. [Accessed: 25th January 2016]

Andon, P.I. et al. (2007) Algebra-algorithmic models and methods of parallel programming. Kiev: Akademperiodika. (in Russian)

Iovchev, V.O. & Mokhnitsa, O.S. (2010) Algebra-algorithmic tools on Web 2.0 platform. Problems in programming. (2-3). p. 547-555. (in Russian)

Doroshenko, A.Yu., Beketov, O.G., Ivaniv R.B., Iovchev, V.O., Myronenko, I.O. & Yatsenko, O.A. (2015) Automated generation of parallel programs for graphics processing units based on algorithm schemes. Problems in programming. (1). p. 19-28. (in Ukrainian)

Oracle. (2016) Java documentation. The Java Tutorials. Lesson: Annotations. [Online] Available from: https://docs.oracle.com/ javase/tutorial/java/annotations. [Accessed: 25th January 2016].

Twitter. (2016) Public streams. [Online] Available from: https://dev.twitter.com/streaming/public. [Accessed: 25th January 2016]

Spring. (2016) Spring Social. [Online] Available from: http://projects.spring.io/spring-social. [Accessed: 25th January 2016]

The Stanford NLP (Natural Language Processing) Group. (2016) Stanford CoreNLP – a suite of core NLP tools. [Online] Available from: http://nlp.stanford.edu/software/corenlp.shtml. [Accessed: 25th January 2016]

Amazon. (2016) Amazon EC2. [Online] Available from: https://aws.amazon.com/ec2. [Accessed: 25th January 2016]

Amazon. (2016) Amazon EC2 Instances. [Online] Available from: http://aws.amazon.com/ec2/instance-types. [Accessed: 25th January 2016]




DOI: https://doi.org/10.15407/pp2016.02-03.096

Refbacks

  • There are currently no refbacks.