DOI: https://doi.org/10.15407/pp2022.01.067

Analytical store for streaming data with huge volume

V.O. Tiurin, А.Yu. Doroshenko, E.V. Savchuk

Abstract


A concept for organizing an analytical data warehouse has been developed, which includes a method of interaction between data producers and a repository, a method of data circuit control, a method of data streaming, a method of storing initial data, a method of data processing and a method of providing secure data access. Other concepts on the market are discussed, namely: SDLF as the leading standard recommended by AWS, IronSource DL using Upsolver, SimilarWeb DL using Upsolver. A comparative analysis was conducted (mostly with SDLF, as its implementation is open, and the implementation by private companies is hidden). The advantages of the proposed concept over the existing ones are examined in detail. Recommendations on how to integrate the concept with data schema control applications are given. A service for streaming data using Apache Beam in Java has been developed. A repository architecture for analytics was designed and developed. A data schema management model was developed as well as a data schema management model and a model for secure access to data. The research that has been conducted can be improved by the experience of implementing the concept in business, as well as by collecting and systematizing knowledge about other standards that will be created.

Problems in programming 2022; 1: 67-74


Keywords


serverless; analytical datastore; BigQuery; AWS; GCP; ETL; message; data streaming

Full Text:

PDF (Ukrainian)

References


Електронний ресурс [01.02.2022]: https:// aws.amazon.com/datawarehouse/

Електроний ресурс [01.02.2022]: https:// cloud.mts.ru/cloud-thinking/blog/datawarehouse/

Електроний ресурс [01.02.2022]: https://azure. microsoft.com/en-us/resources/customer-stories

Електроний ресурс [01.02.2022]: https:// builtin.com/data-science/company-data-lake- questions

Електроний ресурс [01.02.2022]: https:// databricks.com/discover/data-lakes/challenges

Електроний ресурс [01.02.2022]: https:// lingarogroup.com/blog/8-challenges-faced- by-ctos-when-starting-data-lake-projects/

Електроний ресурс [01.02.2022]: https:// www.qlik.com/blog/2020-the-year-cloud- data-warehouses-arrived

Електроний ресурс [01.02.2022]: https://www. castordoc.com/blog/cloud-data-warehousing- the-past-present-and-future

Електронийресурс[01.02.2022]:https://catalog. us-east-1.prod.workshops.aws/v2/workshops/ 501cb14c-91b3-455c-a2a9-d0a21ce68114/en-US

Електроний ресурс [01.02.2022]: https:// docs.aws.amazon.com/prescriptive-guidance/ latest/patterns/deploy-and-manage-a- serverless-data-lake-on-the-aws-cloud-by- using-infrastructure-as-code.html

Електроний ресурс [01.02.2022]: https:// www.upsolver.com/case-studies/ironsource- how-built-petabyte-scale-data-lake

Електроний ресурс [01.02.2022]: https://aws. amazon.com/blogs/big-data/how-similarweb- analyze-hundreds-of-terabytes-of-data-every- month-with-amazon-athena-and-upsolver/

Електроний ресурс [01.02.2022]: https:// docs.confluent.io/platform/current/schema- registry/index.html

Електорний ресурс [01.02.2022]: https:// habr.com/ru/company/alfastrah/blog/547092/

Електроний ресурс [01.02.2022]: https:// www.bigdataschool.ru/blog/kafka-big-data- schema-registry.html

Електроний ресурс [01.02.2022]: https:// cloud.google.com/pubsub/docs/schemas

Електроний ресурс [01.02.2022]: https:// cloud.google.com/dataflow

Електроний ресурс [01.02.2022]: https:// habr.com/ru/post/122479/

Електроний ресурс [01.02.2022]: https:// beam.apache.org/

Електроний ресурс [01.02.2022]: https://aws. amazon.com/ru/lambda/

Електроний ресурс [01.02.2022]: https:// airflow.apache.org/




DOI: https://doi.org/10.15407/pp2022.01.067

Refbacks

  • There are currently no refbacks.