Mobility and Social are at the moment driving the way a user should look the world at. Consumers and Enterprise worlds are at a cross section where they need to change and drive the technological sea which is facing a new set of development and deployment challenges. In this blog post part I would cover the Data Engineering challenges being faced by Mobile and Social infrastructures and how a Big Data technology like MongoDB can help (or helps, I would dare to say) and cater to the challenges. My sub-sequent posts will talk more about Enterprise Mobility and MongoDB as a key player in that.
Mobile, Social Data Engineering challenges:
Mobility world is trying to address a spectrum of use cases in Consumer and Enterprise verticals. Social Engines are revolutionizing the way a consumer will discover and interact with information. There are certain generic challenges which I see when we build Data infrastructure for any of these platforms.
Diversified Data on devices: For any mobile/social platform the amount of data, type of data (structured, semi-structured, unstructured, and transactional) would be one of the biggest challenges. Data is gathered in and/or from variety of devices like tablets, smartphones, readers, smart meters etc. Data storage for all of them cannot be application specific or use case specific. We need a generic Data Store which suits all of them.
Multiple Schemas: The type of data on the Mobile and social platforms various different schemas utilizing many data structures which may include log data, scrap data, user-generated data, images etc. Any relational data model will suffer encompassing all of them together.
Faster Roll outs: Consumer engines require fairly rapid development and deployment cycles. If they follow RAD (Rapid Application Development) methodology they need to have fairly dynamic data technology and infrastructure which could allow them to quickly evolve their data patterns with a fairly smooth development and deployment transitions. Traditional Data engineering technologies do lack in this front.
Wider user base: Globally distributed user base would ask for low-latency and high availability of the systems which they will be using. This means platforms like these will need to support a user base of any scale with reliability and cost-effectively.
Processing and Analytics: Any complex analysis or machine learning algorithms are usually time consuming if we use traditional data modeling mechanisms which will have compulsion to write internal stored procedures. Social platforms will need patterns for aggregation, simplifications, recommendations, analysis, etc. which is quite time consuming if we run the jobs on the relational data model architecture. Managing these in a relational data infrastructure is fairly complex, time consuming and very expensive when the data needs to evolve rapidly.
MongoDB — a solution:
Addressing the given challenges need a fairly reliable, stable, scalable and cost-effective Data Engineering model which in turn would help any Data Architect team to evolve the data as and when
needed. Traditionally we have been taught that define your schema first but the platforms and challenges which we are working towards do require us to break these barriers and come out with solutions which can evolve rapidly with seamless transitions. MongoDB has helped us with some of the key attributes or points which we have observed over the period of time.
Schema less (or Dynamic Schematron): MongoDB is purely based on Document Oriented data model which can accommodate our diversified data and meta data requirements. Its Schema less schematron is pretty well aligned with any modern programming language which eventually helps us in adopting to Agile development methodology. Document oriented data base means you don’t need to look for traditional rows and columns. Its BSON (i.e JSON) oriented document data storage methodology and javascript based query access methods allow us to rapidly grow the data model within the same sprints when we are changing our user interface aspects of any Mobile or Social engines.
Horizontal Scalability: Auto Sharding helps the Data Architecture team to scale horizontally on any of the commodity hardware, cloud or hosted infrastructure fairly quickly with increasing user base for better throughput. Usually scale up is considered as very complex and expensive solution whereas scale out here is pretty well aligned with the type of need the systems will have.
Analytics: Map-Reduce and Aggregation frameworks from MongoDb can empower real-time data analytics on Mobile and social platforms from within database which helps reducing and simplifying the analytics deployment and fairly increase the user experience.
Geo Spatial: MongoDB does have geo-spatial indexing capabilities which in turn would help social and mobile platforms to support location based information storage, retrieval and analysis natively. Overall it improves the user experience with improved productive data.
Enterprise Grade: MongoDB can provide Enterprise class operational simplicity, reliability with state of the art automatic fail over and replica systems and Map-Reduce frameworks.
If we look at the bigger picture MongoDB support an organization to improve Faster Time to Market, operational excellence and better user experience. In my next set of posts I would discuss more detailed relevance on Enterprise Mobility and role of MongoDB as a key data engineering player.
