[IDEA2]: Data aware Data Management Systems – borrowing and building up on ideas implemented in the form of OLTP, OLAP and Big Data systems and a wide variety of other systems including distributed systems (e.g. git) – developing a specification for a uniform client interface to “DATA” irrespective of its size, nature and structure.
For example providing a uniform SQL interface to data irrespective of whether the data is 1 KB, 1 GB or 1 TB and whether it is structured, semi-structured or unstructured and whether it is string, number, xml or binary content.
Requirements
- The system should be aware of the data that it holds e.g. it should be able to adjust its physical schema and/or file system over a period of time as the size, nature and structure changes while ensuring that the client interface and the performance, scalability of the client remain unaffected.
- The system should be able to work in both auto-magic mode which would need minimum administration and also recommendation-engine mode which would generate and send out appropriate notifications at appropriate milestone events and allow the administrator or user to take action.
- The system should follow convention over configuration yet be fully configurable.
Design Notes
- Cloud would be one of the important strategies to address the problem.
- A set of advanced tools which can monitor and scale up/down the cloud infrastructure automatically or programmatically would be required.
- A software platform layer can be built above these two layers (cloud and automation tools)which can automatically instruct the tools to scale up/down the cloud infrastructure and then migrate/setup the physical schema and later choose the appropriate category of algorithms to work with the current form of physical schema.