MYOB: Next-generation data management
Looking for a technology partner with a big data capability, MYOB enlisted the help of DiUS with prototyping and building a next-generation data processing platform.
Awarded Most Innovative Large Company in 2015 by BRW, MYOB exists to make business life easier for its 1.2 million customers in Australia and New Zealand. Recognising that no two businesses are alike, MYOB offers more than 50 products and services helping its customers be more productive by simplifying financial management.
Looking for a technology partner with a big data capability, MYOB enlisted the help of DiUS with prototyping and building a next-generation data processing platform. The platform needed to enable MYOB to bring together data from its vast array of applications and systems, and enable analytics and the development of value added services. The goal was to reduce reliance on a centralised reporting team and enable the whole business to process data as needed, whether it be for reporting, analytics, or reuse back into client-facing products.
DiUS is an Australian-based technology services company that prides itself on helping companies innovate and grow. Working with clients on a strategic level, helping to define and shape their product and technology strategy, DiUS specialises in delivering software with speed, agility and scale that extends the boundaries of its clients’ essential systems to transform the way they do business.
Using an agile and incremental approach, the team embarked on the design and development of a novel, flexible and serverless, self-service data lake and processing engine. The lake was based on AWS S3 and the engine on AWS Lambda. Apache Spark was also integrated for Map Reduce capabilities. AWS DynamoDB was the intermediate job lifecycle storage system.
The next piece of the puzzle was to build out a data mart on AWS Redshift to provide both analysts and product managers with the ability to use their preferred data analytics tools and third party services. Finally, an API was provided to open the data up to internal developers to enable them to upload data, build custom processing applications and enable two-way communication between the internal data platform and the external facing MYOB product platform.
Over the top of the end-to-end system was layered automation, monitoring and metrics to ensure that everything that is expected to arrive and be processed is of the correct size and format, and is producing the expected results. In addition, at the foundation of the system, is a data security and access control layer to ensure that only those with the appropriate credentials can access the various data sources.
The data platform has completely transformed the way data is managed and used within MYOB. No longer is there a centralised internal reporting team responsible for a single data warehouse and its processing. Instead, the data team now enable everyone else in the business to self service both the insertion, combining, and processing of data as needed.