Versioning
Version 1.2
Published effective 13 December 2024
Last reviewed 13 December 2024
Classification Public
Approved by Head of Kineo Courses
Introduction
Kineo offers a range of business support services that become a part of the operations of our clients. We recognise that in providing these services, we are committed to delivering a continuity of service that allows our clients to maintain the continuity of their operations.
Document purpose
The purpose of this document is to explain the:
- Measures we take to mitigate risks to our business
- Technical measures we take to protect our data, and
- Technical and administrative measures we take to recover from a disaster
Intended audience
This document is intended for the following groups with the purpose of:
- Kineo staff seeking guidance on business continuity
- Clients and prospective clients evaluating Kineo as a service provider, and
- Clients raising a specific issue in relation to Kineo’s business continuity.
Definitions
Hosted platform: The primary services offered by Kineo are:
- Hosted platform which consists of the following core applications:
- Kineo Courses delivered through Learnforce Portal
- Kineo Courses delivered through Learnforce Cloud
- Developer solutions including integration
- Generic courseware
These services are collectively referred to as the hosted platform.
Policy statement
Corporate management has approved the following policy statement:
- The company shall develop a comprehensive IT disaster recovery plan.
- A formal risk assessment shall be undertaken to determine the requirements for the disaster recovery plan.
- The disaster recovery plan should cover all essential and critical infrastructure elements, systems and networks, in accordance with key business activities.
- The disaster recovery plan should be periodically tested to ensure that it can be implemented in emergency situations and that the management and staff understand how it is to be executed.
- All staff must be made aware of the disaster recovery plan and their own respective roles.
- The disaster recovery plan is to be kept up to date to take into account changing circumstances.
Objectives
The principal objective of the disaster recovery program is to develop, test and document a well-structured and easily understood plan which will help the company recover as quickly and effectively as possible from an unforeseen disaster or emergency which interrupts information systems and business operations. Additional objectives include the following:
- The need to ensure that all employees fully understand their duties in implementing such a plan.
- The need to ensure that operational policies are adhered to within all planned activities.
- The need to ensure that proposed contingency arrangements are cost-effective.
- The need to consider implications on other company sites.
- Disaster recovery capabilities as applicable to key customers, vendors and others.
The Hosting Infrastructure
Kineo has partnered with Amazon Web Services (AWS) to host and deploy its application. Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS cloud.
Amazon EC2 enables Kineo to launch as many or as few virtual servers as required, configure security and networking, manage storage, and scale up or down to handle changes in forecast traffic.
Importantly, Amazon meets and exceeds a security and compliance requirements ensuring Kineo maintains data protection for its clients. You can read about Amazon's compliance from this link Amazon’s compliance.
Regions and availability zones
Amazon EC2 is hosted in multiple locations world-wide, known as regions. Kineo has deployed its application in three regions across Australia and Europe. All data collected and processed from the Kineo application is retained within either the Australian or European regions.
Each Amazon EC2 region is completely isolated from the other Amazon EC2 regions, and within each Amazon region there are multiple, isolated locations known as Availability Zones. Amazon EC2 provides Kineo with the ability to place resources, such as instances (servers), and data in multiple locations within that region ensuring the highest level of fail over and redundancy in the situation of a disaster.
The Availability Zones in each region are connected through low-latency links. Kineo utilises both the separation of Regions and Availability Zones within AWS to maintain data protection, and implement both disaster recovery and backup.
EC2 instances
Amazon enables Kineo to deploy multiple instances (servers) to meet its business requirements and platform needs. The flexibility of Amazon EC2 enables Kineo to scale both horizontally and vertically to meet its platform needs, and implement solutions to support performance, disaster recovery and backup requirements.
Horizontal scaling
Horizontal scaling is the process of adding more nodes to a system, such as adding a new server to a distributed software application. Kineo uses horizontal scaling to extend its platform by adding additional instances across both in the Australian and Europe regions to meet its business requirements, but important to meet usage demands.
Vertical scaling
Vertical scaling means to add more resources to a single node within a system, such as increasing the CPU or memory requirements for a single server. Vertical scaling is used by Kineo to increase performance to align with forecast user traffic, and manage performance of its application when under load.
Vertical scaling is the process of increasing the instance type (hardware performance) for each instance on Amazon. Kineo uses a variety of instance types depending on the purpose and requirements for the application or software that is run on the instance.
The Platform
The platform consists of the following virtualised servers:
- Production (Primary)
- Beta
- Production (Failover)
- Backup
- Flat file integration, and
- Monitoring
Production (Primary)
This is the cluster of primary production servers which host the application and database. All user traffic is directed to the platform hosted on these servers.
The primary production cluster is divided into four tiers
Web application firewall (WAF)
The WAF helps secure the application by blocking common web application threats.
Content delivery
Platform uses Amazon's S3 and CloudFront services to deliver static content such as video, audio and images to users.
Front end web servers
Front end web servers deliver the Learnforce application to the user.
Database
Platform uses Amazon's Aurora managed database platform to host the database, which provides scalability, redundancy and to-the-minute backups. All database data is encrypted at rest.
Beta
This instance is accessible to Kineo clients to conduct functional and integration development and testing, which is separate from their production (live) environment. The database on this instance is synchronised with the Production (Primary) database on the 28th of each month.
Production (Failover)
The Production (Failover) is a physically and logically separate instance of the Production (Primary) servers, updated in real-time using disk-level replication, including the application and all data collected in the database. When a disaster occurs in the Availability Zone hosting the Production (Primary) servers, end-user traffic is automatically directed to the Production (Failover) instances in the second Availability Zone. This provides redundancy in the case of a disaster within one of the availability zones within an Amazon region.
Backup
This instance is responsible for coordinating and storing all backups for the application and database. The Backup instance is hosted in a separate Availability Zone to both the Primary and Failover servers to ensure that the backups of customer data are logically and physically separate from production data and can be accessed in the case of a disaster in another Availability Zone.
Flat File Integration
This instance is used for uploading files for the flat file Integration offered by Kineo.
Monitoring
This instance is used by Kineo to host our internal monitoring and logging systems. Metrics logged include:
- Application access logs
- Application uptime, and
- Platform performance and utilisation statistics.
Application
This section details information about the following:
- Password security
- Encryption, and
- Data storage.
Password Security
All passwords stored by the application are salted and hashed using the BCrypt algorithm.
Minimum password complexity rules for the application are:
- 6 characters in length.
- Must contain 1 lowercase and 1 uppercase character and 1 number.
These rules are configurable and can be increased in complexity on request.
Encryption
The application uses HTTPS (SSL) to encrypt all data between the application and the user’s browser.
Data storage
All data collected by the application is stored within each geographic region.
- For the Australian Platform, this is AWS Sydney (ap-southeast-2).
- For the Europe Platform, this is AWS Ireland and Frankfurt (eu-west-1 and eu-central-1).
The three storage options provided by Amazon that are used by Kineo to store and collect data are:
- Amazon EBS
- Amazon S3
- Amazon Aurora
All databases and database backups are encrypted at rest with backups retained for seven years.
Australian Platform
The Australian platform services all clients and users located within Australasia, particularly in Australia and New Zealand. The Australian platform is hosted in the AWS Sydney region and uses all the three availability zones in this region. The AWS Sydney region in Amazon hosts the primary platform including primary backups. Redundant backups are also securely stored in Kineo’s head office in Adelaide, Australia.
Instances
The Australian platform consists of the following instances in the AWS Sydney region, across three availability zones:
- Sydney: Availability zone A
- Sydney: Availability zone B
- Sydney: Availability zone C
Sydney: Availability zone A
Availability Zone A hosts the following instances:
- Production (Primary)
- Beta
- Website
- Flat File Integration
- Monitoring
Sydney: Availability zone B
Availability Zone B hosts the following instances:
- Production (Failover)
Sydney: Availability zone C
Availability Zone C hosts the following instances:
- Backup
Backup
Backup is an important aspect of the Kineo hosted platform, and Kineo has implemented a multi-facet backup solution. Backups are implemented both on the Amazon platform, and in a secure location at Kineo head office in the case of the disaster within the entire Sydney region.
For the Australian platform, Kineo has implemented incremental backups of the following:
- The application and its database
- Course content
- User uploaded files and assets
- Kineo corporate website
- All site logs
- Uploaded SCORM courses,
- Infrastructure and instance configuration
Backup instance
The power of Amazon enables Kineo to implement backups using horizontal scaling. A dedicated Amazon instance has been setup to manage incremental backups. The backup frequency is daily (24 hours), and all backups are stored within Amazon AWS. Backup integrity is verified daily.
Disaster recovery
In the case of a disaster, Kineo has implemented a multi staged disaster recovery process
Stage 1
Using file system level replication, the Production (Failover) instances are a real-time replication of Production (Primary) servers, including the application and all data collected in the database.
When a disaster occurs in the Availability Zone hosting the Production (Primary) servers, end-user traffic is automatically directed to the Production (Failover) instances in a separate Availability Zone. This provides a real time fail over solution in the case of a disaster within one of the availability zones within Amazon.
- The expected recovery time objective (RTO) is less than 1 hour.
- The expected recovery point objective (RPO) is less than 1 hour.
Stage 2
In the unlikely case, that a disaster has impacted, both Availability Zone A and B in the Sydney region, Kineo will use backups to restore to a non-impacted Availability Zone within AWS.
- The expected recovery time objective (RTO) is 48 hours.
- The expected recovery point objective (RPO) is up to 24 hours.
Disaster recovery process
In the event of a significant server failure, Kineo will take the following steps to address the issue:
- conduct an immediate assessment of the problem
- identify an action plan and develop initial estimate for the rectification
- send communication and an estimated time of rectification to client contacts
- live server status page
- Implement the action plan. Depending on the circumstance this can include:
- setup of an alternate server
- installation of the application
- transfer of files from backup
- system testing – random account and operational procedures
- re-delegation of domains
When there is a change to the circumstances, Kineo will provide an update to the client contacts detailing the change.
Where domain re-delegation is required, there may be a 12 to 24-hour delay between restoration of system functionality and the system becoming available via the internet, due to the delay in DNS propagation.
Disaster recovery testing and validation
The following activities are performed in order to test the disaster recovery process and validate the integrity of backups:
- Daily - Backups are restored to a simulated environment to ensure integrity of data.
- Monthly - Backups are restored to the Beta environment to ensure availability of all data from backup and for verification.
- Quarterly - Stage 1 disaster recovery is tested. The production failover capability is tested by running the application on the Production (Failover) servers during a maintenance period.
- Yearly - Stage 2 disaster recovery is tested. A new simulated Production instance of the application is built from backup in order to test the processes and procedures required to rebuild the application and infrastructure in case of disaster.
Europe Platform
The Europe platform services all clients and users located within Europe, particularly in United Kingdom. The Europe platform is hosted in two regions across Europe, AWS Ireland and AWS Frankfurt. The Ireland region in Amazon hosts the primary production platform across both Availability Zones A and B, and the Frankfurt region hosts disaster recovery and backups within the Availability Zone A.
Instances
The Europe platform consists of the following instances across two regions:
- Ireland: Availability Zone A
- Ireland: Availability Zone B
- Frankfurt: Availability Zone A
Ireland: Availability Zone A
Availability Zone A hosts the following instances:
- Production (Primary)
- Beta
- Flat File Integration
Ireland: Availability Zone B
Availability Zone B hosts the following instances:
- Production (Failover)
Frankfurt: Availability Zone A
Availability Zone A hosts the following instances:
- Backup
- Disaster Recovery
Backup
Backup is an important aspect of the Kineo Europe hosted platform, and Kineo has implemented a multi-facet backup solution. Backups are implemented solely on the Amazon platform, in both the Ireland and Frankfurt regions
From the Europe platform, Kineo has implemented incremental backups of the following:
- The application and its database
- Course content
- User uploaded files and assets
- All site logs
- Uploaded SCORM courses, and
- Infrastructure instance configuration
Backup instance
The power of Amazon enables Kineo to implement backups using the horizontal scaling. A dedicated Amazon instance has been setup to manage incremental backups in the Frankfurt region. The backup frequency is daily (24 hours), and all backups are stored on Amazon S3 storage. Backup integrity is verified daily.
Disaster recovery
In the unlikely case of a disaster Kineo has implemented a multi staged disaster recovery process.
Stage 1
Using file system level replication, the Production (Failover) instances are a real-time replication of Production (Primary) servers, including the application and all data collected in the database.
When a disaster occurs in the Availability Zone hosting the Production (Primary) servers, end-user traffic is automatically directed to the Production (Failover) instances in a separate Availability Zone.This provides a real time fail over solution in the case of a disaster within one of the availability zones within Amazon.
- The expected recovery time objective (RTO) is less than 1 hour.
- The expected recovery point objective (RPO) is less than 1 hour.
Stage 2
In the unlikely case, that a disaster has impacted the entire Ireland Region the disaster recovery instance in Frankfurt will be used. Kineo will implement the disaster recovery plan to divert user traffic from Ireland to Frankfurt.
- The expected recovery time objective (RTO) is 48 hours.
- The expected recovery point objective (RPO) is up to 24 hours.
Disaster recovery process
In the event of a significant server failure, Kineo will take the following steps to address the issue:
- conduct an immediate assessment of the problem
- identify an action plan and develop initial estimate for the rectification
- send communication and an estimated time of rectification to the client contacts.
- update the live server status page
- Implement the action plan. Depending on the circumstance this could include:
- setup of an alternate server
- installation of the application
- transfer of files from backup
- system testing – random account and operational procedures
- re-delegate domains
When there is a change to the circumstances, Kineo will provide communication update to the client contacts detailing the change.
Where domain re-delegation is required, there may be a 12 to 24-hour delay between restoration of system functionality and the system becoming available via the Internet, due to the delay in DNS.
Disaster recovery testing and validation
The following activities are performed in order to test the disaster recovery process and validate the integrity of backups:
- Daily - Backups are restored to a simulated environment to ensure integrity of data.
- Monthly - Backups are restored to the Beta environment to ensure availability of all data from backup and for verification.
- Quarterly - Stage 1 disaster recovery is tested. The production failover capability is tested by running the application on the Production (Failover) servers during a maintenance period.
- Yearly - Stage 2 disaster recovery is tested. A new simulated Production instance of the application is built from backup in order to test the processes and procedures required to rebuild the application and infrastructure in case of disaster.