Comment Data storage is one of the most complex areas of IT infrastructure, as it needs to fit in with a range of conflicting requirements. Storage architectures have to be fast enough to meet the demands of users and applications, but without breaking the budget. They must deliver enough capacity to meet ever growing volumes of data while being reliable.
Not surprisingly, it turns out that most organisations still have a need for on-premises storage infrastructure, despite the lure of cloud-based storage services that promise to take away all the complexity and replace it with per-gigabyte pricing models. This is for various reasons, such as concerns over data governance, security worries or performance issues with online storage.
Whatever the reason, on-premises storage for most organisations is not going to disappear anytime soon. But it is facing new challenges as data volumes continue to expand, new workloads are being introduced into the enterprise portfolio, and users demand ever higher performance.
For small to mid-market businesses, on-premises storage has long been delivered by network-attached storage (NAS) and storage area network (SAN) platforms, both of which were developed to provide a shared pool of storage. However, while a NAS box attaches directly to a corporate Ethernet LAN and serves up files, a SAN may comprise multiple storage devices connected to servers via a dedicated network, often using high-speed Fibre Channel links.
Another key difference between the two is that while NAS exposes a file system to the network via protocols such as NFS or SMB, a SAN provides block-level storage that looks like a locally attached drive to servers on the SAN. As SAN and NAS has matured, some storage arrays offer unified block and file services in a single device. In this latter case, it is important that the storage platform you choose offers the ability to expand capacity seamlessly across both SAN and NAS environments as your business grows.
Enterprise workloads are evolving and calling for greater levels of performance. The volumes of data that organisations have to deal with also keep growing, and storage systems now have to serve applications such as analytics to make sense of all that data. Meanwhile, storage systems are also expected to deal with the introduction of new challenges such as DevOps, which brings a rapid cadence to the software development process and calls for the on-demand provisioning of new resources.
This demand for fast and easy access to data sets has driven the uptake of flash storage in the enterprise. First this took the form of hybrid storage arrays as a cache or hot tier front-end to a bunch of disk drives to accelerate read and write performance, but all-flash arrays started to gain market share as the relative cost of flash memory chips came down. IDC figures from the third quarter of 2019 show the all-flash array (AFA) market revenue was up 11.3 per cent year on year, for example.
Despite this, organisations are seeing that SAN and NAS systems may soon be too slow for some applications. Partly, this is because many flash-based solid state drives (SSDs) were manufactured with legacy host interfaces like SATA and SAS and came in disk drive form factors, in order to maintain compatibility with enterprise server and storage systems based on hard drives.
Because SSDs based on NAND flash are significantly faster than rotating hard drives, interfaces such as SAS and SATA are actually a bottleneck to the potential throughput of the drive. To fix this, SSD makers started to produce drives that used the PCIe bus, which offers higher speed and connects directly to the host processor in a server or storage controller, but there was no standard protocol stack to support this, which hindered the uptake of such solutions.
Fortunately, a group of storage and chip vendors got together and created Non-Volatile Memory Express (NVMe) as a new storage protocol optimised for high performance, designed from the ground up for non-volatile memory media.
NVMe includes a number of features that provide a significant improvement in I/O performance and reduced latency compared to legacy protocols. NVMe drives typically support four lanes of PCIe 3.0 each for a total of 4GBps bandwidth, compared with 12Gbps (about 1.5GBps) for SAS.
Another feature is support for multiple I/O queues up to 65,535 taking advantage of the internal parallelism of NAND flash storage. Each queue can also support up to 65,535 commands, compared with a single queue for SATA and SAS interfaces, supporting 32 and 256 commands respectively. This means that NVMe storage systems should be far less prone to the performance degradation that SAS and SATA can experience when heavily loaded with requests for data.
Perhaps equally importantly, NVMe allows applications to request data more or less directly from an SSD, while in the traditional storage protocol stack the commands have to pass through multiple layers on their way to the target drives. NVMe also provides a streamlined and simple command set that requires far fewer CPU cycles to process I/O requests than SAS or SATA, according to the NVMe standards body. This delivers higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. For the ultimate application solution performance, customers will want to architect end to end NVMe infrastructure where the host, network, and storage all incorporate NVMe technology.
Switching to NVMe thus speeds data transfers between the SSD and the host processor in a server, or in the case of a NAS box or SAN storage array, between the SSD and the controller. But this just shifts the bottleneck to the host interface or fabric that links the storage device to the systems it serves.
This is because a fabric such as Fibre Channel is essentially just a transport for SCSI commands, as is iSCSI (which transfers SCSI commands over TCP/IP, typically using Ethernet). However, fast your NVMe array is, data is still being sent across the network using a legacy storage protocol stack, with the additional latency that implies.
One answer to this is to extend the NVMe protocol across the network, using it over fabric technologies that are already widely used for storage purposes in enterprises, such as Fibre Channel, and Ethernet. This has collectively become known as NVMe over Fabrics (NVMe-oF).
NVMe-oF differs according to the fabric technology employed. Running it over Fibre Channel (FC-NVMe) is relatively simple for environments already invested in that infrastructure. For others such as Ethernet, this may necessitate the use of special network adapter cards supporting Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE).
This end-to-end implementation of NVMe allows enterprises to scale out their storage infrastructure without losing any of the low latency advantages offered by the protocol, effectively by allowing requested data to be transferred directly from the storage device into the computer systems memory by the network adapter, needing no further intervention by the host processor.
Naturally, there is a great deal of technical detail work in making all of this new technology hang together seamlessly, and organisations will likely find it beneficial to work with a vendor partner that can supply not just the storage, but servers with the necessary adapters and even network switches optimised for NVMe-oF to provide a true end to end NVMe solution. Partnering with one partner can also offer complete support and components validated to work together in an end-to-end solution.
In fact, the growing complexity of storage means that for many small to mid-market businesses, choosing a data platform that can simplify many processes will be vital. This may include intelligently placing data into storage media tiers or providing quality of service (QoS) capabilities to ensure that business-critical applications get the system resources they need.
Having previously stated that organisations still need on-premises storage, it is also true that many are looking to take advantage of cloud-based storage wherever it is advantageous to do so. One of the reasons for doing so is economics, with many cloud providers offering storage services that cost just pennies per GB per month.
While on-premises storage systems have the performance needed by applications, they can easily get cluttered up with data that is no longer in everyday use, so a key feature for any SAN or NAS platform is to enable this to be migrated to a lower cost storage tier such as a cloud storage service, when speed of access is no longer so important.
Some storage array platforms go beyond this to allow customers to deploy a cloud-hosted version of the product onto one or more of the big public clouds such as AWS or Azure. With this scenario, organisations can easily put in place data protection and business continuity strategies by using the cloud-hosted version to replicate snapshots and disaster recovery copies from their on-premises system.
This enables them to build high availability using the consumption-based expense model of the cloud instead of the cost of a physical second site. It is also important to avoid cloud vendor lock in and choose a solution that has the choice of multiple cloud providers.
Multi-cloud management is coming to the forefront of the cloud discussion due to escalating cloud storage costs and the ability to pick and choose multiple vendors gives customers more business leverage.
As technologies such as NAND flash and storage-class memories mature and become less costly, storage arrays based on them will eventually edge out those based on disk drives. NVMe has been designed to make the most of the capabilities these offer and with NVMe-oF now delivering these benefits end-to-end in the data centre, organisations should consider this a key technology to future proofing their storage infrastructure.
Sponsored: Webcast: Build the next generation of your business in the public cloud
- PAM as a Service: Its All a Matter of Trust - Security Boulevard - June 2nd, 2020
- How To Best Adapt Your Business When The World Is Moving Online - Forbes - June 2nd, 2020
- Cloud computing via satellite to drive 52 Exabytes of traffic by 2029: NSR - SatelliteProME.com - June 2nd, 2020
- Multinational Insurance Company Completes Upgrade of Majesco Policy for P&C from On-Premise to Majesco CloudInsurer to Bolster Growth Strategy -... - June 2nd, 2020
- COVID-19 Impact on Healthcare Cloud Computing Market Marked US$ 13 Bn in forecast Years 2025 - 3rd Watch News - June 2nd, 2020
- Cloud computing, future trends to be followed in the industry - Optocrypto - June 2nd, 2020
- You couldn't do this already? AWS adds size and bandwidth growth to FSx for Windows File Server - Blocks and Files - June 2nd, 2020
- Upstream Security Partners With Amazon Web Services to Enhance Automotive Cybersecurity - PRNewswire - June 2nd, 2020
- Improvements on the verify domain error in Office 365 - TechGenix - June 2nd, 2020
- Digital transformation held back by lack of skilled people - ComputerWeekly.com - June 2nd, 2020
- NTT Com internal cloud server hacked, information on 621 customers stolen - DatacenterDynamics - June 2nd, 2020
- Where is the edge in edge computing? And who gets to decide? - ZDNet - June 2nd, 2020
- Cloud-native architectures will define the vRAN future - 5Gradar - June 2nd, 2020
- Developers recall career 'aha' moments that have shaped their Docker experience - SiliconANGLE News - June 2nd, 2020
- HSBC platform uses AI to analyse trading data thousands of times faster - ComputerWeekly.com - June 2nd, 2020
- CloudBolt Releases Version 9.3 of Its Award-Winning Cloud Management Platform - Container Journal - May 31st, 2020
- Kaminario offers cut-price virtual SAN in the cloud - ComputerWeekly.com - May 31st, 2020
- 4 types of mobile security models and how they work - TechTarget - May 31st, 2020
- Increased cybersecurity for the transportation industry - Commercial Carrier Journal - May 31st, 2020
- Cloud-Based Firewalls Are Key to Protecting Employees While Working Remotely - Security Boulevard - May 31st, 2020
- Cloud storage 101: File, block and object storage in the cloud - ComputerWeekly.com - May 31st, 2020
- Cloud Transition During the COVID-19 Exposing the Enterprise Vulnerabilities - EnterpriseTalk - May 31st, 2020
- The Role of Artificial Intelligence in Ethical Hacking | EC-Council Official Blog - EC-Council Blog - May 31st, 2020
- Shelves are well-stocked with cloud-native tools, but simplicity remains a moving target - SiliconANGLE - May 31st, 2020
- Uncover and overcome cloud threat hunting obstacles - TechTarget - May 26th, 2020
- This extraordinary motherboard is being used by server CPU scavengers - TechRadar India - May 26th, 2020
- VMware reduces hardware footprint of its shiny new K8s-on-vSphere toys - The Register - May 26th, 2020
- How Zoom plans to better secure meetings with end-to-end encryption - TechRepublic - May 26th, 2020
- VMware, Dell level up their combined on-prem cloud with much more computing grunt - The Register - May 26th, 2020
- Accelerator Card Market Will Witness Substantial Growth in the Upcoming years by 2027 - WaterCloud News - May 26th, 2020
- Uber India deploys Canon information management solution- Therefore for operational workflow - CRN.in - May 26th, 2020
- Potential Impact of COVID-19 on Research Report prospects the Server Backup Software Market - Cole of Duty - May 26th, 2020
- Do You Know Where Your Servers Come From? Heres Why Securing The Supply Chain Matters - Forbes - May 26th, 2020
- Live analytics without vendor lock-in? It's more likely than you think, says Redis Labs - The Register - May 26th, 2020
- Latest Forecast on Government Cloud Market Emerging Industries, Growth, Remarkable Developments and Key Players| Global Future Prospects 2025 - 3rd... - May 26th, 2020
- Cloud Accounting Software Market Research Report Comprising Development Trends 2020, Key Manufacturers and Competitive Landscape to 2025 - Cole of... - May 26th, 2020
- Gartner: How and why cloud providers need to support their customers through Covid-19 - Cloud Tech - May 22nd, 2020
- The Connection Between Cloud Service Providers and Cyber Resilience - Security Intelligence - May 22nd, 2020
- Google And Dell Pave The Way For File Data In The Cloud - The Next Platform - May 22nd, 2020
- Veeam teams up with Kasten for containerised app backup Blocks and Files - Blocks and Files - May 22nd, 2020
- Hybrid cloud: The key to surviving and thriving during the pandemic - WTOP - May 22nd, 2020
- Global Bare Metal Cloud Market : Industry Analysis and Forecast... - Azizsalon News - May 22nd, 2020
- Exabeam sees more than half of new and add-on recurring revenue from cloud offering - Help Net Security - May 22nd, 2020
- OnShip Brings its Parcel & Freight Shipping Transportation Management Platform to the Cloud with Cameyo - Supply and Demand Chain Executive - May 22nd, 2020
- 'What is Dropbox?': How to use the cloud-based file-storage service for collaboration - Business Insider - Business Insider - May 22nd, 2020
- Couchbase Announces $105 Million Equity Investment Led by GPI Capital to Fuel Its Next Phase of Growth and Cloud Innovation - GlobeNewswire - May 22nd, 2020
- The Register calls for aid, and Microsoft's Rohan Kumar will answer... our questions about SQL Edge and Azure Synapse - The Register - May 22nd, 2020
- What are the different types of cloud load balancing? - TechTarget - May 22nd, 2020
- How data centers will become automated and self-reliant - TechHQ - May 22nd, 2020
- Masayoshi Son says AWS and Microsoft will buy more chipsets from the SoftBank Vision Fund-backed Arm, and not - Business Insider India - May 22nd, 2020
- Chinese IPOs hang in the balance as Senate and Nasdaq change rules - Data Economy - May 22nd, 2020
- Portworx upbeat on container storage revenues Blocks and Files - Blocks and Files - May 22nd, 2020
- New study Global Managed Servers Market 2019 | Growth Opportunities, Investment Feasibility, Market Share And Forecast 2025 - Cole of Duty - May 22nd, 2020
- New Study Finds that IT Pros Are Worried About Corporate Data Security - Database Trends and Applications - May 19th, 2020
- Get your head in the cloud: why cloud is crucial for sustainable business - New Zealand News Centre - Microsoft - May 19th, 2020
- The Global Public Cloud Services Market is expected to grow by $ 221.84 billion during 2020-2024 progressing at a CAGR of 19% during the forecast... - May 19th, 2020
- Traditional or Cloud Antivirus Solutions Which is Best? - PC Tech Magazine - May 19th, 2020
- Moving beyond Covid-19: what does the future of work look like? - ETCIO.com - May 19th, 2020
- AWS unleashes custom Arm processor the Graviton2 in new EC2 M6g instance type - The Register - May 14th, 2020
- Pandemic Shows The Value Of The Public Cloud - The Next Platform - May 14th, 2020
- Jigsaw24 Expands Via24 Cloud Services With Deployment of EditShares EFSv - Broadcasting & Cable - May 14th, 2020
- The age of the ethical cloud is green and for everyone Intelligent CIO Europe - Intelligent CIO Africa - May 14th, 2020
- The Future of Artificial Intelligence: Edge Intelligence - Analytics Insight - May 14th, 2020
- How cloud is accelerating the growth of digital payments - TechHQ - May 14th, 2020
- Live Webinar Preview: Commands & Custom Scripting for Remote Application Installs - Security Boulevard - May 14th, 2020
- Private Cloud Server Market Growth by Top Companies, Trends by Types and Application, Forecast to 2026 - Cole of Duty - May 14th, 2020
- Swarm Theory: Lessons from nature in the advancement of robotics - Techerati - May 14th, 2020
- What are the Differences Between IaaS, PaaS, and SaaS? - stopthefud - May 14th, 2020
- Zoom Settles with NY AG over Privacy and Security Concerns - Security Magazine - May 14th, 2020
- Codestone helps shipping agent to cloud-based infrastructure - Codestone - May 14th, 2020
- Server sales went through the roof in the first three months of 2020. Enjoy it while it lasts, Dell, HPE, and pals - The Register - May 14th, 2020
- Global Cloud Infrastructure Testing Market Research Report 2020 By Size, Share, Trends and Analysis up to 2025. - Cole of Duty - May 14th, 2020
- Digital Harmonic to Bring its Powerful AI-Driven Image and Video Enhancing Solution to the Federal Market - Business Wire - May 14th, 2020
- The role of the data centre in the future of Data Management - Data Economy - May 14th, 2020
- We'd love to come up with a Harbor container ship pun but we're too corona-frazzled. Version 2.0 is out - The Register - May 14th, 2020
- Edge Intelligence: The Next Wave of AI - EE Times India - May 14th, 2020
- Patch by Friday or compromised by Monday: Salt exploit exposes Infrastructure-as-Code tools threat - SC Magazine UK - May 6th, 2020
- Serverless Exists In The Cloud and Both Need Servers - Computer Business Review - May 6th, 2020
- Analysis on Impact of COVID-19- Rugged Servers Market 2020-2024 | Increased Adoption of Cloud Applications to Boost Growth | Technavio - Business Wire - May 6th, 2020
- Privitar Announces New Native Integration With Google Cloud Platform - Business Wire - May 6th, 2020