Comment Data storage is one of the most complex areas of IT infrastructure, as it needs to fit in with a range of conflicting requirements. Storage architectures have to be fast enough to meet the demands of users and applications, but without breaking the budget. They must deliver enough capacity to meet ever growing volumes of data while being reliable.
Not surprisingly, it turns out that most organisations still have a need for on-premises storage infrastructure, despite the lure of cloud-based storage services that promise to take away all the complexity and replace it with per-gigabyte pricing models. This is for various reasons, such as concerns over data governance, security worries or performance issues with online storage.
Whatever the reason, on-premises storage for most organisations is not going to disappear anytime soon. But it is facing new challenges as data volumes continue to expand, new workloads are being introduced into the enterprise portfolio, and users demand ever higher performance.
For small to mid-market businesses, on-premises storage has long been delivered by network-attached storage (NAS) and storage area network (SAN) platforms, both of which were developed to provide a shared pool of storage. However, while a NAS box attaches directly to a corporate Ethernet LAN and serves up files, a SAN may comprise multiple storage devices connected to servers via a dedicated network, often using high-speed Fibre Channel links.
Another key difference between the two is that while NAS exposes a file system to the network via protocols such as NFS or SMB, a SAN provides block-level storage that looks like a locally attached drive to servers on the SAN. As SAN and NAS has matured, some storage arrays offer unified block and file services in a single device. In this latter case, it is important that the storage platform you choose offers the ability to expand capacity seamlessly across both SAN and NAS environments as your business grows.
Enterprise workloads are evolving and calling for greater levels of performance. The volumes of data that organisations have to deal with also keep growing, and storage systems now have to serve applications such as analytics to make sense of all that data. Meanwhile, storage systems are also expected to deal with the introduction of new challenges such as DevOps, which brings a rapid cadence to the software development process and calls for the on-demand provisioning of new resources.
This demand for fast and easy access to data sets has driven the uptake of flash storage in the enterprise. First this took the form of hybrid storage arrays as a cache or hot tier front-end to a bunch of disk drives to accelerate read and write performance, but all-flash arrays started to gain market share as the relative cost of flash memory chips came down. IDC figures from the third quarter of 2019 show the all-flash array (AFA) market revenue was up 11.3 per cent year on year, for example.
Despite this, organisations are seeing that SAN and NAS systems may soon be too slow for some applications. Partly, this is because many flash-based solid state drives (SSDs) were manufactured with legacy host interfaces like SATA and SAS and came in disk drive form factors, in order to maintain compatibility with enterprise server and storage systems based on hard drives.
Because SSDs based on NAND flash are significantly faster than rotating hard drives, interfaces such as SAS and SATA are actually a bottleneck to the potential throughput of the drive. To fix this, SSD makers started to produce drives that used the PCIe bus, which offers higher speed and connects directly to the host processor in a server or storage controller, but there was no standard protocol stack to support this, which hindered the uptake of such solutions.
Fortunately, a group of storage and chip vendors got together and created Non-Volatile Memory Express (NVMe) as a new storage protocol optimised for high performance, designed from the ground up for non-volatile memory media.
NVMe includes a number of features that provide a significant improvement in I/O performance and reduced latency compared to legacy protocols. NVMe drives typically support four lanes of PCIe 3.0 each for a total of 4GBps bandwidth, compared with 12Gbps (about 1.5GBps) for SAS.
Another feature is support for multiple I/O queues up to 65,535 taking advantage of the internal parallelism of NAND flash storage. Each queue can also support up to 65,535 commands, compared with a single queue for SATA and SAS interfaces, supporting 32 and 256 commands respectively. This means that NVMe storage systems should be far less prone to the performance degradation that SAS and SATA can experience when heavily loaded with requests for data.
Perhaps equally importantly, NVMe allows applications to request data more or less directly from an SSD, while in the traditional storage protocol stack the commands have to pass through multiple layers on their way to the target drives. NVMe also provides a streamlined and simple command set that requires far fewer CPU cycles to process I/O requests than SAS or SATA, according to the NVMe standards body. This delivers higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. For the ultimate application solution performance, customers will want to architect end to end NVMe infrastructure where the host, network, and storage all incorporate NVMe technology.
Switching to NVMe thus speeds data transfers between the SSD and the host processor in a server, or in the case of a NAS box or SAN storage array, between the SSD and the controller. But this just shifts the bottleneck to the host interface or fabric that links the storage device to the systems it serves.
This is because a fabric such as Fibre Channel is essentially just a transport for SCSI commands, as is iSCSI (which transfers SCSI commands over TCP/IP, typically using Ethernet). However, fast your NVMe array is, data is still being sent across the network using a legacy storage protocol stack, with the additional latency that implies.
One answer to this is to extend the NVMe protocol across the network, using it over fabric technologies that are already widely used for storage purposes in enterprises, such as Fibre Channel, and Ethernet. This has collectively become known as NVMe over Fabrics (NVMe-oF).
NVMe-oF differs according to the fabric technology employed. Running it over Fibre Channel (FC-NVMe) is relatively simple for environments already invested in that infrastructure. For others such as Ethernet, this may necessitate the use of special network adapter cards supporting Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE).
This end-to-end implementation of NVMe allows enterprises to scale out their storage infrastructure without losing any of the low latency advantages offered by the protocol, effectively by allowing requested data to be transferred directly from the storage device into the computer systems memory by the network adapter, needing no further intervention by the host processor.
Naturally, there is a great deal of technical detail work in making all of this new technology hang together seamlessly, and organisations will likely find it beneficial to work with a vendor partner that can supply not just the storage, but servers with the necessary adapters and even network switches optimised for NVMe-oF to provide a true end to end NVMe solution. Partnering with one partner can also offer complete support and components validated to work together in an end-to-end solution.
In fact, the growing complexity of storage means that for many small to mid-market businesses, choosing a data platform that can simplify many processes will be vital. This may include intelligently placing data into storage media tiers or providing quality of service (QoS) capabilities to ensure that business-critical applications get the system resources they need.
Having previously stated that organisations still need on-premises storage, it is also true that many are looking to take advantage of cloud-based storage wherever it is advantageous to do so. One of the reasons for doing so is economics, with many cloud providers offering storage services that cost just pennies per GB per month.
While on-premises storage systems have the performance needed by applications, they can easily get cluttered up with data that is no longer in everyday use, so a key feature for any SAN or NAS platform is to enable this to be migrated to a lower cost storage tier such as a cloud storage service, when speed of access is no longer so important.
Some storage array platforms go beyond this to allow customers to deploy a cloud-hosted version of the product onto one or more of the big public clouds such as AWS or Azure. With this scenario, organisations can easily put in place data protection and business continuity strategies by using the cloud-hosted version to replicate snapshots and disaster recovery copies from their on-premises system.
This enables them to build high availability using the consumption-based expense model of the cloud instead of the cost of a physical second site. It is also important to avoid cloud vendor lock in and choose a solution that has the choice of multiple cloud providers.
Multi-cloud management is coming to the forefront of the cloud discussion due to escalating cloud storage costs and the ability to pick and choose multiple vendors gives customers more business leverage.
As technologies such as NAND flash and storage-class memories mature and become less costly, storage arrays based on them will eventually edge out those based on disk drives. NVMe has been designed to make the most of the capabilities these offer and with NVMe-oF now delivering these benefits end-to-end in the data centre, organisations should consider this a key technology to future proofing their storage infrastructure.
Sponsored: Webcast: Build the next generation of your business in the public cloud
- EDA moves to the cloud - eeNews Europe - December 29th, 2020
- One Option You Shouldnt Overlook When Setting Up a Security Camera - The New York Times - December 29th, 2020
- This is the best Google Cloud Print alternative - Android Police - December 29th, 2020
- Finding the balance between edge AI vs. cloud AI - TechTarget - December 29th, 2020
- Cybercriminals to focus on remote and cloud-based systems in UAE next year - Gulf Business - December 29th, 2020
- Top 10 Hyperconverged Infrastructure (HCI) Solutions - Datamation - December 29th, 2020
- The Diminishing Role of Operating Systems | IT Pro - ITPro Today - December 29th, 2020
- Building a Better U.S. Approach to TikTok and Beyond - Lawfare - December 29th, 2020
- Legacy IT: The hidden problem of digital transformation - SC Magazine - December 29th, 2020
- TGen Leverages phoenixNAP's Hardware-as-a-Service Powered by Intel to Empower COVID-19 Research - PR Web - December 29th, 2020
- Global Cloud Server Market Share, Competition Analysis, COVID-19 Impact Analysis & Projected Recovery, and Market Sizing & Forecast to 2026 -... - December 29th, 2020
- Private Cloud Server Market Report, History And Forecast 2020-2025, Breakdown Data By Manufacturers, Key Regions, Types And Application - The Monitor - December 29th, 2020
- Bare Metal Cloud Market Poised to Expand at a Robust Pace Over 2025 - Farming Sector - December 29th, 2020
- 4 reasons your business needs to switch to cloud servers - TechEngage - December 19th, 2020
- Microsoft is designing its own ARM-based processor for Surface and cloud servers - Digital Trends - December 19th, 2020
- Dedicated server or cloud server: which one to choose? - Business MattersBusiness Matters - December 19th, 2020
- Cybersecurity expert: After Russian hack, common security tools, including cloud-based multi-factor systems, shown to be less effective in preventing... - December 19th, 2020
- Remote and cloud-based systems to be ruthlessly targeted next year - Help Net Security - December 19th, 2020
- The ROI of Cloud-Based Email - TechDecisions - December 19th, 2020
- The Advantages of Running your ERP Off-Premise During the COVID-19 Era - BBN Times - December 19th, 2020
- The balkanization of the cloud is bad for everyone - MIT Technology Review - December 19th, 2020
- When Is a Good Time For SMBs To Move Their IT Infrastructure To the Cloud? - Entrepreneur - December 19th, 2020
- AWS reveals array of new cloud observability and other new tools at re:Invent - SiliconANGLE News - December 19th, 2020
- TaxBandits Offers Cloud Based Solution For E-filing Forms W-2, 1099, 94x Series, and ACA 1095 for 2020 Tax Year. - CPAPracticeAdvisor.com - December 19th, 2020
- Programmable NICs Will Empower the Future of the Network - CEOWORLD magazine - December 19th, 2020
- Amazon's weird Halo band is now available if you want to pay $4/month for fitness tracking - Android Police - December 19th, 2020
- Microsoft Says Its Systems Were Exposed to SolarWinds Hack - Data Center Knowledge - December 19th, 2020
- Holiday-proof your Business with Cloud Solutions - The Edge Markets MY - December 19th, 2020
- Gurucul Cloud-native Analytics-driven XDR Platform Sets New Standard for Real-Time Threat Detection and Incident Response - Business Wire - December 19th, 2020
- Run Kubernetes at the edge with these K8s distributions - TechTarget - December 19th, 2020
- How Technology Is Revolutionizing the Unorganized Parking Sector in India and the Road Ahead - News18 - December 19th, 2020
- Cloud Server: The advantages and why Kronos Cloud is worth trying - Programming Insider - December 6th, 2020
- Amazon details cause of AWS outage that hobbled thousands of online sites and services - GeekWire - December 6th, 2020
- Can we really trust the Cloud with our data? - The Next Web - December 6th, 2020
- An Introduction to Cloud Computing | Ethical Hacking | EC-Council Blog - EC-Council Blog - December 6th, 2020
- Google Cloud Will Not Be Able To Overtake Microsoft Azure - Forbes - December 6th, 2020
- Google builds out Cloud with Actifio acquisition Blocks and Files - Blocks and Files - December 6th, 2020
- 5 advantages of a cloud disaster recovery plan - BAI Banking Strategies - December 6th, 2020
- Your Digital Transformation Will Be as Successful as the Foundation Its Built On - CMSWire - December 6th, 2020
- Kubernetes: What You Need To Know - Forbes - December 6th, 2020
- 3 cloud computing trends to watch in 2021 - TechHQ - December 6th, 2020
- Amazon Web Service Explains Its Major OutageAnd Other Small Business Tech News - Forbes - December 6th, 2020
- How AWS is computing the future of the cloud - SiliconANGLE News - December 6th, 2020
- ONF Announces Aether 5G Connected Edge Cloud Platform Being Used as the Software Platform for Pronto Project - sUAS News - December 6th, 2020
- International health IT week in review: December 6 - Pulse+IT - December 6th, 2020
- Lenovo boosts low end all-flash array with end-to-end NVMe - Blocks and Files - December 6th, 2020
- Bull of the Day: Baidu (BIDU) - Yahoo Finance - December 6th, 2020
- A re:Invent like no other shows an AWS capitalizing on 2020 chaos - Diginomica - December 6th, 2020
- The Mystery Behind the Aarogya Setu App - TheLeaflet - The Leaflet - December 6th, 2020
- People Cant Vacuum Or Use Their Doorbell Because Amazons Cloud Servers Are Down - Gizmodo Australia - November 26th, 2020
- Inside Innovation: 'In the cloud we trust' doesn't hack it for ransomware protection - Daily Commercial News - November 26th, 2020
- Will edge computing become the new cloud in 2021? - TechRepublic - November 26th, 2020
- Cloud Server Market ? What Factors Will Drive The Market In Upcoming Years And How It Is Going To Impact On Global Industry | (2020-2026) - The... - November 26th, 2020
- IP surveillance: The storage it needs, on-premise and in the cloud - ComputerWeekly.com - November 26th, 2020
- Does AD CS Work in the Cloud? - Security Boulevard - November 26th, 2020
- SKT Unveils its AI Chip and New Plans for AI Semiconductor Business - HPCwire - November 26th, 2020
- Calculating the Total Cost of Hybrid Cloud - Data Center Knowledge - November 26th, 2020
- Cloud Server Market Research Report: Overview With Geographical Segmentation By Revenue With Forecast 2026 - Cheshire Media - November 26th, 2020
- Cryptomining Worm Uses Third-Party Software to Target Cloud - Security Intelligence - November 26th, 2020
- SKT new AI chip changes the company's AI semiconductor bu... - evertiq.com - November 26th, 2020
- Should IT Support Always Be Handled In-House? Are There Other Options? - ABCmoney.co.uk - November 26th, 2020
- Home Automation System Market worth $63.2 billion by 2025 - Exclusive Report by MarketsandMarkets - PRNewswire - November 26th, 2020
- Global Managed Servers Industry Market Growth Graph To Demonstrate Inclination Towards Positive Axis By 2026 - The Courier - November 26th, 2020
- Everett Police will put body cameras on all its officers - Snohomish County Tribune - November 26th, 2020
- NetRange moves browser to the cloud in new smart TV range - Broadband TV News - November 26th, 2020
- Hackers steal and save Spotify login information - Somag News - November 26th, 2020
- Three Reasons Why You Should Invest in Cloud-based Email - My TechDecisions - TechDecisions - November 4th, 2020
- No way to go but up as cloud solutions shape the future of business - CNN Philippines - November 4th, 2020
- The global cyber insurance market is expected to reach a value of $70,671.9 million by 2030, from $5,573.2 million in 2019 - Yahoo Finance - November 4th, 2020
- Moving to cloud-native applications and data with Kubernetes and Apache Cassandra - JAXenter - November 4th, 2020
- How Digital Twins Accelerate the Growth of IoT - IoT For All - November 4th, 2020
- Bluebeam expands its global Studio data infrastructure - Planning, BIM & Construction Today - November 4th, 2020
- The journey to a cloud BSS - Ericsson - November 4th, 2020
- Edge computing strategies will determine the next cloud frontier - TechTarget - November 4th, 2020
- How do we protect the hybrid workplace? - TechHQ - November 4th, 2020
- Evolution of File Sharing and its method - InfotechLead.com - November 4th, 2020
- Amazon: The Coming Graviton3 - Seeking Alpha - November 4th, 2020
- Windows Admin Center is Coming to the Azure Portal - Petri.com - November 4th, 2020
- Sophos Uncovers Attackers Targeting Non-Governmental Organizations in Myanmar With New 'KilllSomeOne' Backdoor - GlobeNewswire - November 4th, 2020
- IBM Delivered An RDi Update, Too - IT Jungle - November 4th, 2020