This fall SVG will be presenting a series of White Papers covering the latest advancements and trends in sports-production technology. The full series of SVGs Tech Insight White Papers can be found in the SVG Fall SportsTech Journal HERE.
Following the height of the 2020 global pandemic, live sports are starting to re-emerge worldwide albeit predominantly behind closed doors. For the majority of sports fans, video is the only way they can watch and engage with their favorite teams or players. This means the quality of the viewing experience itself has become even more critical.
With UHD being adopted by both households and broadcasters around the world, there is a marked expectation around visual quality. To realize these expectations in the immediate term, it will be necessary for some years to up-convert from HD to UHD when creating 4K UHD sports channels and content.
This is not so different from the early days of HD, where SD sporting related content had to be up-converted to HD. In the intervening years, however, machine learning as a technology has progressed sufficiently to be a serious contender for performing better up-conversions than with more conventional techniques, specifically designed to work for TV content.
Ideally, we want to process HD content into UHD with a simple black box arrangement.
The problem with conventional up-conversion, though, is that it does not offer an improved resolution, so does not fully meet the expectations of the viewer at home watching on a UHD TV. The question, therefore, becomes: can we do better for the sports fan? If so, how?
UHD is a progressive scan format, with the native TV formats being 38402160, known as 2160p59.64 (usually abbreviated to 2160p60) or 2160p50. The corresponding HD formats, with the frame/field rates set by region, are either progressive 1280720 (720p60 or 720p50) or interlaced 19201080 (1080i30 or 1080i25).
Conversion from HD to UHD for progressive images at the same rate is fairly simple. It can be achieved using spatial processing only. Traditionally, this might typically use a bi-cubic interpolation filter, (a 2-dimensional interpolation commonly used for photographic image scaling.) This uses a grid of 44 source pixels and interpolates intermediate locations in the center of the grid. The conversion from 1280720 to 38402160 requires a 3x scaling factor in each dimension and is almost the ideal case for an upsampling filter.
These types of filters can only interpolate, resulting in an image that is a better result than nearest-neighbor or bi-linear interpolation, but does not have the appearance of being a higher resolution.
Machine Learning (ML) is a technique whereby a neural network learns patterns from a set of training data. Images are large, and it becomes unfeasible to create neural networks that process this data as a complete set. So, a different structure is used for image processing, known as Convolutional Neural Networks (CNNs). CNNs are structured to extract features from the images by successively processing subsets from the source image and then processes the features rather than the raw pixels.
Up-conversion process with neural network processing
The inbuilt non-linearity, in combination with feature-based processing, mean CNNs can invent data not in the original image. In the case of up-conversion, we are interested in the ability to create plausible new content that was not present in the original image, but that doesnt modify the nature of the image too much. The CNN used to create the UHD data from the HD source is known as the Generator CNN.
When input source data needs to be propagated through the whole chain, possibly with scaling involved, then a specific variant of a CNN known as a Residual Network (ResNet) is used. A ResNet has a number of stages, each of which includes a contribution from a bypass path that carries the input data. For this study, a ResNet with scaling stages towards the end of the chain was used as the Generator CNN.
For the Generator CNN to do its job, it must be trained with a set of known data patches of reference images and a comparison is made between the output and the original. For training, the originals are a set of high-resolution UHD images, down-sampled to produce HD source images, then up-converted and finally compared to the originals.
The difference between the original and synthesized UHD images is calculated by the compare function with the error signal fed back to the Generator CNN. Progressively, the Generator CNN learns to create an image with features more similar to original UHD images.
The training process is dependent on the data set used for training, and the neural network tries to fit the characteristics seen during training onto the current image. This is intriguingly illustrated in Googles AI Blog , where a neural network presented with a random noise pattern introduces shapes like the ones used during training. It is important that a diverse, representative content set is used for training. Patches from about 800 different images were used for training during the process of MediaKinds research.
The compare function affects the way the Generator CNN learns to process the HD source data. It is easy to calculate a sum of absolute differences between original and synthesized. This causes an issue due to training set imbalance; in this case, the imbalance is that real pictures have large proportions with relatively little fine detail, so the data set is biased towards regenerating a result like that which is very similar to the use of a bicubic interpolation filter.
This doesnt really achieve the objective of creating plausible fine detail.
Generative Adversarial Neural Networks (GANs) are a relatively new concept , where a second neural network, known as the Discriminator CNN, is used and is itself trained during the training process of the Generator CNN. The Discriminator CNN learns to detect the difference between features that are characteristic of original UHD images and synthesized UHD images. During training, the Discriminator CNN sees either an original UHD image or a synthesized UHD image, with the detection correctness fed back to the discriminator and, if the image was a synthesized one, also fed back to the Generator CNN.
Each CNN is attempting to beat the other: the Generator by creating images that have characteristics more like originals, while the Discriminator becomes better at detecting synthesized images.
The result is the synthesis of feature details that are characteristic of original UHD images.
With a GAN approach, there is no real constraint to the ability of the Generator CNN to create new detail everywhere. This means the Generator CNN can create images that diverge from the original image in more general ways. A combination of both compare functions can offer a better balance, retaining the detail regeneration, but also limiting divergence. This produces results that are subjectively better than conventional up-conversion.
Conversion from 1080i60 to 2160p60 is necessarily more complex than from 720p60. Starting from 1080i, there are three basic approaches to up-conversion:
Training data is required here, which must come from 2160p video sequences. This enables a set of fields to be created, which are then downsampled, with each field coming from one frame in the original 2160p sequence, so the fields are not temporally co-located.
Surprisingly, results from field-based up-conversion tended to be better than using de-interlaced frame conversion, despite using sophisticated motion-compensated de-interlacing: the frame-based conversion being dominated by the artifacts from the de-interlacing process. However, it is clear that potentially useful data from the opposite fields did not contribute to the result, and the field-based approach missed data that could produce a better result.
A solution to this is to use multiple fields data as the source data directly into a modified Generator CNN, letting the GAN learn how best to perform the deinterlacing function. This approach was adopted and re-trained with a new set of video-based data, where adjacent fields were also provided.
This led to both high visual spatial resolution and good temporal stability. These are, of course, best viewed as a video sequence, however an example of one frame from a test sequence shows the comparison:
Comparison of a sample frame from different up-conversion techniques against original UHD
Up-conversion using a hybrid GAN with multiple fields was effective across a range of content, but is especially relevant for the visual sports experience to the consumer. This offers a realistic means by which content that has more of the appearance of UHD can be created from both progressive and interlaced HD source, which in turn can enable an improved experience for the fan at home when watching a sports UHD channel.
1 A. Mordvintsev, C. Olah and M. Tyka, Inceptionism: Going Deeper into Neural Networks, 2015. [Online]. Available: https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
2 I. e. a. Goodfellow, Generative Adversarial Nets, Neural Information Processing Systems Proceedings, vol. 27, 2014.
- Boosting Weather Prediction with Machine Learning - Eos - November 28th, 2020
- Machine learning - it's all about the data - KHL Group - November 28th, 2020
- Need a Hypothesis? This A.I. Has One - The New York Times - November 28th, 2020
- Artificial Intelligence and Machine Learning Together To Reach the Culmination of Growth By 2023 - thepolicytimes.com - November 28th, 2020
- Metal Geochemistry Meets Machine Learning in the North Atlantic - Hydro International - November 28th, 2020
- Machine Learning-Based Risk Assessment for Cancer Therapy-Related Cardiac Dysfunction in 4300 Longitudinal Oncology Patients - DocWire News - November 28th, 2020
- Postdoctoral Research Associate in Computer Vision and Machine Learning job with DURHAM UNIVERSITY | 235683 - Times Higher Education (THE) - November 28th, 2020
- The 12 Coolest Machine-Learning Startups Of 2020 - CRN - November 19th, 2020
- Utilizing machine learning to uncover the right content at KMWorld Connect 2020 - KMWorld Magazine - November 19th, 2020
- The way we train AI is fundamentally flawed - MIT Technology Review - November 19th, 2020
- DIY Camera Uses Machine Learning to Audibly Tell You What it Sees - PetaPixel - November 19th, 2020
- Machine Learning Predicts How Cancer Patients Will Respond to Therapy - HealthITAnalytics.com - November 19th, 2020
- This New Machine Learning Tool Might Stop Misinformation - Digital Information World - November 19th, 2020
- Fujitsu, AIST and RIKEN Achieve Unparalleled Speed on MLPerf HPC Machine Learning Processing Benchmark - HPCwire - November 19th, 2020
- SiMa.ai Adopts Arm Technology to Deliver a Purpose-built Heterogeneous Machine Learning Compute Platform for the Embedded Edge - Design and Reuse - November 19th, 2020
- Machine learning removes bias from algorithms and the hiring process - PRNewswire - November 6th, 2020
- Using machine learning to track the pandemic's impact on mental health - MIT News - November 6th, 2020
- AI Recognizes COVID-19 in the Sound of a Cough Machine Learning Times - The Predictive Analytics Times - November 6th, 2020
- The consistency of machine learning and statistical models in predicting clinical risks of individual patients - The BMJ - The BMJ - November 6th, 2020
- PathAI and Gilead Report Data from Machine Learning Model Predictions of Liver Disease Progression and Treatment Response at AASLD's The Liver Meeting... - November 6th, 2020
- Google Introduces New Analytics with Machine Learning and Predictive Models - IBL News - November 6th, 2020
- Free Webinar | Machine Learning and Data Analytics in the Pandemic Era - MIT Sloan - November 6th, 2020
- Global Predictive Analytics Market (2020 to 2025) - Advent of Machine Learning and Artificial Intelligence is Driving Growth - PRNewswire - November 6th, 2020
- Machine learning and predictive analytics work better together - TechTarget - October 31st, 2020
- Microsoft Introduces Lobe: A Free Machine Learning Application That Allows You To Create AI Models Without Coding - MarkTechPost - October 31st, 2020
- Amwell CMO: Google partnership will focus on AI, machine learning to expand into new markets - FierceHealthcare - October 31st, 2020
- Microsoft/MITRE group declares war on machine learning vulnerabilities with Adversarial ML Threat Matrix - Diginomica - October 31st, 2020
- 93% of security operations centers employing AI and machine learning tools to detect advanced threats - Security Magazine - October 31st, 2020
- Machine Learning in Insurance Market(COVID-19 Analysis): Indoor Applications Projected to be the Most Attractive Segment during 2020-2027 - Global... - October 31st, 2020
- Leveraging Machine Learning and IDP to Scale Your Automation Program - AiiA - October 31st, 2020
- 5 machine learning skills you need in the cloud - TechTarget - October 31st, 2020
- Machine learning approach could detect drivers of atrial fibrillation - Cardiac Rhythm News - October 31st, 2020
- Vanderbilt trans-institutional team shows how next-gen wearable sensor algorithms powered by machine learning could be key to preventing injuries that... - October 31st, 2020
- Machine Learning & Big Data Analytics Education Market Size And Forecast (2020-2026)| With Post Impact Of Covid-19 By Top Leading Players-... - October 31st, 2020
- The security threat of adversarial machine learning is real - TechTalks - October 31st, 2020
- Bridging the Skills Gap for AI and Machine Learning - Integration Developers - October 23rd, 2020
- Nudges and machine learning triples advanced care conversations - Penn Today - October 23rd, 2020
- Machine Learning and AI Can Now Create Plastics That Easily Degrade - Science Times - October 23rd, 2020
- insitro Strengthens Machine Learning-Based Drug Discovery Capabilities with Acquisition of Haystack Sciences - Business Wire - October 23rd, 2020
- Revolutionizing IoT with Machine Learning at the Edge | Perceive's Steve Teig - IoT For All - October 23rd, 2020
- Mastercard Says its AI and Machine Learning Solutions Aim to Stop Fraudulent Activites which have Increased Significantly due to COVID - Crowdfund... - October 23rd, 2020
- Abstract Perspective: Long-term PM2.5 Exposure and the Clinical Application of Machine Learning for Predicting Incident Atrial Fibrillation - DocWire... - October 23rd, 2020
- Machine-Learning Inference Chip Travels to the Edge - Electronic Design - October 23rd, 2020
- Machine Learning Data Catalog Software Market share forecast to witness considerable growth from 2020 to 2025 | By Top Leading Vendors IBM, Alation,... - October 23rd, 2020
- AI and machine learning: a gift, and a curse, for cybersecurity - Healthcare IT News - October 21st, 2020
- Teaming Up with Arm, NXP Ups Its Place in the Machine Learning Industry - News - All About Circuits - October 21st, 2020
- Machine Learning Capabilities Come to the Majority of Open Source Databases with MindsDB AI-Tables - PR Web - October 21st, 2020
- Soleadify secures seed funding for database that uses machine learning to track 40M businesses - TechCrunch - October 21st, 2020
- NXP Announces Expansion of its Scalable Machine Learning Portfolio and Capabilities - GlobeNewswire - October 21st, 2020
- NXP Invests in Au-Zone to Enhance Machine Learning Capabilities - Mobile ID World - October 21st, 2020
- Factories of The Future Are Using Machine Learning Analytics to Optimize Assets - Embedded Computing Design - October 21st, 2020
- Lantronix Brings Advanced AI and Machine Learning to Smart Cameras With New Open-Q 610 SOM Based on the Powerful Qualcomm QCS610 System on Chip (SOC)... - October 21st, 2020
- EMA Webinar to Uncover How Machine Learning and Predictive Analytics Can Improve Workload Automation Outcomes - PR Web - October 21st, 2020
- How To Choose The Best Machine Learning Algorithm For A Particular Problem? - Analytics India Magazine - October 21st, 2020
- AI and Machine Learning Technologies Expected to Play a Key Role in Expanding Multi Billion Dollar Digital Banking Sector: Report - Crowdfund Insider - October 21st, 2020
- EXCLUSIVE: Amazon AI executive explains three things every business needs to address before using machine lear - Business Insider India - October 21st, 2020
- Photoshops AI neural filters can tweak age and expression with a few clicks - The Verge - October 21st, 2020
- Futurism Reinforces Its Next-Gen Business Commerce Platform With Advanced Machine Learning and Artificial Intelligence Capabilities - Yahoo Finance - October 15th, 2020
- Purebase Enhances Its Board of Advisors with An Expert on Machine Learning and Cheminformatics - GlobeNewswire - October 15th, 2020
- How to Beat Analysts and the Stock Market with Machine Learning - Knowledge@Wharton - October 15th, 2020
- Synopsys and SiMa.ai Collaborate to Bring Machine Learning Inference at Scale to the Embedded Edge - AiThority - October 15th, 2020
- Robotic Interviews, Machine Learning And the Future Of Workforce Recruitment - Entrepreneur - October 15th, 2020
- Top 8 Books on Machine Learning In Cybersecurity One Must Read - Analytics India Magazine - October 15th, 2020
- AI and Machine Learning Can Help Fintechs if We Focus on Practical Implementation and Move Away from Overhyped Narratives, Researcher Says - Crowdfund... - October 15th, 2020
- AI and Machine Learning Can Help FIs Avoid Riskbut They Have Risk of Their Own - PR Web - October 15th, 2020
- Machine learning for rowdy roadies: Cops and tech to rein in traffic offenders - Bangalore Mirror - October 15th, 2020
- Automated ATOs and cybersecurity - FCW.com - October 15th, 2020
- Experian partners with Standard Chartered to drive Financial Inclusion with Machine Learning, powering the next generation of Decisioning - Yahoo... - October 15th, 2020
- Machine Learning Answers: Facebook Stock Is Down 20% In A Month, What Are The Chances It'll Rebound? - Trefis - September 22nd, 2020
- Machine Learning in Education Market Incredible Possibilities, Growth Analysis and Forecast To 2025 - The Daily Chronicle - September 22nd, 2020
- Proximity matters: Using machine learning and geospatial analytics to reduce COVID-19 exposure risk - Healthcare IT News - September 22nd, 2020
- Global Machine Learning Market Tends To Show Steady Growth Post Pandemic With Regional Overview and Top Key Players - Verdant News - September 22nd, 2020
- PREDICTING THE OPTIMUM PATH - Port Strategy - September 22nd, 2020
- AI/ML Remains The Most In-Demand Tech Skill Post COVID - Analytics India Magazine - September 22nd, 2020
- Panalgo Brings the Power of Machine-Learning to the Healthcare Industry Via Its IHD Software - AiThority - September 15th, 2020
- Microchip Partners with Machine-Learning (ML) Software Leaders to Simplify AI-at-the-Edge Design Using its 32-Bit Microcontrollers (MCUs) - EE Journal - September 15th, 2020
- What is 'custom machine learning' and why is it important for programmatic optimisation? - The Drum - September 15th, 2020
- PODCAST: NVIDIA's Director of Data Science Talks Machine Learning for Airlines and Aerospace - Aviation Today - September 15th, 2020
- The Use of Machine Learning to Forecast Progression to Advanced AMD - DocWire News - September 15th, 2020
- How Can Machine Learning Help the Teaching Profession? - FE News - September 15th, 2020