Daniel Moore explains why honesty is the best policy in data science and why the best models and most sophisticated techniques are not what matters most.

Daniel Moore is a lead data scientist at Liberty IT with a decade of experience in the analytics and data science field.

He has a background in biophysics and has worked in diverse fields such as cancer research, drug design, the mobility sector and insurance.

Moore told SiliconRepublic.com that his love for maths and physics were what started him on his STEM career. By the time it came to choosing what to study in university, it was between computer science and human biology. But the latter won out.

I know there was a turning point in my education and one person that shaped how I got here. In your final year of most university degrees, you undertake a research project. Essentially you work with a professor to explore your thoughts on how to solve a novel problem. Mine was Frank, he said.

As part of this, I worked with a new group of people who were all really friendly and devoted time to helping me learn. No question was too stupid. This was my first time programming. I remember how amazing it felt to solve mathematical problems automatically and with incredible speed. I didnt know such a field existed where I could work in biology and explore my love for technology and computers.

This experience made him change his career plans. He did a masters degree in computational biology, now known as bioinformatics, and went to complete a PhD in biophysics and drug design.

I was exposed to the real tangible impact of data science on humans, he said. How we can leverage data to screen for cancer automatically, to design and develop new drugs in a move toward a disease-free world preventing unnecessary suffering and death. I may have got here by chance, but I stayed because the impact and use case were just incredible.

The models you produce as a data scientist are useless if you cannot explain how they work DANIEL MOORE

When you finish your PhD or your degree, one question always gets asked, should I continue in academia or move to industry? However, a good work-life balance was important to me so I decided against studying medicine and chose to explore a career in industry.

It wasnt an obvious choice. I applied to be a lecturer before submitting my CV to a small local start-up company to work as something called a data scientist. It was the first time I heard that term, and with a fear of interviews, I almost didnt go. Looking back, Im so glad I took that opportunity for multiple reasons to experience a career in industry, within a start-up and as a data scientist.

For two years I was fortunate enough to work within the mobility sector on cutting-edge tech such as self-driving cars, biometric wearables and how we can use such sensors to improve driver safety. I got to work with some amazing tech and OEM car brands such as Bentley and Volvo, and experience that start-up culture even working in Silicon Valley for a brief time. However, start-up life is difficult, your input directly shapes the future of the company, and you feel a tremendous amount of responsibility.

I decided to make a change and joined Liberty IT as a senior data scientist. What drew me to this job was the diversity of projects and the impact they could have. Like many, I would hear the word insurance and practically fall asleep with boredom. However, this is certainly not the case here and the projects I get to work on would put this stereotype to shame.

I never realised just how much insurance impacts our everyday lives, making things right when something unfortunate happens. The projects I get to work on impact the everyday person. Moreover, I have always enjoyed learning. The sheer diversity of projects within computer vision, natural language processing, predictive modelling and MLOps provide enough learning for a lifetime.

No one wants to do the same thing day in, day out, and thats the beauty of data science in general its an emerging discipline and that means you need to constantly learn and adapt. It was one of the great aspects of research and now is part of my career as a data scientist. I love it.

As a data scientist, you tend to work on solving various problems by developing statistical models and learning from historical data. However, you often need to explain some insights and your methodology to solve a given problem in the manner that you did.

The difficulty is that you need to explain this to a non-technical individual, the stakeholder. Its all too easy to pull the wool over someones eyes and razzle-dazzle them with fancy terminology and buzzwords. Data science is a disciple where you need to be honest, show in a non-biased way your insights and recognise there are a plethora of methods to solve any given problem.

No matter how experienced you are, sometimes you will not be able to solve that problem, your approach might be incorrect, and youll often be the bearer of bad news: Sorry Mr Stakeholder, its unrealistic to train a model to be 100pc accurate.

Its incredibly challenging to be so honest with your insights. In fact, to progress your career in data science it often feels like you need to develop the best performing model and use the most sophisticated techniques. There have been times in my career when I have chosen to have the easier conversations, to be passive rather than debate why a problem might not be possible to solve with data science.

My advice is always be honest, always have those difficult conversations, push back when you need to and have the confidence to speak openly to your stakeholders regardless of their seniority. Ultimately, people will have more respect and trust for you if you do.

I would say there was a combination of people that influenced my career. My university professor Frank helped give me the confidence to ask the stupidest of questions and sparked my initial interest in research. Irina, my PhD supervisor, showed me what it takes to strive for perfection, and my first boss Gawain helped show me the reality of the commercial world.

On another side, some individuals have explained complex topics in ways that really helped me understand them. Josh Starmer for example, the individual behind StatQuest videos on YouTube. He helped me appreciate that everyone can understand the most complex topics.

The most obvious trait is the logical part of me loves the methodical nature of data science. If you are new to the field, it may feel like there are a million and one ways to analyse data or create a predictive model. However, there is method to the madness and as you gain experience you realise that most projects follow a similar lifecycle.

I think there are a few other areas which are well suited to a career in analytics. I am the type of person who needs to understand fully how something works. This inquisitive nature drives me to ask questions and get to the root of a problem. It helps to be inquisitive and understand how your solution will solve the actual problem.

Lastly, communication and collaboration are everything in data science. You can be the most technically proficient individual, however, the models you produce as a data scientist are useless if you cannot explain how they work and can then collaborate with your team to move your model into production, to consume real-world data and make actionable insights.

Data science is one of the most diverse fields imaginable. Frankly, its overwhelming. The norm for progressing to a more senior role is to deepen your knowledge in one of the three big areas of data science computer vision, NLP or predictive modelling before branching out into another area.

Try to ignore the feeling that you need to be knowledgeable across all of these areas. If you are reading a job description that is asking for knowledge across all three disciplines, they dont need a single data scientist, they need a full team of specialists.

At Liberty IT, one of the questions we ask all new starts is what area excites you the most in data science? Ultimately, if you have never worked on a computer vision project but are eager to explore this side of data science, they will pair you with an experienced individual to help you learn and progress in that area.

I feel like there is a real emphasis on training and development at Liberty IT that makes it much easier to progress your career. Everyone in our team is given the opportunity to take on board other responsibilities that help develop their career, such as interviewing, supervising more junior employees and teaching others within the data science community. Its honestly a really nice place to work.

Lastly, I think there is a misconception about career progression in the tech industry. Many think that a promotion simply means more money. As you progress in your career your responsibility changes. I think its worthwhile reflecting on what type of work you enjoy the most, the technical aspect or the management and strategy side.

Its my opinion that the most interesting careers are the ones that you find difficult to describe to your parents, what exactly it is you do, and where you define the language of your discipline.

Data science and the general field of analytics have been exactly this for me. Its changing rapidly and theres a lot to learn.

If youre starting your career in analytics or data science, imposter syndrome is something you need to be mindful of. Dont panic about not knowing everything. Ask questions and be open to learning new things.

10 things you need to know direct to your inbox every weekday. Sign up for theDaily Brief, Silicon Republics digest of essential sci-tech news.

Visit link:

'Communication and collaboration are everything in data science' - Siliconrepublic.com

What Google Docs did for word processing and Figma did for interface design, Hex hopes to do for data science. Which is to say, make data science and analytics a collaborative process, using a slick web-based user interface.

I spoke to Hex CEO Barry McCardel about how the data stack will change over the 2020s and what it means for data science notebooks, a common tool for data scientists.

We think of ourselves as building the frontend for the modern data stack, McCardel began. While thats the ultimate goal, practically speaking Hex fits into a category of tools known as the data science notebook. It competes with other such products, like Jupyter, Amazon SageMaker and Google Colab. Programming notebooks have actually existed since at least the late 1980s when Mathematica was launched. But in the modern cloud computing era, the open source Jupyter Notebook has been the flag-bearer for the data science notebook industry. Project Jupyter was launched in 2014, as a spinoff of IPython (Interactive Python).

I think at the core, notebooks are really just a very nice way to be able to do iterative analysis, McCardel told me. It basically breaks your code up into chunks, called cells. So you have re-runnable individual chunks. And those chunks can both run as a small unit, but also show you the output. So that was really one of the core innovations with the notebook format, where I can run a small bunch of the code and see what it does maybe Im seeing a chart, or a table, or just a result set, or whatever.

However, McCardel views Hex as more than just a notebook, or a Jupyter in the cloud (which is how he classifies some of Hexs competitors). What we see from our customers is that they arent just looking for a notebook solution, he said. Theyre looking for something that helps them share and helps them bring in people from different types of backgrounds and stakeholders.

Hex is trying to position itself as not just a tool for data scientists, but for data analysts and Business Intelligence (BI) roles both of which are less technical and more business-focused.

In Hex, users can use a no-code interface, or do queries in SQL or Python. Data scientists have traditionally used Python, but according to McCardel, that isnt necessarily the case with Hex users.

This idea that, oh, data scientists are high-end technical and use Python, and SQL is lesser, is not true at all! SQL is really good at a bunch of things. And youll talk to a lot of data scientists who use mostly SQL, and thats totally cool. It doesnt make them less of a data scientist. Theres a lot of really great things you can do [with SQL].

Clearly, though, Hex is eyeing a much broader market than competing tools that simply focus on data scientists. Its reminiscent of many of the popular low-code platforms that have emerged in the enterprise development market in recent years, most of which also target business users.

We want it to be easily accessible for people of all technicality levels, said McCardel, regarding the target users for Hex. People with just baseline data knowledge and curiosity, people who might be coming from spreadsheets or BI tools.

So are Python users adopting Hex too, I asked?

Yeah, its everything Jupyter can do and more. If youre coming from Python-based data science work, Hex will be familiar and powerful and give you a lot of new superpowers.

Another interesting aspect of Hex is that it has seemingly joined forces with two other modern data companies. Snowflake and Databricks, two leading data warehouses, were both investors in Hexs most recent funding round in March. So I asked McCardel, how would a data professional use Hex alongside one or both of those other tools?

So, Hex sits on top of those environments on top of Snowflake and on top of Databricks and helps customers make the most of the data that are in those environments. If you have brought all of your data into a data warehouse, including both of those two, often the next question is: so what now? How do I make this useful and impactful for the organization? Hex really seeks to answer that question.

Ultimately though, a lot hinges on Hexs ability to emulate the likes of Google Docs and Figma in becoming a user-friendly tool that also offers enough oomph to keep power users happy. In this case, the data scientists are the power users.

McCardel admires how Figma allows designers to share their work and bring other people into the process as stakeholders, and he wants to achieve that in Hex for data scientists. Not only that, he said, but Figma translates a lot of those people [stakeholders] into editors. The users become creators in the system, not just viewers.

So I get really excited when I see that type of thing happen in Hex, he continued. We see engineers and product managers and other people come in and actually become editors. Hex is not just for high-end data scientists.

Finally, we talked about where things are headed for the data stack. With tools like Snowflake, DataBricks, and perhaps now Hex, the tools available to enterprise users are increasingly sophisticated and easily accessible via the web. I was curious for McCardels thoughts on what comes next.

The last few years have been this revolution in the data world, on the integration story, he said. Im using Fivetran or Stitch or Airbyte to get my data from source into my warehouse. I now have my data in my warehouse, I can use DBT [data build tool] to transform it. I can use observability tools to monitor for quality. [] I can store this at any scale. I can run queries at any scale, you dont need to worry about provisioning servers anymore. I have been working in data for like 10 years, and its such a huge difference from 10 years ago. Its like the ability as an organization to just have all of my data in one place and have it integrated, clean and ready to go.

However, he added, the story is not over. He thinks the next part of the cloud data revolution will be in frontend tooling, or as he put it, what that new frontend for the data stack is.

Hex is clearly eyeing the data stack frontend, but well just have to wait and see whether it can capture that market as well as Figma did for interface design.

Feature image via Shutterstock.

See the original post here:

Hex Wants to Build the Frontend for the Modern Data Stack - thenewstack.io

Hays Martin Pardey explains what a data analytics professional does, what they can expect in their career and how to develop the necessary skills.

Data analytics has become a critical part of many businesses, but even within the analytics space, there are many roles available, including a data analyst, data engineer, data scientist and data manager. All these roles contribute to the goal of deriving meaningful insight from data.

Data analysts derive insight from data, while data engineers extract and manipulate data from systems and build data capability.

Data scientists, meanwhile, are able to build predictive models that help organisations make decisions based on potential future events, as well as driving automation and artificial intelligence systems. Lastly, data managers look after the data to ensure quality, governance and security.

Data roles used to be mainly centred around extracting simple management information and building reports for key stakeholders so that they could accurately analyse company performance. Now, as organisations become more data-centric, these roles have become complex.

There is a greater focus on ensuring that vast amounts of data can be analysed and accessed at any time across the organisation, as well as be used to build predictive models and power AI systems.

As a result, many organisations have now built internal data practices that employ many different types of data professional.

We have seen salaries for these roles increase steadily over the last few years as demand grows among organisations for better data.

An entry-level data analyst in permanent employment can expect to earn anywhere from 25,000 to 30,000 in the UK, while advanced data engineers and data scientists could even command six-figure salaries. Contract rates vary greatly depending on roles and skills.

Firstly, decide what area of data you want to work in. Are you highly analytical? Do you enjoy number-crunching and solving business problems? Or are you more technical, with a thirst for building data platforms and extracting the right data?

Where possible, try to develop your skills. See whether you can get any hands-on experience where you can actually apply the skills although, admittedly, this is much easier if you are already at an organisation or educational institution.

If youre in employment, look for opportunities to work with data within your current company. Ive heard of companies that are training employees in non-data roles to become data professionals in internal data academies.

For those in higher education, enquire about any projects you can work on. Many universities now have partnerships with major corporate organisations so that you can contribute to real life data projects.

While the data and analytics profession is strong, the need for data insights across all business levels means data skills have become critical even outside the technical sphere.

Research from Digital Realty revealed that more than one in five (21pc) IT leaders globally highlighted that the lack of internal talent to analyse data, and the lack of talent to build technical capacity (21pc), are among the greatest obstacles their organisations are facing when drawing insights from their data.

Luckily, there is a whole host of online courses out there, depending on what area of data you wish to pursue. For example, My Learning, Hays free online learning portal, has lessons in data science and analytics for those interested.

Data analysis forms part of a lot of roles these days. If youre in employment, you will likely have access to data and reporting systems in your current role. Make sure that you are fully trained on how to use them.

Seek out the head of data in your current company and talk to them about what it takes and whether they can provide you with any support. Enquire about learning resources that your employer already provides and whether there are any courses, classes or even modules that are directly relevant to what you want.

The value of being able to work with data effectively is high, so organisations are likely to see the benefit in supporting you in upskilling as they will benefit from your new skillset at the same time.

By Martin Pardey

Martin Pardey is a director for technology solutions at Hays UK with more than 20 years personal recruitment experience in the sector.

10 things you need to know direct to your inbox every weekday. Sign up for theDaily Brief, Silicon Republics digest of essential sci-tech news.

Continued here:

These are the roles available in data and analytics - Siliconrepublic.com

Last year, Aakash Mishra and Frank DAgostino learned an important distinction in data science while competing in the Citadel Boston Regional Datathon. Their team built a model to accurately predict Airbnb real estate prices in the southern United States, but failed to place. Another Harvard team won the competition by linking public trust in the government with increased mortality rates during the COVID-19 pandemic.

That loss taught the two incoming fourth-year students at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) that no matter how good a model is, data scientists should seek to use that model to address a concrete challenge.

You have to understand what is an important issue that needs an answer, and then you need to use your technical know-how to answer that question, said DAgostino, whos pursuing an A.B. in applied mathematics. Its the combination of not only the topic and how you come up with a problem, but also how you approach it in a way thats rigorous enough to be accepted.

Mishra and DAgostino took that lesson into their second attempt at the Boston Regional Datathon earlier this year, with much better results. Along with Viet Vu (A.B. 23, statistics) and MIT masters student Doron Hazan, the team proved a causal relationship between the FDA-approved drug glipizide and an increased rate of heart failure in diabetic patients. Their efforts netted them the $15,000 first prize and a trip to the World Championship Datathon later this year in New York.

What we did this time was very actionable, Mishra said. Knowing the effect of glipizide or another type of drug on diabetic patients, and being able to provide exact numbers linking it to heart failure, could help inform doctors.

The 2022 datathon was a one-day event in which teams were given data sets in the morning, then had six hours to complete their analyses and submit their methodology and reports to the judges. For the SEAS team, the data consisted of 70,000 anonymized patient records and medical histories.

When we originally got the data, it was really messy, said Mishra, whos pursuing an A.B. in computer science. There were a lot of missing rows, parts of the data that didnt make sense, and parts of the data set that werent filled out correctly. We had to figure out what we were going to keep, what we were going to throw away, and what we were going to infer.

Cleaning up the data and deciding what challenge to address took about three hours. The second half of the day consisted of running the models while team members worked on the background portions of the report, then analyzing the results in the final hour.

We all have different backgrounds, Mishra said. Frank is more into data science, Viet is into computational biology, and Im computer science. So, a lot of the things I did early on were making sure we had proper features to train our models on and developing the code for the models themselves. Frank tried to figure out the best way to approach creating these models and what we could infer from them, and Viet came up with the mathematical background for them and what results we could take away.

The Datathon forced the team to draw on numerous lessons from their coursework at SEAS. They derived their results using concepts such as hierarchical linear models and synthetics controls, both of which they learned in Harvard courses, as well as an overall approach to data analysis.

In class, were taught that you have to explore the data first, and then after that you choose some features, Mishra said. Then you want to figure out what the most interesting trends in the data are, and after that try to develop a model. Having that in mind helps with doing this quickly.

That approach paid off for Mishra, DAgostino and their teammates, and they'll need to stick to that formula if they want to capture the $100,000 prize at the world championships in New York City.

Just like in the Boston Regional Datathon, the key will be coming up with the right question to answer using the data.

In a lot of coursework, they kind of just give us the questions in a problem set and we solve them, DAgostino said. In the real world, you dont even know the question half the time. Once you have the question, then its easy to answer it.

Read more:

Harvard team wins Boston Regional Datathon for second straight year - Harvard School of Engineering and Applied Sciences