An overview of how Sonatype have helped their customers with DevSecOps.

We work with companies of any size across multiple industries. Since 2000, 52% of Fortune 500 companies have been replaced by new emerging technologies. Companies are having to digitally transform today. They have to do this to remain competitive, increase revenue, and create new markets. This is not going to end. What is interesting is that as companies digitally transform, the way they build software has changed. 90% of the components that make up an application are open source, which means developers are going outside your organisation and bringing third party code in house.

This year alone we were seeing a 1.5 trillion request for open-source components. The demand is there. There are over 21,000 new releases a day. Developers are doing this because they need to digitally transform as quickly as possible. Open source enables them to do that. That is all good and why demand and supply are so high. However, there is always some risk. Also, in our research, we have seen that one in 10 downloads has a vulnerability. 40% of packages rely on code with a known vulnerability. Understanding what those components are and if they are vulnerable is important.

Bad actors used to wait until a vulnerability was exposed, and then try to attack it. Now they are targeting the projects themselves. They are injecting the malicious code into the project and waiting for it to cascade into supply chains.

The Equifax breach of 2017 was due to an open-source component, a java component. Many people didn’t patch it and upgrade to a more secure version. The bad actors were able to get access to thousands of private data records. We felt that after this we would have seen a change in behaviour after exposing the weakness in Struts. However, 14 months after the breach we still saw 18,000 organisations still downloading this vulnerable version. Do they not have the tools to see the weakness?

Automate faster than evil: you’d have about 45 days to find a weakness and act. Equifax had 3 days. That’s not a lot of time, and time is not on your side. You need the right tools and solutions right away.

We have seen a new wave of attacks, a 430% increase. Bad actors are befriending the committers then injecting malicious code. That malicious code is being downloaded in the software supply chains. We just identified two NPM components last week where this happened. As soon as the developer is bringing these components in, they are already malicious.

What we know is that manual processes to get your hands-on open source, they just don’t scale. If developers have to manually identify, it just doesn’t scale. This means they take on additional risk. Security teams are overloaded as they are researching vulnerabilities manually. This increases the likelihood of a breach. A lot of times now, companies are asking for their vendors to provide a software bill of materials, so they know what’s there. Even on the legal side, there are obligations. Legal teams have to review the licenses. All of this, these processes are manual, and you are slowing down development. It increases the likelihood of you being sued. We are looking for a solution so that applications can be delivered to market faster. They need to have a handle on what is being used.

These solutions have to integrate easily. It has to be fast and scalable. They need remediation guidance so that they can remediate issues just as fast. A big part is culture and change management. We offer a single platform to deliver more secure apps to market faster. For developers, we provide really precise intelligence so that they can identify the best components from the beginning. We provide in-depth remediation advice. The security side, we have a team of 65 researchers looking at 96 million components. That data is what enables security teams to roll out governance processes at an enterprise scale. You can continuously monitor your new apps for new risk. A component might be okay today, but tomorrow it might be vulnerable. There is a way that organisations get a handle on what they are using, and they know that they are delivering these applications to market quickly in a secure way.

Which are the main drivers within digital transformation for DevSecOps?

DevSecOps is also a very interesting thing which is not very alive in Western Europe. We are still getting our hands dirty mitigating breaches. We are developing new types of services to mitigate that better. The DevSecOps solutions – UEBA solutions where we look at user behaviour and do some machine learning solutions. We combine intelligence of different infrastructures, devices, and solutions, and corollate those logs. DevSecOps can be interpreted in several ways. The drivers for a digital transformation are triggering that. A lot of our customers are into digital transformation to get rid of the burden for them as a burden to the IT department. We take away that burden digitising business processes with security questions and related topics and vulnerabilities. A lot of companies in Europe are spending time in digital transformation from an IT structure point of view. If you automate business processes with software, that’s a different discussion.

The main driver is changing customer needs. Digital transformation is like a walk in a forest – it’s a forty steps journey, but you see only the next five. Sometimes, that whole kind of walking journey, things change.

We are involved in a lot of those digital transformation journeys. We see a lot of transformation tracks as well, where customers will get transforming the way they build and run their applications. The only way to do that in a good way is the DevOps way. This means moving it into DevSecOps. In my view, digital transformation is driving this. It’s competing forces. Development today needs to be built much faster. From an operations point of view, it must be stable. It needs to be secure. You build it and bring it into a stable situation. That time frame is too long for the way that businesses today are run, and applications are typically business-driven. It must go much faster. That is why DevSecOps is the only way. The three domains into one approach. You need all that from a security perspective. Security by design.

If a piece of code is developed in one way, if somebody with security insights comes after the development face, then you are too late. Somebody from a security insights point of view should be involved from the very early stage of development.

Time to market for the first release is important. Reaction time is important in terms of a security incident. The challenge is to keep it secure. You need to be able to correctly detect and respond to it by having the capability to fix within a short time frame. You need to be able to recreate and deploy the package quickly.

Equifax didn’t know what was in their application and suffered accordingly. That legacy currently, it’s really important to know what’s inside them, and if there are vulnerabilities to respond faster than the malicious actors around the world. The world is changing – gone are the days when high street or retail was getting in the car or walking. It’s now all through an app and is an application economy. We need to build security into that development process to ensure we are not pushing code into release cycles. We need to make sure it’s clean. It’s the legacy codes that are making us money. Infrastructure must be secured on the network. It’s the new vector of how those malicious actors are in control of people. BA had a breach of records. 22 lines of custom code. It’s easy, unfortunately. Those applications are a battlefront.

It would be nice if safe code repositories could be built. If it were possible to share signatures based on patterns sent throughout networks. So, if there was knowledge of code flaws and vulnerabilities, there isn’t much effort involved in how patterns will be sent throughout networks. If that is the next step, then the final step is to send out information to vendors about security products to be able to detect certain bit patterns in network traffic and to say okay, you’re running a piece of known vulnerable code. It’s a different twist, but theoretically possible.

Unfortunately, some of the ways that applications work mean that you can’t put a signature in because it’s part of the way the app is running. That is how we recognise malicious libraries because it’s unique. We use that as a high-level methodology for understanding if there are vulnerabilities in applications. There is enough noise in the world anyway. Things need to be fixed quickly with precision. The library has a hash, if we know that hash to be vulnerable, we stop people downloading it. We all realise that security is about doing many things and about depth. So, stopping the bad things coming into your organisation. Layered security.

There is a huge business opportunity. If you can add some signatures, you have a band-aid. Even for 80% of those vulnerabilities. Do nothing is not allowed anymore. We can’t sit on our hands.

How do you cope with the challenge to assign people to a DevSecOp position once you define it within your organisation?

What’s the background to become a DevSecOps person? Where does it come from? A security officer coming from development is good. It’s necessary to have someone to speak the right language. It comes at the cost of the security of code. You need to have someone strong in different types of development to tell people to do it better.

Everyone in IT needs to understand how applications are created. Perhaps not from a development background, but to understand what they do. There are a lot of people in IT who do not know how applications are built. This could be a long-term issue as an industry. Developers are poachers – you need your security people to understand what the poachers do.

You need to be sure that the reporting line makes this position very secure. Again, these guys are under huge pressure because it’s a pain in the neck. A good DevSecOps needs to point out where there are issues and layout rules everyone will be able to follow. We need to be able to develop and operate, so you cannot do Fort Knox with everything, and you have to relax on some aspects.

In Belgium, there were recently some upgrades in education for IT guys, so there are now more updated software engineering programmes. DevSecOps is a new area in the industry so you need some experience to kick in there to take up that role, and they are very hard to find on the job market. If you have operations and security guys telling software developers what to do, it’s a natural thing that they don’t speak the same language. You need some very skilled people there.

At some point, we’ll recruit people out of university and train them well, although this is not a good year for that. For older developers, we are more productive because we all started with a yellow book from IBM!

Does everyone have an internal policy for what open source develops are allowed to use?

Peer review. But we also have a lot of internal validation and checks to verify that the code quality is there from a coding point of view and that the product is working as expected and memory usage.

A lot of the companies are in a position where they are in a good position if they have people to review code.

Code review, you still need peer review to look at the code quality. This should be automated, but you can start scoring your code also.

We have seen people implement some successful project and there is a need for tooling to help with that automation because there is an enormous amount of information. You have got to give people tooling and automation to help them do it. Tooling and policy aren’t enough, you need both.

For those who were in DevOps mode before DevSecOps, what are the key integrations for the security component around that?

The first component you introduce – what is that?

Most people do a static analysis first. And then some code quality and then move onto RunTime or open source.

We proposed a simple SQL injection attack on a piece of code on internal stuff. That shows that the developer thinks somebody else will look at that for me. I’ve got a magic guy somewhere that is protecting me from the external world. That’s something we have to change in the culture. You are the last one securing the data. The guy responsible is the guy you see in the mirror. We are all trying to sell the product of the company, the same has to be done with security. We are not too much into DevOps yet, but what we saw as a problem is the shift of responsibility. More people, it’s harder to secure.

From a software security point of view, it’s important to look at possible entry points. Where can a bad actor get his or her hands on the easy entry points? Most of the times, you do that through a web front end. Look at that one as a high priority.

Also, look at the end-user as being the most critical point of entry into the system. This is the attack factor. If the bad guy is inside the organisation, this is a difficulty. If you have cost centre responsibility, you know that you don’t like your other cost centre owner to see your expenses, for instance.

Static analysis is one of the first steps. On top of that, a more theoretical view, pretty much like an infrastructure point of view, is a key thing as well on the development side.

Start with the risk analysis – what are the things that threaten you the most? What are the easiest entry points and the things that take the lowest effort? You need to start from there from a security point of view.

Applications are dealing with data, so key to look at what’s the type of data the application is working with. That’s the risk assessment as well. What is the data classification, is the application dealing with personally identifiable information? That’s the key thing to keep in mind.

Understand your risk so that you can manage it. You should know where to start. You had better pick your battles wisely and you can only start in a decent way doing that after doing a risk analysis.

In a DevSecOps mode, do you believe you still need an external security entity? What would that cover if so?

This question comes regularly, and the answer is yes. It’s key to have somebody else from outside watching on your shoulder. If you are developing code, you always suffer from tunnel vision. That’s the reason why there is something like ISO auditing. You don’t necessarily have the right people internally to do that. There should be some external control there, just to trigger the security by design part.

You always need to have an external challenger. If you do a review of your company internally, they can only review things they know about. An external reviewer will be wider. It will be a pain in the neck, but it’s good. It should be a long one, do not stress them in terms of time and money. You need to show there is no problem with the company and answer honestly.

If you are going for certification, there are two reasons why a company wants that, or it’s about improving your security posture.

For big customers, they have got IT departments that are bigger than security companies so when they ask questions about security it can take ages to answer and open a can of worms. We are on a journey to certification. Your security guy should work on security, not answering questions about security.

Mission-critical IT outsourcing: we work in high-security environments such as banks. We have a security expert and a security team. Most of our clients do audits for us. Some of these companies are way bigger than us.

In a world where it’s already hard to recruit people for DevOps teams, how you manage to add in the Sec component?

Train your people. That’s the answer. That is what we are asking our top management to do. Don’t search the market for the ‘white raven’ because you won’t find it. Check your staff to see if you have interesting people. Not all top managers are willing to hear that answer. Everyone is short-staffed and short on budget. You won’t find those people laying around in the streets.

We rely on tools to help people. We are in an industry that is a bit specific, but all industries are. It takes a while to understand how it works. People cannot be on your shoulder all day long. Mandatory code review for every step. You cannot replace everything with a machine, but tools can definitively help with the first boarding of a new employee in your company. They also know they can’t break stuff easily! If you don’t have checkpoints, you are always one button away from breaking everything.

It’s good to have a policy, it’s not good to rely on automation. No system prevents you from doing something wrong. If you have a policy as we speak about the open source if you say we are not using a piece of code in this company if a developer, then imports code they are fired. In some ways, the most important thing is what is the direction of the company and what is allowed or not. With COVID-19, mask or no mask, the spirit is ‘be careful’. Wear a mask. This is the same as the policy. Don’t be shy to voice it and act. Sometimes you need to take strong action.

You need to understand your risk; at Sonatype we run maiden central and it’s where all the developers get their open source. If organisations ever want to know what their developers are downloading from maiden central, it’s worthwhile investigating to find out more. Are they downloading lots of open sources and there’s lots of vulnerabilities? You might want to investigate it.

You can also start your risk analysis from the past and what you’ve seen. There are different angles of view that you can take here. Otherwise, you are driving blind and you don’t know where to start.