52% of the Fortune 500 companies since 2000 have been replaced. For example, Air B&B disrupting the hotel industry. Organisations, new companies are coming to market, and are displacing existing organisations around technology. Every company today is a software company to some perspective. They need to think about how to digitally transform. Even small businesses have to do this to stay relevant in the COVID-19 environment.

Digital transformation is filling the need for software applications. The way we build this has changed forever. We provide solutions for them to track open-source applications. The growth in open-source usage – demand and supply. We see that in any modern application, 90% is from open-source code. Only 10% is written from scratch. The demand continues to grow. 13 million developers requesting 1.5 trillion open-source components. There are over 21,000 new releases every day.

This helps to innovate quickly and get applications to market faster. It ages like milk, not like wine – there are vulnerabilities and risks with code brought into the supply change. About one in 10 of those downloads contain unknown vulnerabilities. 71% of organisations recognise a related breach due to open-source components. There is some risk. You have to be aware of what that is.

Growth in the next-gen software supply chain attacks. Looking for the projects themselves and injecting malicious code in the projects. When you take that next version, you could be downloading a malicious version immediately.

The reason why is we are all aware of what happened at Equifax. They were using a well-known java component called Struts 2 with a well-known vulnerability. It took some time to identify and remediate it, and in the meantime, they were hacked. What’s fascinating to us is that after the Equifax breach there were a huge number of drop-offs to request that version of Struts 2. It’s well-known and publicly disclosed, so they don’t know they’re using it. We are taking the next version of the component release. You have to automate faster than evil. It takes about 45 for a bad actor to find and exploit a weakness. For Equifax, it was three days. If they didn’t have automated tools or technology to support that, they were open to exploit which is exactly what is happening. What is changing in this dynamic is the next-gen supply chain attacks. We are seeing a 430% increase in those attacks, where someone befriends a project maintainer and helps out. They are provided with credentials and inject malicious code. Now the component is malicious from the start, and they can take advantage of that.

What we and our customers know are that manual processes do not scale. They can’t get a handle on their open-source usage. Developers spend too much time trying to remediate an open-source policy. They put their organisation at greater risk. Security teams are overloaded, and this slows down developers and increases the chance of a breach. In many cases, software technology companies are asked to provide a software bill to show exactly what is being used. Even on the legal side, there is a risk. If lawyers are manually reviewing licenses, they are slowing down development.

Automated governance processes do scale. A complete platform is required for open-source policies, not just looking at applications that are being actively developed. These tools have to integrate into existing DevOps pipelines and have to provide expert remediation guidance. They don’t want to have to think about how to remediate a risk, they want a solution. This can’t just be technology. It’s a whole people process and technology initiative. This is critical to your success.

The Nexus Platform: to help deliver secure apps to market faster. The ability to select the best components from the beginning. It helps them manage dependencies which are critical to remaining secure. A risk score to identify malicious behaviour. From an Ops perspective, a repository to help store stable builds. A component a developer fixes today might be okay, but in a week, go bad.

Sonatype works with customers to make sure they have the right solutions.

What would be your three key steps to implement or mature DevSecOps within an organisation?

The journey is a cultural change. Like many other projects in the corporate environment that’s a huge cultural shift and you need support from management. Quite often, security is quite a hard sell and you do have to become a marketeer in selling the benefits and why it supports the business.

Bank in Frankfurt: an interesting project, they realised the developers were using a lot of open sources and wanted to do something about it. Adopted the carrot and stick methodology. Set an edict and made it their supervisory board policy that you had to have a bill of materials for what was in your applications. It was mandatory. Cost of 50 euros to do it, but if you go to that project team it’s free. Every single project team listened. The interesting thing was getting the 50 euros signed off was the problem! There was a group of people that could help you as the development community – 600 developers, so not a small organisation. In that DevSecOps world, if you just give tooling to developers, they will never use it. You have to do something from the top and something from the bottom.

An audit is a lovely term in most organisations – most people are scared of it. Lots of people take holiday when there’s an audit going on! You have got to put some context around it. For a Tier 1 or Tier 0 application that is mission critical you have to enforce it. It’s about appropriate controls at appropriate stages. Balancing act – business has to carry on, but you have to have compliance. Embrace change.

The risk of catastrophic failure cannot be taken on board at all. It doesn’t matter how long it gets delayed; the business has to stop.

The Equifax was 143 million records and has cost them 2 billion dollars to put right including a 700 million dollar fine. That is a catastrophic failure, and it was solvable. If it’s an internal app, then yes, you can probably say that you don’t want to impose too rigorous a policy, but anything customer-facing you have to be strict.

From a development and design approach, whenever there is a new approach or paradigm and you don’t have control over it, you are inviting risk and trouble. There needs to be some level of governance and aligned governance in terms of security and technology. These are the standards you need to conform to before you bring it into the operational area. If they are not controlled and have risk management around them, there are issues. You need that visibility. It’s important to set up that framework before anything major as a cultural shift. There’s a rush to get into those areas where they are reaping immediate benefits. You need to take stock and control it in a way that’s meaningful and beneficial.

Understand your environment. It’s a responsibility of teams in the whole lifecycle to have some level and control of what’s in that environment and what goes in and out. You need that level of control, but it needs to be managed. An enterprise architect sees gaps and risks where people say “no I don’t have time”. You need to get everybody on board and have accountability and ownership. There needs to be an incentive to do something risk-free and beneficial for the organisation.

Does everyone have a policy around open-source usage?

There is no consistent open-source policy meaningful across an organisation. Groups say they will adhere to policies, but there is not sufficient level of authority. Even if you can deliver the application, there needs to be a sandbox.

Some of the challenges are visibility without having something plugged in or looking at what’s being developed. At that point, you can’t define a policy that well without knowing what’s there. You can’t manage what you don’t know.

If it’s brought into the environment, you have control and visibility. There needs to be a gateway, how you can bring in and take out of the environment. Anything which has become a very fashionable approach, with everyone else doing it, has a big risk particularly where those environments are returning but it is not long-term sustainable. You need a mixture, and governance and visibility. A portfolio of those people, what they are doing and what they are working on. High-level GANT charts. You can have a view of what is being done and you can control it.

There are strict criteria, and if that causes a delay then so be it. It can be frustrating, but it’s needed. It can lead to escalations, and so policies have to be changed. As long as you have a good open dialogue you are okay. Individual developers get very frustrated. Do it quicker, make it look prettier.

The people that have been in development for longer will go along with policies because they understand what it will do to the organisation.

The developers have set paradigms to develop to. They change quite a lot. With legacy products, it’s relatively easy. With newer products, we’ve had to change both paradigms. This has only led to a culture change in one small part of the organisation. It creates issues with feature cycle times as they get out of sync with each other. Teams blame each other for missing deadlines.

Developers like throwing their toys out of the pram.

A lot of teams are not co-ordinated. Some teams are good at delivering some functionality. If any companies globally adopted DevOps. How easy is it to scale? There are considerable issues in terms of control and planning for the developers and development team and what they need to do. But that all comes back to management.

Teams across several different countries working on two different projects have a language and a cultural barrier. It wasn’t smooth to start with and they had a lot of lessons to learn before they achieved a good output. It was a challenge. A lot was learned. This was DevOps kicking off as a capability. Security was on the back foot. We were receiving things from different global teams at the last minute. It was frequently getting overridden. We had to change our approach and go higher up to make sure we could stop things and then we had to think about how we could do things differently. Speaking to the teams individually made a difference. It was quite a challenging time managing it globally. That was the biggest challenge, the different teams had different meanings. An issue that was important somewhere was not a problem elsewhere. Just understanding a single page of what a problem was taking a long time.

Is the solution for the development teams to fix things early in the process? Digital people don’t get risk. From a security point of view, it’s all about the risk and the business risk. We don’t talk to our teams too much about risk. A lot of the automated checking we want them to do as part of the process it has to have a certain level of assurance before it will go to the next level of the process. We have to demonstrate to the management what the risk is to the rest of the organisation.

You have to educate people. Quite often developers come in who have worked in less secure environments. An introductory course for them. They don’t get to log on to tools without going through a cultural learning process. We have a lot of education on our policies and behaviours. Global DevOps, we have been rolling that out for the past three or four years, but there is no such thing as this is DevOps or this is not DevOps. It’s different things to different organisations. We are still trying to find the right balance between wanting to deliver fast and is causing us to ask questions on our policies (some of which were written a long time ago). You can’t become a DevOps rolling out code unless financed. That’s no easy thing. It’s an interesting journey and that’s a theme.

Developers being creative and providing the functionality for the business, but within constraints – providing guardrails for developers is the safe way to create applications. The risk gets signed off.

Developers do pride themselves that the system they have just created is suiting the company needs. Risk is not front of mind. It’s ‘congratulations, nothing happened’.

We have a challenge with developers. A lot of the time they are engaged by an individual part of the business to solve a problem. They stream off work without looking at the bigger framework. Frequently people put together an app using a service and there’s a bit of corporate infrastructure built on external free solutions. There is a whack-a-mole exercise keeping developers doing things the enterprise way. When trying to make something fool proof, never underestimate the ingenuity of fools!

Often the first time you know something is there is when someone phones up for support. It’s a big challenge. A lot of the challenge with modern overlay networks, you don’t know they’re there. All you can do is monitor the network for tunnels and look for who is doing what. It’s a problem of scale.

In terms of the development teams and how they do what they need to do, one thing was to appoint a very strong technical architect/developer to several teams whose responsibility was to provide the link and assurance to the enterprise architecture and lead the development team that was aligned to the global architecture and design principles, and not to interfere with the approach. It helps with the cultural shift, but at the same time having the oversight. That’s a key area. There are a lot of gaps between the different teams.

Putting architects into each development stream is interesting. Is it possible in a large organisation? The cost is enormous.

Put a very strong lead architect to be the leader in that area and guide developers. Either as the leader or the support. It worked well in some areas, but not in others. It should not be someone detached from the people, and they should speak the language of developers and have the skillset to be able to see what’s beneficial for the developers and the business, and also what is aligned to the enterprise. The type of governance you put in could be light, with a visibility of what’s happening. A very minute code problem, you’re not going to bring to senior management. We found it beneficial to have that level of engagement. It’s not a case of having 100 technical architects but should scale to the organisation.

This is a key area. From a security point of view, they are being trained and incentivised to be our security champions within teams. We have been focusing on security developers to be team leaders for that role. Those senior individuals who can influence younger developers.

There needs to be some level of control – not the wild west. And some incentivisation. You need a champion at that level who will also represent a team. As long as they represent them, it fosters a feeling that we are in this together rather than being competitive over code or process.

Can the incentivisation just be done with training or does it have to be a cultural change? Training helps, but it has to have a feeling of what the culture is and be a part of the whole team that has evolved within that DevOps environment and that it is their responsibility. There are some core principles. It seems to be more acceptable and the whole ethos of secure by default is being driven at the moment. That feel-good feeling of releasing something good. We did have security gateways for apps – the teams themselves are seeing their score which is building a bit of healthy competition. It has spurred something that seems to be working.

Microsoft did a Wall of Shame rather than a Hall of Fame – positive seems to work better.

There is no social interaction of the teams coming together and sharing results at present, although the productivity hasn’t gone down. The fun aspect has suffered. There is too much isolation. Some of these people are young and don’t have great facilities at home, they are in rented properties. It has become a great people management thing. It’s down to line managers to monitor this.

All the line managers can report upward – holistic. Doesn’t make up for the personal touch.

Changing culture – if we are in this for a very long time and you are trying to bring people into the organisation, it can be quite tricky. This can become quite challenging.

Can anyone recommend any resources that can be used to train up the existing workforce in DevSecOps?

It’s a ‘depends’ question. It depends on the culture of the organisation. It’s up to you as an organisation how you deploy what DevSecOps is. That needs to be nurtured internally.

Some of this training is outsourced. It doesn’t make up for embedding in our processes, but it’s a good thing.

What should organisations think about when selecting a solution to manage open-source risk?

In terms of supporting that, what we wanted to do was shift left as much as possible and reach a point where we are minimising efforts at the earliest possible stage of development and developers are solving issues and eliminating risk.

Rather than security being the blocker at the end, we moved it further down the pipeline.

Visibility was an issue, so at the same point as it’s got to be usable. We had to show to the business there was an issue. Whatever tooling we went for had to be very ready with components of our pipeline. It had to be quite API friendly and connecting to modern standards DevOps tooling.

Automation was another key area that we wanted to focus on. Every other day we are releasing something, so automation is key.

Not only the ability to spot, but to prevent. A key requirement.

On visibility: Sonatype runs Maven Central. There are 5.7 million libraries there. It has got enormous. We don’t publicly talk about it, but we keep an audit log of who connects to us. If anyone wants to know who connects to us and downloads what, we can provide a report privately. You would be amazed at what your developers are downloading. Everything in technology is about priorities. If they aren’t downloading open source with vulnerabilities, don’t worry about it.