Devops

Si vous voulez déployer vos applications et services dans un cloud public, vers qui allez-vous vous tourner? Très probablement vers l’un des 3 acteurs majeurs au niveau planétaire. Votre choix se fera sûrement pour des raisons politiques plus que techniques ou financières. Je dédierais sûrement un article à ces choix ultérieurement. Le fait est que pour déployer une nouvelle application dans le cloud, le choix est finalement assez simple. En premier vous avez la solution déjà indiquée : un des mastodontes américains. Problème, vous ne voulez pas forcément donner vos données, vos applications et votre argent à une multinationale, quelle que soit sa position vis-à-vis des questions éthiques et légales. Et dans un climat de défiance envers la globalisation, et une tendance à la relocalisation, il semble un peu hypocrite de s’appuyer sur eux. ...

This is a second post in what might become a series regarding DevOps, and DevOps transformation. Unfortunately for you, I speak my mind when I feel like it, so there is no master plan behind my ranting. First post was about choosing a strategy when transforming a managed services team towards DevOps. This second step may not come necessarily after that first step :) You get to choose. In a way, I have the feeling that DevOps is close to what ITIL is : a toolbox where you can select which parts you would like to apply, and the way to implement them. So new topic : how to target or build the team which will start the DevOps initiative in your organization? There are several indicators that you can use to make this selection. None is an absolute, once again, these are pointers to which team would better succeed in applying DevOps principles. There is no sure shot, no magic finger method :) The obvious criteria is to pick a team, of future team members that are already aware of DevOps, and willing to make the move. Of course these would be the people that will help you succeed in the first move, and show the rest of the world that it can be done, and bring very positive outcomes. The second flag that should be raised is tied to one of the main outcome of DevOps : software delivery. If you have a team that is already struggling to get a reliable software delivery in place, and is willing to put the effort into building the organization and tools to get there, then you have a probable winner :) On the opposite, if you try to convince a team that they need to move to DevOps when they are convinced their software is delivered efficiently and reliably, you’re going to hit a wall. Basically it would come down to convincing them that they are doing a poor job and are not even aware of that. Maybe not the right target for your first DevOps project. The last one brings us to my favorite topic, outside of tech stuff : business outcome! Of course, the aim of a transformation within your organization should always be mindful of the business outcome. Imagine you pick a team that is working on a neglected piece of software that no one cares about, and of which customers are just happy of the current situation. I do not think you would get the visibility and traction, not to mention willingness if you try to initiate your DevOps transformation in there. On the other hand, if you manage to get a team working on a new flagship product, that gets the attention of the CxOs, and has a need to show modern and reliable software delivery… you have a winner! Once you’ve chosen your dream team, and start working, remember that DevOps, as always, is about people, not just tools. The key thing to take away stems from common sense, and has been given by someone who did try and implement multiple Agile/DevOps organizations: “if you start with bad people and give them a smart way to work together, they won’t become smart by themselves”. Alright, I must admit the original quote was much harsher, but I cannot write it here :) PS : I tend to keep these article short, and do not dive into each topic. This is done on purpose, not just by laziness. If you need to go deeper in each topic, just ask, or get a good read :) ...

I know, DevOps is still a buzzword. IT is often used to rebrand an old team or practice, and give more sexiness to a marketing slideset. I have seen it used to describe what is essentially an automation team within an Ops team. Good start, but missing the point. Anyway, having been a major actor in a devops transformation, and witness to many, I wanted to give some advice to anyone out there patient enough to listen to me. I think most of what I am going to say will seem trivial, but it would help a few of us to put some words on what is happening to good ol’ outsourcing. I am talking right now about DevOps transformation, but specifically when it happens to a Managed Services team, as in an outsourcing company, or an MSP. This is a difficult spot, as such teams should be able to provide support for DevOps organizations form their customers, and even help them. I will skip the contract and SLA part, as it is a very tricky subject, at least for now. I have seen two ways of approach the subject. First is full ahead transformation, no quarters, no mercy. The second is building a new team dedicated to DevOps, in parallel of the existing one. I have only experienced the second first hand. It was an opportunity as we were bringing a new set of skills and customers into an existing managed services org. I chose to break from the past, and build a new set of tools and processes almost outside of the existing system. This creates the possibility to create and prove the viability of these new ways of working and interacting with your customers. The difficult part is where you have to merge with the existing tools and processes. For example, we had to start with the monitoring tools we had, as we could not ask our 24*7 monitoring team to have multiple dashboards and tools. The fact that we were successful with the other tools and habits we developped allowed us to push for a new monitoring solution a few months after the initial move. And we kept the momentum after that :) The other way I have witnessed, to transform the whole team at once, is challenging and cannot be carried by anyone on the team. I would advise that you start this path if you have a sufficient executive weight, or support from the executive team, because it will be a disruptive path. This is good, but the cost might be high, both in terms of people inside the team being disgruntled and maybe leaving, and in terms of your existing customers (and prospects) who might not understand what you are undetaking. The uptake is that you avoid having two different teams working with different mindsets and toolsets. Obviously I would recommend reading the Devops Handbook, and the Phoenix Project, to get a basic understanding of what you are getting yourself into :) Next up, I will try and help you target which team and/or customer would be a best fit to start your devops journey, stay tuned! Je sais, tout le monde parle de DevOps, à tort et à travers, mais parfois à raison. C’est un terme souvent utilisé pour redorer une image d’équipe ou de practice, et donner une touche sexy sur un jeu de diapositives. Je le vois souvent utilisé pour décrire ce qui n’est essentiellement que de l’automatisation dans une équipe Ops/infrastructure. C’est un bon début, mais ce n’est qu’un début, pas le voyage complet. Néanmoins, ayant été acteur, parfois majeur, et témoin de plusieurs de ces transformations vers le DevOps, je voulais donner mon avis, à qui veut bien l’entendre. Ce que je vais dire va sûrement paraître trivial à certains d’entre vous, mais pourrait donner quelques indices à ceux qui se demandent où va le métier d’infogéreur. Je parle de transformation DevOps, spécifiquement lorsque cette transformation touche une équipe de Managed Services/infogérance. C’est un sujet difficile dans ce contexte, car ces équipes doivent fournir un support moderne à leur client, qui sont parfois déjà organisés en mode DevOps, et même les aider à vivre cette transformation. Je ne vais pas entrer dans les sujets contractuels et SLA, c’est un point très sensible et complexe. En tout cas, pas aujourd’hui. J’ai pu observer deux façons d’approcher ce sujet. La première est une transformation complète et radicale de l’existant, pas de quartiers. La seconde est de construire une nouvelle équipe dédiée DevOps, en parallèle de l’existant. J’ai seulement vécu la seconde méthode personnellement. Il s’agissait d’une opportunité car nous amenions tout un panel de nouvelles technologies et de clients dans une équipe d’infogérance existante. J’ai pu faire le choix de repartir d’une feuille blanche et de bâtir un ensemble d’outils et de méthodes pour desservir des nouveaux clients dans un ...

Nope, my new position is not dead yet, thank you very much. What I mean by this title is usually a meeting in any IT service, after a major incident has been resolved, where all the team members who have worked on the incident gather and discuss what went wrong, and how to improve tools and processes to do better next time. I specify blameless, as it is a very good practice to avoid finger pointing, generally and particularly in these meetings. If you want people to be honest and share their best insights, you have to keep in mind that these post-mortems have to cultivate an atmosphere of trust. The aim is really to find out how the events have unfolded, which information had been gathered, what went wrong, what steps were smart, which ones did not work properly etc. For more information about that, I recommend some DevOps sessions and talks, like this one from @Jasonhand from VictorOps : It’s Not Your Fault - Blameless Post-mortems ...

This is a difficult topic. I have to admit I am still not completely comfortable with all the concepts and functions. However, the thinking is amazingly interesting, and I will take some time to ingest everything. First things first, I will use this post to summarize what I have learned so far. How did I end up reading that kind of work, you ask? Weeeellll, that’s easy :) Brendan Burns, in one of Ignite ‘17 sessions, used the comparison “autonomous vs autonomic” to discuss Kubernetes. This got me thinking on the actual comparison, and aided with our trusted friend, Google, I found a NASA paper about that (https://www.researchgate.net/publication/265111077 _Autonomous_and_Autonomic_Systems_with_Applications_to_NASA_Intelligent_Spacecraft_Operations_and_Exploration_Systems) I started to read it, but it was a bit obscure for me, and scientific English, applied to space research, was a bit too hard for an introduction to that topic of autonomic systems. Some more research, helped by me beloved wife, led to a research thesis, in French, by Rémi Sharrock (https://www.linkedin.com/in/tichadok/). The thesis is available right there : https://tel.archives-ouvertes.fr/tel-00578735/document. This one relates to the same topic, but applied to distributed software and infrastructure, which ends up being way more familiar to me :) The point where I am right now is just over getting the definitions and concepts right. I will try to describe what I understand here about automated, autonomous and autonomic systems. There is some progression from the first to the second, and from the second to the third concept. Let’s start with automated. An automated system is just like an automaton in the old world : something that will execute a series of commands, on the order of a human (or another system). For example, you have a thermostat at home that send the temperature from inside and outside your home to the heater controller. ...

Yes, DevOps is the new black. I might not be the first to use the phrase, but it’s so obviously true. I am currently working on building some kind of offer around DevOps, so you’ll probably see more posts on the topic. But two things struck me recently and I decided I would make a post out of those. Both items are related to the people side of DevOps. The first is the importance of the people involved in your DevOps transformation or organization. The second, corollary to the first, is the recruitment of these people. People matter People are important, that’s obvious. However a customer experience has lately surfaced the importance for a successful DevOps transformation. I may not be able to go into many details but the broad outline is quite simple. The organization is a software team, within a large company. It delivers its own product, to be used by other business units. It has decided to run its own operations. Perfect candidate for DevOps, right? Using a custom approach, based on industry standards, an Agile/DevOps organization is designed and implemented. Fast forward one year. The transformation is quite successful, the stability and quality of the product have improved. The only thing that prevented the team to be outstanding seems to come from the people. Don’t misunderstand me, I am not judging this team and its members. But for an Agile/DevOps transformation to be successful you need the right people, with the right mindset. And not everyone fits the bill. Like some people are more comfortable in an open-space and some prefer a closed-off office. It has been the same with Agile practices, which could not apply to every situation and team. We need to pay extra attention to the people we include in these transformation projects, if we want them to succeed. Recruitment is crucial As a follow-up of the above assessment, recruitment of people for your team is important. Yes, I know, it has always been important. However, take a second look at the title of this post. Done? Alright. Now have a look at the job offers in IT. See any pattern? DevOps is written everywhere. It is somehow justified, as DevOps encompasses many modern practices that we have or would like to implement. Take automation, continuous delivery, continuous deployment, testing, QA etc. The issue is that not every job offer is for a DevOps team or project. Most of the offers are for traditional sysadmins or developers with a hint of DevOps. Which is a good trend, but not a good fit for a full devops profile. So, people matter in DevOps environments, so take care of your profile :) Note : this post was inspired by a LinkedIN post that I cannot find, in french, regarding the abusive use of DevOps in job postings in France. If anyone can find it, I’d love to thank and credit its author!

I already posted about this event a few weeks ago, with a focus around my experience and the organization : https://cloudinthealps.mandin.net/2017/11/03/velocity-london-2017/ This time, I would like to share a short summary of what I have learned during these 4 days. The first two days were a Kubernetes training, so nothing very specific here. I learnt a lot about Kubernetes, which is to be expected :) During the two conference days, I attended the keynotes, and several sessions. The keynotes are difficult to sum up, as they were very different, and each was a succession of short talks. I attended several large-scale conferences in the past, and that was the first time that I felt that the speakers were really on the edge of research and technology. They were not specifically here to sell us their new product, but to share where their work was headed, what the outcomes could be etc. They broached subjects ranging from bio-software to chaos engineering, from blockchain to edge computing. Some talks were really oriented toward IT & DevOps, and some were bringing a completely different view on our world. Overall, it felt energizing to hear some many brilliant minds talk about what is mostly our future! The sessions were a bit more down to earth and provided with data, content and feedbacks that would bring us some changes back home. I was surprised to have most sessions concentrate on general information and feedback, and not so much on specific tools and solutions. I expected more sessions from the toolchains for DevOps (Chef, Puppet, Gitlab, Sensu and so on). Actually, even when the session were presented by these software companies (Datadog, Yahoo, Bitly, Puppet, PagerDuty) they never sold their product. However they used their experience and data to provide very useful insights and feedbacks. What I brought back could be split into two categories : short term improvements/decisions that could be implemented as soon as I got back (which I did partly), and trends that would have to be thought about and analyzed, and then maybe crafted into a new offer or approach. In the first category : • Blameless post-mortems. A lot of data analyzed, with one takeout for us : keep the story focused and short. If you do not have anything to add apart from the basic timeline… maybe you’re not the right team to handle the postmortem :) • Solving overmonitoring and alert fatigue. This talk was a gamechanger for me. What Kishore Jalleda (https://twitter.com/KishoreJalleda) stated was this : you may stop monitoring applications and services that are not respectful. For example, if you get more than X alerts everyday from an application, you may go to the owner of the application and say “as you are generating too much noise, we will disable monitoring for a moment, until the situation comes back to something that is manageable by the 24*7 team” Of course you have to help the product team get back on track and identify what is monitoring and what is alerting (https://cloudinthealps.mandin.net/2017/05/12/monitoring-and-alerting/). And you need top management support before you go and apply that :) • On the same topic, a session about monitoring containers came down to the same issue : how do you monitor the health of your application? Track the data :) The second group covered mostly higher level topics, on how to organize your teams and company for successful DevOps transformation. I noted an ever spreading use of the term “SRE”, which I would qualify of misused most of the time. At least SRE seems now to qualify any team/engineer in charge of running your infrastructure. Another trend, in terms of organization, was the model based on this famous SRE team, to provide tooling and best practices for each DevOps/Feature/Product team. I’ll probably post at length sometime later.

This October I had the opportunity to go at the Velovity conference in London (https://conferences.oreilly.com/velocity/vl-eu). The exact title of the conference is “Build and maintain complex distributed systems”. That’s an ambitious subject. The event had been suggested by a customer who went to one of the US editions and found that it was a brilliant event, both in terms of DevOps subjects covered and in terms the attendees & networking. So here I am, back in London, for 4 days of DevOps and cloud talks. [logo conf] I have started the conference with a special 2-days training on Kubernetes, by Sebastien Goasgen (@sebgoa) [logo k8s] The training was really intense, as Sebastien described many standard objects and tools of the platform, as well as a few custom options that we can use. We played with Minikube on our laptops, which is a really great way to have the experience of a Kubernetes cluster, in a small box. It was really packed, and we had to rush to keep up with Sebastien’s tests and labs, even with his Github repo containing most of the scripts and K8s manifests. I came out of those days a bit tired by all the things I had learned and tested, and a long list of new functions and tools to try, new ideas to explore etc, it was immensely fun, thank Sebastien! The conference itself was rather overwhelming, and a big surprise for me. I am used to large conferences like Vmworld or Tech-Ed where you get the good word for the year to come from an editor and its ecosystem. Most of the sessions in those are obviously diving into the products and how to use them. At Velocity, almost all the keynote speakers were somehow working in research, or such bleeding edge domain that it might not even exist yet. I loved being presented with what might be happening, and by people who are scientists at heart, not just marketing infused with a light touch of technology. Moreover the sessions themselves were mostly feedback on the speakers own experience on a specific domain/issue/subject. Usually they are not into a particular tool or software suite, but rather on how to make things work with DevOps and large distributed systems. Overall, I I really enjoyed this conference, because it very well organized, and small enough to be a human experience. As we have four days with the same group of around 400 people (227 to this day according to the attendee directory), in a rather small area, you cross path often with the same people, and it makes it easy to start conversations. Also they came up with a lot of ribbons that you can attach to your badge, to let other know what you are here for : [photo des ribbons] I was used to much larger conferences, where I always found the networking a bit difficult, if you did not know a few people beforehand. In Velocity’s case, it is so easy, you have only a handful of attendees and speakers, and you can meet everyone informally, during lunch breaks or just by asking. It came as a surprise for me to able just to chat with some of the speakers that have impressed me, like [juergen & Gremlin] just by going and sit with them at a lunch table.

As I was getting ready for Velocity conference (https://conferences.oreilly.com/velocity/vl-eu) and the Kubernetes training by Sebastien Goasguen, I happened to be captured by a spiral of testing. First, I needed to have a K8s cluster running for said training. Sebastien suggested Minikube, which is a nifty way of having a local K8s cluster on your workstation and play with it. As it was too easy, I went through my K8s the hard way (http://cloudinthealps.mandin.net/2017/09/14/kubernetes-the-hard-way-revival/) on Azure again to be able to work on the real stuff, and use kubectl from my Linux env (embedded in windows 10). And I realized that I might have internet issues during the conference and would be happy to have Minikube running. So back to square one and to setting up minikube and kubectl properly on Windows. I tried the easy way, which was to download Minikube for Windows and run it. It obviously failed, and I could not find out why. After some try and fails, I just updated Virtualbox, which I was already using for personnal stuff. I just had then to rest the minkube setup that I had, with “minikube delete” and then start fresh : “minikube start” and voilà, I had a brand new Minikube+kubectl setup fully on Windows 10 (and a backup on Linux and Azure). But as I was working on that, I stumbled on a news there about Azure Stack (https://social.msdn.microsoft.com/Forums/azure/en-US/131985bd-bc56-4c35-bde8-640ac7a44299/microsoft-azurestack-development-kit-201709283-now-available-released-10102017?forum=AzureStack) and specifically the AS SDK, which allows for a one node setup of Azure Stack. This tickled my curiosity gene. A quick Google to find if there was any tutorials or advice on running nested Azure Stack on Azure, and here am I, setting up just that. Keep in mind that the required VM (E16s-V3) is just above 1€/hour, which means 800€ monthly, so do not forget the auto-shutdown if you need to control your costs :) The guide I followed is there : https://azurestack.blog/2017/07/deploy-azure-stack-development-kit-on-an-azure-vm/ I did almost everything using the Azure portal, maybe it might be useful to build a script to do that more quickly. Note that the email with the download link takes some time to be sent, so you might start with that. Or you can use the direct link : https://aka.ms/azurestackdevkitdownloader And this first test did not work out the way I expected. There were many differences between the article, the official doc, and what I encountered while deploying. Back again to first step, I redeployed the VM, redownloaded the SDK, and started from scratch, following the official doc (https://docs.microsoft.com/en-us/azure/azure-stack/azure-stackdeploy), I just added the tweak to skip the Physical host check, in order for the installation to continue even though it was running on a VM. After a few hours, Voilà I had a fully running Azure Stack, within an Azure VM! Now I just have to read the manual and play with it. This’ll be the subject of a future post, keep checking!

Today is another rant day, or, to put it politely a clarification that needs to be made. As you probably know by now, I’m an infra/Ops guy. So monitoring has always been our core interest and tooling. There are many tools out there, some dating back to pre-cloud era, some brand new and cloud oriented, some focused on the application, some on the infrastructure. And with some tuning, you can always find the right one for you. But beware of a fundamental misunderstanding, that is very common : monitoring is not alerting, and vice-versa. Let me explain a bit. Monitoring is the action of gathering some information about the value of a probe. This probe can measure anything, from CPU load to an application return code. Monitoring will then store this data and give you the ability to graph/query/display/export that. Alerting is one of the possible actions taken when a probe reaches a defined value. The alert can be an email sent to your Ops team when a certain CPU reaches 80%, or it could be a notification on your IPhone when your spouse get within 50m of your home. Of course, most tools have both abilities, but that does not mean that you need to mix them and setup alerting for any probe that you have setup. ...